Test and Learn Culture: How to Get Your Leadership Excited About Experimentation

What do Amazon and Facebook have in common? A culture steeped in the test and learn philosophy.

At the heart of all authentic conversion rate optimization strategies, you’ll find one crucial methodology: “test and learn.” Not only is a test and learn culture the foundation of CRO, it’s an attitude you’ll find in the most successful online ventures.

You’re in business to help people. You do that by converting them from the status of interested shoppers to becoming satisfied buyers of your products or services and then serving them so well that they can’t help but pass the word about you on to others.

By developing an internal culture of experimentation, you can speed up the process of finding out what your best prospects want most, and how you can deliver on those desires.

The bad news is that getting to the truth of what your prospects will actually buy can be a real challenge. The good news is that we’re going to show you the most direct path to overcoming that challenge.

In this article, we’ll highlight the benefits of a test and learn organizational framework, provide examples of what can happen when you invest in experimentation, and show you how you can begin building a test and learn culture in your organization.

At the heart of all genuine conversion rate optimization strategies, you’ll find one crucial attitude—test and learn. Click To Tweet

What is a test and learn culture?

“Test and learn” describes a conversion rate optimization process where all changes and investments are considered hypotheses to be tested. Rather than make decisions based entirely on past experience and gut feelings, test and learn organizations set up experiments, observe the results of those experiments, and make decisions based on that data.

Companies that adopt the test and learn methodology choose to invest resources based on decisions backed by proven results rather than on opinion. Continued iterations of test and learn protocols enable an organization to become smarter, and often uncannily innovative.

Why is it important to make test and learn the center of your CRO strategy?

Conversion rate optimization is a means to steady improvement. Those incremental gains, over time, can lead to big leaps in ROI. They tend to be more sustainable than dramatic improvements that may make a flash then disappear.

For instance, you can realize a significant increase in sales with a Black Friday promotion, but the boost you achieve won’t last long before revenue from sales drops back to normal.

The additional business you gain from the right application of CRO principles, however, will stick with you (and continue to build) over the long run. These principles will continue to pay back into your revenues day after day, all year long.

Since the test and learn philosophy is at the heart of CRO, ecommerce VPs and managers who build a culture of experimentation within their organizations will realize many benefits. A test and learn approach:

  • Provides freedom to fail. When failure is expected and accepted instead of condemned, your ecommerce development team will ultimately get to the right solution faster.
  • Lowers the opportunity cost and minimizes major investments in changes that don’t actually work. It’s better to make a series of small tested changes over time to increase the effectiveness of an ecommerce site, rather than try and rebuild from scratch every two years to try and please the CEO. Wholesale redesigns often lose the insights prior experience affords.
  • Provides a lean, evidence-based organization that reduces internal debates by making it clear that decisions are settled based on data and not egos or pecking order.
  • Builds a culture that cultivates teamwork. It’s like working a puzzle together – everyone gets to pitch in, develop ideas, propose the next piece to see if it will fit, and celebrate when portions of the work come together.
Test and learn provides freedom to fail. When failure is expected and accepted instead of condemned your ecommerce development team will ultimately get to the right solution faster. Click To Tweet

Amazon’s growth exploded because of test and learn

A Fortune online commentary by Hal Gregorson (2015) called the willingness to experiment the “one skill that made Amazon’s CEO wildly successful.”

Bezos told Gregorson that “experiments are key to innovation because they rarely turn out as you expect and you learn so much.” The Amazon test and learn culture reflects Bezos’ belief that innovation occurs in direct proportion to the number of experiments performed.

“Experimenters like Bezos see the world as a laboratory,” says Gregorson. “” The willingness to experiment, it turns out, is the chief indicator of how innovative a person or company will be.

Experiments are key to innovation because they rarely turn out as you expect and you learn so much (Jeff Bezos). Click To Tweet

Facebook embraces test and learn

Facebook business customers gained test and learn capability for their Facebook ads in early 2018. That new option gave advertisers the opportunity to compare strategies across campaigns to quickly and accurately determine which efforts are paying off and which aren’t.

Here’s what Rich Chew, Senior Director of Growth at Playstudios said about Facebook Test and Learn:

“We’re excited for Facebook’s new creative split testing to help us easily test new creatives. We’ll use split testing to identify winning creative and introduce it to our main audience alongside other top-performing creative.”

Test and learn tactics aren’t something Facebook management read about and decided to roll out to their customers, though. The phenomenal growth of Facebook has largely been fueled by an internal culture of experimentation.

Have you ever heard someone complain about how often Facebook makes changes to the platform? Any gripes about how difficult it is to keep up with all the changes on Facebook arise from the company’s dedication to test and learn?

Facebook features come, they go, they morph… they do everything but remain static. That’s because the Facebook test and learn culture informs management of results from ongoing experiments; and those results are used to make course corrections rapidly.

Consider this:

Not long ago, smart companies kept a firm grasp on tradition. Changes came slowly and painfully if at all. Today’s growth, though, often requires nimbleness and adaptability.

Dinosaurs in business find it tough to thrive in the computer age; they’re in threat of becoming extinct. That’s the number one reason why it’s crucial to invest in a culture of experimentation.

Enjoying this article?

Subscribe to our newsletter, Good Question, to get insights like this sent straight to your inbox every week.

How to develop a test and learn culture in your organization

If you accept the idea that developing a test and learn culture can help your organization make the most gains in the least amount of time, the question becomes one of how to establish test and learn as the foundation of your own conversion rate optimization efforts.

These ideas can help you do that:

  • Get personal with your motivation to inspire each member of the team. The better your team performs, the better each member of the team looks, and the more opportunities the individual members of the team will enjoy. Not only that, but a high-performing team makes management look good too. Everyone involved in test and learn gains rewards. Yes, teamwork is important – but let’s face it: individual recognition is important too.
  • Highlight the broader benefits of test and learn to give every stakeholder the big picture view of where you want to go. Test and learn procedures provide deep insight to customer needs and behavior. Test and learn will help you optimize customer experience, reduce customer support needs, and increase engagement metrics like acquisition, retention, revenue, and more. Test and learn can produce dramatic upticks in ROI. At The Good, we’ve seen conversion rate optimizations fueled by test and learn make sustainable breakthroughs unachievable by any other means.
  • Stress the need for ongoing experimentation to keep expectations set realistically. Test and learn is not a silver bullet. It’s not the knockout punch, but more like a personal trainer for your site. Executed correctly, your conversion rate optimization work is ultimately concerned with learning more about your customers. Test and learn culture means ongoing experiments. It’s important to remember that the insights come from both winning AND losing tests. Once your team gets the feel for test and learn, there will be just as much value gained from what used to be called “failed tests” as from experiments that turn out the way the team wanted.
  • Involve the team members in the entire process for deeper buy-in. The team can have impact and application beyond just your digital experience. They can be included in the ideation process as well. Report insights often. Make the entire process an open window. Keep the team informed about results related to broader goals and explain what you’re doing next based on your learnings. Properly implemented, a test and learn culture develops a sense of belonging and responsibility within the team. Everyone shares in the feedback, and everyone is invited to make suggestions and come up with new ideas.
  • Provide convincing evidence to show the validity of test and learn. Bookmark this article to cite the examples given above (Facebook and Amazon). Collect other stories of test and learn success – Netflix, AirBnB, and Google are prime examples. Use case studies to provide strong evidence-based support. Check the Results page on our website for real-life examples of how test and learn can pay off big.
  • Count the costs ahead of time to get prepared for the all-important presentations to stakeholders. Make sure you can defend the wisdom behind implementing a legitimate CRO effort based on a test and learn strategy. If you’ll need to call in help from outside your company (and most organizations do), you can ask the CRO consulting agency for help with your presentation. If you’re going with in-house help only, you’ll need to show the wisdom of that decision and establish your team’s ability to conduct the necessary work.

Test and learn cultures don’t develop overnight. Be prepared to dig in for the long haul. You’ll need to prove the value of test and learn by showing results – but those wins are most often preceded by “failures.”

Test and learn is not a silver bullet. It’s not the knockout punch, but more like a personal trainer for your site. Click To Tweet

Getting the team on board for test and learn isn’t always easy, but it is always worth the struggle

Management’s job is to allocate resources to the projects that offer the best opportunity. Once you’re personally convinced of the value of a test and learn culture, you’ll naturally want to share the excitement and potential with others in your company.

Take time to collect the resources and counsel you need to make the best case for test and learn. Establishing a genuine conversion rate optimization initiative with a test and learn culture at your company can easily become a tipping point for abundant gains in sales and customer satisfaction.

For help developing your presentation or implementation, contact The Good. We love seeing companies like yours move from frustration to success.


Jon MacDonald smiling at the camera for The Good

About the Author

Jon MacDonald

Jon MacDonald is founder and President of The Good, a conversion rate optimization firm that has achieved results for some of the largest online brands including Adobe, Nike, Xerox, Verizon, Intel and more. Jon regularly contributes content on conversion optimization to publications like Entrepreneur and Inc. He knows how to get visitors to take action.