how to improve customer experience

How to Create an Effective A/B Test Hypothesis

Here's your comprehensive guide to A/B testing – how to develop a hypothesis, when to employ your test, and what to expect from your results.

The A/B test hypothesis you develop can make or break the effectiveness of your A/B test results. Moreover, your A/B test results can make or break the effectiveness of your ecommerce conversion optimization work. And we all know that your conversion rate can make or break the profitability of your business.

Here’s the quintessential rhetorical question for a digital marketing manager: Would it be worthwhile to learn more about creating an effective A/B test hypothesis?

Whether you were the type of student who worshipped the scientific method and couldn’t wait for chemistry class, or the type who still isn’t sure about the difference between a hypothesis and a hypotenuse, learning how to build a kick-butt A/B test hypothesis can open the door to increased profits and a steadily climbing return on investment for your ecommerce website.

You may already understand the basics of conversion rate optimization (CRO) testing, or you may just be getting started. You may already be familiar with how the A/B test hypothesis is formed, or this may be the first time you’ve heard of it. Either way, this article will help you leverage the power of CRO to get more conversions.

By the time we’re finished here, you’ll know all about the A/B test hypothesis: what it is, how to create it, and how to use it.

What is an A/B test hypothesis?

In ancient Greece, science and philosophy were considered partners. The aim of science was to increase wisdom, and the “hypothesis” (literally “placed under”) was the basis of a well-constructed argument. Socrates, Plato, and Aristotle knew that a sound hypothesis could lead one to a sound conclusion.

That unyielding search for truth led to the development of what we now know as the “scientific method,” which is the basis of the A/B testing we do today.

Here are the components of that process:

  • Observe the situation, define a problem, and ask a question
  • Formulate a hypothesis (a proposed solution or explanation)
  • Test the hypothesis with experiments
  • Collect and analyze the results
  • Interpret the results and keep testing until you’re satisfied with the solution

Without the hypothesis, there can be no test – since proving or disproving the hypothesis is exactly what the test is about!

TIP: While it’s true there is no “bad hypothesis,” (since you always learn something from each test), a tightly constructed A/B test hypothesis will get you closer to the solution quicker. Spend a little more time on developing the hypothesis, and you’ll need to spend less money on testing.

EXAMPLE: TreeRing helps schools and students create less expensive, creative yearbooks. They came to The Good for help getting more leads. We used the scientific process (listed above) to conduct A/B testing aimed at conversion rate optimization.

One part of the process went like this:

  • Our team conducted a conversion audit of the TreeRing website, which highlighted areas of concern based on data like the clicks and movements of their site visitors. We wanted to know how leads were generated and why lead generation conversion wasn’t hitting TreeRing’s goals.
  • We formulated a hypothesis about the dropdown menu in the header. We theorized that moving the link to request a free sample to the top of the menu list would draw more clicks and boost the conversion rate.
  • We tested the hypothesis by using the current menu as the “control” and the new arrangement as the variation for the A/B test.
  • We collected the results and crunched the data. The test configuration sent 42 percent more visitors to the free sample landing page and drew 12 percent more requests for the sample.
  • Since testing proved our first assumption true, we then proceeded to develop and test another A/B test hypothesis to see if we could push the conversion rate even higher.

See the TreeRing case study for more insight.

ab test hypothesis
The graphic above is from an actual A/B testing experiment. See the TreeRing case study (described below) for more information.
A good hypothesis explains why the change of an element will influence the customer behavior. Share on X

Notes on the creation of an A/B testing hypothesis

Your ability to identify potential trouble spots and develop high-probability theories about how to repair them will grow over time. At The Good, our team members are skilled CRO researchers with years of entrenched experience. The more you work with conversion rate optimization and A/B testing in particular, the quicker and easier the process becomes.

To learn how to build an A/B test hypothesis, you need to get in there and practice. There is no substitute for hands-on experience. If you’re not sure how to get started, no problem. Contact The Good. We’ll help you launch a robust CRO testing program.

TIP: Finding something to test isn’t difficult. You can test every part of your ecommerce website – every word of copy, every layout, every configuration. The tough part is deciding which parts you should test. You’ll want to identify the areas that present the most potential for increasing conversions.

Begin with research

Closely observe the path your visitors take on their way to becoming customers and repeat customers. The journey to sales includes many micro conversions that lead to the macro conversion, the desired result – typically placing an order.

Your aim is to find areas where traffic is moving smoothly and areas where traffic is slowing down or dropping off. These are good tools to help you dig deep and uncover root problems. At The Good, we use a comprehensive conversion audit process. Our Stuck Score™ self-evaluation can also provide an excellent starting point.

You must ask yourself, “What business or customer experience problems do we think we can solve for this element of the site, and why do we think those changes will impact a certain KPI?”

The success of your optimization program is directly related to your ability to identify an A/B test hypothesis that will move the needle for your business.

Here’s a short list of tactics you can employ to take your research deeper:

  • Look for case studies and blog articles related to the problems your initial assessment identified
  • Collect and study customer feedback
  • Collect and study feedback from customer service, your sales team, and your support team
  • Use customer surveys to solicit specific feedback
  • Pour over analytics data
  • Employ tools like heatmapping to observe visitor behavior
  • Employ user testing to observe the entire visitor experience

With experience, you’ll get a feel for how many of these you need to employ. For cost-effective CRO testing, you’ll want to make sure your investment is balanced favorably against the potential returns.

Create a list of possibilities and select your hypothesis

Your research will illuminate one or more stuck points – barriers to sales. Choose one of them to investigate further. What could you do to free up the flow of traffic at that point on the sales journey? Where would the metrics you’re observing need to be before you consider the effort successful? How can you achieve those metrics?

Before you settle on a hypothesis, brainstorm for possibilities. There are times when your first hunch is correct, and there are times when the lightbulb comes on after a dozen potential solutions are proposed.

Don’t evaluate the proposals until you’re out of ideas. Take a break, come back to the table and try one more time: What could we change that would move the results we’re seeing towards increased conversions? How could we make that change? What exactly would the change look like? What do we think would happen as a result of the change?

Rank your ideas and choose the theory most likely to create the desired effect. At The Good we use an RDI ranking scale: Risk, Difficulty, and Impact. Each is assigned a one to ten rating. Before investing time and money in a test, we want the average of those scores to be six or better. The winner is your hypothesis.

NOTE: Your hypothesis isn’t a question. It’s a clear and measurable statement that answers your questions. For example, our hypothesis in the TreeRing example given above could be stated like this: “Moving the free sample link from the bottom of the menu to the top of the menu should increase the final conversion rate by at least 10 percent.”

Unanswered questions kill conversions. So the first job of the CRO is to learn the top questions and answer them on the page. Share on X

Should you conduct A/B or multivariate tests?

The simplest, most direct testing method is to focus on changing just one thing at a time. A/B testing does just what it says. It compares version A against version B. In our TreeRing example, we tested leaving the free sample link at the bottom of the menu versus moving it to the top of the menu.

NOTE: In A/B testing, it’s critical that you change only one thing at a time. Otherwise, you won’t be able to clearly determine the cause of the results.

Multivariate testing allows you to test more than one change at a time. You test combinations of variables simultaneously. The benefit to multivariate testing is that you can get to a total solution quicker than with A/B testing. Drawbacks are that it’s more complicated, more expensive, and will require more traffic to get dependable results.

Create your A/B test hypothesis and start testing!

You know what an A/B test hypothesis is, and you know how to create it. You know why testing is critical to conversion rate optimization, and you have access to the data you need to get started.

Use the steps and tips covered here to find a problem, develop a hypothesis, test the hypothesis, and evaluate the results. Use the information gleaned from that process to keep optimizing your conversions at every point on the sale journey.

If you need help (or even if you just want access to the tool), get your free customized Stuck Score™ from The Good. It benchmarks key conversion areas of your ecommerce website to find growth opportunities that may now be hidden.

Many of your competitors think creating an A/B test hypothesis is too complicated. They won’t take the next steps. But now you’re armed with the tools to move forward. That gives you a huge advantage. Move on it!


Find out what stands between your company and digital excellence with a custom 5-Factors Scorecard™.

Jon MacDonald smiling at the camera for The Good

About the Author

Jon MacDonald

Jon MacDonald is founder and President of The Good, a digital experience optimization firm that has achieved results for some of the largest companies including Adobe, Nike, Xerox, Verizon, Intel and more. Jon regularly contributes to publications like Entrepreneur and Inc.