Too Many Tests Can Kill Experimentation Programs: Here’s How

Contrary to popular belief, sometimes less testing is actually better in a conversion rate optimization program.

Let’s imagine a scenario together. You’re hoping to optimize your bestselling product page for more conversions. You think more testing means more opportunities for winning results, so you focus on the velocity of testing rather than the quality of your tests.

After a few weeks, you analyze the results and can’t decipher which tests led to what outcome because everything was active at the same time.

Not only have you wasted time, you’ve also lost a portion of your CRO testing budget.

In order to optimize your site effectively and efficiently, you need to find the delicate balance between your testing cadence, velocity, and analysis.

In this article, we’re going to teach you how to do just that.

What is the problem with running too many tests?

For each test your goal is to validate your hypothesis with data, right? Ultimately, you want to know if the change will increase conversions for your business.

So, unless you’re a giant like Amazon, running too many tests could completely wreck your experimentation program.

First of all, test ideas should come from data or research. So, if you’re a small team running 50 tests each month, you likely aren’t running the right tests for your site. You’re wasting money on the quantity of testing and you aren’t spending enough time and energy coming up with quality test ideas.

Focusing on the quality of ideas helps you run more tests that impact the high-value goals for your brand.

Running too many tests on the same page or audience at once can lead to poor data cleanliness.

For example, if you are running five tests on your product bundling and subscription features all at the same time, with the same goal to increase the average order value, how will you know which one moves the needle? Your data is messy. The results will be difficult to attribute to a single change and your learnings may not be clear enough to provide you with a next step in your optimization roadmap.

Our director of UX and CRO Strategy, Natalie Thomas, says:

“It’s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you’re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.”

What does it look like when you do it right? You can run a homepage test and product page test simultaneously with the primary goal of increasing the conversion rate. But for the secondary goal on the homepage you might be looking at bounce rate, while on the product page you are looking at add to cart.

In this situation, both the homepage and product page test have the same primary goal, but secondary goals are validation of which optimization actually produced the result. You can monitor these different goals to know what works.

How many CRO tests should I run per month?

There’s no one answer to the ideal number of tests you should run. It depends on your CRO goals, the complexity of your site, and your optimization strategy.

For example, The Good works with a major online media organization. Their goal is to register and subscribe new users to their service.

Every test we run is based on complex calculations that analyze what site elements they can afford to adjust and which test ideas will be the most profitable. After all of the research, analysis, and data exploration that we do, we end up running one test per quarter.

Alternatively, we have clients that have sufficient traffic and are willing to test ideas based on a bit of competitive research and user data. They end up running 3-5 tests per month.

While we can’t tell you exactly how many tests to run, the following guidelines can help you determine if you are running too many or too few tests.

As a general rule of thumb:

• If your win rate is low, you need to increase the quality and tone down the quantity.
• If your win rate is high, you’re too cautious, so your testing quality and learnings won’t be very meaningful.

A “good” win rate depends on what, where, and how you’re testing:

What: If you’re testing on a site with an insane amount of data, you might feel comfortable failing regularly because even a small win here and there has a large dollar value in the end. If you don’t have a lot of data, your testing program takes time and you’ll be tempted to make sure every test counts and swings for the trees.

Where: At volume, large companies can start optimizing even the smallest parts of the funnel, like the return customer dashboard and the reordering experience. Smaller organizations may want to focus on only the highest volume landing pages and most popular products. Limited pages mean a limited number of tests, so your cadence or volume of tests won’t be as consistent.

How: Based on the complexity of your product ecosystem and the number of downstream variables you’re analyzing, you may determine that a simple conversion rate is enough to call a winner. For organizations with many variables contributing to the bottom line (e.g., cancellation rates, return rates), post-test analysis could take months. Long post-test analysis cycles may become a limiting factor in your testing velocity and win rate.

Quality vs. quantity of CRO testing

Should you run smaller and more frequent tests? Or should you focus on bigger and fewer tests?

The answer is you should only run the number of tests you can research and manage.

If you can’t monitor all the tests you run it will be difficult to find out exactly which tests lead to specific results.

CRO testing is typically 80% research and 20% experimentation, so the more you research customer pain points and come up with strong hypotheses to solve them, the more you can determine high probability conversion opportunities.

Additionally, the bigger the predicted change in conversion rate, the quicker you see statistical significance. Higher converting tests let you quickly identify and implement a winner, so you can move on to other tests.

Once you shift your focus from trying random tests based on your gut feeling to solving specific problems your customers face, your quantity AND quality of CRO testing will increase significantly

Valuing your experimentation program beyond the number of tests

Many teams struggle to place a value on the efforts of their testing programs. To assess the performance and impact of your present experimentation program, there are other questions you should be asking besides “how many tests am I running?”

Our team at The Good uses one metric across all testing partners to answer the question: What is the value of your CRO testing program?

It comes from this formula: (365 / Days Live) x (Change in Per-Session-Value x Collected Sessions) x (1 / Percent traffic shown in this test as a decimal)

We can estimate the dollar amount expected in revenue gains over 12 months if shown to 100% of sessions on relevant pages. It’s not an exact prediction because it doesn’t account for things like changes in traffic quality and volume, but it’s helpful to gauge the ballpark impact of a test.

If a test was run during peak traffic times, you can get a more conservative estimate of annual revenue with this formula: (change in per session value) x (number of sessions to the page(s) over the last 12 months).

This is just one way to value your experimentation program beyond the number of tests you can run in a month.

Other questions you can consider are:

• What is the ROI on your experimentation program?
• Do your tests teach you about your audience? How valuable is this information?

Running a lot of tests does not automatically mean you are running a successful experimentation program. Sometimes, less testing is actually better.

Other hidden foes of your testing program

Besides running too many tests, there are some other “not-so-obvious” problems that could be holding you back from the results you want.

Lack of buy-in from leadership

An absence of experimentation culture on your leadership team can trickle down and kill your optimization program before you can say “A/B test.”

So, it’s important to map out potential objections before presenting your experimentation ideas. Get buy-in from the stakeholders and then only involve team members who have a “test and learn” mentality.

If your CRO testing program isn’t getting great results or you’re getting too high of a win rate, it means you’re not taking enough risks with your tests.

Leading your program with biases and pre-held assumptions

When too many opinions are considered in an experimentation program, the focus shifts away from research. Instead, it goes towards the majority viewpoint or that of the HIPPO (highest paid person’s opinion).

Team members often have tightly-held assumptions about their customers — what they like, what they don’t like, and why you shouldn’t do a certain thing based on previous experience. If they hold these assumptions too close, they tend to play it safe.

We also know from experience that if too many people contribute to how you structure and conduct your tests, it can put a strain on your CRO program.

Get started with your new and improved experimentation strategy

If you’re struggling to move the needle with your CRO program, then it’s time to rethink your strategy.

While there’s no one-size-fits-all answer to exactly how many tests you should carry out, one thing is for sure. You should prioritize quality before you up your quantity.

Start analyzing what’s going wrong and identifying potential silent killers of your experimentation program.

• Are you placing value on the quantity of your tests over the quality?
• Are you able to monitor and analyze all the tests that you run?
• Do you need to spend more time getting buy-in from your leadership team?
• Are you only testing pre-held assumptions?

These are just a few thought-starters to get you thinking.

If you want a quality-driven CRO firm to help you identify areas for improvement or to set up your ideal testing frequency, we would love to hear from you. Contact us.