a woman holding pages of cro testing results

Too Many Tests Can Kill Experimentation Programs: Here’s How

When optimizing your website, you need to find the balance between quality and quantity of tests. Here, we discuss how you can test more effectively for winning results.

Key Takeaways

By the end of this article, you should have the knowledge and resources to “check the box” in these areas…

  • Why more testing doesn’t always mean better results
  • How you can run more of the RIGHT kind of tests
  • Potential problems with your experimentation program

Contrary to popular belief, sometimes less testing is actually better in a conversion rate optimization program.

Let’s imagine a scenario together. You’re hoping to optimize your bestselling product page for more conversions. You think more testing means more opportunities for winning results, so you focus on the velocity of testing rather than the quality of your tests.

After a few weeks, you analyze the results and can’t decipher which tests led to what outcome because everything was active at the same time.

Not only have you wasted time, you’ve also lost a portion of your testing budget.

In order to optimize your site effectively and efficiently, you need to find the delicate balance between your testing cadence, velocity, and analysis.

In this article, we’re going to teach you how to do just that.

What is the problem with running too many tests?

For each test your goal is to validate your hypothesis with data, right? Ultimately, you want to know if the change will increase conversions for your business.

So, unless you’re a giant like Amazon, running too many tests could completely wreck your experimentation program.

First of all, test ideas should come from data or research. So, if you’re a small team running 50 tests each month, you likely aren’t running the right tests for your site. You’re wasting money on the quantity of testing and you aren’t spending enough time and energy coming up with quality test ideas.

Focusing on the quality of ideas helps you run more tests that impact the high-value goals for your brand.

Running too many tests on the same page or audience at once can lead to poor data cleanliness.

For example, if you are running five tests on your product bundling and subscription features all at the same time, with the same goal to increase the average order value, how will you know which one moves the needle? Your data is messy. The results will be difficult to attribute to a single change and your learnings may not be clear enough to provide you with a next step in your optimization roadmap.

Our director of UX and Strategy, Natalie Thomas, says:

“It’s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you’re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.”

What does it look like when you do it right? You can run a homepage test and product page test simultaneously with the primary goal of increasing the conversion rate. But for the secondary goal on the homepage you might be looking at bounce rate, while on the product page you are looking at add to cart.

In this situation, both the homepage and product page test have the same primary goal, but secondary goals are validation of which optimization actually produced the result. You can monitor these different goals to know what works.

How many tests should I run per month?

There’s no one answer to the ideal number of tests you should run. It depends on your goals, the complexity of your site, and your optimization strategy.

For example, The Good works with a major online media organization. Their goal is to register and subscribe new users to their service.

Every test we run is based on complex calculations that analyze what site elements they can afford to adjust and which test ideas will be the most profitable. After all of the research, analysis, and data exploration that we do, we end up running one test per quarter.

Alternatively, we have clients that have sufficient traffic and are willing to test ideas based on a bit of competitive research and user data. They end up running 3-5 tests per month.

While we can’t tell you exactly how many tests to run, the following guidelines can help you determine if you are running too many or too few tests.

As a general rule of thumb:

  • If your win rate is low, you need to increase the quality and tone down the quantity.
  • If your win rate is high, you’re too cautious, so your testing quality and learnings won’t be very meaningful.

A “good” win rate depends on what, where, and how you’re testing:

What: If you’re testing on a site with an insane amount of data, you might feel comfortable failing regularly because even a small win here and there has a large dollar value in the end. If you don’t have a lot of data, your testing program takes time and you’ll be tempted to make sure every test counts and swings for the trees.

Where: At volume, large companies can start optimizing even the smallest parts of the funnel, like the return customer dashboard and the reordering experience. Smaller organizations may want to focus on only the highest volume landing pages and most popular products. Limited pages mean a limited number of tests, so your cadence or volume of tests won’t be as consistent.

How: Based on the complexity of your product ecosystem and the number of downstream variables you’re analyzing, you may determine that a simple conversion rate is enough to call a winner. For organizations with many variables contributing to the bottom line (e.g., cancellation rates, return rates), post-test analysis could take months. Long post-test analysis cycles may become a limiting factor in your testing velocity and win rate.

Quality vs. quantity of testing

Should you run smaller and more frequent tests? Or should you focus on bigger and fewer tests?

The answer is you should only run the number of tests you can research and manage.

If you can’t monitor all the tests you run it will be difficult to find out exactly which tests lead to specific results.

Testing is typically 80% research and 20% experimentation, so the more you research customer pain points and come up with strong hypotheses to solve them, the more you can determine high probability conversion opportunities.

Additionally, the bigger the predicted change in conversion rate, the quicker you see statistical significance. Higher converting tests let you quickly identify and implement a winner, so you can move on to other tests.

Once you shift your focus from trying random tests based on your gut feeling to solving specific problems your customers face, your quantity AND quality of testing will increase significantly

Enjoying this article?

Subscribe to our newsletter, Good Question, to get insights like this sent straight to your inbox every week.

Valuing your experimentation program beyond the number of tests

Many teams struggle to place a value on the efforts of their testing programs. To assess the performance and impact of your present experimentation program, there are other questions you should be asking besides “how many tests am I running?”

Our team at The Good uses one metric across all testing partners to answer the question: What is the value of your experimentation program?

It comes from this formula: (365 / Days Live) x (Change in Per-Session-Value x Collected Sessions) x (1 / Percent traffic shown in this test as a decimal)

formula showing the value of cro testing program

We can estimate the dollar amount expected in revenue gains over 12 months if shown to 100% of sessions on relevant pages. It’s not an exact prediction because it doesn’t account for things like changes in traffic quality and volume, but it’s helpful to gauge the ballpark impact of a test.

If a test was run during peak traffic times, you can get a more conservative estimate of annual revenue with this formula: (change in per session value) x (number of sessions to the page(s) over the last 12 months).

formula showing how cro testing program affects annual revenue

This is just one way to value your experimentation program beyond the number of tests you can run in a month.

Other questions you can consider are:

  • What is the ROI on your experimentation program?
  • Do your tests teach you about your audience? How valuable is this information?

Running a lot of tests does not automatically mean you are running a successful experimentation program. Sometimes, less testing is actually better.

Other hidden foes of your testing program

Besides running too many tests, there are some other “not-so-obvious” problems that could be holding you back from the results you want.

Lack of buy-in from leadership

An absence of experimentation culture on your leadership team can trickle down and kill your optimization program before you can say “A/B test.”

So, it’s important to map out potential objections before presenting your experimentation ideas. Get buy-in from the stakeholders and then only involve team members who have a “test and learn” mentality.

If your testing program isn’t getting great results or you’re getting too high of a win rate, it means you’re not taking enough risks with your tests.

Leading your program with biases and pre-held assumptions

When too many opinions are considered in an experimentation program, the focus shifts away from research. Instead, it goes towards the majority viewpoint or that of the HIPPO (highest paid person’s opinion).

Team members often have tightly-held assumptions about their customers — what they like, what they don’t like, and why you shouldn’t do a certain thing based on previous experience. If they hold these assumptions too close, they tend to play it safe.

We also know from experience that if too many people contribute to how you structure and conduct your tests, it can put a strain on your program.

Get started with your new and improved experimentation strategy

If you’re struggling to move the needle with your program, then it’s time to rethink your strategy.

While there’s no one-size-fits-all answer to exactly how many tests you should carry out, one thing is for sure. You should prioritize quality before you up your quantity.

Start analyzing what’s going wrong and identifying potential silent killers of your experimentation program.

  • Are you placing value on the quantity of your tests over the quality?
  • Are you able to monitor and analyze all the tests that you run?
  • Do you need to spend more time getting buy-in from your leadership team?
  • Are you only testing pre-held assumptions?

These are just a few thought-starters to get you thinking.

If you want a quality-driven optimization firm to help you identify areas for improvement or to set up your ideal testing frequency, we would love to hear from you. Contact us.

Enjoying this article?

Subscribe to our newsletter, Good Question, to get insights like this sent straight to your inbox every week.

caroline appert profile picture

About the Author

Caroline Appert

Caroline Appert is the Director of Marketing at The Good. She has proven success in crafting marketing strategies and executing revenue-boosting campaigns for companies in a diverse set of industries.