multivariate testing

How Ecommerce Brands Can Run Successful Multivariate Testing

Use this guide to multivariate testing to learn how your ecommerce brand can pinpoint the right combination of web elements that drive conversions.

Trial and error is often the route to success. 

When you figure out what works and what doesn’t, you can do more of the good stuff (and who doesn’t like the good stuff?!).

Luckily, there are plenty of ways you can test your ecommerce website. After all, there are so many elements that come together, from your product pages and everything that populates them to your marketing efforts, checkout process, and post-purchase campaigns.

Multivariate testing allows you to test many different possible combinations, which can be a great opportunity to quickly learn what site features are getting the most meaningful results. 

What is multivariate testing in ecommerce? 

Multivariate testing is the act of testing different combinations of elements on your ecommerce website to find out which converts best. 

For example, you might serve consumers several different versions of the same landing page with a few differences. One version might have three static product photos, another might have a mix of static photos and a handful of user-generated photos, and yet another might have user-generated photos and a video. 

If you’ve got a good amount of traffic, you can quickly start to see which page (and which combination of elements) resonate the best with your audience. If the page with the UGC and video performs better than the photos on the other web pages you’re testing, you can safely assume that’s the option shoppers prefer or are more inclined to click. 

An important starting point in multivariate testing is formulating a hypothesis based on cold, hard data. For example, let’s say you run user testing and see that the majority of users are clicking out of your checkout after 2 minutes of trying to fill out the checkout form. Then using Google Analytics data for your store and benchmarking stats from a reliable research company, you discern that your cart abandonment rate is higher than it should be. 

You then form a hypothesis: “The checkout form is too long/complex. By shortening the form and adding inline validation, we will deliver a better customer experience and better form completion rate.” 

This is the perfect opportunity for multivariate testing. You can create different combinations of form elements, lengths, or instructions, and serve them to different audience segments to see which delivers the best completion rate.

Multivariate testing vs A/B testing: what’s the difference?  

Multivariate testing is not to be confused with A/B testing (although it can be a bit murky). 

A/B testing focuses on one element and tests multiple variations of that specific element (e.g. testing the headline on a landing page). 

Multivariate testing is a way to try out different combinations of multiple elements all at the same time. Say, for example, you want to test a video, an image, and a combination of the two to see which variation gets the best results. 

The pages you create might look something like this:

  • Control: your current page design with no changes
  • Combination 1: just an image
  • Combination 2: just a video
  • Combination 3: an image and a video
ab vs multivariate testing

This way you can isolate which combinations of elements work best together and create the biggest uplift. 

You might also have come across the term split testing, which falls under the umbrella term of A/B testing. In this method, you essentially send traffic to two or more different URLs to see which performs better – think ad traffic seeing the same ad, but being sent to different pages once they click on the CTA. One group might be sent to a product page and the other might be sent to a category page so you can see which journey reaps the biggest rewards. 

You can run both A/B testing and multivariate testing in tandem. Where A/B testing identifies the winning elements in an X vs Y vs Z (e.g. video vs image vs text) formula, multivariate testing is more about how combinations of elements work together in an X1 vs X2 vs Y1 vs Y2 formula (e.g. video + image vs video + text vs text + video + image vs just video).

The benefits (and downfalls) of multivariate testing 

There is a time and place for every type of ecommerce testing. 

One of the most crucial upsides to A/B testing is that you can do it with a slightly smaller audience than multivariate testing. With the latter, you’re likely running a larger number of variants and combinations at once which means you’ll need a bigger sample audience to get significant results. 

Even so, multivariate testing comes with its pros. This method of testing tells you about the interactions between a number of elements on a single page and also shows you what/where different page elements have the most impact all at the same time. 

A/B testing may get you the best headline, but multivariate testing will tell you the best combination of headline + hero image + call to action. 

Enjoying this article?

Subscribe to our newsletter, Good Question, to get insights like this sent straight to your inbox every week.

How to do multivariate testing that increases ecommerce conversions 

Ready to run a multivariate test? Here’s a step-by-step guide to doing it justice. 

1. Identify a problem that needs solving 

Start by determining what you want to test. 

This will depend on your analytics and key business goals. For example, if your conversion rates on product pages are particularly low, you might experiment with combinations of different product names, images, and placement of value propositions like free shipping, sustainability, returns, etc. 

2. Set your goals 

Once you’ve identified a problem, link it to a conversion goal. If your conversion rates are low, you might set a goal to improve them by 5% every month, or if your abandoned cart rate is particularly high, you might set a goal to get it down to 60% in six months. 

When you know what your goals are, you can measure the efforts to see how your multivariate tests are performing – how will you know if the imagery and value proposition combination is working if you’ve got nothing to measure it against? 

Once you have set your goals, you can create hypotheses to test. This should be based on the problem you’ve discovered and a potential solution you’re going to test. 

3. Create your variations

This is the fun part. Now it’s time to start testing the elements you want to test, but to do so you need to put together the different combinations of your product page, checkout process, or any other part of the sales cycle you’re looking to tweak with website optimization. 

For example, if you’re looking to test the performance of your product page click-throughs by testing different visual elements you might create four different combinations: 

  • The control 
  • Combination 1: carousel images and text descriptions
  • Combination 2: just carousel images
  • Combination 3: video and text descriptions
  • Combination 4: video and carousel images and text descriptions 

4. Review your tweaks and run the test

Many ecommerce platforms allow you to run multivariate tests, but you can also use a dedicated testing tool to put together the different pages and the different elements you want to tweak. At this point, you’ll want to go over the changes you’ve made and check that the different tests you’re running are set up properly and have the aim of reaching your goal. 

Once that’s done, you can start running your test. You’ll get more significant results (a.k.a. more data) if you’re serving the pages to a high number of people. For pages that don’t have a large amount of website traffic, you might find that the tests take longer to get the data you need to make an educated decision or they simply don’t work at all (this will be common if you have a limited sample size).

If you don’t think you have high traffic on the pages you want to test, you can start driving traffic to increase the numbers and get more data to work with. You can do this by: 

  • Running social media ads
  • Sending out email marketing campaigns
  • Creating PPC ads on Google 
  • Sharing links on your social media accounts

5. Analyze your results 

Once you’ve run the test to a high enough number of people, you should start to see which variation comes out on top. 

It’s likely you’ve tested a combination of elements at once, so it’s important that you dig into the results from each combination. 

You can use tools like heatmaps, click maps, form Google analytics, and session recordings to see which variation had the best results.

When to use multivariate testing

Multivariate testing is ideal if you’re looking to optimize several different elements on a page. It’s best for isolating changes that, in the right combination, add up to create an optimized product page, checkout process, or another part of the sales cycle. 

If we just tested a new image on our homepage that might fail. If we just updated the language that might fail too. If we test two different images with two different value props, we might find a combination that sticks the landing best.

Ecommerce multivariate testing examples to kick off your experiments

The buying journey often looks very different for each customer, but there is a universal route that they tend to go through that takes in major touchpoints. It’s at these touchpoints that you can implement minor tweaks and carry out multivariate testing to check the outcomes. 

There are a number of combinations you can test on your ecommerce site depending on the length of the sales cycle, who you’re targeting, and the products you’re selling. Here are some examples of multivariate tests you can run with your ecommerce brand: 

  • Homepage
    • The control 
    • Combination 1: evergreen hero image + classic product value proposition
    • Combination 2: limited time offer hero image + urgent messaging
    • Combination 3: evergreen hero image + CTA to purchase best seller 
    • Combination 4: hero product video + CTA to purchase best seller
  • Product media
    • The control 
    • Combination 1: product video and images only
    • Combination 2: product video and icons on product benefits
    • Combination 3: product images and icons on product benefits
    • Combination 4: product images, videos, and icons on product benefits
  • Checkout
    • The control 
    • Combination 1: 10 fields + form field naming conventions version 1 
    • Combination 2: 5 fields with progress bar + form field naming conventions version 1 
    • Combination 3: 10 fields + form field naming conventions version 2
    • Combination 4: 5 fields with progress bar + form field naming conventions version 2

Improve conversions through multivariate testing with The Good 

Testing different combinations on your ecommerce website can really help ramp up your conversions. If you’re consistently tweaking small bits here and there, it can all add up to much higher engagement rates and, ultimately, more sales and customers.

For Fayettechill, multivariate testing changed everything. They came to The Good after realizing their customer experience wasn’t reflecting their brand and as a result, they were losing customers. 

The brand went through The Good’s Conversion Growth Program to identify where they could make the changes. Throughout the program, The Good reviewed site metrics and online sales data to form hypotheses to run tests against. From there, we optimized every page a customer lands on during the buying process which ended up increasing the year-on-year conversion rate by 24% and a revenue increase of 54%.  

Multivariate testing is a key way ecommerce brands can determine what individual elements are working throughout the buying journey and what elements aren’t. 

Small changes can add up over time, leading to a high-performing sales cycle that generates more conversions and sales. 

Start small by identifying problems through testing tools such as heatmaps and analytics, creating hypotheses to test, and measuring your test results over time. 

Enjoying this article?

Subscribe to our newsletter, Good Question, to get insights like this sent straight to your inbox every week.

Natalie Thomas

About the Author

Natalie Thomas

Natalie Thomas is the Director of CRO & UX Strategy at The Good. She works alongside ecommerce and lead generation brands every day to produce sustainable, long term growth strategies.