D&C Episode 047 – Rapid Testing (WP Featured Image)

Drive and Convert (Ep. 047): Rapid Testing – The Next Evolution of CRO?

In this episode, Ryan and Jon talk about the merits and challenges of Rapid Testing, an approach to CRO that requires less time and resources to implement, but also comes with a lower confidence interval.

Listen to this episode:

About This Episode:

Traditional conversion rate optimization is built around achieving statistically significant results through a combination of research and testing. It can be expensive and time-consuming, but it also carries a >90% confidence level.

In this episode, Ryan and Jon talk about the merits and challenges of another optimization approach known as Rapid Testing. It requires less time, traffic, and resources, but also provides less confidence in the efficacy of the results.

Listen to the full episode if you want to learn:

  1. What separates Rapid Testing from traditional CRO methods
  2. Which businesses are a good fit for Rapid Testing
  3. The benefits and challenges of implementing a Rapid Testing program
  4. What kind of results are typical of a successful Rapid Testing program

If you have questions, ideas, or feedback to share, hit us up on Twitter. We’re @jonmacdonald and @ryangarrow.

Subscribe To The Show:

Episode Transcript:

Announcer:
You’re listening to Drive and Convert, a podcast about helping online brands to build a better e-commerce growth engine with Jon MacDonald and Ryan Garrow.

Ryan:
Jon! Pleasure to be here. Today, we’ve got a really cool topic. I think that’s not one that a lot of people talk about. But I think is going to excite a lot of people around the CRO consideration realm. And it’s a buzzy word in this time of life that we’re in. But rapid testing.

Jon:
Yes.

Ryan:
Which is generally something that never comes out of your mouth.

Jon:
Hopefully.

Ryan:
“Just move quick and we’re going to get great answers.” No, it’s the time data scenario in most CRO. But you and your team have come up with some pretty cool things you can do quickly in the CRO space. In fact, somebody that’s not very patient. Rapid testing really speaks to who I am at my core. So I’m excited to hear about this. So kick us off and just tell us at a high level, what is rapid testing in CRO?

Jon:
Yeah, this was formulated specifically for folks like you. So I’m glad we can talk about this today. Yeah, so rapid testing. Okay, as you mentioned traditional, conversion optimization takes time. You really want to be looking at things like AB testing, that requires a certain amount of traffic going through that test, which can require time. In order to really statistically prove out which is the better variant. Or you can do things like user testing, which is a huge lift.
Even if you do remote unmoderated, where you’re running it through a software tool and things of that sort, it can be a problem. So really the goal of rapid testing is to get the cost down, the timeline and effort down. And help have a bit of an insurance policy on what you’re moving forward with. So is it as thorough traditionally we would be CRO? No, but it has its place. And I think that’s why it’s important. Sometimes you don’t need the Ferrari and maybe you only need the Volkswagen. Right? And that’s okay.

Ryan:
It’s not as exciting to drive a Volkswagen, but-

Jon:
I would agree with that.

Ryan:
… there’s a place for it, I guess. Okay, and so most companies across the board, you’re going to recommend full CRO. I know out of the gate, I’ve talked to you enough times with brands. They have a certain size.

Jon:
Yes, yes.

Ryan:
And so if I’m a large enough brand to afford general CRO that you would put in that bucket, why would they even consider rapid testing then? Is there scenarios that make sense for them, that it wouldn’t make sense to a full CRO?

Jon:
Of course. I mean, even when we’re are working with Fortune 500 brands like Adobe, we’re doing rapid testing at times for them.

Ryan:
Really?

Jon:
Because there are situations where they need to move extremely quick. Right? And they need to know how to improve something. But they don’t have time to go do a full user testing study, or for us to go out and interview customers, or for us to even get an AB test. Despite the fact that some of their pages have tens of millions of visitors a day. And they still can’t or don’t want to be able to test through AB testing on their site. So what are the next steps?
And that’s really where rapid testing comes from, is how do we augment these heavier lifts to just be able to do something and prove it out now. Most cases you’re right. I’m going to say, “Hey, you really need to be doing full AB testing.” Or, “You really need to be thinking about doing a more iterative approach. This isn’t just a quick win.” And I think the goal here is going in with your eyes wide open. You know this is not going to be as thorough, and you’re okay with that in this instance.

Ryan:
Got it. So if you’ve been working with a company for years and years. And they know you well, you know them well. Why can’t they just spin up something that’s worked already before in some other version of what they’ve done. And just say, “Yeah, it’s probably going to be good enough let’s go.”?

Jon:
Well, I think that’s where if you look at consumers or behaviors are always changing. Technology is always changing. Your product is always changing.

Ryan:
Mm-hmm (affirmative).

Jon:
And the context that the consumer is coming to your site’s always changing. There’s also a timeline issue, for instance. Right? So a brand that is right now, we’re on the cusp of holiday, Black Friday, Cyber Monday week in the US, especially. But the reality is, it’s way too late to be doing AB testing.

Ryan:
Mm-hmm (affirmative).

Jon:
Right? But if you were launching a new landing pay page, for instance. And you really want to understand and have that insurance policy, that that landing page is going to produce, because it’s such a critical time. What do you do? Do you throw it at the wall and see if it sticks? Well, you may miss that opportunity, or do you take a days or two days, and run some rapid testing before you throw it up? And that’s going to allow you to know if there’s any big holes in the process and go from there.

Ryan:
Okay. So for example, last week if I was going to decide, does this promo make sense or does this landing page work for my Black Friday or Cyber Monday promotional schedule? What does it actually look like to go through that rapid testing?

Jon:
Great.

Ryan:
I mean, I have to have some traffic, I assume, that I want to test promos with.

Jon:
Yeah, actually no.

Ryan:
Oh.

Jon:
These are all things that aren’t done on your site, and they aren’t done using your traffic necessarily. There are some ways we can involve your consumers. But when you involve consumers in this way, you slow down the process dramatic. Right? Because you want to make sure you have the right consumers, you have to compensate them in some ways. And so there’s a lot of things you can be thinking about here. So what are the different types of rapid testing? Well, let’s start with a pretty easy one. You could do heat mapping, but it’s artificial intelligence heat mapping. Right? So you’re doing things that are like eye tracking based on artificial intelligence algorithms, fed with a lot of actual eye tracking studies to see that AI. And then you can train it as well. And so we use those to types of tools, which are great.

Ryan:
And you do have one of your own proprietary versions of that.

Jon:
Yes.

Ryan:
Don’t you?

Jon:
Yes, exactly.

Ryan:
So not everybody can just go off the shelf and find one of these things, I’m guessing?

Jon:
Yeah. There’s definitely options out there. This is something that is, again an enterprise level tool. That if you’re just as small or medium e-comm site, you’re probably not going to want to invest the budget to make that happen on your own. But through us, we have the tools. So we’re able to use it with our customer base, who might not be able to afford it if they want to market for something similar.

Ryan:
Cool.

Jon:
So artificial intelligence tracking is one way. We can also do something where we go to these different types of small task warehouses that exist. And what that means is there are folks all over the globe, who do small tasks for a dollar each task, or 50 cents a task at times, depending on how long it takes. What we can do is put certain types of testing up there. And we can do things like preference tests. We can do things like five second tests. So what’s a preference test? Well, we could put two different banners up or two different page layouts, and ask a very pointed question. Which one of these has the better offer. Right? Which one of these is more intriguing? And there’s a whole psychological way that you want to survey and ask these folks about these. But the goal here, is really to do that rather quickly. So you can get thousands upon thousands of responses within 12 hours.

Ryan:
Whoa!

Jon:
And it still costs a little bit. But it’s moi more minimal than the cost to design a test, do all the data science on it, build that test, then run it, and then implement it on your site. Right?

Ryan:
Mm-hmm (affirmative).

Jon:
So you start thinking about weighing the options there. Okay. Well, this might not be a subset of my exact site visitors, but they fit a reasonable demographic. And so it makes sense to get the large volume of data. Another thing you can do up there is the five second test that I mentioned. Where you put a image up for five seconds and then take it down. And ask people to describe what they remembered. “What do you recall?” Or ask even a more pointed question, “Did you see A or B?” Right?

Ryan:
Mm-hmm (affirmative).

Jon:
And so then you have some options there as well.

Ryan:
I have a weird question that may be slightly different than what you have, how you would think about it.

Jon:
Mm-hmm (affirmative).

Ryan:
But if I’m going into holiday, and one of the things I’d like to be able to know that would fall in the vein of this cool task warehouse that I hadn’t heard of before now. If I already know what my competitor’s offer is, in theory I could leverage this to say, “All right, competitors are offering a BOGO 50% off-

Jon:
Yes.

Ryan:
… and I want to offer 25% off.” Who’s going to win in that offsite to get the traffic. If I’m going head to head in Google ads with them?

Jon:
Yes.

Ryan:
Or I’m going head to head in social. So even before they get to the site, be like-

Jon:
Yes.

Ryan:
… “Which offers more compelling?”

Jon:
This is something that we’ve been doing for brands with this rapid testing. Because it’s a great way, which you bring up another great opportunity with rapid testing, which is offsite testing. So when you typically think about conversion rate optimization, you’re thinking about onsite testing. And that is where we spend 99% of our time. But as you’re calling out, there are definitely opportunities for us to be testing what it’s driving people to the site, and what’s going to be more effective there.

Ryan:
Especially during high conversion period times, where a lot of that research is done offsite. And by the time I get to the site, I already know I’m converting.

Jon:
Right.

Ryan:
If I’m looking for TV and I’m like, “Target’s got it for 200 and Walmart’s got it for 1.99. Great! I already know I’m going 1.99, I’m buying it. My conversion rate is astronomical over the next week if I’m buying from you right now.”

Jon:
Yeah. And again, all of these things are absent of price. Right? One thing that we really don’t want… I mean, we do some testing, and we have on preference testing around pricing. But the reality is lower price is always going to win almost. Right? So it’s not really worth testing pricing in our point of view. And it’s also, that’s not optimization. It’s really, as you know I say it’s margin drain. Right? Because you really just a race to the bottom at that point. So instead we’re thinking this is a great opportunity to be testing which ad catches an eye better. Right?

Ryan:
Mm-hmm (affirmative).

Jon:
If you have the same price, how can you outperform your competitor’s ad? That is a great way to be doing some rapid testing here.

Ryan:
Yeah. Is it going to be free shipping this year? Or is going to be free offer with purchase? Or is it going to be that BOGO? I think brands are having to get a lot more creative right now before they even get to the site on some of those-

Jon:
Yeah.

Ryan:
These cyber deals, that it becomes interesting. And funny, we’ll talk on this at a later episode, probably about some traffic you can get with some of these offers. But there are companies that are sending a lot of traffic to Amazon. And what they do is they get all the lists of promotions going out from Amazon, and they get to pick and choose which ones they’re going to send their traffic to. And so they get money from Amazon to drive social ads to Amazon during holiday periods.

Jon:
Yeah.

Ryan:
And they just going be like, “Oh, I like this offer, this offer, this offer.” I’m like, so you have to not only stand out to the consumer.

Jon:
Mm-hmm (affirmative).

Ryan:
But you’re going to stand out to the person driving the traffic to your site or to Amazon, so that they pick your offer. Which is crazy.

Jon:
Well, and I think you bring up a whole nother opportunity for rapid testing. Which is Amazon. Right? So we’ve started recently using rapid testing to do some optimization of Amazon product pages. Because you really have very, very limited AB testing opportunities within Amazon’s platform. But if we can get the testing off of Amazon’s site in some manner. Then we’re able to really start moving the needle. And so that’s where rapid testing can often come in as well.

Ryan:
Cool. So all of you listening that have not talked to Jon about CRO in your Amazon pages, call him up. There’s some-

Jon:
Yes.

Ryan:
… opportunity there. Amazon has a disgusting amount of volume.

Jon:
It’s a big opportunity area for us, and something that we’re really looking to put our weight behind in 2022.

Ryan:
That’s exciting. I’m excited to see some of that data. And so there’s close timelines. We know we have to get a decision very quickly.

Jon:
Mm-hmm (affirmative).

Ryan:
We can’t fully be test. Initial direction, and so is that more about offer or is that usually more around the design of the page? Where you like, “Eh, does this image go here or here?” Where do you move often see the rapid testing for initial directions?

Jon:
Yeah. Bigger than a placement of an image. If you’re talking two big changes. Right? So two completely different designs of something. Right? And you want to test those two big things, and see which direction is going to help you go down a better rabbit hole. So that way, this is where I like to call it an insurance policy. You’re basically ensuring yourself against diving and spending a lot of resources down a path that isn’t going to perform. And you could have been told that right front with a little bit of research. And so that’s where rapid testing can really be helpful here.

Ryan:
Got it. So it’s like a round, “Hey, do we want a left hand nav or top nav?” I can probably guess which one’s right, but you want to make sure.

Jon:
Well, yeah. Or just a completely, you have a designer, you’re overhauling the design of your site. And they offered you two options for the visual design of your site and all. Right? The color schemes, the type, are you using photos versus illustrations? Different fonts. Right? All of that type of stuff. You can basically export a flat image of your Photoshop mock ups, and do preference testing or rapid testing on those two. And you could even do something where you put one up for five seconds. And then say, “What emotions are you feeling right now, based on what you saw?”

Ryan:
Yeah.

Jon:
Right?

Ryan:
Feelings, oh, man!

Jon:
Can’t see this, but Ryan’s running away.

Ryan:
Yeah.

Jon:
Yeah. So I think that’s where I call it the insurance policy. Because you’re involving users at such an early stage, that you typically wouldn’t have been able to a full conversion optimization. Because otherwise it’d be much harder to set up a whole user test, and start interviewing people about what they’re doing here. Instead of getting tens of thousands of people to respond to the general sentiment. And then you can look at that data and say, “Wow! It’s like 90 to 10, then very, very high bias in one direction.” That’s going to be hard to pass up in terms of data and relevance there. Or it may come in and it’s like 60/40, and then you have to start thinking, okay, maybe I’ll head down that one path a little bit and incorporate some of the other elements, and do another round of rapid testing. Again, it’s not going to be a one and done in most cases. But it is something that’s going to be a lot more quicker and helpful.

Announcer:
You’re listening to Drive and Convert, the podcast focused on e-commerce growth. Your hosts are Jon MacDonald, founder of The Good, a conversion rate optimization agency that works with e-commerce brands to help convert more of their visitors into buyers. And Ryan Garrow of Logical Position, the digital marketing agency offering paper click management, search engine optimization, and website design services to brands of all sizes. If you find this podcast helpful, please help us out by leaving a review on Apple podcasts, and sharing it with a friend or colleague. Thank you.

Ryan:
Okay. And then where do you often see brands trying to stuff something into rapid testing that’s not probably appropriate for that? Where should they be like, “Oh, yeah, I really want to do this, but…”?

Jon:
Yeah. First thing is, it’s the antithesis of rapid testing to have your audience involved, specifically your users. That’s when it starts to slow down and now we’re talking to real optimization, not this rapid testing. Right? And so with that in mind, you need to have some grace on, you’re giving up a few things in essence of speed. So what do I mean by that? Well, first of all, you’re giving up more statistical certainty. Right? You are giving up the ability to exactly test with your site visitors.
So these are folks who may or may not have ever heard of you. They may or may not ever be on your site. But what we can do with that, is get them into your demographics at a high level. Right? But we’re not going to be able to go down to they’ve bought your competitor for instance. Right? It’s going to be much higher level than that. It’s a male between this 20 and 40, with an income above $50,000. Right? That might be the most that we’d be able to do. So you have to keep that in mind too.

Ryan:
So we can get your target demo.

Jon:
Yeah.

Ryan:
But other than that, you’re just getting guidelines to begin something that eventually you will get official CRO on.

Jon:
Right, exactly! And that’s how you want to really want to be thinking about this is, this is a great starting spot and it’s great for early items to help guide. But it’s not the end all be all. You really need to be thinking about what is your next step going to be from here? And knowing that you’re going to have to continue to test at each step. But you’re able to get that feedback much quicker.

Ryan:
Now, is this something that can work for small brands that have only a few $100 of budget that they can start looking at this? Or is this still going to be, “Hey, you’ve got to have 2,500, 3000 bucks right now to do any of this.”?

Jon:
Well, I think if you only have a few $100, you might want to attempt doing some of this yourself. And if you are small or a smaller, or let’s just say SMB e-commerce. It’s really worse spending a couple 1,000 bucks on this, versus a true conversion optimization where you don’t have enough traffic or time or budget. And you can’t spend, let’s just say $10,000 a month. But with this rapid testing you for a couple 1,000, you should be able to have a really clear direction of where to head. So to answer your question, it’s not a couple 1,000 dollars. There’s still some effort involved. But it is way cheaper than doing true conversion optimization. And you get the benefits of the time and the other sources that come with that.

Ryan:
Yeah, that’s why I talk to a lot of brands and say, “It’s your time, money conversation.”

Jon:
Mm-hmm (affirmative).

Ryan:
If you have more time than money, then you’re going to Starbucks, you’re buying some people coffee.

Jon:
Right.

Ryan:
If you have a little more money than time right now, you’re going to be paying a couple 1,000 bucks for rapid testing to get yourself going and jump starting until you can afford full CRO.

Jon:
Yeah. And you think about how much you pay for car insurance every month. Right? Several $100 in most cases. And you look at that and you say, “Okay, I’m paying that for car insurance, but the reality here is how often am I getting in an accident? Or hopefully I’m never using it.” Right? It’s the same thing here. You might spend a couple grand, but the reality is that you’re going to have much better performance. And think of it as that type of insurance policy, what’s the opportunity cost of not doing this, not involving users early? You might get all the way in to development and launch of your new site to only find out, “Wow! That new site performing way worse than my last site.”
And we’ve had a couple of clients who’ve come to us after they’ve tried to do this without involving consumers. They’ve redesigned and relaunched their entire site. And since they didn’t talk to consumers at launched. And they’re like, “Whoa! Our site just totally is half the conversions, half the sales. We’re not sure what’s going on here. Something must be broken.” And you’re like, “Well, it’s not broken. Everything seems to be working okay. It’s the fact that you didn’t involve consumers, you made a ton of assumptions.” And then they end up taking the new site down and putting the old one back up. Because it doesn’t work and they get really concerned really quickly. And so my first thought with that is, well, you should have done some initial rapid testing. You could have been way better off and saved yourself all of that revenue that you spent.

Ryan:
I find this often, actually that particular scenario when I’m talking to a brand that generally targets an older demographic. And they’ve had an old site that for all intensive purposes and for a millennials or gen Zs, it is just clunky. It’s terrible to use. If you’re on, excuse me, but Volusion or Yahoo-

Jon:
Yeah.

Ryan:
… sometimes those sites tend to be older. But it’s worked really well for a long time for them. And their demographic has aged with it, and it’s just easy for them. Then they go hire a web designer because they’re like, “Look, we need to upgrade the site. We need to keep up with the organic algorithm and mobile site speed.” So we do it.

Jon:
Yeah.

Ryan:
But they hire a development agency that’s got younger people that don’t resonate with their target market. And they get a great looking site, maybe if you’re targeting me.

Jon:
Mm-hmm (affirmative).

Ryan:
And I’m not going to your site to buy it. And unfortunately you worked yourself into a beautiful site that doesn’t convert.

Jon:
Yeah.

Ryan:
And because you didn’t think through, at least if I’m going to get a better speedier platform, I need to make it work for my demographic. Even if it doesn’t look pretty to the designers-

Jon:
Right.

Ryan:
… that are actually building it.

Jon:
And that’s a great point.

Ryan:
Yeah. So if it’s not easy, if your web design team doesn’t understand your target demographic. Well, you are going to for sure want to do something like a rapid test with your target demo.

Jon:
Mm-hmm (affirmative).

Ryan:
I mean, there’s too many risks involved, especially if you are a site that has enough revenue that it would be losing, it would be painful.

Jon:
Yes, great point. And that’s why I keep going back to that insurance policy term, when maybe it’s not a 100% accurate. It is, I think a good representation of what you’re looking at with this.

Ryan:
Yeah. I think it’s too often, you can’t make that many assumptions. I mean, we’d all like to make assumptions because it makes our life easier and we just assumed, hey, it worked here, it worked here. It’s going to work here.

Jon:
Mm-hmm (affirmative).

Ryan:
But rapid testing can fix that. But then even then I think, you’ve got products for companies that are small, that can help small brands that need to take some baby steps to start taking some best practices before they can get to real levels of CRO. That I think they need to be considering as well. Great, rapid test can get you going in one direction. But it’s not going to be the iteration of testing, and heat mapping, and the best practices that your SMB product does provide for.

Jon:
Exactly. And so I think this is where optimization is headed from a volume standpoint. Because it really helps, one of my goals has been to democratize CRO. Right? Bring it down from that Fortune 500 boardroom style, and budget and investment level, and time required. How do I bring this to the masses? And one of the ways we’ve been doing that is this rapid testing. And that’s been working out very, very well with us. You mentioned that the artificial intelligence eye tracking has been one way we can do that. We have a user testing site that has brought the user testing overall from an enterprise level down to what SMBs can afford, without that massive annual contract, and all the tool sets, and everything else that you don’t need. To just be able to submit and say, “Hey, I want five user tests on this site.” And we’re able to provide that for you at a reasonable cost.
So there’s a bunch of these things that we’re doing. And I think rapid testing is one of those blocks that I’m really, really excited about. And when you start combining that with things like launching new books, and sharing all of our content out there. We just have this great library of what we’re providing that is a whole suite of things that an SMB site can use. And so that’s really been my goal, is how do we do that at a price that’s affordable, and in a way that’s understandable. And so hopefully this is another one of those tactics

Ryan:
And I can attest you’re doing a good job. We at LP have a lot of smaller clients, but they still find a lot of value in the things you’re doing. And I find a lot of value in helping smaller businesses grow big. I think it’s just more exciting. And yeah, I mean, any agency can probably work with Nike or Adidas and see them grow. They’re Nike and Adidas.

Jon:
Right.

Ryan:
But taking small clients to say, “Man, you’re doing a 1 million last year and now you’re doing 15 million.” That’s just exciting to see that impact it has on a business owner too.

Jon:
Yeah.

Ryan:
Rapid testing, almost everybody can leverage it. And should be leveraging it at some point, before they make massive decisions when regards to their websites. Or even just if you’re gambling on holiday promos, you can take a lot of the risk out.

Jon:
Yeah. Awesome! Well, thank you Ryan. I appreciate the wonderful questions on this today.

Ryan:
Thank you, Jon.

Announcer:
Thanks for listening to Drive and Convert with Jon MacDonald and Ryan Garrow. To keep up to date with new episodes, you could subscribe at www.driveandconvert.com.

About the Author

James Sowers

James Sowers is the former Director of The Good Ventures. He has more than a decade of experience helping software and ecommerce companies accelerate their growth and improve their customer experience.