How Pilot Testing Can Save You Money (and Boost Conversions)

When optimizing your site, user testing can be a significant investment. It’s essential to get it right the first time. That’s why pilot testing is so important. Pilot testing ensures that a user test is calibrated effectively to provide you the best user feedback.

Let’s consider the importance of pilot testing prior to launching a usability study.

Pilot testing ensures that a user test is calibrated effectively to provide you the best user feedback. Click To Tweet

This brief overview could save you thousands of dollars and help get user experience data into play weeks earlier than if you would’ve opted for no pilot testing. That, in turn, can lead to a boost in your conversion rate and a surprising growth in revenue.

Here’s a typical scenario:

An ecommerce VP calls The Good anxious to find out what’s wrong with the results the marketing department is getting from user testing. They’re doing everything just right – according to the book – but gains are dismal.

Invariably, there are three primary complaints:

1. The UX tests that the team spent weeks developing are proving largely ineffective. Insights gained are negligible at best.
2. Users are generating overly simplistic responses. They aren’t thinking hard enough about the issues being observed.
3. Users aren’t getting the process. They need guidance to complete the testing. That’s skewing the results and frustrating the team.

We almost always need to ask just one question to get to the root of the problem: Did you conduct pilot testing in advance of the user testing?

And we almost always get the same answer: “What do you mean by ‘pilot testing’?”

In this article, I’ll cover the essentials of pilot testing: what it is, why it’s important, and how it can make a major difference in your user testing results. Stick with me a few minutes and you’ll understand why pilot testing in marketing is an often-overlooked piece to the conversion optimization puzzle, and how you can leverage it to get a jump on the competition.

What is pilot testing?

Pilot testing is a quality control measure for marketing research and testing. It’s a dress rehearsal conducted to ensure the instructions are clear, the equipment is working properly, and there are no errors in the script.

Pilot testing is a quality control measure for marketing research and testing. Click To Tweet

By making a dry run through your testing protocol with just a few representative users, you can catch any problem areas before bringing the entire group on. In essence, the practice test makes sure the path you want testers to take is free of obstructions.

Here’s the key difference between beta testing and pilot testing:

• Pilot testing is typically performed with the help of only a handful of users. It’s aimed at getting the bugs out of the testing system prior to its full deployment to collect feedback on its ease of use and performance.
• Beta testing calls in many users to try a system pilot testing has already cleared. You know the big mistakes and points of confusion have been cleared. Now you’re fine-tuning the process.

How pilot testing saves your money and time

Pilot testing reduces the chance of errors in your experiments. It leads to higher quality tests by finding and correcting issues with instructions, formatting, and procedures. Pilot testing is the quality assurance step in your experimentation process.

Pilot testing is the quality assurance step in your experimentation process. Click To Tweet

Usability testing costs probably aren’t eating up the lion’s share of your marketing budget, but they can add up – depending on the number of users you’re paying and the costs of test development and review.

Here’s the rub:

When there are mistakes in the formulation of your test, then you may end up having to toss out the results tallied. Pilot testing helps you get it right the first time. That reduces expense and speeds up the process. It also saves headaches.

Pilot testing isn’t the main test, and the results from your pilot should never be included in your UX observations. Pilot testing is like warm-ups before an athletic event. It’s a way of making sure you’re ready to go when the action begins.

Pilot testing checklist

A pilot study to get the real test in order would seem like a no-brainer. There’s a tendency, though, to assume that testing sequences have already been tested by the project designers.

Here’s the problem:

The design team is too close to the issues they’re testing and too familiar with the results they expect to see. That leads to an unavoidable bias.

At The Good, we find most companies who think they’re running a pilot test… aren’t.

Use the following checklist to make sure your pilot test really does prep the main test instead of adding more confusion.

• Pilot test subjects should qualify as candidates for your primary UX test. They should fit the desired profile, and they should never be members of the team that designed the test you’re piloting.
• Pilot testers are disqualified from then taking the main test. You don’t need a lot of pilot participants. Most of the time, one to three subjects are plenty.
• Divide the test into logical sections and observe the time required to work through each. That will help you pinpoint trouble areas and adjust where needed. You don’t want participants to feel rushed. You want as much task-centered, focused feedback as you can get.
• Remember that the purpose of the pilot study run-through is to test the test. You’re looking for buttons that don’t work properly, instructions that aren’t clear, type that’s difficult to read – anything that hinders subjects from concentrating on the tasks they’ve been given to complete.
• A flipside to pilot testing is that you’re also looking for sections of the test that are too easy to perform. Are the instructions inadvertently leading subjects in a predefined direction? If so, that’s not a test at all.
• Don’t conduct the pilot test immediately before the “real test.” You want to provide a buffer of time to allow test designers to make necessary corrections, and there are almost always changes required.
• Keep pilot test subjects in the dark about being pilot testers. They should be given the same instructions and treated in every way just like the “real testers” will be treated. Avoid bias as much as humanly possible.

Save money and time with pilot testing

Pilot testing before conducting usability studies is one of those parts of the plan that seems so plainly required that it’s easily overlooked.

Don’t just ask the test designers if they’ve conducted a pilot study. Go over the checklist provided above with them to be sure you’re talking about the same thing.

Getting team agreement that you’re ready to go isn’t the same as observing a user who’s never seen the test before to make sure the project is optimized for the right people.

For more information about pilot testing, user testing, or conversion rate optimization in general, contact The Good.

Resources:

• How to Test Your Website’s Usability Prior to Launch
• 10 Steps to Choosing the Best Testing Platform for Your Website
• Top Usability Testing Tools: The Complete List (82 Tools & Free Spreadsheet)