How to Conduct High-Impact User Testing: Part 1 – Thoughtful Panel Selection

User testing is the best way to know how people are interacting with your website or product. This Insight takes a closer look at how to properly screen and select users for your testing panel.

This is part 1 of a 3-part series titled, How to Conduct High Impact User Testing. If you enjoy this Insight, make sure to read Part 2: Formulating Tasks and Questions.

User testing is commonly used to reveal a host of usability issues and quick wins for your site. While this is a great application of user research, by the time you’re ready to conduct a test, you often already know the general behavioral metrics (e.g., time on page, exit rates) that indicate key pain points your users are experiencing. 

However, with the context of quantitative and demographic data, well-executed user testing has the potential to reveal not just general usability issues but insight into the expectations of your unique audience. We put together some guidelines to help you get the most out of your user testing program and start improving the user experience on your own site. 

In this article, we’ll cover:

  • The difference between user testing and user research
  • How to screen for a relevant audience 
  • How to select for a fresh perspective
  • When to talk to your core audience vs. niche segments with differing needs 

How Is User Testing Different from User Research?

User testing and user research are two terms that are often used interchangeably. User research is seen as an umbrella term that refers to a variety of methods for researching your users, including interviews, surveys, focus groups, and user testing. 

User testing is a sub-division of user research that allows you to validate design decisions, reveal common pain points, and inform a/b testing by asking members in your core audience to think out loud as they complete specific tasks on your site. During testing, the evaluator observes how users complete the tasks without assistance, and then draws conclusions from the users’ ability to complete the tasks. 

User testing is a highly effective method for identifying pain points in the user experience, but the success of your testing hinges on how you screen and select your users.

So how do you determine which users should be part of your test?

Use screener questions to identify your target audience

Reaching the right audience on your testing panel will open up a host of contextual insights, showing not just how users perform, but what motivates them to make decisions. If you approach user testing with the assumption that users in your target audience (or segments of your target audience) share some unique attributes, you’ll set yourself up to get more accurate test results and understand what your site is missing. 

Reaching the right audience on your testing panel will open up a host of contextual insights, showing not just how users perform, but what motivates them to make decisions. Click To Tweet

To choose the right users for testing, define your core audience demographics by mining your analytics for sample data on age, gender, and location; this will tell you who is converting and who is not at a basic level. Most testing platforms will allow you to screen for gender, age, and location simply by checking a few boxes. 

However, while screening for these demographics is a start, it doesn’t encompass all of the characteristics that are unique to your audience. 

Once demographics have been identified, take it a step further with carefully formulated screener questions. Answers to these questions will connect you to users with psychographic qualities that match your unique audience profile on a deeper level (e.g., price-insensitive homeowners) and provide insight that gets at the heart of why your target audience is not converting better. 

Here’s an example of a screener question that would weed out a group of potential testers based on psychographic characteristics:

What is your approach for shopping for home furniture or appliances?

  • I go low stress. I prefer to make easy decisions from a few options to save myself time. [Reject]
  • I take my time. I aim to find the best deal at the lowest price. [Reject]
  • I take my time. I aim to find the best product, and I prioritize quality over cost. [Accept] 
  • None of the above. [Reject]

Note: Be careful to formulate screener questions that don’t give away the “right” answer among the various responses. When filtering for users who won’t be stuck on the prices of high-end goods, for example, aim to make every optional response sound positive and appealing. Make sure you have a “none of the above” option when applicable so you’re screening out folks to whom the question doesn’t apply.

Avoid talking to users who have already experienced and “learned” your site 

When our strategy team is looking to learn more about how to convert visitors to buyers on a client’s site, we often prefer to first talk to users with a clean slate. Starting with an uninitiated audience will give you a good understanding of how a typical user will experience your site and highlight problem areas that those who have already “learned” your site may naturally overlook. 

To find new users with the right fit, screen for those who have heard of in-market competitors (which assures that you are talking to the right audience) but reject users who have already heard of the brand/site you are testing. Here’s a simple template you can use to screen potential users:

Which of the following brands have you purchased from?

  • Competitor 1 [Accept]
  • Competitor 2 [Accept]
  • Your site [Reject] 
  • Competitor 3 [Accept]
  • Fake company [Reject]
  • None of the above [Reject]

Start by talking to users with the highest intent to purchase

…and avoid users with no intent to purchase.

Screen for the most basic criteria, then try branching out. For instance, if you’re a direct-to-consumer brand selling personal care products online, you may want to ask users how often they shop online or how they shop. Your primary audience may be users who shop online at least once per week. That’s a good place to start. 

Screen for the most basic criteria before you try branching out.

Once you’ve decided which attributes are critical for relevant feedback, you’ll know what you need to screen for in initial testing rounds. For example, you may eventually want to run another test with a group of users who prefer to shop in-store, in bulk, once a month, to understand if there’s an untapped market you could be reaching and how to appeal to them. 

User testing is about more than usability—it’s about the thoughts behind the actions users are taking on your site. Click To Tweet

By treating these as discrete segments you’ll be able to rediscover their needs through iterative user testing cycles and personalize your site experience for your many unique audience segments.

User testing is about more than usability–it’s about the thoughts behind the actions users are taking on your site 

User testing is more than just a usability gauge. By defining your core audience, targeting unbiased users with aligned interests, and starting with those with the highest intent to purchase, you’ll set yourself up to talk to an audience with insightful feedback. 

But remember: Once your panel is set up, the work has just begun. A user test is only as good as the questions you ask, so if you’re finding that your user testing endeavors are not fruitful, check out part two of this series where we’ll walk you through formulating high-impact tasks and questions to get the most from user testing. 

Read next: How to Conduct High Impact User Testing Part 2: Formulating Tasks and Questions

About the authors:

Natalie Thomas is the Director of CRO & UX Strategy at The Good. She works alongside ecommerce and lead generation brands every day to produce sustainable, long term growth strategies.

Maggie Paveza is a CRO Strategist at The Good. She has over five years of experience in UX research and Human-Computer Interaction, and acts as expert on the team in the area of user research.

About the Author

The Good