So, we know that in user testing, we need representative users to perform tasks using our system so that we can see where they succeed and where they fail. A big part of this is that testing with the right people is critical. Because if you run tests with people who don't represent your actual users, you might see problems that your users aren't going to have and you might not see problems that your users are going to have because of the differences between who you test with and who's actually going to be using your system. So, a first step at any user testing scenario is that you need to know who your intended users are. Obviously, if you're working on a system, you're building a system, you are evaluating a system, you should know this already. But if you don't, this is a good time to figure it out. What you need to know is who are your intended users in terms of their expertise? Do they tend to be technically sophisticated or do they tend to be relatively new and inexperienced technology users? Are they experts in the particular domain where your system is operating? What are their characteristics? Are they young? Are they old? What languages do they speak? What nationality are they coming from? What are their behaviors that are relevant to the system? What kinds of things are your users going to do that the system will help them with? Sometimes you need to know what are the attitudes of your intended users. What are their attitudes towards technology for example or towards particular aspects of the domain that you're supporting? So, at a very basic level when thinking about the people who participate in your test, you need to ask, would they actually be users of your system? When setting up a user test, you need to be very clear and very explicit about your recruiting criteria. Your recruiting criteria are clear tests that tell you whether somebody is in or out of your target population. As an example, if you're evaluating a shopping site, you might want to know that people are regular online shoppers and you might articulate that by saying they have to have bought at least two items online within the past three months. This would be an example of a behavioral criteria or a criteria that's based on user's behavior. If we were evaluating an online learning site like edX, we might be interested in knowing whether people have experience with online learning. We might have a criterion like they've completed at least one assignment in an online course in the past year. This is another example of a behavioral criteria. If we were building an app that was meant to support healthcare professionals, we would want to make sure that our users were health care professionals, and so we might articulate our criterion as they have to be currently employed as a physician, nurse, or physician's assistant. This is an example of a criterion that is based around characteristics of the users rather than their behavior. If we were evaluating a privacy assistant or a system that might help people make decisions about how to set up their privacy policies and social media sites for example. We would probably assume that the users of such a system would be people that were concerned about privacy. We might articulate this by saying that in a questionnaire, our participants would need to self-describe as somewhat or extremely concerned about online privacy in order to be members of our target population. This is an example of an attitudinal criteria or one that is based on their attitudes, not necessarily their behaviors or characteristics. Sometimes it's worth making a distinction between the target population for a system and the recruiting criteria that we're going to use for a particular test. In the simplest case, the recruiting criteria are the same criteria that you would use for defining whether somebody was in your target population or not. However, in some cases, if the target population is very broad to a large system that serves lots of different users doing lots of different things, it can be better to conduct multiple focus tests using different recruiting pools. So, for example, if we were evaluating Amazon, we might note that Amazon does lots of different things for lots of different people, and that for example, Kindle users are going to be very different from people that are avid users of Prime video and we would want to recruit separately for those two things and run different tests for those two different products. If we were evaluating edX, we might note that there's probably going to be some difference between different types of learners that participate in the system. We might observe that casual learners of world history, for example, might have different characteristics, it might approach tasks differently than people who are trying to change careers into a high-tech field. When defining your recruiting criteria and your recruiting strategy for a particular test, it's important to make sure that your recruiting criteria and your test design are lined up with each other. The important question is, would these users perform these tasks? For large projects or where you're evaluating many aspects of a large system, you often want to come up with a testing strategy, rather than a single test design. You might end up running multiple tests with different foci and you might want to run both broad and focused tests with different populations and focusing on different tasks. Regardless of how you define your recruiting criteria, you're going to end up with some measures of diversity within the target population that you're looking to recruit for a particular test. You want to think systematically about the differences that you would expect to see in your recruiting population and how those differences might impact the way that users would interact with the system and how they would approach the task that you're asking them to do. You should state those expected differences as diversity criteria or the criteria along which you expect variation to occur. Ideally, if you've defined your diversity categories or your diversity criteria, you want to recruit across those different categories and make sure that you have representation across those different aspects of diversity. Sometimes this can be difficult, especially when you're running a fairly small user test and you're not going to end up recruiting that many participants. Even then, it's important to capture the diversity dimensions or the ways that your users do vary across those diversity dimensions so they can help to inform your analysis later. So, to return to our earlier examples, if we were evaluating a shopping site, we had our recruiting criteria which is that users needed to buy at least two items online within the past three months. But how might people who meet that criteria vary within themselves? Well, we might look, for example, at people who are extremely frequent shoppers. We might call them top shoppers. These might be people who bought more than 20 items online in the past three months. Maybe we have a middle category frequent shoppers who bought 10 to 20 items online, and the occasional shopper who bought two to nine items online. We might want to look at how all of these different types of people interact with the system because we might expect to see differences in what they expect, how familiar they are with the shopping process, and how they perform on our tasks. To return to our healthcare app example, we have the recruiting criteria that our users need to be currently employed as a physician, nurse, or physician's assistant, but we might think about the different ways that users could vary within that general description. We might want to consciously recruit physicians and nurses and physician's assistants and make sure we have representation across those categories. We might also think about whether it matters if they work at a small clinic versus a hospital that could change the way that they interact with health technology for example. We might want to get people from different specialties, whether that's family medicine, or cardiology, or other types of medical professions because these people might have different interactions with a healthcare app as well. Even when we have a clear idea of who we want to recruit for our user test and what types of diversity we would like to see in our participant pool, the realities can sometimes force us to make compromises. So, you may have identified your ideal, recruiting criteria, you've identified your important dimensions of diversity, but it turns out you have limited access to potential users and this is often the case. In these cases, you may need to relax your criteria a bit. That's okay to do; however, it's important to recognize when you're doing that so that when you're reporting the results or when you're analyzing the results, you can take them with a grain of salt and you can understand the differences between what you might have seen in the user test and what might happen with the users who you expect to use the system once it's released into the wild. There are a number of different recruiting strategies that we can use when pulling together participants for a user test. I'm going to go through a few of them here. The first is friends and family, also co-workers, and so basically, using our own social network to recruit people to participate in a user test. Another is proxy or snowball recruiting where we basically use somebody else's social network to fill out the participants for our test. We can use public advertisements where we cast a wide net to try to get people to participate and we can use professional recruiting firms to get very targeted participants when that's something that we need for a particular test. Friends and family recruiting is a form of convenience sampling, which basically means we take what's easy for us to get. It's not necessarily the best-recruiting strategy in terms of getting the most representative users, but often given time and resource constraints, it's something that we need to do. Sometimes your friends or family or co-workers do in fact fit the profile for your target audience. More often, they don't exactly or at least they may not be fully representative of the diversity of users that you would see. But given the choice between recruiting friends and family and not doing a user test, it's definitely better to do something like this. One form of friends and family recruiting that is often used is, testing internally to a company or an organization, which not only is more convenient, but is sometimes preferred by organizations when they're trying to maintain secrecy around the system that they're building, and they're not ready to make it available to the public. Proxy recruiting is when you use a contact to provide access to their social network to help you recruit users, and this is often required when you're trying to recruit users with a particular characteristic, a particular specialty, that isn't well-represented within your social network. For example, if you're building an app for doctors and nurses, and you didn't really know a lot of doctors and nurses, you would probably want to work with a collaborator who did work in the medical field to give you access to some of the people that they know, in order to participate in your test. In these cases, cultivating the relationship with your proxy, becomes a central concern. You need to convince them of the value of your test and the value of giving you access to their social network, and that becomes a major task as part of your recruiting strategy. Working with a proxy might impose some limitations. Their level of access to the users that you want to recruit may impact your recruiting criteria, and just as with friends and family testing, you may have to relax those criteria a bit, based on the need for convenience and the need for access. Snowball recruiting is a form of recruiting where you ask each participant that you recruit for your test to introduce you to others that have those criteria. This is another form of expanding your social network beyond the friends and family and coworkers that you already have, and gaining access to people that might have the characteristics, behaviors, and attitudes that you need for your user test when you don't have direct access to them yourself. Another approach that can mean more time and labor intensive, is to advertise publicly. To basically cast a wide net to try to inform a lot of people about the user test that you're running, and try to get them to come to you and volunteer to be part of your test. The specific form of the public advertisement can be varied, it can range from posted flyers on bulletin boards to online ads, to a blast, to a mailing list that might contain a lot of people that would be a part of your target population. This approach can be effective if you can identify a target rich area. So, for example, you can find places to post your flyers where lots of members of your target population go through, or you can target your online ads to places where your participants are likely to congregate, or you can identify a mailing list that has the people that you'd be interested in. If you're broadcasting an invitation to participate in your test to a wide population, it will often require you to add an additional screening step where you ask people questions, whether that's in an interview or via an online questionnaire, that will help you narrow down whether they actually meet your recruiting criteria. When advertising to a broad population, the wording of the invitation and the incentives become very important, because you don't have that social contact with the individuals, as you would if they were friends or family. A final approach to recruiting for user test is to enlist the services of a professional recruiting firm. So, basically, you can pay a service to deliver users to you that meet the criteria that you give them. Typically, these firms specialize in what's called focus group recruiting. And so their main job is to recruit focus group participants for market research studies, but many of them are familiar with user testing and can provide users that would be appropriate for a test that you want to run. When working with these firms, you need to specify your recruiting criteria, you often need to provide them with a screening questionnaire that they can use to make sure that all of the people they recruit are valid participants for your test, and they do the rest. So, this is great, it's very convenient and it can result in a highly targeted pool of participants for your test. The downside is, they tend to be quite expensive. When you ask people to participate in your user test, you're asking them to give up some time, usually about an hour, and in return, it's common to compensate them in some way. The details of compensation can vary quite a bit by the locale, by the test duration, is it a 30-minute test, is it an hour, is it a 90-minute test, and also by the user population. In many cases, when conducting internal tests, little or no incentive is needed. Often, when testing internally to an organization, little or no incentive is needed, and that's because they're getting paid to do this as part of their job. When recruiting professionals, you often need to pay according to what they're worth or what their typical wage is, which means that if you're recruiting highly paid professionals, it can be a fairly expensive proposition to recruit them for participating in your test. If you're recruiting people in non-work context, so this would apply to friends and family, but also when recruiting for apps that are used in a leisure context or things like that, often, special items or gift certificates that are related to your user populations interests, may be even better than cash, because it may be more appealing to them and doesn't cause them to think about how much their time is actually worth. There's a lot more that could be said about recruiting participants for user studies. I'm just going to point to a couple of references that you could go to for additional reading. The first is a report that was released by the Nielsen Norman Group, on how organizations currently recruit participants for usability studies, and they've distilled 234 tips and tricks for recruiting users. So, there's got to be something in there that can help you. Another reference I recommend, is the Handbook of Usability Testing by Jeff Rubin and Dana Chisnell. This is an excellent resource for all aspects of usability testing, but they have specific recommendations for recruiting that I think are very helpful. To summarize, recruiting is extremely important for user testing, but it's not necessarily easy. In fact, it can be one of the most challenging aspects of running a successful user test. As a result, we often default to convenience sampling or recruiting from within our social network. If this is your only option, given the resources and time that you have available, that's okay. Some data is definitely better than none. However, you should make sure that you have appropriate confidence in your findings. So, if your user test participants differ systematically from the users you expect to be using your system when it's released to a broader population, you should be aware of that and think about how what you found might or might not apply to that broader population. It's very important to make sure that the test that you design and the participants that you recruit are aligned. So, if you have to relax or change your recruiting criteria, you should make sure that the test you've designed is appropriate for the people who are actually going to have participate in your test. When working on larger projects, you probably want to think in terms of planning a strategy for multiple tests to cover the different user populations and the different tasks that people need to perform, and not just about designing an individual test. So, even though recruiting is a challenging aspect of user testing, it's worth the effort to define specific recruiting criteria and do whatever you can to recruit participants who really meet that criteria. It will result in a much more valuable test that gives you a lot more information about how well your system is going to work for users who will be using it, when it's released to the broader population.
