Five (and a Half) Approaches to User Testing

This week, an organization asked me about our approach to usability testing. I proceeded to prattle on about how we like to keep it simple: lose the two-way mirror, constrain the scope of what's tested, test with small numbers of users, re-test often, blah, blah, blah.

After the meeting, I recalled Kira Marchanese's presentation, "Five and a Half User Testing Approaches." Kira is the Director of Internet Communications at the Environmental Defense Fund. If every web redesign team watched this slide show, I could spare them much of my yammering. In this presentation from Forum One's May 2008 seminar Putting Audiences First, Kira de-mystifies testing approaches. She explains that not only is usability testing affordable, but you often can't afford not to test. (Kira calls it "User Testing." Others, including our own director of user experience, prefer the term "usability testing.")

I'll summarize Kira's points here, but I recommend that you watch and listen to her presentation below. (Click the green arrow to hear her audio.) Kira points out that since many organizations will not routinely invest $15,000 to $20,000 on a single round of testing, there are cheaper ways:

| View | Upload your own

Approach #1 - Hire a Professional Under Your Own Roof: One of the largest costs of traditional user testings has been the cost of the facilities. If you have extra computers and a spare conference room, you can save a lot of money by letting a usability professional test on-site. You can also recruit subjects from your own contact lists. Call upon volunteers on your email list. Or if it comes to it, draft friends and family. Look for a spectrum of computer levels and a base familiarity with your work. But you need not obsess over your sample's representativeness.

Approach #2 - Do It Yourself: If money is significantly constrained, you may want to consider conducting the tests yourself. A drawback to this option is that you give up the perspective that an experienced unbiased facilitator can lend to the overall analysis. In addition, a skilled facilitator is much better at not biasing the test by inadvertently leading a subject. In addition, they are trained to draw out the reasons a user might be struggling with a given interface.

Approach: #3 - Go Virtual: Use a service such as www.usertesting.com. You submit a URL and their panel of testers does the task and records their activity. You get the screen recording with audio and written feedback. This best for simple, focused tasks. This is really cheap -- only $20/tester. So a full panel of ten tests is only $200 (plus staff time to review and analyze the recordings afterwards). The main drawback is that there is no moderator to probe and guide the subject or ask questions like "what did you mean by that?"

Approach #4: Go Virtual -- More Basic: In this method, you ask task-like questions in an online survey. You can run different versions of survey. Again works well for very focused tasks such as deciding on terminology or labeling. You can churn a few hundred users through a few variations.

Approach #5 - Go Virtual -- Still Basic: Card Sorting is a useful activity in which you have participants sort physical index cards into piles and label them. (Usability.gov has a good introduction to card sorting). You can do this online too using WebSort.net or similar. It allows people to sort items into folders and give them titles. It's about $50/test (plus staff time).

Approach #5.5: Don't Test: It is perfectly acceptable to skip testing when you realize that you already have good feedback, don't have dramatically new functionality, and need to get to market. You can always evaluate the solution further later.

Technorati tags: usability user experience web design nptech wes08may