Unmoderated Testing

Unmoderated usability testing combines aspects of remote testing and online surveys to collect quantitative data to guide business and design decisions. Unmoderated testing is ideal for usability studies that require large numbers of participants, including benchmarking and comparative testing.

For unmoderated testing, TecEd designs the usability study as we normally would, but then we set up the study using an online tool. The tool automates the sessions, providing our instructions, task scenarios, and post-task questions to participants as needed.

Participants perform the test tasks in their own homes or offices as the tool records their behavior using a website, prototype, or static mockups. After completing each task, participants answer questions about their experience before moving on to the next task. The tasks provide a realistic context for the questions, while the questions improve the rigor and success of unmoderated testing.

Because sessions take place continuously and simultaneously, hundreds of people can take part in the same amount of time it would take TecEd’s researcher to conduct a few participant sessions. The researcher later analyzes the data collected by the tool and reports the results.

Examples of Quantitative Data

Unmoderated usability testing typically collects:

• Task success and failure, and reasons for failure.
• Time on task.
• Number of clicks to complete a task.
• Task clickstreams, the paths used to complete each task.
• Click heatmaps, a visual representation of where and how often users click on each screen.

TecEd’s post-task multiple-choice questions capture additional quantitative data, such as ease of use and satisfaction ratings. We also include open-ended questions to gather qualitative data in the form of participant comments.

Benefits of Unmoderated Testing

Unmoderated testing is useful for achieving:

• Objective quantitative data, which is vital for convincing engineers and executives.
• Statistical confidence through large numbers of participants.
• Broad geographic coverage, whether across a country or around the world.
• Greater convenience and more natural behavior for participants, who complete their sessions from the comfort of their home or office, using their own computers, when their schedule permits.
• Comparisons across user groups, to see how their experiences differ.
• Cost-effective research, with study design and implementation costs spread over hundreds of participants and data analysis streamlined by online tools.

Unmoderated testing is most effective when the tasks to be performed are relatively straightforward and the questions to be answered are fairly specific. For complex products and tasks, moderated testing sessions—conducted in the lab, in the field, or remotely—yields a deeper understanding of user needs, goals, and expectations.

TecEd can help you define a balanced user research program that includes both moderated and unmoderated usability testing. We can use moderated testing to identify issues to investigate in quantitative research, or to refine design ideas before unmoderated testing validates the new design. Alternatively, we can use unmoderated testing to uncover issues for usability testing with a TecEd researcher, or to explore the reasons behind participants’ problems and perceptions.