Benchmarking and comparative testing follow best practices in user experience and help ensure success in the marketplace:
- Benchmarking establishes baseline usability metrics for your product, setting the bar against which to compare and measure the improvements in future releases. Benchmarking tests the released product or its final designs.
- Comparative testing (also called competitive testing) evaluates the strengths and weaknesses of two or more products or designs based on users’ actual experiences. It can gauge how well your product measures up to the competition, or focus on competitors’ products only to generate ideas for new features and functionality.
Both benchmarking and comparative testing measure how well users perform key tasks: careful planning ensures that tasks performed with competing products or designs are as similar as possible. The focus is on quantitative (observational) data, although TecEd collects qualitative data as well for deeper insights into users’ behavior. Both methods call for larger numbers of participants, from as few as 20 or 30 to several hundred, depending on the level of statistical confidence required. Sessions may be moderated or unmoderated. Rigor is a must to ensure reliable, unbiased results.
Examples of Benchmarking and Comparative Data
Benchmarking and comparative usability testing typically collect:
- Task success and failure, and reasons for failure.
- Time on task.
- Number of clicks to complete a task, and the paths followed.
- Number and types of errors.
- Number of assists: for example, how many times participants use online Help or say they would call customer support.
- Ratings of ease of use, user satisfaction, and so on.
- Product preference (comparative testing only).
Benefits of Benchmarking and Comparative Testing
Benchmarking and comparative testing combine benefits from lab testing and unmoderated testing. Above all, these methods quantify the user experience so business and design decisions can be based on data rather than the opinions of stakeholders, designers, or developers.
Choosing TecEd to perform your benchmarking and comparative testing brings additional benefits:
- Strengthens the unbiased quality of the results, because the testing is conducted by a neutral third-party research firm.
- Allows you to focus on translating and applying the results and recommendations to your product.
When you need rigorous benchmarking or comparative testing, do what eBay, Google, Intuit, MSNBC, Team Detroit/Scotts Miracle-Gro, VMware, and Yahoo! did: Turn to TecEd.
