Using Test Results to Improve the Application
You need to test five to eight subjects to make sure that you've observed a high percentage of the usability issues in your application.
The outputs of your usability tests consist of lists of usability issues observed during the tests. For example:
"User didn't understand what the main menu said and hung up."
"User accidentally hit DTMF 3 and didn't understand the error message."
"User said, 'That's really dumb,' when he heard the help message."
If you have a typical application and one or two people to assist with observations, you might end up with a hundred or more usability issues. You need a good method for summarizing the results and deciding, in conjunction with other development team members, how to use the results to improve the application.
The following list briefly outlines a process to group, prioritize, and get agreement on resolutions for usability issues:
Use the affinity analysis method to group issues that are closely related.
Eliminate duplicated issues.
Each team member should assign each usability issue a priority number:
Must fix before any user touches this application.
This is a significant usability issue that should be fixed before the application is deployed.
This can wait until a later release of the application.
This is not a usability issue.
Facilitate a discussion to resolve priority disagreements, if needed.
Develop strategies to resolve all priority 1 issues as well as all priority 2 issues that can be fixed within the available time.