- Developing a VoiceXML Code Prototype
- Overview of a Usability Testing Process
- Designing the Usability Test Scenarios
- Using Test Results to Improve the Application
- Completing the Usability Test Cycle
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Designing the Usability Test Scenarios
Begin the process by developing simple usability test scenarios to run against your VUI script prototype. Those scenarios should be reusable, but they're probably not developed to the level of detail needed to use with the application prototype. Additional scenarios will probably be needed to test key aspects of the prototype.
If you've had exposure to usability testing or human factors work in the past, you may be concerned about your ability to design, develop, and conduct usability tests. You may have visions of megabuck usability test labs and tests designed to track users' tiniest muscle twitches. Although detailed and scientifically accurate usability testing may be required for some applications, most applications can be greatly improved through the use of what Jakob Nielsen refers to as "discount" usability engineering in Usability Engineering.
To perform the usability test scenarios described here, you need a "test lab"—a quiet area free from distractions—equipped with the following:
Comfortable seating.
Access to your test platform.
A speaker phone, if your test platform supports dial-in access. If a speaker phone degrades your speech recognizer's accuracy, you need to come up with an alternative way to monitor application output.
A videotape recorder, preferably set up so that it can record both sides of the users' interactions with your application.
An assistant observer who helps you record usability issues during the tests.
This section describes a low-cost version of usability testing that reveals the majority of user experience problems so that you can resolve them before your application is made available to users. The method uses simple observation and notation methods, in conjunction with two levels of usability test scenarios:
High-level scenarios. Scenarios designed to assess users' initial reactions to the basic features and functions of your application.
Low-level scenarios. More detailed scenarios that specifically test aspects of your application that have high impact on usability.
High-Level Usability Test Scenarios
High-level scenarios assess the usability of an application at the point where a user first contacts it and first contacts major functional areas within the application. High-level scenarios are not too popular with some human factors purists, but they're important in understanding how users feel when they initially access your application or its key activities.
VoiceXML applications are much like web sites. Users abandon them quickly if they have a bad experience or don't perceive the application as valuable. Because it's almost impossible to entice users to try an application again after they've abandoned it, you want to do everything you can to give them a good initial experience.
High-level scenarios should be simple. Their goal is to give users exploratory access to your application so that you can observe how they react. The following sections list examples of high-level test scenarios for the VoiceXML book and its t-shirt sales site.
Access the Application
Give the user a brief explanation of the test: "You are going to dial a voice application that will offer to sell you books and related products. Just dial the number and listen to the prompts. You can experiment if you like. Hang up as soon as you want."
The purpose of the test is to recognize users' "gut reactions" to the application. Observe what they say and do in response to the application. Did they quit quickly, or did they navigate through the application?
Debrief test subjects immediately after the test to find out what they thought about the application's purpose, ease of use, voice quality, and so on.
Record each usability issue for evaluation later.
Navigate the Main Menu
Give the user a brief explanation of the test: "You are going to dial into the voice application and purchase one VoiceXML book. If you have problems, you can try to continue, or just hang up."
The purpose of the test is to recognize users' abilities to perform basic navigation. Observe what they say and do in response to the application. Did they choose to use spoken input, or did they use DTMF? Did they look confused? Did they just hang up?
Debrief test subjects immediately after the test to find out what they thought about the purchasing process.
Record each usability issue for evaluation later.
Optional Tests
You may want to have a few optional high-level tests available for follow-up purposes. For example, if all the test subjects used DTMF, an alternate version of the "purchase a book" test that specifies using spoken input. Or, if all the test subjects used spoken input, an alternate version of the "purchase a book" test that specifies using DTMF input.
Low-Level Usability Test Scenarios
Low-level, detailed scenarios test specific activities in the application. In general, design tests that ensure that every activity in the application is tested to completion of the activity.
In addition to basic application activities, following are some other application functionalities that should be specifically tested:
Access to help
Understandability and desirability of all voices (TTS and recorded) used in the application
Error messages and error recovery
Acceptability and usefulness of treatments (such as "filler" sounds or advertisements) played while the application is waiting during processing or external data retrieval
Acceptability of typical times through the application for key tasks
These functions can be tested within the scenarios you design for basic application activities. The key issue is that they are tested.
Example: Low-Level Usability Test Scenario
We typically develop two versions of each test scenario:
A simple version that states the task in the users' terms
A more detailed version for the usability tester that includes specifics about application setup, timing you want to be sure to measure, prepared debriefing questions, and note-taking space
The user version of the test scenario might include the following steps:
Access the application at 1-800-555-1212.
Use the application to add one book and one t-shirt to your order.
Remove the t-shirt from your order and complete purchasing the book.
The tester version of the scenario might include these steps:
Check to ensure that the application is up and running okay.
Ensure that the user selects both a book and a t-shirt.
Ensure that the user removes the t-shirt before completing the purchase.
Ask the assistant to time the process from successful connection to the application to completion of the purchase.
Watch for any problems in removing the item from the order.
Ask debriefing questions:
"Was it clear to you what you needed to do to remove the t-shirt from your order?"
"Was it confusing to return to completing the purchase after you removed the t-shirt?"