Usability Testing

What is Usability Testing?

Usability testing is the process of working with end-users directly and indirectly to asses how the user perceives a software package and how they interact with it. This process will uncover areas of difficulty for users as well as areas of strength. The goal of usability testing should be to limit and remove difficulties for users and to leverage areas of strength for maximum usability.

This testing should ideally involve direct user feedback, indirect feedback (observed behavior), and when possible computer supported feedback. Computer supported feedback is often (if not always) left out of this process. Computer supported feedback can be as simple as a timer on a dialog to monitor how long it takes users to use the dialog and counters to determine how often certain conditions occur (ie. error messages, help messages, etc). Often, this involves trivial modifications to existing software, but can result in tremendous return on investment.

Ultimately, usability testing should result in changes to the delivered product in line with the discoveries made regarding usability. These changes should be directly related to real-world usability by average users. As much as possible, documentation should be written supporting changes so that in the future, similar situations can be handled with ease.


When to Begin

This process can and should begin as early in the development process as possible. It need not involve a large number of users, in fact 3-10 is probably ideal depending on the scale of the product. These users should ideally be intended users for the product (ie. alpha/beta testers) and should represent a decent cross section of users targeted by the application.

The real question should be "when to end". I believe that this is an incremental process with many beginnings and endings. Ultimately, the job is not done until the product has reached the end of its lifecycle (not just development cycle). Each incremental step should be relatively short with changes and supporting documentation made often (up until initial delivery of the product.) Once a product is initially delivered it can be difficult to make these kind of changes without affecting active users. Once the product is shipped, changes should be considered more carefully with special concern for how active users will be affected and how future users will benefit.

However, it is never too late to start. Even if you are nearing the end of the development cycle, usability testing can still yield enormous results. Sometimes even minor changes to the UI, help system, reports, etc can make the product more appealing to users.


How to Begin

Usability testing can be quite simple. There are 4 basic ingredients that are essential to success. These are

1. The usability testing person/team needs to include a software developer who is open-minded about changes and not offended by criticism. The goal of usability testing is not to criticize, but improve and learn. If any member of the team is not ready to receive criticism with an open-mind, the testing will almost certainly fail. This person needs to have a good working knowledge of the workflow process the application is designed to facilitate and needs to have good communication skills. Good note taking skills are also essential.


2. The users selected as test subjects need to be average users (not all power users, and not all entry-level users.) They should understand what the application is designed to do and should be able to communicate their needs reasonably well. Often, these users will not be able to communicate their needs in technical terms, but will be able to identify problems they are having.


3. The development person/team needs to be prepared to make changes on an incremental basis in relatively fast paced environment. The best situation is to make needed changes quickly so that those changes can be incorporated into the continuing testing process.


4. Patience. The usability testing and refinement process can take some time and will sometimes go in the wrong direction. By keeping changes small and incremental, it is usually easy to back-track and rework the problem without significant setbacks.

There are 3 methods of feedback that need to be incorporated into our testing. These are:

1. Direct user feedback. This type of feedback usually occurs by having the test users use the software alone and report back to the usability team/person. Ideally, reporting back should occur on a regular basis (daily or weekly at most).


2. Observed behavior. This type of feedback can occur in conjunction with direct user feedback. This occurs when the testing team/person observes how users use the software.


3. Computer supported feedback. This type of feedback can occur on an ongoing basis throughout the testing process. As mentioned this is usually quite simple involving timers and hit counters.

Each of these feedback methods should be used to achieve the ultimate goal.


How to best Leverage Feedback Methods

Direct Feedback

1. Provide users with notebooks, pre-printed forms, a simple program, web page, etc to record their problems as they are having them. They tend to forget the details if this is not done.


2. Regularly review the issues users report.


3. Meet with users on a scheduled (somewhat) basis to discuss their issues and make sure that you fully understand it before proceeding. Be prepared for this meeting by reviewing their issues prior to the meeting.


4. Keep a good communication dialog open with all users.


5. Prioritize direct-feedback issues highly. Users need to see results relatively quickly or they get discouraged with the process.

Observed Behavior

1. Use multiple opportunities to observe behavior. Whether you are training them, discussing problems found earlier, or just walking past their desks, these are opportunities to observe their behavior. (I doubt many of us have 1 way glass to watch users, so I won't go into that here.)


2. Take notes (use same forms, software, web pages, etc as users use if possible. This will insure you do not forget what you observed.


3. Compare the notes you are taking against the notes users are taking. If users are not reporting nearly as many problems as you are finding, it is possible that they are not comfortable with the process yet.


4. Keep a good communication dialog open with all users.


5. Don't interfere with their normal process. The goal is not to train them on the "right way", but rather to have the software work "their way".


6. Be prepared to misinterpret user behavior. Sometimes you might observe a user having an apparent problem, but in-fact you are misunderstanding what they are doing.


7. Prioritize observed behavior after direct-feedback. Users are not expecting these changes and they can cause confusion. Carefully review your observations and discuss them with your team and the users.

Computer Supported Feedback

1. It is hard to know where to use computer supported feedback early in the development and testing cycles. Therefore start simple and grow from there.


2. Be careful that the code that supports this does not interfere with the users workflow and by all means does not crash the software. (Been there, done that.)


3. If possible, log all computer-feedback issues into a simple database (Access worked well for me.)


4. When reviewing the log, be very careful to not overlook issues and not misinterpret the data. (Your method need not be statistically valid, just reasonable.)


5. When you see an issue in the data, try and support it through direct feedback and observed behavior methods. This data can be very helpful in knowing what to look for when working with users.


6. Consider leaving this capability in the final product carefully. Only do so, if the log can be disabled safely and it has been thoroughly tested against all issues (especially exceptional growth in log size.)