Usability testing

What is it?

Participants are asked to complete one or more tasks while using a service. There are two types of usability testing sessions: moderated and unmoderated.

• Moderated. Throughout the test session, a moderator asks the participant to talk out loud while completing the tasks. In a separate space, the team watches the session and takes notes via video feed or through a one-way window. A moderator is present throughout the session to guide the participant on what needs to be tested.
• Unmoderated. An unmoderated session is a series of tasks that are inputted into a testing system ahead of time. Then the participant is asked to go into the system to complete the tasks on their own. Although it is more convenient to plan compared to moderated testing, the downside of unmoderated testing is that if a participant gets stuck or does something unexpected, you won’t be able to probe in real time.

This guide will only cover moderated testing. Although it takes more time to plan and execute, it is beneficial for the following reasons:

• you get the benefit of involving stakeholder and team throughout the session
• you can clarify and ask questions in real time
• you have time to ask follow up questions
• you have the flexibility to mix-and-match other user research techniques since you already have the participant present in the session
• if you have complex products that may not be straightforward to test, a moderator can observe and pivot quickly so there’s time to test other important aspects to test on

Purpose of usability testing

Usability testing is about understanding user’s intention when interacting with the product/service. Questions to ask while usability testing is happening include (but not limited to):

• What are the user’s goals?
• How can we better achieve the user’s goals?
• What would the user do if we are not around?

In terms of brand impression for users, what turns out to be the most important isn’t the look and feel of the site, it’s whether when users come to the site, they can accomplish what they want.

Christine Perfetti

Planning your tests

Leading up to your usability test sessions, you will need to consider:

Roles and and responsibilities

Each member participating in user research will have specific roles and responsibilities.

A prototype

The purpose of usability testing is to put something in front of the user and watch them work. This could be a prototype, working website, physical products.

User goals and tasks

To test if your user interacts with the prototype the way it was designed to be, the user will need instructions on a list of goals and tasks to complete while interacting with the prototype.

Number of users to test

The objective is to test up to a point where you can identify trends and patterns and further tests will not give you much new insight moving forward.

• Small sites and applications. Nielsen and Landauer (1993) report that the first five users will uncover about 70% of major usability problems and next few users will find nearly all the remaining problems.
• Information-rich sites. Five to eight users may not uncover all of the usability problems. Many successful organizations have moved in the direction of continuous testing depending on budget and project needs
• Users with different abilities. Test with at least one user with physical or cognitive limitations or who needs assistance with digital applications. Testing with real users can uncover accessibility barriers better than a checklist.

Lab configuration / test location

To be very clear, usability testing doesn’t need a “lab”. You can run testing sessions almost anywhere, assuming you can set up the space properly. However, having a dedicated space helps make frequent testing easier and more seamless. For example, you will have better grasp on the quality of audio recordings, better control of schedules, and be in a space that will be familiar to you.

Before you start testing, ensure users can access the building and the room for in-person testing sessions.

Ask beforehand if they need any special accommodations are needed prior to the test (software, access, etc.). Here are examples of how you can configure the space:

• one room setup. One room where participant uses the product with moderator. Another room is dedicated for observers to watch participant’s interaction with product using recording equipment. This setup is just as effective as two room setup
• two room setup. Observers look into a one-way mirror to see the test session from another room, but not the other way around. This reduces the need for expensive digital cameras and video monitors, but needs a custom-built observation space to maintain correct 3:1 lighting ratio between rooms so the one way mirrors functions properly.
• remote testing. Moderator, participant, and observers dial into a video conference call. Observers will remain quiet and watch what happens on participant’s screen. The benefit of this setup is that you can test users anywhere they would like to be situated. However, unstable internet connectivity may be an issue when conducting the tests.
• on site testing. You may also choose to conduct usability testing sessions in coffee shops, library, or other areas that is convenient to the participant as long as everyone feels comfortable with the situation. For on site testing sessions, do not go alone. It is always good to go with at least one of your teammates because you will need an extra helping hand and it will be easier to debrief with them after each session.

Type of tasks

There are different types of tasks you may use to gather feedback from users. When conducting usability testing sessions, the goal is to find out why your participant is asking the questions they are asking, rather than provide answers to their concerns.

For example, if a user asks “what does it mean by ___?” Rather than answering the question, you can reply “if you were to take a guess, what do you think it means?” This will help you get a good level of understanding based on their answer.

Verb-based tasks

Verb-based tasks is when you ask users to accomplish specific actions with the product. Examples include:

• ask a question to your friend, Scott
• write a sample blog post about your pet
• pay for a product you are interested in

Scavenger-hunt tasks

With scavenger-hunt tasks, you ask users to find specific piece of information. Scavenger-hunt usually always begin with the verb “find”. Examples include:

• find the nearest postal office
• you recently moved from Quebec to Ontario and need to change your health insurance coverage. Show me where you would go to find more information about this. What would you do next?

Interview-based tasks

Before the user arrives to the session, you won’t know what tasks the users will be doing during the session. When the session starts, you will first conduct an interview with the user to get a better idea on what information they will be looking for. You then give users the opportunity to go on the site and accomplish their goals on the fly.

The best time to do interview-based tasks is when sites and products are populated with real information and data. Without real content and data for users to explore, it will be hard to mirror the true experience of the user.