UX Playbook

Usability Testing

Why do we test?

At Hike One, a usability test is a moderated test to gain valuable insights any time in the process. It’s also known as guerilla test, user test or lean user research. A usability test is a good starting point for a redesign or competitor analysis early on. During a project the test can be used to explore (different) concepts or a beta version of the product. At the end of the process we use it to figure out how people receive the product. But also to see how we can improve the design. The end goal of a usability test is always to gather user feedback to improve the design. We use this method in design sprints as well.

“Discover design problems early and define the next steps”

Usability testing is an important part of every project. It depends on the budget and project how many times we do it and in what extent the test is carried out. We prefer multiple, small tests over one large test.

Deliverables & downloads

What do you need?

Tools People Time
The usability test suitcase (reserve through the dashboard.) Interviewer Prepare: max 1 day
Test plan Notetaker Test: max 1 day
Sharpies Testers Documenting: 1/2 day
Post-its Client

How it works

It’s important that preparing a usability test doesn’t take more than one day. Testing itself will take a day and documenting also half a day. On average, a usability test (including preparation and reporting) will take two days. We prepare the test with our own templates. You can use these templates via this page.

A test is not a test without testers. Most of our clients already have a database with customers that are suitable for testing. That’s why we prefer to outsource the recruiting and rewarding of the testers to our client. Clients can offer a gift voucher or other suitable reward to testers. If testers are recruited through a recruitment office, the office will take care of the reimbursement.

If there is no agreement on the reward, we discuss the right reward with our client before the usability test starts. Another great option is to arrange testers at Respondenten.nl. Ask your project manager for more information on that matter.

We always perform the test with at least two people (excluding the participant). One member, preferably the designer, asks questions during the test. The other member writes down the findings in real-time and is there fore the note-taker.

We don’t write a stack of documents about the test and its results. We use our test template to prepare the test. Later, we supplement this document with the bottlenecks we found. We like the client to participate each test (as a second note-taker), so they know what problems are found. The document is more of a reference book.

Preparation

  1. First we determine the main goal of the test. The goal will depend on the project and the phase the project is in.
  2. We use our template to describe general information about the test. For example the occasion, testers and location of the test. We also describe the top tasks and hypotheses in this document. It also includes a checklist. Use this template as well for your interview script.
  3. We need to set up the recruitment of the participants. Either by ourselves, by the client or by a recruitment agency. Make sure you make the participants aware of the reason why we record the interview for internal purposes and he/she gives permission by attending the interview (e-mail template is attached in the testplan template).
  4. Then we collect and prepare our hardware and software. Decide if you are going to set up a usability test local or remote. It’s nice to have someone there to help you with this.
  5. We prepare a physical or digital observation board. During the test we will write down our findings on post-its. It’s important to decide what type of feedback can be written down before you start the test. Remember; it's always about observations, no opinions or solutions. It’s best to write down improvement observations as well as positive observations.
  6. Either print out the interview script when you test locally or view the script on a second screen.

Pilot test

We perform a pilot test with a colleague to check if everything works the way we want to. Sometimes there’s a broken link or a wrong transition, we fix that in this stage. We also take a photo of our setting now. We can add this to our documentation or presentation later on.

Testing

We usually take 60 minutes into account per test participant; 45 minutes for the interview, 15 minutes for discussion in the observation room.
To introduce the test we use a reading script (see our template for an example).The reading script contains general questions to ask the tester upfront. We ask questions like name, age, product knowledge and so on. This helps us to create a quick image of the user and how they use the product. During the introduction we explain to the participant that he can not do anything wrong.
We also ask him to think out loud during the test. We inform the user that by attending this interview he gave permission to record the test (for internal purpose only). This is also a reminder for you to start the recording.

If it’s necessary (ask your client), we let the participants sign a ‘Nondisclosure agreement’(see test plan template). This will give us permission, among other things, to use the footage on our website.

  1. Then we hand out the (printed)tasks one by one. When a task is finished we give the second task, and so on. Let the tester read the task out loud.
  2. We ask our participant questions like "Can you tell me what you think of this?” or “What do you expect if you click this?” Remember to put aside your own opinion as much as possible. Use a script as a guideline.
  3. Try to keep your interview around maximum 45 minutes. Keep track of time. A few minutes before the interview is done, you tell the participant you’re going to the other room to see if your colleague has any questions. Go to the observation room and ask if they have any questions for the participant.
  4. After finishing the interview, thank your participant and give them their reimbursement if it’s available (sometimes this is arranged trough respondenten.nl, for example).
  5. Stop the screenrecord of the interview (if you use screenflow, close the window first and then press save, this is much faster than choosing save first).
  6. When all interviews have been conducted, cut and paste all the saved screenflows to the HD which is included in the suitcase. Export them from screenflow the next day from the HD. This way the suitcase can be used by someone else the next day.
  7. Please save the exported files in a new Dropbox folder in ‘projects-archive’ (make a new folder with the name of your project and make a ‘usability-test_recordings’). This way, our laptops will not run out of diskspace.

If the post-its and brown paper are out of stock in the suitcase, please restock the suitcase with these items. Also, please put all the cables and other equipment back in the original bags.

Observing room

  1. The note-taker writes down bottlenecks, problems and positive findings during the test. If our client participates, we let them write down things as well. Include the corresponding device if you test more devices (like ‘mob’ for mobile app) on the post-its.
  2. Each observant (note-taker and client participants) marks the 3 most important findings when each test is finished. Use stickers to mark the post-its. We collect these post-its after each test and cluster them per task on the whiteboard/brown paper. If there’s time: discuss the findings after each interview. Don’t throw away the other post-its (without stickers). Hang them on the whiteboard/brown paper as well - or collect them (you will digitalize them after the test).
  3. When all tests are finished, we gather again to discuss the most important findings. The interviewer and observant facilitate this. Try to figure out which marked finding will be important to work on for future improvements in your design process. It’s not realistic to think we can fix every bottleneck or problem, so we decide together on what things we’re going to focus.
  4. It’s nice to digitise the bottlenecks instead of having them on the photo only. We only document the ones we’re going to fix. If there’s time, digitise all the other findings (without stickers) in a Google Doc for example, and make a reference in your report.
  5. If the client was not present on sight, we sometimes present the results in a presentation. We don’t think this is a necessity, and it’ll cost you valuable time. The document should be enough. Also, testing should be ‘away of design’. It’s better to focus on the bottlenecks and present the (new) design. If needed, we use the findings from the test to substantiate our design.

Tips for remote testing

As we are forced to test remotely due to corona, hereby some do's and don'ts:

Do's

Don'ts

Remote recruitment agencies

Checklist for project managers / design leads / UX researchers

A few things need to be arranged and managed with the client. Therefore, a checklist:

Preparation
As described above, the designers will take care of the prototype, material (i.e.test devices and recording equipment), script, hypotheses and tasks. As a Project Manager you need to:

Ready the UX playbook
First, ready the details above about our method for usability testing.

Arrange participants
We can’t conduct a usability test without participants. Usually we test with six participants in one day. We plan 60 minutes for each participant (the interview normally takes 45 minutes but could exceed this time limit). Sometimes the client arranges participants themselves. We could also arrange it for them. The experience we have with respondenten.nl is quite good. If we arrange participants for the client we use a price of €140 per respondent. Discuss and decide what characteristics the participants need to have with the designer and client.

Decide on local or remote
The preparation differs for local or remote setup. Together with you client you can decide what fits most. Some differences:

Local Remote
Amount of respondents Limited Limitless
Reading behaviour Easy Hard
Engage Easy Hard
Scope of research Up to 5 tasks 3 tasks or less
Energy-level Runs out after 5 or 6 participants Runs out after 3 participants
Surrounding Unfamiliar Familiar
Communication At testday Up front and during
Technique In control Out of control

Arrange rooms
If we are going to conduct the test at the client’s office, it’s important to arrange two rooms that are located near each other. This way we can assure a reliable connection (with our HDMI cable of 30 meters) independently from wi-fi.

Invite stakeholders (or have it done)
We would like to invite important stakeholders to accompany one or more interviews. To be an influencer during the next steps of the design proces it’s actually a must to be present. It’s fine if people enter or leave the room between participants. We do ask deciders (Product Owner for example) to be present whole day. Normally, the Product Owner invites other people to join, but as you all know the agendas of important people are filled quickly. Sending an invitation early is always a wise decision. Rather already ‘claim’ a day and cancel if necessary, than try to send a last-minute invitation.

Plan two designers (interviewer and observant)
Plan at least two designers to conduct the test, an interviewer and observant. Halfway during the day, the roles will be switched. It’s physically and mentally exhausting to interview six participants on your own. That’s why the interviewer and observant switch roles when the first three interviews are done.

Reserve the usability testing suitcase
The suitcase is filled with all the hardware and software needed to set up a usability test yourself (except the test devices itself, which should be arranged by the designers). At the moment we have one suitcase (in Rotterdam). A suitcase is made for Amsterdam which should be ready in two weeks. Via the device reservation tool on this dashboard (still in progress) you can reserve the suitcase.

Setup

Read the instructions
The designers (observant and interviewer) will take care of the setup using the suitcase. The suitcase is provided with an instruction. It would be nice if you could read this up front, so if necessary you could help set up all things.

Findings

During the test, the observant and all invitees collect the findings in the observation room. You as a project manager could join, but it’s not mandatory.


Arrange an archive folder in Dropbox
All the interviews will be recorded, but these recordings can be quite big.To prevent syncing each of our laptops with big files, please create a folder in projects-archive for these recordings. You could start by creating a client folder with a folder named ‘usability-test_recordings ’for example. We can now store the recordings directly in this folder and share the folder with the client, if needed.

Presenting outcomes

Test Compilation Video

When presenting the outcomes of the usability test to stakeholders, a compilation video is a strong way to convince your audience. Some tips to create one:

Useful links