UX Playbook

Surveys

Example of a survey setup in Typeform

Topic Description
Objective Gather Quantifiable data in an unmonitored way
Owner Arjen Wiersma
Duration 1/2 day - 5 days
Additional Needs

(Online) survey Tool

Participants
Deliverables Presentation

Why a survey?

Surveys are a quick and low-treshold way to collect data on your (potential) topic at the beginning of your project. An online survey often consists of a collection of differentes types of questions. It can be helpful in many cases.

When

What for

Running a survey

  1. Determine the goal of the survey. Why are you making the survey? What will you use the data for? Which data do you need to reach your goal?
  2. Write down your assumptions. This will help you identify what kind of questions you need to ask
  3. Write down your research questions. It is best to write down research questions that you want to see answered with the results. This also helps to summarize the results at the end
  4. Write out all survey questions that you need to gather the data.
  5. Arrange the questions by theme, and re-asses the relevance of all questions. Take out everything that’s not essential, the shorter the survey, the more people will finish it. Asses whether the questions are appropriate for a survey, or better suited for an interview.
  6. Create the survey with an online tool, like Google Forms. Add a title, clear description and reveal the sender.
  7. Have a trial run with a colleague to uncover (mis)interpretations and measure the time. Adjust when necessary.
  8. Distribute the survey. You can do this through e-mail, social media or internet forums. Or place an advertisement and pay per complete survey through (for instance via Respondenten Database ). A good time would be on monday morning or late afternoon.
  9. Stop the survey when you reach a certain amount of respondents. Analyse the data.

How many responses?

Deciding on the amount of respondents is a very tricky question that totally depends on what you want to find out. If you send out a survey to gather qualitative insights, obviously any input is already new knowledge. But as soon as you have quantitative questions, you will need to ask yourself exactly what you need to know and how many people you need for that. E.g. if you want to know how many people would be interested in a feature, you can't have just 2 responses as the only outcomes are: nobody wants it, everybody wants it, or it's perfectly divided into 50% for and 50% against. But even if you have 10 responses, a critical colleague might dispute the results claiming that the results are not representative of the full group.

There are some calculations and look up tables depending on what kind of data you're collecting and what kind of results you're trying to find.

Creating a survey

Start of Questionnaire

It's good to give some information about what the test is about, who it's run by and what the purpose is. Furthermore, explain what will be done with the data

Types of questions

Most tools allow you to add different kinds of questions.

Open
Open questions are great for getting qualitative feedback. Most tools allow the users to write a short answer or a more extensive answer. Also remember that "open" questions elicit better responses. So use who, what, where, when, why and how.

Agree/disagree
Yes/no or agree/disagree questions are quite specific and should be used with enough respondents. A great use of this question is to forward users to another section and let them skip questions that are irrelevant to them.

Scales
Scales are great for getting insightfull quantitative data. With more steps in a scale, you can reveal smaller changes. Watchout that you don't put too many steps in your scale. The most common steps are 5 or 7. If you have many scales, go for the 5. If you have only a few, go for the 7. Also if you go for an even number, you can force users to pick a side. But it's advised to add a neutral option: some users actually don't know or might get frustrated with the research.

One sneaky thing researchers can do is flip the range of scales or mix them up. The purpose of this is to force the user to thoroughly think about every question. Critics don't think this is necessary and claim it leads to frustration with users.

Multiple Choice
Often you want to have a multiple choice questionnaire which allows users to select multiple answers instead of just one. So make sure this is possible in the tool. Many times, it might be nice to add the option "other" in which users can add an option. If that's not possible, follow up with an open question in which users can add their own result.

Qualitative vs Quantitative

As designers we often collect 'soft' qualitative data. Opinions, observations, intuition can all help the design forward, but can fail to convince stakeholders. If you want to make "evidence" based decisions, it can be useful to collect that evidence, both qualitative and quantitative. Surveys allow you to receive quantified results from users. Instead of "I like it", it can be a rating on a scale from 1 to 10.

By quantifying results, you can more easily compare. Especially when you use standardized tests that have been used on other platforms. It also gives you better insight and higher granularity. A design that was "good" and the new design that is also "good" can be shown to have actually decreased in usability when the SEQ score goes down from a 5 to a 4.5.

Although quantitative questions are great for indexing designs, they fail to explain why designs get these scores. So a proper survey always has questions that seek to understand why certain scores were achieved. Both quantitative and qualitative questions should be used in a survey

End of survey

Before you end the survey, add an open question that allows people to comment on the survey itself! This allows you to figure out any problems or when users struggeld with specific questions.

Checklist

Make sure the survey holds up to these checks:

Resources

Tools

The more design focused tools often allow you to show the design simealtenously with the questions, which can be a great benefit!

Tool Summary
Paper Old school cool! Can be more convenient for quick short in-person surveys
Google Forms Many types of questions, pagination and fowarding to different questions. Creates a report and you can link the answers to a google sheet
Typeform Fancy survey tool, similar to google Forms
Maze Usability testing tool that also allows you to add questions
Usability Hub Focused on design evaluation, has it's own survey tool that allows complex logic for forwarding and text analysis
User Zoom -

Standardized Tests

Standardized tests are great because they've been scientifically evaluated which make the results of the test very hard to dispose, they have been used by others so you can compare your results with theirs, and you don't need to think about structure or formatting because that's already decided on.

NPS (Net Promotor Score)

This is a favourite in the business world as it greatly reflects on what people think of your product. Instead of asking "rate it from 0-10", people are asked how likely it is they will recommend this product to others .

The NPS doesn't look at the average score, but divides the results into promotors, passives and detractors. The percentage of detractors are subtracted from the promotors. A score of -100 means nobody likes the product, while a score of positive 100 means everybody would promote it.

Please note that the US audience is on average more agreeable, so there is a european NPS that places responses in promoters earlier.
https://www.netpromoter.com/know/

SUS (System Usability Scale)

The SUS asks 10 questions that should give insight into the usability of a product. Please note that the longer the time is between doing a task and filling out the SUS, the more positive results become. It's best to present the SUS right after the task. Also note that it takes 10 questions, but the results correlate pretty close to the SEQ which is just one single question.
https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

SEQ (Single Ease Question)

The single ease question is, as the name gives away, a single question that asks about the difficulty of performing a task. Although it's a simple question, according to comparisons done by Jeff Sauro, the results are very close to those of the SUS. So it's a great question to add to your toolkit.
https://measuringu.com/seq10/

SUPR-Q (Standardised Usability Percentile Rank Questionnaire)

The SUPR-Q is a specific questionnaire created by Measuring U use to get more insights into usability, credibility/trust, loyalty and appearance, but also produces a single number for easy comparing.
https://measuringu.com/product/suprq/

CES (Customer Effort Score)

The CES somewhat reminds of the SEQ but it's phrased differently and asks about the ease of their experience.
https://blog.hubspot.com/service/customer-effort-score

ATTRAKDIFF

The ATTRAKDIFF asks a load of questions to get a better insight into how users percieve your product. This is not only related to usability. For this reason it might be very nice to get feedback on visual design. It gives a result that is mapped to two axes: hedonic quality and pragmatic quality. Looking at the individual answers gives you a better idea of what these axes mean.
http://www.attrakdiff.de/index-en.html#tab-einsatz

Learn more