So, you have decided to conduct a user testing program. May I offer my congratulations: you have taken the first step towards a glorious future as a data-driven organisation.
But what happens next is vital. Design and conduct your tests incorrectly and you might as well not bother running any testing at all. In this short blog post I set out the four guiding principles we follow in Kooba when engaged in this work. I believe if you follow these laws you can’t go too far wrong.
Ignore one or more, however, and you’re in trouble. So pay attention at the back!
Law 1: don’t compromise on audience
So many user testing programs are holed before the waterline before they even leave the harbour, and for a simple reason: they ask the wrong people.
User testing doesn’t need to involve a lot of users. Five is often enough. But it absolutely has to involve the right users. That doesn’t just mean people who vaguely fit the profile or whoever your recruiter can find on the street. It means users who are genuine prospects for the business.
That makes the task of recruitment hard. But do not cut corners. Define precisely what you are testing and why, and what the prospect looks like. Then do whatever it takes to ensure your test participants match up.
I stress this point because I have seen research (of all types) fail simply because the wrong people were involved. It’s easy to look the other way or assume that ‘close enough’ is good enough. It isn’t.
Law 2: focus on tasks
A good user test asks participants to perform a set of tasks on a site, or some form of site prototype. The intention is to establish where difficulty occurs and how that difficulty might be resolved in the next design iteration.
What it doesn’t typically do is simply ask users to browse a site aimlessly and report on what they don’t or don’t like. (There is an exception of course. User testing might be intended to establish which areas of a site and which navigation elements draw the eye and draw the user in. But much of that can be established with more quantitative heatmap analysis).
It is very important, before user testing, to agree on which tasks are to be tested, and agree the ‘script’ for each test. This isn’t to say that users cannot explore and make mistakes freely (they can), but rather that what we ask them to try and do is consistent.
That stands to reason. If a particular event (signing up for an account, booking a flight, finding a seat) is important to the client, then we want to see how easy that process is for the user. That is where user testing comes into its own.
Law 3: focus on actions not words
Or what I like to call the “Jane Austen Rule”. The great novelist once said, “It isn't what we say or think that defines us, but what we do”, and just as it was for posh English people wandering around large houses in the 18th century, so it is for web users in the 21st.
When observing user tests, and drawing conclusions, be very careful to separate what the user says with the actions they take. We are all humans, and when shown anything we start to comment on it. But our comments are often contradicted by our actions. If in doubt, favour the latter.
So if your user is shown a home screen and says “I can’t see how I sign up here” whilst their mouse pointer unerringly finds the ‘sign up’ call to action and clicks on it (yes, it happens), give more weight to actions rather than words.
A caveat: good user testing absolutely encourages participants to ‘speak their mind’. We want test subjects to talk about what they see, feel, struggle with, like and dislike. They should always be encouraged to talk aloud. We just have to remember to apply a professional filter to what comes out!
Law 4: record everything
The purpose of user testing is to introduce data and objectivity to conversations about design and UX. To be clear, we are not looking for scientific proof - hence the small sample size - but we want to see how our potential designs fare when up against reality.
Given this is the case, it is most important that we don’t allow what happens in a set of user tests to become subject to the accidental and not so accidental lapses in memory that occur when we didn’t capture that evidence - both on film and in the course of our own notes at the time.
It’s just a fact that over time memories fade. It’s also just a fact that a designer with a marked preference for A over B magically forgets that 4 out of 5 users didn’t understand A and couldn’t complete the relevant task. Don’t allow that to happen. Record everything.
When user testing is complete, use your material to pull out common themes and conclusions into a report. But keep the originals. You will always need to go back and remind your stakeholders what happened out there in the real world.
