3 UX Research Methods That You Might Want to Think Twice About Using

The demand for UX work may be rising but every UX professional is going to be judged on their results and they’re going to have to deliver those results with tight budgets. That means we need to be ultra-critical about the UX research methods that we choose and it’s probably fair to say; not all research methods are created equally.

We’ve decided to cast a critical eye over three user research methods, focus groups, surveys and heuristic evaluations and see why these may be techniques that UX professionals should shy away from.

That’s not to say that these techniques are never going to be useful but we do think that they may not offer the same utility as they were once thought to possess.

What’s a Focus Group?

The marketing research standards authority offers this definition; “A marketing research technique for qualitative data that involves a small group of people (6-10) that share a common set characteristics (demographics, attitudes, etc.) and participate in a discussion of predetermined topics led by a moderator. There are opportunities to conduct focus groups with the use of focus group software.”

Author/Copyright holder: Penn State. Copyright terms and licence: CC BY-NC 2.0

So What’s Wrong With Focus Groups?

In theory, they sound pretty good don’t they? You get a bunch of users together who all have similar characteristics, probably characteristics that feature heavily in your user personas, then you ask them a bunch of questions and get their input. How could that be a bad idea?

We’ve come to the conclusion that the trouble with focus groups is that they focus too much on what people say rather than what they actually do. Why’s that a problem? In the immortal words of House M.D. “It’s a basic truth of the human condition that everybody lies”.

That doesn’t mean that your focus groups have been made up by inveterate liars whose sole purpose is to mislead your research. What it does mean, is that quite often we tend to view our actions through rose-coloured glasses and we tend to offer up, what we might do if we were the person that we think you want us to be, rather than the “what we actually do”.

If you look at the recent UK elections all the polls and focus groups etc. predicted that the election would be a close race and that it would be next to impossible that the next British government could be anything other than a coalition because the UK was so politically divided. What happened? The Conservatives won an outright majority.

Millions of pounds were wasted on measuring what people said they would do and not measuring what they would actually do in practice. We don’t know why so many people refused to admit they would vote Conservative, what we do know is that asking people how they vote won’t tell you how they will actually vote.

What’s a Survey?

According to Wikipedia; survey research is a collection of questions that “is often used to assess thoughts, opinions, and feelings. Survey research can be specific and limited, or it can have more global, widespread goals.”

Marketers and UX researchers have been delivering surveys for a long time now. In fact, conducting surveys has become an industry in its own right. Politicians, health authorities, top professional organizations, the world’s biggest companies, etc. have come to rely on surveys and the insights that they provide.

So What’s Wrong With Surveys?

In a nutshell, the same thing is wrong with surveys as is wrong with focus groups – we’re measuring what people say they will do rather than what they will actually do.

The polling data for the UK election wasn’t just drawn from focus groups it was also drawn from survey data. It was also utter tosh.

What Would Be Better than Focus Groups or Surveys?

In essence we don’t really care what people say they will do. That doesn’t matter very much at all when it comes to creating an experience. What matters is what people will actually do and that means our research needs to examine users in the context of their using our products.

When we can observe people’s behaviour we have a much higher chance of getting to the truth than when we ask them to tell us about how they behave.

If we do fall back on surveys and focus groups – it’s probably a good idea to use them to support other areas of research and to very tightly define the questions used so that they provide more valuable data. You can get insights into people’s opinions and perceptions with these tools and that’s not always a waste of time but you need to back that up with hard evidence to suggest that their opinions and perceptions actually change the way they behave.

What’s a Heuristic Evaluation?

Jakob Nielsen defined 10 heuristics for UI usability and then suggested that you take 3 to 5 usability experts and have them walk through the product to see if these heuristic requirements were met. Once they had conducted their walkthroughs – they were to meet up and discuss and review their findings and make recommendations for improving usability.

What’s Wrong With That?

Well, firstly, there’s an issue that there are no users involved in that process. Who cares what a bunch of experts with a formal ranking scale think? There are plenty of ways to break usability rules and still keep your users happy.

Then there’s the small issue that most businesses don’t actually have that many usability experts to hand. So, in practice – a heuristic evaluation almost always ends up being conducted by a single expert rather than a panel of experts. That leads to some fairly serious room for a margin of error creeping in. The whole point of UX research is that we let our users guide us rather than letting a single person inside the business dictate how a product should be.

It can also be argued that Nielsen’s principles are bit vague to be used as a “hard and fast” set of rules for product evaluation.

What Would Be Better than Heuristic Evaluation?

The heuristic evaluation can easily be replaced by an expert review. It offers a wider scope than a heuristic evaluation, takes roughly the same time and while it still doesn’t involve users – it’s much more likely to be used as a precursor to further UX research rather than to be used as a decision making tool.

Summary

All of these methods of UX research have been popular over the years. Perhaps, it’s time for UX professionals to take a closer look and ask, are these tools really still fit for purpose? And then look for better tools.