How to Conduct User Observations

Observing users interacting with a product can be a great way to understand the usability of a product and to some extent the overall user experience. Conducting observations is relatively easy as it doesn’t require a huge amount of training and it can be relatively fast – depending on the sample size of users you intend to observe.

Author/Copyright holder: Mdd. Copyright terms and licence: CC BY-SA 3.0

Observational studies have long been conducted in psychology and the techniques used for user research are very much the same as those used in psychology. They have their strengths and their weakness too. The two most common techniques used for observing users are controlled observations and naturalistic observations, we’ll take a look at both of these techniques in a few moments.

Preparing for User Observations

Before we decide on a technique for our observations; it’s important to do a few things first:

We need to decide what we expect to learn from our observational study – can this be structured (e.g. a checklist developed) or will it be unstructured (e.g. we expect the data to be qualitative rather than quantitative)?
We need to recruit participants for our research and ensure that we have selected a relevant sample size of our user base to make our results meaningful.
We need to recruit observers, ideally trained observers, and define how we will observe participants
We should be able to explain to participants what they will be doing and what they will be observed for
We should be able to explain to participants how their data and any data collected will be used and how it won’t be used

These efforts are common to all forms of user research and can be better understood through further reading – some links are provided for this at the bottom of this article in the references section.

Types of User Observation

The good news is that we learn to observe from a very young age and it’s something everyone can do. So let’s take a look at the two most common forms of observation:

Author/Copyright holder: NASA Goddard Flight Center. Copyright terms and licence: CC BY 2.0

Controlled Observation

Controlled observation tends to take place in a laboratory environment. It is focused on revealing quantitative data though it may also yield qualitative observations.

To carry out a controlled observation it’s best to formulate a series of observations and have the observer carry these out at each step in the process. They can then record their observation quantitatively (e.g. yes/no or a rating scale out of 3 or 5) against each step. This doesn’t preclude them from adding any additional observations freehand against each step – enabling the collection of additional qualitative data.

Controlled observations require that the researcher explain the purpose of the observation to the user and that the participants know they are being observed. It can help to use more than one observer for longer observations or to use video to record the event for further observational work after the fact. If you use video, you must obtain the participant’s consent for this.

The advantages of this approach include:

Easy to reproduce. If you use a quantitative approach – it should be easy to get similar results by repeating the research. This may be important if the results are challenged at a high level in an organization.
Easy to analyze. Quantitative data requires less effort to analyze than qualitative data.
Quick to conduct. While recruitment may take a little time a lab conducted controlled observation is fairly fast to run.

There is a possible drawback to this approach too:

The Hawthorne Effect. This is an understanding that the act of observation of how someone does something can change their approach to carrying out the task.

Author/Copyright holder: Jisc. Copyright terms and licence: CC BY-NC-ND 2.0

Naturalistic Observation

A naturalistic observation involves studying the user “in the wild” and tends to be less structured (though a structured approach is certainly possible to a “in the wild” observation – it’s really just a repeat of a controlled observation but outside of the lab).

This means spending time with a user or group of users and observing their behavior with the product as they use it in day-to-day life. The observer records their observations as they see fit. This is very much an approach which leads to qualitative outputs.

The advantages of this approach include:

More reliable. When people use a product in real life – they are much more likely to encounter the frustrations (and benefits) of real life use than they are in a lab following a set of instructions.
More useful for ideation. Qualitative research can generate lots of ideas for product improvement as it opens up possibilities that aren’t found in quantitative research.

However, there are some disadvantages to using this method too:

It’s difficult to include a representative sample. This kind of research is more expensive and time consuming to conduct than controlled observations and that limits the reach of the research. It’s better to use this kind of research to create ideas and then test those ideas with other forms of research than to rely on the output of the research as gospel.
It’s difficult to make them replicable. Because of the problem with sample sizes – you are going to find very different results if you repeat the research. This is doubly true if you use a different observer (as observers will see different things in even identical situations).
It’s hard to manipulate external variables. For example, if it’s raining when you observe your users working on a smartphone – their behavior is likely to be different to when it’s sunny. You have no control over the weather “in the wild”.

Author/Copyright holder: Pixabay. Copyright terms and licence: Free to Use

What to Look for During User Observation

It can help, when it comes to qualitative observation, to have some touch points to examine:

What are users actually doing? As opposed to what you expected they might do.
What routines do users have with the product? How are they integrating it into their lives?
Record details – adding granularity and specificity to an observation can make it much more meaningful
Ensure you’re examining activities in their whole; look at how the product is used in context with their device and the flow of their lives and not just at the product itself.
Don’t be afraid to get quantitative. If you see an example of behavior that you think may be repeated – make a note of it and look for it in future observations.

It’s important that you don’t spend time analyzing what is going on during a user observation – the key is to just watch and report on the details – analysis comes later. As Jiddu Krishnamurti says; “Observation without evaluation is the highest form of intelligence.”

Reporting on Observations

It is fairly easy to report on controlled observations – the quantative aspects can be condensed into graphs or tables. Qualitative analysis should be treated more cautiously and data compiled and used as a signpost to further research rather than being accepted as absolute truth.

The Take Away

Observational research is easy. Sit, watch and record what you see and hear when a user interacts with a product. Of course, it’s easy in theory – getting to the meatiest of observations takes practice and a keen interest in your subject. However, user observations are a great way to find out more about how a product is used (or indeed how an information visualization is used) and to identify any problems that they encounter.

It is also worth noting that often research will combine observations with a structured interview after the observation exercise has finished to gain further insight into what is happening and why.