Giving people advice on how to do interviews (the research kind, not the hiring kind) is hard. It’s hard for the obvious reasons: there are quite a lot of aspects to consider – from body language to question types. But I think it’s also hard because interviews can be highly structured while feeling very fluid. Especially when it comes to semi-structured qualitative interviews, there can be a contradictory blend of improvisation and planning – which is tricky to explain.

Which made me wonder: could it be helpful to look at interview structures instead? A notation for interviews? Let’s do a little exploration.

Question types

We’ll look at one interview. For the purpose of this exploration I’m not interested in the actual substance of the interview – the participant responses. I’m only looking at when the interview moderator (that’d be me) asks questions.

The moderator can say and ask many different types of things during an interview, we’ll differentiate between a few high-level types of remarks:

Now we can take the interview transcripts and visualise what I asked/said over the course of the interview:

While this is mildly interesting, it doesn’t give much insight into the structure of the interview. We’re missing a lot of information. Starting with: were these questions about the same topic, or about different topics?

Lines of inquiry

Next, I annotated the questions: how does this question relate to the previous question? Is it addressing the same topic, or a new topic? Now we can see how many topics were covered:

Great, but this visualisation won’t do: it doesn’t convey any sense of ‘depth’. While we’re talking about the same topic, we’re learning more and more about it. We’re building understanding, so let’s also visually make it look like we’re building upon the previous questions:

This is starting to show the contours of the interview: we can easily see the different lines of inquiry that were covered during the interview, some with more questions than others. However, this representation is far too mechanical: the equidistant spacing between the questions makes it look like each question received an identical response. In reality, the participant may have given an extensive answer to one question, while responding very succinctly to another.

Using the question timestamps, we can calculate the time between moderator remarks. To keep things simple, let’s assume that when the moderator is not talking, the participant must be. Now we can visualise a more accurate representation of the interview:

Much better. The unevenness of a real conversation, showing how proportionally time was spent on each topic. Now we can see things like: during topic 4, the participant gave many shorter responses, followed by one very long answer for the last (planned) question.

That’s probably as detailed as we can get without digging into the content of the interview. Let’s create one more simplified version:

If you hover over the triangle landscape you can still see the moderator questions.

This simplified version allows us to easily compare interview structures at a glance. So let’s compare. Here are 4 interviews from 4 different studies:

Just from these pointy little colourful depictions we can see wide variety: some of the interviews go very in-depth into one topic, while others cover multiple topics for a much shorter period of time. Which brings me back to where we started: giving interview advice is hard. But at least now it’s hard with visualisations…? Let’s go:

Interview advice

When mentoring people who are starting to do qualitative interviews, I see three common mistakes:

#1 Starting too specific, too soon

Launching straight into the topic of interest to you, the moderator, can seem obvious from your perspective. But jumping into specifics means you may have missed potentially key contextual information, and you may have lost an opportunity to build rapport with the participant.

During the interview we’re building our understanding. If you start with a very detailed question, there’s nothing left to build upon. Instead, start with broader contextual questions. As the conversation progresses, narrow down and zoom in on the areas you are specifically interested in. Here’s an example:

The main objective of that section may have been to find out if they had previous experience with a particular service. However, everything we just learnt about how they do this activity, and what they find hard about it, has undoubtedly given us a much richer understanding of their broader needs.

#2 Trying to cover too many topics

The differences between the two interviews above say it all: given that there’s a finite amount of time available during an interview, carefully consider how you want to spend that time. If you try to cover too many topics, you’ll get a sequence of small triangles of understanding – breadth but likely little depth. If, instead, you focus on one or two topics, you’re much more likely to get an in-depth understanding. The interview may also flow better, as you’re continuing on the same topic rather than switching between topics.

It may sound obvious, but don’t cram too much in. Also, because it may result in you – the moderator –being far too conscious of how many questions there are left to cover in the available time – which brings us to point 3:

#3 Focusing on the questions, not the answers

While it may seem important to ask everything you planned to ask, it’s far more important to listen to the participant’s responses. Interviewing newcomers often find comfort in following their discussion guide/script. I’ve tried to emphasise the importance of some improvisation when giving people feedback on their moderation, but I’d never actually had a look at impromptu questions and instances of repeating what the participant said in my own interviews:

At first glance the proportion of impromptu questions seems to fluctuate per interview. I’d expect more impromptu questions in in-depth interviews, but interestingly it looks like the percentage was comparable for the very focused interview and the more topic-diverse interview. It also looks like impromptu questions led to some of the longest participant responses in these four interviews, which would be interesting to investigate further.

For participants, I think impromptu questions can show you are paying attention to what they are sharing. As a moderator, ad hoc questions can be an opportunity to gain a deeper understanding, or to adapt your discussion guide/script based on the things you’ve learnt so far.

Final notes

That’s the end of the exploration, for now. Imperfect but fun. I’d love to know whether it’d be possible to distinguish between moderator styles if you’d analyse a bunch of interviews from different moderators. As a reflection exercise, it would also be interesting to see the types of responses different types of questions elicit, ideally in terms of content rather than duration. However, those ideas quickly veer into personal data territory and publicly available timestamped (HCI-y/UXR-y) interview transcripts seem to be a rarity.

The notation above has many limitations. Too many. But that’s the point of a first experiment, right?

• I’ve assumed that the time between interviewer remarks is filled with the participant talking. This ignores the possibility of silence, which is a pretty fundamental part of interviews.
• I’ve assumed that discussing a topic for a longer period of time means you are building more and more understanding. Whether that’s true can only be assessed by looking at the content of the participant responses.
• Defining topics is subjective: the level of granularity you choose here can hugely impact the visualisation.
• In interviews you often return to the same topic later in the interview. I haven’t addressed this in the current notation yet. Your starting point for the repeat topic would not be at the bottom, you’d be building on the peak of the previous triangle instead.
• Who plots the same variable on the x and y-axes? Silly stuff, purely for visual clarity.

So: thoughts very welcome. Does anyone know of existing interview notations? Ideas for other ways to approach this?