Decoding User Emotions: Understanding the Hidden Signals in UX Research Interviews

13 min readSep 2, 2023

I recently read a post on Linkedin that spoke about just sending out surveys instead of performing user interviews because they take up a lot of time and cost a lot of money. And this slightly concerned me, because user interviews provide us UXRs with deep knowledge and understanding of our users. While analytics and data-driven insights are valuable, they often lack the depth required to understand the intricacies of human emotions and behaviors. This is why we should perform UX research interviews. Beyond the written or verbal exchange of information, there is a wealth of non-verbal cues and emotions conveyed through body language, facial expressions, and gestures.

This article dives into what non-verbal cues to look out for, what they mean, and how we can use them to drive user interviews to a deeper layer.

The Role of Nonverbal Communication in UX Research

Statistically, when we converse with somebody, only 7% of the communication is verbal. 38% of communication comes from tone of voice, and the majority 55% comes from body language, i.e. facial expressions, gestures, and posture (image). So when we perform a survey or look at data you may only get 7% of the insights you are looking for as the other data is either unavailable or people tend to overlook them.

While words can provide useful insights into a user’s preference and behavior, non-verbal cues offer a unique perspective uncovering emotions and responses that words might fail to express. Just for that reason alone, UX researchers should learn more about non-verbal communication and how to decode it. During interviews, there is a wealth of data to be found by looking at the unspoken language of facial expressions, body language, gestures, and tone of voice.

These non-verbal cues can enrich the verbal data found. A study conducted by Berry et. al (2007) showed that a combination of verbal and non-verbal cues provides a more comprehensive understanding of user experiences and sentiments. By cross-referencing the verbal with the non-verbal cues, UX researchers can see where emotional reactions align or diverge from the verbal explanations, allowing for deeper insights into a user’s behavior.

For me, where this gets really interesting is that non-verbal communication transcends cultural boundaries, making it an invaluable tool for UX Researchers performing cross-cultural research. Several studies done into cross-cultural facial expressions have shown that there is a universal nature when it comes to conveying emotions across different cultures (Matsumoto, 2008., Ekman. P., 1971.). Emotions that were found to be the same cross-culture were; happiness, sadness, surprise, fear, disgust, anger, and rest. However, just because facial expressions tend to be similar, if not the same, this does not apply to gestures. We’ll look at this a little further on.

Another benefit of looking at non-verbal cues during user interviews is that non-verbal cues are unfiltered and come from the subconscious, meaning that non-verbal cues don’t lie. Noticing this during user interviews will allow you to dive deeper into areas the user might not even be consciously aware of.

A study done by Nielsen Norman Group (2016) advocates for the integration of non-verbal cues as a means to identify pain points and delight moments in the user experience. By recognizing non-verbal cues of frustration or confusion, as well as satisfaction or relief, UX designers can create solutions that address these emotional responses correctly.

Facial expressions: The window to emotions

Research, such as the earlier mentioned work of Matsumoto has shown that there is a universal nature to facial expressions. Understanding the meaning behind facial expressions can help us uncover unspoken feelings and provide us with deeper insights into the user experience. A smile, that involves both the eyes and mouth typically signals genuine happiness. Conversely, a raised upper lip and crinkled nose could indicate disgust, furrowed brows signal confusion, and a raised eyebrow skepticism, or disbelief. These facial expressions are similar across cultures. This universality allows UX researchers to understand emotions and feelings regardless of language and cultural background.

To gain a holistic understanding of our user’s behavior and experiences, UX researchers need to cross-reference the verbal data received during the interviews with the recorded facial expressions. For example: if a user describes being enthusiastic about a certain feature, but facial expression shows hesitation, it could mean that there is an underlying uncertainty that’s worth exploring further. These discrepancies are what make user interviews interesting as they could highlight potential areas of friction or happiness that could impact the overall UX. By understanding our user’s emotions, we can help designers craft experiences that resonate with the user.

Body Language: Posture and Gestures

Because facial expressions are fairly universal they are easy to decode. However, understanding body language and gestures requires a bit more studying. Especially when we bring in the cultural differences seen here. A smile can still be faked when someone is uncomfortable but body language doesn’t lie. Understanding this is a crucial skill for UX researchers as it helps us uncover hidden cues about how our users feel.

The Power of Posture

Posture is what we look at to gain an understanding of a user’s comfort and engagement during an interview. This section dives into deciphering posture to gain a deeper understanding of our user’s emotional states. We look at two types of postures

Open and relaxed: This posture typically indicates comfort, friendliness, and willingness to talk. An open and relaxed posture can be identified by looking at the position of the arms and legs. An open and relaxed posture is important to get the most information out of your participant. You can look for signs like open arms and spread legs, however, gender also plays a role here. Men spreading their legs in an open stance can also mean dominance. If a participant is leaning forward it also means they are more engaged in the conversation.

Closed or defensive: Conversely, a closed position is characterized by closed arms, hands clutches, hunched shoulders, sitting back, or being partially turned away. These signals can show discomfort, anxiety, unfriendliness, hostility, and resistance.

The silent expressions of gestures

Next to posture we then look at gestures, as gestures are also great signifiers of user’s sentiments and attitudes. There are three common gestures to dissect. After that, we will also take a look at cultural gestures and where they might differ from the universality of the three common gestures.

Emphatic gestures

These are hand movements to put emphasis on certain feelings. For example, a feature your user feels strongly about. In empathetic gestures we can look at nodding, smiling, and as mentioned earlier, learning forward.

Nodding along in agreement while listening typically signifies that the participant is engaged and in accord with the conversation. This is a positive sign that shows understanding and support of the topic. However, what most of us know as nodding along in agreement is bobbing your head up and down, in Greek and Bulgarian cultures this nodding actually means disagreement. So be wary when interviewing cultures that you have an understanding of what the nod means.
Smiling can be a powerful empathetic gesture as it shows warmth, agreement, and friendliness. Participants that smile during an interview could be expressing satisfaction or enthusiasm.
Leaning forward during an interview shows interest and engagement. It could mean that the participant is eager to learn more and is actively participating in the interview. This is an interesting gesture when discussing product features as you can identify through this gesture whether they are actually interested in a feature or are just being polite.

Nervous/Anxious gestures

Gestures such as fidgeting, avoiding eye contact, and crossing arms are often considered to be nervous or anxious gestures.

Fidgeting can exhibit itself in tapping hands or feet, playing with hair, or shifting around in a seat. Participants that fidget could look for reassurance or can be frustrated. However, when interviewing a neurodivergent user, fidgeting, i.e. stimming can help them manage anxiety, anger, fear but also excitement. In this case, the fidgeting alone cannot be seen as just a nervous gesture but has to be cross-referenced with other gestures and body language to identify whether it is a positive stim of excitement or a nervous stim.
Avoiding eye contact can be a sign that your participant is feeling nervous or is experiencing discomfort. Often when users are feeling uneasy or not confident, they tend to look away or down. Again, when it comes to neurodivergent audiences this cannot be taken as the full truth. For some neurodivergent people, eye contact can be incredibly painful and thus is avoided altogether. A neurodivergent participant avoiding eye contact can feel perfectly confident and at ease even though they are not looking you in the eye.
Crossing arms is a classic defensive posture that people show when they feel resistant or when they disagree with you. When you see a participant crossing their arms in front of you they are not open to the discussion. It is important to make them feel at ease first before continuing the conversation. Unless it is really cold in the room, because people also tend to cross their arms when it’s cold, then maybe crank up the heat first.

Mirroring

Often, during face-to-face interviews, people mirror the posture and gestures of the other person. Mirroring is an interesting concept as you can use it to change how somebody is feeling. For example, if a participant is showing closed-off body language, the interviewer can portray open language in the hopes of the participant mirroring them. In the concept of mirroring, we can look at the following three types.

Gesture Mirroring is when a participant unconsciously copies the gestures or posture of the interviewer. This typically signifies rapport and engagement and is a sign that the participant is comfortable and aligned in thought with the interviewer.
Tone Mirroring, the copying of the tone of speech or speech patterns also means there is a form of engagement. For example, when the interviewer speaks calmly, the user might adopt a similar tone, resulting in a harmonious connection. This technique is particularly interesting when dealing with a frustrated or angry participant.
Timing Mirroring is a slightly more complicated concept and involves both the participant and interviewer responding or reacting in sync with each other during the conversation. When this happens, it suggests a sense of belonging and shared perspective.

When performing and when analyzing interviews it is important to look for a combination of gestures. Just because a participant is learning forward doesn’t mean they are interested in your product feature, they might just have a back ache, but say they are also nodding along and smiling, then it could mean they are indeed interested. Interpretation of body language requires practice with observing and analyzing the data of both verbal and non-verbal data. On top of that, cultural differences, neurodiversity, and individual traits can influence these gestures. Be mindful of these differences when performing UX research.

Listening to the Tone: Vocal Inflections and Emotion

Next up is the tone of voice, listening to changes in volume can be very insightful for UX researchers. Imagine, when you’re having a conversation with a friend, and their tone changes when they get excited, or angry. This is similar to UX research, where the way a user speaks holds crucial insights into their emotions. The tone of voice is not just interesting during user interviews but can also be incredibly insightful during usability testing or any other method where face-to-face research is being performed. Let’s take a look at six common vocal cues and what they mean.

1. Enthusiastic Tone

Excitement: An enthusiastic tone often reflects genuine excitement or delight. The participant may use an enthusiastic tone to express positive feelings, such as happiness, eagerness, or anticipation, in regard to a product or feature.
Interest: When users speak with an enthusiastic tone, it can also indicate a strong interest in the topic under discussion. This can be a valuable cue for UX researchers, as it suggests potential areas of engagement or delight.

2. Frustrated or Anxious Tone

Impatience: A hurried or impatient tone might suggest that the participant is frustrated or anxious about a particular aspect of the user experience. They may be experiencing challenges or roadblocks.
Concern: A concerned tone can indicate that the user has reservations or worries about certain elements of a product or service. It’s crucial for UX researchers to pay attention to these cues to identify potential pain points.

3. Hesitant Tone

Uncertainty: A hesitant tone, marked by pauses, stutters, or repeated words, suggests that the research participant is uncertain or lacks confidence in their response. They may be struggling to articulate their thoughts or feelings.
Ambiguity: Users using a hesitant tone might have mixed feelings or may not have a clear preference. When you notice this during interviews it’s essential to explore further to uncover their underlying sentiments.

4. Monotonous or Flat Tone

Indifference: A flat or monotonous tone can indicate indifference or a lack of engagement. Users using this tone might not be emotionally invested in the conversation or may find the topic uninteresting.
Boredom: Users who speak in a monotonous manner might be signaling that they are not finding the discussion engaging or stimulating. This could be a red flag for potential usability issues. Not to make it more complicated, but sometimes neurodivergent participants could struggle with inflections so they might sound monotonous, but that does not mean they are not enthusiastic. As stated before, it is important to cross-analyze a multitude of signals before jumping to conclusions about what the non-verbal cues mean.

5. Warm and Friendly Tone

Comfort and Trust: A warm and friendly tone reflects a sense of comfort and trust. Participants using this tone may feel at ease, which can foster open and honest communication.
Positive User Experience: A friendly tone often indicates a positive user experience. Users who feel valued and respected are more likely to engage in productive conversations.

6. Agitated or Defensive Tone

Conflict or Disagreement: An agitated or defensive tone can arise in response to conflicts or disagreements. Participants may be expressing frustration or resistance to certain aspects of the user experience.
Usability Issues: Researchers should pay attention to an agitated tone, as it could point to usability issues or design flaws that need addressing.

As with facial expressions and body language, it’s essential to consider tone in the context of the overall conversation. A single instance of tone may not provide a complete picture, and it’s often necessary to probe further to understand the underlying emotions or concerns. You can also cross-check the tone of voice with posture and gestures to identify what emotions the participant is experiencing building a more well-rounded analysis of the user experience. Additionally, cultural differences and individual communication styles can influence tone, so researchers should approach tone analysis with cultural sensitivity and context awareness during UX research.

Creating an Inclusive and Comfortable Environment

During user interviews, or any method of UX research really, the facilitator is the one shaping the quality of insights that are obtained during the research. Ensuring that you set up and maintain an inclusive and comfortable environment is not just important from an ethical standpoint but also from a strategic one. Doing so will make it easier for participants to open up, share their feelings, and provide a more accurate understanding of their experiences. This can be done in multiple ways, but I’d like to highlight three that have worked well for me in the past.

1. Building Trust and Rapport
The foundation of successful UX research interviews is trust and rapport between the interviewer and the participant. As highlighted in Dumas and Redish’s book “A Practical Guide to Usability Testing,” building trust involves establishing credibility, demonstrating empathy, and actively listening to the participant (1999). When participants trust the interviewer, they are more likely to share their honest opinions and emotions.

2. Cultural Sensitivity and Respect
Cultural sensitivity is extremely important in today’s diverse user landscape. Many of us work remotely and interview participants across the world. As a UX researcher, you must be aware of cultural differences that can influence communication styles and nonverbal cues. A study by Hall and Hall, “Understanding Cultural Differences,” emphasizes the importance of cultural respect and the potential pitfalls of cultural misunderstandings (1999). Being culturally sensitive helps create an environment where users feel valued and respected, promoting open and honest communication.

3. Strategies for a Safe Emotional Space
Creating a safe emotional space is crucial for uncovering genuine emotions during interviews. This involves ensuring that users feel secure and unjudged. Techniques such as active listening, non-directive questioning, and providing encouragement can facilitate this, as discussed in the book “Qualitative Research: A Guide to Design and Implementation” by Creswell and Poth (2017). Researchers should also be prepared to handle emotional disclosures with empathy and professionalism. What definitely helped me here is doing courses on coaching, growth mindset, and strength-based training.

Conclusion

In the realm of UX research, user interviews may take time, but they can bring us deep insights into the user experiences of our users. While surveys and data are helpful (and trust me, I love a good dataset), they don’t reveal the full picture of a user's emotions and actions. Non-verbal cues, like body language, facial expressions, and vocal tone can help us get that data.

As a UX Researcher, it is crucial to gain an understanding of these non-verbal signals. It helps us design experiences that genuinely connect with users’ feelings and needs. Understanding non-verbal cues has been a passion project of mine for the better of 15 years, so let me know if you want to chat about it!

References

Berry, M., Butler, J., & Hensel, J. (2007). Can nonverbal cues be used to make meaningful interpretations of interviews? An exploratory study. The Qualitative Report, 12(3), 405–417.

Creswell, J. W., & Poth, C. N. (2017). Qualitative Research: A Guide to Design and Implementation. SAGE Publications.

Dumas, J. S., & Redish, J. C. (1999). A Practical Guide to Usability Testing. Intellect Books.

Ekman, P. & Friesen, W. V. (1971). Constants Across Cultures in the Face and Emotion. Journal of Personality and Social Psychology, 17(2), 124–129.

Hall, E. T., & Hall, M. R. (1990). Understanding Cultural Differences. Intercultural Press.

Matsumoto, D., Yoo, S. H., Fontaine, J., Anguas-Wong, A. M., & Arriola, M. (2008). Mapping expressive differences around the world: The relationship between emotional display rules and individualism versus collectivism. Journal of Cross-Cultural Psychology, 39(1), 55–74.

Mehrabian, A. (1971). Silent messages: Implicit communication of emotions and attitudes. Wadsworth.

Nielsen Norman Group. (2016). UX Research Cheat Sheet: Nonverbal Communication. https://www.nngroup.com/articles/ux-research-cheat-sheet-nonverbal-communication/

Images used in header image:
- Photo by Hiki App on Unsplash
- Photo by engin akyurt on Unsplash
- Photo by Freddy Mishiki on Unsplash
- Photo by Elyas Pasban on Unsplash