What Emotional AI Fails to Grasp About Emotion
Will Silicon Valley downgrade our emotional experiences into emoticons?
This article is the third in a series considering the field of sentiment analysis, also frequently referred to as Emotional AI — the attempt to create machine learning systems that can detect and understand human emotion.
The first article in the series, Can AI Understand Your Emotions?, is a review of both the positive potential of Emotional AI and the current state of hyperbole promoting the field. The second article, Emotional AI Is Not Ready For Prime Time, compares the practical promises of sentiment analysis services with what they’re actually able to deliver. Additional articles include a warning in It Isn’t Emotional AI. It’s Psychopathic AI, a challenge to ultra-rationalism in Should AI Rid Humanity of Emotion?, a retelling of ancient legends with a tech twist in The Mythology of Emotional AI, and a qualitative-critical conclusion, Emotional AI’s Missing Companion.
Are You An Emotional Simpleton?
As we evaluate the quality of the assertions made by advocates of Emotional AI, it’s essential that we take the time to consider what practitioners of sentiment analysis mean when they use the word “emotion.” It’s a tricky issue, given that the experience of emotion may be the most nuanced aspect of our humanity. In the space of any given day, we go through a huge variety of emotions, with feelings that change unpredictably in ways that are difficult to describe, and often confusing, with mixed feelings characterizing the most significant emotional moments we have.
At least that’s how we feel, but Microsoft is eager to tell us that the way we feel about our own emotions is incorrect. We humans believe that we’re emotionally complex, but Microsoft says that actually, our feelings are exceptionally simple: “Interestingly, the number of basic human emotions has been recently ‘reduced’, or rather re-categorized, to just 4; happiness, sadness, fear/surprise, and anger/disgust. It is surprising to many that we only have 4 basic emotions.”
Happy, sad, afraid, and angry? Is that really all there is to our emotional lives?
Of course not. It’s balderdash.
You may experience a feeling of vindication (not the same thing as happy) upon hearing that Microsoft is distorting the research it cites when it claims that humans only really feel four different emotions. The study that Microsoft uses to justify its disconcerting (neither surprise nor digust, really) assertion is from the journal Current Biology, an article titled Dynamic Facial Expressions of Emotion Transmit an Evolving Hierarchy of Signals over Time. The key words in this title are “expressions”, “transmit”, and “signals”. The paper’s authors assert that digital simulations of facial communications of emotion can be lumped into just four categories. The research doesn’t really study emotion itself, but communication about emotion through simulations of human faces projected on a computer screen.
Most people understand that signaling through facial expressions to communicate something about emotion isn’t the same thing as emotion itself — just as saying the word “blue” isn’t the same thing as the color blue. This philosophical distinction is famously represented by artist Rene Magritte in his series of paintings, The Treachery of Images. In the same way that a picture of a pipe is not itself a pipe, a facial expression of emotion is not itself an emotion.
The misunderstanding exhibited by Microsoft is pervasive in discussions of Emotional AI. In her recent article for the Harvard Business Review, Sophie Kleber writes optimistically that “A combination of facial analysis, voice pattern analysis, and deep learning can already decode human emotions,” presuming that emotion is a code, a message sent from one person out in the world, nothing more than a packet of information.
It’s particularly disturbing when large, powerful corporations such as Microsoft, Apple, and Google don’t grasp the distinction between emotion and communication about emotion. That’s because, in failing to recognize the difference between signals of emotion and emotion itself, these corporations are failing to recognize that human beings experience an internal emotional life that has a validity aside from the external actions we take.
Emotional AI technologies scan for clues to emotion in the form of physical measurements of the human body, looking at just a few data points: Heart rate, perspiration, facial expression, and so on. After a brief glance and a moment’s algorithmic calculation, sentiment analysis systems declare that they know our emotions. In doing so, they’re ignoring that emotion isn’t a physical expression. It’s not a cognitive message. It’s an internal subjective experience.
Emotion Is What We Experience, Not What They Measure
It takes a subject to understand a subject. Artificial intelligence, no matter how complex it is, is not a subject. Arguing that sentiment analysis systems can understand emotion is like claiming that a person who has lived in a pitch dark cave since birth can tell us about the color blue. To begin to understand emotions, rather than mirror physical symptomology that indicates some form of emotion, we need to have human-to-human interaction.
Even in human-to-human research, there is a great deal of ambiguity. The most worthwhile qualitative research specializes in the interpretation of this ambiguity.
In one of the classic writings of his field, cultural anthropologist Clifford Geertz discussed how the attempt to understand systems of meaning is always a work of interpretation. He wrote about the same kind of facial expression that Emotional AI software such as Affectiva’s AffdexMe might scan for. The particular focus of his essay was the wink… or was it a twitch?
Two people, even two people from the same culture, Geertz warned, might interpret the same sudden closure of one eye in different ways. One person might perceive the motion as a signal of secret confidence, another as a sexual come on, and yet another as a display of irritation. For that matter, what one observer saw as a wink could be interpreted a mere nervous tic, a twitch. Geertz wrote, “The difference, however unphotographable, between a twitch and a wink is vast; as anyone unfortunate enough to have had the first taken for the second knows.”
A face is, of course, photographable. What’s unphotographable is the meaning a facial expression conveys, because the meaning isn’t an object. It’s held in the mind behind the face, and a subjective interpretation of it is made in the mind of the person watching the expression. On both sides, there isn’t just one right answer about the question of what the facial expression means.
Anthropologists talk about communications as being multivocal, referring to the idea that signals have multiple simultaneous meanings. It’s a gross simplification to think, as the sentiment analysis team at Microsoft seems to, that a pattern of muscular movements on a face means just one thing.
An emoticon almost always means the same thing. It’s a simple, relatively direct code. Emotions are not emoticons, but in order to support their claims of being able to understand emotion using artificial intelligence, sentiment analysis firms seem intent upon reducing our complex feelings to the simplicity of little yellow icons representing a toddler’s vocabulary of emotion: Happy, mad, sad, afraid.
The Machine Model: Emotion Without Consciousness
Sentio Solutions, with its Feel wristband and Emotional AI system, offers a disturbing example of this trend of technology’s infantilization of human emotion. The company claims that its wristband, worn like a Fitbit, can act as an “emotion sensor and mental health advisor”.
Like other sentiment analysis companies, Sentio Solutions reduces the complexity of emotion as we feel it to just a few bland generalizations such as “angry”, “sad”, and “happy”. The Feel service goes beyond that, though, with the chutzpah to tell the people who wear it what emotions they are feeling.
The people at Sentio seem to believe that it’s a common mental health problem for people to be walking around feeling sad or happy without realizing it. Once again, we’re at the level of a toddler’s learning about feelings — if you’re happy and you know it tap your wrist, or if you’re happy and have no clue about it, your wrist will tap you. The Feel wristband responds, sending a message out to a smartphone app to tell users what their emotions are, and then initiates a round of mental health exercises, such as breathing routines and cognitive behavioral therapy.
Unlike a human mental health counselor, the Feel wristband hasn’t had any training in diagnosing psychological disorders. Instead, Sentio Solutions asserts that the Feel wristband can diagnose people’s emotional status, and recommend mental health treatments, just by measuring heart rate, skin temperature, and skin conductivity. Apparently, in the company’s model, emotions will change according to whether we’re sitting still in a drafty room or climbing a series of stairs on a summer’s day.
How could heart rate, skin temperature, and skin conductivity possibly be correlated with the subtle subjectivity of emotion? The key for Sentio Solutions seems to be to redefine emotion to take all the subtlety and subjectivity out of the equation.
The company writes, “What is emotion? Current psychological science suggests that the experience of emotion is constructed out of at least three key “ingredients”: a) Internal sensations from our body; b) Information we collect from the external world (e.g. Where am I? What is happening around me? What is that smell?) ; and c) Mental representations from prior experience (e.g. When else have I felt like this? or How do I expect to feel in this situation? etc.).”
Sentio Solutions doesn’t explain exactly which “current psychological science” indicates this model of emotion. It looks suspiciously similar to a computer science model of information processing. The Feel concept of emotion includes only A) Data input from within the body; B) Data input from outside the body; and C) Data retrieval from memory systems within the brain.
Does that sound like what you experience when you feel an emotion?
Just as emotion is more than just an external form of communication, it also can’t be reduced to a simple formula of data processing. Our emotions are not just physical sensations such as heart rate, mixed with a external sensory perception, along with information about past events.
Emotion is a holistic experience, meaning that it can’t be separated into distinct biological components and still retain its character. The experience of emotion is what’s called an emergent property. It’s not contained in any one of the many biological systems that make it possible, but in the complex combination of them all. Pare emotion down to heart rate and skin condition, and you’re missing what emotion is all about.
Emotion is deeper than skin temperature and conductivity. Emotion is more than an increase of heart rate. Emotion is a subjective experience that is tied into subjective meaning through irrational ideas and subconscious metaphorical associations. It’s cultural. It’s social. It’s profoundly personal. It’s in the mind and body together, not just held on the wrist to be tracked by a Silicon Valley bangle.
Clear terminology
An old adage warns that when the only tool you know how to use is a hammer, every problem in the world begins to look like a nail.
This blinkered perspective, rather than a genuine interest in human emotion, is what’s driving the development of Emotional AI. Silicon Valley has invented facial recognition technology, and so Silicon Valley firms have decided that facial recognition can tell us everything we need to know about emotion. Silicon Valley has invented devices that can make simple medical measurements while wrapped around our wrists, and so Silicon Valley firms have decided that emotion is a thing that can be measured on our wrists by those sensors. Silicon Valley has invented powerful information processing systems, and so Silicon Valley firms have decided that emotion is nothing more than an information processing system that can interface with its computers, as easy as Raspberry Pi.
We might as well research emotion with a hammer.
The tools of Emotional AI can detect physical conditions that are related to emotion, of course, but they can’t come close to measuring emotion itself. Preserving this distinction is necessary if we are to preserve the idea that human experience is valid on its own terms.
The simple value of human experience comes under threat when powerful corporations begin to believe that they are understanding the emotions of their employees and customers by scanning their faces with cameras or measuring their skin temperature. There’s a growing temptation to replace emotionally intimate human-to-human contact in commercial activities with automated check-ins conducted through digital devices. With sentiment analysis services claiming to provide a full understanding of human emotion quickly and cheaply, why would any corporation bother to pay for human researchers to talk to people at length, for human management to conduct qualitative assessments of employees, for human customer service?
Already, we’re seeing the withdrawal of human interaction in commerce, and its replacement with algorithms. We’re told these algorithms have been coded to be emotionally responsive. We all know damn well that they aren’t.
Sentiment analysis has its place, but it simply can’t provide the rich insight and sensitive response that human beings can. In order to precisely define the limited scope of activities in which sentiment analysis can provide a reliable service, we need to articulate a clear set of terms that corporations can use to describe the different levels of psychological interaction that take place in commerce, distinguishing emotion from the physical phenomena and information associated with it. I propose the following set of terminology to delineate the different categories.
Manifestation: A physical or cognitive sign of a sentiment or emotion
Sentiment: An inclination or general mood, shaped by deeper emotion
Emotion: A complex subjective experience that is non-rational, with both conscious and non-conscious elements, interacting with both cognitive and physical manifestations
Emotional AI can measure manifestations and sentiments, but it cannot measure emotion. Sentiments and manfestations can be measured quantitatively because they are relatively discrete. Emotions, however, are inherently subjective, complex, interconnected, ambiguous, and irreduceable. They need to be assessed through intimate, human-to-human contact at the level provided by psychotherapists, or by cultural anthropologists in the tradition represented by Clifford Geertz.
Let sentiment analysis remain at the level of sentiments.
Our emotions must never be allowed to wither into mere emoticons.
This article is the third part of an ongoing series exploring both the capabilities and flaws of sentiment analysis, also referred to as Emotional AI. The goal of these articles is to make the technology more effective in the future by reintroducing human context to the automated processes, while calling attention to the rampant exaggerations and misleading claims that currently threaten the field’s credibility.
The first article in the series, Can AI Understand Your Emotions?, introduced the concept of of Emotional AI in its cultural context. The second article, Emotional AI Is Not Ready For Prime Time, dealt with the practical question of whether present day Emotional AI systems can actually do the tasks their designers say they can do.
Tomorrow’s article will turn the perspective of Emotional AI on its head, asking: If individual human beings behaved in the way we’re programming sentiment analysis systems to work, what would we say about their mental health? After that, the series confronts the question: Should AI Cure Humanity Of Its Emotion?
