Backlash to Zoom Emotion Detecting

 I agree that AI is not fully ready to do this well today,  but that 9t is inevitable, this will happen. And its a good idea to experiment with it now.  Not Dystopian.  

In AI News    Zoom receives backlash for emotion-detecting AI. By Ryan Daws | May 19, 2022 | TechForge Media

Zoom has caused a stir following reports that it’s developing an AI system for detecting emotions.

The system, first reported by Protocol, claims to scan users’ faces and their speech to determine their emotions.   Zoom detailed the system more in a blog post last month. The company says ‘Zoom IQ’ will be particularly useful for helping salespeople improve their pitches based on the emotions of call participants.

Naturally, the system is seen as rather dystopian and has received its fair share of criticism.

On Wednesday, over 25 rights groups sent a joint letter to Zoom CEO Eric Yuan. The letter urges Zoom to cease research on emotion-based AI.  The letter’s signatories include the American Civil Liberties Union (ACLU), Muslim Justice League, and Access Now.

One of the key concerns is that emotion-detecting AI could be used for things like hiring or financial decisions; such as whether to grant loans. That has the possibility to increase existing inequalities. “Results are not intended to be used for employment decisions or other comparable decisions. All recommended ranges for metrics are based on publicly available research,” Zoom explained.

Zoom IQ tracks metrics including ... '   (more at the ink) 

AI That Feels

 A space we examined with how products could link with consumers.   

BUILDING AN AI THAT FEELS

AI systems with emotional intelligence could learn faster and be more helpful

 By MARY CZERWINSKI, JAVIER HERNANDEZ, DANIEL MCDUFF in IEEE

IN THE PAST YEAR, have you found yourself under stress? Have you ever wished for help coping? Imagine if, throughout the pandemic, you'd had a virtual therapist powered by an artificial intelligence (AI) system, an entity that empathized with you and gradually got to know your moods and behaviors. Therapy is just one area where we think an AI system that can recognize and interpret emotions could offer great benefits to people.

Our team hails from Microsoft's Human Understanding and Empathy group, where our mission is to imbue technology with emotional intelligence. Why? With that quality, AI can better understand its users, more effectively communicate with them, and improve their interactions with technology. The effort to produce emotionally intelligent AI builds on work in psychology, neuroscience, human-computer interaction, linguistics, electrical engineering, and machine learning.

Lately, we've been considering how we could improve AI voice assistants such as Alexa and Siri, which many people now use as everyday aides. We anticipate that they'll soon be deployed in cars, hospitals, stores, schools, and more, where they'll enable more personalized and meaningful interactions with technology. But to achieve their potential, such voice assistants will require a major boost from the field of affective computing. That term, coined by MIT professor Rosalind W. Picard in a 1997 book by the same name, refers to technology that can sense, understand, and even simulate human emotions. Voice assistants that feature emotional intelligence should be more natural and efficient than those that do not.

Consider how such an AI agent could help a person who's feeling overwhelmed by stress. Currently, the best option might be to see a real human psychologist who, over a series of costly consultations, would discuss the situation and teach relevant stress-management skills. During the sessions, the therapist would continually evaluate the person's responses and use that information to shape what's discussed, adapting both content and presentation in an effort to ensure the best outcome.

While this treatment is arguably the best existing therapy, and while technology is still far from being able to replicate that experience, it's not ideal for some. For example, certain people feel uncomfortable discussing their feelings with therapists, and some find the process stigmatizing or time-consuming. An AI therapist could provide them with an alternative avenue for support, while also conducting more frequent and personalized assessments. One recent review article found that 1 billion people globally are affected by mental and addictive disorders; a scalable solution such as a virtual counselor could be a huge boon.

There's some evidence that people can feel more engaged and are more willing to disclose sensitive information when they're talking to a machine. Other research, however, has found that people seeking emotional support from an online platform prefer responses coming from humans to those from a machine, even when the content is the same. Clearly, we need more research in this area.

About 1 billion people globally are affected by mental disorders; a scalable solution such as an AI therapist could be a huge boon.

In any case, an AI therapist offers a key advantage: It would always be available. So it could provide crucial support at unexpected moments of crisis or take advantage of those times when a person is in the mood for more analytical talk. It could potentially gather much more information about the person's behavior than a human therapist could through sporadic sessions, and it could provide reminders to keep the person on track. And as the pandemic has greatly increased the adoption of telehealth methods, people may soon find it quite normal to get guidance from an agent on a computer or phone display.

For this kind of virtual therapist to be effective, though, it would require significant emotional intelligence. It would need to sense and understand the user's preferences and fluctuating emotional states so it could optimize its communication. Ideally, it would also simulate certain emotional responses to promote empathy and better motivate the person.

The virtual therapist is not a new invention. The very first example came about in the 1960s, when Joseph Weizenbaum of MIT wrote scripts for his ELIZA natural-language-processing program, which often repeated users' words back to them in a vastly simplified simulation of psychotherapy. A more serious effort in the 2000s at the University of Southern California's Institute for Creative Technologies produced SimSensei, a virtual human initially designed to counsel military personnel. Today, the most well-known example may be Woebot, a free chatbot that offers conversations based on cognitive behavioral therapy. But there's still a long way to go before we'll see AI systems that truly understand the complexities of human emotion.  .... ' 

Can a Robot Understand a Hug?

Connecting touch and sight sensory interactions. 

Robot Can Understand What a Hug Is  By New Scientist, February 9, 2021

A prototype soft robot with nylon skin developed by researchers at Cornell University is imbued with sensory perception that Cornell's Yuhan Hu said lies somewhere between human touch and sight.

The team stretched the skin over a 1.2-meter (3.9-foot) cylindrical scaffold atop a wheeled platform, with a commercial USB camera for interpreting different types of touch on the nylon.

The Cornell researchers then compiled a database from camera images of humans making one of six interactions with the robot's skin, training a neural network to detect and identify different interactions with up to 92% accuracy.

The team matched simple commands to the gestures, and also showed that with a projector added, the robot could display a user interface on its skin for use as a touchscreen...

Cornell University researchers have created a low-cost method for soft, deformable robots to detect a range of physical interactions, using a USB camera inside the robot to capture shadow movements of hand gestures on the robot's skin and classifying them with machine learning software... 

From New Scientist

Reading Emotions in Obscured Faces

Something we have all noticed in the pandemic

ACM NEWS

Humans and Algorithms Struggle to Read Emotions When Faces Are Obscured

By The Conversation

A recent study shows both algorithms and humans struggle to accurately determine different emotions when people's faces are partially obscured by face masks or sunglasses. But artificial systems are more likely to misinterpret emotions in unusual ways.

The study presented images of people with various emotional facial expressions and wearing two different types of masks—the full mask used by frontline workers and a recently introduced mask with a transparent window to allow lip reading. Depending on the type of covering, the accuracy for both people and artificial systems varied. For instance, sunglasses obscured fear for people while partial masks helped both people and artificial systems to correctly identify happiness. Interestingly, artificial systems performed significantly better than people in recognizing emotions when the face was not covered—98.48% compared to 82.72% for seven different types of emotions. ... 

The performance of artificial systems is greatly affected when it comes to categorizing emotion from natural images; even just the sun's angle or shade can influence outcomes  ... 

From The Conversation

View Full Article  

Emotions to Drive Autonomous Vehicles?

Some interesting things have come out of FAU, we participated.   Here another.  But can human emotions be use to autonomously drive?  Even in part? Test it well.

Invention Uses Machine-Learned Human Emotions to 'Drive' Autonomous Vehicles  By Florida Atlantic University

Florida Atlantic University (FAU)'s Mehrdad Nojoumian has designed and patented new technology for autonomous systems that uses machine-learned human moods to respond to human emotions. Nojoumian's adaptive mood control system employs non-intrusive sensory solutions in semi- or fully autonomous vehicles to read the mood of drivers and passengers.

In-vehicle sensors collect data based on facial expressions and other emotional cues of the vehicle's occupants, then use real-time machine learning mechanisms to identify occupants' moods over time. The vehicle responds to perceived emotions by selecting a suitable driving mode (normal, cautious, or alert).

FAU's Stella Batalama said Nojoumian's system overcomes self-driving vehicles' inability to accurately forecast the behavior of other self-driving and human-driven vehicles.

From Florida Atlantic University

Detecting and Using Emotion in Language Interaction

Interesting ideas implied here, but not proven,   We worked on two kinds of 'emotion'  Detecting emotion in consumer reactions, then classifying and determining its strength.   Also  embedding it in bot style interactions with consumers, in context,  and also classifying that interaction and strength.   Could 'style' be considered a kind of classification of use here?  I like the direction suggested, though hard to see how useful it might be.  Powerful potential idea in sales and marketing. 

Expert.ai adds emotion and style detection tools to natural language API  By Damon Poeter  in Venturebeat  May 10, 2021 

Enterprises and investors are increasingly excited about using natural language (NL) processing to assist in tasks like data mining for sales intelligence, tracking how marketing campaigns change over time, and better defending against phishing and ransomware attacks.

Still, AI products using natural language engines to analyze text have a long way to go to capture more than a fraction of the nuance humans use to communicate with each other. Expert.ai hopes the addition of new emotion- and behavior-measuring extensions and a new style-detecting toolkit for its natural language API will provide AI developers with more human-like language analysis capabilities. The company this week announced new advanced features for its cloud-based NL API designed to help AI developers “[extract] emotions in large-scale texts and [identify] stylometric data driving a complete fingerprint of content,” Expert.ai said in a statement.

Based in Modena, Italy and with U.S. headquarters in Rockville, Md., Expert.ai changed its name from Expert System in 2020. The company’s customers include media outlets like the Associated Press, which uses NL software for content classification and enrichment, business intelligence consultants like L’Argus de la Presse, which conducts brand reputation analysis with NL processing, and financial services firms like Zurich Insurance, which uses Expert.ai’s platform to develop cognitive computing solutions.

Freeing people up for higher-order tasks

Expert.ai’s software platform enables natural language solutions that take unstructured language data from sources like social media sites and emails, transforming it into more digestible, usable intelligence before human analysts look at it. An example of a basic NL capability would be to distinguish between different ways a word like “jaguar” is used contextually—-to signify the animal, the vehicle, or the name of a sports team. This allows for process automation steps to be introduced to text gathering, categorization and analysis workloads, freeing up human analysts to perform higher-order tasks with the data.... " ... ' 

Using Emotion and Humor to Combat Science Misinformation

 Interesting thought here,  how do we do it effectively? 

Using Emotion and Humor to Combat Science Misinformation    By University of Utah

April 13, 2021  in CACM

Humor might help combat scientific misinformation published online and on social media, according to "Emotion and Humor as Misinformation Antidotes,"   by Sara K. Yeo and Meaghan McKasy, published in Proceedings of the National Academy of Sciences.

"Recent research sheds light on how funny science and emotions can help explain and potentially overcome our inability or lack of motivation to recognize and challenge misinformation," the authors say. "Understanding how emotion and humor shape the public's understanding of science is one more resource that can aid communicators' efforts to combat misinformation."

Yeo, of the University of Utah, and McKasy, of Utah Valley University, say humor is also linked to people's evaluations of an information source. Yeo's recent research shows that scientists who use humor are perceived as more likable yet retain their credibility as an expert.

From University of Utah

Detecting Emotion: Fundametally flawed?

We did long time tests with emotion  reading, in retail starting with work using MIT 'emotion glasses',  Was interesting for initial tests, but not spot on accurate.    Pay attention to the measurement aspects, and validate with other methods, like comments. Saw some examples used to prompt specifics in advertising copy. 

Discover the stupidity of AI emotion recognition with this little browser game

Does your face match your feelings?

Tech companies don’t just want to identify you using facial recognition — they also want to read your emotions with the help of AI. For many scientists, though, claims about computers’ ability to understand emotion are fundamentally flawed, and a little in-browser web game built by researchers from the University of Cambridge aims to show why.

Head over to emojify.info, and you can see how your emotions are “read” by your computer via your webcam. The game will challenge you to produce six different emotions (happiness, sadness, fear, surprise, disgust, and anger), which the AI will attempt to identify. However, you’ll probably find that the software’s readings are far from accurate, often interpreting even exaggerated expressions as “neutral.” And even when you do produce a smile that convinces your computer that you’re happy, you’ll know you were faking it.  ; .. ' 

AI Behind in Recognizing Emotions

Fairly clear, and also in our experience when the motion was of the 'satire' type.  Faces could be used to signal opposite emotions as well

Artificial intelligence still lags behind humans at recognising emotions  by University College London

When it comes to reading emotions on people's faces, artificial intelligence still lags behind human observers, according to a new study involving UCL.

The difference was particularly pronounced when it came to spontaneous displays of emotion, according to the findings published in PLOS One.

The research team, led by Dublin City University, looked at eight "out of the box" automatic classifiers for facial affect recognition (artificial intelligence that can identify human emotions on faces) and compared their emotion recognition performance to that of human observers.

The researchers found found that the human recognition accuracy of emotions was 72% whereas among the artificial intelligence tested, the researchers observed a variance in recognition accuracy, ranging from 48% to 62%.

Lead author Dr. Damien Dupré (Dublin City University) said: "AI systems claiming to recognize humans' emotions from their facial expressions are now very easy to develop. However, most of them are based on inconclusive scientific evidence that people are expressing emotions in the same way.  .... "

AI Mapping Moods

We took at cut at the basic idea to determine changes in moods that might indicate purchases.  And also mapping to actual behavior.   Note pharmaceutical use implications.

Can AI Map Our Moods?

By Stanford University, February 4, 2021, in ACM

Machine-learning models can map a persons mood swings and volatility from week to week, according to a new study.

Researchers from Stanford University and the University of Michigan developed a machine learning model that can infer a person's mood from their Facebook posts.

The model was trained using public Facebook postings of close to 3,000 volunteers from an earlier study, then tested on a different set of posts from 640 Facebook users who posted an average of 17 times weekly over 28 weeks. This dataset of 18,000 person-weeks, the largest ever compiled on weekly emotional dynamics, has been made public.

The patterns revealed by the model aligned with the predictions based on classical in-person psychological studies.

Said Stanford's Johannes Eichstaedt, "If this kind of approach is used ethically and legally, with strict privacy protection, we could someday have ways to computationally understand the mind. It could help with diagnosis and pharmaceutical evaluation. It could also help us track the psychological impact of traumatic societal events, such as the Covid pandemic."

From Stanford University

AI Decoding Emotions?

A problem we spent much time on, depends on what you expect from such a classification.    Much like: could you hire a human 'expert' to do this?   Training and background?  Implications and risk of error?    What sensors are being used?  At what required level of accuracy and for what prescribed need?  Here a reasonable overview, but does not address the broader need.

How close is AI to decoding our emotions? Emotion AI is becoming a big business. We talked to leading researchers about how good the tech actually is.  In TechnologyReview

 years trying to crack the mystery of how we express our feelings. Pioneers in the field of emotion detection will tell you the problem is far from solved. But that hasn’t stopped a growing number of companies from claiming their algorithms have cracked the puzzle. In part one of a two-part series on emotion AI, Jennifer Strong and the team at MIT Technology Review explore what emotion AI is, where it is, and what it means.  ... " 

Beating Zoom Fatigue

Somewhat of an old idea,  how well can it be done?   And, as we have been seeing with facial recognition, will it be blocked for implying the wrong 'kind' of control?   I would further ask, is 'zoom fatigue' any different from just general in-the-office work fatigue?  Seems if anything it is less controlling/fatiguing since you can avoid and hide from it readily. 

Emotionally aware technology could help us beat Zoom fatigue By Rana El Kaliouby in FastCompany
In times of crises we tend to crave connection, but we’re being asked to do the opposite through social distancing. More emotionally intelligent computers could help bridge the gap.  ... " 

Facial Expressions Don't Tell Whole Story

Blatantly Obvious,  which we explored in a number of cases.  It can be one input into may vectors,  it may tell only a very minor part of the story, or none at all.  It can lead to categorizing useful behavior for kinds of goals.  We worked with various approaches like 'expression glasses' to work with varying context understanding.   Including chatbot interactions.  Note also the whole large amount of work in understanding 'neural' activity.

Facial expressions don't tell the whole story of emotion
by Laura Arenschield, The Ohio State University

Interacting with other people is almost always a game of reading cues and volleying back. We think a smile conveys happiness, so we offer a smile in return. We think a frown shows sadness, and maybe we attempt to cheer that person up.

Some businesses are even working on technology to determine customer satisfaction through facial expressions.

But facial expressions might not be reliable indicators of emotion, research indicates. In fact, it might be more accurate to say we should never trust a person's face, new research suggests.

"The question we really asked is: 'Can we truly detect emotion from facial articulations?'" said Aleix Martinez, a professor of electrical and computer engineering at The Ohio State University.   ... "

Are Your Students Bored? Do They Understand?

Once you have information about emotion based engagement, this naturally follows.  Could further be used to rate the quality of the instruction.    Where does this kind of measurement end, even if  not completely accurate?

Are Your Students Bored? This AI Could Tell You
IEEE Spectrum
Emily Waltz

Researchers at the Hong Kong University of Science and Technology (HKUST) and China's Harbin Engineering University have created an artificial intelligence (AI) system that analyzes students' emotions based on video recordings of their facial expressions to measure their engagement level in a class. The AI system was tested in a classroom of toddlers in Japan and a classroom of university students in Hong Kong. While the visual analytics system was successful in detecting obvious emotions like happiness, it often incorrectly reported anger or sadness in students who were focused on the lectures. Said HKUST computer scientist Huamin Qu, "To address this issue, we need to add new emotion categories, relabel our data, and retrain the model."  ..."

Banning the Prediction of Emotion?

We examined and experimented with aspects of the idea for 20 years.   A number for these efforts are touched on in this blog, see the tags below.  But now its seen as dangerous.  Another thing I see as inevitable,   for now regulation will only occur in the West.

Researchers Criticize AI Software That Predicts Emotions    By Reuters

The perceptions of facial expressions by artificial intelligence is unreliable, according to researchers at the AI Now Institute.

Researchers alarmed by the harmful social effects of artificial intelligence have called for a ban on automated analysis of facial expression in hiring and other major decisions.

The AI Now Institute at New York University is urging a ban on the use of artificial intelligence (AI) that automatically analyzes facial expressions to influence hiring and other decisions.

The Institute cited an academic review of studies on how people interpret moods from facial expressions, which concluded that such perceptions are unreliable, given that emotional communication widely varies across cultures, scenarios, and individuals in a single situation.

The Institute said action against such software-driven “affect recognition” was its top priority, because science does not justify the technology's use.

Institute founders Kate Crawford and Meredith Whittaker warn of the spread of damaging applications of AI despite broad consensus on underlying ethical principles, due to a lack of consequences for violating those principles.   ... ' 

Emotion Sensing Empathetic Robot in Space

An empath assistant.   Will this be a function of all future assistant models as well.  Or a dangerous, too close to home idea?   Good experiment in a closed context.

Emotion-sensing robot headed to International Space Station to aid astronauts' mental health
An intelligent robot equipped with emotion-sensing voice detectors is headed to the International Space Station after launching in Florida, becoming the latest artificial intelligence-powered astronaut workmate in orbit.

Key points:
CIMON is built with a raft of software designed to enable emotion recognition

It will test technologies that could prove crucial for future crewed missions in deep space, where long-term isolation poses risks to astronauts' mental health

Engineers say the concept was inspired by a 1940s comic series about a robot named Professor Simon

The Crew Interactive Mobile Companion 2, or CIMON 2, is a spherical droid with microphones, cameras and a slew of software to enable emotion recognition.

The droid was among 2,585kg of supplies and experiments aboard SpaceX's Falcon 9 rocket, whose launch had been delayed from Wednesday (local time) due to high winds. .... "

New Alexa Emotions

New complexity in voice expression is announced for Alexa.  Need to see this in useful context, but intrigued by possibilities.

Use New Alexa Emotions and Speaking Styles to Create a More Natural and Intuitive Voice Experience      By Catherine Gao

We’re excited to introduce two new Alexa capabilities that will help create a more natural and intuitive voice experience for your customers. Starting today, you can enable Alexa to respond with either a happy/excited or a disappointed/empathetic tone in the US. Emotional responses are particularly relevant to skills in the gaming and sports categories. Additionally, you can have Alexa respond in a speaking style that is more suited for a specific type of content, starting with news and music. Speaking styles are curated text-to-speech voices designed to create a more delightful customer experience for specific content. For example, the news speaking style makes Alexa’s voice sound similar to what you hear from TV news anchors and radio hosts. To learn more, check out our technical documentation for emotions here and speaking styles here.

How Alexa Emotions Work
Alexa emotions use Neural TTS (NTTS) technology, Amazon’s text-to-speech technology that enables more natural sounding speech. For example, you can have Alexa respond in a happy/excited tone when a customer answers a trivia question correctly or wins a game. Similarly, you can have Alexa respond in a disappointed/empathetic tone when a customer asks for the sports score and their favorite team has lost. Early customer feedback indicates that overall satisfaction with the voice experience increased by 30% when Alexa responded with emotions. Check out the following examples and compare them to the neutral tone:  .... ' 

AI Powered Wireless Emotion Sensing

Another ability to sense neuromarketing signals?

EmoSense: an AI-powered and wireless emotion sensing system    by Ingrid Fadelli, Tech Xplore

Researchers at Hefei University of Technology in China and various universities in Japan have recently developed a unique emotion sensing system that can recognize people's emotions based on their body gestures. They presented this new AI- powered system, called EmoSense, in a paper pre-published on arXiv.

"In our daily life, we can clearly realize that body gestures contain rich mood expressions for emotion recognition," Yantong Wang, one of the researchers who carried out the study, told TechXplore. "Meanwhile, we can also find out that human body gestures affect wireless signals via shadowing and multi-path effects when we use antennas to detect behavior. Such signal effects usually form unique patterns or fingerprints in the temporal-frequency domain for different gestures."

Wang and his colleagues observed that human body gestures can affect wireless signals, producing characteristic patterns that could be used for emotion recognition. This inspired them to develop a system that can identify these patterns, recognizing people's emotions based on their physical movements.  .... "

Amazon Reads Emotions by Voice

We tested early models of 'emotion glasses' to detect emotion in interactions with consumer products.

Amazon's rumored wearable device reads your emotions by listening to your voice  Open the pod bay doors, Alexa.     By Michael Simon

 Alexa already brings joy to the millions of people who receive its packages, but now it wants to know how you’re feeling during the other 1,400 minutes in your day. Bloomberg reports that Amazon is developing a new wearable device “that can discern the wearer’s emotional state from the sound of his or her voice.”

Bloomberg describes the device as a wrist-worn gadget marketed as a health and wellness product. It reportedly has “microphones paired with software” and “could be able to advise the wearer how to interact more effectively with others.” The device is being developed by the same teams responsible for the Echo speakers and Alexa voice recognition, Bloomberg says. The same group is is also reportedly working on a home robot companion with “advanced cameras and computer vision software and can navigate through homes like a self-driving car.”

If that sounds like the stuff of sci-fi dystopian nightmares, you’re not alone. We already rely on Alexa to turn out the lights and remind us to send birthday cards, but allowing Amazon to track our emotions is another thing entirely. It’s not hard to imagine a day when the tech could be built into an Echo speaker and suggest the latest season of The Marvelous Mrs. Maisel if it thinks we’re feeling blue. ... "

AI More Human by Adding Dilemma

Wondering if the Prisoners Dilemma is the right thing to use here,  but agree that showing typical human emotions can bring sympathy to a machine as it can to humans.  Context dependent though, and the PD is an contrived context.   Not a big enough sample either.    Also its interesting that these are specifically 'avatars', or human-like agent.   Would it be different if was not an avatar?   Like that the test is being done though.   Perhaps one way of certifying AI agents.

Making AI More Human 
University of Waterloo News

A study by researchers at the University of Waterloo in Canada found that adding appropriate emotions to artificial intelligence (AI) avatars would make humans more accepting of them. Waterloo's Moojan Ghafurian, Neil Budnarain, and Jesse Hoey employed the classic Prisoner's Dilemma game, substituting one of two human "prisoners" with a virtual AI developed at the University of Colorado, Boulder. The researchers utilized virtual agents that evoked either neutrality, appropriate emotions, or random emotions. Participants cooperated 20 out of 25 times with the AI that exhibited human-like emotions, 16 out of 25 times for the agent with random emotions, and 17 out of 25 times for the emotionless agent. Ghafurian said, "Showing proper emotions can significantly improve the perception of humanness and how much people enjoy interacting with the technology." ... .'