Emotional Sensing Examined for the Metaverse

Emotional Sensing,  much examined, new methods for the metaverse?

Nemesysco spinoff unveils emotional detection and AI tools for metaverse  excerpt in Venturebeat

As organizations continue to move toward digital transformation, enterprise decision-makers are seeing a greater need for data-driven decisions backed by high-quality analytics. A key challenge organizations face is the lack of insights into the different types of customer inquiries they receive and the reason for customer contact, according to a report by Gartner. 

Steve Blood, a vice president and analyst at Gartner, said in the report that organizations can use speech/voice analytics to optimize contact center costs with self-service, process improvement and deeper engagement. “Application leaders for customer service and support technology who are under pressure to cut costs should first use voice analytics to discover which types of inquiries could be automated or avoided, before addressing improvements in customer engagement via the phone or digital channel,” said Blood.

Voice analytics holds great promise, with Gartner predicting 60% of organizations with voice of the customer (VoC) programs will supplement traditional surveys by analyzing voice and text interactions with customers by 2025. While an article by McKinsey notes voice analytics can provide a high-quality customer experience in a digital age, it isn’t the only use case for voice analytics. For example, voice analytics can also help organizations improve operational efficiency across the board. 

Another report, by Deepgram and Opus Research, shows 77% of companies are using voice technology to identify new business opportunities, and 62% are using it to increase revenues. The use cases for voice technology are enormous and Israel-based voice analysis technology company, Nemesysco, wants to drive new frontiers in the industry.

Nemesysco is in the process of spinning off a new voice analytics company called Emotion Logic, which will use artificial intelligence (AI) to detect and measure human emotions and improve automated communications and interactions in metaverse worlds.  ... ' 

Directional Smells, Virtual Odors, Stereo Smell

In our innovation enters we tried to enhance smell,  but not usually directionally.  It was expected that the people would determine changes in smell by their own motion.   Might be useful for example in detecting gas leaks in disaster relief operations.  Really a quite interesting change in how aroma sense might be redefined.   But is that useful? 

Digital Nose Stimulation Enables Smelling in Stereo

IEEE Spectrum, Evan Ackerman, May 13, 2021  via ACM

Researchers at the University of Chicago (UChicago) have developed a wireless, nose-worn device that can enable directional smell using tiny electrical impulses. The battery-powered hardware has magnets to keep itself attached to the inside of the nose; it can detect inhalation, and employs electrodes to stimulate the septum and trigger the trigeminal nerve. The model, which connects with external sensors, works well enough that completely untrained people can use it to localize electrically-triggered virtual odors. UChicago's Jas Brooks said, "The sensation our device produces can feel like a 'tickling' or 'sting,' not far from that of wasabi or the smell of white vinegar, except it is clearly directional." The stimulator could be used as an assistive device for people suffering loss of olfactory function.  .. "

Mimicking the Sense of Touch

We examined how consumers interacted with product in retail.

Glove-Like Device Mimics Sense of Touch  University of New South Wales Sydney Newsroom,   Caroline Tang

A soft wearable device developed by engineers at the University of New South Wales (UNSW) Sydney in Australia recreates the sense of touch using haptic technology. The researchers created a three-way directional skin stretch device that is built into the fingertips of the haptic glove. Said UNSW's Thanh Nho Do, "Our soft, wearable haptic glove enables people to feel virtual or remote objects in a more realistic and immersive way. The inbuilt soft artificial muscles generate sufficient normal and shear forces to the user's fingertips via a soft tactor, enabling them to effectively reproduce the sense of touch."

On Multisensory Adventures

Vint Cerf on a favorite topic.   The senses.  He points to two inspirational books on the topic.  We are starting to 'understand' senses, at least if we use as a measure how we are able to mimic them.

Multisensory Adventures
By Vinton G. Cerf
Communications of the ACM, April 2020, Vol. 63 No. 4, Page 7
10.1145/3383671

Google Vice President and Chief Internet Evangelist Vinton G. Cerf 
In this column, I want to draw your attention to two books. One has been published to great acclaim and the other is still in process. They resonate with a visceral intensity for which I was honestly unprepared and surprised. The first, Multisensory Experiences, Where the Senses Meet Technology, by Carlos Velasco and Marianna Obrist, is to be published by Oxford University Press. The authors explore concepts we experience every day but don't necessarily understand fully. We are familiar with the five senses (sight, sound, touch, taste, smell). Our brains transduce these physical phenomena into neural pulses that flood along many pathways and interact in many ways. Interestingly, all these senses are translated into essentially similar neural signals but they are processed in a complex and interconnected neural web producing what we call experience.  .... "

Robotic Light Finger Feels

An idea that creates haptics for robots.

This Clever Robotic Finger Feels With Light
By Wired  in ACM

Robot finger meets actual finger.
A robotic skeleton developed at Columbia University is equipped with 32 photodiodes and 30 adjacent LEDs and is covered by a squishy skin of reflective silicone.

Researchers at Columbia University have developed a robotic skeleton equipped with 32 photodiodes and 30 adjacent LEDs and is covered by a squishy skin of reflective silicone, which keeps the device's own light in and outside light out.

When the robot finger touches an object, the soft exterior deforms, and the photodiodes detect changing light levels from the LEDs.

The system can determine where contact is being made with the finger, and the intensity of that contact.   The 32 photodiodes and the 30 LEDs produce 960 signals, a massive amount of data from a single poke.

The system relies on machine learning to analyze all of the information.
This type of tactile sensing can facilitate robot manipulation, and this new system is a significant improvement over previous robotic fingers that used electrodes overlaid with rubber to sense touch. .... " 

Detecting Odors

I often mention research in this area because we spent considerable time looking at the idea of an 'artificial nose' to effectively  test quality in products, especially coffee, but in other areas as well.    Here more in the area.  Note the use of 'Neuromorphic', or brain-inspired chips to address the problem.

Intel Trains Neuromorphic Chip to Detect Odors
in VentureBeat
By Kyle Wiggers

Intel and Cornell University researchers have trained Intel's Loihi neuromorphic processor to identify 10 materials from their odors, demonstrating how neuromorphic computing could be applied to detect precursor smells and potentially find explosives and narcotics, diagnose diseases, and notice signs of smoke and carbon monoxide. The chip was trained by configuring the circuit schematic of biological olfaction, using a dataset compiling the activity of 72 chemical sensors in response to various scents. The researchers said the method kept Loihi's memory of the scents intact, and the chip has "superior" recognition accuracy compared with conventional techniques. Said Intel's Nabil Imam, "This work is a prime example of contemporary research at the crossroads of neuroscience and artificial intelligence and demonstrates Loihi's potential to provide important sensing capabilities that could benefit various industries." ... '

Teaching Automobiles to Hear

Makes sense to add another sense to the car.

Teaching Tomorrow's Automobiles to Hear
Fraunhofer-Gesellschaft
Christian Colmer  in ACM

Researchers at Germany’s Fraunhofer Institute for Digital Media Technology (IDMT) have developed a system that allows self-driving cars to recognize external noises such as sirens. The team trained the system to recognize the acoustic signature of specific sound events, using machine learning methods that rely on acoustic libraries. In addition, the researchers created beamforming algorithms that enable the system to dynamically locate moving sound sources. The researchers also developed artificial intelligence-based algorithms to distinguish specific noises the system is designed to identify from other noise. Said IDMT's Danilo Hollosi, “Such systems would be able to immediately recognize the siren of an approaching emergency vehicle, for example, so that the autonomous vehicle would then know to move over to one side of the highway and form an access lane for the rescue services." .... '

Better to Design for Multiple Senses

Nice piece in ArsTechnica which makes the point that we should be designing interactive interfaces for multiple senses, not just for the limitations of one latest, easiest one.     And beyond that we should be consider the power of ritual and tactile elements in that interface.     Does this also have lessons for augmented reality?

Why vinyl records survive in the digital age
Don't underestimate ritual and tactility.
By Steven Brykman in Arstechnica. ....

Research on Sound in Shopping

We researched the idea of how sound changed shopping behavior,  in laboratory and real store environments.  Ambient background music can create mood.  Focused multidimensional sound can influence interest in specific areas of a shelf.  Sound can be mixed with visual cues to change attention.   More on this topic can be found here at the 'Sound' and "Senses" tags.  Note how this is a means of augmented reality.    Here is more recent research with expert comments at the link.

What’s the ideal soundtrack for grocery shopping?    by Tom Ryan in Retailwire.

A new study from the University of South Florida finds that loud music in a restaurant or grocery store leads to more unhealthy food choices, while quieter music leads to heathier ones.

Ambient music can influence healthy or unhealthy food buying, according to the researchers, because it directly impacts heart rate and arousal. Softer music has a calming effect, making people more mindful of what people order in a restaurant. Louder environments increase stimulation and stress, leading to less-mindful choices.

Beyond restaurants, the research offers insights into how grocers can influence in-store buying behavior.  ..... " 

Interesting Sounds Podcast

Discovered a podcast called 'Twenty Thousand Hertz',  which is an examination of 'the most interesting stories of the most interesting sounds'  by Dallas Taylor.  There I learned about sound fatigue and sound scape design in casinos. We experimented broadly with this idea designing soundscapes in retail stores.  I had worked with some experimenters in college on sound immersion.  Could have used this podcast.  It is available here and on Alexa and Google Home.  Sound is another sense we don't pay enough attention to in the way it can influence and even prescribe behavior.

AI and Smell Digitization Classification, and Health

We actively worked on digitally reproducing the senses of smell and taste.   Both in their classification and their production.   In one case for taste of products like coffee and also the analysis of aromas produced by perfumes in context.   In either case it can convert such recipes into digital form, which are more manipulable.  Both of these challenges depend on very low amounts in the atmosphere.  And now, Since Machine Learning AI can detect subtle patterns,  there is cause to believe we can solve these problems.   Further, the sense of smell is being used to detect the state of human health.

Seeing, Sensing and Reproducing Color

In the New Yorker:   The Search for Our Missing Colors, By Amos Zeeberg.  You would think that in our digital world we could sense, see and reproduce most any color.  Well, not quite.  What are the implications for the increasingly digital world of a lack of richness?  I recall in a project that sought to archive color advertisements, we were unable to do this even close to usefully.

David Eagleman's: The Brain

Have been watching David Eagleman's six part popular neuroscience program on PBS: The Brain.   Very nicely done non-technical and personal overview of recent learnings in neuroscience.   A number  of his other works are worth examining.   Will follow with more.

More about the series and online streaming.     
Eagleman's site regarding the programs.   #BrainPBS

TED Talk.   Can we create new senses for humans?

Designing for All Senses

In FastcoDesign:
" .. Our perceptions of the world are built on multiple senses. They interact to help us make sense of our surroundings, so a car will seem to be traveling faster if it makes a lot of noise. And whether we want to buy a sports car, or a vacuum cleaner, it’s our multi-sensory impression of a product or service that dictates how we feel about it. And why companies need to learn the skills to design for all five senses.

 Is Multi Sensorial Design a New Approach?
The concept of multi-sensory design has existed for more than a decade. But want I address the idea of multi-sensory design as it applies to brands. I believe that applying multi-sensory design to all the touchpoints (moments of contact with a user) allows a product or service to produce a more complete, and ultimately better brand experience.  ..... " 

Feeling Virtual Reality

In TechDirt: An area we examined to communicate feelings like 'softness' to consumers.    This is a Kickstarter thing, don't know how well it works, but worth an examination.

" ... True virtual reality — the kind we've been envisioning now for decades — is inching ever closer to, well, reality. But while most of the focus is new devices that let you see a virtual world, there are some people out there working on the other senses to round out the simulated experience. This week, we're looking at Gloveone, a haptic feedback system that wants to do for touch what devices like the Oculus Rift are doing for sight. ... " 

Multi Sensory Marketing

Advertising through multiple sense.  In Shopperception.  '... When two or more senses receive stimulation at the same time - cause shoppers to stay in the store for a longer time and buy more products ...  " 

Drone with Oculus Support

IEEE Spectrum reports on a natural progression, the interaction of physical devices, like drones, and Augmented reality interfaces, like the Oculus Rift. display.   The connection between our senses and mobile sensory devices.  A way of extending your mobile range, with many of the issues about privacy and regulation.  We will see more of this developing.  With video and many images.

Wearables as a Sixth Health Sense

In CWorld:  How do you think of wearable computing?  We have become very used to having a device next to us that acts as sensor, library and communicator.  So is it another sense that we need to treat the same way as our others?  How do we combine it with the other senses?   In particular about health:

" .... People will one day depend on wearable computers to monitor not just their activities but a myriad of data about their health, making the devices basically like a sixth sense. ... That's the vision that was laid out during the MIT Tech Conference on disruptive technologies in Cambridge, Mass. this past weekend. The wearable computer market will see the kind of dramatic growth that the smartphone market has over the last decade and wearables will morph from Fitbit-like bracelets to patches that stick to users' skin and sensors embedded in t-shirts and sneakers. ... " 

Data Sonification

A space I looked at for our information innovation group.  A way to use other senses, including sound, as an interface to deliver data and its patterns.  AT&T had done some experimentation in this area.  In the simplest approach, alerts can be transmitted with sound, and the use of pagers in manufacturing spaces were tried.  Today's smartphone can link calls to specific rings. Taking this forward, to delivering more precision describing data with sound, is more difficult.  Can you get your data to scream out to you?   Today it is likely that this method would be used in conjunction with visualization techniques.  A link to more recent work at Drexel University.  See also, Drexel's Excite Center, which looks at innovative interfaces.

IKEA and Augmented Reality Catalog

See furniture in your own apartment using augmented reality in IKEA's 2014 catalog.  A good example of AR in mobile retail.  And its potential for using AR to insert the virtual to measure the physical.  " ... To take advantage of IKEA‘s augmented reality catalog, customers download a program through the Apple App Store or Google Play. The catalog is placed within the room the user wants to decorate, and the application senses and measures the booklet itself to create accurate images. Products then appear on your smartphone’s screen in the correct size and color. IKEA hopes their catalog will help to cut down in accidental purchases, or what they refer to as “square peg, round hole syndrome”. They estimate that seventy percent of people do not know the area of their homes, and fourteen percent have admitted to buying items that were the wrong dimensions for their rooms.... "