Complexity of Personalized Bartending

The task of bartending has often been proposed as a classic automatable human interaction, here a new development of interest.   Social aspects claim to be included.    Could be instructive.

A bartending robot that can engage in personalized interactions with humans    by Ingrid Fadelli , Tech Xplore

A widely discussed application of social robots that has so far been rarely tested in real-world settings is their use as bartenders in cafés, cocktail bars and restaurants. While many roboticists have been trying to develop systems that can effectively prepare drinks and serve them, so far very few have focused on artificially reproducing the social aspect of bartending.

Researchers at University of Naples Federico II in Italy have recently developed a new interactive robotic system called BRILLO, which is specifically designed for bartending. In a recent paper published in UMAP '22 Adjunct: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, they introduced a new approach that could allow their robot to have personalized interactions with regular customers.

"The bartending scenario is an extremely challenging one to tackle using robots, yet it is also very interesting from a research point of view," Prof. Silvia Rossi, one of the researchers who carried out the study and the scientific coordinator of the project, told TechXplore. "In fact, this scenario combines the complexity of efficiently manipulating objects to make drinks with the need to interact with the users. Interestingly, however, all current applications of robotics for bartending scenarios ignore the interaction part entirely."

Silvia Rossi and her colleagues Alessandra Rossi and Nitha Elizabeth John believe that to effectively take on the role of a bartender, a robot should not only be able to interact with humans, but it should also create a "profile" of users. This would allow it to personalize its interactions with regular customers, increasing the likelihood that they will like and continue using the robotic bartending service.

"In the end, we think a barman should not only be the one who remembers your tastes, but also your interests and your daily life. Just like human bartenders, they should sometimes act as close friends," Rossi said.

BRILLO, the bartending robot used by Rossi, Rossi and John consists of a humanoid bust with two robotic arms, which allow the robot to make drinks, and a monitor-based face that can produce different facial expressions. The robot also features a microphone, speaker and camera that allow it to capture images of customers and pick up their body language, process what they are saying and answer back to them.

Paying Older Audience Attention

Brought to my attention, being in the older, more experienced audience myself, struck me as rarely paid attention to.   Good first look at this kind of resource problem.

Your Messaging to Older Audiences Is Outdated   by Hal Hershfield and Laura Carstensen

July 02, 2021

Summary.   Given a rapidly aging population, effective messaging to older people holds national importance for public health as well as marketing of goods and services. Older people make up an incredibly diverse demographic that varies in terms of physical and cognitive ability,...more

One of the most pressing concerns in the early days of the Covid-19 pandemic was how to best communicate information to those who were at greatest risk — particularly, the elderly. Unfortunately, many attempts were riddled with stereotyped depictions of older people as frail, lonely, and incompetent. In doing so, messages from advertisers, public health officials, and policymakers may have failed to resonate with large swaths of their targeted audience. Given a rapidly aging population, effective messaging to older people holds national importance for public health as well as marketing of goods and services.

Arguably, the greatest challenge is market segmentation. Older people make up an incredibly diverse demographic that varies in terms of physical and cognitive ability, economic power, and social connection. Aging is also changing over historical time. Several studies have shown that the incidence of dementia appears to be decreasing over time; some research suggests this is due to higher educational attainment and improvements in cardiovascular health. Today’s older generations are less lonely and happier than their younger counterparts. As a result, market segmentation based on chronological age is becoming increasingly difficult, if not futile.

A more telling predictor of behavior and a better approach to age segmentation may be time left in life rather than time since birth. Healthy versus sick offers more meaningful insight than whether someone is in their 70s or their 80s.  ... 

Challenge for Learning from Human Feedback using Minecraft

Berkeley Bair challenge competition here using a common gaming environment.   Been a long time since I looked at Minecraft.  Short extract of the idea below, more complete look at the link.   Seems a novel look at a broader look at contextual learning. 

 BASALT: A Benchmark for  Learning from Human Feedback   by Rohin Shah    Jul 8, 2021

TL;DR: We are launching a NeurIPS competition and benchmark called BASALT: a set of Minecraft environments and a human evaluation protocol that we hope will stimulate research and investigation into solving tasks with no pre-specified reward function, where the goal of an agent must be communicated through demonstrations, preferences, or some other form of human feedback. Sign up to participate in the competition!

Motivation

Deep reinforcement learning takes a reward function as input and learns to maximize the expected total reward. An obvious question is: where did this reward come from? How do we know it captures what we want? Indeed, it often doesn’t capture what we want, with many recent examples showing that the provided specification often leads the agent to behave in an unintended way.

Our existing algorithms have a problem: they implicitly assume access to a perfect specification, as though one has been handed down by God. Of course, in reality, tasks don’t come pre-packaged with rewards; those rewards come from imperfect human reward designers.

For example, consider the task of summarizing articles. Should the agent focus more on the key claims, or on the supporting evidence? Should it always use a dry, analytic tone, or should it copy the tone of the source material? If the article contains toxic content, should the agent summarize it faithfully, mention that toxic content exists but not summarize it, or ignore it completely? How should the agent deal with claims that it knows or suspects to be false? A human designer likely won’t be able to capture all of these considerations in a reward function on their first try, and, even if they did manage to have a complete set of considerations in mind, it might be quite difficult to translate these conceptual preferences into a reward function the environment can directly calculate.  ...................

Conclusion

We hope that BASALT will be used by anyone who aims to learn from human feedback, whether they are working on imitation learning, learning from comparisons, or some other method. It mitigates many of the issues with the standard benchmarks used in the field. The current baseline has lots of obvious flaws, which we hope the research community will soon fix.

Note that, so far, we have worked on the competition version of BASALT. We aim to release the benchmark version shortly. You can get started now, by simply installing MineRL from pip and loading up the BASALT environments. The code to run your own human evaluations will be added in the benchmark release.

If you would like to use BASALT in the very near future and would like beta access to the evaluation code, please email the lead organizer, Rohin Shah, at rohinmshah@berkeley.edu.

This post is based on the paper “The MineRL BASALT Competition on Learning from Human Feedback”, accepted at the NeurIPS 2021 Competition Track. Sign up to participate in the competition!   

Virtual Reality to Assess Cognitive Abilities

 Heard of this kind of approach somewhere.   Either for determining brain and tactile health, or to use after specialized training, to assess abilities.   

Researchers create virtual reality cognitive assessment

by Center for BrainHealth in TechXplore

Virtual reality isn't just for gaming. Researchers can use virtual reality, or VR, to assess participants' attention, memory and problem-solving abilities in real world settings. By using VR technology to examine how folks complete daily tasks, like making a grocery list, researchers can better help clinical populations that struggle with executive functioning to manage their everyday lives. ... " 

Is There a Need Apologetic AI?

 Now this really struck me.   As a useful consideration.  When humans do tasks they do apologize, sometimes appropriately, sometimes not.  Makes them more human.   Recalled we talked about doing this for one of our pieces of 'brand equity' that talked to consumers.   It was way out there on the limb of the decision tree.    But the reaction was removed.  Was it because of one of our many talks with our lawyers?  Maybe, but I forget.   Sorry.   

Apologetic AI Is A Somewhat Sorry Trend, Especially For Autonomous Cars  

AI developers are beginning to enable self-driving cars to apologize to passengers after driving mishaps, causing Lance Eliot to consider the implications. 

By Lance Eliot, the AI Trends Insider   

That’s the latest trend for AI that directly interacts with people. The notion seems to be that if the AI has to deliver unfavorable news or appears to have made a potential mistake, it ought to be civil about the matter and emit an apology. AI developers are either opting to include the apology-generating capacity or they are being pressed by system designers and managers to infuse the “sorry about that” capability.   

This might seem at first glance as a marvelous addition to an AI system and would presumably be valuable to construct. Sorry to say that the AI being apologetic has both upsides and downsides. 

Let’s begin by considering a context that will help to reveal the pros and cons of AI-powered apologies. Imagine that you apply online for a car loan and the AI system determines that you are not worthy, as it were, and promptly turns you down. The belief is that this would be an ideal moment for the AI to offer you a “heartfelt” apology.   

It might go something like this: Dear loan applicant, it is with great sorrow that I must inform you of the unfortunate news that your request to borrow funds to buy a car is hereby denied. Please know that you are not alone in having been spurned and accept this apology for any discomfort that might arise from this outcome. Sincerely, the AI system that reluctantly rebuffed your request.   

Do you think this apology will make the person feel any better about the AI-powered decision?   ... "

Moving AI Recognized Things

Interesting generalization of a set of common tasks using AI identification.

Robotics Researchers Propose AI That Locates, Safely Moves Items on Shelves

Venture Beat   By Kyle Wiggers in CACM

Two new robotics studies detail methods for locating occluded objects on shelves and solving "contact-rich" manipulation tasks. Researchers at the University of California, Berkeley developed the Lateral Access maXimal Reduction of occupancY support Area (LAX-RAY) system, which predicts an object's location even when only a portion of it is visible. LAX-RAY achieved 87.3% accuracy in a simulation, which translated to about 80% for a real-world robot. Meanwhile, Google developed the Contact-aware Online COntext Inference (COCOI), which uses video footage and readings from a robot-mounted touch sensor to encode dynamics information into a representation, which then permits a reinforcement learning algorithm to plan with “dynamics-awareness,” increasing its robustness in difficult environments.  ... ' 

Multi-Agent VR for Task Application?

Might this be useful for simulating multi-agent complex, interactive tasks in VR?    We encountered such problems when assigning multiple people to do a job and they needed to be informed of the location, status, actions ...   of others in the team.  This inhibited the use of VR solutions.   Like too that this could be done with simple devices.

Novel System Allows Untethered Multi-Player VR
Purdue University News  By Chris Adam

Purdue University researchers have created a virtual reality (VR) system that allows untethered multi-player gameplay on smartphones. The Coterie system manages the rendering of high-resolution virtual scenes to fulfill the quality of experience of VR, facilitating 4K resolutions on commodity smartphones and accommodating up to 10 players to engage in the same VR application at once. Purdue's Y. Charlie Hu said Coterie "opens the door for enterprise applications such as employee training, collaboration and operations, healthcare applications such as surgical training, as well as education and military applications."

Simple Uses of RPA in Business

Good example of the simple use of RPA in business

AI Is Coming for Your Most Mind-Numbing Office Tasks  By Will Knight in Wired

Routine work, like cutting and pasting between documents, is increasingly being automated. But for now, there's little artificial intelligence involved.

In 2018, the New York Foundling, a charity that offers child welfare, adoption, and mental health services, was stuck in cut-and-paste hell.

Clinicians and admin staff were spending hours transferring text between different documents and databases to meet varied legal requirements. Arik Hill, the charity’s chief information officer, blames the data entry drudgery for an annual staff turnover of 42 percent at the time. “We are not a very glamorous industry,” says Hill. “We are really only just moving on from paper clinical records.” ... "

Shows how the company used UiPath to implement Robotic Process Automation (RPA) with UiPath  http://www.uipath.com/

Show Your Robots How to do Chores

The ultimate desire.  Or perhaps have someone else show a robot how to do your chores better.  And archive the results for use.  Much like the inclusion of 'planning with uncertain specifications' ... the humanlike planning ability to simultaneously weigh many ambiguous — and potentially contradictory — requirements ... '

Showing robots how to do your chores
By observing humans, robots learn to perform complex tasks, such as setting a table.
Watch Video at this post.
Rob Matheson | MIT News Office

Training interactive robots may one day be an easy job for everyone, even those without programming expertise. Roboticists are developing automated robots that can learn new tasks solely by observing humans. At home, you might someday show a domestic robot how to do routine chores. In the workplace, you could train robots like new employees, showing them how to perform many duties.

Making progress on that vision, MIT researchers have designed a system that lets these types of robots learn complicated tasks that would otherwise stymie them with too many confusing rules. One such task is setting a dinner table under certain conditions.  

At its core, the researchers’ “Planning with Uncertain Specifications” (PUnS) system gives robots the humanlike planning ability to simultaneously weigh many ambiguous — and potentially contradictory — requirements to reach an end goal. In doing so, the system always chooses the most likely action to take, based on a “belief” about some probable specifications for the task it is supposed to perform.   .... "

Autonomous Drone piloting

Could have used this for our forestry resource measurements, which required expensive and time consuming surveys by road or helicopter.  Further doing this to determine embedded tasks for resource analysis. 

Drones can now scan terrain and excavations without human intervention  by Aarhus University in Techexplore

Drone pilots may become superfluous in the future. New research from Aarhus University has allowed artificial intelligence to take over control of drones scanning and measuring terrain.

A research project at Aarhus University (AU) in collaboration with the Technical University of Denmark (DTU) aims to make measuring and documenting gravel and limestone quarries much faster, cheaper and easier in the future.

The project has allowed artificial intelligence to take over the human-controlled drones currently being used for the task.

"We've made the entire process completely automatic. We tell the drone where to start, and the width of the wall or rock face we want to photograph, and then it flies zig-zag all the way along and lands automatically," says Associate Professor Erdal Kayacan, an expert in artificial intelligence and drones at the Department of Engineering at Aarhus University.  ... " 

Assigning Machines to Tasks

Interesting question when we start to share work.  How is it most effectively done?   Thoughts from Kellogg linked to at the short Podcast linked to below.   But  I would offer that machines are still too simplistic to do this well.   Until now we have just applied them where they work best.

Podcast: How You Should Divvy Up Work between People and Machines

On this episode of The Insightful Leader: strategies for building a happier, more productive workplace.

Machines are taking on more and more new responsibilities at work. But are some jobs better left to humans?

On this episode of The Insightful Leader, Adam Waytz, associate professor of management and organizations at Kellogg and author of the book The Power of Human: How Our Shared Humanity Can Help Us Create a Better World, offers three guidelines for how managers can play to the unique strengths of both people and technology.    ... " 

IBM and MIT on the Future of Work

Posted this previously, now just taking a dive,  worthwhile read

The Future of Work: How New Technologies Are Transforming Tasks- Artificial intelligence - IBM Research and MIT  .... Work is changing ... '

Robots, AI and Machine Learning Will be Changing Your Jobs

All jobs will have their nature changed, not unlike how computing in general has changed many jobs.    MIT and IBM are studying the changes and what they mean.

The Future of Work: How New Technologies Are Transforming Tasks- Artificial intelligence - IBM Research

AI May Not Kill Your Job—Just Change It  via Wired.
Don't fear the robots, according to a report from MIT and IBM. Worry about algorithms replacing any task that can be automated: 

" ... Technology has long brought change to the nature of work, and to the skills required for the most desirable, best-paying jobs. But until recently, new technology – even robotics – has tended to mean automating repetitive or arduous tasks, while often leading to new types of tasks for workers. The emergence of artificial intelligence (AI) and machine learning (ML) poses a new set of opportunities – and challenges – for work and workers. The tasks that can be done by machine learning are much broader in scope than previous generations of technology have made possible. The expanded scope will change the value employers place on tasks, and the types of skills most in demand. As AI and machine learning transform businesses and reshape industries, the innovators of these technologies must consider not only the business implications, but also the societal impact. As a result, the MIT-IBM Watson AI Lab has engaged in a first-of-a-kind research that sheds new light on the reorganization of tasks within occupations by analyzing 170 million online job postings in the US between 2010 and 2017. There is no question that AI and related technologies will affect all jobs. The research reveals how tasks are transforming and what the implications are for employment and wages.   ... " 

47 Page pdf complete report.

Understanding the Digitization of the Workforce

Came to this piece by seeing the mention of O*NET, which we also used to look at the assemblage of work based on looking at component tasks.    Irving does a great job looking at workplace digitization, here a short excerpt:

By Irving Wladawsky-Berger:

The Digitalization of the American Workforce

“Over the past half century, wave after wave of digital innovation has ensured that digitalization - the diffusion of digital technologies into nearly every business and workplace and pocket - has been remaking the U.S. economy and the world of work,” said the Brookings Institution in Digitalization and the American Workforce.  The digitalization of everything has both increased the potential of individuals and societies while contributing to widespread anxiety about its impact on jobs and economic inequalities.  “And yet, for all of the evidence that big changes are underway, surprisingly little data exists to track the spread of digital adoption.” .... 

The Brookings report quantified the spread of digitalization by analyzing the changes in the digital content of 545 occupations between 2002 and 2016 based on O*NET, the most comprehensive data on US occupations sponsored by the US Department of Labor.  O*NET includes highly detailed, task-level information on hundreds of occupations, and includes quantitative measures of knowledge requirements, skills, tools, education, training and work activities. In addition, the report collected information on the occupations’ particular industries, pay, growth rates, geographical locations and distribution across educational and demographic groups. .... '

At Work: Specialists vs Generalists?

Specialists vs Generalists?

I think there is a need for a mix of specialist and generalists.    But at least initially more of the specialist activity will be taken over by devices.     Alerts, Notifications, Assistants, smart contracts, sensors,  ... that will add to the skill and of the number  jobs taken up by generalists.  ....

A Case of the Navy: 

At Work, Expertise Is Falling Out of Favor   In the Atlantic

These days, it seems, just about all organizations are asking their employees to do more with less. Is that actually a good idea? .... 

 " .... Minimal manning—and with it, the replacement of specialized workers with problem-solving generalists—isn’t a particularly nautical concept. Indeed, it will sound familiar to anyone in an organization who’s been asked to “do more with less”—which, these days, seems to be just about everyone. Ten years from now, the Deloitte consultant Erica Volini projects, 70 to 90 percent of workers will be in so-called hybrid jobs or superjobs—that is, positions combining tasks once performed by people in two or more traditional roles. Visit SkyWest Airlines’ careers site, and you’ll see that the company is looking for “cross utilized agents” capable of ticketing, marshaling and servicing aircraft, and handling luggage. At the online shoe company Zappos, which famously did away with job titles a few years back, employees are encouraged to take on multiple roles by joining “circles” that tackle different responsibilities. If you ask Laszlo Bock, Google’s former culture chief and now the head of the HR start-up Humu, what he looks for in a new hire, he’ll tell you “mental agility.” “What companies are looking for,” says Mary Jo King, the president of the National Résumé Writers’ Association, “is someone who can be all, do all, and pivot on a dime to solve any problem.” .... " 

AI to Drive Re-Skilling Efforts

Makes sense,  but the primary challenge will be how these skills are interlinked between each other, and among people and devices.   And how much autonomy and leadership is afforded to the AI aspects.  Still not that common in practice.  Clear that automation has been here for a long time.  Starting with calculators, computers, then on to smartphones and tablets.    Creativity will be harder than repetitive tasks, it always has been.

AI will drive reskilling in problem solving, creativity and collaboration

A study from the Economist Intelligence Unit has found that executives do not believe that artificial intelligence will lead to job losses, but staff will need retraining  By Cliff Saran in ComputerWeek.,  Managing Editor
  
Uncertainty over security and data privacy represent workers’ main concerns over automation, a study from the Economist Intelligence Unit (EIU) has reported.

EIU’s Advance of automation report, based on a survey of 502 executives in Canada, France, Germany, India, Japan, Singapore, the UK and the US, found that just 9% of respondents said they were not using any automation.

More than half of the people surveyed (51%) said they made extensive use of automation, while 40% were moderate users of it, mainly for automating highly repetitive back-office functions.

However, the EIU found that more than a quarter of respondents (27%) expect automation to create opportunities for professional growth, and a similar number (26%) said they believe it could free up time for more human interaction. Another 37% said they believe automation would serve to increase employee engagement. ..... " 

Evaluating the Use-ability of VR

Useful to see the proposal of a measure here, regarding usability. 

How Usable Is VR? 
University of Gottingen

Patrick Harms at the University of Gottingen in Germany has designed an automated process for evaluating the usability of virtual reality (VR). Harms' process, which can detect many issues with user friendliness and usability in the virtual environment, begins by recording testers' individual activities and movement, producing "activity lists." A program (MAUSI-VR) mines those lists for typical user behavioral patterns, then assesses this behavior as it relates to defined irregularities. Said Harms, "This makes it possible...to determine how well users of a specific VR are guided by it and whether they usually have to perform ergonomically inconvenient procedures during its operation." ... " 

Customer Service Help and AI Agents

Inclined to agree, with only the most difficult examples handed over to groups of human or specialty agents.

How will AI powered Customer service Help Take over Support?  in MarutiTech Blog

Automation of services has picked up its fastest pace by now, giving users the much needed facility to fulfill their regular tasks. With advanced systems powered by automated solutions, users can now book a restaurant reservation, order a pizza, book a movie ticket, hotel room and even make a clinic appointment. Customer service industry is gaining much momentum especially due to disruption of Artificial Intelligence – a technological breakthrough that has taken almost every business industry by storm.

By transforming customer service interactions, AI-powered digital solutions are prepared to improve every aspect of your business including online customer experience, loyalty, brand reputation, preventive assistance and even generation of revenue streams. Digital market moguls project that by 2020 more than 85% of all customer support communications will be conducted without engaging any customer service representatives.

This blog delves into the subject a little more to convey how AI-powered customer service can possibly help customer support agents online.  ... "

Swarms of Drones Herding Sheep

Whats most interesting here is the use of swarms of multiple drones performing a group task.   Which is what we tested , but not sheepor dogs,  Or does it?  I quote from the full artcle:  " ... A single drone can do the job of multiple dogs, farmers say. ... " .  But I assume you could look for tasks or subtasks that might be done in parallel. 

Mavic 2 drones New Zealand Farmers Have New Tool for Herding Sheep: Drones That Bark Like Dogs    in The Washington Post  By Peter Holley

 

New Zealand farmers are using drones to herd livestock, with some capable of emitting barks like dogs. One drone, the DJI Mavic Enterprise, can record sounds and play them over a loudspeaker, allowing the machine to mimic its canine counterparts. Shepherd Corey Lambeth said cows are less resistant to drones than to actual dogs, which means the machines move livestock faster, with less stress. The drones also let farmers monitor their land remotely, tracking water and feed levels, and checking on livestock health without upsetting the animals. Said farmer Jason Rentoul, "Being a hilly farm where a lot of stuff is done on foot, the drones really saved a lot of man-hours. The drone does the higher bits that you can't see [from the ground], and you would [otherwise] have to walk half an hour to go and have a look and then go, 'Oh, there was no sheep there.'"... '

Use of Cube Satellites

Recent mention of the successful use of Cube Satellites made me think of yet broader applications, and others are thinking this way.  Back to the swarms of small and relatively cheap robots  to perform tasks as a group.   Could such small devices be very cheaply launched in groups?  See:  https://en.wikipedia.org/wiki/CubeSat   And the recent use of Cube Sats for the recent  Mars Lander.

Tiny satellites could be “guide stars” for huge next-generation telescopes
Researchers design CubeSats with lasers to provide steady reference light for telescopes investigating distant planets.    By Jennifer Chu | MIT News Office 

There are more than 3,900 confirmed planets beyond our solar system. Most of them have been detected because of their “transits” — instances when a planet crosses its star, momentarily blocking its light. These dips in starlight can tell astronomers a bit about a planet’s size and its distance from its star.

But knowing more about the planet, including whether it harbors oxygen, water, and other signs of life, requires far more powerful tools. Ideally, these would be much bigger telescopes in space, with light-gathering mirrors as wide as those of the largest ground observatories. NASA engineers are now developing designs for such next-generation space telescopes, including “segmented” telescopes with multiple small mirrors that could be assembled or unfurled to form one very large telescope once launched into space. ... "