Rewarding Autonomous AIs

Thoughtful piece.    Can a human just provide some sort of reward function?    Or is creating that alone very hard?  Especially if we include some measures of risk as well.  The latter we found in practice at least if you are honest about risk.  This also clouds some the the 'future is unsupervised', things I have heard recently.   What exactly does 'unsupervised' mean?    More than just a simple lack of a measure of success, we discovered.

Stanford researchers teach robots what humans want
Researchers are developing better, faster ways of providing human guidance to autonomous robots.

By Taylor Robota

Told to optimize for speed while racing down a track in a computer game, a car pushes the pedal to the metal … and proceeds to spin in a tight little circle. Nothing in the instructions told the car to drive straight, and so it improvised.

Researchers are trying to make it easier for humans to tell autonomous systems, such as vehicles and robots, what they want them to do. 

This example – funny in a computer game but not so much in life – is among those that motivated Stanford University researchers to build a better way to set goals for autonomous systems.

Dorsa Sadigh, assistant professor of computer science and of electrical engineering, and her lab have combined two different ways of setting goals for robots into a single process, which performed better than either of its parts alone in both simulations and real-world experiments. The researchers presented the work June 24 at the Robotics: Science and Systems conference.

“In the future, I fully expect there to be more autonomous systems in the world and they are going to need some concept of what is good and what is bad,” said Andy Palan, graduate student in computer science and co-lead author of the paper. “It’s crucial, if we want to deploy these autonomous systems in the future, that we get that right.”

The team’s new system for providing instruction to robots – known as reward functions – combines demonstrations, in which humans show the robot what to do, and user preference surveys, in which people answer questions about how they want the robot to behave.

“Demonstrations are informative but they can be noisy. On the other hand, preferences provide, at most, one bit of information, but are way more accurate,” said Sadigh. “Our goal is to get the best of both worlds, and combine data coming from both of these sources more intelligently to better learn about humans’ preferred reward function.”  .... " 

Usupervised Learning is the AI Future

An outline of Yann LeCuns recent talk.    See the LeCun tag below for links to talk and slides.

The AI technique that could imbue machines with the ability to reason
Yann LeCun, Facebook’s chief AI scientist, believes unsupervised learning will bring about the next AI revolution.  .... 

by Karen Hao in Technology Review

The Revolution of AI will be Unsupervised

I attended the ACM Webinar by Yann LeCun  mentioned below.   Well done, about the history and future of AI.   I will point to the slides and audio when they are announced this week.  A great, although often technical Deep Learning and AI.   Too hurried, but still very good if you are willing to go back through it.    A view of the history and predicted future of deep learning.    I think there will be some breakthrough in the addition of 'deep logic' to get reasonably general AI.   And also get better transparency.  Yes, we may get the logic of babies,  but ultimately need to break beyond to that of reasoning adults.  Continue to watch this thread.

Also watching the evolution of intelligence in my granddaughter at age 2, and right, there is lots that is unsupervised.   But you can readily insert so much supervised learning.   And logic does emerge very early and often.

Artificial Intelligence / Machine Learning in Technology Review

The AI technique that could imbue machines with the ability to reason
Yann LeCun, Facebook’s chief AI scientist, believes unsupervised learning will bring about the next AI revolution.   by Karen Hao

At six months old, a baby won’t bat an eye if a toy truck drives off a platform and seems to hover in the air. But perform the same experiment a mere two to three months later, and she will instantly recognize that something is wrong. She has already learned the concept of gravity.

“Nobody tells the baby that objects are supposed to fall,” said Yann LeCun, the chief AI scientist at Facebook and a professor at NYU, during a webinar on Thursday organized by the Association for Computing Machinery, an industry body. And because babies don’t have very sophisticated motor control, he hypothesizes, “a lot of what they learn about the world is through observation.”  ... "

(Update)

The Power and Limits Of Deep Learning
ACM TechTalk
2019-07-11

Yann LeCun
New York University
Facebook AI Research
http://yann.lecun.com
Talk:   https://event.on24.com/wcc/r/2014818/04C58DF355DF00190DE4F046CE243077?


Slides:  https://drive.google.com/file/d/1f0sPHv7ozHafASPwIOfuvF_RvP3FDPY0/view

Unsupervised, Disentangled Learning

Intriguing piece in the Google AI Blog.  Had always thought that unsupervised learning should be one essence of creativity  Not examples we know, but new ones we can come up with.  And here 'disentanglement' implies we can capture and model aspects of a system that are not tied together.    How do these ideas fit together?   A technical Google view:

Evaluating the Unsupervised Learning of Disentangled Representations
Posted by Olivier Bachem, Research Scientist, Google AI Zürich 

The ability to understand high-dimensional data, and to distill that knowledge into useful representations in an unsupervised manner, remains a key challenge in deep learning. One approach to solving these challenges is through disentangled representations, models that capture the independent features of a given scene in such a way that if one feature changes, the others remain unaffected. If done successfully, a machine learning system that is designed to navigate the real world, such as a self driving car or a robot, can disentangle the different factors and properties of objects and their surroundings, enabling the generalization of knowledge to previously unobserved situations. While, unsupervised disentanglement methods have already been used for curiosity driven exploration, abstract reasoning, visual concept learning and domain adaptation for reinforcement learning, recent progress in the field makes it difficult to know how well different approaches work and the extent of their limitations.  .... " 

Intro to GAN's by Jason Brownlee

Good intro, have just passed this on to a group ...

A Gentle Introduction to Generative Adversarial Networks (GANs)
by Jason Brownlee on June 17, 2019 in Generative Adversarial Networks  Follow him, good understandable content. 

Generative Adversarial Networks, or GANs for short, are an approach to generative modeling using deep learning methods, such as convolutional neural networks.

Generative modeling is an unsupervised learning task in machine learning that involves automatically discovering and learning the regularities or patterns in input data in such a way that the model can be used to generate or output new examples that plausibly could have been drawn from the original dataset.

GANs are a clever way of training a generative model by framing the problem as a supervised learning problem with two sub-models: the generator model that we train to generate new examples, and the discriminator model that tries to classify examples as either real (from the domain) or fake (generated). The two models are trained together in a zero-sum game, adversarial, until the discriminator model is fooled about half the time, meaning the generator model is generating plausible examples.

GANs are an exciting and rapidly changing field, delivering on the promise of generative models in their ability to generate realistic examples across a range of problem domains, most notably in image-to-image translation tasks such as translating photos of summer to winter or day to night, and in generating photorealistic photos of objects, scenes, and people that even humans cannot tell are fake.

In this post, you will discover a gentle introduction to Generative Adversarial Networks, or GANs. .... " 

Supervised, Unsupervised and Reinforcement Learning

Nicely done piece, relatively non technical,  with comparison of approaches and visualizations.

Machine Learning Explained: Understanding Supervised, Unsupervised, and Reinforcement Learning - Data Science Central by Ronald Van Loon  ... 

Dolphins Speak

Efforts like this could tell us if there are other ways to efficiently store knowledge and process.

Scientists use artificial intelligence to eavesdrop on dolphins
Algorithm enables research team to sift through millions of echolocation clicks made by marine mammals in the Gulf of Mexico  ....  "

by Josh Gabbatiss Science Correspondent