AI is Teaching us New, Surprising Things About the Human Mind

 ACM TECHNEWS

AI is Teaching Us New, Surprising Things About the Human Mind

By The Wall Street Journal, April 5, 2023

A researcher demonstrates the brain-scanning magnetoencephalography device at New York University.

Artificial intelligence is helping scientists decode how neurons in our brains communicate, and to explore the nature of cognition.

Scientists are gaining new insights into the human mind though artificial intelligence (AI), including the mechanism of communication between neurons, and the roots of cognition.

The University of California, Berkeley's Celeste Kidd and colleagues used a clustering model to find people's opinions tend to diverge about even the most fundamental properties of things. Researchers led by Princeton University's Tatiana Engel used artificial neurons to interpret hundreds of neurons' electrical impulses in animals' brains simultaneously, then trained them to perform identical tasks.

These networks self-organize into reasonable approximations of those in animals, indicating dynamic electrical activity forms the substance of thought, according to Engel.

From The Wall Street Journal

View Full Article -  May Require Paid Subscription

Training Neural Nets on Small Devices

 Good direction.

We Can Train Big Neural Networks on Small Devices

IEEE Spectrum

Matthew Hutson, September 20, 2022

A new training method expands small devices' capabilities to train large neural networks, while potentially helping to protect privacy. The University of California, Berkeley's Shishir Patil and colleagues integrated offloading and rematerialization techniques using suboptimal heuristics to reduce memory requirements for training via the private optimal energy training (POET) system. Users feed POET a device's technical details and data on the architecture of a neural network they want to train, specifying memory and time budgets; the system generates a training process that minimizes energy usage. Defining the problem as a mixed integer linear programming challenge was critical to POET's effectiveness. Testing showed the system could slash memory usage by about 80% without significantly increasing energy consumption.  ...

Robots Adapt to Challenging Terrain in Real Time

Terrain-able robotics have been very impressive in recent years.   Here yet more advances by some relatively unexpected parties.  Ultimately will it become rare to have humans repetitively port around themselves and things they can carry?  

Stumble-Proof Robot Adapts to Challenging Terrain in Real Time. By TechCrunch, July 16, 2021

A new robotic locomotion model capable of real-time terrain adaptation has been developed by a multi-institutional research team.

Engineers at Facebook AI, the University of California, Berkeley (UC Berkeley), and Carnegie Mellon University based Rapid Motor Adaptation (RMA) on the ability of humans and animals to quickly and unconsciously adjust their locomotion to different conditions.

The team trained the system in a virtual model of the real world, where the robot's brain learned to maximize forward motion with the least amount of energy, and to avoid falls by responding to incoming data from physical sensors.

UC Berkeley's Jitendra Malik said the robot employs absolutely no visual input, instead closely monitoring itself.

The RMA system uses a constantly running main gait-control algorithm and a parallel adaptive algorithm that watches internal readings and provides the main model adjustment data in response to terrain changes.

From TechCrunch   Full article.

Challenge for Learning from Human Feedback using Minecraft

Berkeley Bair challenge competition here using a common gaming environment.   Been a long time since I looked at Minecraft.  Short extract of the idea below, more complete look at the link.   Seems a novel look at a broader look at contextual learning. 

 BASALT: A Benchmark for  Learning from Human Feedback   by Rohin Shah    Jul 8, 2021

TL;DR: We are launching a NeurIPS competition and benchmark called BASALT: a set of Minecraft environments and a human evaluation protocol that we hope will stimulate research and investigation into solving tasks with no pre-specified reward function, where the goal of an agent must be communicated through demonstrations, preferences, or some other form of human feedback. Sign up to participate in the competition!

Motivation

Deep reinforcement learning takes a reward function as input and learns to maximize the expected total reward. An obvious question is: where did this reward come from? How do we know it captures what we want? Indeed, it often doesn’t capture what we want, with many recent examples showing that the provided specification often leads the agent to behave in an unintended way.

Our existing algorithms have a problem: they implicitly assume access to a perfect specification, as though one has been handed down by God. Of course, in reality, tasks don’t come pre-packaged with rewards; those rewards come from imperfect human reward designers.

For example, consider the task of summarizing articles. Should the agent focus more on the key claims, or on the supporting evidence? Should it always use a dry, analytic tone, or should it copy the tone of the source material? If the article contains toxic content, should the agent summarize it faithfully, mention that toxic content exists but not summarize it, or ignore it completely? How should the agent deal with claims that it knows or suspects to be false? A human designer likely won’t be able to capture all of these considerations in a reward function on their first try, and, even if they did manage to have a complete set of considerations in mind, it might be quite difficult to translate these conceptual preferences into a reward function the environment can directly calculate.  ...................

Conclusion

We hope that BASALT will be used by anyone who aims to learn from human feedback, whether they are working on imitation learning, learning from comparisons, or some other method. It mitigates many of the issues with the standard benchmarks used in the field. The current baseline has lots of obvious flaws, which we hope the research community will soon fix.

Note that, so far, we have worked on the competition version of BASALT. We aim to release the benchmark version shortly. You can get started now, by simply installing MineRL from pip and loading up the BASALT environments. The code to run your own human evaluations will be added in the benchmark release.

If you would like to use BASALT in the very near future and would like beta access to the evaluation code, please email the lead organizer, Rohin Shah, at rohinmshah@berkeley.edu.

This post is based on the paper “The MineRL BASALT Competition on Learning from Human Feedback”, accepted at the NeurIPS 2021 Competition Track. Sign up to participate in the competition!   

Simulate, Constrain, Repeat, Learn

Berkeley Bair posts an interesting look at Reinforcement Learning.   Made me think,  But once you get beyond the paras below this gets quite complicated and technical.   Anyone who has written significant simulation packages can be amazed at what they can accomplish. And embedded with reinforcement learning to provide direction, consider the possibilities.  And suggestions that anything can be a 'simulation'  gives us pause.   But how accurate can it be in real contexts?  Worth thinking it. 

Learning What To Do by Simulating the Past    By David Lindner, Rohin Shah    May 3, 2021,    Berkeley Bair

Reinforcement learning (RL) has been used successfully for solving tasks which have a well defined reward function – think AlphaZero for Go, OpenAI Five for Dota, or AlphaStar for StarCraft. However, in many practical situations you don’t have a well defined reward function. Even a task as seemingly straightforward as cleaning a room has many subtle cases: should a business card with a piece of gum be thrown away as trash, or might it have sentimental value? Should the clothes on the floor be washed, or returned to the closet? Where are notebooks supposed to be stored? Even when these aspects of a task have been clarified, translating it into a reward is non-trivial: if you provide rewards every time you sweep the trash, then the agent might dump the trash back out so that it can sweep it up again.1

Alternatively, we can try to learn a reward function from human feedback about the behavior of the agent. For example, Deep RL from Human Preferences learns a reward function from pairwise comparisons of video clips of the agent’s behavior. Unfortunately, however, this approach can be very costly: training a MuJoCo Cheetah to run forward requires a human to provide 750 comparisons.

Instead, we propose an algorithm that can learn a policy without any human supervision or reward function, by using information implicitly available in the state of the world. For example, we learn a policy that balances this Cheetah on its front leg from a single state in which it is balancing.  ...."

Reinforcement Learning and Entropy

 Have recently been looking at Reinforcement Learning methods.   And this as a form of simulation-optimization for 'Twin' style models that need training.  Berkeley BAIR  makes some points about entropy (disorder) in a recent  article.   (Technical)  Consideringe the application. See the full article, linked to below,  for sufficient detail.

Maximum Entropy RL (Provably) Solves Some Robust RL Problems  By Ben Eysenbach    Mar 10, 2021    Berkeley BAIR  AI

Nearly all real-world applications of reinforcement learning involve some degree of shift between the training environment and the testing environment. However, prior work has observed that even small shifts in the environment cause most RL algorithms to perform markedly worse. As we aim to scale reinforcement learning algorithms and apply them in the real world, it is increasingly important to learn policies that are robust to changes in the environment.

Robust reinforcement learning maximizes reward on an adversarially-chosen environment.

Broadly, prior approaches to handling distribution shift in RL aim to maximize performance in either the average case or the worst case. The first set of approaches, such as domain randomization, train a policy on a distribution of environments, and optimize the average performance of the policy on these environments. While these methods have been successfully applied to a number of areas (e.g., self-driving cars, robot locomotion and manipulation), their success rests critically on the design of the distribution of environments. Moreover, policies that do well on average are not guaranteed to get high reward on every environment. The policy that gets the highest reward on average might get very low reward on a small fraction of environments. The second set of approaches, typically referred to as robust RL, focus on the worst-case scenarios. The aim is to find a policy that gets high reward on every environment within some set. Robust RL can equivalently be viewed as a two-player game between the policy and an environment adversary. The policy tries to get high reward, while the environment adversary tries to tweak the dynamics and reward function of the environment so that the policy gets lower reward. One important property of the robust approach is that, unlike domain randomization, it is invariant to the ratio of easy and hard tasks. Whereas robust RL always evaluates a policy on the most challenging tasks, domain randomization will predict that the policy is better if it is evaluated on a distribution of environments with more easy tasks.

Prior work has suggested a number of algorithms for solving robust RL problems. Generally, these algorithms all follow the same recipe: take an existing RL algorithm and add some additional machinery on top to make it robust. For example, robust value iteration uses Q-learning as the base RL algorithm, and modifies the Bellman update by solving a convex optimization problem in the inner loop of each Bellman backup. Similarly, Pinto ‘17 uses TRPO as the base RL algorithm and periodically updates the environment based on the behavior of the current policy. These prior approaches are often difficult to implement and, even once implemented correctly, they requiring tuning of many additional hyperparameters. Might there be a simpler approach, an approach that does not require additional hyperparameters and additional lines of code to debug? ... "