Bill Gates Predicts AI Agents will Rule

 Bill Gates Speaks (excerpts)  In Reuters:

SAN FRANCISCO, May 22 (Reuters) - Bill Gates, Microsoft Corp's (MSFT.O) co-founder, on Monday said the technology race to win is the development of the top artificially intelligent agent, poised to disrupt search-engine, productivity and online shopping sites.

"You’ll never go to a search site again," he said. "You’ll never go to Amazon."

Speaking at AI Forward 2023, an event in San Francisco hosted by Goldman Sachs Group Inc (GS.N) and SV Angel, Gates said it would disappoint him if Microsoft were not in the running, though there was a 50% chance the top player to emerge will be a startup... 

Amazon.com Inc (AMZN.O), Alphabet Inc's (GOOGL.O) Google and Microsoft did not immediately respond to requests for comment.

Gates said Inflection AI, co-founded by entrepreneur Reid Hoffman, impressed him.

Various industry players have worked on building a computer assistant, powered by AI, that consumers can command by speech or text to handle a range of tasks for them.

"Whoever wins the personal agent, that’s a big thing," Gates said.  ... ' 

(and where did Alexa and her tribe go?) 

Dream First, Learn Later: DECKARD is an AI Approach

Interesting, new to me.

Dream First, Learn Later: DECKARD is an AI Approach That Uses LLMs for Training Reinforcement learning (RL) Agents

By Ekrem Çetinkaya -May 4, 2023

Reinforcement learning (RL) is a popular approach to training autonomous agents that can learn to perform complex tasks by interacting with their environment. RL enables them to learn the best action in different conditions and adapt to their environment using a reward system.

A major challenge in RL is how to explore the vast state space of many real-world problems efficiently. This challenge arises due to the fact that in RL, agents learn by interacting with their environment via exploration. Think of an agent that tries to play Minecraft. If you heard about it before, you know how complicated Minecraft crafting tree looks. You have hundreds of craftable objects, and you might need to craft one to craft another, etc. So, it is a really complex environment.

As the environment can have a large number of possible states and actions, it can become difficult for the agent to find the optimal policy through random exploration alone. The agent must balance between exploiting the current best policy and exploring new parts of the state space to find a better policy potentially. Finding efficient exploration methods that can balance exploration and exploitation is an active area of research in RL.

🚀 JOIN the fastest ML Subreddit Community

It’s known that practical decision-making systems need to use prior knowledge about a task efficiently. By having prior information about the task itself, the agent can better adapt its policy and can avoid getting stuck in sub-optimal policies. However, most reinforcement learning methods currently train without any previous training or external knowledge. 

But why is that the case? In recent years, there has been growing interest in using large language models (LLMs) to aid RL agents in exploration by providing external knowledge. This approach has shown promise, but there are still many challenges to overcome, such as grounding the LLM knowledge in the environment and dealing with the accuracy of LLM outputs.

So, should we give up on using LLMs to aid RL agents? If not, how can we fix those problems and then use them again to guide RL agents? The answer has a name, and it’s DECKARD.

Overview of DECKARD. Source: https://arxiv.org/abs/2301.12050

DECKARD is trained for Minecraft, as crafting a specific item in Minecraft can be a challenging task if one lacks expert knowledge of the game. This has been demonstrated by studies that have shown that achieving a goal in Minecraft can be made easier through the use of dense rewards or expert demonstrations. As a result, item crafting in Minecraft has become a persistent challenge in the field of AI. ... '

Teaching BabyAGI

Inevitable and scary?  Agents teaching your models bottom up.

Auto-GPT and BabyAGI: How ‘autonomous agents’ are bringing generative AI to the masses

Autonomous agents may mark an important step toward a world where AI-driven systems are smart enough to work on their own, without need of human involvement.

Auto-GPT and BabyAGI: How ‘autonomous agents’ are bringing generative AI to the masses

By Mark Sullivan  in Fastcompany

Over the past week, developers around the world have begun building “autonomous agents” that work with large language models (LLMs) such as OpenAI’s GPT-4 to solve complex problems. While still very new, such agents could represent a major milestone in the productive application of LLMs.

Normally, we interact with GPT-4 by typing carefully worded prompts into ChatGPT’s text window until the model generates the output we want. But most of us lack the skill and patience to sit and write prompt after prompt, guiding the LLM toward answering a complex question, such as “What is the optimal business plan for capturing 20% of the fingernail-polish market?” Quite naturally, developers have been thinking of ways to automate much of that process. That’s where autonomous agents come in.

In general terms, autonomous agents can generate a systematic sequence of tasks that the LLM works on until it’s satisfied a preordained “goal.” Autonomous agents can already perform tasks as varied as conducting web research, writing code, and creating to-do lists....  ... '

AI as an Agent and More

 Thinking agents and more, lot of definitional info in links below, following up definitions:

If you thought the pace of AI development had sped up since the release of ChatGPT last November, well, buckle up. Thanks to the rise of autonomous AI agents like Auto-GPT, BabyAGI and AgentGPT over the past few weeks, the race to get ahead in AI is just getting faster. And, many experts say, more concerning. in VentureBeat

It all started in late March, when developer Toran Bruce Richards, under the name @significantgravitas, launched Auto-GPT, an “experimental open-source application” connected to OpenAI’s GPT-4 by API. Running on Python, Auto-GPT had internet access, long/short-term memory and, by stringing together GPT calls in loops, could act autonomously without requiring a human agent to prompt every action. With just a goal in mind — such as preparing a podcast — it could research information online, for example, and then without being prompted take further action towards the goal, like preparing a list of topics and titles.

Then, on March 29, @yoheinakajima launched BabyAGI, a “task-driven autonomous agent” that leverages GPT-4, Pinecone‘s vector search, and LangChainAI‘s framework to “autonomously create and perform tasks based on an objective” — say, planning and automatically execute a campaign to grow your Twitter following or creating and running a small content marketing business.

MBRL Tuning for Partially Understood Environments

Below is very technical,  but I do like some of the background statements such as 'solving tasks in a partially understood environment ...'.    And the idea of optimizing agents to resolve elements of understanding.  (Which exemplifies the situations we are often in).    So I am not saying I understand this yet, but working through it now for broader application.  As part of my broader study of practical reinforcement learning.

The Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Nathan Lambert, Baohe Zhang, Raghu Rajan, André Biedenkapp    Apr 19, 2021  From BAIR  Berkeley

Model-based reinforcement learning (MBRL) is a variant of the iterative learning framework, reinforcement learning, that includes a structured component of the system that is solely optimized to model the environment dynamics. Learning a model is broadly motivated from biology, optimal control, and more – it is grounded in natural human intuition of planning before acting. This intuitive grounding, however, results in a more complicated learning process. In this post, we discuss how model-based reinforcement learning is more susceptible to parameter tuning and how AutoML can help in finding very well performing parameter settings and schedules. Below, left is the expected behavior of an agent maximizing velocity on a “Half Cheetah” robotic task, and to the right is what our paper with hyperparameter tuning finds.

MBRL

Model-based reinforcement learning (MBRL) is an iterative framework for solving tasks in a partially understood environment. There is an agent that repeatedly tries to solve a problem, accumulating state and action data. With that data, the agent creates a structured learning tool – a dynamics model – to reason about the world. With the dynamics model, the agent decides how to act by predicting into the future. With those actions, the agent collects more data, improves said model, and hopefully improves future actions.  ... " 

Autonomous Manufacturing Marketplace

Intriguing play here, will look into more details... pass them along and I will post an overview.

Fetch.ai partners with FESTO on decentralised manufacturing marketplace

AI blockchain startup Fetch.ai is partnering with industry veteran FESTO to launch a decentralised marketplace for manufacturing.

Fetch.ai is based in Cambridge, UK and has built an impressive team of talent with experience from DeepMind, Siemens, Sony, and a number of esteemed academic institutions. The company is working on decentralised autonomous “agents” which perform real-world tasks.

FESTO was founded in 1925 and currently produces and sells pneumatic and electrical control and drive technology. The company has continued to thrive over the years through consistent reinvention and likes to show off around once a year by developing bionic robots like its ‘flying fox’ which uses machine learning to optimise its flight behaviour with every manoeuvre:

The partnership with Fetch.ai is FESTO’s first foray into blockchain and shows how the company aims to remain at the cutting-edge of technological developments.

Fetch.ai will develop a decentralised manufacturing marketplace for FESTO to help transform the company’s existing control systems and make them more efficient.

Maria Minaricova, Director of Business Development at Fetch.ai, said:

“We are delighted to be able to announce a collaboration with such an advanced technology company like FESTO.

With FESTO’s contribution, we will be able to demonstrate in real life the benefits of autonomous AI agents in manufacturing and supply chain.

We look forward to working further with FESTO to bring about the wide adoption of these ground-breaking manufacturing innovations.”   .... '

Modeling Multi Agent Interacting Systems

 Fascinating piece.  Aren't we all multi agent interacting systems?  Can we effectively model that to understand it in context?   Berkeley Bair takes a took.  Its hard.  Considerable technical challenge.  Starting with urban traffic models.   Early on we used agent modeling to understand how new markets changed.   

EvolveGraph: Dynamic Neural Relational Reasoning for Interacting Systems

Jiachen Li    Nov 18, 2020

Multi-agent interacting systems are prevalent in the world, from purely physical systems to complicated social dynamic systems. The interactions between entities / components can give rise to very complex behavior patterns at the level of both individuals and the multi-agent system as a whole. Since usually only the trajectories of individual entities are observed without any knowledge of the underlying interaction patterns, and there are usually multiple possible modalities for each agent with uncertainty, it is challenging to model their dynamics and forecast their future behaviors.

Figure 1. Typical multi-agent interacting systems.

In many real-world applications (e.g. autonomous vehicles, mobile robots), an effective understanding of the situation and accurate trajectory prediction of interactive agents play a significant role in downstream tasks, such as decision making and planning. We introduce a generic trajectory forecasting framework (named EvolveGraph) with explicit relational structure recognition and prediction via latent interaction graphs among multiple heterogeneous, interactive agents. Considering the uncertainty of future behaviors, the model is designed to provide multi-modal prediction hypotheses. Since the underlying interactions may evolve even with abrupt changes over time, and different modalities of evolution may lead to different outcomes, we address the necessity of dynamic relational reasoning and adaptively evolving the interaction graphs.

Challenges of Multi-Agent Behavior Prediction

Figure 2. An illustration of a typical urban intersection scenario.

We use an urban intersection scenario with multiple interacting traffic participants as an illustrative example to elaborate on the major challenges of the multi-agent behavior prediction task.

First, there may be heterogeneous agents that have distinct behavior patterns, thus using a homogeneous dynamics / behavior model may not be sufficient. For example, there are different constraints and traffic rules for vehicles and pedestrians. More specifically, vehicle trajectories are strictly constrained by road geometry and their own kinematic models; while pedestrian behaviors are much more flexible.  ... ' 

Free Book on Computational Agents

The use of an agent to represent objects, IOTs or people is very useful.  Closer, but not the same as classic optimization models.  The book and comments here are good introductions.   Technical. 

Free book - Artificial Intelligence: Foundations of Computational Agents   Posted by ajit jaokar   in DSC       

Comments:  There are many excellent free books on Python – but Artificial Intelligence: Foundations of Computational Agents is about  a subject not commonly covered

I found the book useful as a introduction to Reinforcement Learning

As the title suggests, the book is about computational agents

An agent observes the world and carries out actions in the environment. The agent maintains an internal state that it updates. Also, the environment takes in actions of the agents, and in turn updates its internal state and returns the percepts. In this implementation, the state of the agent and the state of the environment are represented using standard Python variables, which are updated as the state changes.  .... " 

This structure can be used to model many interesting problems and is the focus of the book. Ultimately, it leads to Reinforcement Learning. ... " 

Questions on the Value of Speaking Robotish

What is the meaning behind declaring something a language?    We have many computing and many human languages.  How much does their form influence the nature of communication between people and people, people and devices?  What kinds of efficiencies occur?   Risks of error?  Learning of context?  Improvement of innovations?   Working with many 'agents'?   So many questions.

Do you speak robot-ish? Interpreters may soon be in the house
by Nancy Owano , Tech Xplore

(Tech Xplore)—OpenAI is a non-profit artificial intelligence research company. In describing their work, they state that they are "working towards the next set of breakthroughs." That is no exaggeration.


What if you are told machine learning is so yesterday in language processing. "By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing," wrote Igor Mordatch, Pieter Abbeel in their paper "Emergence of Grounded Compositional Language in Multi-Agent Populations," now on arXiv.

However, they stated, just capturing statistical patterns is not enough for agents to intelligently interact with humans. With that in mind, they investigated if and how, "grounded compositional language can emerge as a means to achieve goals in multi-agent populations."

The authors noted that "Development of agents that are capable of communication and flexible language use is one of the long-standing challenges facing the field of artificial intelligence." Looks as if they are taking up that challenge 

In a recent blog post, Igor Mordatch, Pieter Abbeel, Ryan Lowe, Jon Gauthier and Jack Clark, wrote about their new research where agents develop their own language..... " 

More information: * Learning to communicate: openai.com/blog/learning-to-communicate/
* Emergence of Grounded Compositional Language in Multi-Agent Populations, arXiv:1703.04908 [cs.AI] arxiv.org/abs/1703.04908

Abstract
By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing, including in machine translation, question answering, and sentiment analysis. However, for agents to intelligently interact with humans, simply capturing the statistical patterns is insufficient. In this paper we investigate if, and how, grounded compositional language can emerge as a means to achieve goals in multi-agent populations. Towards this end, we propose a multi-agent learning environment and learning methods that bring about emergence of a basic compositional language. This language is represented as streams of abstract discrete symbols uttered by agents over time, but nonetheless has a coherent structure that possesses a defined vocabulary and syntax. We also observe emergence of non-verbal communication such as pointing and guiding when language communication is unavailable. .... "

Exploring Exploration

A very old proposition has said that if only we could make a system as inquisitive as a child, and had it able to learn and extend its knowledge, we could get to intelligent machines.   An ultimate form of Machine learning.   We are not close to that, but the attempt continues.  I like the term: Exploring Exploration.  Bair takes a look and explains the idea and what is being done.  So how do children explore to learn?  And can we use that as a model for machines?  Extensive and not too technical report.

Exploring Exploration: Comparing Children with RL Agents in Unified Environments
Eliza Kosoy, Jasmine Collins and David Chan    Jul 24, 2020

Despite recent advances in artificial intelligence (AI) research, human children are still by far the best learners we know of, learning impressive skills like language and high-level reasoning from very little data. Children’s learning is supported by highly efficient, hypothesis-driven exploration: in fact, they explore so well that many machine learning researchers have been inspired to put videos like the one below in their talks to motivate research into exploration methods. However, because applying results from studies in developmental psychology can be difficult, this video is often the extent to which such research actually connects with human cognition.  ... " 

AI Agents Learning to Communicate

Machines learning and adapting using language.   As suggested,the environment/context is also important.

AI agents can learn to communicate effectively  by University of Gothenburg

A multi-disciplinary team of researchers from Chalmers and University of Gothenburg has developed a framework to study how language evolves as an effective tool for describing mental concepts. In a new paper, they show that artificial agents can learn how to communicate in an artificial language similar to human language. The results have been published in the scientific journal PLOS ONE.

This research lies on the border between cognitive science and machine learning. There has been an influential proposal from cognitive scientists that all human languages can be viewed as having evolved as a means to communicate concepts in a near-optimal way in the sense of classical information theory. The Gothenburg researchers' method for training the artificial agents is based on reinforcement learning, which is an area of machine learning where agents gradually learn by interacting with an environment and getting feedback. In this case, the agents start without any linguistic knowledge and learn to communicate by getting feedback on how well they succeed in communicating a mental concept.

Reconstructing colors

"In our paper we have studied how agents learn to name mental concepts and communicate by playing a several rounds of a referential game consisting of a sender and a listener. We have especially focused on the color-domain which is well studied in cognitive science. The game works as follows; the sender sees a color and describes it by uttering a word from a glossary to the listener which then tries to reconstruct the color.

Both agents receive a shared reward based on how precise the listener's reconstruction was. The words in the glossary have no meaning at the outset; it is up to the agents to agree on the meaning of the words during multiple rounds of the game. We see that the resulting artificial languages are near-optimal in an information-theoretic sense and with similar properties as found in human languages," says Mikael Kågebäck, researcher at Sleepcycle, and whose Ph.D. dissertation at Chalmers contained some of the results presented in the paper. .... " 

Google AI Making Decisions on the Fly

Intriguing direction making decisions like we often do,  in a series to address changes in context

Google’s AI enables robots to make decisions on the fly
  By Kyle Wiggers in Venturebeat

In a paper published this week  (technical) on the preprint server Arxiv.org, a team of Google Brain; Google X; and University of Calfornia, Berkeley researchers describe an extension to existing AI methods that enable an agent — for instance, a robot — to decide which action to take while performing a previous action. The idea is that modeling an agent’s behavior after that of a person or an animal will lead to more robust, less failure-prone systems in the future.

The researchers point out that while AI algorithms have achieved success in video games, robotic grasping, and manipulation tasks, most use a blocking observe-think-act paradigm — an agent assumes that its environment will remain static while it “thinks” so that its actions will be executed on the same states from which they were computed. This holds true in simulation but not in the real world, where the environment state evolves as the agent processes observations and plans its next actions. ...."

Virtual Agents

A broader take than chatbots, more addressed to solving specific customer service issues.    We used the term in loose combination with agent-models to deliver goal-specific assistants.     Was always important to have very specific goals to drive their actions.  Usually using simulation style prediction.   But we often used simpler methods when appropriate.    The definitions online vary.   A number of vendors now talking this are covered in this article.

No, IBM is not the only relevant player in virtual agents
  by Matt Marshell

Last month, IBM General Manager of Data and Watson AI, Rob Thomas, told VentureBeat that IBM was the only major enterprise provider in the red-hot area of virtual agents.

Virtual agents are software that can chat with customers through text, voice, or web chat. “There really are no big players, except for us,” Thomas said at the time. He called the rest of the virtual agent providers “fireflies,” because they are small and there are so many of them..... "

Dimensional Analysis of Failure and Fixing

Brought to my attention.  In Facebook Engineering Blog.   We also attempted to use neural methods to determine when agents, machine or otherwise, were likely to fail,  and afterwards what were the necessary means of repairing the problem.      Scale may be the biggest problem.  I realize the below is more about machine-agents, which we also saw, but could it be broadened to human agent behavior?

Fast dimensional analysis for root cause analysis at scale
Nikolay Pavlovich Laptev,  Fred Lin  Keyur Muzumdar,  Mihai-Valentin Curelea

What the research is: 
A fast dimensional analysis (FDA) framework that automates root cause analysis on structured logs with improved scalability. When a failure event happens in a large-scale distributed production environment, performing root cause analysis can be challenging. Various hardware, software, and tooling logs are often maintained separately, making it difficult to detect issues across multiple logs. Additionally, at scale, there could easily be millions of entities, each with hundreds of features, making it difficult to debug issues.

Our proposed FDA framework combines structured logs from a number of sources and provides a meaningful combination of features. That information arms engineers with actionable insights and helps them determine where to begin their investigation. And improved Apriori/FP-Growth algorithms sustain analysis at Facebook scale.

In the figure above, our system finds highly correlated features that explain the exception spike (red line).

How it works:
The FDA framework first fetches structured logs from various sources. Log data is deduplicated at query time (deduping significantly improves algorithm performance). Each duplicated row will have a samples column that counts original row frequency. The resulting data is then one-hot encoded (a Boolean 0/1 value, depending on whether a given feature is present in a given row) to transform it into a schema that fits the frequent pattern mining formulation. Frequent pattern mining is applied to identify frequent item-sets.   .... " 

More Aims at Explainable AI

More interesting moves in providing explanation to AI.  Here from Ga Tech.   As in the previous recent  post, this asks the question:   What does explainable mean?   And to whom in what context?  Simplicity is good, if everyone agrees to its truth and value.

A Breakthrough in Explainable AI   By Joe Dysart in CACM
A new artificial intelligence (AI) agent offers easy-to-understand English explanations of the AI's analysis and decisions.

Such tools of explanation are considered critical by those working in AI who fear users may be reluctant to embrace AI applications that make recommendations whose rationales are shrouded in mystery.

"As AI pervades all aspects of our lives, there is a distinct need for human-centered AI design that makes black-boxed AI systems explainable to everyday users," says Upol Ehsan, a doctoral student in the School of Interactive Computing at the Georgia Institute of Technology (Georgia Tech) and lead author of the study.  "Our work takes a formative step toward understanding the role of language-based explanations and how humans perceive them."

Adds Devi Parikh, an assistant professor in Georgia Tech's School of Interactive Computing, "Just like human teams are more effective when team members are transparent and can understand the rationale behind each others decisions or opinions, human-AI teams will be more effective if AI systems explain themselves and are somewhat interpretable to a human."

Eshan and his research team set out to solve the explainable AI problem by developing an AI agent that could offer easy-to-understand explanations to humans in certain settings.

For their research, the team—including researchers from Georgia Tech, Cornell University, and the University of Kentucky—decided to create an AI agent that would analyze and explain moves in the video game "Frogger." The game is an ideal choice for developing an AI agent, given its simplicity: the entire goal of the game is to move an animated frog across a screen, enabling it to dodge oncoming vehicles and other animated hazards.

The researchers trained their AI agent by first asking gamers to play Frogger as they explained the rationale behind each action they took, move by move."  ... ' 

Humans Assisted Cognitively

On assistance and related topics,  humans doing more with less.    Cognitive limits.

Humans: Doing More With Less
A Talk By Tom Griffiths  in The Edge

Imagine a superintelligent system with far more computational resources than us mere humans that’s trying to make inferences about what the humans who are surrounding it—which it thinks of as cute little pets—are trying to achieve so that it is then able to act in a way that is consistent with what those human beings might want. That system needs to be able to simulate what an agent with greater constraints on its cognitive resources should be doing, and it should be able to make inferences, like the fact that we’re not able to calculate the zeros of the Riemann zeta function or discover a cure for cancer. It doesn’t mean we’re not interested in those things; it’s just a consequence of the cognitive limitations that we have.

As a parent of two small children, a problem that I face all the time is trying to figure out what my kids want—kids who are operating in an entirely different mode of computation, and having to build a kind of internal model of how a toddler’s mind works such that it’s possible to unravel that and work out that there’s a particular motivation for the very strange pattern of actions that they’re taking.

Both from the perspective of understanding human cognition and from the perspective of being able to build AI systems that can understand human cognition, it’s desirable for us to have a better model of how rational agents should act if those rational agents have limited cognitive resources. That’s something I’ve been working on for the last few years. We have an approach to thinking about this that we call resource rationality. And this is closely related to similar ideas that are being proposed in the artificial intelligence literature. One of these ideas is the notion of bounded optimality, proposed by Stuart Russell.   .... "

TOM GRIFFITHS is the Henry R. Luce Professor of Information, Technology, Consciousness, and Culture at Princeton University. He is co-author (with Brian Christian) of Algorithms to Live By. Tom Griffiths's Edge Bio Page