Start with the Basics for Machine Learning

Pretraining and related topics. 

Machines Learn Better if We Teach Them the Basics

A wave of research improves reinforcement learning algorithms by pre-training them as if they were human.      By Max G Levy     In QuantaMagazine

Imagine that your neighbor calls to ask a favor: Could you please feed their pet rabbit some carrot slices? Easy enough, you’d think. You can imagine their kitchen, even if you’ve never been there — carrots in a fridge, a drawer holding various knives. It’s abstract knowledge: You don’t know what your neighbor’s carrots and knives look like exactly, but you won’t take a spoon to a cucumber.

Artificial intelligence programs can’t compete. What seems to you like an easy task is a huge undertaking for current algorithms.

An AI-trained robot can find a specified knife and carrot hiding in a familiar kitchen, but in a different kitchen it will lack the abstract skills to succeed. “They don’t generalize to new environments,” said Victor Zhong, a graduate student in computer science at the University of Washington. The machine fails because there’s simply too much to learn, and too vast a space to explore.

The problem is that these robots — and AI agents in general — don’t have a foundation of concepts to build on. They don’t know what a knife or a carrot really is, much less how to open a drawer, choose one and cut slices. This limitation is due in part to the fact that many advanced AI systems get trained with a method called reinforcement learning that’s essentially self-education through trial and error. AI agents trained with reinforcement learning can execute the job they were trained to do very well, in the environment they were trained to do it in. But change the job or the environment, and these systems will often fail.

ARTIFICIAL INTELLIGENCE

Computers Evolve a New Path Toward Human Intelligence   NOVEMBER 6, 2019

To get around this limitation, computer scientists have begun to teach machines important concepts before setting them loose. It’s like reading a manual before using new software: You could try to explore without it, but you’ll learn far faster with it. “Humans learn through a combination of both doing and reading,” said Karthik Narasimhan, a computer scientist at Princeton University. “We want machines to do the same.”

New work from Zhong and others shows that priming a learning model in this way can supercharge learning in simulated environments, both online and in the real world with robots. And it doesn’t just make algorithms learn faster — it guides them toward skills they’d otherwise never learn. Researchers want these agents to become generalists, capable of learning anything from chess to shopping to cleaning. And as demonstrations become more practical, scientists think this approach might even change how humans can interact with robots.

“It’s been a pretty big breakthrough,” said Brian Ichter, a research scientist in robotics at Google. “It’s pretty unimaginable how far it’s come in a year and a half.”

Sparse Rewards

At first glance, machine learning has already been remarkably successful. Most models typically use reinforcement learning, where algorithms learn by getting rewards. They begin totally ignorant, but trial and error eventually becomes trial and triumph. Reinforcement learning agents can easily master simple games.

Consider the video game Snake, where players control a snake that grows longer as it eats digital apples. You want your snake to eat the most apples, stay within the boundaries and avoid running into its increasingly bulky body. Such clear right and wrong outcomes give a well-rewarded machine agent positive feedback, so enough attempts can take it from “noob” to High Score.

But suppose the rules change. Perhaps the same agent must play on a larger grid and in three dimensions. While a human player could adapt quickly, the machine can’t, because of two critical weaknesses. First, the larger space means it takes longer for the snake to stumble upon apples, and learning slows exponentially when rewards become sparse. Second, the new dimension provides a totally new experience, and reinforcement learning struggles to generalize to new challenges .... '