Adobe and MIT Identify

 Robotic applications emerge...

Researchers use AI to identify similar materials in images

This machine-learning method could assist with robotic scene understanding, image editing, or online recommendation systems.

Adam Zewe | MIT News Office    Publication Date:May 23, 2023

A robot manipulating objects while, say, working in a kitchen, will benefit from understanding which items are composed of the same materials. With this knowledge, the robot would know to exert a similar amount of force whether it picks up a small pat of butter from a shadowy corner of the counter or an entire stick from inside the brightly lit fridge.

Identifying objects in a scene that are composed of the same material, known as material selection, is an especially challenging problem for machines because a material’s appearance can vary drastically based on the shape of the object or lighting conditions.

Scientists at MIT and Adobe Research have taken a step toward solving this challenge. They developed a technique that can identify all pixels in an image representing a given material, which is shown in a pixel selected by the user.

The method is accurate even when objects have varying shapes and sizes, and the machine-learning model they developed isn’t tricked by shadows or lighting conditions that can make the same material appear different.

Although they trained their model using only “synthetic” data, which are created by a computer that modifies 3D scenes to produce many varying images, the system works effectively on real indoor and outdoor scenes it has never seen before. The approach can also be used for videos; once the user identifies a pixel in the first frame, the model can identify objects made from the same material throughout the rest of the video.

Four images shown horizontally of person walking with luggage. First, image still shows red dot on yellow pants material. Second and third images are animations, but the third image shows pink pants. Fourth, monochrome version animation is shown, with luggage and shoes barely visible in black background.

The researchers' technique can also be used to select similar materials in a video. The user identifies a pixel in the first frame (red dot in the far-left image on the yellow fabric) and the system automatically identifies objects made from the same material throughout the rest of the video.

In addition to applications in scene understanding for robotics, this method could be used for image editing or incorporated into computational systems that deduce the parameters of materials in images. It could also be utilized for material-based web recommendation systems. (Perhaps a shopper is searching for clothing made from a particular type of fabric, for example.)

“Knowing what material you are interacting with is often quite important. Although two objects may look similar, they can have different material properties. Our method can facilitate the selection of all the other pixels in an image that are made from the same material,” says Prafull Sharma, an electrical engineering and computer science graduate student and lead author of a paper on this technique.

Sharma’s co-authors include Julien Philip and Michael Gharbi, research scientists at Adobe Research; and senior authors William T. Freeman, the Thomas and Gerd Perkins Professor of Electrical Engineering and Computer Science and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Frédo Durand, a professor of electrical engineering and computer science and a member of CSAIL; and Valentin Deschaintre, a research scientist at Adobe Research. The research will be presented at the SIGGRAPH 2023 conference.

A new approach

Existing methods for material selection struggle to accurately identify all pixels representing the same material. For instance, some methods focus on entire objects, but one object can be composed of multiple materials, like a chair with wooden arms and a leather seat. Other methods may utilize a predetermined set of materials, but these often have broad labels like “wood,” despite the fact that there are thousands of varieties of wood.

Instead, Sharma and his collaborators developed a machine-learning approach that dynamically evaluates all pixels in an image to determine the material similarities between a pixel the user selects and all other regions of the image. If an image contains a table and two chairs, and the chair legs and tabletop are made of the same type of wood, their model could accurately identify those similar regions.

Before the researchers could develop an AI method to learn how to select similar materials, they had to overcome a few hurdles. First, no existing dataset contained materials that were labeled finely enough to train their machine-learning model. The researchers rendered their own synthetic dataset of indoor scenes, which included 50,000 images and more than 16,000 materials randomly applied to each object.

“We wanted a dataset where each individual type of material is marked independently,” Sharma says.

Synthetic dataset in hand, they trained a machine-learning model for the task of identifying similar materials in real images — but it failed. The researchers realized distribution shift was to blame. This occurs when a model is trained on synthetic data, but it fails when tested on real-world data that can be very different from the training set.

To solve this problem, they built their model on top of a pretrained computer vision model, which has seen millions of real images. They utilized the prior knowledge of that model by leveraging the visual features it had already learned.   .. ' 

Robotic Echolocaton

More examples of biomimicry here and hint at possible mapping application, 

Flying Robot Echolocates Like a Bat to Avoid Hitting Walls,  By New Scientist, February 6, 2023

This drone uses a buzzer and microphone set-up to navigate by echolocation.

A robot using the echolocation technique can make a map of its environment and localize itself at the same time.

Frederike Dümbgen and colleagues at Canada's University of Toronto and the Swiss Federal Institute of Technology, Lausanne have equipped a flying robot to use bat-like echolocation to map its surroundings using a simple microphone and speaker.

The robot’s speaker emits sound bursts infused with a range of frequencies, which bounce off walls and are recorded by the microphone when they come back., An algorithm then uses interference patterns caused by the sound waves to model the environment's surfaces.

The researchers tested the system on a drone rigged with a buzzer and four microphones, and on a wheeled robot with a built-in speaker and microphone. The drone could map walls with up to 2-centimeter (0.7-inch) accuracy from 0.5 meters away when stationary, and with 8-centimeter (3.1-inch) accuracy when airborne.

From New Scientist

View Full Article  

Seoul Patrols Streets with Robots

Interesting approach but unlikely to be tolerated today in the West. 

Self-Driving Robot Patrols Seoul Streets

EuroNews, Roselyne Min, November 19, 2022

South Korea has launched its first autonomous urban patrol robot, which patrols the streets of Seoul in search of dangerous situations. HL Mando, which developed "Goalie," said the robot is like a "moving surveillance camera," able to go where fixed CCTV cameras cannot. Equipped with satellite navigation and remote sensing technology, Goalie can avoid pedestrians and obstacles. HL Mando's Young-ha Cho said, "When the robot sees a dangerous situation or hears a sound like 'help me’, the control center operates the robot to move there and can check whether it is really a dangerous situation or not." HL Mando livestreams but does not store the footage from the robot, and encrypts its communications with the control center.  ... ' 

Robots that Write their Own Code

Interesting general challenge, 

Robots That Write Their Own Code

WEDNESDAY, NOVEMBER 02, 2022

Posted by Jacky Liang, Research Intern, and Andy Zeng, Research Scientist, Robotics at Google

A common approach used to control robots is to program them with code to detect objects, sequencing commands to move actuators, and feedback loops to specify how the robot should perform a task. While these programs can be expressive, re-programming policies for each new task can be time consuming, and requires domain expertise.

What if when given instructions from people, robots could autonomously write their own code to interact with the world? It turns out that the latest generation of language models, such as PaLM, are capable of complex reasoning and have also been trained on millions of lines of code. Given natural language instructions, current language models are highly proficient at writing not only generic code but, as we’ve discovered, code that can control robot actions as well. When provided with several example instructions (formatted as comments) paired with corresponding code (via in-context learning), language models can take in new instructions and autonomously generate new code that re-composes API calls, synthesizes new functions, and expresses feedback loops to assemble new behaviors at runtime. More broadly, this suggests an alternative approach to using machine learning for robots that (i) pursues generalization through modularity and (ii) leverages the abundance of open-source code and data available on the Internet.

Given code for an example task (left), language models can re-compose API calls to assemble new robot behaviors for new tasks (right) that use the same functions but in different ways.

To explore this possibility, we developed Code as Policies (CaP), a robot-centric formulation of language model-generated programs executed on physical systems. CaP extends our prior work, PaLM-SayCan, by enabling language models to complete even more complex robotic tasks with the full expression of general-purpose Python code. With CaP, we propose using language models to directly write robot code through few-shot prompting. Our experiments demonstrate that outputting code led to improved generalization and task performance over directly learning robot tasks and outputting natural language actions. CaP allows a single system to perform a variety of complex and varied robotic tasks without task-specific training.  ... ' 

Robots Killing Cockroaches

Robotic Insecticide

Robotic Laser Can Target, Kill Cockroaches

By New Scientist, October 27, 2022

The device is equipped with a laser, two cameras, and a small computer running an AI model that can be trained to target certain types of insect.

Researchers at the U.K.'s Heriot-Watt University developed a device that can kill cockroaches using a laser, two cameras, and an artificial intelligence (AI) model capable of targeting specific insect types.

The device could would be more cost-effective and environmentally friendly than insecticides, according to the researchers. The researchers were able to kill a common cockroach in about a second from 1.2 meters away using a 1,600 milliwatt laser.

They said the AI would have to be trained to target the laser at the cockroach's abdomen, which kills it quickly, rather than other parts of its body.

However, the dangerous impact of lasers on human eyesight means the device likely would not be suitable for home use.

Said Rakhmatulin, "If we talk about industrial or agricultural applications, it's a very serious possibility. It's very cheap, even compared to pesticides, because laser prices are not so high."

From New Scientist

View Full Article -   May Require Paid Subscription

Flexible Hand Grip

Safe? 

Flexible Robot Hand with Precise Grip Lifts 1,000 Times Its Own Weight

By New Scientist, August 4, 2022

Researchers at the Italian Institute of Technology have developed artificial muscles using a three-dimensional (3D) printer and tiny actuators that convert energy into movement.

Dubbed GeometRy-based Actuators that Contract and Elongate (GRACE), the actuator membranes are made from a flexible resin and move like a human muscle. The artificial muscles are stronger and more flexible due to pleats in the membranes that fold and unfold.

The researchers tested an 8-gram actuator that was able to lift 8 kilograms.   They also created a robotic hand with a wrist by connecting 18 actuators of varying sizes.

The hand could bend its fingers, twist its palm, and rotate at the wrist when pressure was applied to the different actuator membranes.

Said Italian Institute of Technology's Corrado De Pascali, "We started from the traditional artificial muscle and developed a new class of artificial muscles made of a single monolithic component."... 

GRACE actuator membranes are three-dimensionally printed from a resin that enables them to stretch and contract like a human muscle... 

From New Scientist    

View Full Article - May Require Paid Subscription

3D Printing Method to Make Robotic Materials

 Notable use of piezoelectric effect,  detecting obstacles.    See full article below.  

Engineers Create Single-Step 3D Printing Method to Make Robotic Materials

By UCLA Samueli School of Engineering, June 17, 2022

University of California, Los Angeles (UCLA) engineers and colleagues designed a one-step three-dimensional (3D) printing process for manufacturing robots.

Critical to the all-in-one approach is the design and printing of piezoelectric metamaterials, which can change shape and move in response to an electric field, or generate electricity in response to physical forces.

The researchers developed the metamaterials to bend, flex, twist, rotate, expand, or contract rapidly.  They constitute an internal network of sensory, moving, and structural components that can move in response to programmed commands.

UCLA’s Huachen Cui said the two-way piezoelectric effect permits the robots to “detect obstacles via echoes and ultrasound emissions, as well as respond to external stimuli through a feedback control loop that determines how the robots move, how fast they move, and toward which target they move.”

From UCLA Samueli School of Engineering

View Full Article   

IKEA Robotic Furniture Assembly

 Had heard a previous overview of this, if this could be done effectively it  would likely increase sales.  As I understand it is still a proposal, tell me if otherwise.

Need help building IKEA furniture? This robot can lend a hand  by Caitlin Dawson, University of Southern California  in Techxplore

As robots increasingly join forces to work with humans—from nursing care homes to warehouses to factories—they must be able to proactively offer support. But first, robots have to learn something we know instinctively: how to anticipate people's needs.

With that goal in mind, researchers at the USC Viterbi School of Engineering have created a new robotic system that accurately predicts how a human will build an IKEA bookcase, and then lends a hand—providing the shelf, bolt or screw necessary to complete the task. The research was presented at the International Conference on Robotics and Automation on May 30, 2021.

"We want to have the human and robot work together—a robot can help you do things faster and better by doing supporting tasks, like fetching things," said the study's lead author Heramb Nemlekar. "Humans will still perform the primary actions, but can offload simpler secondary actions to the robot."

Nemlekar, a Ph.D. student in computer science, is supervised by Stefanos Nikolaidis, an assistant professor of computer science, and co-authored the paper with Nikolaidis and SK Gupta, a professor of aerospace, mechanical engineering and computer science who holds the Smith International Professorship in Mechanical Engineering.

Adapting to variations

In 2018, a robot created by researchers in Singapore famously learned to assemble an IKEA chair itself. In this new study, the USC research team aims to focus instead on human-robot collaboration.

There are advantages to combining human intelligence and robot strength. In a factory for instance, a human operator can control and monitor production, while the robot performs the physically strenuous work. Humans are also more adept at those fiddly, delicate tasks, like wiggling a screw around to make it fit.

The key challenge to overcome: humans tend to perform actions in different orders. For instance, imagine you're building a bookcase—do you tackle the easy tasks first, or go straight for the difficult ones? How does the robot helper quickly adapt to variations in its human partners?  ... ' 

Amazon Wants a Rolling Home Assistant

Have now seen a number of rolling smart home assistants fail,  will this take it forward by using the Alexa infrastructure?

Amazon continues work on mobile home robot as it preps new high-end Echo, says report
Prototypes for the wheeled robot can be summoned using voice commands
By James Vincent in TheVerge

 Amazon is still working on a mobile home robot, according to a report from Bloomberg’s Mark Gurman. It’s also planning to add a high-end Echo to its lineup of Alexa devices.

We first heard about Amazon’s plans to build a wheeled home robot in April last year. The project is reportedly codenamed “Vesta” (after the Roman goddess of the hearth), and rumors suggest it’s a sort of “mobile Alexa” that’s able to follow users around their homes.

Today’s report doesn’t add significantly to this picture, but it seems Amazon is still keen to build the mobile device. It was apparently slated to launch this year but wasn’t ready for mass-production. Engineers have reportedly been pulled from other projects to work on Vesta, and Gurman reports that prototypes are “waist-high and navigate with the help of an array of computer-vision cameras.” They can also be summoned using voice commands.   ....  "

First Kroger - Ocado Fulfillment Center

Continued high tech improvements by Kroger.   Adds a number of jobs here.

Kroger breaking ground on $55M Butler County facility that will bring more than 400 jobs... 

Kroger and Ocado are set to break ground this morning in Butler County for the first-ever U.S. high-tech customer fulfillment center.

Kroger is investing $55 million to build the 335,000-square-foot facility at 6266 Hamilton Lebanon Road in Monroe, which will be the first of 20 in America.It’s expected to create more than 410 new jobs.MORE: Kroger’s robot-powered warehouse to create over 400 jobs in Monroe Kroger purchased the more than 98-acre property along Ohio 63 on March 20 for $3.1 million, according to Butler County Auditor’s Office records.The Ohio Tax Credit Authority voted last December to approve an 8-year, 1.362 percent tax credit for the project. .... " 

More in Forbes with video.

Driverless Platforms

I was pointed to this as a advertisement in IEEE Spectrum, and found it interesting.  Especially dealing with the current and potential autonomy of such systems.   Have worked with a  number of advanced tracking systems in supply chain applications.

" .... KVH Industries is a leading manufacturer of solutions that provide global high-speed Internet, television, and voice services via satellite to mobile users at sea and on land. These solutions include the mini-VSAT Broadband network, TracPhone satellite communications systems, and TracVision satellite television systems. KVH is also a premier manufacturer of high performance sensors and integrated inertial systems for defense and commercial guidance and stabilization applications. ... " 

 " ... The Navigation Technology Behind Driverless Platforms ... Precision sensors are critical components in driverless vehicles. These sensors are integrated with LIDAR, cameras and other systems to enable obstacle avoidance and relative navigation. Learn why KVH sensors are preferred for autonomous platforms, even in GNSS-denied environments.  ... "