A New Language Interface for Object Detection

 A New Language Interface for Object Detection

Pix2Seq: A New Language Interface for Object Detection  in the Google AI Blog

Friday, April 22, 2022

Posted by Ting Chen and David Fleet, Research Scientists, Google Research, Brain Team

Object detection is a long-standing computer vision task that attempts to recognize and localize all objects of interest in an image. The complexity arises when trying to identify or localize all object instances while also avoiding duplication. Existing approaches, like Faster R-CNN and DETR, are carefully designed and highly customized in the choice of architecture and loss function. This specialization of existing systems has created two major barriers: (1) it adds complexity in tuning and training the different parts of the system (e.g., region proposal network, graph matching with GIOU loss, etc.), and (2), it can reduce the ability of a model to generalize, necessitating a redesign of the model for application to other tasks.

In “Pix2Seq: A Language Modeling Framework for Object Detection”, published at ICLR 2022, we present a simple and generic method that tackles object detection from a completely different perspective. Unlike existing approaches that are task-specific, we cast object detection as a language modeling task conditioned on the observed pixel inputs. We demonstrate that Pix2Seq achieves competitive results on the large-scale object detection COCO dataset compared to existing highly-specialized and well-optimized detection algorithms, and its performance can be further improved by pre-training the model on a larger object detection dataset. To encourage further research in this direction, we are also excited to release to the broader research community Pix2Seq’s code and pre-trained models along with an interactive demo.  .... 

Google Announces 3D Object Detection

 Could lead to very  interesting applications.  More phone based marketing and location possibilities? And another round of complaints like we have seen from face recognition. 

Real-Time 3D Object Detection on Mobile Devices with MediaPipe

Wednesday, March 11, 2020

Posted by Adel Ahmadyan and Tingbo Hou, Software Engineers, Google Research

Object detection is an extensively studied computer vision problem, but most of the research has focused on 2D object prediction. While 2D prediction only provides 2D bounding boxes, by extending prediction to 3D, one can capture an object’s size, position and orientation in the world, leading to a variety of applications in robotics, self-driving vehicles, image retrieval, and augmented reality. Although 2D object detection is relatively mature and has been widely used in the industry, 3D object detection from 2D imagery is a challenging problem, due to the lack of data and diversity of appearances and shapes of objects within a category.

Today, we are announcing the release of MediaPipe Objectron, a mobile real-time 3D object detection pipeline for everyday objects. This pipeline detects objects in 2D images, and estimates their poses and sizes through a machine learning (ML) model, trained on a newly created 3D dataset. Implemented in MediaPipe, an open-source cross-platform framework for building pipelines to process perceptual data of different modalities, Objectron computes oriented 3D bounding boxes of objects in real-time on mobile devices.  ... "

Identifying Birds from Behind

Not banned for bias yet.   So at least we can continue to fine tune accuracy as needed and apply it to studies to help the avian world.

Birdwatching AI can recognise individual birds from behind
in NewScientist   By Michael Le Page

Artificial intelligence has been trained to recognise individual birds, which is more than we humans are capable of. The system is being developed for biologists studying wild animals, but could be adapted so that people can identify individual birds in their surroundings.

André Ferreira at the Center for Functional and Evolutionary Ecology in Montpellier, France, started the project while studying how individual sociable weavers contribute to their colonies. This is normally done by putting coloured tags on their legs and sitting by nests to watch them, which is very time-consuming. Ferreira tried filming the colonies instead, but often the coloured tags weren’t visible in the footage, so he and his colleagues turned to AI. ... "  

Recognizing Objects

More advances in fast vision systems.

Optical System Could Lead to Devices That Can Recognize Objects Instantly
UCLA Newsroom
Matthew Chin
March 4, 2020

An optical neural network developed at the University of California, Los Angeles (UCLA) Henry Samueli School of Engineering that concurrently works with multiple wavelengths of light could potentially lead to devices that instantly recognize objects without additional computer processing, with potential applications for robots and autonomous vehicles. The network is a maze with an array of translucent wafers made of different materials like plastic or glass, engineered at a smaller scale than the wavelength of light to split beams into various directions. Said UCLA's Aydogan Ozcan, “There is richer information when you can see colors through different wavelengths of light. Most scenes naturally contain information in vivid color, so the more wavelengths that a network can ‘see,’ the more it increases the amount of information it can process.” .... ' 

Heartbeat Identification

Another example of pattern recognition,  here with lasers at a distance.

The Pentagon can now identify people by measuring their heartbeats in DigitalTrends
pentagon heartbeat identification

As if facial recognition and digital fingerprinting weren’t scary enough, the Pentagon has reportedly developed a method for identifying and tracking people through their heartbeat.

Heartbeats are as unique and distinctive as fingerprints, but are distinct in that they can be read from a distance. And it’s this that the Pentagon is taking advantage of, according to a report in the MIT Technology Review.

Developed for identifying combatants in war zones, the idea is to identify individuals by listening in to their cardiac signatures using an infrared laser. Unlike other identification methods like facial recognition, it’s impossible to disguise a heartbeat in any way. The laser method also works through clothing at a distance of up to 200 meters (219 yards). In the future, this range could be extended to be even longer.

“I don’t want to say you could do it from space,” Steward Remaly of the Pentagon’s Combating Terrorism Technical Support Office told the MIT Technology Review, “but longer ranges should be possible.”  ... " 

Lenses for Dog Recognition

Have also been seeing this for livestock applications like recognizing horses and pigs.  Imagine other applications.

Object recognition taken beyond humans.  Dogs, and some indication cats are next.

Snapchat lenses now officially work on dogs
Because pets can't take selfies on their own. ....

By AJ Dellinger, @ajdell

New Means of Seeing Objects

Closer to biomimicry, the article says a not deeply technical description:

New AI computer vision system mimics how humans visualize and identify objects
 UCLA Samueli School of Engineering

Summary:
Researchers have demonstrated a computer system that can discover and identify the real-world objects it 'sees' based on the same method of visual learning that humans use.
Researchers from UCLA Samueli School of Engineering and Stanford have demonstrated a computer system that can discover and identify the real-world objects it "sees" based on the same method of visual learning that humans use.

The system is an advance in a type of technology called "computer vision," which enables computers to read and identify visual images. It is an important step toward general artificial intelligence systems -- computers that learn on their own, are intuitive, make decisions based on reasoning and interact with humans in a more human-like way. Although current AI computer vision systems are increasingly powerful and capable, they are task-specific, meaning their ability to identify what they see is limited by how much they have been trained and programmed by humans.

Even today's best computer vision systems cannot create a full picture of an object after seeing only certain parts of it -- and the systems can be fooled by viewing the object in an unfamiliar setting. Engineers are aiming to make computer systems with those abilities -- just like humans can understand that they are looking at a dog, even if the animal is hiding behind a chair and only the paws and tail are visible. Humans, of course, can also easily intuit where the dog's head and the rest of its body are, but that ability still eludes most artificial intelligence systems. .... " 

Machine Learning Tackles Simultaneous Speech and Object Recognition

Could lead to at least greater efficiency for assistants.  Note the data involved. 

Machine-Learning System Tackles Speech and Object Recognition, All at Once     MIT News  by Rob Matheson

Researchers at the Massachusetts Institute of Technology (MIT) have developed a system that can learn to identify objects within an image, based on a spoken description of the image. When provided with an image and an audio caption, the system can highlight in real-time the relevant regions of the image being described. The system learns words directly from recorded speech clips and objects in raw images, and associates them with one another. The researchers trained the model on a total of 400,000 image-caption pairs, and held out 1,000 random pairs for testing. Said researcher David Harwath, “We wanted to do speech recognition in a way that’s more natural, leveraging additional signals and information that humans have the benefit of using, but that machine learning algorithms don’t typically have access to. We got the idea of training a model in a manner similar to walking a child through the world and narrating what you’re seeing.” ... '