3D Scene Understanding

Intelligence allows us to quickly understand our world, here a good example, our visual world, based on sensors.   To establish context we can depend on.    Many new kinds of sensors now, and means of using them. 

In the Google AI Blog Technical 

3D Scene Understanding with TensorFlow 3D,   Thursday, February 11, 2021

Posted by Alireza Fathi, Research Scientist and Rui Huang, AI Resident, Google Research

The growing ubiquity of 3D sensors (e.g., Lidar, depth sensing cameras and radar) over the last few years has created a need for scene understanding technology that can process the data these devices capture. Such technology can enable machine learning (ML) systems that use these sensors, like autonomous cars and robots, to navigate and operate in the real world, and can create an improved augmented reality experience on mobile devices. The field of computer vision has recently begun making good progress in 3D scene understanding, including models for mobile 3D object detection, transparent object detection, and more, but entry to the field can be challenging due to the limited availability tools and resources that can be applied to 3D data.

In order to further improve 3D scene understanding and reduce barriers to entry for interested researchers, we are releasing TensorFlow 3D (TF 3D), a highly modular and efficient library that is designed to bring 3D deep learning capabilities into TensorFlow. TF 3D provides a set of popular operations, loss functions, data processing tools, models and metrics that enables the broader research community to develop, train and deploy state-of-the-art 3D scene understanding models.

TF 3D contains training and evaluation pipelines for state-of-the-art 3D semantic segmentation, 3D object detection and 3D instance segmentation, with support for distributed training. It also enables other potential applications like 3D object shape prediction, point cloud registration and point cloud densification. In addition, it offers a unified dataset specification and configuration for training and evaluation of the standard 3D scene understanding datasets. It currently supports the Waymo Open, ScanNet, and Rio datasets. However, users can freely convert other popular datasets, such as NuScenes and Kitti, into a similar format and use them in the pre-existing or custom created pipelines, and can leverage TF 3D for a wide variety of 3D deep learning research and applications, from quickly prototyping and trying new ideas to deploying a real-time inference system. ... " 

Understanding Scenes Using Neural Nets

A powerful kind of visual intelligence, with many applications wen integrated with cameras and video.   Ultimately this kind of 'common sense', here visual sense, style reasoning can make conversational and assistant systems appear more intelligent.

Google researchers created an amazing scene-rendering AI

A neural network from Google's DeepMind has impressive spatial reasoning skills.
 By Timothy B. Lee in Arstechnica

New research from Google's UK-based DeepMind subsidiary demonstrates that deep neural networks have a remarkable capacity to understand a scene, represent it in a compact format, and then "imagine" what the same scene would look like from a perspective the network hasn't seen before.

Human beings are good at this. If shown a picture of a table with only the front three legs visible, most people know intuitively that the table probably has a fourth leg on the opposite side and that the wall behind the table is probably the same color as the parts they can see. With practice, we can learn to sketch the scene from another angle, taking into account perspective, shadow, and other visual effects. ... " 

Pupil Arousal in Shopping

A different kind of eye tracking.   Non conscious reactions that are also related to other neuromarketing methods.

" ... Abstract:  The present study proposes arousal as an important mechanism driving buying impulsiveness. We examined the effect of buying impulsiveness on arousal in non-shopping and shopping contexts. In an eye-tracking experiment, we measured pupil dilation while participants viewed and rated pictures of shopping scenes and non-shopping scenes. The results demonstrated that buying impulsiveness is closely associated with arousal as response to viewing pictures of shopping scenes. This pertained for hedonic shopping situations as well as for utilitarian shopping situations. Importantly, the effect did not emerge for non-shopping scenes. Furthermore, we demonstrated that arousal of impulsive buyers is independent from cognitive evaluation of scenes in the pictures.   ... " 

Google does Scene Analysis

In DigitalStrategy: Don't know where the IP rests today, but here is Google also doing scene analysis on images.  Again I can see some interesting applications with Ad copy analysis, and real time interaction between displays and shoppers.   See also, related work at Stanford.

Stanford Building Stories from Pictures

This was a sub problem solution we needed for an expert system that worked with archives of advertising copy.  Later I worked with a startup that aimed to classify photographs for their potential use in Ads.   Good progress in this space.

Stanford team creates computer vision algorithm that can describe photos
Computers only recently began to get the software needed to discern unknown objects; now machine-learning takes computer vision to the next level with a system that can describe objects and put them into context. Coming soon, better visual search?

Stanford Professor Fei-Fei Li, director of the Stanford Artificial Intelligence Lab, leads work on a computer vision system. ... Computer software only recently became smart enough to recognize objects in photographs. Now, Stanford researchers using machine learning have created a system that takes the next step, writing a simple story of what's happening in any digital image. ... " 

Update:  Here is a link to Google's work on image scene description.

Video based Detection Methods

Some good points about video detection and analysis.  This area continues to get more sophisticated and pervasive.  Expect it to get even more common:
" .... It's very likely that you've been on camera from the moment you left home today -- recorded as you rode in the elevator, walked on the street, bought coffee at the local deli, withdrew money, and as you've moved throughout your office building. While you're at work, cameras might be recording the events in your home, capturing the nanny's interaction with your children and when your cat drinks from her water bowl. Your image is part of the crowd scene in the camera advertisement on a billboard in Times Square, passersby are looking at you on the video display at an electronics store, the game system in your living room is analyzing your gestures, and your face is being analyzed as you go through security at the airport.... "