Turning Snapshots into 3D

Image transformation of value for complex manufacturing error examination type applications.

NVIDIA’s Tiny New AI Transforms Photos Into Full 3D Scenes in Mere Seconds  By Jason Dorrier -Mar 27, 2022    in Sigularity WEB     Demo images at the link.

There was a time when converting an old photograph into a digital image impressed people. These days we can do a bit more, like bringing vintage photos to life à la Harry Potter. And this week, chipmaker NVIDIA performed another magic trick.

Building on previous work, NVIDIA researchers showed how a small neural network trained on a few dozen images can render the pictured scene in full 3D. As a demo, the team transformed images of a model holding a Polaroid camera—an ode to Andy Warhol—into a 3D scene.

The work stands out for a few reasons.

First, it’s very speedy. Earlier AI models took hours to train and minutes to render 3D scenes. NVIDIA’s neural network takes no more than a few minutes to train and renders the scene in tens of milliseconds. Second, the AI itself is diminutive in comparison to today’s hulking language models. Large models like GPT-3 train on hundreds or thousands of graphics processing units (GPUs). NVIDIA’s image rendering AI runs on a single GPU.

The work builds on neural radiance fields (NeRFs), a technique developed by researchers at UC Berkeley, UC San Diego, and Google Research, a couple years ago. In short, a NeRF takes a limited data set—say, 36 photographs of a subject captured from a variety of angles—and then predicts the color, intensity, and direction of light radiating from any point in the scene. That is, the neural net fills in the gaps between images with best guesses based on the training data. The result is a continuous 3D space stitched together from the original images.

NVIDIA’s recent contribution, outlined in a paper, puts NeRFs on performance enhancing drugs. According to the paper, the new method, dubbed Instant NeRF, exploits an approach known as multi-resolution hash grid encoding to simplify the algorithm’s architecture and run it in parallel on a GPU. This upped performance by a few orders of magnitude—their algorithm runs up to 1,000 times faster, according to an NVIDIA blog post—without sacrificing quality.  .... '