Improving Digital Images with Machine Learning

Akin to somethings we experimented with in a startup years ago.  Taking it much farther, it seems.   Need to see more about this!   A sort of knowledge extraction process from images.

ML Approach Brings Digital Photos Back to Life,     By Texas A&M Engineering News

A machine learning (ML) technique developed by researchers at Texas A&M University (TAMU) enables users to produce novel views of a scene from a single photo.  TAMU's Nima Kalantari said, "We can download and use any image on the Internet, even ones that are 100 years old, and essentially bring it back to life and look at it from different angles."

Kalantari and graduate student Qinbo Li trained a deep learning network to generate novel views based on a single input image by showing it a set of over 2,000 images and corresponding novel-view images.

Each input image was converted into a multiplane image, training the network to infer the location of objects in the scene.

From Texas A&M Engineering News

High Quality Images of Moving Objects

I recall having to solve this problem for diagnosing from images of manufacturing machine parts.

Capturing Moving Subjects in Still-Life Quality
EPFL News (Switzerland)
June 18, 2020

Researchers at the Swiss Federal Institute of Technology in Lausanne (EPFL) Advanced Quantum Architecture Laboratory and the University of Wisconsin-Madison (UW-Madison) Wision Laboratory have developed a method for capturing extremely clear images of moving subjects. UW-Madison's Mohit Gupta borrowed EPFL's SwissSPAD camera, which generates two-dimensional binary images at a resolution of 512 x 512 pixels. EPFL's Edoardo Charbon said SwissSPAD captures 100,000 binary images per second, as an algorithm corrects for variations; the researchers built a high-definition image of a moving subject by combining these photos. The team aims to repeat the experiment with the MegaX camera, which Charbon said "is similar to SwissSPAD in many ways; it's also a depth-sensing camera, thus it can generate [three-dimensional] images."   ... " 

Transforming Objects in Image Editing

Interesting use of the ideaof learned transformation.   Could be used for example in editing images with their technical details to represent contextual details.

AI Taught to Instantly Transform Objects in Image-Editing Software
New Scientist
Donna Lu

A team of researchers from NVIDIA, Adobe Research, and Aalto University in Finland has developed an artificial intelligence (AI)-controlled program that lets the user transform the shape of objects in images, as well as permitting adjustments to lighting and perspective. The researchers built the program using generative adversarial networks (GANs), which are made up of two competing AIs: a generator that creates an image, and a discriminator that decides whether the image is real or fake. After several rounds, the generator can produce images so realistic that the discriminator can no longer tell the difference. The team analyzed existing GANs to determine the way they each modify images into components. The program provides a simple yet powerful way to create images with existing AIs, without the massive computational requirements to train new algorithms for specific tasks.  .... "

Recovering “Lost Dimensions” of Images, Video

Fascinating idea I could have used long ago.  Projections, for example, from advertising copy, archived and needed for reapplication. 

Recovering “Lost Dimensions” of Images, Video
MIT News   Rob Matheson

Researchers at the Massachusetts Institute of Technology (MIT) have developed a model that recovers valuable data lost from images and video that have been "collapsed" into lower dimensions. Captured visual data often collapses data of multiple dimensions of time and space into one or two dimensions called "projections." The researchers created a "visual deprojection" model that uses a neural network to learn patterns that match low-dimensional projections to their original higher-dimensional images and videos. The model takes in new projections and uses what it has learned to recreate the original data. During testing, the model synthesized accurate video frames showing people walking by extracting information from single-one-dimensional lines. The model also recovered video frames from single, motion-blurred projections of digits moving around a screen. ..."

Augmenting Movie Making Using VR

Makes sense to use a visual medium to show alternative views to be chosen or combined.   Using game techniques.   No mention of a potential AR capability when combining shots with physical sets, but would be similar. How this would influence creativity remains to be seen.  It is likely that people who work with visual choices will start to see more of this kind of capability, with the further ability of seeing measures like cost, predicted audience reactions, continuity measures and more. So how might this work in other industries?  With these measures also leading to different kinds of optimization of overall goals.   On we go.

Directors Using VR on Set to Find the Perfect Shot 
New Scientist
By Andrew Rosenblum  in ACM

Technicolor has developed a virtual reality (VR) headset that allows film directors to instantly see what a scene will look like with computer-generated imagery added. Previously, directors had to send footage of human actors to render farms (groups of powerful computers) that would add graphical elements in a process that could take days to complete. The VR headset incorporates a stack of high-powered chips originally designed for gaming, which provide directors the opportunity to see immediately how a scene will look with the addition of computer-generated elements. The director can be immersed in the three-dimensional world of the scene and find the best camera angles. Independent filmmaker Kevin Margo said the technology can reduce the cost of visual effects up to 20% by trimming the expenses of lighting, and of the process to combine the elements into a single image. ... "

Fast.ai Students Show Us How to De-Crapify

Some very intriguing image and video filtering and improvement results, with lots of images and audio.   Join their newsletter.

From  Fast.AI
Deep learning is transforming the world. We are making deep learning easier to use and getting more people from all backgrounds involved through our:  free courses for coders, software library, cutting-edge research, community  ....  

Decrappification, DeOldification, and Super Resolution
In this article we will introduce the idea of “decrappification”, a deep learning method implemented in fastai on PyTorch that can do some pretty amazing things, like… colorize classic black and white movies—even ones from back in the days of silent movies, like this ....

The genesis of DeOldify

DeOldify was developed at around the same time that fast.ai started looking at decrappification, and was designed to colorize black and white photos. Jason Antic watched the Spring 2018 fast.ai course that introduced GANs, U-Nets, and other techniques, and wondered about what would happen if they were combined for the purpose of colorization. Jason’s initial experiments with GANs were largely a failure, so he tried something else - the self-attention GAN (SAGAN). His ambition was to be able to successfully colorize real world old images with the noise, contrast, and brightness problems caused by film degradation. The model needed to be trained on photos with these problems simulated. To do this, he started with the images in the ImageNet dataset, converted them to b&w, and then added random contrast, brightness, and other changes. In other words, he was “crappifying” images too!

The results were amazing, and people all over the internet were talking about Jason’s new “DeOldify” program. Jeremy saw some of the early results and was excited to see that someone else was getting great results in image generation. He reached out to Jason to learn more. Jeremy and Jason soon realized that they were both using very similar techniques, but had both developed in some different directions too. So they decided to join forces and develop a decrappification process that included all of their best ideas. ... "