How Generative AI Could Create Assets for the Metaverse

Have my doubts of how effectively , but interesting thought  ... 

How Generative AI could Create Assets for the Metaverse | Jensen Huang

Check out the on-demand sessions from the Low-Code/No-Code Summit to learn how to successfully innovate and achieve efficiency by upskilling and scaling citizen developers. Watch now.

The metaverse skyrocketed into our collective awareness during the height of the pandemic, when people longed for better ways to connect with each other than video calls. Gaming’s hot growth during the pandemic also pushed it forward. But the metaverse became so trendy that it now faces a backlash, and folks aren’t talking about it as much.

Yet technologies that will power the metaverse are speeding ahead. One of those technologies is generative AI, which uses deep learning neural networks to produce creative concept art and other ideas based on simple text prompts. ... 

Jensen Huang, CEO of AI and graphics chip maker Nvidia, believes that generative AI will be transformational and it’s just getting started. One of its biggest applications could be with the metaverse, which has huge demands for content as developers need to fill out virtual worlds with 3D assets. And numerous companies like Stable Diffusion, Promethean AI and Ludo AI are using these technologies to automatically generate artwork and other assets for gaming and metaverse applications. Nvidia has its own research going on this front.

Many metaverse companies are hoping that generative AI will help provide the resources to help them build out their worlds. Huang believes you will see progress when you enter more and more prompts — such as text to flesh out a concept — and the concept imagery gets better and better. And he also believes that when it becomes reusable across different Omniverse applications, then it will be clear that generative AI has reached a more mature stage.

EVENT

Intelligent Security Summit

Learn the critical role of AI & ML in cybersecurity and industry specific case studies on December 8. Register for your free pass today.

Register Now

I recently caught up with Huang for a short interview on the metaverse and gaming. Our GamesBeat Summit: Into the Metaverse 3 event is coming on February 1 to February 2.

Here’s an edited transcript of our interview: ...  

Using Synthetic Data

Synthetic data training from real data.

When It Comes to AI, Can We Ditch the Datasets?

MIT News, Adam Zewe, March 15, 2022

Massachusetts Institute of Technology (MIT) researchers have demonstrated the use of a generative machine-learning model to produce synthetic data, based on real data, to train another model for image classification. Researchers showed the generative model millions of images containing objects in a specific class, after which it learned those objects' appearance in order to generate similar objects. MIT's Ali Jahanian said generative models also learn how to transform underlying training data, and connecting a pretrained generative model to a contrastive learning model enabled both models to work together automatically. The results show that a contrastive representation learning model trained only on synthetic data can learn visual representations that rival or top those learned from real data. In analyzing how the number of samples influenced the model's performance, researchers determined that, in some cases, generating larger numbers of unique samples facilitated additional enhancements.  ... ' 

Future of AI is Generative

Looks to be a good talk, about to experience.

THE HIVE THINK TANK LECTURE: THE FUTURE OF AI IS GENERATIVE NOT DISCRIMINATIVE

On May 26, 2021, Steve Omohundro gave a lecture on “The Future of AI is Generative not Discriminative” to The Hive’s excellent “Think Tank” group:

Here’s the video of the talk:  https://youtu.be/4I3FVZ1J6nY  

The Future of AI is Generative not Discriminative 5/26/2021 from Steve Omohundro

and the abstract:

The deep learning AI revolution has been sweeping the world for a decade now. Deep neural nets are routinely used for tasks like translation, fraud detection, and image classification. PwC estimates that they will create $15.7 trillion/year of value by 2030. But most current networks are “discriminative” in that they directly map inputs to predictions. This type of model requires lots of training examples, doesn’t generalize well outside of its training set, creates inscrutable representations, is subject to adversarial examples, and makes knowledge transfer difficult. People, in contrast, can learn from just a few examples, generalize far beyond their experience, and can easily transfer and reuse knowledge. In recent years, new kinds of “generative” AI models have begun to exhibit these desirable human characteristics. They represent the causal generative processes by which the data is created and can be compositional, compact, and directly interpretable. Generative AI systems that assist people can model their needs and desires and interact with empathy. Their adaptability to changing circumstances will likely be required by rapidly changing AI-driven business and social systems. Generative AI will be the engine of future AI innovation.

KEYNOTE SPEAKER: Steve Omohundro

Identification Improvements Using Blurred Fingerprints

Fingerprint clear up analysis.    An example of the use of Generative Adversarial Networks.  But note the comment on auditing the results in court.

AI Clears Up Images of Fingerprints to Help with Identification

New Scientist, Matthew Sparkes, June 28, 2021

West Virginia University researchers have trained an artificial intelligence (AI) model to clean up distorted images of fingerprints from crime scenes to improve identification. The researchers developed a generative adversarial network by creating blurred versions of 15,860 clean fingerprint images from 250 subjects. They trained the AI using nearly 14,000 of these pairs of images; when they tested its performance on the remainder, they found the model to be 96% accurate at the lower end of the range of blurring intensity, and 86% at the higher end. Forensic Equity's David Goodwin said the use of neural networks to manipulate images would have trouble standing up in court because they cannot be audited like human-generated code, and the inner workings of these models are unknown.   ... '

AI Changing Science

The race is on.   But how creative can this be?   The problems I see still require lots of data gathered in a given exploratory context.  The lots of data is one thing, but the context is harder yet.  Now you could test lots of hypotheses this way,  but the combinatorics get very big.   Quite a challenge.

 How Artificial Intelligence Is Changing Science  in QuantaMag
The latest AI algorithms are probing the evolution of galaxies, calculating quantum wave functions, discovering new chemical compounds and more. Is there anything that scientists do that can’t be automated?

Rachel Suggs for Quanta Magazine
Dan Falk

Alice and Bob Meet the Wall of Fire - The Biggest Ideas in Science from Quanta – Available now!
No human, or team of humans, could possibly keep up with the avalanche of information produced by many of today’s physics and astronomy experiments. Some of them record terabytes of data every day — and the torrent is only increasing. The Square Kilometer Array, a radio telescope slated to switch on in the mid-2020s, will generate about as much data traffic each year as the entire internet.

The deluge has many scientists turning to artificial intelligence for help. With minimal human input, AI systems such as artificial neural networks — computer-simulated networks of neurons that mimic the function of brains — can plow through mountains of data, highlighting anomalies and detecting patterns that humans could never have spotted.

Of course, the use of computers to aid in scientific research goes back about 75 years, and the method of manually poring over data in search of meaningful patterns originated millennia earlier. But some scientists are arguing that the latest techniques in machine learning and AI represent a fundamentally new way of doing science. One such approach, known as generative modeling, can help identify the most plausible theory among competing explanations for observational data, based solely on the data, and, importantly, without any preprogrammed knowledge of what physical processes might be at work in the system under study. Proponents of generative modeling see it as novel enough to be considered a potential “third way” of learning about the universe...  "

GANs: Generative Adversarial Networks

Brought to my attention as useful

Computer Science > Machine Learning
NIPS 2016 Tutorial: Generative Adversarial Networks
Ian Goodfellow

This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs). The tutorial describes: (1) Why generative modeling is a topic worth studying, (2) how generative models work, and how GANs compare to other generative models, (3) the details of how GANs work, (4) research frontiers in GANs, and (5) state-of-the-art image models that combine GANs with other methods. Finally, the tutorial contains three exercises for readers to complete, and the solutions to these exercises. .... " 

Augmenting Human Intelligence

Quite an interesting piece.   Obvious at one level.  Its always about augmenting human intelligence,  at least in any near term view.   So then its about how well you can create the most useful tools.  Yes, it is about reasoning, but also about doing valuable things.  About inserting AI into valuable human process.   That's a challenge in every piece of work I do.

Using Artificial Intelligence to Augment Human Intelligence   (In Distill,Via O'Reilly) 

By creating user interfaces which let us work with the representations inside machine learning models, we can give people new tools for reasoning.

" .... Our essay begins with a survey of recent technical work hinting at artificial intelligence augmentation, including work on generative interfaces – that is, interfaces which can be used to explore and visualize generative machine learning models. Such interfaces develop a kind of cartography of generative models, ways for humans to explore and make meaning from those models, and to incorporate what those models “know” into their creative work.

Our essay is not just a survey of technical work. We believe now is a good time to identify some of the broad, fundamental questions at the foundation of this emerging field. To what extent are these new tools enabling creativity? Can they be used to generate ideas which are truly surprising and new, or are the ideas cliches, based on trivial recombinations of existing ideas? Can such systems be used to develop fundamental new interface primitives? How will those new primitives change and expand the way humans think?  ..... " 

Imaginary People

We are not far from generating very convincing faces of people.  Even making them dynamic.  And if you put these on humanoid robots?   Give them Generative AI?   In the Verge: 

" .... As we get better at making, faking, and manipulating human faces with machine learning, one thing is abundantly clear: things are going to get ~freaky~ fast.

Case in point: this online demo hosted (and, we presume, made) by web developer AlteredQualia. It combines two different research projects, both of which use neural networks. The first is DeepWarp, which alters where subjects in photographs are looking, and the second is a work in progress by Mike Tyka dubbed Portraits of Imaginary People. This does exactly what it says on the tin: feeding a generative neural network with a bunch of faces and getting it to create similar samples. .... "

Generative Networks

Its all about learning.  But learning does not just mean saving away information.  Its about constructing an architecture that allows that information to be accessible and and used in context. Ontologies are one such approach.   Generative methods are another.  Ultimately a big challenge for advancing AI.

How generative artificial networks are accelerating AI learning  By Larry Alton  In VentureBeat:
 
" ... One of the biggest limiting factors of artificial intelligence (AI) systems is that they can’t think or conceptualize the world the way humans can. .... 

Google’s GANs

Google researcher and AI expert Ian Goodfellow is working on AI that belongs to a group of “generative models,” which are designed to create images and sounds comparable to those you’d find in the real world. This is a deceptively difficult task, as AI programs must first conceptually understand what it is they’re trying to replicate, a leap forward in intuitive thinking that has historically been reserved for human beings.

Goodfellow is attempting to accomplish this using something called generative artificial networks, or GANs, which are sets of two dueling, semi-competing AI algorithms designed to continuously one-up each other. For example, one AI may be programmed to generate imagery that looks realistic, while the other AI will be programmed to distinguish real images from machine-generated ones. Over time, the image generator will get better at generating realistic images, and the “judge” will get better at discerning them.  .... "