Parallel Domain’s API lets Customers use Generative AI to build Synthetic Datasets

Interesting Direction

Parallel Domain’s API lets customers use generative AI to build synthetic datasets

Rebecca Bellan@rebeccabellan / 11:00 AM EDT•June 19, 2023

Parallel Domain is putting the ability to generate synthetic datasets into the hands of its customers. The San Francisco-based startup has launched a new API called Data Lab that stands on the shoulders of generative AI giants, giving machine-learning engineers control over dynamic virtual worlds to simulate any scenario imaginable. 

“All you have to do is you go to GitHub, you install the API, and then you can start writing Python code that generates datasets,” Kevin McNamara, founder and CEO of Parallel Domain, told TechCrunch.

Data Lab allows engineers to generate objects that weren’t previously available in the startup’s asset library. The API uses 3D simulation to provide a foundation upon which an engineer, through a series of simple prompts, can layer the real world in all its randomness on top. Want to train your model to drive on a highway with a cab flipped over across two lanes? Easy. Think your robotaxi should know how to identify a human dressed in an inflatable dinosaur outfit? Done.

A 5G eSIM cloud platform allows for seamless connectivity across borders for businesses wishing to scale rapidly in the global marketplace

The goal is to give autonomy, drone and robotics companies more control over and more efficiency in building large datasets so they can train their models quicker and at a deeper level.

“Iteration time now goes to essentially how fast can you, as an ML engineer, think of what you want and translate that into an API call, a set of code?” said McNamara. “There is a near infinite, unbounded level of stuff a customer could type in for a prompt, and the system just works.”

Back to Synthetic Data

Have not talked this for sometime, still interesting.

What is Synthetic Data? The Good, the Bad, and the Ugly

Sharing data can often enable compelling applications and analytics. However, more often than not, valuable datasets contain information of sensitive nature, and thus sharing them can endanger the privacy of users and organizations.

A possible alternative gaining momentum in the research community is to share synthetic data instead. The idea is to release artificially generated datasets that resemble the actual data — more precisely, having similar statistical properties.

So how do you generate synthetic data? What is that useful for? What are the benefits and the risks? What are the fundamental limitations and the open research questions that remain unanswered?

All right, let’s go!

How To Safely Release Data?

Before discussing synthetic data, let’s first consider the “alternatives.”

Anonymization: Theoretically, one could remove personally identifiable information before sharing it. However, in practice, anonymization fails to provide realistic privacy guarantees because a malevolent actor often has auxiliary information that allows them to re-identify anonymized data. For example, when Netflix de-identified movie rankings (as part of a challenge seeking better recommendation systems), Arvind Narayanan and Vitaly Shmatikov de-anonymized a large chunk by cross-referencing them with public information on IMDb.

Aggregation Another approach is to share aggregate statistics about a dataset. For example, telcos can provide statistics about how many people are in some specific locations at a given time — e.g., to assess footfall and decide where one should open a new store. However, this is often ineffective too, as the aggregates can still help an adversary learn something about specific individuals.

Differential Privacy: More promising attempts come from providing access to statistics obtained from the data while adding noise to the queries’ response, guaranteeing differential privacy. However, this approach generally lowers the dataset’s utility, especially on high-dimensional data. Additionally, allowing unlimited non-trivial queries on a dataset can reveal the whole dataset, so this approach needs to keep track of the privacy budget over time.   ... '

Empirical Optimization with Divergent Fixed Point Algorithm – When All Else Fails

 Of Technical Interest, worth signing into:

Empirical Optimization with Divergent Fixed Point Algorithm – When All Else Fails

 DECEMBER 9, 2022 EXPLAINABLE AI FEATURED POSTS MACHINE LEARNING SYNTHETIC DATA VISUALIZATION

Entitled “Empirical Optimization with Divergent Fixed Point Algorithm – When All Else Fails”, the full version in PDF format is accessible in the “Free Books and Articles” section, here. Also discussed in details with Python code in my book “Synthetic Data”, available here.

While the technique discussed here is a last resort solution when all else fails, it is actually more powerful than it seems at first glance. First, it also works in standard cases with “nice” functions. However, there are better methods when the function behaves nicely, taking advantage of the differentiability of the function in question, such as the Newton algorithm (itself a fixed-point iteration). It can be generalized to higher dimensions, though I focus on univariate functions here.

Perhaps the attractive features are the fact that it is simple and intuitive, and quickly leads to a solution despite the absence of convergence. However, it is an empirical method and may require working with different parameter sets to actually find a solution. Still, it can be turned into a black-box solution by automatically testing different parameter configurations. In that respect, I compare it to the empirical elbow rule to detect the number of clusters in unsupervised clustering problems. I also turned the elbow rule into a fully automated black-box procedure, with full details offered in the same book .... '

About the author

Vincent Granville is a pioneering data scientist and machine learning expert, co-founder of Data Science Central (acquired by  TechTarget in 2020), founder of MLTechniques.com, former VC-funded executive, author and patent owner. Vincent’s past corporate experience includes Visa, Wells Fargo, eBay, NBC, Microsoft, and CNET. Vincent is also a former post-doc at Cambridge University, and the National Institute of Statistical Sciences (NISS).  

Vincent published in Journal of Number Theory, Journal of the Royal Statistical Society (Series B), and IEEE Transactions on Pattern Analysis and Machine Intelligence. He is also the author of multiple books, including “Intuitive Machine Learning and Explainable AI”, available here. He lives  in Washington state, and enjoys doing research on spatial stochastic processes, chaotic dynamical systems, experimental math and probabilistic number theory.  

Why Synthetic Data Is Key to Paving the Way for Smart Cities

Good thoughts here for other applications as well

Why Synthetic Data Is Key to Paving the Way for Smart Cities

By Spiceworks, November 18, 2022

Illustration of a futuristic smart city.

Gartner predicts synthetic data will overshadow real-world data in artificial intelligence (AI) models by 2030. It saves valuable time and energy while also providing the opportunity to counteract incumbent bias in the base data.

Of course, model training should not be limited to one data source or the other. Synthetic and real-world data can be complementary in efficiently progressing training at a lower cost, with greater applicability and without privacy or bias issues. But, it stands that synthetic data is essential for developing smarter AIs capable of recognizing and preventing corner cases in smart cities.

The potential for uniting synthetic and real-world data in the pursuit of smart cities is highly promising for the future of smarter AI.

From Spiceworks

View Full Article    

Synthetic Data Can Offer ML Performance Improvements

 Previously mentioned, considering use of idea in test.

MIT News, Adam Zewe, November 3, 2022

A team of researchers at the Massachusetts Institute of Technology (MIT), the MIT-IBM Watson AI Laboratory, and Boston University found synthetic data can improve machine learning (ML) model performance. The researchers amassed the Synthetic Action Pre-training and Transfer dataset of 150,000 video clips to train three ML models on a wide range of human actions. They found the three models outperformed models trained with real video clips on four of six datasets, yielding the highest accuracy for datasets featuring clips with "low scene-object bias." MIT-IBM Watson AI Lab's Rogerio Feris said, "The ultimate goal of our research is to replace real data pretraining with synthetic data pretraining." Feris said that while there is a cost in creating an action in synthetic data, “Once that is done, then you can generate an unlimited number of images or videos by changing the pose, the lighting, etc."  ... ' 

New Book on Synthetic Data

The data is most often,  the thing.    Look forward to taking a look.

New Book: Synthetic Data

Published on November 28, 2022, By Vincent Granville

Author of Intuitive Machine Learning and Explainable AI.

84 articles

Synthetic data is used more and more to augment real-life datasets, enriching them and allowing black-box systems to correctly classify observations or predict values that are well outside of training and validation sets. In addition, it helps understand decisions made by obscure systems such as deep neural networks, contributing to the development of explainable AI. It also helps with unbalanced data, for instance in fraud detection. Finally, since synthetic data is not directly linked to real people or transactions, it offers protection against data leakage. Synthetic data also contributes to eliminating algorithm biases and privacy issues, and more generally, to increased security.

Terrain generation, evolution and morphing (in chapter 11)

This book is the culmination of years of research on the topic, by the author. Not only it integrates all the material from his previous book “Intuitive Machine Learning and explainable AI”, but it also contains all but the most advanced math from his book on stochastic simulations. The author also added more recent advances with applications to terrain generation (with animated data), synthetic universes and experimental math. The latter is an infinite source of synthetic data to build and benchmark new machine learning techniques. Conversely mathematics benefits from these techniques to uncover new insights related to the most famous unsolved math problems. Chapter 14 on the Riemann Hypothesis illustrates this point, with new state-of-the-art research results on the subject.  ... ' 

Using Synthetic Data

Synthetic data training from real data.

When It Comes to AI, Can We Ditch the Datasets?

MIT News, Adam Zewe, March 15, 2022

Massachusetts Institute of Technology (MIT) researchers have demonstrated the use of a generative machine-learning model to produce synthetic data, based on real data, to train another model for image classification. Researchers showed the generative model millions of images containing objects in a specific class, after which it learned those objects' appearance in order to generate similar objects. MIT's Ali Jahanian said generative models also learn how to transform underlying training data, and connecting a pretrained generative model to a contrastive learning model enabled both models to work together automatically. The results show that a contrastive representation learning model trained only on synthetic data can learn visual representations that rival or top those learned from real data. In analyzing how the number of samples influenced the model's performance, researchers determined that, in some cases, generating larger numbers of unique samples facilitated additional enhancements.  ... ' 

Synthetic vs Real Training Data?

The real world is messy and needs direction.

Are You Still Using Real Data to Train Your AI?

500+IEEE Spectrumby Eliza Strickland / 3d//keep unread//hide

It may be counterintuitive. But some argue that the key to training AI systems that must work in messy real-world environments, such as self-driving cars and warehouse robots, is not, in fact, real-world data. Instead, some say, synthetic data is what will unlock the true potential of AI. Synthetic data is generated instead of collected, and the consultancy Gartner has estimated that 60 percent of data used to train AI systems will be synthetic. But its use is controversial, as questions remain about whether synthetic data can accurately mirror real-world data and prepare AI systems for real-world situations.

Nvidia has embraced the synthetic data trend, and is striving to be a leader in the young industry. In November, Nvidia founder and CEO Jensen Huang announced the launch of the Omniverse Replicator, which Nvidia describes as “an engine for generating synthetic data with ground truth for training AI networks.” To find out what that means, IEEE Spectrum spoke with Rev Lebaredian, vice president of simulation technology and Omniverse engineering at Nvidia.

Rev Lebaredian on...

What Nvidia hopes to achieve with Omniverse

Why today’s real-world data isn’t good enough

Why autonomous vehicles need synthetic data

Overfitting, algorithmic bias, and adversarial attacks

The Omniverse Replicator is described as “a powerful synthetic data generation engine that produces physically simulated synthetic data for training neural networks.” Can you explain what that means, and especially what you mean by “physically simulated”?  .... ' 

Amex Uses Synthetic Data for Rare Fraud Patterns

Something we modeled even before machine learning to test 'nearness' to rare,  or even nonexistent patterns. 

Companies Beef Up AI Models with Synthetic Data  By The Wall Street Journal, July 28, 2021

Companies are building synthetic datasets when real-world data is unavailable to train artificial intelligence (AI) models to identify anomalies.

Dmitry Efimov at American Express (Amex) said researchers have spent several years researching synthetic data in order to enhance the credit-card company's AI-based fraud-detection models.  Amex is experimenting with generative adversarial networks to produce synthetic data on rare fraud patterns, which then can be applied to augment an existing dataset of fraud behaviors to improve general AI-based fraud-detection models.

Efimov said one AI model is used to generate new data, while a second model attempts to determine the data's authenticity. Efimov said early tests have demonstrated that the synthetic data improves the AI-based model's ability to identify specific types of fraud.

From The Wall Street Journal 

View Full Article - May Require Paid Subscription

Snythetic Data in Medical Research

Sounds like a bad idea, but you can use the idea of synthetic data to construct innovative models as starting points for serious research.   We did this to understand how product sold in mock stores,  then moved the models to real stores.   It often gave us real insight to the data and its use.  I remember working with Mitre years ago and even then they used the concept of mock data contexts.  Always can be a good way to think through model data design. We did not do enough of it. 

The People in This Medical Research Are Fake. The Innovations Are Real.  By Dov Lieber, April 6, 2021, The Wall Street Journal 

Medical researchers and data scientists are generating artificial patients algorithmically from real-life datasets to accelerate the development of innovations with real-world applications. Allan Tucker at the U.K.'s Brunel University London said, "The key advantage that synthetic data offers for healthcare is a large reduction in privacy risks that have bugged numerous projects [and] to open up healthcare data for the research and development of new technologies." The Covid-19 pandemic fueled demand for synthetic-data solutions as medical providers and researchers raced to understand the pathogen and develop treatments. Israel is a major testbed, using the MDClone startup's platform for creating synthetic data from medical records, for example. Not all synthetic-data research relies on real-life medical records: U.S. nonprofit Mitre's open source Synthea tool can generate populations of artificial patients from scratch, using publicly available data sources.  ... ' 

Using Synthetic Data

Worked with elements of the idea for some time the enterprise. In fact after a while we learned that its often very useful to have synthetic data alongside 'real' data,  it can be used to tease out some less than  common patterns of interest and use.  New methods make it much easier to develop.

Synthetic Data: Even Better than the Real Thing?  By Karen Emslie  Commissioned by CACM Staff  March 9, 2021

Synthetic data can be useful in any real data-based context: researchers have demonstrated the use of synthetic data in object detection, in crowd counting, in machine learning for healthcare, and even in marine science for the detection of Western rock lobsters.

Our lives are inextricably intertwined with data. It is fundamental in software development, artificial intelligence (AI) training, and product testing; it is deployed across industry, social media, and in decision-making. According to a 2020 report by market research firm International Data Corporation, "More than 59 zettabytes (ZB) of data will be created, captured, copied, and consumed in the world this year."

This is a mind-boggling amount of data, but it is not always available to those who want to make use of it. Innovators working on emerging technologies, such as autonomous vehicles, may find relevant data rare and prohibitively expensive. Access to developers is often limited, due to confidentiality.

Synthetic data, generated from simulations based on real data, has emerged as an answer. It is not a new concept, but recent developments have boosted its accuracy and usability. Add societal issues such as privacy, the General Data Protection Regulation (GDPR), and even the impact of the Covid-19 pandemic on data gathering and access, and the arguments on behalf of synthetic data appear even stronger.

Synthetic data can be useful in any real data-based context: researchers have demonstrated the use of synthetic data in object detection, in crowd counting, in machine learning for healthcare, and even in marine science for the detection of Western rock lobsters.

A group at the Massachusetts Institute of Technology (MIT) led by principal research scientist and Data-to-AI group leader Kalyan Veeramachaneni, have launched an updated set of open-source tools for producing synthetic data. The work is part of the Synthetic Data Vault (SDV), an online ecosystem that allows users to create synthetic data from their own data sources.

Veeramachaneni first experimented with synthetic data back in 2012, to tackle data access bottlenecks in an online learning platform. He realized it could also provide a solution to a problem he had encountered in industry during conversations about data access for machine learning (ML).

"All those conversations come to a grinding halt when we say, 'How can we get access to the data? For that we have to go through this process, and then what do we do next?' It takes three to six months to actually get access to the data," explained Veeramachaneni.

His group set out to build general-purpose tools that would allow anyone to create synthetic data from real data. By 2016, they had succeeded in creating statistical models using datasets from Kaggle, and sampling from those to create synthetic data.

The next step was to take a "much, much more comprehensive" approach by simultaneously creating algorithms, software, and tools that could address any enterprise data type. The result was the Synthetic Data Vault.

The researchers use three types of modelling techniques to generate synthetic data: a classic technique based on Bayesian networks, a mathematical tool from economics called Copulas, and deep learning (DL).

"Deep learning-based synthetic data generation started for images, that's where you see all those deep fakes, and there was a very popular technique called generative adversarial networks (GANs)," said Veeramachaneni.

The MIT group adapted GAN methods used on pixel-based images to work on tabular data. The trick is to generate realistic-looking data, said Veeramachaneni, but it is a fine balancing act, "You don't want it to be so real that it can actually enable you to detect some personal information about someone if it belongs to humans."

The latest tools in the SDV ecosystem support scalability, testing, and interaction with data science teams. To prove the functionality of algorithms and software, users need to come up with edge cases. As Veeramachaneni explained, "Slowly and steadily, we have seen a lot of people coming to it, using it, telling us where it's working, where it's not working, and that's essentially driving us to make it much better."

When the Covid-19 pandemic shut down MIT's Data-to-AI labs, the group spotted another use case. Sensitive data is often housed on one or two computers. Veeramachaneni said the team had to work out how keep their own machines up and running, "Then we were like, 'wouldn't it help to just have synthetic data, so that everyone can have their data on their local machine at home?'"

Privacy and access make a solid case for synthetic data use, but there are others  ... " 

On Synthetic Data

 Was asked about this, seems it has not come up for some time.   We used it to set up software and analyses for coming real data.  Some MIT thoughts.

The real promise of synthetic data   by Massachusetts Institute of Technology

After years of work, MIT's Kalyan Veeramachaneni and his collaborators recently unveiled a set of open-source data generation tools — a one-stop shop where users can get as much data as they need for their projects, in formats from tables to time series. They call it the Synthetic Data Vault. Credit: Arash Akhgari

Each year, the world generates more data than the previous year. In 2020 alone, an estimated 59 zettabytes of data will be "created, captured, copied, and consumed," according to the International Data Corporation—enough to fill about a trillion 64-gigabyte hard drives.

But just because data are proliferating doesn't mean everyone can actually use them. Companies and institutions, rightfully concerned with their users' privacy, often restrict access to datasets—sometimes within their own teams. And now that the COVID-19 pandemic has shut down labs and offices, preventing people from visiting centralized data stores, sharing information safely is even more difficult.

Without access to data, it's hard to make tools that actually work. Enter synthetic data: artificial information developers and engineers can use as a stand-in for real data.

Synthetic data is a bit like diet soda. To be effective, it has to resemble the "real thing" in certain ways. Diet soda should look, taste, and fizz like regular soda. Similarly, a synthetic dataset must have the same mathematical and statistical properties as the real-world dataset it's standing in for. "It looks like it, and has formatting like it," says Kalyan Veeramachaneni, principal investigator of the Data to AI (DAI) Lab and a principal research scientist in MIT's Laboratory for Information and Decision Systems. If it's run through a model, or used to build or test an application, it performs like that real-world data would.

But—just as diet soda should have fewer calories than the regular variety—a synthetic dataset must also differ from a real one in crucial aspects. If it's based on a real dataset, for example, it shouldn't contain or even hint at any of the information from that dataset.

Threading this needle is tricky. After years of work, Veeramachaneni and his collaborators recently unveiled a set of open-source data generation tools—a one-stop shop where users can get as much data as they need for their projects, in formats from tables to time series. They call it the Synthetic Data Vault.  .... "

Synthetic Data to Support Research

In a much less complex sense we used a method, connected to real sensors on systems, to construct sample datasets.  The resulting ML could then be used as a testbed for proposed solutions.   We were not worried in this case about the security of the data, but I can see how that could be effectively used.

Producing Data Synthetically to Help Treat Diseases Like Covid-19
Aalto University
June 25, 2020

Finnish Center for Artificial Intelligence (FCAI) researchers have developed a machine learning-based method that can synthetically generate research data and help in designing better treatments for diseases like Covid-19. The team based a newly released application on the technique, allowing academics and companies to exchange data while maintaining the privacy of individuals involved in the research. Researchers enter the original dataset into the app, which produces a synthetic dataset that can be shared securely with other scientists and companies. FCAI investigators are currently using synthetic data to construct a model that anticipates whether a subject's coronavirus test is positive or negative, based on certain biomarkers.  ... " 

Synthetic Data for Machine Learning

Synthetic data is something you use when you don't have real, sensor or measurement based data.   Its not un-common when you don't have enough real data ... It can come from simulations, forecasts, derived from metadata,  crowdsourced.   Its often used to set up initial tests... see how a model reacts to outliers, work with management, test hypotheses ....  Have never seen it used exclusively, but could imagine some cases where it might.

Will 2020 Be the Year of Synthetic Data?
by Andrew White  |  January 9, 2020  |  In the Gartner Blog

Digital is a reality.  It is evolving to accommodate and exploit AI to help automate work and drive innovation; and so the hype around AI and ML continues.  For an increasing number of organizations, leaders are beginning to ask, “Where is the beef?”  All the proof of concepts and investments need to start paying the bills.  See Toolkit: Presentation for Key Findings From the 2020 Board of Directors Survey. ... "

Synthetic Data

Companies may often have mixes of real and synthetic data,  early on we used simulations to create streams of data that were realistic for particular context. Synthetic data can also be assembled from snippets of data from other sources. Behavioral data is a good example. Good to think of a plan to make this available.

Deep learning with synthetic data will democratize the tech industry
From Evan Nisselson in TechCrunch.

" .... Synthetic data is computer-generated data that mimics real data; in other words, data that is created by a computer, not a human. Software algorithms can be designed to create realistic simulated, or “synthetic,” data.

This synthetic data then assists in teaching a computer how to react to certain situations or criteria, replacing real-world-captured training data. One of the most important aspects of real or synthetic data is to have accurate labels so computers can translate visual data to have meaning.

Since 2012, we at LDV Capital have been investing in deep technical teams that leverage computer vision, machine learning and artificial intelligence to analyze visual data across any business sector, such as healthcare, robotics, logistics, mapping, transportation, manufacturing and much more. Many startups we encounter have the “cold start” problem of not having enough quality labelled data to train their computer algorithms. A system cannot draw any inferences for users or items about which it hasn’t yet gathered sufficient information.

Startups can gather their own contextually relevant data or partner with others to gather relevant data, such as retailers for data of human shopping behaviors or hospitals for medical data. Many early-stage startups are solving their cold start problem by creating data simulators to generate contextually relevant data with quality labels in order to train their algorithms.  ... "