Back to Synthetic Data

Have not talked this for sometime, still interesting.

What is Synthetic Data? The Good, the Bad, and the Ugly

Sharing data can often enable compelling applications and analytics. However, more often than not, valuable datasets contain information of sensitive nature, and thus sharing them can endanger the privacy of users and organizations.

A possible alternative gaining momentum in the research community is to share synthetic data instead. The idea is to release artificially generated datasets that resemble the actual data — more precisely, having similar statistical properties.

So how do you generate synthetic data? What is that useful for? What are the benefits and the risks? What are the fundamental limitations and the open research questions that remain unanswered?

All right, let’s go!

How To Safely Release Data?

Before discussing synthetic data, let’s first consider the “alternatives.”

Anonymization: Theoretically, one could remove personally identifiable information before sharing it. However, in practice, anonymization fails to provide realistic privacy guarantees because a malevolent actor often has auxiliary information that allows them to re-identify anonymized data. For example, when Netflix de-identified movie rankings (as part of a challenge seeking better recommendation systems), Arvind Narayanan and Vitaly Shmatikov de-anonymized a large chunk by cross-referencing them with public information on IMDb.

Aggregation Another approach is to share aggregate statistics about a dataset. For example, telcos can provide statistics about how many people are in some specific locations at a given time — e.g., to assess footfall and decide where one should open a new store. However, this is often ineffective too, as the aggregates can still help an adversary learn something about specific individuals.

Differential Privacy: More promising attempts come from providing access to statistics obtained from the data while adding noise to the queries’ response, guaranteeing differential privacy. However, this approach generally lowers the dataset’s utility, especially on high-dimensional data. Additionally, allowing unlimited non-trivial queries on a dataset can reveal the whole dataset, so this approach needs to keep track of the privacy budget over time.   ... '

McKinsey on the Frontier of Artificial Intelligence

An 80 page non-technical overview of AI today.  Perhaps not enough detail, but a useful exec view of where we are, where we are going , and some reasonable domain specific descriptions.

Mckinsey Global Institute
Artificial Intelligence, the Next Global Frontier

 Since its founding in 1990, the McKinsey Global Institute (MGI) has sought to develop a deeper understanding of the evolving global economy. As the business and economics research arm of McKinsey & Company, MGI aims to provide leaders in the commercial, public, and social sectors with the facts and insights on which to base management and policy decisions. The Lauder Institute at the University of Pennsylvania ranked MGI the world’s number-one private-sector think tank in its 2016 Global Think Tank Index for the second consecutive year.

MGI research combines the disciplines of economics and management, employing the analytical tools of economics with the insights of businessleaders. Our “micro-to-macro” methodology examines microeconomic industry trends to better understand the broad macroeconomic forces affecting business strategy and public policy. MGI’s in-depth reports have covered more than 20 countries and 30 industries. Current research focuses on six themes: productivity and growth, natural resources, labor markets, the evolution of global financial markets, the economic impact of technology and innovation, and urbanization.

Recent reports have assessed the economic benefits of tackling gender inequality, a new era of global competition, Chinese innovation, and digital globalization. MGI is led by four McKinsey and Company senior partners: Jacques Bughin, James Manyika, Jonathan Woetzel, and Frank Mattern, MGI’s chairman. Michael Chui, Susan Lund, Anu Madgavkar, Sree Ramaswamy, and Jaana Remes serve as MGI partners. Project teams are led by the MGI partners and a group of senior fellows and include consultants from McKinsey offices around the world. These teams draw on McKinsey’s global network of partners and industry and management experts. Input is provided by the MGI Council, which coleads projects and provides guidance; members are Andres Cadena, Sandrine Devillard, Richard Dobbs, Katy George, Rajat Gupta, Eric Hazan, Eric Labaye, Acha Leke, Scott Nyquist, Gary Pinkus, Oliver Tonby, and Eckart Windhagen. In addition, leading economists, including Nobel laureates, act as research advisers.

McKinsey & Company is a member of the Partnership on AI, a collection of companies and non-profits that have committed to sharing best practices and communicating openly about the benefits and risks of artificial intelligence research. The partners of McKinsey fund MGI’s research; it is not commissioned by any business, government, or other institution. For further information about MGI and to download reports, please visit   www.mckinsey.com/mgi. 

AI Report Card from the Verge

Nicely done, non-technical view of the condition of AI circa 2017.   Covering many points I would also make.  AI is not only about machine learning,  a solution to difficult statistical pattern recognition problems, it is about how well we can deliver intelligent cognitive task solutions to specific contexts.  Like the home, industry, vehicles, supply chains, retail, healthcare and others.   In other words, how can AI systems do what humans can do?

The Verge overview covers many things I have covered here, often quoting their reporting  Also what I have done in a number of consulting projects over many years.   Great progress, but still a long way to go.

Power BI no Tableau yet

A friend is continuing to take a closer look at Microsoft's Power BI, and I am following.  A means for doing simple data visualization starting with Excel tables.  Good impressions so far, continuing to evolve.  Worth a look,  An overview comparision.