Netflix Digital Experiments with Content Decisions

With lots of interesting links to presentations I am looking at.   How Netflix works with its content has always interested me.   Exploring this:

Netflix at MIT CODE 2020

Netflix Technology Blog

In November, Netflix was a proud sponsor of the 2020 Conference on Digital Experimentation (CODE), hosted by the MIT Initiative on the Digital Economy. As well as providing sponsorship, Netflix data scientists were active participants, with three contributions.

Eskil Forsell and colleagues presented a poster describing Success stories from a democratized experimentation platform. Over the last few years, we’ve been Reimagining Experimentation Analysis at Netflix with an open platform that supports contributions of metrics, methods and visualizations. 

This poster, reproduced below, highlights some of the success stories we are now seeing, as data scientists across Netflix partner with our platform team to broaden the suite of methodologies we can support at scale. Ultimately, these successes support confident decision making from our experiments, and help Netflix deliver more joy to our members!   ... ' 

Automating and Optimizing Experiment Data Collection

 Collecting data from processes, and using some automatic method of choosing, pre-analysis, cleansing, visualizing and tagging, associating with metadata .... can be very useful.   Here a more complex example.

SMART Algorithm Makes Beamline Data Collection Smarter
By Lawrence Berkeley National Laboratory

The "data deluge" in scientific research stems in large part from the growing sophistication of experimental instrumentation and optimizing tools — often using machine- and deep-learning methods — to analyze increasingly large data sets. But what is equally important for improving scientific productivity is the optimization of data collection — aka "data taking" — methods.

Toward this end, Marcus Noack, a postdoctoral scholar at Lawrence Berkeley National Laboratory in the Center for Advanced Mathematics for Energy Research Applications (CAMERA), and James Sethian, director of CAMERA and Professor of Mathematics at UC Berkeley, have been working with beamline scientists at Brookhaven National Laboratory to develop and test SMART (Surrogate Model Autonomous Experiment), a mathematical method that enables autonomous experimental decision making without human interaction. A paper describing SMART and its application in experiments at Brookhaven's National Synchrotron Light Source II (NSLS-II) are described in "A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering," published in Scientific Reports.

"Modern scientific instruments are acquiring data at ever-increasing rates, leading to an exponential increase in the size of data sets," says Noack, lead author on the paper. "Taking full advantage of these acquisition rates requires corresponding advancements in the speed and efficiency not just of data analytics but also experimental control."

The goal of many experiments is to gain knowledge about the material that is studied, and scientists have a well-tested way to do this: they take a sample of the material and measure how it reacts to changes in its environment. User facilities such as Brookhaven's NSLS-II and the Center for Functional Nanomaterials offer access to high-end materials characterization tools. The associated experiments are often lengthy, and complicated procedures and measurement time is precious. A research team might only have a few days to measure their materials, so they need to make the most of each step in each measurement.  .... " 

A/B Testing for Startups

A/B Testing .... of experimentation for Startups.   Intriguing paper.

Experimentation and Startup Performance: Evidence from A/B Testing   by Rembrand Koning, Sharique Hasan, and Aaron Chatterji in HBS Working Knowledge

Is experimentation the right strategy for startups? This analysis of the adoption of A/B testing technology by 35,000 global startups provides evidence that a strategy based on repeated experimentation will improve performance over time. However, the benefits of experimentation vary. Experimentation helps younger startups “fail faster,” while older firms may discover new, high-growth products.

Author Abstract
Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing business ideas. This paper provides the first evidence of how digital experimentation affects the performance of a large sample of high-technology startups using data that tracks their growth, technology use, and product launches. We find that, despite its prominence in the business press, relatively few firms have adopted A/B testing. 

However, among those that do, we find increased performance on several critical dimensions, including page views and new product features. Furthermore, A/B testing is positively related to tail outcomes, with younger ventures failing faster and older firms being more likely to scale. Firms with experienced managers also derive more benefits from A/B testing. Our results inform the emerging literature on entrepreneurial strategy and how digitization and data-driven decision-making are shaping strategy  .... ' 

Rapid AI Experimentation with AWS

Via O'Reilly:

Rapid AI experimentation and innovation on Amazon Web Services

Dan Mbanga explores how accelerating AI experimentation has influenced innovations such as Amazon Alexa, Prime Air, and Go.

This is a keynote from the Artificial Intelligence Conference in New York 2018. See other highlights from the event.

This keynote was sponsored by Amazon Web Services.
The Artificial Intelligence Conference in San Francisco, September 4-7, 2018 ... "

Robots Do their own Experiments

There are robots as we typically interpret that term.  There is also robotic process automation (RPA).  Which does not have to look anything like a 'robot'.   And further just automation, such as in a manufacturing line, which in modern times is run by process automation.     All influence the jobs we do, and how we integrate human labor.

These Robots Are Learning to Conduct Their Own Science Experiments
Carnegie Mellon professors plan to gradually outsource their chemical work to AI.  by Jeff Wise in Bloomberg.

Inside a lab at Carnegie Mellon University in Pittsburgh, a robot arm lifts a bottle filled with chemical reagents and carries it over a bank of test tubes, where it dispenses a precise number of drops into each one. The arm swivels, replaces the bottle, swivels again, and picks up another container. Gracelessly, tirelessly, the machine thrums on, carrying out test after test. The experiments are part of an ongoing project to determine the ideal chemical makeup for high-capacity electric car batteries. Soon, machines won’t just run the experiments—they’ll devise them, too.

Over the next few months, an artificial intelligence algorithm will gradually take over the planning of experiments based on the battery test runs. Once fully functioning, this robot graduate student will decide how to modify the concentrations of the ingredients it’s testing. “It’s automating not only the manual part of doing the experiment but also the planning part,” says Brian Storey, the Toyota Research Institute scientist leading the project.

Science has long been considered one of the human activities least likely to be farmed out to robots. That’s changing as sensors, sequencers, and satellites churn out digital information by the terabyte. “We just cannot handle the amount of data anymore,” says Manuela Veloso, who heads Carnegie Mellon’s machine learning department. It’s a daily concern for biotech companies and a wide range of other businesses struggling to make sense of the unprecedented swell of raw information.

AI software designed to identify and sort patterns has been deployed across a wide swath of science, from marine biology (identifying wild dolphin vocalizations from hydrophone recordings) to astronomy (detecting the presence of planets from subtle fluctuations in the brightness of thousands of stars). To discover the Higgs boson, the so-called God particle, an algorithm sifted billions of particle tracks generated within the Large Hadron Collider in Switzerland. AI is fast becoming an essential part of university science curricula.

Automating the process of discovery doesn’t just free up researchers’ time. It could potentially change what sorts of discoveries are made. “I can easily imagine cases in which AI would recommend experiments to try to synthesize a chemical molecule that you wouldn’t think possible, but the AI will be able to do it,” says Barnabás Póczos, a Carnegie Mellon machine learning professor collaborating on the Toyota project.  .... "

Vetting Design with Rapid Experimentation

Reminds me of the idea of develop and test.   A kind of brute force of examination of options.  Always a good way to start thinking about a problem, but does depend on how quickly you can create designs and then test them.

Julie Stanford on vetting designs through rapid experimentation

The O’Reilly Design Podcast: Quickly test ideas like a design thinker.   By Nikki McDonald 

In this week’s Design Podcast, I sit down with Julie Stanford, founder and principal of user experience agency Sliced Bread Design. We talk about how to get in the rapid experimentation mindset, the design thinking process, and how to get started with rapid experimentation at your company. Hint: start small.  .... " 

Chaos Engineering in Practice

Don't remember ever hearing of Chaos Engineering, but had done some engineering in Chaotic, non predictable situations.  It should be noted that this is not about chaos theory, in a mathematical sense, but rather testing and adjusting complex systems.   Nora Jones in InfoQ describes it in practice.

Free 81 page Book on the topic via O'Reilly,

Where they describe it and its development and use by Netflix:

Building Confidence in System Behavior Through Experiments.

" .... With so many interacting components, the number of things that can go wrong in a distributed system is enormous. You’ll never be able to prevent all possible failure modes, but you can identify many of the weaknesses in your system before they’re triggered by these events. This report introduces you to Chaos Engineering, a method of experimenting on infrastructure that lets you expose weaknesses before they become a real problem.

Members of the Netflix team that developed Chaos Engineering explain how to apply these principles to your own system. By introducing controlled experiments, you’ll learn how emergent behavior from component interactions can cause your system to drift into an unsafe, chaotic state.  .... "

How might this be integrated with forms of process modeling.  like BPM?  Could the testing be applied to a process model?

Running A/B Experiments

On the risks of A/B testing.    Remember the process well.  A podcast:  " ....  Andrea Burbank discusses the risks, benefits, and lessons from running a single huge experiment with hundreds of moving parts, and with long-term engagement as the metric of success. ... " 

Graph Analytics and Gephi

In Forbes, an article on Graph Analytics.    Good overview,  it will become an important Bigdata concept, but no mention of Gephi,  an Open Graph Viz Platform.   Its free for experimentation.  It is not the same thing that Forbes describes, but is an excellent starting point.  A means to produce and visualize very complex, data based graph structures. Have used it for supply chain and physical system problems and mentioned it a number of times in this blog.

(Brought to my attention by @KirkDBorne)

Brain Marketing

In the HBR:  Using the activity of the brain to understand purchasing and marketing decisions.   We called this neuromarketing, but that term is never used in the article.   " ... Brain Marketing: Is the Product Worth the Price?  Are consumers more likely to buy if they see the price before the product, or vice versa? Uma Karmarkar and colleagues scan the brains of shoppers to find out. ... " .  

The approaches described uses fMRI, while today most experimentation by retailers and manufacturers use very different electroencephalogram methods.

" ... By the time you decided to buy a product, you knew both what you were buying and how much it cost. But was your decision affected by whether you saw the price or the product first? That's the question at the heart of new experimental research that uses neuroscience tools to shed light on how our brains make purchasing decisions. ... " 

Cheap Hack-able Roomba

In Mashable: " ... iRobot builds a vacuum-free Roomba for students to hack ... "     This is a successor to the iRobot Create, which we used to experiment with simple robotic demonstration ideas. This version has no vacuum, so its simpler, cheaper.  Can be hacked for experimentation with algorithms in Python, for which there is a experimenters community.

C-Level Execs and Big Data

A natural next step.  The key will be how to make the translation from the analyst speak to the business strategy and process language.   " ... An overwhelming majority of executives from companies that have brought big data projects to production are pleased with the results, according to new research by Accenture Digital. However, they also cite security as the major challenge they face.

"One thing we've seen in the past year is that people are not really talking about what is big data anymore," says Vince Dell'Anno, managing director and global information management lead, Accenture Analytics, part of Accenture Digital. "They've done their experimentation. They've got some understanding of it. What they're talking about now is what happens when they need to scale to support, 1,000 or 2,000 or 5,000 users. That's where challenges like security and integration come into play." ... " 

The Reciprocity Advantage

Reading The Reciprocity Advantage: A New Way to Partner for Innovation and Growth.  by Bob Johansen and Karl Ronn.  Nicely done with real life enterprise examples, including IBM, TED, Microsoft, Google Apple Store and more.

I like in particular an outline about how each company partnered and what the partnership exchanged.  Also what assets where involved.   Then  the scaling advantage involved.  I also like the emphasis on experimentation being done to figure out what the real value is and how it will be developed.   Included in that is how 'gameful engagement' can be used to figure out how the partnership will work.   The latter parts of the book provide direct  'How to' steps.  Well done.

" ... Bob Johansen, best-selling author and Distinguished Fellow at IFTF, has partnered with innovation expert Karl Ronn to uncover what they believe will be the biggest innovation opportunity in history in their forthcoming book, The Reciprocity Advantage. Through illustrative examples from leading organizations and an eye to the future, Johansen and Ronn present a four-part model that shows how organizations can give away assets to form partnerships, leading to collaborative experimentation and scalable new business opportunities.  ... 

" ... Johansen and Ronn explain that reciprocity, an old concept, will be reimagined within a future context of connective technologies, generational shifts, and new organizational models to provide a distinct competitive advantage. To win in this world, businesses will need to  (1) uncover their right-of-way, the underutilized resources they already own that they can share with others; (2) find partners who can help them do what they cannot do alone; (3) experiment to learn through cloud-based systems; and (4) scale the results, but only when the business’ reciprocity advantage is desirable, viable, and ownable. If it’s not scalable, it’s not worth doing—and the authors describe technologies that make scaling up faster than ever. .... " 

More and ordering information.   Video.   @Reciprocityadv

Tom Peters on 21st Century Organization

In McKinsey:   An interview that takes some interesting directions: ' ...  " ... Well, one answer to that, as far as I’m concerned, is “I don’t know.” My real bottom-line hypothesis is that nobody has a sweet clue what they’re doing. Therefore you better be trying stuff at an insanely rapid pace. You want to be screwing around with nearly everything. Relentless experimentation was probably important in the 1970s—now it’s do or die. It takes a certain confidence, though. The first partner I worked for at McKinsey had the self-assurance to look a chief executive officer in the eye and say, “We don’t know what the hell’s going on. Can we play with this together?” .. ' 

Kroger Looks to Digital for Shopper Experience

Another example of increasing use of data to understand and anticipate shopper experience.

Kroger looks to digital to improve shopping experience 
Kroger's recent acquisitions of Harris Teeter and Vitacost.com are part of its experimentation with improving the shopping experience through offering more digital options to help customers save time and money, according to the retailer's Director of Digital and E-Commerce Matt Thompson. Currently, about 20% of shoppers prepare for a grocery trip online, and 60% of all consumers consider grocery shopping their least favorite chore, he said  ... " 

Reciprocity Advantage: Innovation and Organization

New book by my two former colleagues: The Reciprocity Advantage.

" ... Bob Johansen, best-selling author and Distinguished Fellow at IFTF, has partnered with innovation expert Karl Ronn to uncover what they believe will be the biggest innovation opportunity in history in their forthcoming book, The Reciprocity Advantage. Through illustrative examples from leading organizations and an eye to the future, Johansen and Ronn present a four-part model that shows how organizations can give away assets to form partnerships, leading to collaborative experimentation and scalable new business opportunities.  ... 

" ... Johansen and Ronn explain that reciprocity, an old concept, will be reimagined within a future context of connective technologies, generational shifts, and new organizational models to provide a distinct competitive advantage. To win in this world, businesses will need to  (1) uncover their right-of-way, the underutilized resources they already own that they can share with others; (2) find partners who can help them do what they cannot do alone; (3) experiment to learn through cloud-based systems; and (4) scale the results, but only when the business’ reciprocity advantage is desirable, viable, and ownable. If it’s not scalable, it’s not worth doing—and the authors describe technologies that make scaling up faster than ever. .... " 

More and pre-order information.   Video.   I plan to read and comment here later.

Dark Stores for Fulfillment

Had not heard the term 'dark stores' for some time, but experimented with the concept some years ago. It also allows experimentation with the structure of the store as well.     Then becomes a hybrid of store and warehouse.

In Retailwire: " ... Although online grocery shopping hasn't yet taken off in the U.S., it's quite popular in the U.K. Orders are fulfilled at a nearby store and delivered by truck. Pickers are given a large cart with separate bins for separate orders, and they use a tablet to efficiently navigate the store. With enough orders, you can imagine those customers slowing down the pickers. The grocery store layout isn't really conducive to both types of foot-traffic. Thus the dark store was born.

Just as you might expect, the dark store has no customers and is used strictly for picking and fulfillment. Its location and layout are similar to traditional stores, but there are no price tags, no endcap advertising, and no checkout lines. It's a neighborhood warehouse, complete with fresh, frozen and dry goods.   ... "  

Teradata's Data Lab

It is very useful to have a lab to experiment with your data.  Here is Teradata's approach.
 " ... Promote Exploration & Experimentation
A Teradata Data Lab lets you explore and examine new ideas by combining new data with existing data so its easy to identify new trends and insight or react to immediate business issues. ...  " 

Visualizing Data from High Performance Computing

Simulation creates a great deal of data, so it naturally links well with visualization   Every industrial simulation we wrote for the enterprise was linked to a visualization to understand the subtleties of parameter changes and experimentation.    In Computer Org:   Largely non technical coverage.

" ... For computational simulations, the era of "big data" ended before it began. We're actually living in the era of infinite data — in which the stream pouring forth from computational models and simulations can be as voluminous as outputing every value at every timestep, drowning disks and researchers alike with high-cadence, arbitrarily large datasets. Rather than struggling to make models bigger, the challenge is now to keep them under control.  ... " 

Links to a number of other articles and resources.  See link below for more coverage in this blog on HPC.

My Updated One Page Bio

I have updated my one page bio, it continues after the paragraphs below.  Feel free to contact me about consulting, strategic and tactical.

Franz A. Dill

Franz Dill’s academic background is in astrophysics and mathematics, with degrees from the University of Pennsylvania and the University of Florida.  He worked developing tank battle simulations in the 1970s for the office of the Joint Chiefs at the Pentagon, where he won a Defense Department scholarship for information Technology.  

For thirty years he worked for the Procter & Gamble Company.  At P&G he reported to the CIO as part of an information technology research organization which scouted and vetted emergent technology use.   He worked directly with the CEO developing their first executive information systems.  This led directly to P&G’s Business Sphere data focused experience.  He led the company in the application of modeling to enterprise supply chains.   He managed work with many academic and vendor partners.   .....