Delta's Digital Black Swan

In MIT Sloan:  A reminder of the problems of building centralized systems.  And a good definition of the Black Swan:   " ...   Taleb wrote that black-swan events have three characteristics: “rarity, extreme impact, and retrospective (but not prospective) predictability.” .... I don’t know if the power failure at Delta — and the chain of unexpected events that followed it — qualifies as a black swan by Taleb’s standards, but it must have felt that way to CEO Ed Bastian. The day after the failure, he apologized for the second time and ruefully explained that over the past three years, Delta has invested “hundreds of millions of dollars in technology infrastructure upgrades and systems, including back-up systems, to prevent what happened yesterday from occurring.”   ... " 

(Registration required)

Black Swans and the Price of Oil

All systems that predict future states need to predict future related context.

Black swans and barrels: How to think about the future of oil prices
No one can be sure where oil prices are headed. McKinsey’s Scott Nyquist answers the questions business leaders are asking about the state of the market. .... " 

Changes in Managing Risk

Noting the need to connect risk directly to relevant decision process.

How Managing Risk Has Changed    (Podcast and Transcript)
The problem with many catastrophic risks isn’t just that their impacts, when they hit, are so massive. It’s also that their odds of occurring in any given short time frame are very small, so that planning for them has to be handled as a long-term priority while the proverbial sun is shining. And neither companies nor individuals are particularly apt at taking serious, long-term action to prepare for low probability, high consequence events.

Enter the Wharton Risk Management and Decision Processes Center, which was created 30 years ago to help individuals, businesses, governments and global organizations to be better prepared for those longer range, more unpredictable dangers.

Knowledge@Wharton spoke with Howard Kunreuther and Robert Meyer, co-directors of the Wharton Risk Management and Decision Processes Center, and executive director Erwann Michel-Kerjan about the center’s research and how managing risk has changed over the past few decades. ... " 

Considering Risk Assessments

In CWorld: Good short basic piece on basic risk assessments. Good place to start.  I would further get a professional to do the analysis that really knows the domains involved.   Start with visual methods to display the risk metrics.  Double and triple check your numbers.  You may get many numbers that vary wildly, so show them to multiple experts.  Create a risk portfolio with estimates of costs to address each risk.  Keep a log of near misses and past issues.    Do list 'black swans' and what their effects might be.

Unintended Consequence Algorithms

This is not uncommon.  And things are getting more and more complex. Many more variables.   Black boxes are out there.  But be ready to explain how your box works, or show its track record compared to the current process, to those that make the decisions.   Putting these algorithms in charge will require better risk management.

In the Atlantic: Not Even the People Who Write Algorithms Really Know How They Work ...  The web's information filters are making assumptions about you based on details that you might not even notice yourself.  ...  " 

Big Data Ducklings

In Teradata Mag:   " ... Ugly Duckling or Black Swan?  Big Data gives businesses the periheral vision to detect and respond to catastropic events before it is too late. ... " .  Thoughtful piece,  but I define black swans as those that cannot be completely predicted.  So this is more like risk management, which has its own literature.  If I build a portfolio of risks, and can define potential occurrences, then Black Swans are very low probability possibilities.  How low?  Completely unanticipated?   Events that I do not have lots of data about, are by definition not 'big' data.  Yes I know it is not all about 'big', which is part of the reason I do not like the term.  So lets just call it data analytics under unusual contexts.

GE Predix Cloud Predicts Machine Failures

We worked on a similar project in collaboration with Los Alamos Labs. In particular to do reliability and failure prediction for systems that were composed of elements that only rarely failed.   So called 'Black Swans'.    This was eventually licensed out via a third party.  Examining the difference between the ideas.   Will report back here with more information.  A related project looked at out of stock condition on a store shelf as a failure that could be predicted by multiple sensory inputs.

Pointer to some of the above work in R&D Magazine.
This was eventually offered for use by KPMG Consulting. in 2005.

In FastCompany:
GE wants to give industrial machines their own social network with Predix Cloud ... GE is selling a new service that promises to predict when a machine will break down, so technicians can preemptively fix it. .... "

Black Swans and Big Data

In TeraData Mag:  Good piece that looks at the inevitability of these events, and how Data can still help. Leads to the suggestion that predictive analytics will allow us to be prepared for classes of Black Swans.  Segmenting appropriate responses.  And then, obviously addressing them with the same patterns:   

" ....  A data-driven analysis or simulation designed to determine an organization’s ability to deal with a crisis situation can help it be ready for the day a black swan lands on its front steps. Krishna notes that this type of “stress testing” can gauge the level of readiness. “[Preparation] is really all about imagining the unimaginable, understanding what’s going to blow up ... [and] to be able to determine what corrective actions are needed,” he explains. “That is becoming very much an accepted approach ... certainly something that, from a regulatory standpoint, is becoming mandatory for a number of financial institutions.”

Businesses that identify a possible event early on and take evasive actions are usually in the best position to ride out the crisis. Krishna advises companies to carefully map out the exact steps they will need to take in various types of crisis situations. “Then document those actions so that if any of the expected scenarios occur, there will be no second-guessing,” he points out. “It’s simply a matter of executing what’s already been documented; executing that game plan, if you will.” ... ' 

Black Swans Circling?

In the HBR Blog.  Provocative piece on the number of changes occurring that could influence the near term future of P&G.   And many other consumer goods companies.   I take issue to some degree with the term being used ... A black swan is a metaphor for a unique surprise with potential major effect.  None of the examples given are surprises.  P&G has know for some time that they will be very important, and I agree.   Its not only the existence of these issues, but the degree to which they will exist that is key. As the article says the co-occurrence of even a few of these things could be disastrous.  How can we diminish or prevent their influence?  Article is a very good check list of cautions.

Taleb on Accepting Uncertainty and Volatility

Nassim Nicholas Taleb reviewing his new book on Accepting Uncertainty, Embracing Volatility  in  Knowledge@Wharton.    Over the years I have talked to many vendors that believed that they could directly solve the uncertainty problem, despite the obvious volatility of the data about world we live in.  Once again Taleb does an excellent job of describing this space well.  I am still looking for something I can use directly to better simulate the volatile world we live in.   " ... The day before a big game, regardless of the sport, a team's coach or star player is often asked, "How will you stop the opposing team tomorrow?" The answer typically goes something like this: "We can't worry about the other team. We just have to play our game." That, in a very simplified nutshell, is the essence of Nassim Nicholas Taleb's highly polemical, always thought-provoking new book, Antifragile: Things That Gain from Disorder. Here, though, the opponent is not another team's slugger, quarterback or point guard, but the future and change ... " 

Econophysicists and the Swans

Ultrafast Trades Trigger Black Swan Events Every Day, Say Econophysicists The US financial markets have suffered over 18,000 extreme price changes caused by ultrafast trading, according to a new study of market data between 2006 and 2011  .... "

Messy Analytics

The conclusion of a three part series on practical aspects of using analytical methods by Frank Buytendijk.   I had missed the first two parts, but now plan to go back and read them.  " ... First, when you do statistical analysis, resist the temptation to remove the outliers. Improbable scores or data are usually filtered out of the dataset because it is noise "messing up" the model. However, the outliers might actually represent the most interesting bits. They could be the early warning signal for a black swan coming or could represent new business opportunities that others – following best practices –neatly filter out. If the model is your lens, you won't see any change coming. You won't get any weird new ideas. What you see is what you've always seen. All the model does is confirm your hypothesis. Outliers deserve extra attention.  ... "

Black Swans

A conversation with Nassim Taleb on living with Black Swans. In KnowlegeAtWharton.   And more about black swan theory.

The Black Swan

Have read Nassim Nicholas Taleb's book: The Black Swan: The Impact of the Highly Improbable. Highly recommended for modelers or those who think about their application.

Taleb, once a very successful derivative and options trader, now a professor at the University of Massachusetts, takes you on a wild and often idiosyncratic ride through financial modeling. No equations in the book, but it helps to have some basic statistics and econometric background. As close to a page-turner as a book like this can be.

Taleb writes about what he considers the total inadequacy of currently used modeling methods. This is mostly a full-steam attack on the mis-use of Gaussian methods (the Normal or Bell curve), which are the basis of modern portfolio theory, forecasting, regression and just about any statistical method that claims to be predictive. The Gaussians' small tails make it incapable of modeling anything even close to improbable. Our connected world is getting more improbable, thus these methods are increasingly wrong.

To be clear, it's not that the use of Gaussian methods are always wrong, though Taleb's style sometimes implies that. He is making the case that they have been used to underpin all of financial modeling methods, even when it makes no sense.

Along the way, Taleb tells a personal story (very unusual for a statistics book!) and trashes modern portfolio theory, Black-Scholes, Wharton, the Nobel Prize Committee, the use of narrative and most of the last twenty years of econometrics. He has received threats from the normally staid econometric community. He suggests that that Mandelbrot's scalable fractal methods are a better approach than Gaussian methods.

He says that modern practitioners of financial methods agree with him, though most academics do not. The formal methods do not work. Evidenced by the 1998 LTCM crisis, which came close to bringing down the entire global financial system. The formal methods could not deal with a 'black swan', a very improbable event by normal distribution standards.

The practitioners respond that the current portfolio risk methods are all they have today to create useful models. Mandelbrot's models do not give the predictions that current formal methods do. But if the predictions are wrong?

This is a big deal to large companie who use many techniques like forecasting, marketing mix and simulation models that are based on what Taleb is saying are flawed fundamentals in a world of increasing improbabilities. Also, WSJ Review.