Superforecasting Update

For some time have followed GoodJudgment  and their 'superforecasting' services.   Most recently got a new overview with considerable efforts and claims.    Since very accurate forecasting would seem to be fundamentally unbeatable in many domains, why believe it?  May depend on the definition of a correct forecast in a given context.  See their work with returns to normality with Covid.  And 'Right ways to think about the future'.

Why Superforecasting?

When you make decisions based on precise probability forecasts, rather than hunches, the benefits are game-changing.

Gain a competitive edge

Making decisions without precise probability forecasts is like playing poker without counting cards. Outsmart your competition with well-calibrated forecasts.

Manage risk

Convert strategic uncertainty into manageable risk to avoid costly mistakes.

Seize opportunities

Identify expected outcomes before others recognize them so that you can capitalize on opportunities quickly and with confidence.  From groundbreaking theory to powerhouse practice

In 2011, IARPA – the US intelligence community’s equivalent to DARPA – launched a massive competition to identify cutting-edge methods to forecast geopolitical events. Four years, 500 questions, and over a million forecasts later, the Good Judgment Project (GJP) – led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania – emerged as the undisputed victor in the tournament. GJP’s forecasts were so accurate that they even outperformed intelligence analysts with access to classified data.

Good Judgment Inc is now making this winning approach to harnessing the wisdom of the crowd available for commercial use. Our clients benefit from the externally validated forecasting methodology that made the Good Judgment Project so successful.

Today, Good Judgment’s professional Superforecasters deliver unparalleled accuracy on forecasting questions across the political, economic and social spectrum. And, we train others to apply this evidence-based methodology within their own teams.  .... '

Superforecasting Short Course from The Edge

Quite interesting, free, have followed the concept for some time, but have yet to apply it.  Videos embedded at the link.   Good to see this followup.  Is enough said about embedded risk analysis?  Following.

A Short Course in Superforecasting—Philip Tetlock: An EDGE Master Class
(ED. NOTE: In 2015, Edge presented "A Short Course in Superforecasting" with political and social scientist Philip Tetlock. Superforecasting is back in the news this week thanks to the UK news coverage of comments by Boris Johnson's chief adviser Dominic Cummings, who urged journalists to "read Philip Tetlock's Superforecasters [sic], instead of political pundits who don't know what they're talking about.")

PHILIP E. TETLOCK, political and social scientist, is the Annenberg University Professor at the University of Pennsylvania, with appointments in Wharton, psychology and political science. He is co-leader of the Good Judgment Project, a multi-year forecasting study, author of Expert Political Judgment, co-author of Counterfactual Thought Experiments in World Politics (with Aaron Belkin), and co-author of Superforecasting: The Art & Science of Prediction (with Dan Gardner). Further reading on Edge: "How to Win at Forecasting: A Conversation with Philip Tetlock" (December 6, 2012). Philip Tetlock's Edge Bio Page.

CLASS I — Forecasting Tournaments: What We Discover When We Start Scoring Accuracy
It is as though high status pundits have learned a valuable survival skill, and that survival skill is they've mastered the art of appearing to go out on a limb without actually going out on a limb. They say dramatic things but there are vague verbiage quantifiers connected to the dramatic things. It sounds as though they're saying something very compelling and riveting. There's a scenario that's been conjured up in your mind of something either very good or very bad. It's vivid, easily imaginable.

It turns out, on close inspection they're not really saying that's going to happen. They're not specifying the conditions, or a time frame, or likelihood, so there's no way of assessing accuracy. You could say these pundits are just doing what a rational pundit would do because they know that they live in a somewhat stochastic world. They know that it's a world that frequently is going to throw off surprises at them, so to maintain their credibility with their community of co-believers they need to be vague. It's an essential survival skill. There is some considerable truth to that, and forecasting tournaments are a very different way of proceeding. Forecasting tournaments require people to attach explicit probabilities to well-defined outcomes in well-defined time frames so you can keep score.   ... " 

Good Judgment Formalizes Superforecasting

Been following the idea of 'superforecasting' for some time.   All business is dependent on accurate forecasts.    If they are analytical-numerical or the hunch of decision makers, or crowdsourced consumer opinions.   Or as more common a combination of these.

Good Judgment names Superforecaster® Warren Hatch as its new CEO

New York, 1 October – Good Judgment is pleased to announce that Warren Hatch has been named chief executive officer, effective immediately. Dr. Hatch was previously President of the firm. As CEO, Hatch will direct Good Judgment’s strategy and oversee its global team.

“Warren brings the strengths of a Superforecaster to the role of Good Judgment CEO – he’s smart, analytical, open-minded, and determined to do whatever it takes to get the job done right,” said Phil Tetlock, Good Judgment co-founder, Wharton professor, and author of Superforecasting: The Art and Science of Prediction. “Warren’s academic credentials in international politics and history combined with his Wall Street experience uniquely position him to lead Good Judgment effectively. I look forward to working with him and our colleagues, both internal and external, as Good Judgment continues to develop innovative approaches for making sense of an uncertain world so that our clients can make better decisions.”  .... '

Superforecasting Update

Just received their latest newsletter, of interest:

Goodjudgment:  How Can Superforecasting     Improve your Decisions?

Good Judgment’s co-founder Philip Tetlock literally wrote the book on state-of-the-art crowd-sourced forecasting.

Now, the training, techniques, and talent that helped the Good Judgment Project win a massive government-sponsored forecasting competition can help your organization manage strategic uncertainty. .... " 

In this newsletter, we share the Wall Street Journal's view on Superforecasting, an update from our partnership with Eurasia Group, Superforecasters view on EU Article 7 sanctions, Superforecaster Frederic Bush's article in Two Plus Two on Superforecasting in Poker, and reveal the GJ Open crowd's tip for the World Cup winner.  .... " 

See also, IARPA's sponsorship of work in superforecasting.

Forecasting Challenge

I had mentioned superforcasting  again, was brought to my attention that they have had a hybrid competition under way for a while.  Are looking for volunteers.  Worth a look.

Superforecasting for Better Prediction

I was reminded today again of the need for the best forecasting possible when doing many kinds of analytics.     I have covered the concept of Superforecasting here a number of times.   Basically the idea that there are people out there who have a style of prediction that is significantly better than others, and can be used to provide this value.  

See  the book  “Superforecasting: The Art and Science of Prediction,” by Philip E. Tetlock and Dan Gardner on this, and my tag below. Also their frequently updated Good Judgement blog, which contains updated news and challenges.   They are even looking for volunteer forecasters there to compete.

A must read.   I will be following more closely and reporting on items from their blog now.  I taught forecasting, and this rings very true in value, beyond the usual analytic methods, or even Machine Learning or Big Data,

Superforecasting and Decision Makers

This is such an important issue.  So many claims are made for good forecasting,  but it is only as good as it is integrated into good decision making.   Forecasting is an analytic method,  but of little use if it can't contribute to value and less risk in decisions.  And as the authors suggest, forecasts are always 'wrong'.  Which calls for a modification of contextual decisions, based on required operations.   In Knowledge@Wharton:

Are You a Superforecaster? What Good Decision-makers Have in Common
 Wharton's Barbara Mellers and Michael Platt discuss their research on superforecasters.

While nobody can get forecasts right 100% of the time, research shows that there are certain kinds of people who are better at forecasting outcomes than others. Wharton marketing professors Barbara Mellers and Michael Platt, who are also Penn Integrates Knowledge (PIK) professors at the University of Pennsylvania, examine the intersection of marketing, psychology and neuroscience to understand the traits that “superforecasters” share and that can lead to better decision making. They recently talked about their research and its implications on the Knowledge@Wharton show on Wharton Business Radio on SiriusXM channel 111. (Listen to the podcast at the top of this page.)

Podcast, and an edited transcript of the conversation follows.  .... "

Philip Tetlock and Super Forecasting

Recently covered in the BBC with an introductory article and broadcast.    About the gradual improvement of forecasting and the use of tournaments to drive them.  This topic has been covered here for some time.    Precision forecasting is fundamental for strategic and tactical organization planning. " ... Most intelligence agencies rely on the informed hunches of trained specialists. But there are people from ordinary walks of life who routinely do far better at predicting events than the experts. Tara Isabella Burton meets some of them. ... " 

Superforecasting and Straight Thinking

In this week's NYT book review, a piece on  “Superforecasting: The Art and Science of Prediction,” by the psychologist Philip E. Tetlock.   I have mentioned the work here a number of times.   This overview does a good job presenting key elements.   Also a review of “Mindware: Tools for Smart Thinking,” by the psychologist Richard E. Nisbett.  I have not read the latter,  but based on the review it is a look at principles of thinking that are well known, but could serve as an introduction for students at all ages.

More on Superforecasting

Previously mentioned the concept of superforecasting, see tag below.  Here more about the book and idea.   Podcast and text in K@W:

Why an Open Mind Is Key to Making Better Predictions
 .... So what makes a good forecaster? In the new book, Superforecasting: The Art and Science of Prediction, Wharton management professor Philip Tetlock and co-author Dan Gardner look into what makes people good forecasters. Tetlock, who is also a professor of psychology at Penn’s School of Arts and Sciences, recently spoke to Knowledge@Wharton about his decades of research on the topic and how you can incorporate some of these forecasting techniques into your own life. .... "
 

Superforecasting with Tournaments

I m a student of unusual markets and forecasts,  and this forecasting approach with  tournaments is described in some detail in a recent Edge article.  Quite an eminent science group of participants in the project are commenting.    Note mention of the Good Judgement Project.  Interesting connection to crowd sourcing methods.

" .... When IARPA originally launched this project, they thought that beating the unweighted average of the crowd by 20 percent would be an ambitious goal in year one, 30 percent in year two, 40 percent in year three, and 50 percent in year four. The Good Judgment Project, for reasons that are interesting, was able to beat IARPA's fourth year benchmark in the first year and in all subsequent years. For reasons that are also maybe a little less interesting, other teams were not. I say the reasons are less interesting, I don’t think it was due to them not having the right research expertise. There were issues of mismanagement, of how they went about it. We had a way better project manager.

Putting that to the side, the Good Judgment Project was able to do far better than IARPA or any of the other researchers who were consulted on the design of the project thought possible. We were able to knock out some pretty formidable competitors. Slide twenty-nine tells you what the four big drivers of performance were in the tournament: Getting the right people on the bus, the benefits of interaction, the benefits of training, and the benefits of that strange algorithm that I call the “extremizing algorithm.”
 ... " 

Introduction to Superforecasting

Despite my background in enterprise forecasting,   Superforecasting topic was new to me. Tales from a recent conference. A favorite topic.   Book by U Penn prof  Philip E.Tetlock:   Superforecasting: The Art and Science of Prediction.  Also related:  The Good Judgement Project.   Somewhat Delphi method like.