Superforecasting Short Course from The Edge

Quite interesting, free, have followed the concept for some time, but have yet to apply it.  Videos embedded at the link.   Good to see this followup.  Is enough said about embedded risk analysis?  Following.

A Short Course in Superforecasting—Philip Tetlock: An EDGE Master Class
(ED. NOTE: In 2015, Edge presented "A Short Course in Superforecasting" with political and social scientist Philip Tetlock. Superforecasting is back in the news this week thanks to the UK news coverage of comments by Boris Johnson's chief adviser Dominic Cummings, who urged journalists to "read Philip Tetlock's Superforecasters [sic], instead of political pundits who don't know what they're talking about.")

PHILIP E. TETLOCK, political and social scientist, is the Annenberg University Professor at the University of Pennsylvania, with appointments in Wharton, psychology and political science. He is co-leader of the Good Judgment Project, a multi-year forecasting study, author of Expert Political Judgment, co-author of Counterfactual Thought Experiments in World Politics (with Aaron Belkin), and co-author of Superforecasting: The Art & Science of Prediction (with Dan Gardner). Further reading on Edge: "How to Win at Forecasting: A Conversation with Philip Tetlock" (December 6, 2012). Philip Tetlock's Edge Bio Page.

CLASS I — Forecasting Tournaments: What We Discover When We Start Scoring Accuracy
It is as though high status pundits have learned a valuable survival skill, and that survival skill is they've mastered the art of appearing to go out on a limb without actually going out on a limb. They say dramatic things but there are vague verbiage quantifiers connected to the dramatic things. It sounds as though they're saying something very compelling and riveting. There's a scenario that's been conjured up in your mind of something either very good or very bad. It's vivid, easily imaginable.

It turns out, on close inspection they're not really saying that's going to happen. They're not specifying the conditions, or a time frame, or likelihood, so there's no way of assessing accuracy. You could say these pundits are just doing what a rational pundit would do because they know that they live in a somewhat stochastic world. They know that it's a world that frequently is going to throw off surprises at them, so to maintain their credibility with their community of co-believers they need to be vague. It's an essential survival skill. There is some considerable truth to that, and forecasting tournaments are a very different way of proceeding. Forecasting tournaments require people to attach explicit probabilities to well-defined outcomes in well-defined time frames so you can keep score.   ... " 

Good Judgment Formalizes Superforecasting

Been following the idea of 'superforecasting' for some time.   All business is dependent on accurate forecasts.    If they are analytical-numerical or the hunch of decision makers, or crowdsourced consumer opinions.   Or as more common a combination of these.

Good Judgment names Superforecaster® Warren Hatch as its new CEO

New York, 1 October – Good Judgment is pleased to announce that Warren Hatch has been named chief executive officer, effective immediately. Dr. Hatch was previously President of the firm. As CEO, Hatch will direct Good Judgment’s strategy and oversee its global team.

“Warren brings the strengths of a Superforecaster to the role of Good Judgment CEO – he’s smart, analytical, open-minded, and determined to do whatever it takes to get the job done right,” said Phil Tetlock, Good Judgment co-founder, Wharton professor, and author of Superforecasting: The Art and Science of Prediction. “Warren’s academic credentials in international politics and history combined with his Wall Street experience uniquely position him to lead Good Judgment effectively. I look forward to working with him and our colleagues, both internal and external, as Good Judgment continues to develop innovative approaches for making sense of an uncertain world so that our clients can make better decisions.”  .... '

Superforecasting with Tournaments

I m a student of unusual markets and forecasts,  and this forecasting approach with  tournaments is described in some detail in a recent Edge article.  Quite an eminent science group of participants in the project are commenting.    Note mention of the Good Judgement Project.  Interesting connection to crowd sourcing methods.

" .... When IARPA originally launched this project, they thought that beating the unweighted average of the crowd by 20 percent would be an ambitious goal in year one, 30 percent in year two, 40 percent in year three, and 50 percent in year four. The Good Judgment Project, for reasons that are interesting, was able to beat IARPA's fourth year benchmark in the first year and in all subsequent years. For reasons that are also maybe a little less interesting, other teams were not. I say the reasons are less interesting, I don’t think it was due to them not having the right research expertise. There were issues of mismanagement, of how they went about it. We had a way better project manager.

Putting that to the side, the Good Judgment Project was able to do far better than IARPA or any of the other researchers who were consulted on the design of the project thought possible. We were able to knock out some pretty formidable competitors. Slide twenty-nine tells you what the four big drivers of performance were in the tournament: Getting the right people on the bus, the benefits of interaction, the benefits of training, and the benefits of that strange algorithm that I call the “extremizing algorithm.”
 ... " 

Introduction to Superforecasting

Despite my background in enterprise forecasting,   Superforecasting topic was new to me. Tales from a recent conference. A favorite topic.   Book by U Penn prof  Philip E.Tetlock:   Superforecasting: The Art and Science of Prediction.  Also related:  The Good Judgement Project.   Somewhat Delphi method like.