Just received their latest newsletter, of interest:
Goodjudgment: How Can Superforecasting Improve your Decisions?
Good Judgment’s co-founder Philip Tetlock literally wrote the book on state-of-the-art crowd-sourced forecasting.
Now, the training, techniques, and talent that helped the Good Judgment Project win a massive government-sponsored forecasting competition can help your organization manage strategic uncertainty. .... "
In this newsletter, we share the Wall Street Journal's view on Superforecasting, an update from our partnership with Eurasia Group, Superforecasters view on EU Article 7 sanctions, Superforecaster Frederic Bush's article in Two Plus Two on Superforecasting in Poker, and reveal the GJ Open crowd's tip for the World Cup winner. .... "
See also, IARPA's sponsorship of work in superforecasting.
Showing posts with label IARPA. Show all posts
Showing posts with label IARPA. Show all posts
Wednesday, August 01, 2018
Thursday, March 01, 2018
IARPA Wants better Machine Learning
Machine Learning: The Good, the Bad, and the Ugly
Government Computer News, Matt Leonard
Although machine-learning technology is still in its early stages, the U.S. Intelligence Advanced Research Projects Activity (IARPA) has been studying machine learning since 2006. Some of its early efforts include the Biometrics Exploitation Science and Technology program, which developed tools for facial recognition that have since been widely adopted. Meanwhile, other projects focused on natural-language processing and formed the basis for the use of machine learning in more complex applications, such as predicting cyberattacks based on conversations in hacker forums and the market price of malware, forecasting military mobilization and terrorism, and developing accurate three-dimensional models of buildings from satellite imagery. IARPA also is studying ways of improving neural networks, the fundamental architecture upon which machine learning is built. For example, the Machine Intelligence from Cortical Networks program aims to reverse-engineer the algorithms of the brain, while the agency also is starting to examine how quantum computing will affect machine learning. .... "
Government Computer News, Matt Leonard
Although machine-learning technology is still in its early stages, the U.S. Intelligence Advanced Research Projects Activity (IARPA) has been studying machine learning since 2006. Some of its early efforts include the Biometrics Exploitation Science and Technology program, which developed tools for facial recognition that have since been widely adopted. Meanwhile, other projects focused on natural-language processing and formed the basis for the use of machine learning in more complex applications, such as predicting cyberattacks based on conversations in hacker forums and the market price of malware, forecasting military mobilization and terrorism, and developing accurate three-dimensional models of buildings from satellite imagery. IARPA also is studying ways of improving neural networks, the fundamental architecture upon which machine learning is built. For example, the Machine Intelligence from Cortical Networks program aims to reverse-engineer the algorithms of the brain, while the agency also is starting to examine how quantum computing will affect machine learning. .... "
Saturday, August 29, 2015
Superforecasting with Tournaments
I m a student of unusual markets and forecasts, and this forecasting approach with tournaments is described in some detail in a recent Edge article. Quite an eminent science group of participants in the project are commenting. Note mention of the Good Judgement Project. Interesting connection to crowd sourcing methods.
" .... When IARPA originally launched this project, they thought that beating the unweighted average of the crowd by 20 percent would be an ambitious goal in year one, 30 percent in year two, 40 percent in year three, and 50 percent in year four. The Good Judgment Project, for reasons that are interesting, was able to beat IARPA's fourth year benchmark in the first year and in all subsequent years. For reasons that are also maybe a little less interesting, other teams were not. I say the reasons are less interesting, I don’t think it was due to them not having the right research expertise. There were issues of mismanagement, of how they went about it. We had a way better project manager.
Putting that to the side, the Good Judgment Project was able to do far better than IARPA or any of the other researchers who were consulted on the design of the project thought possible. We were able to knock out some pretty formidable competitors. Slide twenty-nine tells you what the four big drivers of performance were in the tournament: Getting the right people on the bus, the benefits of interaction, the benefits of training, and the benefits of that strange algorithm that I call the “extremizing algorithm.”
... "
" .... When IARPA originally launched this project, they thought that beating the unweighted average of the crowd by 20 percent would be an ambitious goal in year one, 30 percent in year two, 40 percent in year three, and 50 percent in year four. The Good Judgment Project, for reasons that are interesting, was able to beat IARPA's fourth year benchmark in the first year and in all subsequent years. For reasons that are also maybe a little less interesting, other teams were not. I say the reasons are less interesting, I don’t think it was due to them not having the right research expertise. There were issues of mismanagement, of how they went about it. We had a way better project manager.
Putting that to the side, the Good Judgment Project was able to do far better than IARPA or any of the other researchers who were consulted on the design of the project thought possible. We were able to knock out some pretty formidable competitors. Slide twenty-nine tells you what the four big drivers of performance were in the tournament: Getting the right people on the bus, the benefits of interaction, the benefits of training, and the benefits of that strange algorithm that I call the “extremizing algorithm.”
... "
Thursday, May 14, 2015
SocioPragmatic Models of Text Analysis
Mona Diab of GWU gave the CSI talk today:
Towards Building Effective Computational Sociopragmatics Models of Human Cognition
Slides. For talk recording, go here, and follow instructions using Replay ID: 471310086028
Though technical, the implications can be understood from the slides.
Excellent talk on the use of modeling to analyze text to better understand its meaning from a social and pragmatic - usage perspective. Including alignment and participation in groups. Detection and analysis of subgroups. Looks at how this can be used for multiple languages. Its use for marketing engagement was also mentioned and briefly discussed. Sponsored by DARPA and IARPA
" .. Before joining GWU, Mona was Research Scientist (Principle Investigator) at the Center for Computational Learning Systems (CCLS). She is also co-founder of the CADIM group which is one of the reference points on computational processing of Arabic and its dialects. Her Link.
Towards Building Effective Computational Sociopragmatics Models of Human Cognition
Slides. For talk recording, go here, and follow instructions using Replay ID: 471310086028
Though technical, the implications can be understood from the slides.
Excellent talk on the use of modeling to analyze text to better understand its meaning from a social and pragmatic - usage perspective. Including alignment and participation in groups. Detection and analysis of subgroups. Looks at how this can be used for multiple languages. Its use for marketing engagement was also mentioned and briefly discussed. Sponsored by DARPA and IARPA
" .. Before joining GWU, Mona was Research Scientist (Principle Investigator) at the Center for Computational Learning Systems (CCLS). She is also co-founder of the CADIM group which is one of the reference points on computational processing of Arabic and its dialects. Her Link.
Wednesday, June 25, 2014
IARPA Seeking Brain-like Algorithms
In CACM: Interesting take. I am always skeptical when a requirement to be 'brain like', is included. It is an option, but not the only option, or even the best inspiration:" ... The U.S. Office of the Director of National Intelligence (ODNI) wants to develop technology that will enable computers to think like humans. ODNI's Intelligence Advanced Research Projects Activity (IARPA) believes algorithms that utilize the same data representations, transformations, and learning rules as those employed by the brain have the potential to revolutionize machine intelligence. ... "
Thursday, May 29, 2014
IARPA Sharp Meets Big Data
I was reminded of the IARPA Sharp project again. To which I have a loose connection. Some of the statements below are certainly how we define intelligence ... being adaptive under varying and often contradictory data. This means something different again under the strain of big data. Big Data means we need analytical methods as part of our understanding of the data itself. More thoughts on this will follow.
Strengthening Human Adaptive Reasoning and Problem-Solving (SHARP)
Adaptive reasoning and problem-solving are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated—often under stressful, time-sensitive conditions. In such contexts, one’s ability to accurately update one’s mental models, make valid conclusions, and effectively deploy attention and other cognitive resources is critical. .... "
Strengthening Human Adaptive Reasoning and Problem-Solving (SHARP)
Adaptive reasoning and problem-solving are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated—often under stressful, time-sensitive conditions. In such contexts, one’s ability to accurately update one’s mental models, make valid conclusions, and effectively deploy attention and other cognitive resources is critical. .... "
Monday, February 03, 2014
IARPA Explores Methods to Build Human Analytical Skills
A project to which I am connected started last week. Will continue to provide updates in this space as I can.Adam Russell: IARPA Explores Methods to Build Human Analytical Skills
The Intelligence Advanced Research Projects Activity has kicked off a multi-year collaborative research effort to design new tools and work to help analysts solve complex problems.
IARPA launched the Strengthening Human Adaptive Reasoning and Problem-Solving project to further study cognitive skills in healthy and high-performing people, IARPA said Monday.
“The long-term goal of SHARP research is to develop evidence-based tools and methods that can improve the quality of human judgment and reasoning in complex, real world environments,” said Adam Russell, an IARPA program manager. ... "
More on IARPA Sharp.
Broad concepts covered in Dan Hurley's book.
IARPA thread in this blog.
Thursday, January 02, 2014
Adaptive Reasoning and Human Problem Solving
Brought to my attention. Solicitation by IARPA still open. Below the synopsis. Agree strongly that this is a big topic. Its about the data, AND finding new ways to make it useful in the business process solving context. That context it shifting constantly.
" ... Adaptive reasoning and problem-solving (ARP) are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated-often under stressful, time-sensitive conditions. In such contexts, an ability to apply inductive and deductive reasoning to complex, ambiguous and/or novel problems is critical. Accordingly, optimizing an analyst's adaptive reasoning could pay large dividends in the quality of their analytic conclusions and information products. Given adaptive reasoning tests' high predictive value for performance and productivity, proven methods for strengthening adaptive reasoning and problem-solving could have significant benefits for society in general, as well as for individuals whose work is both analytical and cognitively demanding. Intriguingly, some recent research suggests that these capabilities may be strengthened, even among high-performing adults. Despite some promising results, however, there are methodological and practical shortcomings that currently limit the direct applicability of this research for the Intelligence Community. ... "
" ... Adaptive reasoning and problem-solving (ARP) are increasingly valuable for information-oriented workplaces, where inferences from sparse, voluminous, or conflicting data must be drawn, validated, and communicated-often under stressful, time-sensitive conditions. In such contexts, an ability to apply inductive and deductive reasoning to complex, ambiguous and/or novel problems is critical. Accordingly, optimizing an analyst's adaptive reasoning could pay large dividends in the quality of their analytic conclusions and information products. Given adaptive reasoning tests' high predictive value for performance and productivity, proven methods for strengthening adaptive reasoning and problem-solving could have significant benefits for society in general, as well as for individuals whose work is both analytical and cognitively demanding. Intriguingly, some recent research suggests that these capabilities may be strengthened, even among high-performing adults. Despite some promising results, however, there are methodological and practical shortcomings that currently limit the direct applicability of this research for the Intelligence Community. ... "
Friday, December 20, 2013
IARPA: Intelligence Advanced Research Projects Activity
" ... The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high-payoff research programs that have the potential to provide the United States with an overwhelming intelligence advantage over future adversaries. IARPA works very closely with the various members of the Intelligence Community to ensure that its programs address relevant future needs and to facilitate the transition of demonstrated capabilities. However, IARPA is not an operational organization, and it neither collects raw intelligence nor produces and disseminates intelligence analyses. To ensure organizational agility, IARPA focuses on long-term, 3-5 year programs rather than the short-term time horizons.
IARPA tackles some of the most difficult challenges across the intelligence agencies and disciplines, and results from its programs are expected to transition to its IC customers. IARPA does not have an operational mission and does not deploy technologies directly to the field.
It is about taking real risk
It is not about "quick wins", "low-hanging fruit", or "sure things"
Failure is completely acceptable as long as...
It is not due to failure to maintain technical and programmatic integrity
Results are fully documented
“High-risk/ High-payoff” is not a free pass for stupidity
IARPA brings the best minds to bear on our problems
IARPA sponsors full and open competition to the greatest possible extent
IARPA will not start a program without a good idea and an exceptional person to execute it
IARPA’s cross-community focus ensures its ability to...
Address cross-agency challenges
Leverage expertise from across the community (both operational and R&D)
Work transition strategies and plans with agency partners
High standards of technical rigor
All IARPA programs are structured according to the Heilmeier framework
Technical excellence and technical truth are the hallmarks of all IARPA programs .... "
Thursday, October 20, 2011
Crowdsourcing for Intelligence
In the CACM: University studies crowdsourcing for intelligence. " ... The U.S. intelligence community is studying how to tap the power of crowdsourcing through a multi-university effort. George Mason University professors Charles Twardy and Kathryn Laskey are organizing an online team of more than 500 forecasters who make educated guesses about a series of world events. Their team will vie with four other teams at several universities for grant money supplied by the U.S. Intelligence Advanced Research Projects Activity (IARPA).... "
Subscribe to:
Posts (Atom)