Hybrid Forecasting

2020 – 2021 ISIT’s Carolyn Meinel and Dawna Coutant participated in IARPA’s recent hybrid forecasting research in the Forecasting Counterfactuals in Uncontrolled Settings (FOCUS) program.

Oct. 2021: ISIT’s involvement in IARPA’s Hybrid Forecasting Competition finally paid off. Although we didn’t win a contract, we went ahead and did the research anyhow. See ” What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?” in the International Journal of Forecasting.

History of IARPA’s Hybrid Forecasting Competitions

In March of 2020, IARPA ended this program early because its heavily funded contractors were unable to do any better than human forecasters alone, and did far worse than the hybrid forecasting team of ISIT and KaDSci, funded on a shoestring entirely from IR&D (some details here)

Hybrid forecasting comprises techniques to combine machine forecasting and human judgments. This long has been practiced in the field of meteorology, by running several computer models (machine forecasting) in parallel, often giving significantly different results. Using these models as inputs, meteorologists make judgments on the basis of their expertise. Next, after observing the resulting weather, both the meteorologists and those who design and run the computer models will refine their techniques.

Beginning in 2011, the Intelligence Advanced Research Projects Activity (IARPA) funded competitions in both machine forecasting of such events, and, separately, human forecasting. Both techniques, using only open source information, have outperformed traditional intelligence community techniques.

Beginning in 2017, IARPA had been attempting to combine human and machine forecasting via the Hybrid Forecasting Competition.

Performers in the Hybrid Forecasting Competition were:

Raytheon BBN Technologies Corporation (Prescience team) (with subcontracts to American Center for Democracy; Wright State Applied Research Corporation; Tufts University; System of Systems Analytics, Inc.; Lumenogic, Inc.; and Ipsos Public Affairs, LLC).

The University of Southern California (SAGE team) (with subcontracts to Barnard College; Ward Associates; University of California, Irvine; Stanford University; Regina Joseph, Independent Consultant; Pavel Atanasov, Independent Consultant; Manuel Cebrian, Independent Consultant; Kun Zhang, Independent Consultant; and Fordham University).

Test and Evaluation: MITRE Corporation, Good Judgment, Inc, and Cultivate Labs. The control forecasting group this team is running is Carbon. Because this the control team, Carbon’s forecasters had no assistance from machine intelligence.

We were not at all surprised when, in 2018, the top forecaster out of about 1000 Carbon competitors was Dmitry Sarin. He also is a Senior Research Fellow at ISIT. ISIT’s Chief Technology Officer, Carolyn Meinel, participated in another aspect of this T&E effort.

We were surprised, however, when the Carbon team overall surpassed all the Hybrid Forecasting Teams. One of the teams, led by HRL Labs, was subsequently cut from the competition. The other teams, Prescience and SAGE, entered the second year of competition. They did so poorly that the third year of competition was cancelled.

This surprising outcome of no measurable benefit from combining human and machine intelligence was concerning. By the end of 2018, by far the most accurate forecasting was being done by an elite group of “superforecasters,” which included ISIT’s Dawna Coutant, Carolyn Meinel, and Dmitry Sarin.

Then in 2019, as noted above, ISIT and KaDSci, LLC did even better with a hybrid forecasting system funded on a shoestring through IR&D. ISIT’s Dawna Coutant and Carolyn Meinel participated on this team.

More on ISIT’s human forecasting technologies IR&D as conducted by Carolyn Meinel, Dawba Coutant and Dmitry Sarin, here –>

ISIT continues to engage in internal research activities in pursuit of this hybrid forecasting initiated by IARPA.