.: Hybrid Forecasting


Hybrid forecasting comprises techniques to combine machine forecasting and human judgments. This long has been practiced in the field of meteorology, by running several computer models (machine forecasting) in parallel, often giving significantly different results. Using these models as inputs, meteorologists make judgments on the basis of their expertise. Next, after observing the resulting weather, both the meteorologists and those who design and run the computer models will refine their techniques.

Beginning in 2011, the Intelligence Advanced Research Projects Activity (IARPA) has funded competitions in both machine forecasting of such events, and, separately, human forecasting. Both techniques, using only open source information, have outperformed traditional intelligence community techniques.

Beginning in 2017, IARPA has been attempting to combine human and machine forecasting via the Hybrid Forecasting Competition.

Current performers in the Hybrid Forecasting Competition are:

Raytheon BBN Technologies Corporation (Prescience team) (with subcontracts to American Center for Democracy; Wright State Applied Research Corporation; Tufts University; System of Systems Analytics, Inc.; Lumenogic, Inc.; and Ipsos Public Affairs, LLC).

The University of Southern California (SAGE team) (with subcontracts to Barnard College; Ward Associates; University of California, Irvine; Stanford University; Regina Joseph, Independent Consultant; Pavel Atanasov, Independent Consultant; Manuel Cebrian, Independent Consultant; Kun Zhang, Independent Consultant; and Fordham University).

Test and Evaluation: MITRE Corporation, Good Judgment, Inc, and Cultivate Labs. The control forecasting group this team is running is Carbon. Because this the control team, Carbon’s forecasters had no assistance from machine intelligence.

We were not at all surprised when, in 2018, the top forecaster out of about 1000 Carbon competitors was Dmitry Sarin. He also is a Senior Research Fellow at ISIT. Another Senior Research Fellow here, Carolyn Meinel, participated in another aspect of this T&E effort.

We were surprised, however, when the Carbon team overall surpassed all the Hybrid Forecasting Teams. One of the teams, led by HRL Labs, was subsequently cut from the competition. The other teams, Prescience and SAGE, now are entering the second year of competition.

This surpising outcome of no measurable benefit from combining human and machine intelligence is concerning. So far, by far the most accurate forecasting is being sone by an elite group of "superforecasters," which includes ISIT's Dawna Coutnant, Carolyn Meinel, and Dmitry Sarin.

More on ISIT's human forecasting technologies IR&D as conducted by Carolyn Meinel and ISIT's Dmitry Sarin, here -->

ISIT continues to engage in internal research activities in pursuit of this hybrid forecasting thrust by IARPA.