.: ISIT's FOCUS Activities

Imagine this -- during our FOCUS competitions, we had to forecast the outcomes of bots battling bots in computer games. These games included Civilization V , Recon Blind Chess, and Goofspiel. All IARPA gave us to figure out who would win was examples of various pairs of bots fighting each other, all programmed by coders at the Johns Hopkins Advanced Physics Lab. But the games we had to predict all were by bots who had never fought each other before. Then, just to make it harder, those sadists would throw in a wild card from time to time This cranky toddler bot wild card below really happened.

Goofspiel problem

In these bots vs bots games, we were competing against teams from Raytheon and HRL Laboratories.


.: Yeehaw! Our Good Judgment 2.0 team smoked them bots and the Raytheon and HRL teams! Over and over again!

Oh, and if this wasn't hard enough, holy cow! Next IARPA told us we had to teach Amazon Mechanical Turk workers (AKA mTurkers) how to predict what happens, for example, after that toddler bot messed up Red bot's game.

IARPA told us that in just one of these sadistic prediction problems, we had to prove that we could teach mTurkers to achieve a forecast accuracy measured by a Cohen's d of 0.8, which means they had to be about 80% better forecasters than they could be with pure luck. This goal was slightly higher than what the best of our team (like Carolyn) could do, and only with lots of experience and use of Bennett's FOCUS platform for collaborative learning. To make it even harder, we had to distill all we had learned into a two hour training session for these mTurkers.

At first, we thought this was impossible. We were delightfully shocked when the mTurkers made it to a Cohen's d of 0.81 at forecasting Goofspiel game points.

We also won when IARPA asked us to forecast COVID-19 deaths across many nations during the first half year of the COVID-19 pandemic. We were better than the big modelers, including those at the Johns Hopkins Advanced Physics Lab. Yeehaw! IARPA even gave Dawna and Carolyn each a badge for our accurate predictions: Top Ten best COVID-19 forecasts, IARPA FOCUS program.

A big part of our success was our FOCUS platform, programmed by Amory Bennett of Podia, a teaching software company.

Below, the start page for a team with a two week deadline for a group of forecasts. The left hand column lists forecasts we can make for bot vs bots games of Civilization V and Recon Blind Chess. "Cleric," Shaman," and "Minstrel" are bots that play Recon Blind Chess.

program2 week competitionm

Below, an example of a Civilization V question.

Civ V Question

The screenshots above were taken our activities with the Good Judgment Project 2.0 team, competing in IARPA's FOCUS program. ISIT's Dawna Coutant and Carolyn Meinel participated on this team. Yay, team!

Top ten COvid-20 forecastersFOCUSGood Judgment 2.0 team logo

More screenshots from our competitions --->

.: Management and Organization

The Institute's focus areas. Click on a specific area for more information.