Executive Summary
Can probabilistic forecasting tools such as the wisdom of the crowd help stakeholders—the experts and policymakers at the table—reduce uncertainty and disagreement on big questions? Although studies have shown that crowd forecasting is an effective tool for generating accurate forecasts on well-defined questions, it has yet to find a regular home in the policymaker tool kit. Policymakers are often interested in big questions that are not directly forecastable, and quantitative forecasts on their own lack the context in which to understand their importance.
To address these obstacles, the Center for Security and Emerging Technology (CSET) trialed a method for incorporating crowd forecasting into a stakeholder debate on a big question: What is the future of the relationship between the U.S. Department of Defense (DOD) and Silicon Valley? Based on stakeholder interviews, we broke the big question down into forecastable components relating to the state of the relationship and factors that affect the relationship. We then elicited forecasts on those components from stakeholders and the crowd, using Foretell, the crowd forecasting platform piloted by CSET. The stakeholders’ forecasts provide the context—such as areas of disagreement or uncertainty—in which to understand the wisdom of the crowd. In general, we expect the crowd’s forecasts will be most useful where stakeholder disagreement on the big question can be traced to disagreement on specific forecasts. Under these circumstances, the crowd is akin to a reliable third party that can be invoked to arbitrate a disagreement.
After interviewing and surveying 17 stakeholders, we divided them into two cohorts based on their overall expectations for the DOD-Silicon Valley relationship. The improve cohortexpects the relationship to improve over the next five years, and the same/worse cohortexpects it to stay the same or worsen. We then looked for differences between the cohorts’ forecasts and views on how a factor affects the DOD-Silicon Valley relationship. For forecast disagreements, we compared the cohorts’ forecasts with the views of the crowd—comprising more than 2,700 forecasts from nearly 400 forecasters—to see whether the crowd’s view is more in line with the improve cohort or the same/worse cohort.
The process succeeded in identifying forecast and factor disagreements that might help explain why the stakeholders have different expectations for the DOD-Silicon Valley relationship. The stakeholders’ forecasts generally correlate with their expectations for the relationship, and the crowd’s views are closer to the same/worse cohort more often than the improve cohort. The correlations between forecasts and overall expectations are strongest—and the difference between the cohorts’ forecasts statistically significant—for three metrics:
- Forecast disagreement on percentage of Americans with low confidence in the military. The same/worse cohort forecasts lower confidence in the military. The crowd agrees with the same/worse cohort.
- Forecast disagreement on “big tech” revenue. The improve cohort forecasts higher revenue. The crowd agrees with the same/worse cohort.
- Forecast disagreement on value of “big tech” DOD contracts. The improve cohort forecasts a higher value of “big tech” DOD contracts. The crowd agrees with the same/worse cohort.
The stakeholders generally agree on how changes in a factor would affect the DOD-Silicon Valley relationship. The one exception was also where we saw the most significant disagreement between the cohorts:
- Factor disagreement on strength of U.S. tech sector. Nearly half of the improve cohort believes a stronger U.S. tech sector would improve the DOD-Silicon Valley relationship. None of the same/worse cohort agrees.
This study was a proof of concept for how probabilistic forecasting tools could be incorporated into policy debates on a big question. We hope this effort—or efforts like it—can achieve the following:
- A more-focused debate that zeroes in on the underlying points of disagreement.
- Adjustments in stakeholders’ expectations that reduce overall levels of uncertainty and disagreement.
- A record of what the stakeholders and crowd expected to happen that makes it more likely surprising developments will be noticed and accorded their due weight.
Visit www.cset-foretell.com/issue-campaigns-dod-sv to view the forecast questions and see the most up-to-date data.