Combining Human and Machine Predictions Using "Boosting" Algorithms

This project will focus on new approaches for combining human and machine predictions in order to produce a more accurate combined predictor. To do this, we will use algorithms developed for machine learning (specifically, Freund and Schapire’s AdaBoost algorithm), and explore ways that human agent predictions can be combined into stronger predictors.

One of the key problems for intelligent systems at many levels—from animals to human societies—is predicting future events. At the organizational level, for instance, timely and accurate predictions of product sales, competitor actions, and economic conditions can be extremely valuable for businesses, and predictions of troop movements and terrorist actions can be crucial to military and law enforcement organizations. With the increasing availability of massive databases and powerful computational tools for analyzing them, computer predictions are becoming increasingly important in many of these situations. But it is also often useful to combine these machine predictions with additional insights and information provided by human predictions.

This project will focus on a new approach for combining predictions that uses “boosting algorithms” developed for machine learning. (Freund and Schapire, 1996, Schapire, 2002). Boosting algorithms are known for their ability to combine “weak” classifiers, which can be only slightly better than a random guess, into “strong” classifiers, which can be extremely accurate. The lack of constraint on the weak classifiers suggests that the classifiers can be motivated humans or mixed collections of motivated humans and computational agents, as well as computational agents acting alone. As long as a weak classifier does better than chance, it is a candidate to join in a linear sum with other weak classifiers in the construction of a strong classifier.

We consider several important research questions: Can people be put together with each other and with computational agents with a boosting algorithm to make better predictors? Under what circumstances is equal one-person–one-vote weighting adequate? What complications are introduced by human inconsistency? Is there a way to combine AdaBoost concepts with prediction-market methods? And what influence should AdaBoost have on motivation and elicitation, and vice versa, so as to produce the best top-to-bottom systems of prediction.

To attempt to answer these questions, we will begin by using AdaBoost to analyze data we have already collected in a previous experiment that involved both humans and computers predicting plays in a football game. Based on the results of this initial analysis, we expect to design additional experiments and data analysis approaches. For example, we may collect and analyze other types of data, such as about product sales or economic conditions. More generally, we expect to experiment with alternative approaches for combining human and machine knowledge.