Application of Bayesian Truth Serum to Forecasting, and Collaborative Discussion

As human knowledge grows, the problem of combining multiple opinions to find truth is becoming both more important and more difficult to solve (Tetlock, 2006).

Majority rule is perhaps the oldest aggregation method, but it fails to work if the truth is contrarian -- if most people happen to be wrong. If there are experts in the crowd, their voices may not be heard. The Bayesian Truth Serum (BTS) is a new approach to the problem of eliciting and combining opinions. It defines expertise as meta-knowledge, which is knowledge of how other people will answer the same question. Using a Bayesian model of belief formation, it can be shown that the BTS can identify experts and find correct answers even when majority opinion is wrong (Prelec & Seung, MIT working paper 2009). We propose to develop this approach experimentally in two contexts, one relatively traditional (forecasting) and the other more speculative (open-ended, scored discussion).