!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> Streamline Training & Documentation: A Study of Google's Prediction Markets

Tuesday, July 07, 2009

A Study of Google's Prediction Markets

In January of this year, Bo Cowgill (Google, Inc.), Justin Wolfers (Wharton School), and Eric Zitzewitz (Dartmouth College) published the results of their research (pdf) concerning how prediction markets at Google function, both in terms of the accuracy of the predictions generated and in terms of what they tell us about how information moves around the company.

With respect to the accuracy of predictions generated, the question can be rephrased as "How efficient are Google's prediction markets? How reliably do the prices in the markets reflect the actual probabilities of the events the markets are assessing? (An example of an event would be "Gmail having between X and Y users" at the end of the current quarter.)

Cowgill, Wolfers, and Zitzewitz (CWZ) found that the efficiency of Google's prediction markets is somewhat reduced by four biases:
  • Optimism — Participants tended to overestimate the likelihood of outcomes favorable to Google. This was the most pronouced bias CWZ found, and it was particularly strong on and following days when Google stock appreciated.

    Note that this bias "exists entirely in the two categories of contracts where outcomes are most directly under the control of Google employees: company news (e.g., office openings) and performance (e.g., project completion and product quality). Markets on demand and external news with implications for Google are not optimistically biased."

  • Aversion to betting on extreme outcomes — Participants tended to underprice extreme incomes.

  • Attraction to favorites and aversion to longshots — Participants tended to slightly overprice outcomes with short odds, and to slightly underprice outcomes with long odds.

  • Aversion to short selling — CWZ found that returns from purchasing securities (as opposed to selling securities) were negative and statistically significant on average.

    CWZ comment, "As further evidence of short aversion, in order book snapshots collected each time an order was placed, we found 1,747 instances where the bid prices of the securities in a particular market added to more than 1, implying an arbitrage opportunity (from buying a bundle of securities for $1 and then selling the components). In [contrast], we found only 495 instances where the ask prices added to less than 1 (implying an arbitrage opportunity of buying the components of a bundle for less than $1 and then exchanging the bundle)."
In their discussion of these four biases in prices, CWZ note that they were
partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period [second quarter of 2005 to third quarter of 2007], suggesting that corporate prediction markets may perform better as collective experience increases.
With respect to the question of how Google as an organization processes information (more precisely, "information and beliefs about prediction market topics"), CWZ report several findings concerning correlations in trading behavior:
  • Close geographic proximity matters. Market participants "who share an office or whose offices are located within a few feet on the same floor" appear to make correlated trades.

  • Organizational proximity matters. "[T]he single best explanator [of participants' displaying correlated trading] is being within one or two steps on the organization chart (i.e., sharing a manager, being someone’s manager, or being someone’s manager’s manager)."

  • Work history matters. Participants with a history of reviewing each other's code or overlapping on a project also tended to display correlated trading patterns.

  • Social connections — measured in terms of a self-reported professional relationship, self-reported friendship, and the number of overlapping email lists — don't seem to play much of a role in explaining correlated trading.

  • Demographic factors do not seem to play a role in explaining correlated trading.
CWZ close with a caveat:
[O]ur results ... tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters more than work relationships — if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.