!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
Streamline Training & Documentation
Streamline Training & Documentation
Sunday, November 15, 2009
The Pharmer's Market
A group of researchers — Ragu Bharadwaj, Eric von Hippel and Fiona Murray at MIT's Sloan School and Peter Coles at Harvard Business School — have spearheaded development of a prediction market at Crowdcast.com that focuses on questions relating to the pharmaceuticals industry.
The current questions on which individuals with pharmaceutical knowledge are invited to weigh in, deal with drug candidates for breast cancer treatment:
Will AMG 706 make it through Phase 2 Trials to Phase 3 by July 2010?
Will ABT-869 make it through Phase 2 trials to Phase 3 by March 31, 2010?
When will Phase 3 Trials for breast cancer be announced?
Will AZD2281 make it through Phase 2 trials to Phase 3 by March 31, 2010?
When will Phase 3 Trials for breast cancer be announced?
Will BIBW 2992 make it through Phase 2 trials to Phase 3 by March 31, 2010?
When will Phase 3 Trials for breast cancer be announced?
Will BMS-599626 make it through Phase 1 trials to Phase 2 by March 31, 2010?
When will Phase 2 Trials be announced?
Will BMS-690514 make it through Phase 1 trials to Phase 2 by October 2010?
Will Cediranib make it through Phase 2 trials to Phase 3 by Jan 1, 2010?
When will Phase 3 Trials for breast cancer be announced?
Will Dasatinib make it to Phase 3 trials by July 1, 2010?
Will Lonaprisan make it through Phase 2 trials to Phase 3 by Jan 1, 2010?
When will Phase 3 Trials be announced?
The goal is to predict progress through clinical trials more accurately than is possible with alternate methods (e.g., spreadsheet modeling).
For more about the context of hugely expensive drug development into which the Pharmer's Market fits, you can read an article by Natasha Singer published in today's New York Times.
One of the companies offering a platform for organizations wishing to set up prediction markets — Spigit — has partnered with IBM to run a series of prediction markets related to IBM's Smarter Cities initiative.
According to IBM,
SmarterCities is an integrated, multi-year program [that] is part of IBM's smarter planet agenda. The program was created to bolster economic vitality and the quality of life in cities and metropolitan areas by sparking new thinking and meaningful action across the city ecosystem — from mayors to citizens.
Hutch Carpenter of Spigit has provided a good overview of Smarter Cities in a blog post.
Once you have learned about the context — from reading Carpenter's post and/or browsing the Smarter Cities website — you can home in on the twenty questions for which IBM has set up prediction markets. For convenience, I've reproduced the questions below (lightly edited), along with the answers people are being asked to assign probabilities to. (The markets close on September 13 at 11:59 pm.)
The aforementioned probabilities are expressed in terms of the portion of an allotment of 100 tokens that a particular market participant decides to "invest" in each option for a particular question.
For example, for the first question below, a participant might invest 20 tokens in the first option, 30 in the second, 20 in the third, 15 in the fourth, and 15 in the fifth. This would indicate that she thinks the second option is most likely to improve education outcomes, the first and the third are somewhat less likely to do so, and the fourth and the firth are least likely to help.
The Smarter Cities questions ...
Education
Which approach will be most effective in enabling better education outcomes within a major city?
Provide real-time information on student achievement to teachers
Provide individualized lesson plans and activities using digital educational content
Enable digital devices and access for students at home
Provide online teacher training and collaboration capabilities
Foster shared services and best practice exchange across school districts and higher educational institutions
In order to increase the proportion of the population completing high school by 10% over the next five years, major cities will begin transforming education in what way?
Provide an educational experience that combines traditional classroom learning with hands-on experience outside of the classroom (e.g., internships in museums and cultural institutions, media experiences, volunteer work, etc.)
Increase the use of online and/or remote education to increase access to a wide range of course offerings
Invest in health and social services that are integrated with education to assure students are prepared to participate fully in school
Decentralize decision making to the school level (for budget, hiring, curriculum, teacher training, etc.) and equip schools with decision-support and analysis technologies
Strengthen accountability for academic standards, curriculum, teacher quality, and allocation of resources at the City/Mayoral level to assure quality and equity of services, economies of scale and a broader range of services
Transportation
Which company offers the best portfolio regarding Smarter Transportation?
IBM
Telvent
Siemens
Accenture
none of the above
In a major city, what will need to be improved in order to make transportation more efficient?
Public safety
The utility grid
Emergency healthcare services
University and school location planning
Government services (such as subway ticketing systems, road charging, department of motor vehicle system, etc.)
What enhancement can a major city make over the next year to be a global technology leader in public transportation?
Allow users to pay for transportation service (tolls, trains, buses, taxis, parking) through mobile devices
Launch citywide social networks for citizens to report conditions of roads, accidents, air quality, etc., to government and other citizens
Embed sensors in city infrastructure, utilities, and public transportation to monitor traffic violations, air quality levels, people congestion, etc.
Implement traffic modeling and prediction technology to inform transportation network operators and travelers of upcoming traffic conditions and route alternatives incorporating all modes of transportation
Faster public transportation that connects a major city with its suburbs
What transportation enhancement will a major city, like New York, need to make to relieve its traffic congestion?
Impose congestion fees for travel into the city based on city zones and time of the day
Integrate mobile devices with pay systems to pay for all types of road charging to avoid bottlenecks
Launch real-time social network systems that track and share traffic-related data with all citizens, offering choice of alternate routes
Increase transportation capacity by improving infrastructure (build more roads, bridges, and tunnels) and services (bus and train)
Implement a real-time parking system with highly accurate information to avoid traffic generated by circling for a parking spot
Utilities
Which of the following will be the most important to the rapid deployment and adoption of Smart Grids?
Acceleration of the government stimulus programs
Alignment of objectives with regulators and policy makers
Technology maturity
Over the next five years, what changes should a major city first implement to reduce energy waste and use its resources efficiently?
Decentralize power generation so energy is close to point of use
"Instrument" demand and supply through smart meters [i.e., install meters that enable electricity customers to monitor their usage, with a view both to conserving and to shifting usage from peak to off-peak times]
Develop new energy models using renewable energy
Which of the following will reduce household energy consumption the most within a major city like New York?
Implement metering to furnish citizens usage statistics
Generate awareness and share tips related to energy consumption and waste
Promote energy-efficient appliances
Which of the following should be a primary objective for a major city over the next five years?
Become a low-carbon city (e.g., Chicago, Malaga)
Implement a smart grid (e.g., Malta)
Become a zero waste eco-city (e.g., Masdar)
Government Services
The current economic crisis will change plans for high priority projects in a major city in which way over the next few years?
High priority improvement projects will continue without any significant changes.
High priority improvement projects will be adjusted to make use of economic stimulus dollars to prioritize which get addressed first.
High priority improvement projects will be delayed until the economy improves and tax revenues recover.
If you were a mayor of a major city, which method would you use to assess the needs of your city, the business community and your citizens?
Comparisons to other cities on key metrics
Conducting surveys of business needs through your chamber of commerce
Surveying the needs of citizens through websites, email, or snail mail
In 2011, what will be the primary method for citizens to communicate with their smarter city governments?
Automated call centers (such as 311 service in New York City)
Internet Website (via forms)
Text messaging services (such as tweeting, instant messaging, etc.)
Automated sensor devices without human involvement
Visiting government offices
What immediate step should a major city government take over the next year to emerge as a leader in e-governance?
Provide a social platform to enable two-way communication between citizens and government
Model, analyze and predict changes to all parts of a city via a “dashboard” for decision support made available to businesses and citizens
Integrate a digital experience via mobile devices across the city (e.g., subway tickets, department of motor vehicle forms and processes, museums and cultural events, etc.)
Public Safety
Over the next five years, what transformation will large cities make to their public safety systems to reduce the physical/personal crime rate against people, property, and infrastructure by half (50%)?
Actively involve communities in data collection, neighborhood watch, and crime reporting
Deploy more police resources
Install widespread video surveillance to monitor crime and enhance situational awareness
Standardize data formats for smarter crime analysis, enabling faster response and predictive capabilities for crime hotspots and upswings
Integrate public safety systems with other relevant sub-systems, such as healthcare, transportation, and education, to monitor emergencies and anomalies
If a large city wants to improve its overall public safety position (i.e. reducing traffic fatalities, decreasing gang violence, improving emergency response capabilities), in which public safety area (or related “city sub-system”) should it target investment over the next year?
Transportation and highway infrastructure improvements
Education and after-school activities
Interoperable communications
Video surveillance to monitor activity
Crime analytics to more effectively deploy resources
Healthcare
Which of the following sub-system improvements will be most effective in providing immediate benefit to healthcare delivery for citizens in a leading smarter city?
Traditional education combined with services like healthcare
Consolidated view of public safety information (e.g., crime and surveillance, pandemic outbreaks, accidents, etc.)
A means to enable interactive communication or exchange of information electronically between citizens and healthcare systems
Green energy practices to reduce pollution and improve air quality
Deployment of technologies for remote diagnostics, etc., that will enable access to basic healthcare for all a city’s citizens
Over the next five years, what will major city hospitals do to increase efficiency and deliver better quality healthcare to citizens?
Adopt an electronic means of capturing patient information
Implement social collaboration tools for staff (such as instant messaging, wikis, social network sites, etc.)
Allow patient access to their electronic records and [electronic?] patient interaction with their care-givers
Increase the use of technology-assisted medical procedures by half (50%)
Other
What are the top challenges large cities (population over 5 million) in emerging markets will face during the next five years?
Inadequate utilities (electricity, telecommunications and water)
Inadequate social services (healthcare, education)
Poor public safety / inadequate police, fire, and emergency services
Inadequate transport infrastructure (roads, rail, air)
What region will recover most quickly from the current global economic crisis?
In January of this year, Bo Cowgill (Google, Inc.), Justin Wolfers (Wharton School), and Eric Zitzewitz (Dartmouth College) published the results of their research (pdf) concerning how prediction markets at Google function, both in terms of the accuracy of the predictions generated and in terms of what they tell us about how information moves around the company.
With respect to the accuracy of predictions generated, the question can be rephrased as "How efficient are Google's prediction markets? How reliably do the prices in the markets reflect the actual probabilities of the events the markets are assessing? (An example of an event would be "Gmail having between X and Y users" at the end of the current quarter.)
Cowgill, Wolfers, and Zitzewitz (CWZ) found that the efficiency of Google's prediction markets is somewhat reduced by four biases:
Optimism — Participants tended to overestimate the likelihood of outcomes favorable to Google. This was the most pronouced bias CWZ found, and it was particularly strong on and following days when Google stock appreciated.
Note that this bias "exists entirely in the two categories of contracts where outcomes are most directly under the control of Google employees: company news (e.g., office openings) and performance (e.g., project completion and product quality). Markets on demand and external news with implications for Google are not optimistically biased."
Aversion to betting on extreme outcomes — Participants tended to underprice extreme incomes.
Attraction to favorites and aversion to longshots — Participants tended to slightly overprice outcomes with short odds, and to slightly underprice outcomes with long odds.
Aversion toshort selling — CWZ found that returns from purchasing securities (as opposed to selling securities) were negative and statistically significant on average.
CWZ comment, "As further evidence of short aversion, in order book snapshots collected each time an order was placed, we found 1,747 instances where the bid prices of the securities in a particular market added to more than 1, implying an arbitrage opportunity (from buying a bundle of securities for $1 and then selling the components). In [contrast], we found only 495 instances where the ask prices added to less than 1 (implying an arbitrage opportunity of buying the components of a bundle for less than $1 and then exchanging the bundle)."
In their discussion of these four biases in prices, CWZ note that they were
partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period [second quarter of 2005 to third quarter of 2007], suggesting that corporate prediction markets may perform better as collective experience increases.
With respect to the question of how Google as an organization processes information (more precisely, "information and beliefs about prediction market topics"), CWZ report several findings concerning correlations in trading behavior:
Close geographic proximity matters. Market participants "who share an office or whose offices are located within a few feet on the same floor" appear to make correlated trades.
Organizational proximity matters. "[T]he single best explanator [of participants' displaying correlated trading] is being within one or two steps on the organization chart (i.e., sharing a manager, being someone’s manager, or being someone’s manager’s manager)."
Work history matters. Participants with a history of reviewing each other's code or overlapping on a project also tended to display correlated trading patterns.
Social connections — measured in terms of a self-reported professional relationship, self-reported friendship, and the number of overlapping email lists — don't seem to play much of a role in explaining correlated trading.
Demographic factors do not seem to play a role in explaining correlated trading.
CWZ close with a caveat:
[O]ur results ... tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters more than work relationships — if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.
How much participation does a prediction market need?
In April 2008 McKinsey published an edited and abridged transcript (pdf) of a roundtable on prediction markets moderated by Renée Dye, a consultant in McKinsey's Atlanta office. The participants were four experts on prediction markets:
Bo Cowgill, who is intimately involved with Google's prediction markets.
Todd Henderson, who teaches at the University of Chicago law school and conducts research on prediction markets.
Jeff Severts, a VP at Best Buy whose responsibilities include forecasting and helping oversee Best Buy's prediction markets.
The discussion is quite substantive and well worth reading in its entirety (eleven pages). Here I will call attention to a helpful sidebar Dye provides in which she summarizes the decisions you have to make in setting up a prediction market at your own organization.
Dye cites six key decisions:
How to define the variable the market will forecast. "Express [the variable] in a precise, intuitive unit (such as '2nd-quarter revenue, in euros, for new product X') to avoid confusion among participants."
With whom to share the results. Dye points out that results can be embarrassing to management (e.g., a prediction that a product under development will fail in the market). Results can also raise legal issues (e.g., a prediction that future financial results will show that the company's current stock price is too high).
Who should participate. "Markets involving only internal participants are easiest to organize." On the other hand, if you include appropriate external participants, you will generally increase the accuracy of the results.
Dye also points out, "Front-line employees often are the most active and excited participants."
The nature of the market. "Markets with real-time buying and selling of contracts yield rich, continuous results but require large numbers of participants, some of whom may need training."
"Simple surveys and other single-point forecast mechanisms are easier to administer. Companies getting started may want to proceed gradually through a series of increasingly sophisticated experiments."
Incentives. Cash can present legal issues, since it can make your prediction market look suspiciously like a gambling operation. Fake money can be a workable substitute. Modest prizes, such as t-shirts, have worked well at companies like Google (pdf).
See here for Adam Siegel's suggestion of using access to top management as an incentive. (Siegel is a co-founder of Inkling, a company in Chicago that provides a prediction market platform that anyone can use to set up a prediction market. See this earlier post.)
The role of experts. "Departments dedicated to forecasting [e.g., marketing] will see the establishment of a prediction market as a threat."
Dye argues that shifting the mindset of experts concerning their role is important. Rather than being the people "with all the answers," experts should view their role as formulating the right questions and helping with "analyzing the answers [yielded by the prediction market] in creative ways and using them to guide decision making.".
BTW, if you want to track academic work in the field of prediction markets, one source you can use is the Journal of Prediction Markets, published since 2007 by the University of Buckingham Press.
As a follow-on to my earlier post on competitive analysis, I'd like to call attention to a helpful article on "How to Make Sense of Weak Signals" in the Spring 2009 issue of the MIT Sloan Management Review.
This article by Paul Schoemaker, research director of the Mack Center for Technological Innovation, and George Day, a professor of marketing at the Wharton School, covers a broader range than just the task of scoping out the competition. Schoemaker and Day describe a straightforward way of attending to and responding to weak signals of all sorts that are relevant to one's business, as omens either of emerging opportunities or of looming threats.
A weak signal is defined as a
seemingly random or disconnected piece of information that at first appears to be background noise but can be recognized as part of a significant pattern by viewing it through a different frame or connecting it with other pieces of information.
Schoemaker and Day divide the process of making effective use of weak signals into three phases:
Actively scan for weak signals. Three strategies to consider:
Tap local intelligence, i.e., information distributed among various individual locations in which the organization has a presence.
Leverage extended networks, i.e., networks encompassing partners, suppliers, customers, etc.
Mobilize search parties, i.e., task forces set up to monitor specific areas of interest.
Amplify interesting weak signals to help in deciding what they mean. Three strategies to consider:
Test multiple hypotheses. E.g., you might want to use red teams (MSWord) "to collect and synthesize information to prove that the current plan is wrong and needs to be changed."
Canvass the collective wisdom of your organization. E.g., you might want to try a prediction market.
Develop diverse scenarios. "Scenario planning systematizes the hunt for weak signals that may foreshadow fundamental shifts in the marketplace and society at large ..."
Probe further, clarify, and act. Three strategies to consider:
Seek new information to "confront reality," i.e., you need to recognize developments that make planning and executing an effective response imperative.
Encourage constructive conflict "to ascertain and interpret the facts as they are."
Trust seasoned intuition. "It takes many years of experience, with good feedback, to develop reliable intuition. But once it has been honed, intuitive hunches should be viewed as valuable inputs, along with more analytical ones, for the judgment process."
Schoemaker and Day conclude by reiterating the point with which they begin their article: "The major problem [in monitoring and responding to weak signals] is that managers are insufficiently aware of cognitive and emotional biases that can cloud their judgment when interpreting weak signals."
Issues to Consider when Tapping Collective Intelligence
In the Winter 2009 issue of the MIT Sloan Management Review, Eric Bonabeau, CEO of Icosystem Corp., reviews what we know about the best ways to apply collective intelligence to decision making (the alternative being to depend on a small number of decision-makers who may suffer from biased thinking and/or poor information).
The techniques for tapping collective intelligence Bonabeau mentions include:
Information markets
Wikis
Crowdsourcing
Social networks
Collaborative software
What is especially helpful in Bonabeau's article is his discussion of specific issues one should consider when deciding if and how to use collective intelligence techniques:
Loss of control — Depending on collective intelligence can produce an undesirable outcome, an outcome so unpredictable that the organization is not prepared to deal with it, lack of clarity concerning who is responsible for bad decisions, and public relations problems if outsiders are involved and come up with embarrassing ideas.
Diversity vs. expertise — The best outcome in some situations is produced by ensuring that a wide variety of perspectives are considered. In other situations, it may be best to sacrifice diversity in order to ensure that needed expertise dominates the analysis and decision making.
Engagement — "... organizations must provide a continuous flow of new, enthusiastic participants to keep engagement high, or they need to provide incentives to sustain people's motivation over time."
Policing — to control any mischief-making or malicious input.
Intellectual property — An organization must both manage its own intellectual property when sharing information with those whose input is sought, and it must "determine whether and how it will assume ownership" of intellectual property that arises from ideas contributed by people outside the organization.
Mechanism design — The organization must answer such questions as who gets to participate, whether everyone's input receives equal weight, and whether decision making will be distributed (a number of people contribute to one decision) or decentralized (many people are empowered to make their own independent decisions).
Bonabeau summarizes his key point:
For many problems that a company faces, there is potentially a solution out there, far outside of the traditional places that managers might search, within or outside the organization. The trick, though, is to develop the right tool for locating that source and then tapping into it.
Bonabeaue also emphasizes the importance of identifying appropriate metrics and indicators for assessing the performance of collective intelligence tools the organization adopts.
Suppose your company has developed a new strategy, and you've been given the job of serving as devil's advocate (or as a member of a strategy review panel). How should you go about probing the strategy's soundness?
In an article in the September issue of the Harvard Business Review, Paul B. Carroll and Chunka Mui suggest a half dozen questions to ask:
Is this a realistic strategy for long-term success?
What can we learn from history?
Do vital information and dissenting views about strategies reach decision makers?
Have we assessed the true advantages — and liabilities — that come with scale?
Have we considered all our options?
Would we bet on it?
Once answers to these and any other relevant questions have been carefully discussed, the devil's advocate (or review panel) prepares a report of its findings. A company that believes in intelligent risk management will take the findings seriously in deciding whether the draft strategy requires revision.
Intrade's State-by-State Predictions for the Presidential Election
If you want to follow the shifting fortunes of the US presidential candidates, as reflected in their state-by-state share prices on Intrade's prediction market, the US map at electoralmap.net provides a graphical picture that is updated at least weekly.
The candidates' share prices reflect how the Intrade market participants collectively view John McCain's and Barack Obama's respective prospects for winning each state's electoral votes.
For example, if the price for Obama in a particular state is "50," this means that participants collectively view Obama's chances of winning the state as 50-50. If Obama's price is above "50," participants are giving Obama a better than even chance of winning the state. Conversely, if Obama's price is below "50," participants are giving Obama a better than even chance of losing the state.
You can view electoralmap.net maps dating back to May 24. If you want to watch how Intrade participants have shifted their state-by-state predictions, click on "animated map."
There is a tendency of some people to overgeneralize concerning the value of blogs, with the direction of generalization being decidedly negative. (See, for instance, this earlier post.) In fact, I encountered such dismissive overgeneralization in a conversation with one of my brothers just a few days ago.
Because so many thousands of blogs are, at best, of no interest to a particular individual and, at worst, of no value to anyone beyond the blogger himself/herself, blogging as a whole gets written off.
This refusal to pay attention — selectively — to the blogosphere amounts to passing up valuable opportunities to learn about subjects of interest, and to see how experts evaluate new information and ideas as they emerge.
For example, Andrew McAfee, an associate professor of business administration at the Harvard business school, wrote a post for his blog earlier this month explaining how his blog has helped him reach a large audience with his knowledge, ideas, and views concerning Enterprise 2.0. There are comparable examples of influential blogs in just about any area you'd care to mention. There are worse ways to spend time than monitoring a few blogs dealing expertly with subjects you want to stay up-to-date on.
Thanks to an article by David Leonhardt in today's New York Times, we have an update on how one popular prediction market — Intrade — is doing in predicting the outcomes of the current round of primary elections.
The predictions have been less than accurate, notably, in the case of New Hampshire, which raises the obvious question, Why?
Leonhardt cites the analysis of Barry Ritholtz on his well-trafficked blog, The Big Picture. In a January 11 post, Ritholtz points to three problems with the political prediction markets: their thinness, their small trading volumes, and the low number of dollars at risk.
The small scale of the Intrade market means that
if a significant portion of the participants are biased, they can skew the prices. An example is Ron Paul's non-zero Intrade odds of winning any primary in which he is running.
sluggish response to new information, with eventual overreaction all too common. Leohhardt suspects that this is due to a shortage of smart money in the market.
Leonhardt concludes his article by noting that as more traders recognize the opportunity to make money off the inefficiency of Intrade's political markets, the level of inefficiency will drop, i.e., more smart money will make Intrade's prices more accurately reflective of true probabilities.
As Paul Krugman points out in his blog, at least in the case of the New Hampshire primary, prediction markets trading on the basis of people's bets concerning who would win were no more prescient about the actual outcome than the pollsters and pundits who had Barack Obama coming out on top by several percentage points.
You can see charts for Iowa and New Hampshire contract prices for the major Democratic and Republican candidates here. The extreme volatility for Clinton and Obama in the January 7-9 timeframe is apparent.
The Handbook's page on prediction markets provides a helpful summary of types of prediction markets, research on their nature and reliability, and ideas for further research.
The list of benefits of prediction markets is a helpful reminder of why the topic is important. As summarized by the (typo-prone and grammar-challenged) collective authorship of the Handbook, prediction markets are valuable because:
Traders can correct their own biases, assuming they can see how others are voting.
Traders get a bigger picture that has a high signal-to-noise ratio. The aggregated prediction is a reasonably good summary statistic of many people's reading of the situation.
Organizations can be more agile, since agility depends partly on being able to better anticipate the future.
Availability of internal prices (shadow prices) leads to more precise asset allocation. For example, using a prediction market at a firm could lead to more individualized service if it informs sales staff about how much it would cost to accelerate orders to satisfy a particularly important customer.
Contingent contracts can aid decision making. (However, one needs to be cautious about interpreting contract prices as probabilities for contingencies, since it is easy to mistake correlation for causation.)
You can read here about what several of MIT's collective intelligence researchers — Thomas W. Malone, Alex (Sandy) Pentland, Tomaso Poggio, Drazen Prelec, and Josh Tenenbaum — have to say concerning "prediction economies," a step beyond individual prediction markets.
London-based consultant Jed Christiansen has put together the following 2:49 video explaining the ABCs of prediction markets. If you're new to prediction markets, and you don't mind encountering a pitch for Christiansen's firm at the end, the video is worth viewing.
You can get a quick introduction to the Delphi technique for forecasting in the September issue of the Harvard Business Review. Robert Duboff, CEO of HawkPartners, a marketing consulting firm, explains that the Delphi technique involves recruiting about 20 experts in the area you are assessing (e.g., the prospects for an investment that is under consideration) and asking them — by phone, email, or in person — to evaluate possible outcomes.
First, the experts are asked to identify the outcomes they consider possible, and then to rate the likelihood of each. Duboff suggests that this initial set of ratings be made in the absence of any comments from the experts explaining their predictions. Your choice concerning whether to solicit comments at this stage of the process depends on whether you think allowing comments will inhibit or enrich the generation and consideration of ideas.
The experts' ratings are tabulated and presented to them. After discussion, the experts rate the predicted outcomes again.
Additional rounds are conducted until a pre-determined stop point is reached. This can be consensus on the ratings, or it might be a maximum number of rounds if consensus does not emerge.
You can read a history of the Delphi technique in its Wikipedia entry, which also provides a helpful comparision to prediction markets, reproduced below in slightly edited form.
Advantages of prediction markets, relative to the Delphi technique, include:
They can motivate people to participate over a long period of time and to reveal their true beliefs.
They aggregate information automatically and instantly incorporate new information into the forecast.
Participants do not have to be selected and recruited by a facilitator; they themselves decide whether to participate.
Advantages of the Delphi technique include:
Delphi allows a broader range of problems to be formulated.
For many people, it is easier to reveal one's opinion in response to a questionnaire than to translate it into market prices.
It may be easier to maintain confidentiality with Delphi.
Delphi is not vulnerable to manipulation by participants.
The transparent exchange of knowledge in Delphi allows participants to learn from each other and to introduce new ideas into the discussion.
Only 5 to 20 experts are necessary for conducting a Delphi session.
You can find a wealth or information on the Delphi technique and other forecasting methods at http://www.forecastingprinciples.com, a site created by J. Scott Armstrong, a professor of marketing at the Wharton School. Prof. Armstrong has also created the Delphi Decision Aid, an easy-to-use tool for implementing the Delphi technique.
The Wall Street Journal recently published an important column that provides a useful reminder of how prediction markets can assist with decision making. In a nutshell:
These markets often predict more accurately than experts. Why? They draw on the knowledge of people who might otherwise be ignored. Their anonymity frees participants from pressures to agree with opinion leaders. And they create straightforward profit incentives that encourage participants to search for better information.
Robert Hahn, executive director of the AEI-Brookings Joint Center, and Paul Tetlock, a finance professor at the University of Texas at Austin, wrote this column in the context of concern over how regulatory restrictions on Internet gambling may inhibit use of prediction markets for research purposes. (The Iowa Electronic Markets are spared because they qualify as an educational tool.)
Hahn and Tetlock explain that the AEI-Brookings Joint Center has published a plan, "endorsed by more than 20 leading researchers" that "suggests the creation of a safe harbor for small-stakes, not-for-profit prediction markets to encourage experimentation." The sort of research contemplated includes investigating
how to increase the depth of the markets and make them less susceptible to manipulation. It could also address politically contentious questions, such as how to prevent criminals from benefiting from the use of these markets ...
One can only hope that this highly visible advocacy for freedom to research the ins and outs of prediction markets will bear fruit.
The Chicago Tribune published an article in February about inkling, a company Adam Siegel and Nathan Kontny launched in late 2005 to enable interested parties to participate in public prediction markets and to run secure, private markets.
By registering for a free inkling account, you get access to the inkling interface for setting up your own prediction market. This is an easy way to get hands-on experience with the concepts and process involved.
And then, because the inkling application is coded as a widget, it is easy to add a prediction market you've created to your blog or website if you decide that that's something you'd like to do.
Now that the election is over, we can take a quick look at how various trading sites did in predicting the outcome. A sampling:
As the Wall Street Journal reports today, TradeSports on Friday was giving the Democrats only a 30% chance of winning the Senate. As Tuesday approached, the odds in favor of the Democrats increased, though not in steady fashion.
To get an idea of the volatility of the betting on the outcome of the fight for control of the Senate, you can look at the graph of daily midnight closing prices provided by the Iowa Electronic Markets site.
CasualObserver.net touts its success in predicting the election results more accurately than the Rasmussen Reports poll. CasualObserver made its comparison as of Tuesday morning, i.e., in advance of release of any exit poll data and actual vote counts.
As a follow-on to mypreviousposts concerning prediction markets, here's a link to Tradesports, a trading site that is currently featuring collective predictions concerning the outcome of today's elections. Or, if you prefer the Iowa Electronic Markets, click here. Or you can visit the Intrade site. Enjoy.
Prompted by a brief article by Cass Sunstein in the September 2006 issue of the Harvard Business Review, I'm revisiting today the topic of prediction markets, discussed in two previousposts.
In his article, Sunstein, a professor of both law and political science at the University of Chicago, makes a point that should almost go without saying: A prediction market should be used only in situations in which the correct answer is the answer participants are most likely (but not guaranteed) to give to whatever question is being asked.
This circumstance will prevail only if participants have access to relevant information about the subject under consideration. Each individual participant may very well have only incomplete information. The beauty of the prediction market is that it enables consolidation of what everybody knows into an answer with a high probability of being accurate.
Sunstein offers this example:
A computer company executive could sensibly rely on an internal prediction market if she is asking about completion dates for the company's own products in development. But should the manager ask employees about completion dates for competitors' products? That wouldn't be a good bet. When most people are not likely to be right because the group has little relevant information, it's best to ignore their judgments — and to try to find an expert instead.
I hasten to add that Sunstein's basic position is to endorse the power of prediction markets. Last month he published Infotopia: How Many Minds Produce Knowledge, which deals with the great benefits we can realize through using information-pooling tools like prediction markets, wikis, and open-source software.
As a follow-on to this earlier post on prediction markets, I'd mention a Business Weekonline report that provides several examples of how companies are using prediction markets to help with decision-making. By aggregating knowledge from a broad base of individuals, a prediction market can often yield highly accurate indications of future outcomes. They also help top management get a clearer picture of employees' thinking.
Writer Rachael King explains that
[t]he markets are particularly useful in areas such as consumer goods or technology, where change is rapid and companies need to adapt quickly or get left behind.
Specific types of decisions prediction markets can help with include:
The last item is particularly interesting to me because I've read so often about the unreliability of conventional methods of predicting a movie's box office. You can get an idea of how movie-oriented prediction markets work by visiting the Hollywood Stock Exchange website.
The Business Week report provides a handy list of Do's and Don'ts for setting up a prediction market. One Do is investing in education for participants, so they understand how the market works and are not discouraged by complexity in placing their bets. Also, middle managers will be more willing to reduce their role as information gatekeepers if they are given a persuasive explanation of the value of the prediction market.
