It’s taken a while, but I want to return to my article on using games to tap the wisdom of crowds. First and foremost, I’d like to bring attention to the writings of Raph Koster. Raph informed me that he’s been thinking about this idea as well for quite some time now. However, Raph one-upped me: he found someone who actually tested the theory! From Raph’s blog:

What [Byron Reeves] showed was a mockup of a Star Wars Galaxies medical screen, displaying real medical imagery. Players were challenged to advance as doctors by diagnosing the cancers displayed, in an effort to capture the wisdom of crowds. The result? A typical gamer was found to be able to diagnose accurately at 60% of the rate of a trained pathologist. Pile 30 gamers on top of one another, and the averaged result is equivalent to that of a pathologist — with a total investment of around 60-100 hours per player.

Remarkable stuff! Raph also happened to blog about my article. Some of the comments on his post are worth drawing attention to:

Crowd intelligence can fail (and fail spectacularly) when there’s too much information passed between members of the crowd. Members start to alter their opinions based on the opinions of others, which skews the results. The online communities that build up around any popular game would seem to promote exactly this kind of skew.

An excellent point. Herd mentality can indeed disrupt the wisdom of crowds. However, the online communities that build up around games are not necessarily too different from the online communities that build up around publicly-traded equities (ever visit the forums at finance.yahoo.com?) Active discussion communities are not inherently bad. That said, there are also ways to negate “excessive” information sharing. Some examples:

• Use competition to discourage group-think. The scope of information-sharing is typically more limited when players (in any game genre) are working to best other players. Of course, blocks of information-sharing players will still form (in formal teams or otherwise) but that’s not necessarily a critical problem.
• Online game communities typically form (the most persuasive) opinions about the objective aspects of a design mechanic; i.e. “you’re better off using the shotgun than the pistol, except when you’re fighting at a great distance.” But if a challenge and its feedback mechanism both incorporate real-world data, as I suggested in my earlier article, it becomes harder for any individual (or the community as a whole) to form clear strategies around.
• Encouage population diversity to decrease the likelihood of groupthink. Distributing a game in different countries and courting players of different ages are both examples.

The subject of group-think probably deserves an entire post to itself, but you get the idea.

Game gods can’t help but always build into the warp and woof of every game their ideal vision of How Things Should Be. Their particular take on things, their demographics, class, experience, education, etc. all skew the very platform that you’re supposed to be using for some supposedly open-ended search.

A rather cynical comment (and one that ignores the entire range of “non-game-gods” — such as informed academics — who are trying to make serious games). In response, I’ll simply re-iterate: you can avoid many problems by incorporating real-world data into a game’s feedback mechanisms. If a game uses real-world, real-time data to judge the merits of a player’s actions, it should be less susceptible to simulation problems.

You can aggregate guessing, common sense, opinion and these are valuable insofar as they relay information, but it should not replace expertise and specialization. There is granularity to every subject matter that requires well trained decision making processes. Aggregated opinion says nothing and is invalid without applied knowledge making it actionable.

I think the results speak for themselves: when you combine players’ efforts, their accuracy equals that of the pathologist. Get a few hundred thousand players working on this problem (perhaps without even realizing it) and you’ve got the greatest and most cost-effective “second opinion” system in the history of health care — at least. And remember: a few hundred players aren’t susceptible to individual problems, like, for example, your doctor’s nasty divorce proceedings the morning of your examination.

And, from my original post, two comments by Ben Mattes:

A big reason games like SimCity can be as accessible (and successful) as they are is that they take the complex (city planning) and streamline it to a much more manageable level. Anything that isn’t “fun” about running a city is left out of the game (as it should be).

No arguments here. SimCity (as it is currently designed) wouldn’t be suitable for tapping the wisdom of crowds, and the game probably wouldn’t be fun if every aspect of it were modified to incorporate substantial amounts of real-world, real-time data. That’s why I think it makes more sense to focus on shaping just one aspect of an otherwise purely entertainment-centric gameplay experience (i.e. the cancer screening in Star Wars Galaxies, or weather prediction in Roller Coaster Tycoon.)

Would you really feel comfortable proposing a new zoning restriction for LA (taking into consideration all of the political, financial, moral and ethical issues) based off of the group-think of even a million SimCity gamers?

No, because I doubt I could design a (fun) game that would feed players all the relevant real-world information they’d need to make an informed decision. Too many variables. Maybe Will Wright could pull it off — he’s much smarter than I am. ;-)

I suppose the bottom line is, it’s easy to imagine lots of problems that you can’t solve by trying to tap the wisdom of crowds via games. Problems with no “correct”, objective answer (i.e. “does God exist” or “should abortion be legal?”) Problems that are impossible to communicate to the average person in a reasonable amount of time (i.e. PhD-level mathematics equations). Etc. But there are problems that, while quite complex, are possible to communicate to the player, along with a reasonable amount of real-world support data (i.e. cancer scans.) That’s what I’d like to focus on.