Wisdom, Deferred

Back in August 2007, I wrote the following about the XBLA 1st party (aka MS-published) title review process:

I’ve put systems in place to hopefully help reduce the risk of my own tastes (or lack of vision) from polluting the portfolio. I can’t really discuss the details, but they include a sort of “wisdom of crowds” feedback loop, in which indie submissions are screened and rated by a group of my colleagues within Microsoft (who are asked NOT to discuss the submissions with each other before rating them — mainly to avoid group-think.) The wisdom of crowds can make my forecasts more accurate, and it can help compensate for any subconscious biases I have. Unfortunately, what I don’t believe it can do is help me identify future mega-hits (i.e. “the next Geo Wars“.)

The process I referred to in that post actually kick-started in March of 2007, so it’s been about 18 months since I started changing the way that the XBLA team reviews incoming submissions from independent developers. If you had asked me way back then when I expected to be able to judge the results of the process change, I would have said “a year from now, at the latest.” Turns out, I would have been way off. A year and a half later, only six games that we have reviewed under the new process have launched on Marketplace. Many of the games greenlit in the few months following March ’07 are still in development and/or finally nearing release.

How long does it take to make an XBLA game, and why?

When I first joined the XBLA team, the conventional wisdom was that most games would take six to nine months to go from greenlight to launch. Today, it’s clear that the range is much wider — anywhere from four months for simpler ports (developed by experienced studios) to well over two years for an original game like Castle Crashers. The average development time has obviously risen, though average dev team size does not appear to have risen so much.

I’d attribute some of the rise in development times to the increased size limit for XBLA games, but the much more significant factors seem to be:

1. There are now many more studios that view XBLA as a venue via which they can make a name for themselves, promote original IP, and generate a non-trivial return on their investment, as opposed to in the early days when participation in XBLA seemed more opportunistic. Many of the studios that we’re working with now are highly motivated to create a standout game.
2. There is now much more competition, in many forms. When XBLA first launched, LIVE users had relatively few options competing for their cash — on Marketplace or in retail, for that matter. But today, Marketplace is swimming with games, DLC, and demos, and retail is overflowing with compelling new and used games. The bar has risen tremendously, and XBLA developers have had no choice but to rise with it — which means putting more time and energy into their games. This seems like the most significant factor, to me.
3. Developers are becoming more and more ambitious in their use of LIVE (both in terms of multiplayer modes, and in terms of stuff like creative use of leaderboards, content sharing, etc.) As any experienced developer will tell you, online features — especially LIVE multiplayer — tend to greatly increase the complexity of a development project. One of my colleagues believes that adding robust LIVE multiplayer mode(s) to a game can increase the development schedule by 50% to 100%.
4. The certification process has often proven to be more of a challenge than many independent developers expected, despite the significant assistance our production and test teams offer in preparing for and managing that process. Developers who have gone through certification once have much less trouble the second time around, but we’re still working with plenty of developers who are new to console game development.

What have those six games taught me?

I don’t like to draw conclusions from six of anything — I don’t consider that a statistically significant pool of information. However, it seems that I was right when I said a “wisdom of crowds” process would not be useful in predicting mega-hits. One of the six games I referred to earlier was Braid. The average participant in my review process predicted that Braid would not be one of the top 50% of games on XBLA, by sales. Only two people predicted that it would be in the top 25% of games!

What I really want to know is if the process helped me avoid red-lighting games that were likely to perform reasonably well (not necessarily become mega-hits) — in other words, if it helped me overcome my own personal blind spots and biases. I’ll need to wait a bit longer to find out, but that’s OK. There’s one thing this job teaches you, above and beyond all else. Can you guess what?

Patience.  😉