Wisdom, Deferred

Back in August 2007, I wrote the following about the XBLA 1st party (aka MS-published) title review process:

I’ve put systems in place to hopefully help reduce the risk of my own tastes (or lack of vision) from polluting the portfolio. I can’t really discuss the details, but they include a sort of “wisdom of crowds” feedback loop, in which indie submissions are screened and rated by a group of my colleagues within Microsoft (who are asked NOT to discuss the submissions with each other before rating them — mainly to avoid group-think.) The wisdom of crowds can make my forecasts more accurate, and it can help compensate for any subconscious biases I have. Unfortunately, what I don’t believe it can do is help me identify future mega-hits (i.e. “the next Geo Wars“.)

The process I referred to in that post actually kick-started in March of 2007, so it’s been about 18 months since I started changing the way that the XBLA team reviews incoming submissions from independent developers. If you had asked me way back then when I expected to be able to judge the results of the process change, I would have said “a year from now, at the latest.” Turns out, I would have been way off. A year and a half later, only six games that we have reviewed under the new process have launched on Marketplace. Many of the games greenlit in the few months following March ’07 are still in development and/or finally nearing release.

How long does it take to make an XBLA game, and why?

When I first joined the XBLA team, the conventional wisdom was that most games would take six to nine months to go from greenlight to launch. Today, it’s clear that the range is much wider — anywhere from four months for simpler ports (developed by experienced studios) to well over two years for an original game like Castle Crashers. The average development time has obviously risen, though average dev team size does not appear to have risen so much.

I’d attribute some of the rise in development times to the increased size limit for XBLA games, but the much more significant factors seem to be:

  1. There are now many more studios that view XBLA as a venue via which they can make a name for themselves, promote original IP, and generate a non-trivial return on their investment, as opposed to in the early days when participation in XBLA seemed more opportunistic. Many of the studios that we’re working with now are highly motivated to create a standout game.
  2. There is now much more competition, in many forms. When XBLA first launched, LIVE users had relatively few options competing for their cash — on Marketplace or in retail, for that matter. But today, Marketplace is swimming with games, DLC, and demos, and retail is overflowing with compelling new and used games. The bar has risen tremendously, and XBLA developers have had no choice but to rise with it — which means putting more time and energy into their games. This seems like the most significant factor, to me.
  3. Developers are becoming more and more ambitious in their use of LIVE (both in terms of multiplayer modes, and in terms of stuff like creative use of leaderboards, content sharing, etc.) As any experienced developer will tell you, online features — especially LIVE multiplayer — tend to greatly increase the complexity of a development project. One of my colleagues believes that adding robust LIVE multiplayer mode(s) to a game can increase the development schedule by 50% to 100%.
  4. The certification process has often proven to be more of a challenge than many independent developers expected, despite the significant assistance our production and test teams offer in preparing for and managing that process. Developers who have gone through certification once have much less trouble the second time around, but we’re still working with plenty of developers who are new to console game development.

What have those six games taught me?

I don’t like to draw conclusions from six of anything — I don’t consider that a statistically significant pool of information. However, it seems that I was right when I said a “wisdom of crowds” process would not be useful in predicting mega-hits. One of the six games I referred to earlier was Braid. The average participant in my review process predicted that Braid would not be one of the top 50% of games on XBLA, by sales. Only two people predicted that it would be in the top 25% of games!

What I really want to know is if the process helped me avoid red-lighting games that were likely to perform reasonably well (not necessarily become mega-hits) — in other words, if it helped me overcome my own personal blind spots and biases. I’ll need to wait a bit longer to find out, but that’s OK. There’s one thing this job teaches you, above and beyond all else. Can you guess what?

Patience.  ;-)

17 Responses to Wisdom, Deferred

  1. You seem interestingly complacent, even blithe, about this tortuous bureaucratic process. A two-year approval process — doesn’t that, in itself, strike you as a danger sign? If not, what about three years? Five years? Say, I forget, how long was the development cycle for Windows Vista?

  2. Allen — you’ve misunderstood. Some games have a two year *development* process. Not approval process. As in, they take two years to *make* after they’ve been approved. :-)

    Update: Oh, it just occurred to me that you might think most indie games are generally proposed to XBLA in a finished state. They aren’t — not even close. They’re generally text proposals with mock screenshots and/or video, and an early prototype if we’re lucky.

  3. It’s really hard to predict mega-hits. Too many factor to take into accounts, some that are not even related to the game itself.

    Timing is one big modifier (what would have happen if Braid would have been released along side GTA or Halo?).

    Presence aka: marketing (John did a really great job about marketing his game, ensuring it was known to most).

    Flavor of the month (sometimes, great title just don’t sell as well as expected even though they are praised by all).

    But I do think the process (having a board instead of one person) is a good way to ensure you have more variable to help you judge a title.

    Pat

  4. On the range of development times, two years is a big underestimate. I know that Castle Crashers has been in development at least as long as Braid (they first showed Castle Crashers at Comic-Con 2005) which puts both games at about 3.5 years in development.

Leave a Reply