The Benefit of User-Testing a Prototype

I’ve been wondering if it’s possible to quantify how “useful” it is to user-test a game’s playable prototype before full-scale development begins in earnest. I’m defining a “user-test” as “spending a few weeks observing and interviewing a group of twenty or thirty diverse consumers as they play your prototype.” Unfortunately, a really good analysis requires data that I doubt anyone has access to: the success rates of roughly equivalent games, that were and were not tested at the formal prototype stage, and were released into approximately equivalent market environments. Among other figures.

Given that I’m unlikely to see such numbers anytime soon, I had to settle for a simple experiment. I created a very basic excel model that calculates the average return for a game project that has been carefully tested (and potentially killed outright) at the end of prototyping, versus the average return for a game that semi-automatically proceeds from prototype to full development. A visual representation of the model (decision tree) can be viewed below:

It is important to note that in both cases (“testing” and “no testing”), you need not assume that the prototype is either greenlighted or killed, and nothing in between. The model permits you to assume that in both cases, changes to the prototype may occur. The cost of those changes is included in the total development cost. The real question is this: How much would user-testing need to increase your odds of success in order to be “worth it”?

There are several large assumptions built into my model, and several large market factors that I have ignored entirely. For example, I assume that the average next-gen title with a healthy budget will “fail” 80% to 85% of the time. Failure is defined as $8M in sales. These are obviously very, very rough guesstimates. My model does not take into account marketing costs, platform fees, inflation, team morale effects, etc, etc. I tried to include those factors, but it cluttered the model without substantially altering the outcome. Suffice to say, please don’t try to scrutinize this with a microscope. It is not intended to be perfect, or even close. It simply want to start a conversation, and perhaps get a glimmer of insight into an important question.

Anyway, onto the good stuff. My base assumptions: $400K prototype cost, $30K testing cost, $10M full dev cost, 20% chance of totally killing the project after prototype testing, 20% chance of success with the help of testing feedback, and 15% chance without (in other words, a very conservative success rate increase of just 5% with testing). Success breeds $100M in sales, Failure breeds $8M. You can change these numbers in the excel file, and the outcome values will automatically be updated. Go ahead and play with them.

My results: user-testing, on average, results in ~$1M greater sales. This figure skyrockets if you assume that testing is worth more than just a 5% increase in your odds of success. On the other hand, if you decrease the development budget and/or increase “failure case” sales by any significant percentage, testing ceases to be worthwhile. Assume a much higher base level of success (in all cases), and testing also ceases to be wortwhile.

One interesting (if obvious) takeaway: as development budgets increase, testing your prototypes on potential customers becomes an ever more worthwhile risk-limiting device. And regardless, if you believe that user-testing in relatively early stages of development could boost your chances by even just 5% or 10%, you really should consider doing it.