The Passion Trough and the Ho-hum hump

Seth Godin has an interesting post up about weak analysis of data - in this case, a poor reading of data by a NYTimes blogger's analysis of Kindle review data on Amazon. I think it holds some lessons for the games industry, if you'll bear with me.

Its worth reading both posts, but here's the synopsis: Analysis of the reviews of all three Kindle products (breaking them down into a pie-chart of 1-5 star ratings) shows an increase in the percentage of one-star ratings over time. Conclusion: Kindle customers growing more dissatisfied over time.

There are a number of reasons this is erroneous. Many 1-star reviews are by non-owners ("I'll never by Kindle because..."), the early adopters more passionate than others, they are different products, etc.

However, the more interesting thing to me were these comments by Seth:

Amazon reviews never reflect the product, they reflect the passion people have for the product. As Jeff Bezos has pointed out again and again, most great products get 5 star and 1 star reviews. That makes sense... why would you be passionate enough about something that's sort of 'meh' to bother writing a three star review?

...

The Kindle has managed to offend exactly the right people in exactly the right ways. It's not as boring as it could be, it excites passions and it has created a cadre of insanely loyal evangelists who are buying them by the handful to give as gifts.

I think the lessons here are to Ignore graphs intended to deceive, and to understand the value of the negative review.

Point being that the negative reviews have value as well. For one thing "there's no such thing as bad publicity" (not true of course, but there IS a downside to NO publicity at all. The sound of crickets chirping is not accompanied by the sound of cash registers ringing). Another thing is that the negative reviews let you know who *are not* your customers.

So, what's this got to do with games?

Well, the industry puts some stock in review aggregators like Metacritic, and others are claiming this may not be indicative of a game's potential sales.

However, Seth's post made wonder whether we're looking at the right thing. Take the following fictitious graph:

The vertical axis represents number of reviews, and the horizontal axis represents 1 through 5 star ratings. Series A represents what I call the "passion trough" - reviews polarized toward 1 and 5 star ends of the spectrum (Seth's point about passionate reviewers). Series B represents the opposite, what I call the "Ho Hum Hump" - reviews clustered in the 'meh' range. Each of my fictitious products get 150 reviews

So, which is preferable?

Well, for one thing, it depends what you consider a "3" to mean. If that's a passing grade, then series B is preferable - two thirds of people gave you a passing grade. Series A gets only just over half.

Traditional thinking would be aiming to satisfy this. Do the best you can, for everyone - even if it costs you some of the more passionate customers. Better a 3-star with everyone than a 5-star with only a few people. (Some of the tradeoffs we've seen to 'mainstream' titles might lead you to call this the 'compromise chasm' :-)

I think this would be the wrong conclusion though.

For one thing, per Seth's point, I'm guessing the reality would be that Series B would get far fewer reviews, all other things being equal. It inspires little passion in people. Whereas A is more likely to inspire reviews - both good and bad.

Secondly, For Series A, on third of the reviewers are VERY passionate about the product, and therefore perhaps likely to buy it. For Series B, all those people giving it the middle of the road review are also people with a lot of alternative products to choose from.

Someone will need to crunch the numbers to determine if the above is indeed the case. If I'm right though, then we're looking at the wrong thing by looking at average score. We should be looking at standard deviation, total number of reviews, etc - if looking at Metacritic at all. Not to mention looking at user reviews vs press reviews, but that's a whole other topic.

My gut tells me you are way better off with the trough than the hump.

Correlation vs Causation, and the MetaCritic MetaQuestion

This piece on Gamasutra offers an interesting take on the metacritic issue, concluding that review scores are among the least important of the factors affecting game sales.

The same subject came up at a round-table discussion at MIGS that was lead by EEDAR's Jesse Divnich.

An interesting snippet from the Gamasutra piece that is worth chewing on a little:

Analyst Doug Creutz says "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game"

This highlights a point I brought up at the MIGS round table: That there's a difference between correlation and causation. Scores can be correlated to sales, but not necesarily affect them.

The correlation is fairly straight forward. Most game reviews are written by reviewers who fit the mold of the "typical gamer" if there is such a thing. A high meta-critic score is a small sampling of people who fit the demographic of the customer. "9 out of 10 people gave this game a thumbs up". These reviews serve as this indicator *even if not a single consumer ever reads them*.

Now, whether the average consumer consults these reviews and uses them to decide on a purchase of one game over another, and how that factors in versus everything else vieing for their attention is another matter. I have no idea whether there is any causation here, but it is certainly a more tenuous assertion than the correlation above.

Does it matter though? Of course, and here's why.

If you beleive in the correlation, then you can use meta-critic as an indicator of sorts. The publishers seem to be doing this, and there has been plenty of talk about developers having bonuses tied to MC scores and the like.

Now, using carrots'n'sticks motivators for developers, and tying those to MC scores is being done to drive behavior, I assume. It is essentially the publisher telling the team "Please go do what it takes to acheive 90% or better".

If you beleive only in the correlation, that the reviewers are essentially a sample group of gamers, then you focus on building a great game that they and everyone else will find enjoyable. [A cynic like me would say you also focus on building a marketing frenzy that will have everyone salivating for the title, so that reviewers are ready to write their 98% review before they've laid hands on the game - but again, you are doing nothing for the reviewers that you wouldn't do for the consumer as well]

But if you beleive in causation, then you focus part of that effort in gaming Meta-Critic itself. You go out and try to influence reviewers, beleiving that gaming a high score out of the system will result in high sales.

So the meta-level question about metacritic is whether you beleive it serves as a focus group, or as a marketing tool. I beleive its the former, but choose your own opinion and proceed accordingly.

Addendum: As I was writing this I had an interesting epiphany: If viewing MC as a 'focus group' of sorts, then it would be interesting to treat as such. Do games score extremely high with a subset of the focus group and low with another? And if so, how do those fair vs those with a more homogenous set of scores. Does an 80 MC title with scores ranging from 60-100 fair better or worse vs one with scores ranging from 75-85. In short, does MC standard deviation indicate something? Hmmm.... time to curl up with Excel and a glass of wine...

Another big game, another set of reviews questioned

Games journos are calling attention to themselves again and questioning the value of game reviews that are rushed to print in order to scoop the competition.

We saw a rash of this conversation around the GTA4 release. As part of that, it was implied that the game's publisher further egged it on by incenting higher review scores by restricting allocation of pre-release copies of the game, etc.

This time around, it's reviews of Little Big Planet. The games servers were down for a while, so arguably the reviewers were reviewing the game without looking at some of it's most important features.

Kotaku discusses the topic here, once again showing that the Brians are capable of seeing the big picture & implications.

[Kotaku's serious side aside, am I the only one that thinks that their renaming Epic's Cliffy B to "Dude Huge" is one of the funniest things on the intertubes?]