What it means to be a critic, one's approach to being critical, and how that relates to the larger idea of "criticism" versus a traditional "review" are topics that any writer or developer will give you endless opinions on.
Mass Effect 3 senior designer Manveer Heir and I have been exploring this idea, a conversation prompted by the vocal response to Simon Parkin's Uncharted 3: Drake's Deception review, which served as a review of Uncharted 3 and the entire Uncharted series.
The response was overwhelmingly positive last week, and that makes me awfully happy. You'll see more of this--promise.
In part two, we contemplate whether dropping review scores, which some publications have tried, could be a potential solution. Hmm.
Without further delay, here's the second part of our conversation. If you missed part one, read it here.
Note: This exchange took place over email, and I've done minimal editing to reflect the casual style.
--
Patrick,
The middle ground is simple: we drop review scores. Sorry, I learned my negotiation tactics from the Republicans. But in all seriousness, I think when it comes to reviews you may be right that publishers put a big emphasis on the reviews on their own accord, but publications are the ones that created the overinflated scoring that plagues the industry. My suggestion to fix that is simple. All review scores go to five stars with no half-star given. Make sites make a call on how good/bad a game is and put their name on it. A five star system will not lead to inflation. Anything more than five review points will naturally lead to inflation. You see it on sites that rate out of 10. You definitely see it on sites that rate out of 100. Of course, your industry won't ever change to this in the same way mine won't stop using Metacritic as the metric for all things.
Also, saying that publishers put an emphasis on review scores, while correct, doesn't address the fact that sites still have an obligation to make sure their information is being used in a responsible manner when possible. This means, if a site disagrees with its reviews being used on Metacritic, it should get them pulled from the site or make changes to how its scores are interpreted. I'm not sure how possible that is, but I have to imagine a site can ask to be taken off Metacritic, right? In college football, which I am a big fan of, the AP Poll asked to be taken out of the convoluted BCS formula that attempts to match up the two best teams at the end of the year in a national championship game. They did this because they disagreed with how their poll was being used, and because they understood that by letting the BCS use their poll, it made them, in fact, complicit with the problems that the BCS represented in it's fallacious attempt to match up #1 and #2. This is the part where I would quote Peter Parker, but that's become very hackneyed lately, so I'll just say I think responsibility is on both sides here.
But this whole path would be painful for me as a developer. If, all of a sudden, all reviews were out of five and three was average and everyone adhered to this, a 60% score would be average, instead of 80% as it currently is. That would crush developers/publishers, and customers wouldn't know how to parse the new scores. We've crafted a bed of nails, and now we are lying on it. To try to get up from this will be incredibly painful and difficult.
I think trying to change the system we currently have is incredibly difficult. There is inertia working against us. People will have a hard time adjusting, from the writers, to the developers, to the publishers, to the readers/players. This just creates chaos and will take long to occur, and, as you suggest, not happen anytime soon, if ever. So what's the real middle ground? I think it's to create something wholly new, different. Something that doesn't have the preconceived notions and expectations that reviews have. And I think that thing is called criticism. Yes, I'm repeating my earlier point, but it's because we've pointed out why the "good fight" here is so futile. Our only options are to do something different or just be ok with the status quo. I, for one, am absolutely not ok with the status quo. Not as a reader, and not as a developer.
If we introduce critical analysis, we can train our readers on something new. We'll still have these review things that they want and read, but we'll slowly start putting more emphasis on the criticism. The discussion of authored narrative vs. player-driven. The discussion of the role of modern military shooters in our world. The rise of social networking and connectivity among our games. These are interesting topics, and there are multiple sides to the discussion and these topics can be framed by using specific games to make a persuasive argument. I want to read that. I bet many writers want to write that. And if we offer it to our consumers, I think there will be an appetite for it. But there will not be an appetite for it if we don't offer it. Readers, like players, don't know what they want. They just want what they currently like; they have little vision for the future. If you asked a player after Call of Duty 3 what they wanted, they would have wanted Call of Duty 4 to be more World War II awesomeness. It took someone with vision to take that game to the modern day. And you know what? It seems crazy to think this now, but there was resistance against that and it took guts to make that move. We need an equivalent move amongst the journalists.
So let's not get rid of reviews. But let's offer a side dish of real pointed criticism. And over time, let's make that critical side dish bigger and bigger and more prominent. And, you never know, one day maybe it'll become bigger than the reviews itself. That's the middle ground to me. Something new. Or maybe you have a different idea for something totally new we could do to fill this gap?
--
Manveer,
Why don't you chill out, Gingrich? Review scores aren't going anywhere, just like Metacritic--or an equivalent aggregator--isn't disappearing, either. I'm with you on the five-star scale, though, which is what we have here at Giant Bomb. You can opt out of being listed on Metacritic, which is what Adam Sessler did rather publicly for both G4tv.com and X-Play, after criticizing the organization at the Game Developers Conference a few years ago. Adam is actually a bit like Hulk when he gets angry.
And you're right that a publication should opt-out of Metactitic if they're being misrepresented, but it's no secret that publishers are less likely to provide a publication with review code for a game ahead of release if they are not on Metacritic..
There are few things more frustrating than flipping through a publication with a review scale that goes from 0 to 100. Ugh, ugh. I've never regularly written for a publication that asked me to review a game on that scale, and I can't imagine it, either. If someone can tell me what the difference is between a 72 and a 73, I'd love to know. It makes the reviewers job more difficult, and does nothing to help the developer, an entity whose role is often forgotten when it comes to the job of a review.
I don't believe it's the job of a reviewer to take things like "this may put the developer out of business" when considering a review, but a good review should also serve as a guide for a developer to understand what did and didn't work. A review has failed when the writer becomes a backseat developer, using their soapbox to wax on and off about their design "insight."
And while the battle for "hits" is perhaps another conversation altogether, it's worth noting. Reviews drive an enormous amount of traffic to gaming web sites, and part of that traffic is driven by the score, especially if it's at one end of the scale or the other. That's not to suggest publications are skewing scores in order to generate traffic--the amount of conspiracy theories about the gaming media with any evidence is laughable--but there's financial motive to keep reviews a center piece.
The impasse we're finding ourselves at, I believe, is whether a "review" can also function as "criticism." Games For Windows Magazine (aka Computer Gaming World) dropped review scores for a short period, hoping to force readers to spend more time contemplating the text. They ended up ditching this plan, as people stopped reading the reviews as much. In an experiment under John Davison's leadership, GamePro stopped doing print reviews, and instead aggregated the more timely online reviews to provide an overall perspective of the response to a particular game. That was also eventually dropped.
I'm not advocating that we shouldn't challenge what the reader wants, but that there are expectations. As a developer, you do the same, awkward dance with players. As a reporter, I've constantly pushed against the notion that we should always give the reader what they think they want. If we applied the same philosophy to food, everyone should be a-okay with McDonalds and shut up, simply because it's popular. You demand more when you know what to demand, and it's up to us to provide.
Let me bring around the question to that prompted this dialogue. Are you so opposed to "criticism" being part of a "review" because it's implied that criticism will, inherently, skew more negative? Maybe it's a problem with the label itself. Review come from a weird place where games used to (and, in some cases, still are) broken down by graphics, sounds, gameplay, which is completely unfair, crazy, and should stop. Criticism, however, sounds negative. In the case of Parkin's Uncharted 3 review, he ended up giving the game an 8/10, but as most of his text was spent critiquing Naughty Dog's philosophical approach, he pissed off two sects of people: those who wanted a higher score, those wondering why it wasn't way lower.
It's clear that reviews are in a transitional state, and maybe that's the real problem--it just needs more time?
Look for the final part of our conversation tomorrow, which means I have to work on my day off. Crap.
148 Comments