Criticizing Metacritic

What’s a “Metacritic”?

Over the past couple of years, Metacritic and the scores it hands out to today’s games have become very important to us as an audience, and like anything that gains a little traction, the website has since become a lightning rod for controversy both within and outside of the industry. 

Some say the website is a blessing and that the simplified verdicts it hands out serve to cut through the crap of the review scene. Some say it’s a curse, that it generalizes without transparency, that its system is fundamentally flawed. Still others know relatively little about it, hearing the many voices of dissent as more of a distant thrum. Up until recently, I belonged to that third group; though I knew of Metacriticand its controversies, I hadn’t done much by way of actually educating myself on the subject. A couple of days ago I set out with the intent to correct that little problem and recorded what I found. For anyone interested, here it is. 

Reviews? Who Needs Reviews?

Reviews are a writer’s expert opinion on game X after they have spent enough time with it to feel confident about their conclusions. Review scores are that little number (or letter, or other symbolic measurement icon) floating next to a game’s review on any given website or in any given magazine, and the system that governs these little numbers is referred to as the numbered review system. Now despite their size, these little digits and symbols are remarkably important and have caused a slew of heated debates throughout our industry’s history with developers, publishers and critics alike all voicing their discontent with the system at some point or other. In fact, some publications are so fed up with the idea that they’ve thrown the system out entirely, opting to try to coerce their readers into actually reading their reviews by not offering a “one sentence sum-up.” 

Developers have sunk or swam by this murky science for years and reviewers have come into conflict or lost their jobs all because of a single digit number at the bottom of a page. Now, if you’re already thinking that this system could use some fool-proofing (though some would suggest the scorched earth approach), you’re probably right. What it doesn’t need is another entirely superfluous and equally flawed system heaped on top of it, right?

Enter Metacritic. 

Big, Broken Tent

To the average person, Metacritic gives the appearance of being an all-inclusive think tank, a repository for every review out there. But this average person might also suspect pulling in every review from the edge of eternity for every game under the sun to be an impossible task, with the latter impression being closer to the truth. Metacritic does not compile scores from every publication, or even close to every publication. Instead, Metacritic claims to pull only the best reviews from the most respected reviewers at the most prestigious publications via an undisclosed algorithm formulated in-house. So just to clarify, who exactly determines the people and places that belong to the aforementioned and flattering adjectives?

Well, Metacritic does. Oh, and they’re not saying how. 

Not only does Metacritic pick and choose which scores to pull, but if it sees a review without any kind of scoring mechanic attached to it, it will attach its own score and pull that review anyway. At the face of it that might not seem like such a bad thing, but when considering the site’s growing influence within the industry and its complete lack of transparency in operation, its terrifying. Metacritic has set itself up in such a way that it can, at any time, directly influence a game’s review score simply by pulling a review that does not conform to the numbered system and assigning one to it anyway. Coming from a website preaching ethics that claims to provide a service to which integrity is invaluable, doesn’t that seem like a rather large loophole? 

The bottom line is that no one but the men and women behind the curtain really know where Metacritic’s loyalties lie, and at this point they are quickly becoming one of the most influential sources of information in a fifty-billion dollar (and growing) industry. So the question is, why isn’t anyone checking up on them? 

I Like Where This is Going…

In any situation, many voices are better than one voice. The numbered review system, as it exists now, is flawed but serviceable. Metacritic is dangerous because it threatens to overburden that system with its own (again, superfluous) weight and silence the many voices of reviews in favor of just one: its own.

Thanks for reading,
End_Boss.

What’s a “Metacritic”?

Over the past couple of years, Metacritic and the scores it hands out to today’s games have become very important to us as an audience, and like anything that gains a little traction, the website has since become a lightning rod for controversy both within and outside of the industry. 

Some say the website is a blessing and that the simplified verdicts it hands out serve to cut through the crap of the review scene. Some say it’s a curse, that it generalizes without transparency, that its system is fundamentally flawed. Still others know relatively little about it, hearing the many voices of dissent as more of a distant thrum. Up until recently, I belonged to that third group; though I knew of Metacriticand its controversies, I hadn’t done much by way of actually educating myself on the subject. A couple of days ago I set out with the intent to correct that little problem and recorded what I found. For anyone interested, here it is. 

Reviews? Who Needs Reviews?

Reviews are a writer’s expert opinion on game X after they have spent enough time with it to feel confident about their conclusions. Review scores are that little number (or letter, or other symbolic measurement icon) floating next to a game’s review on any given website or in any given magazine, and the system that governs these little numbers is referred to as the numbered review system. Now despite their size, these little digits and symbols are remarkably important and have caused a slew of heated debates throughout our industry’s history with developers, publishers and critics alike all voicing their discontent with the system at some point or other. In fact, some publications are so fed up with the idea that they’ve thrown the system out entirely, opting to try to coerce their readers into actually reading their reviews by not offering a “one sentence sum-up.” 

Developers have sunk or swam by this murky science for years and reviewers have come into conflict or lost their jobs all because of a single digit number at the bottom of a page. Now, if you’re already thinking that this system could use some fool-proofing (though some would suggest the scorched earth approach), you’re probably right. What it doesn’t need is another entirely superfluous and equally flawed system heaped on top of it, right?

Enter Metacritic. 

Big, Broken Tent

To the average person, Metacritic gives the appearance of being an all-inclusive think tank, a repository for every review out there. But this average person might also suspect pulling in every review from the edge of eternity for every game under the sun to be an impossible task, with the latter impression being closer to the truth. Metacritic does not compile scores from every publication, or even close to every publication. Instead, Metacritic claims to pull only the best reviews from the most respected reviewers at the most prestigious publications via an undisclosed algorithm formulated in-house. So just to clarify, who exactly determines the people and places that belong to the aforementioned and flattering adjectives?

Well, Metacritic does. Oh, and they’re not saying how. 

Not only does Metacritic pick and choose which scores to pull, but if it sees a review without any kind of scoring mechanic attached to it, it will attach its own score and pull that review anyway. At the face of it that might not seem like such a bad thing, but when considering the site’s growing influence within the industry and its complete lack of transparency in operation, its terrifying. Metacritic has set itself up in such a way that it can, at any time, directly influence a game’s review score simply by pulling a review that does not conform to the numbered system and assigning one to it anyway. Coming from a website preaching ethics that claims to provide a service to which integrity is invaluable, doesn’t that seem like a rather large loophole? 

The bottom line is that no one but the men and women behind the curtain really know where Metacritic’s loyalties lie, and at this point they are quickly becoming one of the most influential sources of information in a fifty-billion dollar (and growing) industry. So the question is, why isn’t anyone checking up on them? 

I Like Where This is Going…

In any situation, many voices are better than one voice. The numbered review system, as it exists now, is flawed but serviceable. Metacritic is dangerous because it threatens to overburden that system with its own (again, superfluous) weight and silence the many voices of reviews in favor of just one: its own.

Thanks for reading,
End_Boss.

I think most READERS understand Meta-Critic for what it is.

I appreciate review scores, and personally I think the average consumer would benefit more from a review score than if reviews did not feature them. The world revolves around scores, whether it be movies, video games, or even our school system. Everything is a numbers game.
 
 The reason I feel review scores help consumers is that not everyone has the time to read several paragraphs from numerous sources to decide whether or not a game is for them, and for that, I believe that sites like metacritic.com has it uses. However, scores do not translate well in metacritic, and because of that I feel rottentomatoes has a better approach to the way they scale films. So, while metacritic may help determine what games are universally loved,  a game with mixed reviews is not automatically bad. Having a site that compiles the reviews would help the consumer get a better sense of what he or she is getting into.
 
Finally, just because a game receives a high score on metacritic does not mean the game will appeal to me. It's also important for readers to recognize that.

I think I had this discussion a while ago, but I don't like review scores.

The defense most metacritic Luddites use is "they haven't steered me wrong yet" and the logic there is they're only buying games that score 9 or better. Except you're also not even trying games x, y, and z specifically because of their meta average. Generally speaking, more divisive games(as in games that wouldn't score as well) are geared towards more specific audiences. So really, if you're looking at metacritic you're missing out on a lot of niche games and only scooping out what has the widest mainstream appeal and largest advertising budget. So you're basically a casual gamer. And there's nothing wrong with that, but it is what it is.

Metacritic-related contract clauses are stupid. Publishers should be judging success by sales. Denying bonuses based on what a collection of random blogger say is just wrong. Every reviewer in the world would (probably) agree that they should no say in how much a developer should be paid. Not to mention, gaming reviews hold less value now than ever before. Unlike the old days of magazines and the monolithic GameSpot, reviewers are no longer gaming's gatekeepers. Games are incredibly varied now and cater to so many different tastes that no one has the final authority about the quality of a product. And like everything else, neither does metacritic and publishers should stop using it as one.
 
Having said that, I don't want metacritic to go away. I use it fairly regularly, along with reviews. Sometimes niche games don't get reviews on my followed sites and I've go to other places to find out more about a game. That's where metacritic comes in and directs me directly to the sites that liked the game. I get to know why or what they liked and it helps me in my buying decision. It's a valuable tool for the consumer that's being horribly misused by the industry.

Metaphorically think of it as a compass, it points the score in a more accurate direction. You get a general idea how good a game is or not where all these highs and low scores are like waves in the ocean. I depend on it because I don't trust any individual sites, statistically it gives me a general idea.   
 
Of course i'll probally get a million replies why i'm wrong, i'm used to it, I stand by metacritic and gamerankings forever.

the only real game reviews i trust are my own, when i finally play the game.  metacritic, like all review sites, should be taken more light-hearted.  high scores are always good to see for our favorite games but really they are there to confirm or contest our own personal bias. this, paired with the misconception of a 100 point scale and reviewer bias (gasp!), i feel causes most of the uproar towards review sites. 
 
@HitmanAgent47 - i support your ideas and would like to subscribe to a newsletter.

Fact: People are lazy. Most of them don't want to read long reviews. Scores are perfect for these kinds of people. 
 
I don't use Metacritic for the score, but to have a place to easily go to reviews for specific games. If I want to read IGN's review for Deadly Premonition, I'm not going to go to their website and go through 5 different pages. I'm going to go to Metacritic, find IGN, and hit a button.

You hit a lot of the nails on the head for someone out side the industry.  The developer/publisher clauses is a very big deal in the industry, and it extends to more than that.  Marketing & PR teams in general have the final metacritic score often as an indicator of their 'value' to the company.  Additionally, when Metacritic makes up a score they often get it deadwrong.  For God of War III they recently assigned a 100 score to a review that started out with the line "God of War III is far from perfect..." 

All the commentators so far make good points. As someone who uses GameRankings.com regularly, I personally don't want to see the aggregate review sites go away, nor do I think they need to be "kept in check" or have any kind of regulatory actions taken against them. When you start to influence review scores like that, you get the Kane & Lynch Gamespot fiasco that caused this site to happen. 
 
It's sad that these sites are becoming the final word on every new release and the sole factor in a fast majority of gamers' purchases. Personally, the only thing GameRankings has done is caused me to buy more games than I normally would. When I see a game score a 90 or above that I otherwise wouldn't be interested in, I have to wonder what everyone sees in this thing and concede that maybe I just need to give it another chance. Usually, this causes me to find a game I'm glad I didn't pass up. 
 
But I've been a gamer forever, and I know what I like. So I still bought Infinite Undiscovery and enjoyed it, despite it's 68% rating on GameRankings (which I only saw after beating the game). Did I care? No, I can completely understand why most people saw an underwhelming and stereotypical action-RPG. Gamers need to use these sites for what they are: a quick way to confirm or deny your preconceptions about a game, plus easy access to many different individual reviews. As an alternative, should gamers be buying a game based on a glitzy trailer and a single magazine review, only to find out it wasn't what they were expecting? That's what we had in the past, and obviously it wasn't working if this is where we are.
  
The biggest problem in games journalism isn't aggregate sites, but the numbers themselves. Look at RottenTomatoes.com, there are movies regulary scoring in the teens, with 50-60 being around average. Anything above a 90 is fucking rare. Now look at Metacritic and Gamerankings. All 80s, with the occasional 40 or 50 for shovelware. Are movies really so bad that people should be playing Mean Girls DS instead of seeing Clash of the Titans? More reviewers need to remember that the scale isn't 5 to 10. And so do gamers. If publishers are really starting to look at Metacritic as a payscale, then the problem doesn't lie with the websites or the developers, it lies with us. If you review games, be honest. And if you play them, give that weird RPG a chance. Fun can't be summed up in two digits.

All I use Metacritic for is to read the abysmal reviews, quite entertaining really.

Good read. Never really payed much attention to MetaCritic, so I suppose I'd be the third one too. After reading this, I understand just how influential it is and how dangerous and potentially dangerous it is and could become.

Metacritic does not work since in many journalistic outlets now there is not a numerical percentage score given to games during reviews. A 5* rating from Giant Bomb is NOT 100%, and on the same scale a 1* rating is not 20% (though in the case of Tony Hawk Ride 20% is being kind). And anyway Just Dance has a 47% rating on Metacritic and it is currently UK No. 1, so there is little major consumer actions taken towards these ratings.

The thing I hate about Metacritic is how they interpret review scores on a 100-point scale. Like, who are you to say that this Giant Bomb four-star review warrants an 86? I guess they read the review and try and get what number seems to fit the review the best, but on a 100-point scale, that is completely arbitrary. 
 
I think it should use one of the following: 

• Giant Bomb five-star system ("extremely good," "pretty good," "alright," "flawed," "poor," essentially)
• Roger Ebert's quick-review "thumbs system" (two thumbs up, one thumb up, one thumb down, two thumbs down)
• RottenTomatoes' "fresh or rotten" system (liked it or didn't)

Nicely done.

 Hey End_Boss,

I wanted to chime in to clarify a few things about our processes at Metacritic. Thanks for starting the conversation.  I was one of the founders of the site, and I've been the games editor from the start in January of 2001. I've selected all of the publications we've ever tracked on Metacritic, and if there is any degree of skepticism about Metacritic's process or our integrity, that lands on me, and I do my best to answer every email about our methods. Nobody at my parent company (we merged with CNET - now CBS Interactive in 2005) has ever asked me to add or remove a publication, or to change a publication's weighting. I take my job extremely seriously.

Metacritic is a site for the consumer - a tool to help you decide what game to buy, rent, or download, and a way for users to chime in about their own experiences with their games. Generally, if one game has a 90 Metascore and another a 70, it's an indication that the body of gaming critics thinks the first is the better game. You're correct that the gaming industry has embraced Metascores as a gauge of quality, but I have absolutely nothing to do with how publishers integrate our numbers into their contracts with developers, PR teams, etc. Again, Metacritic's mission is to help the consumer buy better games and avoid bad ones. And obviously some critics might have a different understanding about what an "80" means, and what 4-stars means, and some critics are tougher graders than others, but if I properly vet the critics I cover and they are consistent in the applications of their own scales, and they review a threshold number of games that are released each month, those publications have a "fair" impact on the Metascores,  and there is integrity in the ultimate Metascore as an indicator of quality. (We could have a much longer discussion on this point.)

We cover approximately 150 game review publications.  I've estimated that there are about 1,200 publications that regularly review games, so we're covering a relatively small slice of the whole. Choosing which publications to track is one of the most important elements of my job, and I spend a great deal of time reading reviews, questioning new sites about their processes, and re-evaluating the sites that I currently track in order to highlight reviews from the highest quality critics. Once we decide to pick up a publication, we track EVERY review it publishes, so long as we have a page for those reviews on Metacritic. So we are not picking and choosing reviews to post.  If IGN posts 15 reviews on a given day, all 15 of those reviews will be found on Metacritic within a reasonable period of time.  And yes, we do weight our individual publications based on how much they are respected in the gaming community, the strength of their writing and analysis, and their scoring integrity. Most of the same factors I employ in judging whether to cover a publication are used in establishing how much weight to assign that publication. Furthermore, if a publication scores its review, I have no discretion to change that score.  If GameSpot posts a 7.5, it shows up on Metacritic as 75. If GiantBomb posts a 4-star review, it goes up as 80 (straight mathematical conversion), and if GameRevolution posts a "C", it converts to 50.  Now you may disagree with the conversion of a "C" to 50 instead of 75, but we are consistent in our conversions.  Much is made about the fact that we reserve the right to assign as score to a review when it is unscored. We do this in other sections of Metacritic (Music, Movies, TV), but not in games. Every publication I currently track scores its own reviews. (For a year or two, I covered The New York Times and Variety in the games section, but even though they didn't publish scores on their reviews, I received scores directly from the critics, which I posted on our site. So there was no need for me to estimate their scores.)

Should scores from certain critics be weighted more than others?  Should the opinion of a skilled 20 year veteran of game criticism at a highly respected gaming magazine be worth more than a critic with less experience from which to draw his or her opinions?  We think so.  It's a perfectly valid opinion that all opinions should be worth the same, and there are many other aggregators out there who provide this type of service.  Again, we feel ours is the best method in offering our users the most sound advice about what games to play.

Best,

Marc Doyle
metacritic.com

Uh, wow. What an interesting response from Mark Doyle.

"  ...if GameRevolution posts a "C", it converts to 50.  Now you may disagree with the conversion of a "C" to 50 instead of 75, but we are consistent in our conversions. "

I just want to be the asshole in this thread and go:
 
Oh look, it's this thread again.

Metacritic can be compared to GPA 
 
- Both are numerical averages 
- Both have non-consistent bases: an art major is part of the same GPA system as a science major when applying to a faculty, just as a racing game is different from a first person shooter
- Both have inconsistent judges - students have different combinations of professors from the last student just as games have different combinations of reviewers from the last game
- Both are based on subjective criteria; the majority of GPA is determined ultimately by assignments, lab reports, short answers, long answers, presentations and essays, all of which are subjective
- Both are widely used as a go-to reference for quality
- Both are not all-powerful; a low-GPA student can be quite capable while a high GPA student can be a dunce, just as a low-scoring game might turn out good while a high-scoring game might be quite bad 
- Both are under constantly shifting standards of quality that are not fully defined

 
GPA scores and Metacritic scores are formulated almost under the same methodology, yet one is widely accepted everywhere while one is constantly bashed for being unfair or unrepresentative. Metacritic is not perfect but I don't think anybody ever purported it to be. It merely serves as a guide and it's also the best system we have to use right now to determine the general quality of a media product, just as GPA is the best system we have to determine the general quality of a student. What can be said is that a game which gets a high score is more likely to actually be good to more people than a game that gets a low score. It is a matter of basic statistics that the probability of an occurrence decreases the further the standard deviations go. Also, to suggest otherwise is to say that games get their scores by pulling numbers out of hats, not based on any reasoning whatsoever, which is plainly wrong otherwise you wouldn't be going to sites like GB for reviews.  
 
I'm glad that Marc Doyle wrote up a response to clarify some misconceptions about Metacritic, it gets a bad rap from many people who are too paranoid or ignorant to understand what it's actually about. 

Metacritic has its use, but I don't pay much attention to the aggregate scores. I like to see all the scores, or a good bit of them, in separation, like the Famitsu/EGM scoring method, where you see several professional opinions at once instead of an average of those opinions, which often waters down the true impact of the game's reception.  If a game gets a 9, 7, 7, and a 6, those numbers mean a lot more to me than baking it all together into an average 7.25 (or 73, as I guess it would be) and calling it a day. Why did it get a 9? Why did one person think it was excellent and another thought it was average? Does that really mean the game is average, as its aggregate score would say? Plus, the user reviews on Metacritic are atrocious. For games and movies. Everything's a 0 or a 10, and the opinions are poorly articulated and thinly backed.  A more focused site like this one is far superior for user reviews and opinions.

They could go the Rotten Tomatoes route and instead of giving an aggregate grade based on individual percentage scores they would simply deem either review as a recommendation or a condemnation (with probably a middle of the ground neutral score for conditional recommendations). Its generally easy to tell if a reviewer like or hates a game, and we wouldnt have to worry about any shady score conversion process. However, i think to a certain extent people are overreacting. Rotten tomatoes hasn't harmed the Movie industry and its quite common for very successful movies to have profoundly awful rotten tomato aggregate values. Developers know that good games do not necessarily mean big sellers. Im pretty sure they also take sales into account when determining how to reward employees.

"...it forces publishers to demand more from their developers, license owners to demand more from their licensees, and eventually, hopefully, the games get better. " 

"I have absolutely nothing to do with how publishers integrate our numbers into their contracts with developers, PR teams, etc "

@End_Boss said:

" @MarcDoyle: Perhaps we'd be surprised, perhaps we wouldn't. Unfortunately, as long as that algorithm (or whichever term you'd like to affix to it) remains private, us skeptics go nowhere. Still, I appreciate the courage it took to respond to my post. God knows internet forums aren't always the most hospitable of places. So thank you, again, for trying to shed a little light on the subject. "   

Nice blog endboss, got a nice discussion going on here as well.
 
For me reviews have always been highly personal. I like to know about the writer, know what they like and enjoy and if I can relate to their interests on a wider level. There needs to be a degree of empathy with the writer for a review to be successful. For example Vinny might give a favourable review to a dragonball Z game. As someone who listens to the bombcast i'm aware that Vinny is a Dragonball Z fan and that might, in some minor way, influence his review. The same review from Ryan would be completely different - as both writers have different personalities and interests. The point is that i'm aware of these conflicting opinions and I can relate to them and sympathise them - it's not as mechanical as "Vinny game this game a good review so I automatically go and buy it". In contrast have you ever noticed that sometimes your friend will tell you to go buy a game because "it's awesome" and you do even though it reviewed badly? Because they are your friend and you know them you can relate to why they enjoy the game. It's knowing when to trust a persons opinions.
 
To me metacritic has no sense of personality. There is nothing there I can relate to, no understanding of the dozens of voices that are being mixed together. It's too robotic to really provide anything useful. Metacritic needs to be subjective about which publications are chosen to source, which is perfectly reasonable. The line needs to be drawn somewhere otherwise every bugger with a blog starts showing up and rocking the boat. I can see a use for it, but to place such emphasis on something so mechanical and hollow seems unnecessary to me.