Why is Metacritic so bad?

Lots of people love to shit all over it and blah blah blah but I never got the reason.

and no, the awarding bonuses shit should not count. If people get into car accidents because they were texting on their phone does not make the cell phone a bad thing (probably a more extreme comparison than this warrants but WHATEVER). Metacritic is merely a aggregator of review scores, and if metacritic wasn't there, game companies could do it themselves if they really wanted to (or they could use the other one, gamerankings or whatever the fuck it's called). I like it because I like reading reviews on occasion. I'll go to that site, click on sites I... mildly trust, and it's easier than going to each site individually. because I am lazy. and people posting 1 star review scores on games the internet doesn't like is much better there than doing it on, say, amazon.

I mean, what's the big deal?

I think it's fair, it aggregates every review. Just doesn't do it good enough.

From what I gather, it's the fact that it doesn't actually give an accurate representation of scores for games. For example 3/5 on Giant Bomb translates to something like 60/100 which isn't really the same thing, at least not in my eyes. I just don't like it because scores mean nothing to me, it's about the text man.

which is why I don't care about the score. I care about the reviews, and it handily collects them all into one webpage that I can check whenevz.

True that. Just don't read the user reviews!

@azteck: I trust gamefaq user reviews more than metacritic ones and that's really sad but as I said it's better than doing it on Amazon or something where people actually buy shit from.

What @aztecksaid.

But also, it can never take into account all of the little variations between scoring systems that every outlet has, a 60% score on one site might mean an "average" game but on another 60% is an "okay" game. Then with the problem of trying to convert a "B" rank to a 100-point scale, you can start to see why Metacritic could be called "inaccurate".

And to your point @animasta , the bonuses issue is important, because people are losing out on them based on an inaccurate, flawed system.

What does Metacritic do that is good?

I don't necessarily think it's bad, but it is open to manipulation and once you find a good reviewer, why would you ever use Metacritic?

When you drill down and see how the scores are formed there, it's crazy the "sources" they use, most of them nobody has ever heard of and probably have no editorial oversight so any input they give is questionable at best and without knowing their tastes and biases how can you put an appropriate weight on their score?

Until reviews are required to justify/explain their scores, aggregating them is just as much a waste of time as the scores are themselves.

It's because it's the place users go to bash a video game and drag down its score down for example DmC.

I hate the principle of weighted average. Just because a site is more popular than another one does not mean its reviews should count for more in the average metascore. Jeff is pretty famous as far as game reviewers go. Probably more so than anyone at IGN so why should an IGN review count for more than a Giant Bomb review?

That's where metacritic fails. They have no business deciding what reviews are more important than others.

Gamerankings is just compiling reviews in one place, puts an average score that nobody should pay any attention too anyway and just use the site as a convenient way of finding multiple reviews quickly.

Think about the impact that weighted average can have. You said bonuses shouldn't count but take New Vegas as an example. They missed on their bonuses by 1 point. What if the real average was in fact 1 point higher but just because some of the more generous reviews ended up being given by smaller sites they lost those bonuses?

This doesn't make the business surrounding bonuses tied to metacritic scores any less shitty but in cases like this metacritic's model can have a real impact on a studio.

@serhulse: but my point is that if metacritic wasn't there they'd do it themselves.

Main reason against metacritic & been seen in the past is the user reviews with that score being dumped into the press reviews (and sometimes they are loaded with good or fake reviews from publishers). Just stick with gamerankings as it is just a collection of the gaming journalism press reviews & the scores if you care about that or you can just use it as a one stop site to read a bunch of reviews with one click.

Personally, I dislike Metacritic because of the user submitted reviews that work on a 0 or 10 point system, and the general roid rage/circle jerk mentality they sometimes have going on there.

0 = Didn't like/Didn't play but want to piss people off

10 = Liked/Didn't play but want to piss people off

It's still a decent site for seeing how well received a game was, but I don't base my purchases on reviews anyway.

While there are variations, scores on metacritic are accurate enough to get a rough impression of what quality level to expect.

The problem isn't necessarily Metacritic itself.

It's rather the way people use it. Metacritic is just a quick and dirty review aggregator, but unfortunately people put way too much value in its' rankings that can have pretty negative real world consequences.

namely publishers using it as criteria tieing funding allocation to developers based on MC ratings and users/consumers assuming the MC score is the "real score".

You said that shouldn't count, but I don't see how it doesn't. Most companies (whether it be cell phone manufacturers etc) do take such things into account. Perhaps not out of altruism and more out of fear of potential litigation but they often do. MC could certainly take upon themselves to change their methodology to make their scores used more properly but they clearly are more than ok with how their scores are abused.

Because their title is misleading. Where am I supposed to get reviews of Tom McShea and Alex Navarro NOW!?

@animasta:

1. It uses a weighed average, which is based on the popularity of a site, which makes no sense; A 2/5 with the proper logical backing isn't any less valid on a small site vs a big one.
2. While you may not care about the scores, most people do (and I'm making that assumption because people always get in an uproar about games receiving a certain score but almost never about specific language in a review. Jeff's 8.8 on Twilight Princess, for example) and that devalues the weight of the actual logic behind the criticism. It seems like the vast majority of people don't care if a number is backed up by logic, only that it is high when they like something and low when they don't, for whatever reasons they've decided on. Metacritic helps facilitate this.
3. The bonus thing does count because of how arbitrary it is. Why should a score of X be worth a bonus while a score of X-1 is not? The idea of getting a bonus based on critical reception only exists because of how popular Metacritic is, and that's where your metaphor falls apart (cell phones might cause accidents but they aren't the sole reason for them, whereas Metacritic is the sole reason in this case. If banning cell phones in cars would stop all accidents from occurring, I'm not sure you'd find too many people arguing against that). It pushes too much weight onto the score and not the review.
4. It doesn't add anything of value. It's just an aggregation of links, which you can get from Google. Look at it this way; you probably only trust a few sites, so why not just bookmark those? If the death of Metacritic means that we can do away with this "good reviews = bonus" scheme at the cost of your laziness, I'm not sure you've made your position very defensible. Plus, half the time it misses sites (even some larger ones) which Google does not.

I'm not saying that you should do away with all review aggregation, just that they cause more harm than good, especially when they get popular enough to start displacing the already tenuous balance between the weight of scores and review text.

@animasta: That's true, which is why I favour abolishing scores. But that's a different topic altogether.

The aggregate score is what's the problem though, doesn't matter if it's Metacritic's or a PR agency's. People hate Metacritic now because they are the ones doing it now.

It's useful to me because it's a convenient place to find links to most reviews of a game. The "metascore" nonsense is unimportant.

Accepting all reviews.

Metacritic puts the emphasis on only a few things in a review: score, outlet, and popularity. This is a valuable tool if all you want is a general sense of a game - "do a lot of people like this game?" It's as useful sales numbers - "lots of people thought this would be good." Neither are great gauges for an individual's needs and desires (unless your opinions always follow mainstream lines).

The important thing for any individual is the nature of the reviewer and the words that reviewer uses to justify their opinion. Telling me Giantbomb gave Tomb Raider a 4 out of 5 tells me nothing. Telling me that Brad gave it 4 of 5 is actually useful - I know his likes and dislikes enough to compare them to my own. Explain to me that he had complaints about it's inconsistent tone and lack of "tomb raiding", and I start to be able to form a value judgement on the game. Metacritic white washes all that nuance away.

I think Metacritic is a great tool for consumers. It's a good way to get a quick, broad view of how something was received. It's not my problem publishers use it to be complete fucking dicks.

If you were smart enough, you would just not look at the User Scores. Most people hate they site cause they pay more attention to the User Score then the actual Critic Score which is the one that matters.

I say it is not bad, you just have to USE it correctly. That is the thing with processed data or even just raw information you have to know what it means so that you can use it. What Metacritic does is aggregate reviews, which last time I looked was not a crime. I think some of the issue is that some sites do no like the fact that their review scores A through F (+/-) score are conversed to a 1% to 100% scale. The other issue is there is little consistency, some games are not reviewed by some outlets so there is a shot-gun approach to any score. There are weights to scores and all the rest, but in the end some games get reviewed one way and some another. But, that is what I mean by "using" teh data correctly. If you know the people & sites who reviewed Crysis 3 are different then the people & sites who reviewed Tomb Raider, then you know 78 Metascore and a 86 Metascore are not comparable.

Metacritic is just a tool, it is not one people should use in teh way they do use it, but so what. There are people who will use a hammer to put in a screw, and then use pliers to pull the screw out. Sure, a screw driver would be better, but that is besides the point you can do it that way...it is just messy. You can talk about games by their Metacritic score..it is just messy.

@mirado: Brilliant post. Arbitrary weighted values/averages are the key problem for me.

I really don't think it's bad. I think how it's used by publishers sucks more than anything, but that's not metacritics fault.

This is the issue with all user review systems, you just have to know that. Different sort of example, but if you go to say the Professor Layton 3DS game reviews, you will see a huge number of one star reviews that are people saying they bought it and didn't realize it was for 3DS instead of DS and complaining and saying it's shit. Is that the game's fault? No. But the scores are there.

3D Only Sucks!!!

,

November 1, 2012

By flbchbm "Val" (Florida) - See all my reviews = Fun:1.0 out of 5 stars This review is from: Professor Layton and The Miracle Mask (Video Game)

I bought this latest Layton series game because all the others are great. When I saw 3DS on the box I was asured it would fit because at the bottom of the package it states "Playable in 2D and 3D mode..." I have a DS-XL and the game does not fit. There is a protrusion to one side that prevents the cartridge from engaging fully into the slot. NOT FAIR. I like my DS-XL and have no plans to replace it. It sucks that the series has ended for non 3D users. I don't like the 3D effect. 2D is good for me. Since the box is already opened and I can't return it, I'm going to see if cutting off the protrusion will work. It's all about more $$$$$ at every turn.

I believe there is a fundamental misunderstanding of what a "review score" is which is just aggregated (pun!) into Metacritic. I have no problem with a site that scrapes reviews from a wide range of sources and presents "stats" on them but people need to realize that this is no more "factual" than any single review it uses.

If I have any complaint about Metacritic, it would be it shouldn't bother with "user reviews". Metacritic wants to scrape user reviews form sights like Amazon and Giant Bomb then that would be fine but hosting their own is in my opinion not a great idea.

It's just lazy in execution and the fact that publishers use it as a measure of quality is just another layer of unpleasantness.

Exactly, but that's the publishers fault.

You can't ask 'Why is metacritic so bad?' and then dismiss criticism of the primary function of metacritic. You might use it as just a list of reviews, but you could use google for that.

The main function of metacritic is to assign an average review score based on lots of reviews, and it sucks at doing that. They even manage to fuck up the fairly simple math involved in converting a score from one rating system to their percentage system, which results in everything being skewed way too high. According to Metacritic a 5 Star review scale equates to a 20%-100% scale, which is so dumb it makes me want to cry.

There are lots of reasons why it's probably a bad influence, which is a whole other discussion and not necessarily their responsibility.

People are lazy, and looking up a number that combines and averages all the reviews out there gives them a quick idea of how good a game is.

Is it always accurate? No. Is it an indication of how much Random Person is going to like this game? No. But enough people use it. (Remember -- a VAST majority of video game buyers don't regularly keep up on video game news & visit websites regularly. We're the informed minority.) In fact, enough people use it as a quick idea of how good a game is, that companies consider the MetaCritic numbers to be a literal score. I'm sure they also have plenty of research that shows that X score equals however many more sales.

So MetaCritic overly reduces everything to a score, one that is often used in internet arguments. It detracts from the discussion, and some of the more overly sensitive among us get upset when games they like or hate get scores they don't want. Which goes back to MetaCritic detracts from actual discussion.

as a website or product- it's fine. it's simply an aggregator. it's what it's used for that's problematic. all the compensation-related stuff aside, it's often used as a target for review-bombing and the like. if you just go there to get a general digest of the critical response, whatever, it's fine. but you can't really use it as a indicator for anything, since it's simply an average, and an average doesn't tell the whole story of a game. numerical averages are somewhat meaningless when applied to the continuum of subjective opinion.

Well if you look at how they score giant bomb reviews, a 3 star game is a 60 in their eyes. Which means that Dead Space 3 was a 6 out of 10 which we all know it isn't even with the missteps it did have.

@artisanbreads: Sure, nevertheless the stigma carries over to the site as well.

The way it aggregates scores isn't perfect, but as a site Metacritic is fine. I think the real problem is how certain publishers have actually come to treat it as a genuine barometer of how a game was received, and certain chunks of the gaming audience have come to treat it as some huge objective indicator of how good or bad certain games are.

@animasta:

As a consumer I like metacritic. But if I were a writer I would hate it. I guess I would feel as if scores alongside reviews would be a necessary evil but that I would hate scoring games because scores are basically arbitrary. I would feel that the meaning and rationale of the review is written in the text of the review and not the final score I decide to put on the game. Metacritic takes out the text-aspect of the reviewing process and leaves consumers with an all-too-short summary and a score that, without the text, is almost meaningless.

Eh I guess? I don't agree. Not saying it's a perfect site but it's not the fault of an information source how it ends up being used by a person or entity. I could make an extreme example here but I think that's self explanatory.

If game publishers didn't treat Metacritic as gospel, we wouldn't have this conversation.

Basically i have no problem with metacritc for consumers. My personal issues with are publishers using it to incentivise team members to 'do a better job'.

Nothing is inherently bad about Metacritic. It's raw, albeit curated, data. More statistical data can never be perceived as immoral. How publishers use that data is another conversation. It's a moot topic though. If Metacritic never existed, larger publishers would find another way to retrieve that sort of information. At least with Metacritic the data is completely open to publishers, developers, and consumers.

Because it weights scores in a way that is not transparent and completely unfair at times, it puts a number ahead of the written content for a world with ADD, and for some arbitrary reason it's the gospel that publishers seem to use. Remember Obsidian and that whole thing about Bethesda not paying them a bonus because of a 1 point difference? That's why Metacritic is bad. There are uses to it, make no mistake. If something has an average score of 30% chances are it's shit, but it's the way that Metacritic can get to that average of 30% that's the problem. And it most certainly should not be used to withhold bonuses. That's just fucking stupid.

The problem with metacritic is that it's based on the idea that games can be objectively criticized. All metacritic does is pump out a score saying "this game is X good". While having that number might be valuable in itself it's completely useless without context. According to metacritic one of the best games ever is Mario Galaxy, yet many people would be quite disappointed if they bought that game purely because it's a platformer.

It serves no other function than saying if a game is good or bad at what it's trying to do, which anyone could find out in about 2 minutes by browsing a few different websites themselves.

Metacritic as a site is perfectly fine. It's how it's used within the industry that makes it bad.

It's pretty cool product to work on

When a product is praised by journalists and users bash it; it gives me a warning message and I appreciate it or if a product's metascore is proportional to the user review score it's probable that the judgements made on both side reflects the actual quality of the game.

Few examples:

http://www.metacritic.com/game/pc/dragon-age-origins

http://www.metacritic.com/game/pc/dragon-age-ii

http://www.metacritic.com/game/pc/natural-selection-2

http://www.metacritic.com/game/pc/mass-effect-2

http://www.metacritic.com/game/playstation-3/mass-effect-3

While I was working on it, I always looked at the forums and there was always some kind of balance between what was said on them and the ratio metascore/user-score. I never look at the metascore alone because I feel like it's meaningless...

I don't like metacritic or rottentomatoes for similar reasons. A lot of people just look at the number and replace that with an opinion even before trying a product. Just because a film has like a 90% or whatever on RT doesn't mean it's good, it mostly means that it's crowd pleasing and/or relatively inoffensive. To be contemporary, Argo is a great example of this. I haven't check its RT score in a while, but I saw the film opening day and while it was enjoyable it was just kind of pedestrian. Not thematically rewarding or even all that engaging, just an easy way to spend two hours. When I walked out I said to myself "that's the crowd-pleasingest crowd pleaser I've seen in a while, and of all the movies I've seen so far this year it's the one I could probably recommend to anyone and have no reservations about it because it's just this competently made heist movie." So yeah, in a way that's a positive review, and if that's how the majority of critics felt (I'm not sure, I didn't really read many reviews of it) then its RT score would obviously be off the charts despite it not being all that special or interesting. A RT score could mean something is good, just like a highly ranked game on Metacritic could be good, but to find that out one would have to either find reviewers generally aligned with their tastes or play/watch the product.

In a modern context this becomes even more troubling because some people will see a score of like 75 or whatever and just assume that it's garbage because it doesn't have a 90. When it comes to RT I've found that films in the mid to low 80s all the way down to the 50s tend to be the most engaging and accomplished films because many have a strict artistic vision and are much more focused on expanding one's cultural palate, and as such, in my opinion, are more rewarding. But it's so easy for people to point at an aggregate number and feel like there's some sort of legitimate science behind it as proof, which is particularly troubling in mediums where it's already tough to strike a balance between objectivity and subjectivity.

Metacritic, at least from what I see is a lot more used by the corporate side of things as a quick look of general quality. For the common person, the issue with Metacritic comes from the fact that it takes all scores and flatly makes them out of 100. For sites like Giant Bomb and The Escapist, who use 5 Star systems, 2/5 does not really mean a 40, even though is a direct number comparison. But the bigger problem comes when you find places with bigger scales that mean different things. Game Informer will rate a middle of the road game at a 7/10, while Destructoid will rate that same game a 5/10, assuming both publications agree it is middle of the road. However, when that happens, a 50/100 on Metacritic is a lot harsher than a 70/100, making the higher scoring publication a lot better for the game publisher. In doing that though, Metacritic does not really give any sort of obvious disclaimer saying that these scores may reflect differently than the final number Metacritic gives out of 100.

Also, this site is definitely a lot more about seeing the number and nothing else, as you only get a short blurb of the actual review. In that, the problem with Metacritic itself isn't so much Metacritic, but the blind faith that their number is a fair reflection that actually mirrors your own tastes. One has to remember though, Metacritic has no soul to the final score it assigns, it is simply an aggregate of the numbers taken from a ton of disparate publications. Really the best policy gamers could give to themselves is to find the reviews from the writers they like, and to actually read the review, and understand the publication's scale.

The problem with Metacritic is that when it aggregates the scores it changes the value of the scores. And usually this wouldn't matter but since the industry is hiring and firing devs based on the scores it does matter. I think Obsidian lost out on a big bonus b/c New Vegas scored an 84 instead of 85 which resulted in them having to lay off a bunch of people. It also weights the scores based on how popular the site is which is just silly. Metacritic isn't inherently bad, but I think the implementation of how it aggregates scores and how its used in the industry is problematic.

metacritic is evil for two reasons

1) They are not transparent about the process used to determine the final score. (Technically they can make up the score and there's no way for us to know) (Someone told me it's based on the credibility of the review site, but in that case shouldn't this be displayed somewhere? We aren't entitled to know which review sites we should trust?)

2) They cherry pick which reviews get listed. Some sites show up for some games but not for others even if the site reviewed both.

Metacritic is a site that has failed in the purpose it poses itself as having. It has itself become the reviewer, rather than a database of reviews. The site exists for its own sake now, rather than for us lowly sheep who don't need to know how scores are decided, and who don't need to see a particular review.