Giant Bomb News

139 Comments

Metacritic Finds, Bans Group of Users Unfairly Scoring Games

Users gaming the system to target Bastion, Toy Soldiers: Cold War and others are gone.

When things go sideways on Metacritic, developers don't have many options for appeal.
When things go sideways on Metacritic, developers don't have many options for appeal.

Supergiant Games and Signal Studios were in awkward positions earlier this month, after the average user review score for each game plummeted overnight and seemingly without justification.

It was unclear who was responsible for the sudden change, and there are limited tools for those affected that don't come across as...weird.

The developers of Bastion and Toy Soldiers: Cold War asked fans to submit reviews to Metacritic, which resulted in a slight bump for each but an average score ultimately much lower than it was previously.

After speaking to Supergiant and Signal, I ran a story earlier this week about the challenges the studios faced in addressing these issues.

Metacritic has since identified the users responsible, both banning the users themselves and removing their scores putting things out of balance.

The users were targeting certain games, and hitting them with zero scores without a written review, raising eyebrows.

"A publisher of a different game let us know that he noticed that his game had been 'bombed' by a bunch of zero ratings in an unusually short period of time," said Metacritic co-founder Marc Doyle in an email to me today. "While investigating those ratings, we noticed a group of user accounts and activities that were clearly illegitimate and violated our terms of use."

Doyle's team found a group of users had been coordinating similar attacks on other games within Metacritic's database, allowing Metacritic to fix the "bombings" that had been targeted at other games.

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher. I've never heard of a publisher leveraging user reviews as a metric for payment because of situations like this, but especially for small studios, perception is king, which is why user reviews remain very important.

Signal Studios president and creative director Douglas Robert Albright III had proposed a fix to the problem.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," he said. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
139 CommentsRefresh
Avatar image for mrklorox
Posted By MrKlorox

Holy shit, is this actually Metacritic acknowledging their system isn't perfect and attempting to take steps to right the wrongs!?

Avatar image for youngfrey
Posted By YoungFrey

I think an article like this really suffers from not linking to the outside material in question.  You spend 11 paragraphs talking about metacritic and it gets no links?  It might be common practice, but it's lame and anti-reader.  Not even ones to the games in question?  I hate it when other sites do that, and I hate it here.
Avatar image for joey_ravn
Posted By JoeyRavn

I don't like where this is heading. Then again, I don't use Metacritic, nor like Bastion. So no problem for me, for the time being.

Avatar image for dystopiax
Posted By DystopiaX

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

That has nothing to do with review scores, and that sentiment was around before metacritic was a thing.

Avatar image for sammo21
Posted By Sammo21

Metacritic user reviews are complete trash.

Avatar image for lockwoodx
Posted By lockwoodx

Metacritic caves in I see.

Avatar image for elsolar
Posted By Elsolar

metacritic is cancer

Avatar image for ravenlight
Posted By Ravenlight

Nice job snapping up this scoop, Patrick. Glad to see Metacritic took action in this situation.

Avatar image for winternet
Posted By Winternet

See, that wasn't so difficult. Too bad a couple of devs and websites had to bitch about it for them to actually do what they are supposed to do.

Avatar image for krakn3dfx
Posted By Krakn3Dfx

This seems like a slippery slope to me when it comes to choosing who can and can't post user reviews for games. I'm not much for Metacritic anyway, but to have them picking and choosing which user reviews are valid and which aren't makes them even less reliable than they were before.

Avatar image for tourgen
Posted By tourgen

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

well if scores determine royalty payouts and bonuses, a publisher could hire some people to bomb scores as a cost-saving measure.

Avatar image for splodge
Posted By Splodge

@tourgen said:

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

well if scores determine royalty payouts and bonuses, a publisher could hire some people to bomb scores as a cost-saving measure.

That's a bit of a stretch. I doubt any publisher would risk the kind of horrible PR and fallout that would occur if they got caught doing that.

Avatar image for windfall259
Posted By WindFall259

If Metacritic makes so much attention to developers, then it should be held to a higher standard.

Avatar image for 234rqsd2323d2
Posted By 234r2we232

Great. Now they should ban the people raising scores of games for their friends.

Avatar image for mormonwarrior
Posted By MormonWarrior

I like Metacritic, because it gives a good idea of what games are high-quality and which are simply okay, and I don't have all the time in the world to play every game to find out for myself if it's good.

However, I've always known that the user reviews on there are so completely trash they are totally useless. Where you see Halo Reach, a quality shooter, getting either 10 "Best game EVAR" or 0 "So overrated it's terrible HALO SUX" reviews. The normal reviews have some semblance of reason and rhyme behind them. I especially look for the ratings from sites I trust.

Avatar image for alex_carrillo
Posted By Alex_Carrillo

Metacritic is an utter shitshow.

@DystopiaX said:

@SinisterInfant said:

If I were to put on my conspiracy theorists hat, It could be publishers themselves posting bad reviews to drive down the metacritic score. Both of those games are now several weeks past their initial release window. From a publisher view point the majority of sales have likely been made. If they owe money to the developer based on a score, they could drive the score down pay the developer for the lower score then delete their comments and restore the score. Seems like an easy way to anonymously save some funds.

Would be dumb, plenty of people still buy games a couple weeks after they're out, and payouts are based more on critic than user reviews.

Wouldn't that affect future sequels should they happen? A publisher wouldn't burn potential dollars for a few bonus bucks to the developers.

Avatar image for largo6661
Posted By Largo6661

@DystopiaX: Thats is true. But i do feel that metacritic has made this issue more pronounced. The words in a review matter less and less these days. most poeple i now never even read reviews they just go to metacritic and look at a number. Its not the user reviews i have a problem with its how a dozen or more reviews can be reduced to a simple digit seems really unfair to me. And personally the idea of having one all incompasing site to tell me whether a game is good or not makes me uneasy, due to the fact that its so easy to effect a games score on metacritic.

Avatar image for bkbroiler
Posted By bkbroiler

Woah, Metacritic is used to determine royalties and bonuses? I find that incredibly shocking.

Avatar image for dystopiax
Posted By DystopiaX

@Largo6661 said:

@DystopiaX: Thats is true. But i do feel that metacritic has made this issue more pronounced. The words in a review matter less and less these days. most poeple i now never even read reviews they just go to metacritic and look at a number. Its not the user reviews i have a problem with its how a dozen or more reviews can be reduced to a simple digit seems really unfair to me. And personally the idea of having one all incompasing site to tell me whether a game is good or not makes me uneasy, due to the fact that its so easy to effect a games score on metacritic.

I think that you have a problem more with how people use metacritic than metacritic itself. If used properly it's actually a great tool, for example if I want to read alot of reviews I'll go to the metacritic review and read the summaries of scores, and click on the ones and read the fulll reviews of the ones that interest me... I think that no one should just look at the score a game got but that problem doesn't stem from metacritic.

Avatar image for krelle
Posted By Krelle

@bkbroiler said:

Woah, Metacritic is used to determine royalties and bonuses? I find that incredibly shocking.

Apart from sales, Metacritic is the only way for 'a suit' to judge the success of a game/studio.

Well, that or flipping a coin..

Avatar image for largo6661
Posted By Largo6661

@DystopiaX: Yeah your right when used like that it is helpful. Its just a shame that the majority of its user base probably doesn't do that.

Avatar image for mavs
Posted By mavs

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

Avatar image for boiglenoight
Posted By Boiglenoight

And Douglas Albright's comment seems very reasonable.

Avatar image for ryanwho
Posted By ryanwho

Now Metacritic has integrity. lol

Avatar image for crusader8463
Posted By crusader8463
@mavs said:

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

Buy, rent/borrow/pirate, pass. All you need for a review score.
Avatar image for craigbo180
Posted By craigbo180

Metacritic needs to ban Jim Sterling/destructoid reviews if they really wanted to be helpful.

Avatar image for winsord
Edited By Winsord

It's just a matter of time. Though, forcing people to actually write a review instead of just giving it a score without explanation is definitely a step in the right direction. Maybe it's taken a little longer than it should have, but good on Metacritic for at least addressing this issue.

Avatar image for swoxx
Posted By Swoxx

Why would you put stock into what random people on the internet say anyways? They're all shitheads and dicks.

Except you Giant Bomb community, of course. You're a sweet little safe haven. <>

Avatar image for aajf
Posted By aajf

So the entertaining upshot of all this is that Bastion's user score is actually now inflated, since Greg Kasavin called for grateful players to combat the spam with their own positive reviews. I can't say I have a problem with that outcome.

Glad to see Metacritic at least attempting to get things together. I think it can be a great resource if approached with proper caution by the reader.

Avatar image for hef
Posted By Hef

If metacritic didn't exist neither would double fine. Their first game, psychonauts, sold like shit. But it got really good rreviews from critics. This allowed them to say to EA: Hey, actually market my crazy new awesome metal game and I guarantee you sales. Of course it still didn't sell very well and reviewed slightly worse than psychonauts, but they got a new engine out of it. And that new engine DID lead to 4 awesome downloadable games. I say keep on keepin on metacritic.

Avatar image for whylessness
Posted By whylessness

Bastion is my shit.

Avatar image for sinful
Posted By Sinful

metacritic, a concept that makes no sense and is easily corrupted.

Avatar image for sleepydoughnut
Posted By SleepyDoughnut

Yay, a happy ending to this story

Avatar image for gildermershina
Posted By Gildermershina

@mavs said:

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

That's why the five star system is so great. Too much gradation is bad I think, and too much weight is put on getting a number to put in the metacritic cruncher as opposed to actual review text - thankfully that text does at least make it onto metacritic.

Avatar image for subyman
Posted By subyman

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating.

Avatar image for enigma777
Posted By Enigma777

I bet it was some random forum.

Avatar image for stise
Edited By stise

@subyman said:

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating

Fuck you man, SmokeDAWG420 is my ONLY trusted source for video game news and reviews.

Avatar image for napalm
Posted By Napalm

@MrKlorox said:

Holy shit, is this actually Metacritic acknowledging their system isn't perfect and attempting to take steps to right the wrongs!?

Temporarily, at least. It's not as if things won't be awful by this time tomorrow.

Avatar image for mike76x
Posted By Mike76x

What about the people that gave all three versions of Dragon Age 2 zero's before it was released, or bitched about DRM that wasn't there?

I'd trust the ratings on Amazon.com, or any retailer before I'd give Metacritic any thought whatsoever.

Avatar image for steelknight2000
Posted By steelknight2000

I always ignore user reviews anyway when it comes to games and movies.

I do think it's important for developers and publishers to consider criticism from a third party such as the users and press in order to get a less biased view of whether or not their game sucks.

Avatar image for sagesebas
Posted By sagesebas

Metacritic needs to get rid of the option to submit a user review and they need to make a minimum length for those reviews also why does anybody use metacritic for bonus payouts why don't they just, I dunno determine payouts based on how well the game sells

Avatar image for mattclassic
Posted By MattClassic

What kind of sad group of individuals takes the time to organize rating bombings on random indy video games? Jesus, their lives must be completely devoid of joy.

Avatar image for the_official_japanese_teabag
Posted By the_OFFICIAL_jAPanese_teaBAG
@stise said:

@subyman said:

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating

Fuck you man, SmokeDAWG420 is my ONLY trusted source for video game news and reviews.

You mean RepThePlantDawg420
Avatar image for librariangmr
Posted By librariangmr

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher.

Wait, what? Seriously? How awful :(

Avatar image for acheron
Posted By Acheron

Good on them for actually looking into it and fixing the problem. I give Metacritic a 9/10.

Avatar image for donpixel
Posted By DonPixel

Metacritic = Troll Nation

Avatar image for crazyhorse23
Posted By Crazyhorse23

I guess this is good. I just wish that metacritic good be less dominated by trolls. Judging by this story, developers really care what the scores are for their games, but it seems to me that with all the trolling, devs can never really be sure whether or not their product is really all that good.

Avatar image for lacke
Posted By lacke

There is nothing wrong with Metacritic. It's the people who are abusing it who's at fault. Why blame Metacritic for publishers determining bonuses based on review scores? Why blame Metacritic for consumers who only look for a review score instead of reading a text? That consumer would probably not read any text on any of those websites either.

I'm not a fan of Metacritic and rarely ever use it but I don't think it's the cause for those actions. It just makes them easier.

Avatar image for lonely_ogre
Posted By Lonely_Ogre

That's cool, always nice to hear stuff like this! :)

Avatar image for getz
Posted By Getz

@BestUsernameEver said:

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

Yeah, metacritic is the target.

I'm with you; this smacks of anti-metacritic sentiment. A way to show off how flawed the system really is by abusing that system (e.g. Lulzec). Of course, I'm not condoning this bullshit because it still hurts innocent people in the process and is extremely childish.