Giant Bomb News

139 Comments

Metacritic Finds, Bans Group of Users Unfairly Scoring Games

Users gaming the system to target Bastion, Toy Soldiers: Cold War and others are gone.

When things go sideways on Metacritic, developers don't have many options for appeal.
When things go sideways on Metacritic, developers don't have many options for appeal.

Supergiant Games and Signal Studios were in awkward positions earlier this month, after the average user review score for each game plummeted overnight and seemingly without justification.

It was unclear who was responsible for the sudden change, and there are limited tools for those affected that don't come across as...weird.

The developers of Bastion and Toy Soldiers: Cold War asked fans to submit reviews to Metacritic, which resulted in a slight bump for each but an average score ultimately much lower than it was previously.

After speaking to Supergiant and Signal, I ran a story earlier this week about the challenges the studios faced in addressing these issues.

Metacritic has since identified the users responsible, both banning the users themselves and removing their scores putting things out of balance.

The users were targeting certain games, and hitting them with zero scores without a written review, raising eyebrows.

"A publisher of a different game let us know that he noticed that his game had been 'bombed' by a bunch of zero ratings in an unusually short period of time," said Metacritic co-founder Marc Doyle in an email to me today. "While investigating those ratings, we noticed a group of user accounts and activities that were clearly illegitimate and violated our terms of use."

Doyle's team found a group of users had been coordinating similar attacks on other games within Metacritic's database, allowing Metacritic to fix the "bombings" that had been targeted at other games.

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher. I've never heard of a publisher leveraging user reviews as a metric for payment because of situations like this, but especially for small studios, perception is king, which is why user reviews remain very important.

Signal Studios president and creative director Douglas Robert Albright III had proposed a fix to the problem.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," he said. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
139 CommentsRefresh

Avatar image for craigaa1028
Posted By CraigAA1028

I wish Giant Bomb could do this every once in awhile for user review ratings, but since the team at GB is small I don't expect it or blame them for not policing it. I removed almost all my user reviews and decided never to post them here again since a team of about 10 trolls would just wait for me to post a review and then bomb it with negative feedback within hours of it going up. Trolls fucking suck

Avatar image for murdersmash
Posted By MURDERSMASH

@Hailinel:

As sad as that is, Metacritic can't be blamed for such behavior. That's the publishers who are putting so much stock in the site. In my opinion, the only thing off the top of my head that they're doing seriously wrong is using some strange weighting system to determine the overall score.

Avatar image for big_jon
Posted By big_jon

Good, douche bags.

Avatar image for boopie
Posted By Boopie

it was always obvious it's Sony fanboys no mystery here at all

Avatar image for gbpacker
Posted By gbpacker

Because now metacritic actually matters...

Avatar image for mistermouse
Posted By MisterMouse

good thing they went in and banned the accounts

Avatar image for meatsim
Edited By MeatSim

Nobody should be putting any kind of stock in user ratings with no text reviews.

Avatar image for meowshi
Posted By Meowshi

This sucks considering how highly-regarded Metacritic is.

Of course, Metacritic shouldn't be as important as it is.

Avatar image for obinice
Posted By obinice

I had never even heard of metacritic. I wonder how popular it is in the UK.

Avatar image for mindchamber
Posted By MindChamber

At first Im like... if the metacritic score is so important that it effects the success of a game then maybe these users should be charge for fraud, seeing how employees are given bonuses etc based on it.. Then again.. These trolls show just how flawed metacritic really is .. and hopefully developers will stop relying on an easily manipulated system to judge their games by.

Avatar image for iam3green
Posted By iam3green

well good that they fixed it. pretty weird to just have a random attack like that. usually when a raid like that happens it gets around the net.

Avatar image for assinass
Posted By AssInAss

Anyone got reminded of the old Gamespot user review days?

Avatar image for example1013
Posted By Example1013

Man, some people are thick.

Avatar image for jmrwacko
Posted By jmrwacko

Trolls should be thrown in prison, where trolling for attention only results in unnecessary penetration.

Avatar image for williamrlbaker
Posted By WilliamRLBaker

What the hell bastion was an amazing game...

Avatar image for aaron_g
Posted By Aaron_G

people need to get lives if they have time to do this.

Avatar image for imotep12
Posted By imoTEP12

I doubt doing this will permanently stop any trolls.

Avatar image for huser
Posted By huser

@EPGPX: I agree on the stupidity of using review aggregator's as a measure for bonuses, but using the movie industry as the example isn't exactly the proper analogy. Everyone knows who Spielberg, Lucas, etc are. Until game devs have that level of facetime and recognition, they have far less power to ask for 200 million dollar budgets and huge paydays up front for their work. Given that, game devs are going to have to find some metric to reward successes. Gamesales are certainly one option, but it probably has to be something.

Avatar image for doe3879
Posted By doe3879

Reading the title and half way down the article,

I thought it was about PC Leader Board Score hacking and they're doing something about it, and I go "ABOUT TIME!!!!

but no :(

Avatar image for bog
Posted By BoG

Good. However, eliminating trolls should be business as usual, not some huge breakthrough.
Avatar image for hilts
Posted By Hilts

The solution is even simpler. Get rid of user reviews from Metacrtic all together. Who honestly looks at them?

Avatar image for epgpx
Posted By epgpx

The idea of Metacritic seems rather stupid and it's more laughable when you say publishers pay out bonuses/royaltie checks to developers based on metacritic scores. The system is flawed and as the article states, the system can easily be abused both ways and just imagine how many retarded developers depend on false scores just to be rewarded bonuses is hysterical. Actually, they're not stupid at all cause they're getting more money based on unreliable false scores and its the publishers who are major dumb asses relying on such stupidity that baffles me.

How does one depend on a review aggregator score for bonuses is beyond me. That's just dumb. I mean I'm pretty sure the movie business doesn't do this. Unless they somehow use the Rotten Tomatoes tomato-meter for bonuses too, then this is just one big farce.

Avatar image for elazul
Posted By Elazul

@creiij: Well yeah, most people here probably don't care about metacritic, but for people working at developers who could potentially be stripped of bonuses and royalty payments because of stunts like this, it's kind of a big deal.

Avatar image for creiij
Posted By creiij

It's good they ban people who give games a zero so we can get honest opinions about a game. But seriously, who really cares about Metacritit? It's not a professional review site since anyone can register and give scores (unless you give a zero then you will get banned).

I stick to sites like IGN, Giantbomb and RPS for reviews and Youtube or Giantbomb for some gameplay footage.

Avatar image for rambostyrer
Posted By rambostyrer

This is why I more and more stick to stuff like Totalbiscuits "WTF IS" series and giantbomb quicklooks......

Online
Avatar image for vinsanityv22
Posted By vinsanityv22

that fix sounds questionable. I mean, that system works pretty great for Gamefly, but still - it's really easy to give your game a 1 and write a terribly-written sentence like, "Dis game is teh suck Call of Duty is better only n00bs and kiddiez play dis game"

Avatar image for kosayn
Edited By Kosayn

Metacritic is what it is, but it shouldn't be used for bonuses. I think the critical community for gaming is still too embedded and too prone to repeating sound bites as their own opinions. Needs more time to grow.

Also, what's funny is how little abuse there actually is of things like this that are basically anonymous and without consequences. Jackasses are more of a rare and valuable commodity than you might think.

Avatar image for jazz_lafayette
Posted By Jazz_Lafayette
So Metacritic decided to fix this problem - which apparently involved ToS violations - only after gaming press pointed it out. Encouraging stuff. Thanks for the heads up anyway, Patrick.
 
@gladspooky said:

They should just ban all reviews. Problem solved.

Corrected.
Avatar image for jackelbeaver
Posted By jackelbeaver

>people who complain that originality in gaming is dieing, that all that sells is big fat shooters.

and then nobody buys big original AAA games that ninja theory, platinum games, or grasshopper studios makes.

self fulfilling prophecy

Avatar image for solidejake
Posted By solidejake

They should ban G4/X-Play.

Avatar image for xeiphyer
Posted By Xeiphyer
@SeriouslyNow said:

@Levio said:

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Oh I'm sorry, I meant corruptible. Does that please you?

The reviewers aren't getting the bonuses, the team that made the game is, so its not corrupt to be paying a bonus based on how well a game reviews.
 
 It is pretty disgusting though, but it makes sense. It does encourage the developers to make a good game, but honestly, anyone making games is trying to make a good game because the industry is highly competitive and everyone in it wants to be there and works hard to stay there.
Avatar image for seriouslynow
Posted By SeriouslyNow

@Levio said:

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Oh I'm sorry, I meant corruptible. Does that please you?

Avatar image for nunsandroses
Posted By nunsandroses

@lacke: The nature of Metacritic condones people ignoring the text of a review, and simply looking at the score. Not to mention, they have a totally messed up way of converting scores (letter grades converted on an idiotic scale). The fact that they attempt to convert everything to a score out of 100 is where they go wrong, not that they aggregate reviews.

Rotten Tomatoes does a good job of both using a simple scale ("like it/don't like it"), and highlighting outlets that are higher-profile than others. They give the user control over granularity in how it aggregates, rather than taking on the responsibility themselves. Unfortunately, this doesn't stop its users from changing "72% of critics like this" into "This movie got a 72/100", which is pretty lame. :\

Avatar image for addfwyn
Posted By Addfwyn

@Grognard66 said:

Only MS titles targeted for this (and specifically XBLA Summer of Arcade) - so is this the work of immature fanboys or negative viral marketing by Sony paying their PR firms to do this?

Fanboys, sure maybe, doesn't seem that likely. Sony paying PR firms to do this? Yeah, that's just insane conspiracy theory shit. Sony isn't going to do something like that, they'd know that it would be found out and would blow back on them HUGELY if they tried to do something like that.

I'd put it at about the same level as the kids who hack companies with too much time on their hands, and not some sinister plot. Good on MC for doing something about at it at least.

Avatar image for levio
Posted By Levio

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Avatar image for ncjoker
Posted By NcJoker

still sounds fishy to me. who were these people, why did they act this way, and how do we know they won't just make new accounts on Metacritic?

Avatar image for grognard66
Posted By Grognard66

Only MS titles targeted for this (and specifically XBLA Summer of Arcade) - so is this the work of immature fanboys or negative viral marketing by Sony paying their PR firms to do this?

Avatar image for seriouslynow
Posted By SeriouslyNow

Bonuses based on review scores. Definitely not corrupt.

Avatar image for hailinel
Posted By Hailinel

@SithLibrarian said:

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher.

Wait, what? Seriously? How awful :(

Yep. Some developers are only paid bonuses if their games surpass a particular metacritic score threshold. It is messed up.

Avatar image for getz
Posted By Getz

@BestUsernameEver said:

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

Yeah, metacritic is the target.

I'm with you; this smacks of anti-metacritic sentiment. A way to show off how flawed the system really is by abusing that system (e.g. Lulzec). Of course, I'm not condoning this bullshit because it still hurts innocent people in the process and is extremely childish.

Avatar image for lonely_ogre
Posted By Lonely_Ogre

That's cool, always nice to hear stuff like this! :)

Avatar image for lacke
Posted By lacke

There is nothing wrong with Metacritic. It's the people who are abusing it who's at fault. Why blame Metacritic for publishers determining bonuses based on review scores? Why blame Metacritic for consumers who only look for a review score instead of reading a text? That consumer would probably not read any text on any of those websites either.

I'm not a fan of Metacritic and rarely ever use it but I don't think it's the cause for those actions. It just makes them easier.

Avatar image for crazyhorse23
Posted By Crazyhorse23

I guess this is good. I just wish that metacritic good be less dominated by trolls. Judging by this story, developers really care what the scores are for their games, but it seems to me that with all the trolling, devs can never really be sure whether or not their product is really all that good.

Avatar image for donpixel
Posted By DonPixel

Metacritic = Troll Nation

Avatar image for acheron
Posted By Acheron

Good on them for actually looking into it and fixing the problem. I give Metacritic a 9/10.

Avatar image for librariangmr
Posted By librariangmr

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher.

Wait, what? Seriously? How awful :(

Avatar image for the_official_japanese_teabag
Posted By the_OFFICIAL_jAPanese_teaBAG
@stise said:

@subyman said:

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating

Fuck you man, SmokeDAWG420 is my ONLY trusted source for video game news and reviews.

You mean RepThePlantDawg420
Avatar image for mattclassic
Posted By mattclassic

What kind of sad group of individuals takes the time to organize rating bombings on random indy video games? Jesus, their lives must be completely devoid of joy.

Avatar image for sagesebas
Posted By sagesebas

Metacritic needs to get rid of the option to submit a user review and they need to make a minimum length for those reviews also why does anybody use metacritic for bonus payouts why don't they just, I dunno determine payouts based on how well the game sells

Avatar image for steelknight2000
Posted By steelknight2000

I always ignore user reviews anyway when it comes to games and movies.

I do think it's important for developers and publishers to consider criticism from a third party such as the users and press in order to get a less biased view of whether or not their game sucks.