Giant Bomb News

139 Comments

Metacritic Finds, Bans Group of Users Unfairly Scoring Games

Users gaming the system to target Bastion, Toy Soldiers: Cold War and others are gone.

When things go sideways on Metacritic, developers don't have many options for appeal.

Supergiant Games and Signal Studios were in awkward positions earlier this month, after the average user review score for each game plummeted overnight and seemingly without justification.

It was unclear who was responsible for the sudden change, and there are limited tools for those affected that don't come across as...weird.

The developers of Bastion and Toy Soldiers: Cold War asked fans to submit reviews to Metacritic, which resulted in a slight bump for each but an average score ultimately much lower than it was previously.

After speaking to Supergiant and Signal, I ran a story earlier this week about the challenges the studios faced in addressing these issues.

Metacritic has since identified the users responsible, both banning the users themselves and removing their scores putting things out of balance.

The users were targeting certain games, and hitting them with zero scores without a written review, raising eyebrows.

"A publisher of a different game let us know that he noticed that his game had been 'bombed' by a bunch of zero ratings in an unusually short period of time," said Metacritic co-founder Marc Doyle in an email to me today. "While investigating those ratings, we noticed a group of user accounts and activities that were clearly illegitimate and violated our terms of use."

Doyle's team found a group of users had been coordinating similar attacks on other games within Metacritic's database, allowing Metacritic to fix the "bombings" that had been targeted at other games.

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher. I've never heard of a publisher leveraging user reviews as a metric for payment because of situations like this, but especially for small studios, perception is king, which is why user reviews remain very important.

Signal Studios president and creative director Douglas Robert Albright III had proposed a fix to the problem.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," he said. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
139 Comments
  • 139 results
  • 1
  • 2
  • 3
Posted by Hailinel

@SithLibrarian said:

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher.

Wait, what? Seriously? How awful :(

Yep. Some developers are only paid bonuses if their games surpass a particular metacritic score threshold. It is messed up.

Posted by SeriouslyNow

Bonuses based on review scores. Definitely not corrupt.

Posted by Grognard66

Only MS titles targeted for this (and specifically XBLA Summer of Arcade) - so is this the work of immature fanboys or negative viral marketing by Sony paying their PR firms to do this?

Posted by NcJoker

still sounds fishy to me. who were these people, why did they act this way, and how do we know they won't just make new accounts on Metacritic?

Posted by Levio

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Posted by Addfwyn

@Grognard66 said:

Only MS titles targeted for this (and specifically XBLA Summer of Arcade) - so is this the work of immature fanboys or negative viral marketing by Sony paying their PR firms to do this?

Fanboys, sure maybe, doesn't seem that likely. Sony paying PR firms to do this? Yeah, that's just insane conspiracy theory shit. Sony isn't going to do something like that, they'd know that it would be found out and would blow back on them HUGELY if they tried to do something like that.

I'd put it at about the same level as the kids who hack companies with too much time on their hands, and not some sinister plot. Good on MC for doing something about at it at least.

Posted by NunsandRoses

@lacke: The nature of Metacritic condones people ignoring the text of a review, and simply looking at the score. Not to mention, they have a totally messed up way of converting scores (letter grades converted on an idiotic scale). The fact that they attempt to convert everything to a score out of 100 is where they go wrong, not that they aggregate reviews.

Rotten Tomatoes does a good job of both using a simple scale ("like it/don't like it"), and highlighting outlets that are higher-profile than others. They give the user control over granularity in how it aggregates, rather than taking on the responsibility themselves. Unfortunately, this doesn't stop its users from changing "72% of critics like this" into "This movie got a 72/100", which is pretty lame. :\

Posted by SeriouslyNow

@Levio said:

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Oh I'm sorry, I meant corruptible. Does that please you?

Posted by Xeiphyer
@SeriouslyNow said:

@Levio said:

@SeriouslyNow said:

Bonuses based on review scores. Definitely not corrupt.

Are you sure you know what corrupt means?

Oh I'm sorry, I meant corruptible. Does that please you?

The reviewers aren't getting the bonuses, the team that made the game is, so its not corrupt to be paying a bonus based on how well a game reviews.
 
 It is pretty disgusting though, but it makes sense. It does encourage the developers to make a good game, but honestly, anyone making games is trying to make a good game because the industry is highly competitive and everyone in it wants to be there and works hard to stay there.
Posted by solidejake

They should ban G4/X-Play.

Posted by jackelbeaver

>people who complain that originality in gaming is dieing, that all that sells is big fat shooters.

and then nobody buys big original AAA games that ninja theory, platinum games, or grasshopper studios makes.

self fulfilling prophecy

Posted by Dark_Lord_Spam
So Metacritic decided to fix this problem - which apparently involved ToS violations - only after gaming press pointed it out. Encouraging stuff. Thanks for the heads up anyway, Patrick.
 
@gladspooky said:

They should just ban all reviews. Problem solved.

Corrected.
Edited by Kosayn

Metacritic is what it is, but it shouldn't be used for bonuses. I think the critical community for gaming is still too embedded and too prone to repeating sound bites as their own opinions. Needs more time to grow.

Also, what's funny is how little abuse there actually is of things like this that are basically anonymous and without consequences. Jackasses are more of a rare and valuable commodity than you might think.

Posted by vinsanityv22

that fix sounds questionable. I mean, that system works pretty great for Gamefly, but still - it's really easy to give your game a 1 and write a terribly-written sentence like, "Dis game is teh suck Call of Duty is better only n00bs and kiddiez play dis game"

Posted by rambostyrer

This is why I more and more stick to stuff like Totalbiscuits "WTF IS" series and giantbomb quicklooks......

Posted by creiij

It's good they ban people who give games a zero so we can get honest opinions about a game. But seriously, who really cares about Metacritit? It's not a professional review site since anyone can register and give scores (unless you give a zero then you will get banned).

I stick to sites like IGN, Giantbomb and RPS for reviews and Youtube or Giantbomb for some gameplay footage.

Posted by Elazul

@creiij: Well yeah, most people here probably don't care about metacritic, but for people working at developers who could potentially be stripped of bonuses and royalty payments because of stunts like this, it's kind of a big deal.

Posted by EPGPX

The idea of Metacritic seems rather stupid and it's more laughable when you say publishers pay out bonuses/royaltie checks to developers based on metacritic scores. The system is flawed and as the article states, the system can easily be abused both ways and just imagine how many retarded developers depend on false scores just to be rewarded bonuses is hysterical. Actually, they're not stupid at all cause they're getting more money based on unreliable false scores and its the publishers who are major dumb asses relying on such stupidity that baffles me.

How does one depend on a review aggregator score for bonuses is beyond me. That's just dumb. I mean I'm pretty sure the movie business doesn't do this. Unless they somehow use the Rotten Tomatoes tomato-meter for bonuses too, then this is just one big farce.

Posted by Hilts

The solution is even simpler. Get rid of user reviews from Metacrtic all together. Who honestly looks at them?

Posted by BoG

Good. However, eliminating trolls should be business as usual, not some huge breakthrough.
Moderator
Posted by doe3879

Reading the title and half way down the article,

I thought it was about PC Leader Board Score hacking and they're doing something about it, and I go "ABOUT TIME!!!!

but no :(

Posted by huser

@EPGPX: I agree on the stupidity of using review aggregator's as a measure for bonuses, but using the movie industry as the example isn't exactly the proper analogy. Everyone knows who Spielberg, Lucas, etc are. Until game devs have that level of facetime and recognition, they have far less power to ask for 200 million dollar budgets and huge paydays up front for their work. Given that, game devs are going to have to find some metric to reward successes. Gamesales are certainly one option, but it probably has to be something.

Posted by imoTEP12

I doubt doing this will permanently stop any trolls.

Posted by Aaron_G

people need to get lives if they have time to do this.

Posted by WilliamRLBaker

What the hell bastion was an amazing game...

Posted by jmrwacko

Trolls should be thrown in prison, where trolling for attention only results in unnecessary penetration.

Posted by Example1013

Man, some people are thick.

Posted by AssInAss

Anyone got reminded of the old Gamespot user review days?

Posted by iam3green

well good that they fixed it. pretty weird to just have a random attack like that. usually when a raid like that happens it gets around the net.

Posted by MindChamber

At first Im like... if the metacritic score is so important that it effects the success of a game then maybe these users should be charge for fraud, seeing how employees are given bonuses etc based on it.. Then again.. These trolls show just how flawed metacritic really is .. and hopefully developers will stop relying on an easily manipulated system to judge their games by.

Posted by Obinice

I had never even heard of metacritic. I wonder how popular it is in the UK.

Posted by Meowshi

This sucks considering how highly-regarded Metacritic is.

Of course, Metacritic shouldn't be as important as it is.

Edited by MeatSim

Nobody should be putting any kind of stock in user ratings with no text reviews.

Posted by MisterMouse

good thing they went in and banned the accounts

Posted by gbpacker

Because now metacritic actually matters...

Posted by Boopie

it was always obvious it's Sony fanboys no mystery here at all

Posted by big_jon

Good, douche bags.

Posted by MURDERSMASH

@Hailinel:

As sad as that is, Metacritic can't be blamed for such behavior. That's the publishers who are putting so much stock in the site. In my opinion, the only thing off the top of my head that they're doing seriously wrong is using some strange weighting system to determine the overall score.

Posted by CraigAA1028

I wish Giant Bomb could do this every once in awhile for user review ratings, but since the team at GB is small I don't expect it or blame them for not policing it. I removed almost all my user reviews and decided never to post them here again since a team of about 10 trolls would just wait for me to post a review and then bomb it with negative feedback within hours of it going up. Trolls fucking suck