Giant Bomb News


Power to the People, Until the Power's Out of Control

A negative review is one thing, but 20 of them? In an hour? Out of nowhere? The problems developers face when Metacritic goes awry.

Signal Games is not the only developer to experience a rash of strange, out-of-nowhere negative reviews on Metacritic.

Signal Studios had a problem last week.

The tiny developer of Toy Soldiers: Cold War discovered its game had been hit with a series of negative, score-only (meaning no text) user reviews on Metacritic overnight, dragging its overall user rating down. While the scores assigned by critics are important, so are user reviews--anyone visting Metacritic is free to sort games by user reviews, too.

Metacritic has been a lightning rod of criticism over the years, due to its ties to developer pay.

"There is a rash of fake negative user scores going about Metacritic and it hit TS:CW (and other games)," said Signal Studios community manager Logan DeMelt last week. "This means we need to hear your voice!"

DeMelt started mobilizing fans to add more reviews to the pile, incentivizing them with free download codes for Toy Soldiers: Cold War.

"If you write a user review," he said, "just being honest (we aren't bribing for positive) we will do the following: You write a review & post it, and we will put your name in for a chance for a prize on Friday. Every 10 new posts, we will drop a code out."

Of course, there are problems with this approach, especially in terms of the perception of a developer asking people to write reviews, albeit not advocating positive or negative, by dangling prizes around. DeMelt even got some Twitter flack about the concept from a friend, Sucker Punch community manager Colin Moore.

When I talked to Signal Studios president and creative director Douglas Robert Albright III on the phone last week, he told me he'd asked DeMelt to stop the promotion, understanding how people might interpret it.

"It doesn't matter what your intentions are, it's what the perception is," said Albright. "If that's the perception, then we'll just stop doing it. Because, honestly, he [DeMelt] had no intention of bribing people to get good scores."

"It was actually my fault," he continued. "I looked at the Metacritic and contacted them and told them 'This has been spammed.' We're not a bunch of dudes with a bunch of money laying around or whatever. It affects us, right? You can search by user scores and stuff like that. It's clearly spam. Metacritic just responded with this generic thing. All the intent was 'Well, this isn't our fault, we're just going to go on Metacritic and [ask users to] review the game.'"

Signal Studios noticed a spike in negative reviews overnight, without review text to accompany it.

Like it or not, Metacritic has become important to the games industry. It's a system with faults, and one that's come under enormous criticism over the years. But game publishers have few ways to determine success outside of sales--so they turn to Metacritic. Metacritic determines bonuses and royalty payouts for many. Thus, developers have reason to pay close attention.

The situation prompts hard questions for developers, especially small ones with financial destinies tied to something partially out of its control. There are few options. Asking for reviews could be perceived wrong, but what do you do if the negative reviews seem fake? How do you prove that? If you can't prove it but know you're right, do you gamble the possible backlash?

The motivations behind the user or users spamming Metatritic with negative reviews are unknown. Blind Internet rage? Sheer boredom? A new form of spam?

"I don't imagine there is some conspiracy," said Albright. "I think some folks just do annoying crap because they can. Like the kids who hack the leaderboards or idiots who deploy viruses. Ever play mailbox baseball?" comment.

In any case, Albright hasn't received any evidence worth acting on--and neither has another studio.

The only reason Signal Studios even realized something was goofy on its Metacritic page was thanks to Supergiant Games noticing a similar explosion of negative user reviews overnight for Bastion in early September.

Signal Studios was tipped off to the Metacritic issue by Supergiant Games.

"I think it was sitting at around a mid-8 on Xbox 360 and at a 9 on PC, but on September 2 it had dropped into the 6s," said Supergiant Games creative director Greg Kasavin to me over email. "No additional user reviews were posted on that day (or at least no new negative reviews were posted), but I noticed that we had 20 new 'negative' user ratings that were entered for both versions of the game. This struck me and the rest of us as highly suspicious, because we were gathering new user ratings at a much slower rate than that in previous days. The idea that all of a sudden we would get 40 extremely disappointed people come and give us a 0 out of 10 rating all at the same time seemed very dubious."

Supergiant Games mentioned the bizarre nature of the user reviews over its Twitter and Facebook accounts and left it at that.

"It's the first time we've ever complained about something via our Facebook and Twitter so I didn't do it lightly," said Kasavin.

Bastion's user review score went up as a result--7.7 on Xbox 360, 8.0 on PC--but the damage was done. Kasavin flagged the issue with Metacritic but was told little could be done. Metacritic said it's very careful with user reviews, especially so if the user doesn't actually write a review. On Metacritic, it's possible to submit a "review" with only a score, making it difficult to determine whether someone simply registered an account and their negative review was their first submission--or spam.

Supergiant mentioned the Metacritic issue and users responded with positive reviews.

"We just had to let it go," he said. "We value Metacritic as a service and are the highest rated XBLA [Xbox Live Arcade] game released so far this year according to them. There are some really great user reviews of our game on there, and if ratings-bombing continues to be an issue for other products, I trust it's something that the team there will investigate a solution more closely."

I contacted Metacritic about this pattern of issues. Metacritic told me each review has a "report abuse" option, which sends the review to Metacritic's team to possibly delete the review or ban the user.

"If any interested party feels that there has been a group of illegitimate user ratings (score only) entered for its game, they can contact me through the website and we'll investigate the issue," explained co-founder Marc Doyle. "We track each rating and can delete any that appear to be illegitimate or suspicious to us after a staff review of the rating data in question."

For now, for better or worse, that's the way the system works.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," said Albright. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
231 CommentsRefresh
Posted by AceBlack19

Reading metacritic user reviews is an exercise in endurance. The negative Bastion user reviews were filled to the brim with misinformation and lies about the games controls and plot.

While a website where everyone can pitch in their own scores is a great idea in theory, its flawed in practice when idiots who never take the time to actually play the game (or at least click the options menu) decide they're allowed to put in their own two cents.

Posted by Pop

That's some bullshit, there should just be a no text user score, so that this kind of spam couldn't happen. Cause trolls don't like to spend time writing and stuff. And it sucks that game developers have to depend on metacritic which doesn't have the best system. New whiskey site idea super score game and movie site :P. Put the top men on it right away.

Posted by spilledmilkfactory

I always thought of the user reviews on Metacritic as kind of a joke. Most of them are either 10s or 0s. But it's still bullshit that some people step all over these developers' hard work for no discernable reason. Even if that affects one sale, that's money that an indie dev is losing

Posted by Grognard66

Stupid fanboys. It's not too hard to figure out what happened here. Two exclusive Summer of Arcade games being targeted for negative reviews? I sense a juvenile Playstation fanboy...

Posted by Parsnip

I'm always baffled that the metacritic user ratings matter so much to publishers and devs.

Posted by swamplord666

I don't know if it's still the case, but there was a time when exlusive AAA titles would bomb simply for the fact that fanboys from the opposing console would write it just to be trolls.

User reveiws are an awesome concept, but it should never be the end all. And sadly it's the case a little too often :/

Edited by ProfessorEss

  @WilliamHenry said: 

Really, you don't understand why scores are needed? Its because people are fucking lazy.

Yeah, and I'm saying reviewers should either say: 
A. I believe scores are valuable because a lot of people don't have time or are too lazy to read my review 
B. I don't believe in scores so fuck anyone who's too lazy to read what I think about it. 
Just not: 
C. I don't believe scores, but here's a number just in case you're too lazy to read my review.

@Pop  said: 

That's some bullshit, there should just be a no text user score, so that this kind of spam couldn't happen. Cause trolls don't like to spend time writing and stuff. And it sucks that game developers have to depend on metacritic which doesn't have the best system. New whiskey site idea super score game and movie site :P. Put the top men on it right away.

I'd go so far as to say not just text but a minimum character count so at the very least a troll has to hold down his M key for a good 15-20 seconds. And get them off the game's main page. 
Posted by Ghost_Cat

We have come to a point in modern society where numbers (and associated color schemes) mean more than in depth discussion. Numbers suddenly suddenly mean everything, and consumers assume them as fact without doing a little bit of detailed researched. Whatever happened to reading?

Posted by Shuborno

@metalsnakezero said:

This is a really broken system if things like this is happening.

It's always been a broken system.

"Statistics" generated without a public methodology are not useful for public discourse.

Posted by lacke

@Vinny_Says: I think it's more sad that we can't take user reviews more seriously because of blatant bias.

Posted by GeekDown

I use Metacritic a lot, but I never read the user reviews. I'm not even a registered member and I think that's the case with most of it's userbase. People are going there to see how well a game, album or movie is being received by professional journalists. I don't think developers should worry too much though.

Posted by Six

what score would metacritic get if its scoring system was used on itself and users rated metacritic based on the service it provides?

Edited by hermes

@WilliamHenry said:

@Vegsen said:

Proves why Metacritic is a BAD IDEA.

Metacritic is not a bad idea, its actually a great idea, hows its used is the problem. How can having a large number of different reviews all in one place be a bad idea? The fact that bonuses are paid based on Metacritic scores and the bad policing of spam reviews are the problems, not Metacritic itself.

All reviews should require text. Just picking a number isn't sufficient enough. All reviewers should also have their reviews approved for a period of time to make sure they're not spam or fanboys.

Metacritic as an aggregation of sites reviews is a great idea. Metacritic as an average of points is a terrible idea.

Other sites like rotten tomatoes seems to get it right by paying attention to the review text more than applying a formula to incompatible sites numeric reviews.

Posted by TadThuggish

The way to fix Metacritic is to never use Metacritic.

Posted by jayjonesjunior

@Undeadpool said:

I remember when this exact thing happened for Dragon Age II...and I'm sure it will absolutely happen for Gears of War 3. It's really said that publishers prize User Reviews this highly for this exact reason. Yeah, it wouldn't be terribly smart to just tell your user base you flat-out don't care unless they're "critics," but tying THIS much to them is just primed for this kind of random mayhem.

I'm pretty sure Dragon Age 2 deserved that.

Posted by BisonHero

Unless user reviews are the ONLY form of reviews available to the website (like with Amazon), I don't think any website really benefits from user-submitted scores. Too much potential for trolling, or for idiots with an axe to grind who just rate something as either a 0/10 or a 10/10, despite very few games deserving either of those scores.

Wanna submit a review of a game? Become a games journalist. That takes enough effort that it tends to weed out most trolls.

Edited by benjaebe

@ProfessorEss said:

As long as (pretty much) every reviewer out there is playing right into Metacritic's hands by sending them numbers that they know full well are going to be mathematically converted to a 100 point scale I can't see how we can place all the blame squarely on Metacritic.

You can't blame Sessler or Giant Bomb for "sending" Metacritic numbers which are reinterpreted without their consent, nor should you expect them to change their methods of reviewing because Metacritic exists and is perverting their work.

EDIT: To make the point more clear, they aren't agreeing to having their reviews appear on Metacritic. Metacritic aggregates the reviews and converts them without any reviewer input.

Posted by DonPixel

Metacritic is troll country

Posted by NegativeCero

I understand that he wasn't trying to bribe positive reviews, but that is how I took it when I first read it. The problem I have is how is someone supposed to write a review for a game they haven't played, at least I assume they haven't since they would be going for that code. If they are doing it for the code, clearly the "promotion" worked. I get that user reviews are dumb anyway, but I hate the way they went about this.

Posted by gbrading

Look no further than a little game called Kane and Lynch if you want to see a prime example of this sort of score skewing in action.

Posted by roughplague

it's a silly thing of course, but it DOES factor into sales, they can lose a lot by a super low (spammed) user critic rate, because some people see a score, and they go "oh, I guess that's a shit game, not buying that" regardless of if it's critics or spam assholes.

Posted by CharlesSurge

No gamer I know cares about a Metacritic score.

Posted by spartan1017

metacritic is stupid. gamerankings is bettter

Posted by imsh_pl

I'm pretty sure that not many people take the users reviews seriously, there are like only 0s and 10s out there.

Posted by NcJoker

usually when you can't find who is responsible, it is the party with something to gain. in this case, that would be the companies paying the developers. if they are able to anonymously lower the meta critic rating of the game, then they are able to save money through paying lower bonuses, all while taking no risk. So, why wouldn't companies engage in this practice? The solution is for developers to refuse contracts which rely on a meta critic rating, rather than waiting for meta critic to solve the issue. Or, require account verification

Posted by Kartana

user-scores --> Lol!

Posted by Undeadpool

@jayjonesjunior: It deserved to have people spam 0/10 reviews before the game even came out? How the game turned out is irrelevant at that point, cause that's just straight-up lying. Also didn't make one lick of difference for the game itself.

Posted by Owlright

I don't use Metacritic except very rarely to check professional reviews, but I know a lot of my friends do put stock in the scores on Metacritic. It sucks for smaller studios like this that such a thing is going to affect their sales and perception. I don't think asking for reviews is a good way to deal with the problem, but realistically what else can they do? Sucky situation :|

I'm not a fan of Metacritic's set-up, and I wish that they'd employ a system similar Rotten Tomatoes, which seems to work fairly well. Not perfectly, mind, but certainly better than Metacritic's current set-up.

Posted by confideration

So basically Metacritic says: it's the Internet. SORRY.

Posted by Boiglenoight

I use Metacritic for professional reviews. If I'm interested in a game and it's gotten a low 80, I may look to user reviews for perspective but review bombs are obvious. If light of one, I'll just rely on impressions from game forums.

Posted by LiquidSwords

Bloggers, Metacritic, and Rotten Tomatoes, they are all the same: Worthless.

Posted by notha

@Yodzilla said:

I didn't think anyone paid attention to the user reviews. The aggregated "real" reviews yes but the user reviews have always been garbage.

last week, in an attempt to find some good obscure games, i browsed the metacritic scores by user score because a lot of the games didn't have critic scores.

Posted by LordXavierBritish

More and more I'm starting to think games journalism needs to just throw away the entire concept of review for something more akin to Quick Looks and Let's Plays.

More often than not I've been sold on a game more from seeing people actually play and enjoy it than reading a long list of, sometimes dubious, generalized opinions.

Posted by thedez

No one site should have all that POWER.

Posted by Fuga

hahaha, you can write a review without any words? wow

Posted by Manatassi

When it comes to games I completely ignore User Reviews. They are completely Useless.

Edited by kpaadet

I find it hard to believe it is as big of a deal as this article makes it out to be, especially considering they didnt even notice until someone else told them it was going on. What sane person has ever even gone Metacritic to read the user reviews?

Edited by 234r2we232

I've never used Metacritic, but I can understand why people do. Their fast-food approach to scoring seems to be made legitimate by some of the sites they leech from, but it's entirely why I find it to be not the least bit helpful. It's just another Internet thing that can be abused by lonely Internet jerks.

Reviews are only useful when you know enough about the reviewer - whether they're trustworthy, and what they enjoy as a person. Isn't that kind of the base imperativeness to the usefulness of a review anyway?

It's funny they included the Bastion thing in there "we complained via Twitter, and our game's score went up"... jerks :/

Posted by TheHBK

That game sucks ass anyway.

Posted by DFSVegas

He did his due diligence. He made it clear that they wouldn't favor positive reviews. That's the sound of a man who has faith in his product, and just want to try and right a wrong.

So naturally, CONTROVERSY. There always has to be some in this world where we truly have no real problems.

Posted by Arkasai

Metacritic needs some kind of community management, like real ID on Amazon...or some way of putting a limit on the number of anonymous accounts someone can post from. The ability to one-man astroturf a title undermines the whole purpose of having user ratings.

Posted by oldskooldeano

@LordXavierBritish said:

More and more I'm starting to think games journalism needs to just throw away the entire concept of review for something more akin to Quick Looks and Let's Plays.

More often than not I've been sold on a game more from seeing people actually play and enjoy it than reading a long list of, sometimes dubious, generalized opinions.

I'd say that's one of the reason GiantBomb is the format that it is, and why we are all here. That and Brad's dulcet tones of course.

Posted by BoG

I didn't realize that developers put so much emphasis on Metacritic, nor did I realize how (at least in the user review aspect) poorly run the site is. I don't think there is much of an excuse for this. If Metacritic wants to remain reputable, they should fix this.
Posted by HeadHitsConcrete

This is just actual proof that my ignoring of Metacritic scores is sensible. Before I did it because I disagree on a personal basis with most of what is popular, accepting that I have strange taste in music, movies, and games.

Posted by Curufinwe

They shouldn't allow user scores without a user review to back it up.

Posted by Swoxx

People are pricks

Posted by BRNK

@kpaadet said:

I find it hard to believe it is as big of a deal as this article makes it out to be, especially considering they didnt even notice until someone else told them it was going on. What sane person has ever even gone Metacritic to read the user reviews?

As the article mentions, many companies and employees get bonuses based on their game's Metacritic score... so for better or for worse, there can be a lot riding on that number.

Posted by WiqidBritt

exactly why I don't pay attention to "user" reviews, especially in aggregate form like this

Posted by Delerat

@benjaebe said:

A 4/5 on Giant Bomb becomes an 80, even if that's not the reviewers intention. And there's nothing they can do about it.

It feels like this should be on equal ground with misquoting someone or misrepresentation or something. They are taking people's words and skewing them to fit their system.

Posted by Wazzit

No one Fan should have all that POWER