Giant Bomb News

231 Comments

Power to the People, Until the Power's Out of Control

A negative review is one thing, but 20 of them? In an hour? Out of nowhere? The problems developers face when Metacritic goes awry.

Signal Games is not the only developer to experience a rash of strange, out-of-nowhere negative reviews on Metacritic.

Signal Studios had a problem last week.

The tiny developer of Toy Soldiers: Cold War discovered its game had been hit with a series of negative, score-only (meaning no text) user reviews on Metacritic overnight, dragging its overall user rating down. While the scores assigned by critics are important, so are user reviews--anyone visting Metacritic is free to sort games by user reviews, too.

Metacritic has been a lightning rod of criticism over the years, due to its ties to developer pay.

"There is a rash of fake negative user scores going about Metacritic and it hit TS:CW (and other games)," said Signal Studios community manager Logan DeMelt last week. "This means we need to hear your voice!"

DeMelt started mobilizing fans to add more reviews to the pile, incentivizing them with free download codes for Toy Soldiers: Cold War.

"If you write a user review," he said, "just being honest (we aren't bribing for positive) we will do the following: You write a review & post it, and we will put your name in for a chance for a prize on Friday. Every 10 new posts, we will drop a code out."

Of course, there are problems with this approach, especially in terms of the perception of a developer asking people to write reviews, albeit not advocating positive or negative, by dangling prizes around. DeMelt even got some Twitter flack about the concept from a friend, Sucker Punch community manager Colin Moore.

When I talked to Signal Studios president and creative director Douglas Robert Albright III on the phone last week, he told me he'd asked DeMelt to stop the promotion, understanding how people might interpret it.

"It doesn't matter what your intentions are, it's what the perception is," said Albright. "If that's the perception, then we'll just stop doing it. Because, honestly, he [DeMelt] had no intention of bribing people to get good scores."

"It was actually my fault," he continued. "I looked at the Metacritic and contacted them and told them 'This has been spammed.' We're not a bunch of dudes with a bunch of money laying around or whatever. It affects us, right? You can search by user scores and stuff like that. It's clearly spam. Metacritic just responded with this generic thing. All the intent was 'Well, this isn't our fault, we're just going to go on Metacritic and [ask users to] review the game.'"

Signal Studios noticed a spike in negative reviews overnight, without review text to accompany it.

Like it or not, Metacritic has become important to the games industry. It's a system with faults, and one that's come under enormous criticism over the years. But game publishers have few ways to determine success outside of sales--so they turn to Metacritic. Metacritic determines bonuses and royalty payouts for many. Thus, developers have reason to pay close attention.

The situation prompts hard questions for developers, especially small ones with financial destinies tied to something partially out of its control. There are few options. Asking for reviews could be perceived wrong, but what do you do if the negative reviews seem fake? How do you prove that? If you can't prove it but know you're right, do you gamble the possible backlash?

The motivations behind the user or users spamming Metatritic with negative reviews are unknown. Blind Internet rage? Sheer boredom? A new form of spam?

"I don't imagine there is some conspiracy," said Albright. "I think some folks just do annoying crap because they can. Like the kids who hack the leaderboards or idiots who deploy viruses. Ever play mailbox baseball?"

...no comment.

In any case, Albright hasn't received any evidence worth acting on--and neither has another studio.

The only reason Signal Studios even realized something was goofy on its Metacritic page was thanks to Supergiant Games noticing a similar explosion of negative user reviews overnight for Bastion in early September.

Signal Studios was tipped off to the Metacritic issue by Supergiant Games.

"I think it was sitting at around a mid-8 on Xbox 360 and at a 9 on PC, but on September 2 it had dropped into the 6s," said Supergiant Games creative director Greg Kasavin to me over email. "No additional user reviews were posted on that day (or at least no new negative reviews were posted), but I noticed that we had 20 new 'negative' user ratings that were entered for both versions of the game. This struck me and the rest of us as highly suspicious, because we were gathering new user ratings at a much slower rate than that in previous days. The idea that all of a sudden we would get 40 extremely disappointed people come and give us a 0 out of 10 rating all at the same time seemed very dubious."

Supergiant Games mentioned the bizarre nature of the user reviews over its Twitter and Facebook accounts and left it at that.

"It's the first time we've ever complained about something via our Facebook and Twitter so I didn't do it lightly," said Kasavin.

Bastion's user review score went up as a result--7.7 on Xbox 360, 8.0 on PC--but the damage was done. Kasavin flagged the issue with Metacritic but was told little could be done. Metacritic said it's very careful with user reviews, especially so if the user doesn't actually write a review. On Metacritic, it's possible to submit a "review" with only a score, making it difficult to determine whether someone simply registered an account and their negative review was their first submission--or spam.

Supergiant mentioned the Metacritic issue and users responded with positive reviews.

"We just had to let it go," he said. "We value Metacritic as a service and are the highest rated XBLA [Xbox Live Arcade] game released so far this year according to them. There are some really great user reviews of our game on there, and if ratings-bombing continues to be an issue for other products, I trust it's something that the team there will investigate a solution more closely."

I contacted Metacritic about this pattern of issues. Metacritic told me each review has a "report abuse" option, which sends the review to Metacritic's team to possibly delete the review or ban the user.

"If any interested party feels that there has been a group of illegitimate user ratings (score only) entered for its game, they can contact me through the website and we'll investigate the issue," explained co-founder Marc Doyle. "We track each rating and can delete any that appear to be illegitimate or suspicious to us after a staff review of the rating data in question."

For now, for better or worse, that's the way the system works.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," said Albright. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
231 Comments
Posted by White_Silhouette

@Delta_Ass: it was from the futureshop website, but yeah the FF symbol is their logo.

Posted by PrivateIronTFU

@Delta_Ass: Seinfeld. Nice.

Posted by DaBuddaDa

Metacritic is the most useless, idiotic concept that has plagued the entire industry. Developers need to stop giving a flying fuck about it, and publishers need to learn that Metacritic scores mean shit all for sales numbers. It is completely, utterly useless.

Posted by lokilaufey

@White_Silhouette: Did you ever see the Amazon page for the Star Wars BluRay set? That had 1.5-2 out of 5 stars for weeks up until release and that is a site that even requires text with the scores. All this from people who had never seen or touched the set, but were going off of clips or information they had heard that they were unhappy about. People "reviewing" products yet to be released because they are unhappy about what's been shown off or have an agenda to push has been going on for awhile and probably will continue.

Posted by Fizzy

Interesting. I always wondered why some awesome games received oddly low scores on metacritic.

Posted by AndrewB

Amazon gets review bombed just as frequently. Recently I noticed a ton of negative reviews for Deus Ex Human Revolution based solely on the point that it requires a Steam install. It's a valid point of criticism, but not enough to justify an instant 1 star review.

Other games get lambasted for their DRM.

Posted by nmarchan

There's also the fact that people who work in the industry have been caught voting down competing games on metacritic while voting up their own. There was a thread on GAF a few months ago, where a few metacritic reviewers were giving super high scores for Bioware RPGs and super low scores to... I think it was the new Witcher game. Turns out when you look up their username, it brought you to a linkedin account showing they worked for EA or something like that.

I wouldn't assume these rushes of negative reviews are all from fans. Do some research, you may actually find something worse going on.

Edited by Three0neFive

Reminds me of when Bioware was blaming DA2's low Metacritic score on /v/. Because it's impossible that, you know, that game was just a fucking awful yaoi fanwank.

But that's besides the point. lol, metacritic.

Posted by krazy_kyle

Review scores are very unreliable to the point where I do not bother with Metacritic or any other large gaming site like gamespot or IGN. I just use my own initiative now and it has served me pretty well so far. I still read reviews to see what the reviewer has to say and at times I see the reviewer mentioning plenty of good things about the game but the game would get a 7/10 for example. If a game looks good, I buy it, if it looks bad, I don't bother. People need to think for themselves instead of relying on review scores to decide on whether to buy a game or not.

Posted by SaturnPrime

The sad fact is that user reviews on the internet should not be using such a stratified scale for anonymous ratings. The larger the scale, the more weight a 0/10 or 10/10 vote bombing will skew results by mean average or even median scores. Most users don't understand or differentiate the ten point scale with 0.5 or 0.1 increment on established sites like IGN or Gamespot had; why would we allow a ten point scale for anonymous submitters. An unrealistic number of ratings are always on the extremes on Amazon, Yelp, Rotten Tomatoes as well. This is not unique to Metacritic, except ratings at Metacritic get tied to bonuses and financial progressions for unfortunate reasons. A thumbs up, thumbs down vote or 5 point scale would help, but not solve the issue. Personally I have ignored user ratings on Metacritic and would not display that user score on the games page or in search results without a click-through to a user reviews page.

Posted by dox

@Vinny_Says: The only reason they care is because consumers review those sources. If consumers didn't care then developers would not care.

Edited by Hyperfludd

Great write-up, and it'll hopefully add a reason for publishers to not depend so heavily on the site.

Posted by OriginalGman

Metacritic's interface is ass. Just use Gamerankings. Problem solved.

Posted by Fizzy

@krazy_kyle said:

Review scores are very unreliable to the point where I do not bother with Metacritic or any other large gaming site like gamespot or IGN. I just use my own initiative now and it has served me pretty well so far. I still read reviews to see what the reviewer has to say and at times I see the reviewer mentioning plenty of good things about the game but the game would get a 7/10 for example. If a game looks good, I buy it, if it looks bad, I don't bother. People need to think for themselves instead of relying on review scores to decide on whether to buy a game or not.

That's what I usually do. I can't understand why games that look amazing get horrible reviews because of stuff like "Oh, the story is boring."....I can understand if a story is boring, but if it's not a story driven game it shouldn't warrant a 6 or 7 because of it.

Edited by MormonWarrior

User reviews on those sites are nothing but spam. The vast majority of them are like this: (CoD: Black Ops) "COD SUX ITS LAME NOT REAL GAMING TOTALY BROKN. 1/10" I don't care who you are, a good game is a good game even if you're not a fan. The regular Metacritic score is a little better. At least they somewhat mask the hyperbole. Sometimes.

When taken in the right context, Metacritic is really useful. I can see that "Gee, Brad's review of Metroid Other M is quite a bit higher than what most people think and the complaints others have concern me. Maybe I'll GameFly it before buying it." It gives me an idea of what to expect from a game rather than blindly buying it. I don't put too much stock in the ratings anymore, but it often helps me sift the amazing games from the merely good ones so I can prioritize. Anything below the 70's is not worth playing ever.

Posted by SomeDeliCook

Reviews don't mean a damn thing compared to word of mouth.

Posted by MadLaughter

This same thing happened to Dragon Age 2. That game has legitimate issues, but there was a huge bombing of negative scores as the swell of frustration about so many pre-order bonuses occurred.

Posted by Vexxan

Proves why Metacritic is a BAD IDEA.

Posted by Mercanis

Metacritic is a reductive blunt instrument. Find writers you know and trust instead of using silly aggregators.

Posted by wrathofconn

Man, this is fucking stupid.

Posted by sirdesmond

@Vinny_Says said:

It's sad devs put so much importance on metacritic, even sadder when they care about user reviews where anyone can write anything

It's sad that devs have to put so much importance on Metacritic because their jobs, amount of pay, and future opportunities depend on the publishers that put so much importance on Metacritic because they need a collection of subjective numbers to tell them if something was a success or not.

Edited by MeatSim

They said how to fix it. Only allow a user rating if they have written a text review.

Posted by prestonhedges

@dox said:

@Vinny_Says: The only reason they care is because consumers review those sources. If consumers didn't care then developers would not care.

Citation needed.

Posted by Shaanyboi

Fuck metacritic. I hate aggregated review sites... Worst fucking thing.

Posted by Sammo21

This is why Metacritic shouldn't allow user reviews at all, especially before the game is even released to the public.

Posted by mordukai

Developers know Metacritic is crap. We know Metacritic is crap. Now someone please let the suits in the publishing house know about that.

Posted by dvorak

User scores are just a little bit more useless than critic scores. But we already knew that.

The joke is that the only people that actually get paid more if they receive better reviews are the devs.

Posted by Viking_Funeral

@Undeadpool: To be fair, regarding Dragon Age 2, it was the fans voicing their frustration at what they saw as bought reviews. It is the general consensus these days that Dragon Age 2 is not a great game, or even that horrible, rather being merely mediocre, but at the time the fans were seeing 94 & "Game of the decade!" reviews pop-up on Metacritic. Either reviewers were getting a very different experience than the fans, or something happened.

Then it was revealed that members of BioWare's team were stealth-posting 10 reviews on Metacritic. That, in addition to the horrible reaction BioWare had to the backlash, made Metacritic user reviews one battleground of many between the upset fans and BioWare.

Posted by JustinAquarius

I knew publishers took MetaCritic very seriously, but I didn't think that meant they took the USER reviews seriously too.

Posted by RobertOrri

@Mordukai said:

Developers know Metacritic is crap. We know Metacritic is crap. Now someone please let the suits in the publishing house know about that.

Yes.

Posted by HellBrendy

Metacritic can die in a fire. It's nothing but problems to the business.

Posted by Rem45

Great piece Patrick! And Metacritic should really review its policy for allowing 'random' reviews in an attempt to grief certain game developers.

Posted by Claude

I use metacritic. I like having a bunch of reviews to choose from and read. The user reviews and their overall score I take with a grain of salt. But I like bottom feeding too.

I'm not sure if metacritic fixed this, but I remember users spamming low scores before a game was even released a while back.

Posted by cikame

I hate metacritic.

Posted by PrivateIronTFU

Metacritic is not the problem. The game developers and publishers (and users) who take Metacritic so seriously are the problem. If you want to hate something, hate them.

Posted by DrRandle

Metacritc and numerical critical scores in general are a disease to this and every industry. There is no true value to any of these systems. It is all ludicrous. You cannot truly tell the quality of the game with a random number grasped from the air. Reading any number of user reviews on Apps on Amazon's App Store, for example, shows why people should not be allowed to just throw scores into the air. It's terrible, and nobody is providing any real qualitative statements to back it up.

Nobody gives numerical scores when they critique Van Gough. Nobody assigns an arbitrary number of stars to Mozart. Thumbs do not apply to the works of shakespeare. So if we are truly going to examine and critique video games, in order to both learn how to make better ones, and in form people what is good and bad about each one, then we need to move away from this system of instant gratification.

Posted by wickedsc3

@RobertOrri said:

@Mordukai said:

Developers know Metacritic is crap. We know Metacritic is crap. Now someone please let the suits in the publishing house know about that.

Yes.

I'm sure they know its crap. But a bad review is a bad review, no matter where it comes from you don't want your game getting negative reviews.

Edited by ProfessorEss

Y'know I'd put a lot more blame on Metacritic if I didn't think the whole reviewing process was kinda jacked from top to bottom. 
 
If reviews are written totally subjectively (because apparently even a modicum of objectivity has been deemed and excepted as impossible) and require a number score even though every reviewer out there seems to be against them, then what's the use of any of it to begin with? 
 
And why do they require number scores? That's my big question, and Giant Bomb is a great example. If Giant Bomb truly doesn't believe scores are important (possibly even detrimental), or that their five point scale doesn't apply to standard mathematical conversions then why is there a score at all? If the stars truly represent what they say then why not simply rate the game with a qualifying statement instead of something that can be so clearly translated into a number? 
  

@benjaebe: I do think Sessler nails it here but he raises the same question. He says "we hate scores but, you know, we have to so..." but no one ever seems to explain why they have to. I realize people have bosses, companies have policies, by how can I accept a reviewer who hates scores as having any integrity if they're putting a score that they don't believe in on every piece that they do?
 
As long as (pretty much) every reviewer out there is playing right into Metacritic's hands by sending them numbers that they know full well are going to be mathematically converted to a 100 point scale I can't see how we can place all the blame squarely on Metacritic. 
 
...oh and user reviews, man I can't understand how any pub, dev, or customer would be bothered with that bag of shit. The fact that Bastion's score could drop so drastically so fast is disturbing, but the fact that a Twiitter from Kasavin could raise it back up so quickly is almost just as disturbing.

Posted by Fobwashed

This exact thing is a pretty huge problem in the XBLI section on XBL as well. It seems like any game that reaches the top 60 gets a surge of 1 star ratings almost without fail. Whether this is the work of a small group of individuals trying to bring down the ratings of specific games or a rival dev going out of his/her way to try to bring their own game back into the spotlight is as of yet unknown but Microsoft doesn't plan on doing anything about it. Maybe ever.

It's sad.

Posted by WilliamHenry

@Vegsen said:

Proves why Metacritic is a BAD IDEA.

Metacritic is not a bad idea, its actually a great idea, hows its used is the problem. How can having a large number of different reviews all in one place be a bad idea? The fact that bonuses are paid based on Metacritic scores and the bad policing of spam reviews are the problems, not Metacritic itself.

All reviews should require text. Just picking a number isn't sufficient enough. All reviewers should also have their reviews approved for a period of time to make sure they're not spam or fanboys.

Posted by BisonHero

@Delta_Ass said:

"Excuse me I'd like to give a user review of 1.0."

"Certainly. May I ask why?"

"For spite."

"Spite?"

"That's right. I don't care for the developer."

"I don't think you can review a game for spite."

"What do you mean?"

"Well if there was some problem with the game, if it were unsatisfactory in some way, then we could do it for you, but I'm afraid spite doesn't fit into any of our conditions for a review."

"That's ridiculous, I want to review it. What's the difference what the reason is?"

You can't review a game based purely on spite."

"Well so fine then... then I don't like it and then that's why I'm giving it a 1."

"Well you already said spite so..."

"But I changed my mind..."

"No... You said spite... Too late."

This is pretty much it. Or it's 4chan morons "doing it for the lulz".

Posted by N7

@JoeyRavn said:

What worries me most is that if I sincerely dislike a game (for example, Bastion) and give it a low score, my rating will be dismissed as "spam". What an awesome tool Metacritic is.

Either you fucking love a game, or you're just a butthurt loser spammer.

Metacritic has never been good for user reviews. It seems like only the worst of the youtube commenters go there and post even more bullshit. Why are publishers unaware of that?

Posted by Matoya

Who the fuck buys games based on their metacritic USER reviews?

Posted by Undeadpool

@Paul_Is_Drunk: The issue then becomes (beyond "two wrongs don't make a right") what did these people actually hope to accomplish? I remember hearing about the "stealth reviews" and seeing the 0/10 user reviews popping up at around the same time, so I'm not sure you can completely say that one was in reaction to another. I actually think DAII turned out fine (better than Origins in some ways, worse in others), it was just weird seeing THAT many people get THAT dedicated to absolutely ruining this game and then have it amount to nothing. DAII outsold Origins (which is merely a statement of fact and not a measure of quality) and yeah, most people have now just kind of shrugged their shoulders and agreed that it wasn't that great.

Posted by WilliamHenry

@ProfessorEss said:

Y'know I'd put a lot more blame on Metacritic if I didn't think the whole reviewing process was kinda jacked from top to bottom.

If reviews are written totally subjectively (because apparently even a modicum of objectivity has been deemed and excepted as impossible) and require a number score even though every reviewer out there seems to be against them, then what's the use of any of it to begin with?

And why do they require number scores? That's my big question, and Giant Bomb is a great example. If Giant Bomb truly doesn't believe scores are important (possibly even detrimental), or that their five point scale doesn't apply to standard mathematical conversions then why is there a score at all? If the stars truly represent what they say then why not simply rate the game with a qualifying statement instead of something that can be so clearly translated into a number?


@benjaebe: I do think Sessler nails it here but he raises the same question. He says "we hate scores but, you know, we have to so..." but no one ever seems to explain why they have to. I realize people have bosses, companies have policies, by how can I accept a reviewer who hates scores as having any integrity if they're putting a score that they don't believe in on every piece that they do? I'm not singling out Giant Bomb or Sessler but they're good examples because they have some clout to try and change this if they really believed this.

As long as (pretty much) every reviewer out there is playing right into Metacritic's hands by sending them numbers that they know full well are going to be mathematically converted to a 100 point scale I can't see how we can place all the blame squarely on Metacritic.
...oh and user reviews, man I can't understand how any pub, dev, or customer would be bothered with that bag of shit.

Really, you don't understand why scores are needed? Its because people are fucking lazy. They don't want to read a review, they just want to see a number and make their decision based on that.

Posted by RichieJohn

@Meatsim: Agreed. A written review should be required. I also they should be moderated to some degree before they're included in the actual user score.

Posted by LLJK_Jasta

@RobertOrri said:

@Mordukai said:

Developers know Metacritic is crap. We know Metacritic is crap. Now someone please let the suits in the publishing house know about that.

Yes.

+1

Posted by prestonhedges

@BisonHero said:

@Delta_Ass said:

"Excuse me I'd like to give a user review of 1.0."

"Certainly. May I ask why?"

"For spite."

"Spite?"

"That's right. I don't care for the developer."

"I don't think you can review a game for spite."

"What do you mean?"

"Well if there was some problem with the game, if it were unsatisfactory in some way, then we could do it for you, but I'm afraid spite doesn't fit into any of our conditions for a review."

"That's ridiculous, I want to review it. What's the difference what the reason is?"

You can't review a game based purely on spite."

"Well so fine then... then I don't like it and then that's why I'm giving it a 1."

"Well you already said spite so..."

"But I changed my mind..."

"No... You said spite... Too late."

This is pretty much it. Or it's 4chan morons "doing it for the lulz".

"I don't think the game's any good."

"You're just a butthurt spammer! Troll! Everything's amazing and nothing is bad!"

"Except this. I wasted money on this game."

"If it's bad, why not write a ten-thousand paragraph review about it, huh?!"

"Because I've wasted enough time on this game already? And who reads these reviews, anyway? It's not like I'm getting paid for this shit."

"Troll! Spam! Lulz! Butthurt! BBS Door Games!"

Posted by kpaadet

Metacritc is not crap, publishers deciding bonuses etc off Metacritc score is crap.

Posted by sugetipula

I think the oficial Metacritic scores are fine( they do represent the majority of profesional reviewers after all),

but the user ones are bullshit. Whoever says not to listen to the profesional scores and just go by user ratings are idiots.