Giant Bomb News

231 Comments

Power to the People, Until the Power's Out of Control

A negative review is one thing, but 20 of them? In an hour? Out of nowhere? The problems developers face when Metacritic goes awry.

Signal Games is not the only developer to experience a rash of strange, out-of-nowhere negative reviews on Metacritic.

Signal Studios had a problem last week.

The tiny developer of Toy Soldiers: Cold War discovered its game had been hit with a series of negative, score-only (meaning no text) user reviews on Metacritic overnight, dragging its overall user rating down. While the scores assigned by critics are important, so are user reviews--anyone visting Metacritic is free to sort games by user reviews, too.

Metacritic has been a lightning rod of criticism over the years, due to its ties to developer pay.

"There is a rash of fake negative user scores going about Metacritic and it hit TS:CW (and other games)," said Signal Studios community manager Logan DeMelt last week. "This means we need to hear your voice!"

DeMelt started mobilizing fans to add more reviews to the pile, incentivizing them with free download codes for Toy Soldiers: Cold War.

"If you write a user review," he said, "just being honest (we aren't bribing for positive) we will do the following: You write a review & post it, and we will put your name in for a chance for a prize on Friday. Every 10 new posts, we will drop a code out."

Of course, there are problems with this approach, especially in terms of the perception of a developer asking people to write reviews, albeit not advocating positive or negative, by dangling prizes around. DeMelt even got some Twitter flack about the concept from a friend, Sucker Punch community manager Colin Moore.

When I talked to Signal Studios president and creative director Douglas Robert Albright III on the phone last week, he told me he'd asked DeMelt to stop the promotion, understanding how people might interpret it.

"It doesn't matter what your intentions are, it's what the perception is," said Albright. "If that's the perception, then we'll just stop doing it. Because, honestly, he [DeMelt] had no intention of bribing people to get good scores."

"It was actually my fault," he continued. "I looked at the Metacritic and contacted them and told them 'This has been spammed.' We're not a bunch of dudes with a bunch of money laying around or whatever. It affects us, right? You can search by user scores and stuff like that. It's clearly spam. Metacritic just responded with this generic thing. All the intent was 'Well, this isn't our fault, we're just going to go on Metacritic and [ask users to] review the game.'"

Signal Studios noticed a spike in negative reviews overnight, without review text to accompany it.

Like it or not, Metacritic has become important to the games industry. It's a system with faults, and one that's come under enormous criticism over the years. But game publishers have few ways to determine success outside of sales--so they turn to Metacritic. Metacritic determines bonuses and royalty payouts for many. Thus, developers have reason to pay close attention.

The situation prompts hard questions for developers, especially small ones with financial destinies tied to something partially out of its control. There are few options. Asking for reviews could be perceived wrong, but what do you do if the negative reviews seem fake? How do you prove that? If you can't prove it but know you're right, do you gamble the possible backlash?

The motivations behind the user or users spamming Metatritic with negative reviews are unknown. Blind Internet rage? Sheer boredom? A new form of spam?

"I don't imagine there is some conspiracy," said Albright. "I think some folks just do annoying crap because they can. Like the kids who hack the leaderboards or idiots who deploy viruses. Ever play mailbox baseball?"

...no comment.

In any case, Albright hasn't received any evidence worth acting on--and neither has another studio.

The only reason Signal Studios even realized something was goofy on its Metacritic page was thanks to Supergiant Games noticing a similar explosion of negative user reviews overnight for Bastion in early September.

Signal Studios was tipped off to the Metacritic issue by Supergiant Games.

"I think it was sitting at around a mid-8 on Xbox 360 and at a 9 on PC, but on September 2 it had dropped into the 6s," said Supergiant Games creative director Greg Kasavin to me over email. "No additional user reviews were posted on that day (or at least no new negative reviews were posted), but I noticed that we had 20 new 'negative' user ratings that were entered for both versions of the game. This struck me and the rest of us as highly suspicious, because we were gathering new user ratings at a much slower rate than that in previous days. The idea that all of a sudden we would get 40 extremely disappointed people come and give us a 0 out of 10 rating all at the same time seemed very dubious."

Supergiant Games mentioned the bizarre nature of the user reviews over its Twitter and Facebook accounts and left it at that.

"It's the first time we've ever complained about something via our Facebook and Twitter so I didn't do it lightly," said Kasavin.

Bastion's user review score went up as a result--7.7 on Xbox 360, 8.0 on PC--but the damage was done. Kasavin flagged the issue with Metacritic but was told little could be done. Metacritic said it's very careful with user reviews, especially so if the user doesn't actually write a review. On Metacritic, it's possible to submit a "review" with only a score, making it difficult to determine whether someone simply registered an account and their negative review was their first submission--or spam.

Supergiant mentioned the Metacritic issue and users responded with positive reviews.

"We just had to let it go," he said. "We value Metacritic as a service and are the highest rated XBLA [Xbox Live Arcade] game released so far this year according to them. There are some really great user reviews of our game on there, and if ratings-bombing continues to be an issue for other products, I trust it's something that the team there will investigate a solution more closely."

I contacted Metacritic about this pattern of issues. Metacritic told me each review has a "report abuse" option, which sends the review to Metacritic's team to possibly delete the review or ban the user.

"If any interested party feels that there has been a group of illegitimate user ratings (score only) entered for its game, they can contact me through the website and we'll investigate the issue," explained co-founder Marc Doyle. "We track each rating and can delete any that appear to be illegitimate or suspicious to us after a staff review of the rating data in question."

For now, for better or worse, that's the way the system works.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," said Albright. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
231 Comments
Posted by Max_Hydrogen

@LiquidSwords: Indeed, I have never, ever cared what Rotten Tomatoes has to say and the only reason I know of Metacritic's existence is because they talk about on GB; I have never even been on Metacritic's site.

If something is too popular or unpopular, I usually find that feel the opposite way about it.

Posted by Atary77

When will companies realize that Metacritic ,is freaking BROKEN!?

Online
Posted by MrMazz

Good job Patrick

I have never understood how devs and publishers use Metacritic to determine bonuses and what not You would think sales would be the driving factor. I would not give on F if my game on metacritic was a 24 but I sold 300 million copies casue I'd be rich

Posted by SpudBug

great article. This is a really shameful process. Metacritic should be more proactive about tracking down the source of these negative reviews.

I wouldn't be surprised to see competition for these publishers getting on metacritic and rating their games down. This kind of shit is going to make them irrelevant.

Edited by Zatoichi_Sanjuro

Metacritic determines bonuses and royalty payouts for many.

I wonder how many Metacritic accounts some devs have.

Posted by Zatoichi_Sanjuro

@SpudBug said:

great article. This is a really shameful process. Metacritic should be more proactive about tracking down the source of these negative reviews.

They need a synergy based paradigm, obviously.

Posted by DrLariat

@TheHBK said:

That game sucks ass anyway.

Dude you are so edgy I cut myself on your post.

Posted by Jerr

I think requiring justification in the score through actual text would be the best thing. What people don't think about as much (as it more affects consumers rather than developers) is the possibility of users spamming games with overly positive reviews, either to cause mischief or express their fanboydom. Of course developers wouldn't care, but I think that it would give consumers looking to purchase the game an inflated sense of how good it is perhaps. Just a view from another perspective on the same issue.

Posted by Nomin

So what, there are as many positively biased user reviews as negative ones anyway. Think the user ratings as some sort of 'hype' meter then it all makes sense. The critics ratings number actually is a fairly accurate representation on the game's reception on the media circles, no wonder the publisher and developers trust them as a source alongside million other figures. I think Metacritic is no less an authoritative source on a game's quality than any other single games 'review' sites, because it is simply a representation of a glut of them.

Posted by beeftothetaco

Fanboys and trolls from /v. That is all.

Posted by BlainN

Thank you for drawing attention to this abuse Patrick.  I hope it lights a fire under CBS Interactive to take these issues seriously.  And they are serious, despite what many of my fellow commenters seem to think.
 
Gaming is big business.  Big businesses require lots of people who know specialized skills: finance specialists, marketers, managers; HR folks.  It'd be great if all of those people were gamers as well, but that's not realistic.  How many people in the recording business don't know any Beethoven aside from the opening of the fifth symphony?  How many people in the publishing business have never read Hemingway?  How many people in the film business have never seen Bergman or Fellini?  These people don't make artistic decisions.  They make business decisions.  Their shareholders demand it.  And sites like Metacritic are the closest things to impartial opinions they've got.
 
And both scores matter.  Some people distrust critics and want the opinions of buyers.

Posted by BooDoug187

I love how the fix is simple, make the person write a word review, but does the site do it, "No we dont want to upset our users" what kind of excuse is that? Hell a simple sentence of "I hate this game it sucks" is better than some asshole 4chan wanna be trolls rating random games 0 "for the lutz!"

Posted by bransonhuggins

here's my thing. If you are only basing your opinion on a game, based on and arbitrary score given by other people, then maybe you shouldn't be playing to begin with. I for one, know that if I played based squarely on that, I would have missed a lot of good games. Just because someone gave something a bad review or score, doesn't mean you won't like it. Maybe you should go out there, try it for yourself, and then, I know, this is shocking, make your own opinion. Just saying.

Edited by tourgen

sounds like issues I've had with credit reporting agencies. yeah they suck but you don't get a choice whether you use them or not, and their information is always inaccurate. I don't think this deserves a news article. game devs just need to police metacritic. paying for reviews would be one way to do that.

actually, just cut right to it and PAY METACRITIC for good reviews.

Posted by ProfessorK

It sucks that it seems to be the norm to grief anyone, anywhere, anytime now and days. I hope they revise the system and take the issue into consideration.

Posted by Tordah

This wouldn't be an issue if Metacritic took any responsibility at all for how their site works. How can you allow textless reviews if those reviews still count towards the average score? It just doesn't make sense.

I know that even if they did require written reviews there would be a lot of bullshit spam reviews, and the resources required to filter every user review on a regular basis would be significant, but that's the only solution I see working.

Edited by Boopie

did that guy just refer to GiantBomb as "some random blog" ???

Posted by DragonBloodthirsty

What's that? It's a system where anonymous "this game sucks" reviews can cut into your pay, and you don't know who to look at first? I'll give you a hint: He pays the check that gets smaller.

Edited by ProfessorEss

@benjaebe: I didn't mean to sound like I was calling them out, and I should've said "making scores available" as opposed to "sending" - but I stand by my point that if you hate numeric scores, and you hate Metacritic using the scores you give games maybe consider not giving them scores. Or at least give us an explanation as to why you (especially reviewers with clout) have to give numeric scores.

Posted by Dizzyhippos

haters gonna hate

Posted by megalowho

Who pays for these kind of "services"? Or is it just trolling? So much shady shit goes on it's hard to tell sometimes. Great article, interesting problem.

Posted by BabyChooChoo

Another reason why I wish some people were just outright banned from the internet...

Posted by Rawrz

Why would anyone ever go by user scores? Most people just seem to have a habit of loving something and giving it a 10 or not liking it and just giving it a 1.

Posted by RedSox8933

Recently I've only read reviews from Giant Bomb. I believe that one of Jeff's main goals of the website was to reestablish the byline, and for me he and the crew have succeeded in doing this. I know their tastes, and I know when to trust a review and when to disregard it and buy the game anyway. And for me, that is the most important thing about any kind of critical journalism (if that's a thing).

Posted by CornBREDX
"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," said Albright. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversite and some standards."

I completely agree with this. I'm not big on scores, but I understand the reason Developers like Metacritic. They want to be able to judge what the public perception of the game is based solely on individual experiences. Metacritic helps funnel that into one place.

This is sound, in theory.

Metacritic should know by now how useful they are to developers and should have things in place that keep these kinds of things from happening.

That being said, I really have problems with the idea that Metacritic affects bonuses/pay. Again I get why it's useful from a development standpoint. It can really give you a sense of how much individual players (no matter how small majority the people who actual write them may actually be) enjoy or dislike your game and sometimes even why. However, this should determine how Developers make games in the future, I don't like the idea that this information can affect what they get paid. If they put in the work, they should still get paid. Sales will often determine what you're worth as a developer- although I suppose I have no defense to the argument that sales don't mean those who bought the game actually thought it was good, but on the same front I believe whether or not they buy another your bonuses or extra pay of whatever kind should come from this game not the possibility of better made next games.

I don't know, just my thoughts. Apparently I have a lot of them on this topic so I'll just leave it at that.

Interesting article.

Edited by shotgunsullivan

I have never used metacritic to help me decide if I want to buy a game, but I do like looking at it after a game has been out for about a month to see where it finally lands. If I want to know if a game is any good, I come here.

Posted by Branthog

Misleading title. It has nothing to do with "the people". If you put stock in shit that some website most people never even visit says (based on completely random visitors rating things) to the point that you base salaries and bonuses and careers and layoffs on it, then that's on you not on the people.

Edited by Dark_Lord_Spam

Fuck. FUCK. This is genuinely disheartening. I can't fathom intentionally attempting to obliterate any person's livelihood, but when they're trying their damnedest to bring innovation, ingenuity, and interest to an increasingly stale market? Really? Are these scumbag spammers trying to express disdain for indie projects? Are they truly just fucking bored? I wish I could go buy these games immediately to show some meager support (I would've already bought Bastion), but I unfortunately don't own a 360. Fuck this whole mess of a situation, man.

Posted by lolzords

The efficiency of metacritic, with its glanceable, if shallow, review consolidation if an enticing service that offers a truly useful service for those of us with little time in the day to digest full reviews, however, it is a shame that such erroneous errors can arise with such services. Circumstances that can have such a negative, monetary effects on game developers and must be remedied.

Posted by FoxMulder

I almost never solely rely on critic scores when it comes to anything. I have loved not so good buggy games and hated critically praised ones as well. I will look at scores and use them to sway, but if I am interested in something 100% nothing is going to stop me. As fo user reviews...I NEVER look at those unless they are on here.

Posted by MordeaniisChaos

I still can't believe bonuses are awarded based on anything but sales, as that's all the publisher should care about: how much money did they get. Sales should speak for themselves.

Edited by Xeirus

@RedSox8933 said:

Recently I've only read reviews from Giant Bomb. I believe that one of Jeff's main goals of the website was to reestablish the byline, and for me he and the crew have succeeded in doing this. I know their tastes, and I know when to trust a review and when to disregard it and buy the game anyway. And for me, that is the most important thing about any kind of critical journalism (if that's a thing).

Yup, once you realize what this site is about and have a feel for the staff you're crazy to go to a random website and trust it. GB is really the only place I trust anymore. Watching Jeff's at-home video the other day pretty much reminded me why I trust the site, he sort of touches on the subject of things like this in the video and I loved it.

@Dark_Lord_Spam said:

Fuck. FUCK. This is genuinely disheartening. I can't fathom intentionally attempting to obliterate any person's livelihood, but when they're trying their damnedest to bring innovation, ingenuity, and interest to an increasingly stale market? Really? Are these scumbag spammers trying to express disdain for indie projects? Are they truly just fucking bored? I wish I could go buy these games immediately to show some meager support (I would've already bought Bastion), but I unfortunately don't own a 360. Fuck this whole mess of a situation, man.

I agree, it's extremely frustrating, especially when I've seen into the dev as much as we have of Bastion and you realize there's tons of other small companies out there like them trying to make a living and people just shit on it for no reason other than it's funny or they're bored, like you pointed out.

Also, Bastion is available on Steam for next to nothing, buy it and support them please.

Posted by lockwoodx

The toy soldiers QL looked shitty, the TNT was boring with people begging you to play more Age of Empires, and I didn't agree with the review. I think Metacritic was right on the money and I trust them most of the time.

Posted by manwithnoname

that sucks

Edited by Brackynews

Imagine if Jim Carrey's or George Clooney's or Spielberg's paycheck was affected by Metacritic, or Rotten Tomatoes, or way back to Siskel & Ebert?

How quickly would that break Hollywood? Clearly there is a correlation where more sales means more residuals, but once the movie is done, you get the percentage in your contract, or SAG gets on the studio's ass. And then you're on to the next job.

Remember these business fuckers with the metacritic carrot on a money stick, are the same fuckers with the "fast track to production" carrot for QA testers. And we know how that goes.

Posted by prestonhedges

@Dark_Lord_Spam said:

Fuck. FUCK. This is genuinely disheartening. I can't fathom intentionally attempting to obliterate any person's livelihood, but when they're trying their damnedest to bring innovation, ingenuity, and interest to an increasingly stale market? Really? Are these scumbag spammers trying to express disdain for indie projects? Are they truly just fucking bored? I wish I could go buy these games immediately to show some meager support (I would've already bought Bastion), but I unfortunately don't own a 360. Fuck this whole mess of a situation, man.

I doubt Supergiant or anyone else is going to go bankrupt just because someone on a website clicked a button. This is just PR wankery at its finest.

Edited by vinsanityv22

People are f***ing stupids. I don't get why forums and boards have moderators, but Metacritic (or anyone else) user reviews do not. Although maybe Gamefly does actually moderate them; they seem to have great user reviews. Even the 1's (out of 10) usually have some text, even just a line or two, about why that person hated the game. With awful grammar, naturally. But not writing anything? You're just being a di*k and loser.

Metacritic needs to get on top of this sh*t. And it would be great if we could just beat up people with mental problems like this; anyone who does sh*t like this randomly is a bane on society. On the internet AND in real-life.

Posted by TPoppaPuff

So how long before Activision starts downvoting their own games at launch to save from paying out bonuses to their developers incentivized bonuses based on Metacritic? Or have they been doing that already?

Posted by Hokucho

I was not aware of this!

Posted by FrEeZe

This is why you can't trust user reviews for video games. There's too much stupid factor.

There are the 0 reviews and then the counter-0 reviews and then the counter-counter-0 reviews.

Posted by Kurtdyoung

So at one point they were going to give away free codes for the game, if people wrote positive reviews for said game? Wouldn't most people writing those reviews have never actually played the game? I guess its the same as the bad reviews but still weird.

Posted by shinyidol

Hi Patrick. It is Collin with two l's

Your friend,

Collin

Posted by afrofools

What's the point of giving people free codes to a game they already played.... unless<end>

Posted by bransonhuggins

Look, if you are going to a site, and basing your buying off of what score something got, then you have other problems that aren't going to be fixed here.

As someone else put, if you look at most sites, especially now about say MW3 or BF3 you will see this, "MW3 sucks ass, BF3 is better" or "BF3 sucks. MW3 is going to be amazing". Well, really people, all you do is have pissing contest.

Just because you don't like a game, doesn't mean it sucks. I am not a huge fan of GoW, but i will tell you right now, it's an amazing game. It's just not a game that is for me. does that take away from it being good at all? NO. Why? Because it wasn't made for ME. It was made for a lot of people, and I just happen to not like it much. But I have enough of a brain to know that just because I don't like something, it doesn't mean it sucks. It just means, I didn't like it.

On the filp side of that, there have been plenty of games that people have said suck, but that I have to say I have enjoyed. Just because someone comes on and gives a bad review, doesn't mean that you shouldn't play it. Make up your own mind, and form your own opinion.

As far as Amazon goes, listen people, if you go onto Amazon, and you give something a 1 star review, and your gripe is with the shipper or anything BESIDES THE ACTUAL PRODUCTS, then shut the fuck up. You are here to review a product, so review it. No one cares if it didn't get there in time to meet your standards. that in no way represents the product, that represents the shipper.

Posted by thebigJ_A

@bransonhuggins said:

...snip...

As far as Amazon goes, listen people, if you go onto Amazon, and you give something a 1 star review, and your gripe is with the shipper or anything BESIDES THE ACTUAL PRODUCTS, then shut the fuck up. You are here to review a product, so review it. No one cares if it didn't get there in time to meet your standards. that in no way represents the product, that represents the shipper.

Amen. I hate that. I'm an avid reader who's recently fallen in love with my new Kindle, so I'm reading user reviews on Amazon a lot lately. It seems like every fourth review is some twit who's pissed his book came a day late or some bull. You find the same thing on Newegg. Some nitwit fried his new motherboard or some other asinine thing and puts up a zero review.

The one good thing on Newegg, is I've actually often seen the manufacturers, or the Newegg people, replying to these reviews and finding out wtf is really up. I wish Amazon had that. Wouldn't be a solution for Metacritic, I don't think. It's just too big.

Posted by prestonhedges

@shinyidol said:

Hi Patrick. It is Collin with two l's

Your friend,

Collin

http://blog.eu.playstation.com/author/cmoore/

Shit, he's right. Even the typos themselves are pointing it out when it happens.

Posted by k4el

@TadThuggish said:

The way to fix Metacritic is to never use Metacritic.

QFT.

The best solution for that site is for everyone to ignore it. It's deeply flawed and overly influential.

Posted by BrontoThunder

Why can't people do this to games that deserve it (more)?

Posted by mlarrabee

Ryan & Vinny: "Now it's the real internet." ... "Hackers. Spammers. Dick-pill ads. Walgreens is having a special."

I think Steam should aggregate review scores from a different system.

I thought the internet jackasses had reached apogee, but I guess not.

Posted by markusespar

Even with professional reviews, the mean score (based on an aggregation of dissimilar data points) is a fundamentally flawed metric. I'd love to see someone create a Metacritic competitor that allowed users to custom-aggregate scores based on sources they trust, whether those sources be professional games journalists or other users.