Giant Bomb News

231 Comments

Power to the People, Until the Power's Out of Control

A negative review is one thing, but 20 of them? In an hour? Out of nowhere? The problems developers face when Metacritic goes awry.

Signal Games is not the only developer to experience a rash of strange, out-of-nowhere negative reviews on Metacritic.

Signal Studios had a problem last week.

The tiny developer of Toy Soldiers: Cold War discovered its game had been hit with a series of negative, score-only (meaning no text) user reviews on Metacritic overnight, dragging its overall user rating down. While the scores assigned by critics are important, so are user reviews--anyone visting Metacritic is free to sort games by user reviews, too.

Metacritic has been a lightning rod of criticism over the years, due to its ties to developer pay.

"There is a rash of fake negative user scores going about Metacritic and it hit TS:CW (and other games)," said Signal Studios community manager Logan DeMelt last week. "This means we need to hear your voice!"

DeMelt started mobilizing fans to add more reviews to the pile, incentivizing them with free download codes for Toy Soldiers: Cold War.

"If you write a user review," he said, "just being honest (we aren't bribing for positive) we will do the following: You write a review & post it, and we will put your name in for a chance for a prize on Friday. Every 10 new posts, we will drop a code out."

Of course, there are problems with this approach, especially in terms of the perception of a developer asking people to write reviews, albeit not advocating positive or negative, by dangling prizes around. DeMelt even got some Twitter flack about the concept from a friend, Sucker Punch community manager Colin Moore.

When I talked to Signal Studios president and creative director Douglas Robert Albright III on the phone last week, he told me he'd asked DeMelt to stop the promotion, understanding how people might interpret it.

"It doesn't matter what your intentions are, it's what the perception is," said Albright. "If that's the perception, then we'll just stop doing it. Because, honestly, he [DeMelt] had no intention of bribing people to get good scores."

"It was actually my fault," he continued. "I looked at the Metacritic and contacted them and told them 'This has been spammed.' We're not a bunch of dudes with a bunch of money laying around or whatever. It affects us, right? You can search by user scores and stuff like that. It's clearly spam. Metacritic just responded with this generic thing. All the intent was 'Well, this isn't our fault, we're just going to go on Metacritic and [ask users to] review the game.'"

Signal Studios noticed a spike in negative reviews overnight, without review text to accompany it.

Like it or not, Metacritic has become important to the games industry. It's a system with faults, and one that's come under enormous criticism over the years. But game publishers have few ways to determine success outside of sales--so they turn to Metacritic. Metacritic determines bonuses and royalty payouts for many. Thus, developers have reason to pay close attention.

The situation prompts hard questions for developers, especially small ones with financial destinies tied to something partially out of its control. There are few options. Asking for reviews could be perceived wrong, but what do you do if the negative reviews seem fake? How do you prove that? If you can't prove it but know you're right, do you gamble the possible backlash?

The motivations behind the user or users spamming Metatritic with negative reviews are unknown. Blind Internet rage? Sheer boredom? A new form of spam?

"I don't imagine there is some conspiracy," said Albright. "I think some folks just do annoying crap because they can. Like the kids who hack the leaderboards or idiots who deploy viruses. Ever play mailbox baseball?"

...no comment.

In any case, Albright hasn't received any evidence worth acting on--and neither has another studio.

The only reason Signal Studios even realized something was goofy on its Metacritic page was thanks to Supergiant Games noticing a similar explosion of negative user reviews overnight for Bastion in early September.

Signal Studios was tipped off to the Metacritic issue by Supergiant Games.

"I think it was sitting at around a mid-8 on Xbox 360 and at a 9 on PC, but on September 2 it had dropped into the 6s," said Supergiant Games creative director Greg Kasavin to me over email. "No additional user reviews were posted on that day (or at least no new negative reviews were posted), but I noticed that we had 20 new 'negative' user ratings that were entered for both versions of the game. This struck me and the rest of us as highly suspicious, because we were gathering new user ratings at a much slower rate than that in previous days. The idea that all of a sudden we would get 40 extremely disappointed people come and give us a 0 out of 10 rating all at the same time seemed very dubious."

Supergiant Games mentioned the bizarre nature of the user reviews over its Twitter and Facebook accounts and left it at that.

"It's the first time we've ever complained about something via our Facebook and Twitter so I didn't do it lightly," said Kasavin.

Bastion's user review score went up as a result--7.7 on Xbox 360, 8.0 on PC--but the damage was done. Kasavin flagged the issue with Metacritic but was told little could be done. Metacritic said it's very careful with user reviews, especially so if the user doesn't actually write a review. On Metacritic, it's possible to submit a "review" with only a score, making it difficult to determine whether someone simply registered an account and their negative review was their first submission--or spam.

Supergiant mentioned the Metacritic issue and users responded with positive reviews.

"We just had to let it go," he said. "We value Metacritic as a service and are the highest rated XBLA [Xbox Live Arcade] game released so far this year according to them. There are some really great user reviews of our game on there, and if ratings-bombing continues to be an issue for other products, I trust it's something that the team there will investigate a solution more closely."

I contacted Metacritic about this pattern of issues. Metacritic told me each review has a "report abuse" option, which sends the review to Metacritic's team to possibly delete the review or ban the user.

"If any interested party feels that there has been a group of illegitimate user ratings (score only) entered for its game, they can contact me through the website and we'll investigate the issue," explained co-founder Marc Doyle. "We track each rating and can delete any that appear to be illegitimate or suspicious to us after a staff review of the rating data in question."

For now, for better or worse, that's the way the system works.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," said Albright. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+
231 Comments
Posted by DarthOrange

Sounds like people mad at the internet because there penis growing pills didn't work.

Posted by doctorworm

According to Metacritic, Dragon Age 2 has a score of 79. I don't think I trust Metacritic anymore..

Posted by yami4ct

@bransonhuggins said:

Look, if you are going to a site, and basing your buying off of what score something got, then you have other problems that aren't going to be fixed here.

As someone else put, if you look at most sites, especially now about say MW3 or BF3 you will see this, "MW3 sucks ass, BF3 is better" or "BF3 sucks. MW3 is going to be amazing". Well, really people, all you do is have pissing contest.

Just because you don't like a game, doesn't mean it sucks. I am not a huge fan of GoW, but i will tell you right now, it's an amazing game. It's just not a game that is for me. does that take away from it being good at all? NO. Why? Because it wasn't made for ME. It was made for a lot of people, and I just happen to not like it much. But I have enough of a brain to know that just because I don't like something, it doesn't mean it sucks. It just means, I didn't like it.

On the filp side of that, there have been plenty of games that people have said suck, but that I have to say I have enjoyed. Just because someone comes on and gives a bad review, doesn't mean that you shouldn't play it. Make up your own mind, and form your own opinion.

As far as Amazon goes, listen people, if you go onto Amazon, and you give something a 1 star review, and your gripe is with the shipper or anything BESIDES THE ACTUAL PRODUCTS, then shut the fuck up. You are here to review a product, so review it. No one cares if it didn't get there in time to meet your standards. that in no way represents the product, that represents the shipper.

My thoughts exactly. CoD isn't my cup of tea, yet I'm willing to admit it's a very well made game. Conversely, you (in the general sense not talking to anyone in particular) may not like FFXIII as much as I did, but that doesn't make it garbage. Metacritic does have its uses, though. If I'm looking at a game I might want to rent or buy, I will typically run over to MC. I than look at the excerpts from different sources with different scores, and click through some positive and some negative to get a balanced perspective. For me, it's more about aggregating the reviews all in one place rather than me have to go through sites like IGN, which is a terribly designed site, 1UP etc. individually to find their reviews. It's just easier that way.

Posted by SpaceInsomniac

Anyone who pays attention to USER reviews on metacritic is an idiot. They're completely worthless, and should be done away with. That won't happen, though, as they mean idiots reading reviews from other idiots, and having a pissing contest about them that results in a shitload of page views.

I'm sure the owners of Metacritic really couldn't care less about those 0 scores, otherwise they'd take action against that sort of crap themselves in the first place. I can't think of ANY game this console generation that deserves a 0 or 1 out of 10.

Posted by P_Pigly_Hogswine

@markusespar said:

Even with professional reviews, the mean score (based on an aggregation of dissimilar data points) is a fundamentally flawed metric. I'd love to see someone create a Metacritic competitor that allowed users to custom-aggregate scores based on sources they trust, whether those sources be professional games journalists or other users.

It's an interesting point. It's been an issue with reviews of all sorts since long before the internet. For every independent, highly-trusted critic there's a swathe of critics sitting in official or distributor circles who dispense high scores like they're going out of fashion. Truth is you trust certain reviewers more so than even certain sites or publications. It all depends on your personal taste and also the natural ebb and flow of critic's credibility. When in doubt, play the demo, I suppose. ;)

Posted by Kazona

This proves how utterly useless metacritic is.

Posted by Chubbaluphigous

Meta Critic's user scores have always been garbage. They have been the target of fanboy rage for a long time now. If a game is a big exclusive, then it is guaranteed to get voted down. Also, there are a lot of assholes out there, who just want to vote something down. I have never trusted user ratings when there are no written reviews attached. They have no accountability, and no way to verify that they are a credible source for reviewing a game.

I only ever use Meta Critic as a source for links to the different reviews. I have my list of reviewers that I tend to agree with, and Meta Critic a is a good way to get links to all of them at the same time. Putting any stock what so ever in the user reviews is foolish on the both the developer and end user sides.

Posted by Teran

Well... one relatively simple solution. Don't let anyone cast a vote without posting a review.

I know this won't magically fix things over night, the system could still be abused however there are a lot fewer people out there willing to post a review (of any quality) than there are people who will vote a score with two mouse clicks... All they'd need at that point is a netflix style button you can push letting you report a review if it's not really a review. Even fewer harassers would bother taking the time to write a quality review.

Posted by AaronChance

The best solution is to ignore ALL reviews, and just buy what looks cool. Lots of games have demos, others have plenty of footage online. You don't need someone's opinion guiding by their own personal taste, which you may or may not agree with.

Posted by Balex1908

Unfortunately Metacritic isnt going away, so they have to fix this.

And this problem has a really easy solution, dont let people rate a game without writing their reasons behind the rating. I assume Metacritic wont do anything about this until it happens to something big like CoD or BF.

Posted by MoseSSesoM

Never liked Metacritic, always seemed to easy a system to abuse for the good or bad.

Posted by mewarmo990

People go to Metacritic for the user reviews? There's a reason it's called Metacritic and not Metafanboy...

Posted by halim51

Metacritic should change it so the user reviews have no effect on the score.

Posted by selbie

What the fuck is Metacritic doing by tallying user reviews!?!?!?? Sure people like to have their say about a game, but it should not be mashed in with the media's own PROFESSIONAL critiques. If they separated the two, then this whole issue wouldn't even exist!

It's like if Giantbomb combined its scores with player reviews. It's stupid and unprofessional, which says a lot about Metacritic.

@markusespar said:

Even with professional reviews, the mean score (based on an aggregation of dissimilar data points) is a fundamentally flawed metric. I'd love to see someone create a Metacritic competitor that allowed users to custom-aggregate scores based on sources they trust, whether those sources be professional games journalists or other users.

YES. THIS. And it wouldn't be hard to do either. You could simply add games to your tracking list and, when the reviews come out, the tracked game's aggregate score will represent the opinions you trust.

Posted by Lozz

I love your articles!

Posted by knightlyknave

While the scores assigned by critics are important, so are user reviews

Here's where you're wrong. While reviewers can sometimes be mistaken, users are on the whole idiots. Users either post because they loved it or hated it. No one who plays a game that's average will go to the effort to post a review, but critics have to. Why would I want to listen to a bunch of trolls/fan boys? At least it's possible a professional critic might have an unbiased review even if this is not always the case.

Posted by mnzy

3rd Strike Online Edition got 0's from reviews saying "THEY SHOULD HAVE MADE MEGA MAN LEGENDS. CAPCOM SUCKS" and so on.
But GIantbomb also has user reviews like that.
 
I think the only chance to do something about it is, that people have to care about their account. Use Facebook-Login for example.

Posted by MaddProdigy

@Cretaceous_Bob: Wow you really don't understand what he meant? How thick are you? I understand oversite is a far cry from oversight, but geez. Good thing you pointed that out, otherwise more people would have been as confused as you are!

Posted by brownsfantb

I have an idea! Let's not care about Metacritic anymore!

Seriously, I don't remember the last time a specific review or group of reviews actually impacted what I bought. I guess I'm the exception to the rule but I go more on word of mouth and what I read and see about the game before it comes out and decide if I want to play it. If a game I like gets bad reviews, I don't really care. I don't like a lot of games that got great reviews too.

Posted by SpaceInsomniac

@selbie said:

What the fuck is Metacritic doing by tallying user reviews!?!?!?? Sure people like to have their say about a game, but it should not be mashed in with the media's own PROFESSIONAL critiques. If they separated the two, then this whole issue wouldn't even exist!

The two already are separated. User scores have absolutely no impact on the metascore, which is only based on critic reviews. There's the metascore, and the users score, and neither of them have anything to do with one another.

Posted by YukoAsho

The problem isn't Metacritic. It's the dipshits that have placed such an absurdly high emphasis on it.

Let Jim Sterling explain.

Unfortunately, we're in a strange period in the growth of gaming where reviews are becoming a sort of bible for many. As a result, Metacritic has unfortunately become a battlefield for fanboys to fight in, and companies starved for any means to gauge reactions in the public use it as a sort of bible.

It's going to take a long time to shake this off... It may never happen, now that the internet is being carved up among various fanboy camps. Hopefully corporations will find an alternative means to gauge user satisfaction before the fanboy war burns a truly classic game.

Edited by Gildermershina

This is the reason I don't really trust people's unsolicited opinions. The user reviews on metacritic, even the ones where they write actual reviews, are mostly valueless and reactionary. It's the same with anything though. To a lot of people, if they don't like, say, a new album by a band, to them it's the worst thing in the world, and they need to tell everybody. The mere fact that anybody else might like it makes them very angry.

Although it's all ultimately subjective, I think good reviewers strive toward some objectivity. Call me elitist, but unless you can write at least a cogent paragraph - hell, even a sentence - about something, then personally I'm not really interested in your opinion.

Edited by Cretaceous_Bob

@MaddProdigy said:

@Cretaceous_Bob: Wow you really don't understand what he meant? How thick are you? I understand oversite is a far cry from oversight, but geez. Good thing you pointed that out, otherwise more people would have been as confused as you are!

Or I was trying to reduce the amount of time a professional writer had a blatant typo in public view. How thick are you?

Posted by Hilts

Its all about trusted sources pal. I dont know em - I dont trust em!

Posted by HarrySound

NEVER LISTEN TO THE USERS UNLESS YOU COMPETELY TRUST THEM.

The internet and gaming is full of too many kids to be taken seriously.

Posted by sungahymn

Man...People can be such jerks.

Posted by BlainN

It looks like Metacritic has taken action.
 
http://www.gamasutra.com/view/news/37425/Metacritic_Takes_Step_To_Remove_Spam_User_Reviews.php
 
I'm sure this won't be the last of it, but the price of credibility is eternal vigilance.

Posted by Nymphonomicon

The problem with empowering people is that a lot of people are dicks and will abuse that power. If Metacritic wants honest reviews, they're going to get people giving zeroes with no written content, expecting people to given anymore (while at least some will) is seriously over-estimating our species.

Posted by polyorpheus

@AaronChance: True, but it helps. I like to read a variety of reviews and read through them. I know the biases of the reviewers and who are closest to what I like. This doesn't apply to games I know I will buy like Skyrim or any Valve game, though.

Posted by Katkillad

If you are using metacritic for video game reviews you are doing it wrong.

Posted by NellyK

I can't believe bonuses and royalties can be based on something like Metacritic. The video game industry is truly twisted.