Log in or sign up to comment
139 CommentsRefresh
Avatar image for patrickklepek
Posted By patrickklepek
When things go sideways on Metacritic, developers don't have many options for appeal.
When things go sideways on Metacritic, developers don't have many options for appeal.

Supergiant Games and Signal Studios were in awkward positions earlier this month, after the average user review score for each game plummeted overnight and seemingly without justification.

It was unclear who was responsible for the sudden change, and there are limited tools for those affected that don't come across as...weird.

The developers of Bastion and Toy Soldiers: Cold War asked fans to submit reviews to Metacritic, which resulted in a slight bump for each but an average score ultimately much lower than it was previously.

After speaking to Supergiant and Signal, I ran a story earlier this week about the challenges the studios faced in addressing these issues.

Metacritic has since identified the users responsible, both banning the users themselves and removing their scores putting things out of balance.

The users were targeting certain games, and hitting them with zero scores without a written review, raising eyebrows.

"A publisher of a different game let us know that he noticed that his game had been 'bombed' by a bunch of zero ratings in an unusually short period of time," said Metacritic co-founder Marc Doyle in an email to me today. "While investigating those ratings, we noticed a group of user accounts and activities that were clearly illegitimate and violated our terms of use."

Doyle's team found a group of users had been coordinating similar attacks on other games within Metacritic's database, allowing Metacritic to fix the "bombings" that had been targeted at other games.

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher. I've never heard of a publisher leveraging user reviews as a metric for payment because of situations like this, but especially for small studios, perception is king, which is why user reviews remain very important.

Signal Studios president and creative director Douglas Robert Albright III had proposed a fix to the problem.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," he said. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Avatar image for wemibelec90
Posted By wemibelec90

Glad to hear someone is cracking down on these jerks.
Avatar image for zackhoagie
Edited By ZackHoagie

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

Avatar image for elcalavera
Posted By elcalavera

Very nice.

Avatar image for bestusernameever
Posted By BestUsernameEver

@elcalavera said:

Very nice.

Is it?

Avatar image for bestusernameever
Posted By BestUsernameEver

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

Yeah, metacritic is the target.

Avatar image for prestonhedges
Posted By prestonhedges

They should just ban all user reviews below 100%. Problem solved.

Avatar image for tesla
Posted By Tesla

Well, good on Metacritic for actually doing something about this.

Avatar image for mayorfeedback
Posted By MayorFeedback

Busted.
Avatar image for cale
Posted By CaLe

Why would they have targeted Bastion? Who did it ever hurt?

Avatar image for largo6661
Posted By Largo6661

see this is why metacritic is bullshit. its beyond me why anyone puts any kind of value on whats posted on that abomination of a site. hope it dies in a fire!

Avatar image for giantstalker
Posted By Giantstalker

You'd think this would happen every day, considering the barrier for entry is so minimal.

Avatar image for somejerk
Posted By SomeJerk

Dear Industry,

Stop taking user reviews seriously, and start taking reviews seriously. If they all say the game is shit but it sells like nude shots of the latest idol, fix it. If they all think the game is great but it sells only a few copies, learn to market properly. Holy shit.

Avatar image for dystopiax
Posted By DystopiaX

@Largo6661 said:

see this is why metacritic is bullshit. its beyond me why anyone puts any kind of value on whats posted on that abomination of a site. hope it dies in a fire!

Most devs only look at staff scores, not user ones, so that doesn't matter. It would be like saying Giantbomb should get the axe cause you disagree with the user reviews on the site.

Avatar image for laserbolts
Posted By laserbolts

Fuck metacritic.

Avatar image for sinisterinfant
Posted By SinisterInfant

If I were to put on my conspiracy theorists hat, It could be publishers themselves posting bad reviews to drive down the metacritic score. Both of those games are now several weeks past their initial release window. From a publisher view point the majority of sales have likely been made. If they owe money to the developer based on a score, they could drive the score down pay the developer for the lower score then delete their comments and restore the score. Seems like an easy way to anonymously save some funds.

Avatar image for falling_fast
Posted By falling_fast

while they're at it they can ban all the critics who are giving Gears of War 3 less than perfect scores *smirk*

Avatar image for mcpaul15
Posted By mcpaul15

I definitely approve. Metacritic user reviews are some of the biggest shams on the internet

Avatar image for dystopiax
Posted By DystopiaX

@SinisterInfant said:

If I were to put on my conspiracy theorists hat, It could be publishers themselves posting bad reviews to drive down the metacritic score. Both of those games are now several weeks past their initial release window. From a publisher view point the majority of sales have likely been made. If they owe money to the developer based on a score, they could drive the score down pay the developer for the lower score then delete their comments and restore the score. Seems like an easy way to anonymously save some funds.

Would be dumb, plenty of people still buy games a couple weeks after they're out, and payouts are based more on critic than user reviews.

Avatar image for baconbot
Posted By Baconbot

@ZackHoagie: Stupidity probably.

Good news to hear and good on Metacritic for cracking down.

Avatar image for autobzooty
Posted By autobzooty

metacritic slowly realizing that the service it provides makes no sense.

Avatar image for cynic04
Posted By Cynic04

WTF, why would they go after the little developers? Not a good way to earn sympathy.

Avatar image for slaker117
Posted By Slaker117
@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

Well, both games were Xbox Live titles that aren't on PSN. I'd like to believe platform fanboyism is dead, but that seems like a possible motive in this case.
Avatar image for vr00m
Posted By vr00m

Why would they count outliers anyway, especially ones with no written review.

Avatar image for honk
Posted By Honk

It's absolutely awesome that someone is cracking down on these shenanigans. Unacceptable behavior like this can't be allowed.

Avatar image for zackhoagie
Posted By ZackHoagie

@Baconbot: The weird thing is though is that they specifically targeted two Summer of Arcade games instead of say, Resistance 3 or Hard Reset or Gears 3 or what have you. As shitty and console-warsish as it sounds, I have a pretty good feeling this might of been Sony fans.

Avatar image for pxabstraction
Posted By PXAbstraction

For once, Metacritic does something useful.

Avatar image for zackhoagie
Posted By ZackHoagie

@Slaker117: Dude, look at these comments at a story about Sony's PSN woes or what have you and you'll realize console warriors are still fighting the stupid, stupid, stupid fucking fight.

Avatar image for fobwashed
Posted By Fobwashed

I use metacritic and before them, gamerankings not to get an aggregate review but to get a list of reviews from sites in one place with blurbs. I never paid much mind to the user reviews though. I DO check and read user reviews for purchases I make on amazon, newegg and I look at the aggregate for movies on RottenTomatoes.

Good on them for actually doing something when they could just have easily have not cared. I wonder what their barrier to entry is for the critic review section.

Avatar image for delta_ass
Posted By Delta_Ass

@SinisterInfant said:

If I were to put on my conspiracy theorists hat, It could be publishers themselves posting bad reviews to drive down the metacritic score. Both of those games are now several weeks past their initial release window. From a publisher view point the majority of sales have likely been made. If they owe money to the developer based on a score, they could drive the score down pay the developer for the lower score then delete their comments and restore the score. Seems like an easy way to anonymously save some funds.

Publishers don't give bonuses based on user reviews. It's based on the aggregate review scores from professional publications. What you suggest is insane.

Avatar image for jimbo
Posted By Jimbo

Cool, they should worry more about the 'Everything with an ad budget is amazinggg! 10/10' bullshit coming out of the reviewers. Also about their own incompetence when it comes to converting review scores, 5* scales converting 10% too high etc.

Avatar image for john1912
Posted By John1912

Yea WTF? I can see people being random dicks and give 0s. But a group of people actively working to do so? Why?

Avatar image for troncek
Edited By Troncek

Meta who?

I almost never use that site. If I'm interested in a game, I check out videos, interviews, articles and other stuff, so by the time the game is actually released, I have gathered more than enough information to make a decision.

I also don't read other people's reviews (including here on GB), because they usually don't tell me anything new or useful.

It's sad to see industry insiders rely so much and put weight on this stuff.

Avatar image for lightsoda
Posted By lightsoda

Good on them.

Avatar image for baconbot
Posted By Baconbot

@ZackHoagie: Probably true.

Avatar image for ssully
Posted By SSully

I wonder if they banned me. I made an account and scored all the games as 10's who got hit by these guys.

Avatar image for jackg100
Posted By JackG100

What kind of tards think that internetpolls give a fair picture of reality anyway?

Avatar image for vexxan
Posted By Vexxan

Great to see something was done about it.

Avatar image for eshchan
Posted By eshchan

@patrickklepek Patrick, are you sure developers of Bastion also asked its fans to write positive review on Metacritic? I know PR for Signal Studios did it but I thought SuperGiantGames did not.

Avatar image for donutfever
Posted By DonutFever

I think it's weird that small teams were being targeted.

Avatar image for ninjacommando
Posted By NinjaCommando

@ZackHoagie: trolls dont have a target they just want any type of response...

Avatar image for corvak
Posted By Corvak

User reviews make no sense. At least, it makes no sense for them to be given equal stature as a professional review. Professional reviewers typically get their jobs through good writing practices, and for being a fair judge of a game, though it is, of course, still the reviewer's opinion of the game. This is no different than major online storefronts like Amazon - who hire people to track down and remove manufacturers or publishers leaving fake reviews praising their products, or attacking a competing brand.

Because reviews are opinion pieces, just reading one site isn't the answer either, which is why Metacritic exists. Because so many of us, both gamers and industry people wrongly feel that the score is the be-all end-all of every review, Metacritic's aggregate score becomes the most accurate way to gauge the reception of a game. Sadly, we cannot break down video games into objective hard numbers to compare them - it just doesn't work - the "fun" of a video game isn't something that can be measured by a machine.

Personally, I like that they line up all the professional reviews for me to read, instead of having to dig through pages of ads and clutter. Giant Bomb is probably the best laid out gaming news site i've found - The others tend to be crammed with as much content as humanly possible onto the front page, or set up like a blog, where everything older than six hours has been pushed off the bottom.

If we start ignoring user scores, Metacritic or sites like it can be a way for quality releases like Bastion to get noticed without spending millions on mass marketing. If we continue to be distracted by numbers, it'll just be another battle in the pointless console war, and that's a war the trolls and moronic fanboys always win.

Avatar image for stormharbinger
Posted By StormHarbinger

Great job Patrick, it's hard not to see your article and exposure as having an impact on Metacritic's actions. I'd love to think they would correct the problem on their own, but...

Anyway, kudos to you for making an impact time and again.

Avatar image for phrosnite
Edited By phrosnite

That's why I've always said that User reviews or overall users' score meant nothing.

Avatar image for rapid
Edited By Rapid

Why, Why hurt developers??? If they were trying to make a statement couldn't they at least have actively rated 100?

Avatar image for klei
Posted By Klei

I've never cared for User Reviews before anyways and I will never. Most guys who writes scores played the game for 30 minutes. heck, that is, if they even played it. They usually overhype it or bitch it to death. It's pretty rare to find level-headed reviews.
Avatar image for mordukai
Edited By mordukai

Too little, too late. This feels like a "sweeping it under the rug" type of solution to this situation. The damage is already done snd there has to be a long term solution that will prevent this from happening in the first place. If it wasn't for those developers and this article pointing out the problem then Metacritic would have done nothing at all. They simply don't care. Metacritic needs a major overhaul and they reviews are taken. As an idea Metacritic is great but it was executed poorly. Metacritic must realise that it's playing with the livelihood of people. I hope this case won't get swept under the rug because this issue must be addressed.

Avatar image for largo6661
Posted By Largo6661

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

Avatar image for boiglenoight
Posted By Boiglenoight

Good for Metacritic.

Avatar image for heatdrive88
Posted By heatDrive88

Who's to say that the people who are running these illegitimate accounts purposely dropping the Metacritic score aren't the same people who will be fiscally responsible for handing out bonuses to the developers, as a method of cutting the cost of that paying out that bonus to developers?