Something went wrong. Try again later

Giant Bomb News

139 Comments

Metacritic Finds, Bans Group of Users Unfairly Scoring Games

Users gaming the system to target Bastion, Toy Soldiers: Cold War and others are gone.

When things go sideways on Metacritic, developers don't have many options for appeal.
When things go sideways on Metacritic, developers don't have many options for appeal.

Supergiant Games and Signal Studios were in awkward positions earlier this month, after the average user review score for each game plummeted overnight and seemingly without justification.

It was unclear who was responsible for the sudden change, and there are limited tools for those affected that don't come across as...weird.

The developers of Bastion and Toy Soldiers: Cold War asked fans to submit reviews to Metacritic, which resulted in a slight bump for each but an average score ultimately much lower than it was previously.

After speaking to Supergiant and Signal, I ran a story earlier this week about the challenges the studios faced in addressing these issues.

Metacritic has since identified the users responsible, both banning the users themselves and removing their scores putting things out of balance.

The users were targeting certain games, and hitting them with zero scores without a written review, raising eyebrows.

"A publisher of a different game let us know that he noticed that his game had been 'bombed' by a bunch of zero ratings in an unusually short period of time," said Metacritic co-founder Marc Doyle in an email to me today. "While investigating those ratings, we noticed a group of user accounts and activities that were clearly illegitimate and violated our terms of use."

Doyle's team found a group of users had been coordinating similar attacks on other games within Metacritic's database, allowing Metacritic to fix the "bombings" that had been targeted at other games.

Metacritic is often used to determine royalty and bonus payouts for developers, though its exact use varies from publisher to publisher. I've never heard of a publisher leveraging user reviews as a metric for payment because of situations like this, but especially for small studios, perception is king, which is why user reviews remain very important.

Signal Studios president and creative director Douglas Robert Albright III had proposed a fix to the problem.

"The way to fix Metacritic user reviews is to simply require a written review and verify user accounts," he said. "If it was just some random blog I'd say whatever. But this is a major news review aggregator that should have better oversight and some standards."

Patrick Klepek on Google+

139 Comments

Avatar image for mike76x
Mike76x

559

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By Mike76x

What about the people that gave all three versions of Dragon Age 2 zero's before it was released, or bitched about DRM that wasn't there?

I'd trust the ratings on Amazon.com, or any retailer before I'd give Metacritic any thought whatsoever.

Avatar image for napalm
napalm

9227

Forum Posts

162

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By napalm

@MrKlorox said:

Holy shit, is this actually Metacritic acknowledging their system isn't perfect and attempting to take steps to right the wrongs!?

Temporarily, at least. It's not as if things won't be awful by this time tomorrow.

Avatar image for stise
stise

222

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By stise

@subyman said:

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating

Fuck you man, SmokeDAWG420 is my ONLY trusted source for video game news and reviews.

Avatar image for enigma777
Enigma777

6285

Forum Posts

696

Wiki Points

0

Followers

Reviews: 0

User Lists: 8

Edited By Enigma777

I bet it was some random forum.

Avatar image for subyman
subyman

729

Forum Posts

2719

Wiki Points

0

Followers

Reviews: 2

User Lists: 0

Edited By subyman

Good that they got it sorted out. I, however, do not use metacritic and have only been to their site a handful of times. I prefer a quicklook and a written review over SmokeDAWG420's numeric rating.

Avatar image for gildermershina
Gildermershina

411

Forum Posts

361

Wiki Points

0

Followers

Reviews: 0

User Lists: 5

Edited By Gildermershina

@mavs said:

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

That's why the five star system is so great. Too much gradation is bad I think, and too much weight is put on getting a number to put in the metacritic cruncher as opposed to actual review text - thankfully that text does at least make it onto metacritic.

Avatar image for sleepydoughnut
SleepyDoughnut

1269

Forum Posts

133

Wiki Points

0

Followers

Reviews: 0

User Lists: 5

Edited By SleepyDoughnut

Yay, a happy ending to this story

Avatar image for sinful
Sinful

218

Forum Posts

48

Wiki Points

0

Followers

Reviews: 0

User Lists: 1

Edited By Sinful

metacritic, a concept that makes no sense and is easily corrupted.

Avatar image for whylessness
whylessness

42

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By whylessness

Bastion is my shit.

Avatar image for hef
Hef

1239

Forum Posts

486

Wiki Points

0

Followers

Reviews: 1

User Lists: 7

Edited By Hef

If metacritic didn't exist neither would double fine. Their first game, psychonauts, sold like shit. But it got really good rreviews from critics. This allowed them to say to EA: Hey, actually market my crazy new awesome metal game and I guarantee you sales. Of course it still didn't sell very well and reviewed slightly worse than psychonauts, but they got a new engine out of it. And that new engine DID lead to 4 awesome downloadable games. I say keep on keepin on metacritic.

Avatar image for aajf
aajf

240

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By aajf

So the entertaining upshot of all this is that Bastion's user score is actually now inflated, since Greg Kasavin called for grateful players to combat the spam with their own positive reviews. I can't say I have a problem with that outcome.

Glad to see Metacritic at least attempting to get things together. I think it can be a great resource if approached with proper caution by the reader.

Avatar image for swoxx
swoxx

3050

Forum Posts

468

Wiki Points

0

Followers

Reviews: 0

User Lists: 4

Edited By swoxx

Why would you put stock into what random people on the internet say anyways? They're all shitheads and dicks.

Except you Giant Bomb community, of course. You're a sweet little safe haven. <>

Avatar image for winsord
winsord

1642

Forum Posts

86

Wiki Points

0

Followers

Reviews: 0

User Lists: 13

Edited By winsord

It's just a matter of time. Though, forcing people to actually write a review instead of just giving it a score without explanation is definitely a step in the right direction. Maybe it's taken a little longer than it should have, but good on Metacritic for at least addressing this issue.

Avatar image for craigbo180
craigbo180

1763

Forum Posts

42988

Wiki Points

0

Followers

Reviews: 2

User Lists: 2

Edited By craigbo180

Metacritic needs to ban Jim Sterling/destructoid reviews if they really wanted to be helpful.

Avatar image for crusader8463
crusader8463

14850

Forum Posts

4290

Wiki Points

0

Followers

Reviews: 7

User Lists: 5

Edited By crusader8463
@mavs said:

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

Buy, rent/borrow/pirate, pass. All you need for a review score.
Avatar image for ryanwho
ryanwho

12011

Forum Posts

-1

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By ryanwho

Now Metacritic has integrity. lol

Avatar image for boiglenoight
Boiglenoight

605

Forum Posts

154

Wiki Points

0

Followers

Reviews: 2

User Lists: 3

Edited By Boiglenoight

And Douglas Albright's comment seems very reasonable.

Avatar image for mavs
mavs

399

Forum Posts

2

Wiki Points

0

Followers

Reviews: 0

User Lists: 2

Edited By mavs

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

This has been true since IGN became popular, so at least since 1999. Anything below 8 has always been a bad score. What Metacritic introduced was a way for publishers to wield metascores as an objective evaluation against developers.

Avatar image for largo6661
largo6661

340

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By largo6661

@DystopiaX: Yeah your right when used like that it is helpful. Its just a shame that the majority of its user base probably doesn't do that.

Avatar image for krelle
Krelle

180

Forum Posts

130

Wiki Points

0

Followers

Reviews: 0

User Lists: 3

Edited By Krelle

@bkbroiler said:

Woah, Metacritic is used to determine royalties and bonuses? I find that incredibly shocking.

Apart from sales, Metacritic is the only way for 'a suit' to judge the success of a game/studio.

Well, that or flipping a coin..

Avatar image for dystopiax
DystopiaX

5776

Forum Posts

416

Wiki Points

0

Followers

Reviews: 1

User Lists: 0

Edited By DystopiaX

@Largo6661 said:

@DystopiaX: Thats is true. But i do feel that metacritic has made this issue more pronounced. The words in a review matter less and less these days. most poeple i now never even read reviews they just go to metacritic and look at a number. Its not the user reviews i have a problem with its how a dozen or more reviews can be reduced to a simple digit seems really unfair to me. And personally the idea of having one all incompasing site to tell me whether a game is good or not makes me uneasy, due to the fact that its so easy to effect a games score on metacritic.

I think that you have a problem more with how people use metacritic than metacritic itself. If used properly it's actually a great tool, for example if I want to read alot of reviews I'll go to the metacritic review and read the summaries of scores, and click on the ones and read the fulll reviews of the ones that interest me... I think that no one should just look at the score a game got but that problem doesn't stem from metacritic.

Avatar image for bkbroiler
bkbroiler

1739

Forum Posts

438

Wiki Points

0

Followers

Reviews: 0

User Lists: 11

Edited By bkbroiler

Woah, Metacritic is used to determine royalties and bonuses? I find that incredibly shocking.

Avatar image for largo6661
largo6661

340

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By largo6661

@DystopiaX: Thats is true. But i do feel that metacritic has made this issue more pronounced. The words in a review matter less and less these days. most poeple i now never even read reviews they just go to metacritic and look at a number. Its not the user reviews i have a problem with its how a dozen or more reviews can be reduced to a simple digit seems really unfair to me. And personally the idea of having one all incompasing site to tell me whether a game is good or not makes me uneasy, due to the fact that its so easy to effect a games score on metacritic.

Avatar image for alex_carrillo
Alex_Carrillo

321

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By Alex_Carrillo

Metacritic is an utter shitshow.

@DystopiaX said:

@SinisterInfant said:

If I were to put on my conspiracy theorists hat, It could be publishers themselves posting bad reviews to drive down the metacritic score. Both of those games are now several weeks past their initial release window. From a publisher view point the majority of sales have likely been made. If they owe money to the developer based on a score, they could drive the score down pay the developer for the lower score then delete their comments and restore the score. Seems like an easy way to anonymously save some funds.

Would be dumb, plenty of people still buy games a couple weeks after they're out, and payouts are based more on critic than user reviews.

Wouldn't that affect future sequels should they happen? A publisher wouldn't burn potential dollars for a few bonus bucks to the developers.

Avatar image for mormonwarrior
MormonWarrior

2945

Forum Posts

577

Wiki Points

0

Followers

Reviews: 15

User Lists: 21

Edited By MormonWarrior

I like Metacritic, because it gives a good idea of what games are high-quality and which are simply okay, and I don't have all the time in the world to play every game to find out for myself if it's good.

However, I've always known that the user reviews on there are so completely trash they are totally useless. Where you see Halo Reach, a quality shooter, getting either 10 "Best game EVAR" or 0 "So overrated it's terrible HALO SUX" reviews. The normal reviews have some semblance of reason and rhyme behind them. I especially look for the ratings from sites I trust.

Avatar image for 234rqsd2323d2
234r2we232

3175

Forum Posts

2007

Wiki Points

0

Followers

Reviews: 0

User Lists: 16

Edited By 234r2we232

Great. Now they should ban the people raising scores of games for their friends.

Avatar image for deactivated-61abb009b221e
deactivated-61abb009b221e

398

Forum Posts

522

Wiki Points

0

Followers

Reviews: 2

User Lists: 5

If Metacritic makes so much attention to developers, then it should be held to a higher standard.

Avatar image for splodge
splodge

3309

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By splodge

@tourgen said:

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

well if scores determine royalty payouts and bonuses, a publisher could hire some people to bomb scores as a cost-saving measure.

That's a bit of a stretch. I doubt any publisher would risk the kind of horrible PR and fallout that would occur if they got caught doing that.

Avatar image for tourgen
tourgen

4568

Forum Posts

645

Wiki Points

0

Followers

Reviews: 4

User Lists: 11

Edited By tourgen

@ZackHoagie said:

So... why exactly were they doing this? I mean, trolling is pretty unreasonable by design, but usually trolls have some sort of selection process for their targets.

well if scores determine royalty payouts and bonuses, a publisher could hire some people to bomb scores as a cost-saving measure.

Avatar image for krakn3dfx
Krakn3Dfx

2746

Forum Posts

101

Wiki Points

0

Followers

Reviews: 4

User Lists: 3

Edited By Krakn3Dfx

This seems like a slippery slope to me when it comes to choosing who can and can't post user reviews for games. I'm not much for Metacritic anyway, but to have them picking and choosing which user reviews are valid and which aren't makes them even less reliable than they were before.

Avatar image for winternet
Winternet

8454

Forum Posts

2255

Wiki Points

0

Followers

Reviews: 0

User Lists: 6

Edited By Winternet

See, that wasn't so difficult. Too bad a couple of devs and websites had to bitch about it for them to actually do what they are supposed to do.

Avatar image for ravenlight
Ravenlight

8057

Forum Posts

12306

Wiki Points

0

Followers

Reviews: 0

User Lists: 2

Edited By Ravenlight

Nice job snapping up this scoop, Patrick. Glad to see Metacritic took action in this situation.

Avatar image for elsolar
Elsolar

16

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By Elsolar

metacritic is cancer

Avatar image for lockwoodx
lockwoodx

2531

Forum Posts

6

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By lockwoodx

Metacritic caves in I see.

Avatar image for sammo21
sammo21

6040

Forum Posts

2237

Wiki Points

0

Followers

Reviews: 18

User Lists: 45

Edited By sammo21

Metacritic user reviews are complete trash.

Avatar image for dystopiax
DystopiaX

5776

Forum Posts

416

Wiki Points

0

Followers

Reviews: 1

User Lists: 0

Edited By DystopiaX

@Largo6661 said:

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

That has nothing to do with review scores, and that sentiment was around before metacritic was a thing.

Avatar image for joey_ravn
JoeyRavn

5290

Forum Posts

792

Wiki Points

0

Followers

Reviews: 2

User Lists: 3

Edited By JoeyRavn

I don't like where this is heading. Then again, I don't use Metacritic, nor like Bastion. So no problem for me, for the time being.

Avatar image for youngfrey
YoungFrey

1363

Forum Posts

10811

Wiki Points

0

Followers

Reviews: 0

User Lists: 4

Edited By YoungFrey

I think an article like this really suffers from not linking to the outside material in question.  You spend 11 paragraphs talking about metacritic and it gets no links?  It might be common practice, but it's lame and anti-reader.  Not even ones to the games in question?  I hate it when other sites do that, and I hate it here.
Avatar image for mrklorox
MrKlorox

11220

Forum Posts

1071

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By MrKlorox

Holy shit, is this actually Metacritic acknowledging their system isn't perfect and attempting to take steps to right the wrongs!?

Avatar image for rjaylee
rjaylee

3804

Forum Posts

529

Wiki Points

0

Followers

Reviews: 3

User Lists: 2

Edited By rjaylee

Who's to say that the people who are running these illegitimate accounts purposely dropping the Metacritic score aren't the same people who will be fiscally responsible for handing out bonuses to the developers, as a method of cutting the cost of that paying out that bonus to developers?

Avatar image for boiglenoight
Boiglenoight

605

Forum Posts

154

Wiki Points

0

Followers

Reviews: 2

User Lists: 3

Edited By Boiglenoight

Good for Metacritic.

Avatar image for largo6661
largo6661

340

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By largo6661

@DystopiaX: i have no problem with review scores or peoples opinions on a game, What i have a problem with is what metacritic has done to gaming culture. I mean 5 years ago 7 was a good score now a 7 is a terrible score, and devs say things like "hope this scores at least an 8 on metacritic or we are done". I hate the way a games success often lives and dies on its metacritic score. At the end of the day its just an arbitrary number. My main problem with the site is how it down plays the importance of the words writen in the review and pretty much focuses in on a number to convey to a person whether a game is good or not, and then whats next? metacritis scores for reviewers or maybe specific devs? its just a dumb site. its the review itself thats important not the number added at the end.

Avatar image for mordukai
mordukai

8516

Forum Posts

398

Wiki Points

0

Followers

Reviews: 0

User Lists: 1

Edited By mordukai

Too little, too late. This feels like a "sweeping it under the rug" type of solution to this situation. The damage is already done snd there has to be a long term solution that will prevent this from happening in the first place. If it wasn't for those developers and this article pointing out the problem then Metacritic would have done nothing at all. They simply don't care. Metacritic needs a major overhaul and they reviews are taken. As an idea Metacritic is great but it was executed poorly. Metacritic must realise that it's playing with the livelihood of people. I hope this case won't get swept under the rug because this issue must be addressed.

Avatar image for klei
Klei

1798

Forum Posts

0

Wiki Points

0

Followers

Reviews: 1

User Lists: 4

Edited By Klei

I've never cared for User Reviews before anyways and I will never. Most guys who writes scores played the game for 30 minutes. heck, that is, if they even played it. They usually overhype it or bitch it to death. It's pretty rare to find level-headed reviews.
Avatar image for rapid
rapid

1962

Forum Posts

1338

Wiki Points

0

Followers

Reviews: 0

User Lists: 8

Edited By rapid

Why, Why hurt developers??? If they were trying to make a statement couldn't they at least have actively rated 100?

Avatar image for phrosnite
phrosnite

3528

Forum Posts

1

Wiki Points

0

Followers

Reviews: 0

User Lists: 5

Edited By phrosnite

That's why I've always said that User reviews or overall users' score meant nothing.

Avatar image for stormharbinger
StormHarbinger

26

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By StormHarbinger

Great job Patrick, it's hard not to see your article and exposure as having an impact on Metacritic's actions. I'd love to think they would correct the problem on their own, but...

Anyway, kudos to you for making an impact time and again.

Avatar image for corvak
Corvak

2047

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By Corvak

User reviews make no sense. At least, it makes no sense for them to be given equal stature as a professional review. Professional reviewers typically get their jobs through good writing practices, and for being a fair judge of a game, though it is, of course, still the reviewer's opinion of the game. This is no different than major online storefronts like Amazon - who hire people to track down and remove manufacturers or publishers leaving fake reviews praising their products, or attacking a competing brand.

Because reviews are opinion pieces, just reading one site isn't the answer either, which is why Metacritic exists. Because so many of us, both gamers and industry people wrongly feel that the score is the be-all end-all of every review, Metacritic's aggregate score becomes the most accurate way to gauge the reception of a game. Sadly, we cannot break down video games into objective hard numbers to compare them - it just doesn't work - the "fun" of a video game isn't something that can be measured by a machine.

Personally, I like that they line up all the professional reviews for me to read, instead of having to dig through pages of ads and clutter. Giant Bomb is probably the best laid out gaming news site i've found - The others tend to be crammed with as much content as humanly possible onto the front page, or set up like a blog, where everything older than six hours has been pushed off the bottom.

If we start ignoring user scores, Metacritic or sites like it can be a way for quality releases like Bastion to get noticed without spending millions on mass marketing. If we continue to be distracted by numbers, it'll just be another battle in the pointless console war, and that's a war the trolls and moronic fanboys always win.

Avatar image for ninjacommando
NinjaCommando

106

Forum Posts

0

Wiki Points

0

Followers

Reviews: 0

User Lists: 0

Edited By NinjaCommando

@ZackHoagie: trolls dont have a target they just want any type of response...

Avatar image for donutfever
DonutFever

4057

Forum Posts

1959

Wiki Points

0

Followers

Reviews: 0

User Lists: 35

Edited By DonutFever

I think it's weird that small teams were being targeted.