An interesting review experiment to consider

I was watching this week's I Love Monday and notice the ranting from Brad against Resident Evil 6, which leads me to think about Gamespot's low score for RE6 (4.5/10), which again leads me to want to research about this topic:

Can we compare the review scores of Giantbomb and Gamespot for each game, convert them into consistent 5 or 10 point scale, then sort them into before and after the CNET merger, and see if the difference on scoring have a smaller Standard Deviation (σ, or on layman's term, the score from Giantbomb team will be more similar to that from Gamespot team, after CNET merger) ?

--

Anyone interested in doing this, or knows a systematic way of calculating such statistics, I would be glad to know more...

--

NOTE: I am NOT suggesting, even slightly, that Giantbomb/Gamespot review system is compromised; but I do believe "husband and wife looks alike": when you pitch a bunch of reviewers closer together, the exchange of idea might just affect scoring.

You're gonna need a lot of reviews to show anything statistically significant.

http://en.wikipedia.org/wiki/Sample_size_determination#Required_sample_sizes_for_hypothesis_tests

@lazarenth said:

You're gonna need a lot of reviews to show anything statistically significant.

http://en.wikipedia.org/wiki/Sample_size_determination#Required_sample_sizes_for_hypothesis_tests

true and valid concerns: while we don't have a lot of reviews after the merger, note that the total number of reviews isn't that large neither, so we might just have large enough a sample.

Seems like a waste of time to me.

Maybe they both just didn't like the game. If you disagree with someone's opinion, fine, but don't try and justify it by making up some conspiracy. Don't be one of those guys...

do people really care that much about review scores

@FlarePhoenix said:

Maybe they both just didn't like the game. If you disagree with someone's opinion, fine, but don't try and justify it by making up some conspiracy. Don't be one of those guys...

Didn't play RE6 (or previous REs), cannot say if i agree or disagree with them.

Conspiracy is possibly not the best word to describe this sort of experiment... at least the result wouldn't be used in a way that argue such conspiracy is valid.

@df said:

@FlarePhoenix said:

Maybe they both just didn't like the game. If you disagree with someone's opinion, fine, but don't try and justify it by making up some conspiracy. Don't be one of those guys...

Didn't play RE6 (or previous REs), cannot say if i agree or disagree with them.

Conspiracy is possibly not the best word to describe this sort of experiment... at least the result wouldn't be used in a way that argue such conspiracy is valid.

Hmm it's kind of what it sounds like to me "Both Gamespot and Giant Bomb didn't like a game, so there must be something going on causing them to think alike".

A lot of people seem to dislike Resident Evil and it's silly to think that the staffs opinions of a game will be changed by being closer to gamespot guys. Brad out of all of them is always the one who really gets behind a game or really hates it no matter what the others say.

I'm one of those guys.

There's only one opinion important to how good a game is.

Whose opinion? The person who is playing it, obviously.

@FlarePhoenix said:

Maybe they both just didn't like the game. If you disagree with someone's opinion, fine, but don't try and justify it by making up some conspiracy. Don't be one of those guys...

@FlarePhoenix said:

@df said:

@FlarePhoenix said:

Maybe they both just didn't like the game. If you disagree with someone's opinion, fine, but don't try and justify it by making up some conspiracy. Don't be one of those guys...

Didn't play RE6 (or previous REs), cannot say if i agree or disagree with them.

Conspiracy is possibly not the best word to describe this sort of experiment... at least the result wouldn't be used in a way that argue such conspiracy is valid.

Hmm it's kind of what it sounds like to me "Both Gamespot and Giant Bomb didn't like a game, so there must be something going on causing them to think alike".

This is not a reaction to RE6 review, but it certainly got me thinking about the possible ways of idea exchange between reviewers.

It's equally possible that "Brad never talked to Kevin about this game" or "They discussed a lot and share similar views" (note that Brad's review is not out yet).

--

"misery loves company", I am just wondering if this is a factor on game review scoring, that's all.

It's not a coincidence that both gamespot and giantbomb hate resi 6 it's a nice lookin lg game but the gameplay mechcanics are ancient it's just not fun to play

I swear standard deviation is a layman's term.

ZOMG Giantbomb sold out. They got no more integrity.

I don't know where people are getting that this is an attack on GB and GS for the low reviews, as it didn't seem that way to me at all. But, I agree with these guys:

@laserbolts said:

Seems like a waste of time to me.

@Bell_End said:

do people really care that much about review scores

What if Resident Evil 6 is just a bad game an everybody agrees? I've yet to see that scenario considered.

@LikeaSsur said:

What if Resident Evil 6 is just a bad game an everybody agrees? I've yet to see that scenario considered.

I had that covered already.

@FlarePhoenix said:

Maybe they both just didn't like the game.

@Bell_End said:

do people really care that much about review scores

apparently so ... wich is sad people actually need Brad or someone else to say "don't play this game" and they'll do it rather than forming their own opinion.

@AngelN7 said:

@Bell_End said:

do people really care that much about review scores

apparently so ... wich is sad people actually need Brad or someone else to say "don't play this game" and they'll do it rather than forming their own opinion.

Well if reviews actually just said "don't play this game" you might actually have a point, but most of them tend to go into more detail and give reasons as to why you should or should not play the game. If you read the review and go "oh I don't like when this happens in games" you might go out and try the game anyway, or if you go "oh that doesn't sound like it will bother me" you're probably going to move on. For example, any talk about the quality of online multiplayer I basically ignore, because I never play online. It has no relevance to me.

Also, games cost a lot. This isn't like a movie where you can drop twenty dollars, and sit for a couple of hours. Sixty to One Hundred dollars is quite a bit of money to drop on a product, and you can't really be doing it for every single game that comes out. So you kind of need reviews to be able to tell you why you should or shouldn't play a game. You can read multiple reviews to get differing opinions and, as I said, you should know what you like and dislike and be able to match that up to what is been said in the review.

But hey, if you can afford every release that comes out that's pretty impressive.

@Bell_End said:

do people really care that much about review scores

Of course! Otherwise they would need to read, and we all know, reading is for suckers.

@FlarePhoenix said:

@AngelN7 said:

@Bell_End said:

do people really care that much about review scores

apparently so ... wich is sad people actually need Brad or someone else to say "don't play this game" and they'll do it rather than forming their own opinion.

Well if reviews actually just said "don't play this game" you might actually have a point, but most of them tend to go into more detail and give reasons as to why you should or should not play the game. If you read the review and go "oh I don't like when this happens in games" you might go out and try the game anyway, or if you go "oh that doesn't sound like it will bother me" you're probably going to move on. For example, any talk about the quality of online multiplayer I basically ignore, because I never play online. It has no relevance to me.

Also, games cost a lot. This isn't like a movie where you can drop twenty dollars, and sit for a couple of hours. Sixty to One Hundred dollars is quite a bit of money to drop on a product, and you can't really be doing it for every single game that comes out. So you kind of need reviews to be able to tell you why you should or shouldn't play a game. You can read multiple reviews to get differing opinions and, as I said, you should know what you like and dislike and be able to match that up to what is been said in the review.

But hey, if you can afford every release that comes out that's pretty impressive.

The thing is I don't need to buy the game there's plenty of media and first hand experiences (demos, trailers, walktroughs) out there wich to me are better than someone else's opinion because even if you aling with it you can never truly tell if that person went with the same frame of mind (a jaded gamer that can't stand the same thing again, hates the franchise for some reason, hates the genre ect) as you will when playing that game maybe that person sees how others are reactinng and say "hey maybe I don't like this I didn't thought about that" or "right I didn't notice that mechanic when I played but since it's bad I'm gonna say is bad too" wich is why I never listen or read any reviews in the case of Resident Evil 6 that E3 demo got me worried I saw bad animations, weird aiming , and over the top action then I played the demo and got the same negative feelings that's all I needed to know because I saw it and I experience it.

If you think Giant Bomb changed its' reviews due to new ownership then you obviously have no idea about what the hell this site is all about.

Waste of time. If Brad says a game sucks I trust that Brad actually thinks that game sucks.

In this case, I played the demo and tend to agree.

Well, we could do that, but I don't really see the point of this experiment. Even if you could see a change in review trends, there are so many things that affect a final score, it wouldn't really prove anything about one review team influencing the other. Heck, even the hypothesis seems rather shaky, to my knowledge the staff of the two websites don't seem to have that much interaction on a day-to-day basis.

@AngelN7 said:

@FlarePhoenix said:

@AngelN7 said:

@Bell_End said:

do people really care that much about review scores

apparently so ... wich is sad people actually need Brad or someone else to say "don't play this game" and they'll do it rather than forming their own opinion.

Well if reviews actually just said "don't play this game" you might actually have a point, but most of them tend to go into more detail and give reasons as to why you should or should not play the game. If you read the review and go "oh I don't like when this happens in games" you might go out and try the game anyway, or if you go "oh that doesn't sound like it will bother me" you're probably going to move on. For example, any talk about the quality of online multiplayer I basically ignore, because I never play online. It has no relevance to me.

Also, games cost a lot. This isn't like a movie where you can drop twenty dollars, and sit for a couple of hours. Sixty to One Hundred dollars is quite a bit of money to drop on a product, and you can't really be doing it for every single game that comes out. So you kind of need reviews to be able to tell you why you should or shouldn't play a game. You can read multiple reviews to get differing opinions and, as I said, you should know what you like and dislike and be able to match that up to what is been said in the review.

But hey, if you can afford every release that comes out that's pretty impressive.

The thing is I don't need to buy the game there's plenty of media and first hand experiences (demos, trailers, walktroughs) out there wich to me are better than someone else's opinion because even if you aling with it you can never truly tell if that person went with the same frame of mind (a jaded gamer that can't stand the same thing again, hates the franchise for some reason, hates the genre ect) as you will when playing that game maybe that person sees how others are reactinng and say "hey maybe I don't like this I didn't thought about that" or "right I didn't notice that mechanic when I played but since it's bad I'm gonna say is bad too" wich is why I never listen or read any reviews in the case of Resident Evil 6 that E3 demo got me worried I saw bad animations, weird aiming , and over the top action then I played the demo and got the same negative feelings that's all I needed to know because I saw it and I experience it.

Yeah, because demos and trailers aren't specifically made to show you the best parts of the game or anything like that; that would just be silly :P. Walkthroughs are slightly better, but it is a lot different to watch a game than it is to actually play it. Your experience is going to be different. As I said before, you should be able to confirm your own gaming tastes with what is coming through in the review.

Reviews are a useful tool to use when it comes to purchasing decisions. If someone is just looking at the scores and going "I'm only going to buy the 4 - 5 star rated games" that might be a problem, but that's an entirely different issue. There is nothing wrong if you don't read reviews, but I'm curious why you seem to actively hate them.

@FlarePhoenix: I don't hate them I just don't understand people that just go "Hey [reviewer] said this game is bad so it must be bad I wont buy it" because if that's all you need then that person misses on a lot of great games that the partciular reviewer doesn't like , the reviewers might have first hand experience but that's his/her not mine's that's why I don't pay attention to them.