Posted by Charlaxy (11 posts) -


Transcript:


Inspired by a forum discussion of review rating scales, and a question I received, I want to talk about why I choose not to use a rating scale for reviews on my blog that involves numbers or stars or anything else.


Although I am fine with giving a numerical score to games and may do so on my ratings posted on other sites, I don't like it, for one simple reason: reviewing games is not scientific and quantitative.


Beyond the basic question of whether or not the game is functional, everything else about it can be seen as a matter of taste. There are games that some people would give a 10 that I think are the most boring things in the world. This is because we have different ideas about what makes a game fun, what makes for a compelling story, etc.


I can compare a game to other games I've played to try to determine if I consider it above or below average, but again, this depends on my taste and what other games I've played. I think it's best to rate games on whether or not you would recommend it, and who you would recommend it for. I'm not going to recommend a shooter to someone who only likes RPGs, for example. The way I like to recommend a game is, "If you like X kind of games and the Y genre, then you'd probably like game Z." For example, "If you like playing co-op platformers with your friends and you're a fan of Super Mario 3, then you should try New Super Mario Bros. Wii."


As for rating components of a game such as the graphics and sound: I understand that there are games where these elements are legitimately bad, but it's difficult to give a numerical rating to them. I pretty much rate them on whether or not they were satisfactory for what the game is. Players of different types of games have different expectations for graphics. Were the graphics in the Scott Pilgrim game horrible because they were 2D and pixelated and didn't fully take advantage of the processing power of the latest video cards? No, they're just a certain style. What about Minecraft? Should it get lots of points off for bad graphics/sound? Some people argue yes, but others would say no, it's just a different type of game that has different standards. It's not trying to be like other 3D ones, so it shouldn't be judged on the same scale.


Additionally, expectations for graphics change over time. How would you rate the sound and graphics of the original Super Mario Bros.? 8 out of 8 bits? Comparing it to new releases on a current 1-10 scale, it'd probably be 1/10. Of course, no one would rate it that way. Why should a game that is great for its time and that has amazing gameplay have points deducted because it doesn't compare to the latest titles in the amount of polygons it uses? Most would argue that the graphics of Half-Life should be judged by a different standard than Half-Life 2 is.


Now, this brings me to my point: if these older games are still considered great titles because of their gameplay and despite their limited graphics, then is it fair to lower the score of new releases because they don't utilize the "best" graphics currently available to them? And because a game's graphics vary by style, can you even objectively say which graphics are the best?


Again, look at Minecraft: it's a sensation even though it doesn't have the latest in graphics. Apparently, it doesn't need them to be a game that people like. So, why should I lower a game's 1-10 score because of the graphics alone? This is just one example of why it's difficult to determine what, exactly, should go into a numerical score, and that's part of the reason why I argue against them.


As for rating a game's graphical/sound appeal, I think it's best to just show the game. This is why I include a video of gameplay with my reviews. Let people judge for themselves whether or not it's acceptable to them. Let them see several minutes of uninterrupted gameplay — not just a few specially chosen screenshots or cut scenes — and use that in conjunction with the full text review of the game to see if it's something that they'd like. Video is worth more than words, in this respect. It's a lot better than trying to assign a numerical score to something that isn't quantitative.


So, I show the game, I describe it, I point out the good and bad, and I make a recommendation. I can assign a number value to that, based on my taste and whether I think it's above or below average in comparison to what else I've tried. The number really doesn't mean anything other than that, so it's not something important to me. The description and recommendation matter more, and the number gives the false impression that the review is somehow scientific.


Maybe I'm just too picky about this. Maybe everyone already understands and agrees with my views, and they want to see numbers anyway, just for the sake of having some kind of scale for quick-comparisons, however inaccurate it may be. That's fine. However, I won't assign numbers to anything on my blog, and this is why.

#1 Posted by Charlaxy (11 posts) -


Transcript:


Inspired by a forum discussion of review rating scales, and a question I received, I want to talk about why I choose not to use a rating scale for reviews on my blog that involves numbers or stars or anything else.


Although I am fine with giving a numerical score to games and may do so on my ratings posted on other sites, I don't like it, for one simple reason: reviewing games is not scientific and quantitative.


Beyond the basic question of whether or not the game is functional, everything else about it can be seen as a matter of taste. There are games that some people would give a 10 that I think are the most boring things in the world. This is because we have different ideas about what makes a game fun, what makes for a compelling story, etc.


I can compare a game to other games I've played to try to determine if I consider it above or below average, but again, this depends on my taste and what other games I've played. I think it's best to rate games on whether or not you would recommend it, and who you would recommend it for. I'm not going to recommend a shooter to someone who only likes RPGs, for example. The way I like to recommend a game is, "If you like X kind of games and the Y genre, then you'd probably like game Z." For example, "If you like playing co-op platformers with your friends and you're a fan of Super Mario 3, then you should try New Super Mario Bros. Wii."


As for rating components of a game such as the graphics and sound: I understand that there are games where these elements are legitimately bad, but it's difficult to give a numerical rating to them. I pretty much rate them on whether or not they were satisfactory for what the game is. Players of different types of games have different expectations for graphics. Were the graphics in the Scott Pilgrim game horrible because they were 2D and pixelated and didn't fully take advantage of the processing power of the latest video cards? No, they're just a certain style. What about Minecraft? Should it get lots of points off for bad graphics/sound? Some people argue yes, but others would say no, it's just a different type of game that has different standards. It's not trying to be like other 3D ones, so it shouldn't be judged on the same scale.


Additionally, expectations for graphics change over time. How would you rate the sound and graphics of the original Super Mario Bros.? 8 out of 8 bits? Comparing it to new releases on a current 1-10 scale, it'd probably be 1/10. Of course, no one would rate it that way. Why should a game that is great for its time and that has amazing gameplay have points deducted because it doesn't compare to the latest titles in the amount of polygons it uses? Most would argue that the graphics of Half-Life should be judged by a different standard than Half-Life 2 is.


Now, this brings me to my point: if these older games are still considered great titles because of their gameplay and despite their limited graphics, then is it fair to lower the score of new releases because they don't utilize the "best" graphics currently available to them? And because a game's graphics vary by style, can you even objectively say which graphics are the best?


Again, look at Minecraft: it's a sensation even though it doesn't have the latest in graphics. Apparently, it doesn't need them to be a game that people like. So, why should I lower a game's 1-10 score because of the graphics alone? This is just one example of why it's difficult to determine what, exactly, should go into a numerical score, and that's part of the reason why I argue against them.


As for rating a game's graphical/sound appeal, I think it's best to just show the game. This is why I include a video of gameplay with my reviews. Let people judge for themselves whether or not it's acceptable to them. Let them see several minutes of uninterrupted gameplay — not just a few specially chosen screenshots or cut scenes — and use that in conjunction with the full text review of the game to see if it's something that they'd like. Video is worth more than words, in this respect. It's a lot better than trying to assign a numerical score to something that isn't quantitative.


So, I show the game, I describe it, I point out the good and bad, and I make a recommendation. I can assign a number value to that, based on my taste and whether I think it's above or below average in comparison to what else I've tried. The number really doesn't mean anything other than that, so it's not something important to me. The description and recommendation matter more, and the number gives the false impression that the review is somehow scientific.


Maybe I'm just too picky about this. Maybe everyone already understands and agrees with my views, and they want to see numbers anyway, just for the sake of having some kind of scale for quick-comparisons, however inaccurate it may be. That's fine. However, I won't assign numbers to anything on my blog, and this is why.

#2 Posted by Immuniity (241 posts) -

Are you a robot?

#3 Posted by DrPockets000 (2859 posts) -

Great views, but loosen up with the voice.

#4 Posted by Vodun (2365 posts) -

Do you watch/listen to Feedback over on G4? They had an interesting discussion about this a while back and it seems most reviewers agree with your point of view. But pretty much every site gives numerical scores because they need it to be "marketable". They want to be seen on Metacritic and the like. I don't remember their exact arguments, but the jist of it was that they wouldn't make any money if they didn't put a number at the end of the review.
 
That just feels very wrong to me. Aren't reviewers supposed to be the helping hand in the market jungle? The hand that doesn't care about sales and the like, but gives honest opinions in a way that only helps the reader and cares nothing for the business? I'm obviously very naive in this thinking as everyone needs to get paid, but at the same time I have a hard time trusting someone to help me make a decision about a purchase...when they themselves have a motive behind helping me make this purchase. 
 
Basically, I completely agree with you and I think scores and a faux-science approach to reviewing is ludicrous. It's opinions and feelings, you can't measure it! 
 
I also don't think there's enough debate about it as I feel most people genuinely interested in gaming agree with this opinion, yet it remains the norm to put a stupid numerical value on quality.

#5 Posted by owl_of_minerva (1455 posts) -

My view is that scores can useful as shorthand for stating what you think the game's significance is, but only in relation to other games of its genre, rather than an attempt to assess its 'objective worth' whatever that might be. So there can be no comparison between scores of games from different genres, thus getting rid of a lot of pointless and dumb discussion comparing scores. A five-star scale is more than adequate for this task, and does away with the fallacy of false precision underlying other review scales. So 5s and 4s are essential games of the genre, 3s most likely only for fans as they have drawbacks, and 2s and 1s for insignificant entries. Giantbomb's scale has it right, I would say.

#6 Posted by Kazona (3055 posts) -

I think a good reviewer rates a game on case-by-case basis, and on its own merits. They can compare it to other games--within reason--but should keep it within the realms of the same genre. A FPS, for instance, should not be compared to an RPG, but comparing Gears of War 2 to Vanquish isn't something I have a problem with; they are, in essence, the same kind of game.  
 
Another criteria for rating games, in my opinion, would be that the person reviewing it should actually have a vested interest in it, and it should, first and foremost, be written for people who share that interest. For example: had Starcraft 2 gotten bad ratings, most RTS fans would know to steer clear of it. With the high praise it got, however, most of those fans feel pretty safe in spending their money on it, while those who don't like the genre will, usually, still pass up on it. 
 
There are exceptions, of course. I for one have never been a huge fan of RTS games, but I did buy Starcraft 2 because of the praise it got, and I regret doing so. To be honest, though, the numerical score had little to nothing to do with that. It was mostly Brad's talk about it on the Podcast, and all the hyped threads that lead me to buying the game. You could say that I fell for the hype. Had I not fallen victim to that, however, I would have more than likely taken the score at face value, and realized that for an RTS fan, this game would be a real treat, but that, despite its high score, I still should have thought twice about buying it. 
 
In short: I think you can assign a score to a game, so long as one doesn't make the mistake of comparing it to too many other things. A game should be rated as is, and not "as compared to."

#7 Posted by Charlaxy (11 posts) -
@Immuniity: Yes.
#8 Posted by Charlaxy (11 posts) -
@Vodun: I don't look at G4 much, but I remember watching a video where one of their commentators talked about what BS he thought Metacritic's 100 point score system was, and some of what you said about how reviewers are forced to adapt to it.  Anyway, it's good to know that people actually paid to do reviews professionally also agree with this, although they still have to adhere to the system anyway.  I understand that numbers are useful for a quick way to see whether the review is going to be favorable or not, but I'm disappointed with the heavy reliance upon scores, and how some reviewers assign a number to everything.  Some visitors to my site have asked me why I don't score graphics and sound on a numerical scale, and that's what caused me to really think about what utter nonsense that seemed to be.  Additionally, I was forced to give something a 1-10 score along with one liners about its pros/cons/"WTF" moments in order to get a review posted on another site.  I felt like I was selling games short by summarizing reviews with just those four things.  But, I can't force anyone to actually read the review anyway.  I just hope that most readers know better than to think that's an adequate way to summarize a game  -- and it seems like they do.
#9 Posted by Icemael (6271 posts) -
@owl_of_minerva said:
" My view is that scores can useful as shorthand for stating what you think the game's significance is, but only in relation to other games of its genre, rather than an attempt to assess its 'objective worth' whatever that might be. So there can be no comparison between scores of games from different genres, thus getting rid of a lot of pointless and dumb discussion comparing scores. A five-star scale is more than adequate for this task, and does away with the fallacy of false precision underlying other review scales. So 5s and 4s are essential games of the genre, 3s most likely only for fans as they have drawbacks, and 2s and 1s for insignificant entries. Giantbomb's scale has it right, I would say. "
If we assigned scores to games like that, we'd end up in situations were we'd have to give shitty games five-star ratings on account of them being the very best in genres of nothing but terrible games, and that's just not acceptable.

Games should be rated on how heartily we wish to recommend them -- and nothing else. Naturally, this means we have to rate them not only against all other games, but against everything we know of -- and that is the only true and proper way to make value judgments.
#10 Posted by Video_Game_King (34664 posts) -
@owl_of_minerva said:
" My view is that scores can useful as shorthand for stating what you think the game's significance is, but only in relation to other games of its genre, rather than an attempt to assess its 'objective worth' whatever that might be. So there can be no comparison between scores of games from different genres, thus getting rid of a lot of pointless and dumb discussion comparing scores. A five-star scale is more than adequate for this task, and does away with the fallacy of false precision underlying other review scales. So 5s and 4s are essential games of the genre, 3s most likely only for fans as they have drawbacks, and 2s and 1s for insignificant entries. Giantbomb's scale has it right, I would say. "
I agree with this, but I still use a 10 point scale for recreational purposes. That's why almost none of my blogs ever mention a score of any kind.
#11 Edited by owl_of_minerva (1455 posts) -
@Icemael:  I agree that there should be some implicit reference to the state of gaming, otherwise the scores would descend into relativism. I was simply prioritising mechanics. How heartily we would recommend games should not in the last instance be based on how good Espgaluda is compared to Gabriel Knight, for example, as that'd be equally absurd. I just can't fathom how there's any possible comparison between the two beyond the psyche of the person reviewing them and the sensation of enjoyment. Ideally, a reviewer would be able to compare a new game to all prior entries in the genre, thus establishing the maximally informed judgment about the game's significance. 
Are new genres even possible? Every 'new' genre that I've seen since has the 90s has been a novel combination of elements taken from other genres. Also, genres are sufficiently saturated that every one has attained some level of quality. I take your point, but I don't think your counter-example is likely to arise.
 
@Video_Game_King: How much would Fragile Dreams get, given the damage it's done to your psyche?
#12 Posted by Video_Game_King (34664 posts) -
@owl_of_minerva: 
 
7.5/10, a respectable score.
#13 Edited by Icemael (6271 posts) -
@owl_of_minerva said:

" How heartily we would recommend games should not in the last instance be based on how good Espgaluda is compared to Gabriel Knight, for example, as that'd be equally absurd. "

A reviewer should never compare the two in his review -- that'd just be a waste of time -- but whether he wants it or not, he will compare them in his mind and, if he rates games with any honesty, give the one he likes the most the higher score. 
 
@owl_of_minerva said:

" Are new genres even possible? Every 'new' genre that I've seen since has the 90s has been a novel combination of elements taken from other genres. Also, genres are sufficiently saturated that every one has attained some level of quality. I take your point, but I don't think your counter-example is likely to arise. "

It really depends on what level you look at. There are the cardinal genres (action and strategy/puzzle), then there are sub-genres of those (platforming, shooter, RTS, SRPG, adventure et cetera), then there are sub-genres of those (first-person shooter, third-person shooter et cetera) and so on. New genres pop up fairly often -- for example, we got cover-based shooters not too long ago -- at the level of sub-genres of sub-genres of sub-genres. The broadest genres, however, will never increase in number, as they already cover everything that can be done.
#14 Posted by Sin4profit (2873 posts) -

i got as far as 1 min before i felt like i was listening to the "side effects" section of a cootchie cream commercial and had to stop.

#15 Posted by Vodun (2365 posts) -
@Charlaxy: Maybe a 3 point scale is enough? That way it's possible to either be negative, meh or positive. Then the different levels of each you feel wouldn't come into play. 
 
Or if you want to go very subtle, just take a photo of yourself and have that as a score. Your facial expression should show your feelings of the game, it's hard to start an argument about how your smile was bigger for Call of Duty than for Battlefield. Although I'm sure some would....