Diary two in this Dark Athena series looks at the day-in, day-out life of a quality assurance tester.
Last week, we heard from The Chronicles of Riddick: Assault on Dark Athena developer Starbreeze about what it takes to fix up a game after the all-important focus group feedback. Now, take a look at this up-close account of the testing required to find bugs while those improvements are taking place. Quality assurance, or "testing," is sort of like the video game industry's equivalent of the mail room; relatively easy to break into without prior experience working with games, and also a good entry point from which to work your way up into more design-related positions. So anyone interested in a gaming career may want to pay attention here.
QA Life
Hugo Hirsh QA Manager, Starbreeze Studios
My name is Hugo Hirsh, and I am the QA Manager here at Starbreeze. I have been working in quality assurance for almost nine years now, and worked on hundreds of games during that time.
When I started working on Riddick a year ago, I was very surprised with the quality of the game, both from a visual perspective and from a gameplay stance too. It felt almost finished then, now it feels like a completely different game, and much better for it.
A typical day for me will start with looking over the new versions of the game that our machines make overnight. This saves us a lot of time, and allows us to launch straight into the latest code when we arrive. I’ll check that the three versions (PC, 360, and PS3) work then start making disc versions of them. Whilst the discs are burning, my team of highly trained robot-ninja QA and I will have a quick look over the nightly builds to check for any glaring errors (multiplayer menu missing, black screen on turning on the console, that kind of thing), then start the playthroughs.
After a year playing the same, slowly evolving game, playthroughs can become blindingly fast. Riddick is at least a ten hour game on the easiest difficulty setting for the latest of the two campaigns. The hardest difficulty would take about 15 hours. At one point we managed to complete the game in under an hour. Since then, more levels have been added, so now it takes us about three hours to complete the Dark Athena campaign.
Once playthroughs are completed and the discs being tested from, we move onto more detailed tasks such as acquiring every collectable in the game (well over 100 at the time of writing) and checking that reported bugs have been fixed. Every day at 4pm we turn the fun up to 11 with an hour-long multiplayer session. A chance for scores to be settled and generally a lot of smack to be talked.
This also gives us invaluable time balancing the weapons and tweaking the maps. If it’s not fun for us to play, why should we expect other people to have fun playing it? Then for the last hour of the day we are generally hunting bugs, trying to find useful reproduction steps for existing bugs or crashes.
I have had some of the most memorable experiences in my career playing Riddick. For a game that I usually spend eight hours a day playing, five days a week, I do not hate it. I don’t think I will ever hate it. Most other games I have worked on start to irk me through bad design choices or extremely poor quality overall after only a few days or weeks. Riddick has consistently outperformed my expectations.
A great example of this was early on in development. Placeholders were common, and if an achievement was earned, a note would appear on screen informing you. I was testing the main decks area, where Riddick controls a Ghost Drone. If you waited long enough, the Ghost Drone would eventually be able to access the area Riddick was controlling it from. In a "What happens if I..." moment, I snuck up behind Riddick and shot him in the head. A note appeared on the screen saying "Darwin Award?" and I was reloaded to the previous checkpoint. I had not laughed that hard since I came to Sweden.
Another memorable bug turned out to be nothing more than a single missing character in a text file somewhere. To solve it took taking an additional four QA guys repeatedly hammering the multiplayer for months on end over the summer. Crash after crash after crash, kilos of torn out hair and blood later it was finally resolved. QA can be very repetitive and mundane at times, but mostly it is rewarding and a joy to see the evolution of a game before your very eyes.
Last week, we heard from The Chronicles of Riddick: Assault on Dark Athena developer Starbreeze about what it takes to fix up a game after the all-important focus group feedback. Now, take a look at this up-close account of the testing required to find bugs while those improvements are taking place. Quality assurance, or "testing," is sort of like the video game industry's equivalent of the mail room; relatively easy to break into without prior experience working with games, and also a good entry point from which to work your way up into more design-related positions. So anyone interested in a gaming career may want to pay attention here.
QA Life
Hugo Hirsh QA Manager, Starbreeze Studios
My name is Hugo Hirsh, and I am the QA Manager here at Starbreeze. I have been working in quality assurance for almost nine years now, and worked on hundreds of games during that time.
When I started working on Riddick a year ago, I was very surprised with the quality of the game, both from a visual perspective and from a gameplay stance too. It felt almost finished then, now it feels like a completely different game, and much better for it.
A typical day for me will start with looking over the new versions of the game that our machines make overnight. This saves us a lot of time, and allows us to launch straight into the latest code when we arrive. I’ll check that the three versions (PC, 360, and PS3) work then start making disc versions of them. Whilst the discs are burning, my team of highly trained robot-ninja QA and I will have a quick look over the nightly builds to check for any glaring errors (multiplayer menu missing, black screen on turning on the console, that kind of thing), then start the playthroughs.
After a year playing the same, slowly evolving game, playthroughs can become blindingly fast. Riddick is at least a ten hour game on the easiest difficulty setting for the latest of the two campaigns. The hardest difficulty would take about 15 hours. At one point we managed to complete the game in under an hour. Since then, more levels have been added, so now it takes us about three hours to complete the Dark Athena campaign.
Once playthroughs are completed and the discs being tested from, we move onto more detailed tasks such as acquiring every collectable in the game (well over 100 at the time of writing) and checking that reported bugs have been fixed. Every day at 4pm we turn the fun up to 11 with an hour-long multiplayer session. A chance for scores to be settled and generally a lot of smack to be talked.
This also gives us invaluable time balancing the weapons and tweaking the maps. If it’s not fun for us to play, why should we expect other people to have fun playing it? Then for the last hour of the day we are generally hunting bugs, trying to find useful reproduction steps for existing bugs or crashes.
I have had some of the most memorable experiences in my career playing Riddick. For a game that I usually spend eight hours a day playing, five days a week, I do not hate it. I don’t think I will ever hate it. Most other games I have worked on start to irk me through bad design choices or extremely poor quality overall after only a few days or weeks. Riddick has consistently outperformed my expectations.
A great example of this was early on in development. Placeholders were common, and if an achievement was earned, a note would appear on screen informing you. I was testing the main decks area, where Riddick controls a Ghost Drone. If you waited long enough, the Ghost Drone would eventually be able to access the area Riddick was controlling it from. In a "What happens if I..." moment, I snuck up behind Riddick and shot him in the head. A note appeared on the screen saying "Darwin Award?" and I was reloaded to the previous checkpoint. I had not laughed that hard since I came to Sweden.
Another memorable bug turned out to be nothing more than a single missing character in a text file somewhere. To solve it took taking an additional four QA guys repeatedly hammering the multiplayer for months on end over the summer. Crash after crash after crash, kilos of torn out hair and blood later it was finally resolved. QA can be very repetitive and mundane at times, but mostly it is rewarding and a joy to see the evolution of a game before your very eyes.
Always wanted to get into Q&A but never really knew how. I can imagine it gets a tad boring after playing the same alpha / beta stage of a game for months / years on end but it would be worth it.
The new Riddick game sort of has me psyched, I'd really like to see more on the character of Riddick. So hopefully this feeds my wanting!
they didn't polish enough. The demo had plenty of frame rate drops and the aliasing was quite noticeable. So much for all that time they were left to work on the game.
Yeah, I worked in QA over at EA for two, plus years and now I'm in Wireless QA for an independent company. It's stressful work, but it can be very rewarding if you have a good development/production team who helps keep a two way communication between QA and development so you don't end up spinning your wheels writing bugs that will never get fixed. I can't wait for this game. I've already played Riddick on the Xbox AND PC, but I'd really be playing this one for Dark Athena.
"they didn't polish enough. The demo had plenty of frame rate drops and the aliasing was quite noticeable. So much for all that time they were left to work on the game." Ok, first of all, aliasing is just a part of reality, its gunna be there. Sure, its ugly, but whatever. Also, its a DEMO. They do not take a chuck of the finished game and put it out, they put a quick demo out that usually will end up being of lesser quality then the full game. For exacmple, the demo for FEAR 2 had MAJOR slowdown issues, but the full retail game improved on that a pretty decent amount.
Ah this takes me back to my letter to VG&CE about wanting to be a "beta tester", that was published when I was 12. Adorable. I'm a decent bughunter, but I doubt my patience could handle the rigors of being a professional Reviewer, never mind a year of QA with the same game. On a very good day I lay out a menu of 5 titles.
I expect it's also frustrating to be so integral to the development process, and yet have your feedback dismissed for various reasons.. time, budget, writing, etc. How many games do we players think could've used more QA, and the QA team said the same!? It's a business and at some point the code has to go out the door. The state of title updates and Day-1 patches must be especially irritating to the people whose job it is to prevent such things. I can only imagine that publishers are skimping on the QA budgets with the ability of pushing mandatory patches. It used to be that a random console game crash was unheard of. (These days the hardware crashes more often. ;)
Haha, awesome. Darwin Award eh? I have to get that achievement when the game comes out! I have already pre-ordered the game too, just a little more than a month to go! :)
I actually did like the demo. The graphics don't bother me whatsoever. It's not about the graphics at all. If it looked like the first game and plays the way it does in this demo, I would love it. That said, I didn't experience any frame rate issues. The game sounds awesome and I can't wait for it to come out.
Being an indie game dev with no QA team to do my dirty work, it's absolutely frustrating. You keep scanning and scanning your code, trying to find whats wrong. I've spent upwards of 4 hours scanning code before realising that I had misspelled a variable or something similar. What's even more frustrating is bugs that shouldn't happen, but do. Once, I added AI code that made an enemy take cover behind the nearest object (simple code, only a few lines long), and then the walking animation refused to run. I took out the code, and it worked. It was like, WTF!!! because there was absolutely no explaination for that bug.
I hope that "Darwin Award" thing wasn't a secret achevement in the final game, i'd hate to see that dude loose his job cause he told everybody how to get an achievement. :)
I can't wait NOT to buy this game. I tried the 360 demo. It's just, plain awful. All it reminds me of is that painful sequel to Pitch Black that left nothing to be desired. That was 2 1/2 hours of my life that I can never get back, so avoiding this game at all cost, would be best. Even thinking about this game feels like such a fuckin' waste of time.
Demo got me pretty excited for this game. I loved Butcher Bay, and they seemed to have added just the right amount of stuff to Dark Athena. Hopefully it'll be polished for release, because the atmosphere is fantastic, and it would be a shame to see the same frame rate problems in the retail version.
And multi-player -- I really hope they don't blow it with this because it looks pretty amazing.
I played the ps3 demo and the 360 demo and have to say the ps3 version looks alot worse then the 360 one (hopefully this will be fixed when it comes out!) ps....and yes my ps3 & 360 both R using hdmi at 720p on lcd
Interesting stuff. Although working in QA doesn't seem like a great job to me. I guess it would be worth tolerating for the chance of a better place at a developer.
24 Comments