The Undiscovered Country (or "Brad, please be excited")

On the last couple of bombcasts Brad has talked about the incremental improvements in game graphics and slightly lamented the loss of the experience of being presented technical feats that leave us awestruck. Having listened to Brad over the years it’s clear he and I had very parallel formative gaming years. We’re about the same age and grew up being exposed to the same games, technologies and gaming zeitgeist and I understand exactly where he is coming from. We were absolutely spoiled in the 90’s. I vividly remember the first time I saw Quake 2 at the local PC shop running at a resolution of 800x600 at 60fps on SLI Voodoo2 cards and being floored. Although at times I can step back and look at Uncharted 3, Halo 4, or Gears of War 3 and marvel at the technical complexity that is being pushed out of what is now comparatively low-end hardware, there is a lack of those “holy shit” moments in graphics compared to days past. Without going in to much detail I’d like to explain what’s left to do in graphics, why those “hot damn” moments are potentially gone forever in graphics, but also to present two places where I’m confident those moments have yet to occur.

Computer graphics, specifically the realistic representation of light transportation to our eyes, is a solved problem at the theoretical level. We know exactly how light bounces around the world and interacts with our eyes and it can neatly be summed up by a set of equations (BRDF, BTDF, BSSRDF). From here on out everything is engineering the implementation of those equations serving two purposes. Firstly, for mathematical reasons the above equations cannot be directly computed, but instead we sample and approximate them to a degree that we can call good enough for the intended purpose. The techniques involved in these approximations are still actively developed. The second engineering task is marrying the math to the available computing power. In games we need to push out a rendered frame at least thirty times a second and in the offline rendering world frames need to be rendered fast enough that directors can iterate over the content, that films come out on time, and the amount of electricity and computing resources used doesn't blow the budget. As computers become more powerful and our approximations of the math become robust enough for all intended purposes it isn't out of the question that all computer rendering will collapse to a unified set of practices across both real-time and offline rendering. This trend is already taking place: at this year’s Siggraph conference there is a clear trend next-gen gaming engines (including Unreal, Cryengine, and Unity) to the use of physically-based renders, where all of the shaders and material properties are constrained to being built with a fixed set of physically plausible parameters found in the above equations. The purpose is to give physical justification to the way each material interacts with light and in doing so not only provide higher quality rendering, but more importantly avoid some of the hackish tricks that can cause production pipeline issues. Meanwhile in the off-line film rendering space almost all techniques across all the major houses (ILM, WETA, Sony, and even Pixar) have collapsed down to similar single-pass ray/path tracing techniques. What was once a diverse set of techniques and approaches is slowly becoming more and more unified even between movies and games. (As a side note please don’t think this means everything will look the same, a common complaint between games that share the same middle-ware. Despite the unified theoretical approach the end results can stylistically look completely different)

More to the point of this article, back in the 90’s off-line rendering was evolving as much as real-time rendering and in both areas there was a vast undiscovered country with quality leap frogging itself year-over-year. We were equally blown away by the first liquid morphing T-1000 in Terminator 2 as we were the OpenGL graphics of Quake 2. Each successive Pixar film brought forward strides in rendering quality and complexity. Today the world is different. Although there are still quality improvements to be had, for the most part film visuals have plateaued be it the newest Pixar or the newest super hero movie. In a film like Avatar, the end game of real-time game rendering is before our eyes and we are just waiting on content production efficiency and computer rendering power to achieve parity.

Thankfully, I still see two areas ripe for “holy shit” moments. The first comes from the current hot kid on the block: Virtual Reality (VR). My day job is in a human factors/cognitive science/psychology lab at a University where I get exposed to lots of game related technology, including VR and advanced simulations. My quick summary of nearly all of VR up until 12 months ago: it sucked. All the things the Oculus folks mention in their pitch are entirely true: professional and consumer VR is a mess of really expensive, laggy, uncomfortable stuff. I don’t want to pile on to the already overblown hype that the Rift enjoys, but it really is a game changer. For me that title gets bestowed upon it because of how the cost and replaceable, widely available parts will allow for easy experimentation and iteration. You've heard all that praise before, so how exactly will this make our jaw drop? Well, again working at the lab I have exposure to perhaps a little more hardware than your average person and more importantly a lot of smart people who know both the engineering and perceptual psychology factors required to start making something great. Recently we've put together a demo that did two important things. Firstly, the software was tweaked to match the physiology of the person, namely the interpupillary distance (IPD) and height of the user was correctly modeled. Most things, like the infamous Rift roller coaster, are very generic in configuration: height, lens type, FOV, and IPD are not tweaked per user. Secondly we attached a set of ten position/orientation tracking sensors to limbs so that a full body tracking system could be implemented. In doing so you were able to look down in the game world and see your own body and move all your limbs as you would in the real world. It was far from perfect, there were the odd Kinect-style jerking twists and the lag wasn't yet invisible, but it was compelling… really, really this-is-totally-unlike-anything-before compelling. Just as the magazine and usenet text of yester-year couldn't convey the first time you saw really good accelerated 3D, the feeling you get interacting with the world in good VR is essentially ineffable in a way that no bump in 3D ever was for me. To be less poetic: reserve judgment on VR until you actually use a good sensor setup with content made specifically for it, because holy shit. Just as proper 3D polygonal graphics added a thick trunk to the genius of gaming experiences, so too will VR if it takes off. Further these basic position sensors and optics are just the beginning, there is much more open ground to provide sensory cues involving touch that have only started to be explored in the academic space. There is so much ground being covered in the near term its almost difficult to catch your breath and look that far ahead.

Another form of evidence to this end is the involvement of the developer community. In many ways the VR scene parallels the 90’s graphics scene. We have disruptive new hardware that's going to iterate yearly. We have a lot of academic material, in this case perceptive psychology, which we can draw on to point towards how to apply that hardware. We have a lot of technical and engineering hurdles to break, but each time we break new doors are going to open. Each of these doors could open new opportunities for mechanics and experiences and each of them has the opportunity to provide our sought after astonished impressions. It should be no surprise to see some of the legends from 90’s computer graphics such as Michael Abrash (inventor of graphics mode X, programmer on the original WindowsNT graphic stack, Quake programmer) and John Carmack (Wolfenstein, Doom, Quake) are jumping on board at the expense of doing further graphics research and development.

The second “jaw drop” is far more speculative and likely much further off. Game “AI” is essentially the cognitive equivalent of fish layered with beautiful animation, voice acing and scripted sequences on top. However, there is the chance that someday game “AI” reaches the point where we can hold conversations with it. Whether through trickery or through some kind of true “AI” the first time we say something using natural language, not through a button press, not constrained to some keywords or selected from small set of dialog choices, but the first time we speak our minds, have an avatar turn around and give us an on-the-fly, intelligent response is going to be jaw dropping. However, I’d imagine in some ways that by then the world will have changed by then. It’s only a slim prospect that gaming would be the first application of such a technology, so by the time it appears in games it may be common place elsewhere and thereby less impressive. More importantly unlike graphics, where we have hard physics and math, and perceptive psychology, where we have some hard knowledge, some general notions and much in progress work, when it comes to high-level cognition required to understand and generate the semantic content of language we are really, really in the dark. How far out such a technology would be, if even possible at all, is open… but just imagine!

The 90’s were amazing for videogames and as someone who loves the technology behind it all the leaps in graphics played a substantial role in my excitement and current nostalgia. I have such fond memories of it all and feel privileged to have lived through the experience of that progress. However, I have never been more excited about gaming and the technologies surrounding it than I am now. There are diverse reasons for this (the ease of opportunity for creators, the breadth of different experiences and outlets to encounter them, the prevailing culture), but for me jaw-dropping technical progress has not fallen off the list. Graphics still have a way to go and there will be a few spectacles left along the way, but the way we embody ourselves within game worlds is about to be provided with a new and exciting form and there is always the prospect of the citizens of those worlds interacting with us in ways that are almost unfathomable by today standards. There is still so much undiscovered country in the technology behind games and so much opportunity to be blown away by its progress.

13 Comments

Pachter cast! (and my E3 platform predictions)

I absolutely loved hearing Michael Pachter on the Bombcast. He has a really unique vantage point in contrast to your average gamer or gaming press staffer. There are so many great topics to talk about in the Pachter-cast I felt compelled to write a blog post with my 2 cents. I’m going to stick with hardware and platforms for this post.

Kinect  

I completely agree that MS got a more than a little lucky on Kinect sales.   Kinect absolutely will need to come out swinging this year, but I also think there are dormant possibilities for the device.   Like the members of the podcast too often people try to shoe-horn the halo’s and gear’s into a kinect style game, but the “hardcore” could take on many other forms.   I’d argue some of the genres that are still “best on the PC” could be molded in experiences better than a gamepad would enable: a Civilization or Telltale/Heavy Rain style adventure game could fit.

More importantly for Kinect in the hardcore space, is the “Better with Kinect” idea that already appears to be a sure thing with box shots leaking for games like Ghost Recon: Future, Forza 4 and who knows what else. I’ve always been curious about the potential for hybrid play using both a controller and a Kinect. Hacking mini-games, dialogue-trees, potion-mixing, inventory management, and NPC interaction seem ripe for experimentation. The only question is how annoying the context switching between gamepad and Kinect would be, but if done right the experience could be magical.

I don’t expect to see something that blows me away with Kinect, but in some ways don’t want to. I want to see the device naturally integrate into sensible extensions of games and mechanics I already find fun adding that little bit of extra interactivity or suspension of disbelief. I’ll be more than satisfied with my purchase when this happens.

The ‘Next Generation’ is already on the PC  

Jeff made the comment on the cast that you used to be able look to the PC for the indication of the move to the next thing. This usually dealt with hardware providing graphical leaps, but in this case, I think we will see this through monetization strategies.

If the mobile games the general population play on the go are being dominated by iOS I would argue the games the general population play at home are being dominated by Facebook.   I really feel, despite cringes of the Giantbomb crowd, this stuff is going to creep into consoles, and I think especially with MS this will happen in an unexpectedly big way.   MS have tested the waters before: 1 vs 100, game room, family zone style content, etc. They also have probably the best micro-transaction platform out there.   They also have a phone that needs more than just iOS ports. They also have failed a number of times to build an MS PC platform brand: they lost the games we play to Steam and have watched the opportunity to be a part of the games as service revenue stream slip away to facebook (tho MS has a $240M+ stake in facebook and has also been a part of their developer tool chain in the past). I think the can hit many birds with a stone by bringing forward a unified platform of web/facebook like games that play in your browser, your windows phone and on XBLA. Perhaps even in collaboration with facebook, though how revenue splitting would happen is the big hindrance there. While Sony (NGP) and Nintendo (3DS) take perhaps their last stand on the mobile front, MS will leap frog them into socially networked games.

The barrier will be low as possible. Most of the games will be free. You will play them with your friends. The potential player base will speak to developers. You will buy themes, avatar items, bonuses and skip-aheads. My bold prediction: A ‘game room’ for board games, Fable, Banjo or Viva Pinata, and Age of Empires Online will all be coming to your couch, phone and browser in a unified, social-gaming way. A developer platform based on HTML5/Silverlight will come out for developers that’s less complicated than XNA, but provides more of the monetization mechanics.

Nintendo or "Developers, Developers, Developers"

Next up was the failure of the 3DS and in a more general sense the future direction of Nintendo. Again I agree with every part of the discussion, but there is an important factor that I think can be overlooked in the gaming world that can point to the successes or failures of platforms and peripherals: follow the developers. As an individual in about 10 minutes of paperwork and $100 I can be an official iOS developer, and with $0 more dollars and one additional step I can make that game using Unreal Engine. Where do I even go to sign up for a 3DS dev kit? I’m sure I need to be a proper corporation, probably need a publishing deal, need to shell out a ton of cash and go through many weeks of process to get going. Then when I’m done I need to cert, produce carts and fit in someone’s release window. Whether you are an indie developer or EA, Nintendo has to either come up with a very answer to the question “Why should I bother?” or just start making a more lean SDK and cert process.

This applies to everyone everywhere trying to be a platform (even if you aren't making games!), but especially the mobile space. Apple is making the barrier to entry for serious developers, big and small, fundamentally nothing. So is UDK. So is Facebook SDK. In a more minor way so are Steam and XNA. Triple AAA games are still going to happen with big proprietary tech, on enormous budgets, through pain-in-the-ass development processes. However, in my opinion, Nintendo or anyone else that doesn’t lower the barrier to entry is going to begin to wither away into a niche that may eventually kill them. If you depend on AAA titles and then they fall flat (remember Lair ?) you’re in trouble.   To those who say, “quality control is essential to the platform,” I hand you one stack of shit Wii games and then point out that although hordes of garbage iPhone/XNA community games exists, nobody plays them and they don’t hurt the perception of any brand’s quality.

The same logic applies to MS and Sony. They need XNA Kinect integration ASAP and also need the PC SDK out there. If any device can benefit from community tinkering and accidental mechanic inventions it’s the Kinect. Sony has already pledged and XNA like Move SDK that bridges your PC to the PS3, but how easy it is to use and how much of it can be used to develop as PS3 game remains to be seen.

Sony and the Wii 2/Cafe  

I don’t have much to else to say about either of these. I think as much as I complained about Nintendo’s developer practices when speaking about the 3DS I think Nintendo may have learned their lesson in regards to the EA’s and the Ubisofts in the home console space. The Wii 2 will probably have its quirks, but will also fit in the development pipeline alongside the PS3 and 360 if you are a professional studio.

As for Sony I can’t see them doing anything that innovative with the Move or on the PS3, so if they do it will be the biggest surprise of E3 for me. To me the NGP is a device that the world doesn’t need or want despite its cool features. As for PS3 with MS going off on Kinect and the social gaming direction and with Nintendo potentially doing something crazy with Wii 2, Sony’s best move this year may be position itself as the best place to play the kind of games people from this site play. Developers seem to be coming around on the platform, the price is finally reasonable, and perhaps a hardcore focused PSN could fend off the clutter slowly accumulating over Xbox Live. 

3 Comments