On the last couple of bombcasts Brad has talked about the incremental improvements in game graphics and slightly lamented the loss of the experience of being presented technical feats that leave us awestruck. Having listened to Brad over the years it’s clear he and I had very parallel formative gaming years. We’re about the same age and grew up being exposed to the same games, technologies and gaming zeitgeist and I understand exactly where he is coming from. We were absolutely spoiled in the 90’s. I vividly remember the first time I saw Quake 2 at the local PC shop running at a resolution of 800x600 at 60fps on SLI Voodoo2 cards and being floored. Although at times I can step back and look at Uncharted 3, Halo 4, or Gears of War 3 and marvel at the technical complexity that is being pushed out of what is now comparatively low-end hardware, there is a lack of those “holy shit” moments in graphics compared to days past. Without going in to much detail I’d like to explain what’s left to do in graphics, why those “hot damn” moments are potentially gone forever in graphics, but also to present two places where I’m confident those moments have yet to occur.

Computer graphics, specifically the realistic representation of light transportation to our eyes, is a solved problem at the theoretical level. We know exactly how light bounces around the world and interacts with our eyes and it can neatly be summed up by a set of equations (BRDF, BTDF, BSSRDF). From here on out everything is engineering the implementation of those equations serving two purposes. Firstly, for mathematical reasons the above equations cannot be directly computed, but instead we sample and approximate them to a degree that we can call good enough for the intended purpose. The techniques involved in these approximations are still actively developed. The second engineering task is marrying the math to the available computing power. In games we need to push out a rendered frame at least thirty times a second and in the offline rendering world frames need to be rendered fast enough that directors can iterate over the content, that films come out on time, and the amount of electricity and computing resources used doesn't blow the budget. As computers become more powerful and our approximations of the math become robust enough for all intended purposes it isn't out of the question that all computer rendering will collapse to a unified set of practices across both real-time and offline rendering. This trend is already taking place: at this year’s Siggraph conference there is a clear trend next-gen gaming engines (including Unreal, Cryengine, and Unity) to the use of physically-based renders, where all of the shaders and material properties are constrained to being built with a fixed set of physically plausible parameters found in the above equations. The purpose is to give physical justification to the way each material interacts with light and in doing so not only provide higher quality rendering, but more importantly avoid some of the hackish tricks that can cause production pipeline issues. Meanwhile in the off-line film rendering space almost all techniques across all the major houses (ILM, WETA, Sony, and even Pixar) have collapsed down to similar single-pass ray/path tracing techniques. What was once a diverse set of techniques and approaches is slowly becoming more and more unified even between movies and games. (As a side note please don’t think this means everything will look the same, a common complaint between games that share the same middle-ware. Despite the unified theoretical approach the end results can stylistically look completely different)

More to the point of this article, back in the 90’s off-line rendering was evolving as much as real-time rendering and in both areas there was a vast undiscovered country with quality leap frogging itself year-over-year. We were equally blown away by the first liquid morphing T-1000 in Terminator 2 as we were the OpenGL graphics of Quake 2. Each successive Pixar film brought forward strides in rendering quality and complexity. Today the world is different. Although there are still quality improvements to be had, for the most part film visuals have plateaued be it the newest Pixar or the newest super hero movie. In a film like Avatar, the end game of real-time game rendering is before our eyes and we are just waiting on content production efficiency and computer rendering power to achieve parity.

Thankfully, I still see two areas ripe for “holy shit” moments. The first comes from the current hot kid on the block: Virtual Reality (VR). My day job is in a human factors/cognitive science/psychology lab at a University where I get exposed to lots of game related technology, including VR and advanced simulations. My quick summary of nearly all of VR up until 12 months ago: it sucked. All the things the Oculus folks mention in their pitch are entirely true: professional and consumer VR is a mess of really expensive, laggy, uncomfortable stuff. I don’t want to pile on to the already overblown hype that the Rift enjoys, but it really is a game changer. For me that title gets bestowed upon it because of how the cost and replaceable, widely available parts will allow for easy experimentation and iteration. You've heard all that praise before, so how exactly will this make our jaw drop? Well, again working at the lab I have exposure to perhaps a little more hardware than your average person and more importantly a lot of smart people who know both the engineering and perceptual psychology factors required to start making something great. Recently we've put together a demo that did two important things. Firstly, the software was tweaked to match the physiology of the person, namely the interpupillary distance (IPD) and height of the user was correctly modeled. Most things, like the infamous Rift roller coaster, are very generic in configuration: height, lens type, FOV, and IPD are not tweaked per user. Secondly we attached a set of ten position/orientation tracking sensors to limbs so that a full body tracking system could be implemented. In doing so you were able to look down in the game world and see your own body and move all your limbs as you would in the real world. It was far from perfect, there were the odd Kinect-style jerking twists and the lag wasn't yet invisible, but it was compelling… really, really this-is-totally-unlike-anything-before compelling. Just as the magazine and usenet text of yester-year couldn't convey the first time you saw really good accelerated 3D, the feeling you get interacting with the world in good VR is essentially ineffable in a way that no bump in 3D ever was for me. To be less poetic: reserve judgment on VR until you actually use a good sensor setup with content made specifically for it, because holy shit. Just as proper 3D polygonal graphics added a thick trunk to the genius of gaming experiences, so too will VR if it takes off. Further these basic position sensors and optics are just the beginning, there is much more open ground to provide sensory cues involving touch that have only started to be explored in the academic space. There is so much ground being covered in the near term its almost difficult to catch your breath and look that far ahead.

Another form of evidence to this end is the involvement of the developer community. In many ways the VR scene parallels the 90’s graphics scene. We have disruptive new hardware that's going to iterate yearly. We have a lot of academic material, in this case perceptive psychology, which we can draw on to point towards how to apply that hardware. We have a lot of technical and engineering hurdles to break, but each time we break new doors are going to open. Each of these doors could open new opportunities for mechanics and experiences and each of them has the opportunity to provide our sought after astonished impressions. It should be no surprise to see some of the legends from 90’s computer graphics such as Michael Abrash (inventor of graphics mode X, programmer on the original WindowsNT graphic stack, Quake programmer) and John Carmack (Wolfenstein, Doom, Quake) are jumping on board at the expense of doing further graphics research and development.

The second “jaw drop” is far more speculative and likely much further off. Game “AI” is essentially the cognitive equivalent of fish layered with beautiful animation, voice acing and scripted sequences on top. However, there is the chance that someday game “AI” reaches the point where we can hold conversations with it. Whether through trickery or through some kind of true “AI” the first time we say something using natural language, not through a button press, not constrained to some keywords or selected from small set of dialog choices, but the first time we speak our minds, have an avatar turn around and give us an on-the-fly, intelligent response is going to be jaw dropping. However, I’d imagine in some ways that by then the world will have changed by then. It’s only a slim prospect that gaming would be the first application of such a technology, so by the time it appears in games it may be common place elsewhere and thereby less impressive. More importantly unlike graphics, where we have hard physics and math, and perceptive psychology, where we have some hard knowledge, some general notions and much in progress work, when it comes to high-level cognition required to understand and generate the semantic content of language we are really, really in the dark. How far out such a technology would be, if even possible at all, is open… but just imagine!

The 90’s were amazing for videogames and as someone who loves the technology behind it all the leaps in graphics played a substantial role in my excitement and current nostalgia. I have such fond memories of it all and feel privileged to have lived through the experience of that progress. However, I have never been more excited about gaming and the technologies surrounding it than I am now. There are diverse reasons for this (the ease of opportunity for creators, the breadth of different experiences and outlets to encounter them, the prevailing culture), but for me jaw-dropping technical progress has not fallen off the list. Graphics still have a way to go and there will be a few spectacles left along the way, but the way we embody ourselves within game worlds is about to be provided with a new and exciting form and there is always the prospect of the citizens of those worlds interacting with us in ways that are almost unfathomable by today standards. There is still so much undiscovered country in the technology behind games and so much opportunity to be blown away by its progress.