I found the 3/28/2016 Oculus stream hard to watch. The problem is locking the camera to the head tracking. The person playing the game doesn't experience this, they are moving their eyes to compensate for their own head movement to stay locked on a visual object. I can only think of two possible solutions:

1. unlock the spectator camera from the head tracking. A fixed forward facing cockpit view for Eve Valkyrie would have been much easier to watch (but maybe boring)
2. Add a whole layer of eye motion compensation. It would need both hardware eye tracking and software to calculate the final spectator view, not a simple solution. And I don't know if this would really work, people scanning a scene with their eyes, or darting their eyes around, might be even harder to watch than the head motion.

In any case, I'm not looking forward to more head-tracking based VR streams.

Finally VR has highlighted that eyes exist and work independently but cooperatively with each other. I have had reservations about this for years for standard non VR first person games, where you are very much moving a head camera and no thought about the eye and head movement combination.

My day job is IT but in my spare time make music and play around with trying to create a different 1st person experience that bases direct movement on the eye direction and the head movement dynamically aligns to the eye direction.

I have a UE4 build that does demonstrate this and some older,not great videos on YouTube.

http://youtu.be/ij8eXXckRDs

I have been meaning to make update videos with vocal commentary explaining what is going on.

Using his eye movement and dynamic head works without VR headsets and has a "you have to try it to see the difference". What I would really love is for an already released games level assets (something like Bioshock Infinite, Gone Home or Alien Isolation for example) and use it with this simulation as it makes every environment nicer to explore, again this statement is vague and hard to explain.

I have always wanted to see what the streamers look at during their quicklooks VR or no VR.

Some of them got me kinda nauseous, well only one so far as i haven't finished the GB stream yet. It was with Dan because he moves around a lot in general.

Yeah eye tracking would be ideal, but I have to imagine that would be quite difficult to pull off. I mean kinect still has trouble sometimes tracking arms and legs and they have been working on that for a while.

This has already been solved with a couple of Vive launch games. They put in automatic smoothing for first person shots and player chosen camera angles. Not to mention mixed reality with a green screen which is also fantastic.

https://www.youtube.com/watch?v=-mrtlYjJRZ0

Eye tracking isn't that difficult. The New 3DS does it to keep the 3D stable. I have zero doubts that the next wave of headsets will use eye tracking for foveated rendering.

If GB can dedicate a 2.5mx2.5m space for the Vive stream it's going to be an amazing thing. I have a feeling they are going to go with the minimum though which can cause issues with quickly swinging controllers.

I guess I'm just immune to this stuff, watched the entirety of the stream with 0 problems whatsoever, I had no idea people could get nauseated by simple watching video footage.

I dont understand what the problem is. They said it was like wearing goggles. If you want to look to the side a little bit you can just use your eyes. It you want to look all the way around you have to turn your head. No need to track the eyes.

@flasaltine said:

It's not strictly necessary, but gaze tracking opens up a number of new rendering techniques like foveated rendering that could be incredibly important as VR matures.

I imagine the next generation of headsets may include headtracking. I can also see some pretty cool applications for it in video games. Like a horror game that knows which direction you're looking at, and can put objects in your periphery which will then disappear as soon as you look directly at them.

Eye tracking doesn't sound like it'd solve the spectator issue. I'm not even sure what eye tracking would do other than to help with stuff like tooltips and other things that could pop up depending on what the user is looking at, other than that the head tracking and a large enough field of vision should be perfectly sufficient for the user, as perspective doesn't shift noticeably when you move your eyes, just your head. You should be able to look at different parts of the rendered screen without noticeable issues.

For the spectator end of things there is image stabilizing software (famously used to show how some blurry and shaky bigfoot footage was just a guy in a suit) and something like that should probably be put in the pipeline when converting what the VR user sees into 2D for spectators to watch.

Worth noting that they're also gesturing and nodding/shaking their heads the same as they would in normal conversation, so a bunch of the time the head-bob was actually that. Eye tracking wouldn't fix that, and I'm not really sure how the GB dudes would fix it without becoming emotionless robots.

@teddie: Thats exactly the type of problem eyetracking would solve. Eyemovement is what keeps your vision stable when you talk and make gestures.

I don't know. (I should preface everything I say with that.) I think it might not be enough. Your eyes move really, really fast; and they dart all around to focus on different objects in a scene. The whole reason eye tracking is hard is because your eyes move so fast that sensors and computer graphics rendering can't keep up; the New 3DS, for instance, visibly switches as it tracks your eyes. If you can see the switch, there's too much latency; and it takes upwards of 120hz refresh rates to keep up with eye movement. Besides, just try watching any footage from existing eye tracking tech: imagine the camera moving around at -that- rate. It'd be nearly impossible to process.

I disagree with the point that perspective does not change much until you move your head. Put you head straight centre and don't move your eyes. Notice the position of object in your peripheral vision. Now keep your head still and look to the far right. Still keeping your head straight in the original direction and your eyes right is there not a significant change in the visible image?

A slightly different angle on things you saw before and all new objects that were not in the original view. A large change in visual information. These changes large and small go a long way to making your brain create the immersion of our environment.