Google Stadia - input lag, display lag, and VR

I'm curious what everyone's thoughts are on how Google Stadia will change how we think about input lag. If the controller inputs are streamed over WiFi to the Stadia servers, then it sends the video with your inputs to a display, does this negate some of the input/display lag? This could be a good thing for VR if they can also stream inputs from VR headsets and controllers. I'm hoping someone with more knowledge can shed some light on this.

Its going to be terrible in Australia for the foreseeable future so I hope it doesnt lead to a wave of developers making mushy feeling games to compensate for a system that I won't even use.

Digital fou dry clocked the lag at 166ms which is almost identical the lag on xbox one x for ass creed. But they also qualified this comparisson by saying that the game has unusually high input lag on xbone, but they also said that when it eventually runs at 60fps itll be less than that of the one x

I think there will be a lot of effort to minimize it, but there is already that effort

The main thing is: Most games are fairly input lag insensitive. You care. I care. We have our TVs and monitors set up to have as little postprocessing as possible. My sister and her husband? They have all that shit cranked up and don't notice.

For a twitch shooter, yeah, this is a problem. For a Call of Duty or even a Halo? Those games are already "mushy feeling". Same with Souls games. It is really the fighting games that are going to be impacted by this and even those have various solutions regarding buffering and the like.

That being said: I have no idea how any of this would negate input lag. Now instead of controller to console we have controller to receiver to router to data center to "console". Same with the resulting display. Instead of console to tv it is "console" to router to your basement to receiver to TV.

@gundato: Stadia’s controllers will connect directly to wifi so no receiver required.

I do wonder, however, if multiplayer games will break even on input lag. Since the game everyone’s playing is on the server farm there’s none of the p2p lag you get with multiplayer games currently.

@haneybd87: Fair enough. But I am pretty sure the distance from your router to the data center down the street (let alone a few states over) is the real cost

@gundato: Oh I’m sure but at least you get however many 10s of milliseconds the receiver would add.

Google's data center locations seem very widespread, which bodes well for the latency problem. It's a threshold though. The input lag will be worse than my 5ms local monitor, but probably better than a lot of TVs out there. As long as it is below the threshold where people notice, they will be fine.

I just wonder what the magic number is when people start to notice input lag. It's way easier to detect with a mouse compared to controller. If you've ever tried using a mouse on a high input lag projector, the cursor 'swims' behind where it is supposed to be. That was 200ms if I remember right (aka 2/10 of a second).

@pappafost: From playing 1 game I couldn’t feel any difference between it and on XB1 but AC isn’t exactly the type of game that needs that fast response time to feel good. Other types of games will be problematic, I think.

@cikame: only 5 mega bits :O

They way they presented the whole controller bit was fairly misleading. First of all the controller inputs are not streamed over WiFi, WiFi doesn't have that kinda range. What they were saying is that the controller connects through WiFi to your normal connection. Then they seemed to imply that that connection would be some sort of back channel directly to Google servers, it's not, it still gets processed like any other internet signal. What they actually said was that the game client you're operating will have a direct local network connection with the game server you're playing on (when playing multiplayer games) removing any sort of client-server latency. While they will provide fairly local servers that should minimize input lag the signal will still be subject to the whims of your normal internet connection.

That is unless they go the extra step where you end up setting up a separate internet connection directly to their server. But as far as I can tell they didn't mention anything like that, yet.

@cikame: I'm right there with ya in the slow lane...

My internet waffles somewhere between 40~ and 50~ down with anywhere from 2 to 10 up, whole thing on this is last year I tried to play Red Dead 2 by remote playing it to my PC (just cause really) and while it was what most would call playable there was still enough input lag that I wasnt willing to use it for more then 10 or so minutes. For me personally any amount of input lag is a hard pass.

I think the presentation overall was smoke and mirrors AC: Odyssey from the stream had massive input lag and hands on impressions are saying the same thing, Google hasn't solved anything the input lag is there and considerably noticable too.

For streaming to really work the games are going to need to be built with streaming in mind which I don't see happening anytime soon as its in a developers best interest to have their stuff everywhere, Google will need to get their checkbook out big time for that to happen.

@mikewhy: I went over to my brothers for vacation after he got a new TV, played some Destiny on it and everything felt off in comparison to my home setup. I later see a movie playing on it and can easily see he had the "soap opera" mode on, i switched it to game mode the next time i played some Destiny and played perfectly fine after that. He also plays games so im not sure how he didn't notice the input lag, other then "well that's how its always been so that's how it must play"

I used PlayStation Now when it was in beta, it was "good enough" and perfectly fine for the most part, but there was still a little lag and compression when compared to playing directly off my PS3. If i didn't already own a PS3 and a large portion of the streaming library i would consider paying a monthly/yearly fee even with some of those cons. But Stadia is different in that its hardware is not locked, which comes with some benefits over just streaming fixed console hardware over the internet. And as far as VR i probably wouldn't try that over internet. the input lag being one problem, and the 60fps Stadia is promoting when 80+ is the recommended for not getting motion sickness in VR is the other. End of the day as some have said its still streaming over the internet, so lag spikes and input latency are going to be a thing, its more of is it "good enough" like what the crew said during the stream? Do the benefits out way all the new cons streaming adds?

@mikewhy: Now people will have their TVs in heavy processing mode AND have the extra input lag from streaming over the internet. I only tried project stream on my Gsync monitor so there was minimal input lag there but I can see how things would be extremely noticeable on a TV not in game mode.

A few points about VR stuff: the generally accepted number for motion-to-photon latency in VR is 20ms. Basically, between you turning your head, to light from the headset display hitting your retinas and showing you that your viewpoint has moved, anything more than 20 milliseconds runs the risk of making you sick; any streaming-based solution is going to be an order of magnitude too slow. To get convincing, smooth movement, you also have to render somewhere between 60-90 FPS minimum, depending on what reconstruction techniques you're using, which will chew bandwidth. Plus, you have to render everything twice, and any disagreement between those viewpoints is going to be glaringly obvious, if not upsetting: even minor compression artifacts will stand out like a sore thumb because they won't be consistent across both eyes. Two HD viewpoints will also punish bandwidth even further.

Basically, they'll have to break a few laws of physics (or do a lot more client-side processing) to make streaming VR even a remote possibility. That said, for 2D games, there's some promise in the concept.

I'm from Australia so my opinion doesn't count, and may never given our internet.

But, if it works... and is sensibly priced. And there's rock solid guarantees about being able to access the shit I bought... Sure why not. Curious if it's going to be a store front or a paid service though.

@moondogg: Good luck on rock solid guarantees is all I will say to that.

My biggest concern is availability in my region. Since when is 17 out of 50 European countries "most of Europe"?

That said, Digital Foundry's analysis shows a decent input lag, seemingly not significantly worse than native gameplay with v-sync turned on. Although it was conducted on Google's own network. I'd imagine if they do bring it to my country, the ping shouldn't be too bad though.

VR is completely out of the question. We need less than 20ms latency for VR and you don't even get that with regular flat gaming.

Overall, the input lag will definitely be noticeable to anyone who has played with v-sync off and is sensitive to that kind of stuff and it will affect your experience where input lag matters. But honestly, people will still play competitive games locally. Rocket League, Counter-Strike, DotA, LoL, that kind of games will still be played locally as long as we don't get like 1ms latency streaming, which won't happen any time soon. But for games like Assassin's Creed, the v-sync/streaming level of input latency is almost completely unnoticeable, especially with a gamepad.

I'm a bit surprised by the low reply count on this thread.

While watching the coverage I had more deja-vu to tablets than to anything more groundbreaking or "disruptive" which seems to be the current buzzword regarding this. I remember back then being super worried that tablets would take over desktop computing, but for me it's not happened and I don't think Stadia will change much.

Some technical thoughts I had during the press conference:

• Isn't the US riddled with data caps? Sure there's google fiber, but not everyone has access to that.
• The way they are talking about assigning GPUs per streaming instance to basically one person sounds horribly inefficient. It is more efficient overall than your averge joe running a standalone GPU which only gets a few hours of use a day. However it supposedly being housed in "edge nodes", that means that those GPU farms will see increased usage during the local regular gaming hours.
• On the flip side I guess they wouldn't go to waste as the rest of the GPU compute time would be sold off to GCP (Google Clould Platform) customers which shouldn't be that latency dependent and could be from anywhere around the world.
• There was no talk about any kind of special encoding/decoding so I guess most of the cpu/gpu found in phones and computers in the past few years should be enough. Still there could be a custom ASIC (application-specific integrated circuit) later down the line that is "Stadia Optimized" which they might also sub-license and or make exclusive to the Pixel line of smartphones for a few months to push those hardware sales.
• Their main argument is that you don't stream from remote servers, but from their geographically closer edge nodes. Eh, sure, your ISP is probably part of a IX (Internet Exchange), but it's not like you directly are. There's also business customers you contend with there. Unless of course this goes back into better paying ISP getting more "Stadia Optimized" access that they then sell to you. See also "net neutrality". I don't know if IX connections are even subject to data discrimination laws.
• They again didn't talk too much what their cheat way to do is. I was thinking, maybe something like a weird WAN accelerator over UDP that could circumvent some nasty routing inequalities some ISPs have.
• Another thing I was expecting was more cheating with deep learning. That art style transpose tool seemed to me more like something they would come up with. That they don't run all the GPU processes, but just try to guess the next image. I mean they might still do that. Run everything at 30 fps and deep learn every other frame. Maybe even a quarter of that.
• Tied to that, most intros and most gameplay frames will probably be matched against each other across sessions, which would mean there could be a lot of computing efficiently. What I mean especially for intros you wouldn't get your rendered cutscene, but a playback of the one someone else got a few weeks ago and heck from when you press the "play" button, that intro is already getting streamed and cached in your browser. But yeah even gameplay could be very effectively be compressed, provided you don't stray too far from the path most people take. If you do you might experience some jitter because you're the asshole who decided to try and fuck around.
• This last point is more about basically caching and saving a lot of frames and not having to render everything ever time for every user, which would be how I envision them getting the most out of their GPU farms. This would of course mean developers being incentivized to make games that stream well. Something like MGS4 could be very low cost then.
• All those things combined could make it so that games get this weird deep fake vibe to them, in addition to the streaming artifacts.
• To be fair to people like John Carmack, most gamers are used to something else as far latency goes, but this goes back to my initial argument that this is disruptive for the casual gamers and not the core. It would be funny however considering the push for adaptive sync (FreeSync and recently G-Sync)
• Google also doesn't live in a vacuum. Considering the competition's tech, I can't see how this isn't compromised from the start.
• Now that I think about tech though, does this mean that it's built on dxvk (a Vulkan-based translation layer for DXGI, D3D10 and D3D11), which would mean that it's probably running on Linux instances? I'm sure Google has a generous bulk discount for Windows licenses from Microsoft, but that could bleed back into gaming and Linux.

Non-technical thoughts:

• More than being about game playing, it feels more like a move to get Twitch users back too whatever that other livestreaming service they had was called (I seriously can't find the name of it because of all the Stadia news)
• Adding on top of that since Twitch is owned my Amazon it does seem even more a sensible business move to try to compete on that front.
• However unlike Twitch, if Google runs Stadia like a core Google product or an offshoot of GCP and doesn't let it be it's own thing they will end up with another Google+ instead of an early Youtube. What I mean is that they have to let this thing be weird in a way that the internet and especially gaming tends to be.
• Not just being a dumpster for Twitch archives might also be something they want. However, can you blame streamers who struggle with demonetization?
• The thing that seems most gross is thinking of Youtube having even more of a say on what gets streamed. I am not some conspiracy nut who wants to hear jokes about race, disabilities or borderline illegal pornography, but Youtube has a clear stance of what they think their userbase should be and their bias towards positivity leaves out those who want or have to push the envelope.

Sorry for the wall of text. Bottom line for me personally is that I might try it at some point, but I am against subscription services. I am however not infallible and a creature of comfort, but I do not currently see how it'll be hard to sit this one out.

Google with Stadia is targeting the casual crowd who don't spend a lot, tend to have lower bandwith broadband packages with data caps, most likely using wifi trying to play a game in movie mode in an enviroment with lots of devices connected at once. First impressions for a lot of normal folk could be quite horrific.

@an_ancient: In regard to video encodes I could see that being somewhat of a problem. For example when YouTube implemented VP9 many devices couldn’t decode VP9 and were never updated to because that’s up to the device manufacturer. So I could see some people getting a poorer compression method when streaming Stadia because of issues like this.

People have been crying about it endlessly. Just watch any video on youtube about anyone trying to discuss this and the comments are filled with everyone crying that it's DOA and that it's going to kill the industry. I say good, the games industry needs some disruption with how gross and abusive it business has gotten in the last 15 years. Most my movies and games I already buy digitally anyway since I really just do not have the room and it's just far more convenient for me.

As for physical releases, they are not going anywhere. Look at CD's and DVD's. We have Blu-ray and 4k releases yet still we get DVD releases. The blu-ray market is not nearly as successful as DVD was yet, just about every modern film gets a physical blu-ray. Those markets are not nearly as big as they used to be and with streaming videos and music is everywhere. Still, You can buy any new physical release you want be it video or music. Hell, there are rumors a new Ipod Touch of all things is coming. If CD's have yet to die from itunes back in the day and streaming now, your games will be fine when streaming video games become the norm.

The Google WiFi controller is a fascinating idea. It seems like the best way to skip a huge set of problems. By doing only WiFi they are skipping the Bluetooth latency problem entirely. They are side-stepping all the traditional problems of connecting a controller to phones and tablets. Just DON'T even try to connect it. I don't know if they can pull this off but just think of the sheer AUDACITY of it. A WiFi internet controller?!?

Wasn't Fiber's biggest issue the number of lawsuits from AT&T/Comcast/Warner?

Also, while Google have a huge track record for cancelling services, have they ever cancelled anything that people pay for?

As far as data caps...well the United States is a big country over a big geographics area. There are indeed data caps on some services. Some have data caps but the caps is in the hundred of gigabytes range, and some have data caps as low as 5GB, much like many cell phone 3G and 4G services. Therefore in the US, some companies have no caps, some have caps so big most people don't go over, and some have harsh data caps. There are currently 2650 Internet service providers in the United States: 895 DSL “Digital Subscriber Line” providers; 222 Copper providers (Business T1/T3 connections, etc); and 450 Cable companies. Of those there are 203 providers with data caps tracked in this database: BroadbandNow database.

So, given the wide geography and over 2600 providers it hard to compare any country's Internet to the United States. If anything the United States is very good at making sure there are "haves" and "have nots" in our nation. :-( With that said, most people in the US like 90% have access to "middling internet" that could run Stadia perfectly fine at the 1080p level.

This was my initial thought also, but I realized there is a relatively trivial technical solution - just render and transmit a video with a higher resolution larger than the viewport, the headset can immediately move the viewport within the larger area as the user's head starts to move. At that point the only limitation is hardware, they could render the entire sphere around the user and completely eliminate it latency, though obviously bandwidth and resource consumption would be prohibitive.

As a non-game developer it took me about 30-seconds to come up with the idea after originally hearing the VR reference and panning it, so I assume it isn't a novel one to people in the industry.

@luthair: VR needs more than just rotational movement. Your solution only solves rotational movement. Head translation still needs new frames to be rendered and that's where the motion sickness would heavily set in, instantaneously, if there were to be that kind of latency.

You can begin to emulate this with lens effects, zooming the viewport in/out and slightly warping the image (e.g. trapazoids). Recall that we aren't trying to get this perfect, we're trying to trick the brain to until the rendered image catches up. Sure, you won't be able to do this for someone whipping themselves around like a madman, but for typical users, typical games, and normal latency it might be enough.

@luthair: Yeah, it's not a novel idea. But it's a good idea, which is why practically every headset does something similar. (you can look up "Timewarp") It's not as exactly as you describe: they don't bother rendering larger image: they reproject the final image in the direction of where your old head orientation was when the frame was rendered, and stretch out the edges to fill the gap -which you don't really notice in practice because they happen around the periphery. This actually cuts out a lot of apparent lag and VR sickness, so it will work well with streaming too, since reprojection will be done client-side.

Translational movements aren't so simple but A) the perceived lag isn't as bad as rotational and B) humans are slow to move and (more importantly) slow to change, so predictive algorithms might close the gap (afaik sensors in current gen headsets already employ a bunch of predictions - they need extend it out more). Whether streaming latency is small enough and consistent enough that the predictions will be useful, I don't know...

Basically what goosemunch said. We already have to use all these methods to get the latency low enough when playing locally. Doing all of that over streaming just cannot be possible yet. Maybe if 5G is as incredible as people say, we might get there.

I am very very very excited for this technology. I will be able to play high end games on my shitty work computer while on breaks. Ill never really have to upgrade my PC again. The savings will be great and being able to play the game on all my devices is going to be awesome. As far as data caps its basically been proven it will use less data to stream the game than to download the game and the updates.