Warning: I don't write much. At all. Ever. I'm sorry if this is difficult to read, if my thoughts are not conveyed in an efficient or understandable manner, or it just makes no god damned sense whatsoever. Before this is done it will probably be article length which will be a nice accomplishment for me - and a terrible, awful experience for you.

While watching the stream on May 21st revealing the Xbox One, there was some interesting revelations about what the machine was focusing on for the wider market. Lots of people were upset about the multimedia treatment the new console was getting. Even worse was that it turned out the machine would be running three operating systems - something that would clearly leech power away from the gaming side of it. The internet's run rampant about this subject.

The reason I'm writing this is because I've spent the last two years mired in virtual machines and hypervisors. I thought I'd take the chance to explain what they are, how the work, and talk a little bit about what it means for resource management.

The hell is a hypervisor?

The Xbox One is running three operating systems: One is for gaming, one is for the Windows kernel used for apps, and a third to bring them together. This all sounds pretty intensive. What you need to know is that third third operating system is a hyper-visor.

I'm sure some of you are familiar with programs such as VMware Workstation, Parallels Desktop, Oracle's Virtual Box, or Microsoft's Virtual-PC. These are considered type 2 (or Hosted) hypervisors. These are designed to interface with the operating system they are installed within. They then provide emulation of hardware for the virtual machine (or guest OS) to run on.

That isn't what the Xbox One uses. Instead, it uses a type 1 (Bare-Metal) hypervisor. This means it is an operating system designed to interface directly with hardware and provide an environment for the guest OS to work within, all while managing the resources available to those guest operating systems. Comparatively, this means much lower overhead, and the ability to allow operating systems to use actual hardware, not just emulated counterparts.

One of the major advantages of the hypervisor is isolation of operating systems and tasks. Often virtual machines are used in a sandbox environment because you can test programs without any risk of them interfering with the host or other guests on that machine. What this means for you is if the apps side crashes it won't do anything to the games side, and vice-versa. Another advantage is you can suspend the operating systems so that they pick up exactly where you left off. You can even do it individually. This is where we get into resource management.

What resources are needed for this?

Straight up, you're losing some of your memory and a couple CPU cycles to the hypervisor. Another bloated Microsoft operating system, right? Up front this sounds awful, but it's not as bad as you think. As I'm writing this I'm running Hyper-V 2012. Microsoft has been trying to get into the hypervisor business market for a while, and they've made headway by offering their hypervisor for free, you pay for support though. If nothing else, at least Microsoft has experience with building these things.

Disclaimer: Windows Hyper-V 2012 is designed to run a variety of operating systems. This means it has a larger foot print than the Xbox One's hypervisor will have. I need you to understand this before you click this spoiler block.

The hypervisor used on the Xbox One, while definitely based on the same technology, is not Hyper-V 2012. One of the things that has always been lauded in the console versus PC argument is that consoles are a closed environment. Each console is expected to be identical or at the very least have a minimum of resources. Microsoft (especially as a creator) knows exactly what's going into that machine. They can tailor the hypervisor to run those specific operating systems for games and apps. The foot print of the hypervisor will be much smaller than that of Hyper-V 2012.

Why on earth do you care about this?

The whole point behind everything above is to accurately express what resources are available to games. We've all seen the information stating that games are provided with 5 gigabytes of memory, 6 of the CPU cores, and supposedly 90% of the CPU cycles. That's going up against the PS4's 8 gigabytes, and a similar processor. Granted, the memory on the PS4 has almost twice the bandwidth, but there are other pieces in the Xbox One that make up for it. Still sounds like a losing hardware battle.

Here's the reality though. We have 8 gigabytes of memory. 5 of which are untouchable and only for games. A developer can always expect at the very least to have those 5 gigabytes. Let's assume that the hypervisor on this thing takes up half a gig of memory. That leaves 2.5 gigabytes available to the Windows kernel. It's not dedicated, but it has access to that 2.5. That doesn't mean your apps will always consume 5/16th of your Xbox One's memory.

The beauty of a hypervisor is we can shut off or suspend operating systems without any effect to others on that same machine. Got a game that needs to load up 6 gigabytes into memory? That's fine - the apps side won't even get in the way. How about 7 gigabytes? I think that's kind of outlandish but let's just suppose. If apps can't run within that last half gig of memory we can just suspend the whole thing and give the whole 7.5 gigabytes to the game.

That's it.

I should probably make it clear I'm assuming things about the Xbox One's hypervisor (I hope that's obvious) and this is all based off of knowledge of other hypervisors and how they work. I'm also not an authority on hypervisors, so if I'm wrong about something please clear it up. I'd rather people have real information than assume that I'm right about everything.

If you managed to read the whole thing, thank you. Or more likely, I'm sorry.

Great article! Hypervisors is just a fancy term for software virtual machines run on. There is nothing magical about this tech which has been around at least 2005ish (when I first ran into it) where desktop hardware has the support today. I haven't seen how Windows Hyper-V works but I have used Xen quite a bit and you wouldn't know the difference between it and the "native OS" each host unless you inspected system variables and states.

The way this is implemented for XB1 is actually a great technical example where abstractly designers wanted "a machine to run games" and "a machine to run apps" to be sperate and protected form flakiness in the other.

Why did they integrate these three OS's into single set of components rather than having a second/third smaller set of RAM or CPU/GPU just for the other sides of the console. Is it was simple as just meaning that if those aren't in use then the gaming side can use it? Because I fell like a more consistent set of components - even if, in instances when the other apps aren't in use, this cannot be increased - would be more useful (I know they have the minimum 6Gig for games, but I think they may have been better off saying 6 Gig for games and then 2 Gigs for the rest rather than getting in this issue over there promoted 8Gig not being wholly usable by games).

It is expensive to add "a second/third smaller set of RAM or CPU/GPU" and it takes up space.

There should be a law against using words like "hypervisor" unless they refer to truly mindblowing tech. Can you see through time with it? No? Then reign back the name a bit, ya bastards.

Seriously, though, thanks for the explanation. That was actually really well written and easy to understand.

@extomar: I only ran into virtual machines in 2008 or so. They've been around since the late 60's. Some serious IBM insanity back then. It wasn't until around 2005 that they started to pop up though.

@dixavd: Probably the number one consideration for "do I need a hypervisor" is if it saves you money. Instead of building 2 separate machines within the Xbox One they can support both the games and apps on one machine through the use of the hypervisor.

And thank you for saying this wasn't awful EXTomar and ShaggE. I'm going to go for a victory swim and hope an alligator gar doesn't bite my nipple.

Great explanation, thanks.

This is one of those cases where "old is new again" where you are completely correct that the idea has been around since the 1960 IBM mainframes but was relegated to very large, very complex, and very expensive systems. Now that Intel and AMD have added "vmx" support on chip it is inexpensive and easy to use and doesn't required an IBM technician to setup.

So yeah, "Mainframe" started to die off because you could just buy a truck load of "Microcomputers" for the same cost of a "Mainframe" but in the end the resiliency, redundancy and scaling of Mainframe tech has reappeared in PC.

@toxeia: If you have verifiable credentials, you should message the staff about this post. The y might want to edit it and throw it up on the site as a story. This is interesting and relevant info.

I can't wait to see the performance differences in the new consoles. I probably won't buy them but I'm still super curious.

@brendan: My only credential is a degree in computer networking and security, and the only source I used is my experiences with virtual machines and information from the Xbox reveal :(

Well, something more advanced than a 'supervisor' is a hypervisor. Its is just the way we construct new words in English. In the tech industry where 'metric levels' of functionality often are used, you find more prefix usage to show levels.

@toxeia: lol well I guess that might not work, but this piece should be seen by more people. Amongst all the fear without knowledge out there at the moment this is a bright spot of information

At the very least this should be noted by a mod and put in the community highlights

The use of a hypervisor was one of the few things the Xbox One reveal had me interested in from a technical side, since I've been working with VMware vCenter and ESXi hypervisors for a while.

Yes.

I was joking.

I'm glad you wrote that up. When I saw the thread title, I thought I was about to write a little missive to try and explain it myself.

Now, what I really want to know is what else other than background downloading can/is the little ARM chip in the PS4 being used for?

I keep pondering if it was beefy enough could it be running an Android based UI for the XMB and incorporate the Sony Android App store stuff they put on their Android phones.

The PS3 had a hypervisor before the OtherOS function was removed. It was used to keep Linux from accessing protected parts of the system.

Thanks for teaching me a thing or two. I use virtual machines at work from time to time, but I wasn't aware of the distinction between type 1 and 2 hypervisors.

@toxeia: Congrats; this post is the number two result (at least for me) on Google when searching for xbox hypervisor!

@fluxbit: Oh no, I don't like that at all... Now everyone will know I'm a shame. But at the same time, fuck you Neogaf.

Knowing that it's so high up there... maybe I should take the time to re-read and edit it. It's definitely "first draft" quality.

Thanks. That clears it up a bit.

My only question is will it bluescreen? When MS says 3 OSs, I think 3 bluescreens at the same time. Like 3D bluescreen.

I just hope X1 is stable.

The hypervisor is still there, it's part of the security architecture. Which is why it was immediately obvious the Xbone is using virtual machines - the machine must not be hackable via the inevitable app-side exploits (or game exploits for that matter).

Thanks for providing your take. It's hard to explain this to some gamers, and I hope more people understand this.

I'm not interested in the X1 yet, but I can certainly respect that it has a competitive design.

xbox 360 also had a hypervisor

I think a lot have missed one of the main advantages of this solution in that you can suspend an instance (Running Dead Rising 3) to the HDD and resume another one from HDD (Running Forza 5) game state in tack.

So when we say loading a game it will really be resuming an instance with the current game state where you were before

Ultravisor is going to be so awesome

So what exactly does this mean?

@the_laughing_man: Essentially, it just means that MS has systems in place to manage the overhead associated with running these multiple operating systems. It's a fairly standard system for managing virtualization (I'll be it with a decently wide variety in specific implementations).

I have some experience using Wind River's Hypervisor, and I can attest to how much it can really help maintain performance and security on embedded systems with hard real-time performance requirements. I've never used them on the scale of a device like the Xbox One, but I assume the hypervisor footprint/overhead probably isn't much bigger than what it is on the smaller embedded devices I've worked on, so that footprint-to-benefit ratio is probably pretty high assuming everything is optimized properly.

@chaser324: so I can still order pizza on X1

@the_laughing_man: I'm guessing that DominOS and P'xone will be added post-launch.

If "Xbox, Pizza" worked the war would be over

lol... wow how did the memory portion of this article get totally ignored. it's like everything with microsoft the good stuff is often ignored. if its not negative its nothing at all

A message on Twitter sent me. Oh, you're here already.

@re_kinect said:

Hi there, new poster. I'm not sure if you're an astroturfing bot (given how you're spreading your message via posts like this and in Twitter, replying to weeks old conversations when people bring up the 5GB quoted RAM figure - as happened to me and the people I was talking to on Twitter a week ago) or an enthusiastic fan of the new Xbox One hardware/software system.

Why PCs and consoles are not the same

The problem with the supposition in the original post, from my perspective (SoftEng but not an OS guy), is that is assumes that the game can take more and not have an issue. The 5GB point is well below the 7.5GB of RAM that the XboxOS + Windows has available to it (once you assume a 500MB block for the hypervisor, as you say I think this is more than where it actually will turn out to be on the dedicated spin they're making rather than the 2012 version for all x86 PCs) but that doesn't mean a game can just request more RAM from the VM system and get it. What if the snap-in apps (these are being displayed by the Xbox One right now so you can't throw their RAM to the HDD and resume them later, they are live) are using that 2.5GB of RAM? That is why these fixed platforms have these bounds, because you know you are guaranteed to have 5GB of RAM for you. The reason why PC games need more silicon to run at the same performance is because their code has to assume the RAM allocated isn't guaranteed, it could be paged out to the HDD when needed. Consoles are lean because that RAM is guaranteed (and if you need more then there may be a paging system but you also need to know you're about to hit it and be able to expect the performance death that comes when you need to load the page back in - titles that do streaming assets are managing their own 'paging' system, the articles about how Rage works with the levels of storage for the texture assets provides great primer material for those wishing to know more).

Why consoles have hard limits

A game could be developed (assuming the hypervisor allows it and doesn't enforce the 5GB limit) that uses 7.5GB. But what would happen if that was run on an Xbox One which used some snap-in app (that has every right to consider the 2.5GB of RAM it + Windows OS is being given the play with) that was eating the same RAM and because it is visible this isn't a case of saving it out to HDD and loading it back up (which would destroy any idea of instant switching MS have given users up to this point - which is why I think it is unlikely they will pass through certification and game that uses 7GB of RAM)? Then you would have two apps fighting to get accesses on this data that the OS was desperately paging in and out of the system depending on which bits were being accessed each frame. Both would have miserable performance is a realistic outcome, I would say especially on a machine with a slower RAM but the obvious limiting factor of the HDD where those paged RAM blocks were living is way, way slower and the factor that kills the performance (as in it takes several seconds to load 1GB of data from HDD to RAM due to the speed of reading from a HDD - even a sequential read that assumes no fragmentation or need to write out the pages being replaced).

I don't know the specifics of this console, I've not had a chance to play with it or get the NDA documents explaining how it works. But talking about generalities, I think if MS have given developers a 5GB line then they are going to play to that 5GB line because in the future app developers for the other OS are going to base their assumptions on how much RAM they can play with on the same rule book. This does mean that some RAM for most users will be sitting idly there storing no data (or more likely storing cached data for other apps that are not live so they can resume faster - just like modern Windows none of the RAM will actually be sitting empty it'll just be sitting there inactive with cached loads from when there was spare time to load it on when they didn't unload it from previous use).

@shivoa: It also can't be understated, look at what the consoles are getting out of 512mb of ram for the 360 and 256 for the ps3 now. 5gb is a huge chunk of ram compared to last generation so its going to be interesting to see what happens in the later years of the console, who knows 5gb may be so large they get lost finding things to put in it (extreme pipe dream I know.)

Hmmm interesting, I learned something new today! Thanks for sharing your knowledge and no, it wasn't difficult to read your write-up.

@dark: Oh definitely. These new consoles have lots of RAM, the muttering about the PS4 being a 4GB device until very recently isn't shocking and having both machines having more than that available purely for gaming (and unified) is great. The limitations of the current consoles (spin round in Rage and cry at the tradeoffs they made that result in that little gap that they can't keep the textures for the stuff behind you in the RAM; the endless advancements in streaming assets and subtly loading in the detail to cause less obvious jumps as you play that plagued earlier titles, especially early UE3) have developed some very nice standard designs that will make the next gen very healthy when it comes to providing large detailed worlds that load in before you get close enough to need it. Ok, so titles like WoW are getting very old and offer seamless worlds but there the texture density (and art choices that support it) help to keep the memory requirements lower but this gen of consoles has solidified streaming assets rather than load screens.

While 5GB isn't infinite, thanks to the compute power of the GPUs to thrash through the textures (with all the diffuse, spec, normal, etc used for a modern material system with physically based shading methods) and the expanding scope of what coders will use the RAM for to keep a world 'alive', it is certainly nothing to be sniffed at. 5GB is a good volume of RAM to have available (PCs will continue to mainly have 2GB or less of textures near the GPU and need to swap them for the 4-16GB of system RAM cache stuff) but 7GB is better (and that PS4 figure isn't officially confirmed is it?). Will games come out that actually need it? I'm going to bet for not, they might be slightly better at hiding any loads or having the most detailed surfaces close to the camera but I'm going to say Digital Foundry are going to have to work hard to find those differences caused by the available RAM (the RAM speed may have far more to do with stuff like that being a concern/easy pickings for comparison articles, not to mention 18 GCN blocks vs 12 to process the render at 1080p).

The problem is we're walking into this with the assumption that a second machine needed to be built at all. That I'm even typing this in response now means I have at my disposal already everything that will be included on the app side of the Xbox One. They've made a solution that's in search of a problem. It solves that problem for the one guy out there that loves to post on Twitter from his 360, but finds the whole process too slow for his liking, at the expense of resources for the games presumably everyone is buying the system for. Incidentally, fairly sure that guy isn't Major Nelson. It really should be. You'd think they'd give him a per-tweet bonus if he used the Twitter app on 360.

Glen's the man of the hour,

He's the king of his cube,

Status call reports have finally met their rival...

Burning the candle at both ends on his way to the top...

He knows one day he just could become, hypervisor!

@shivoa: I don't have those... uhh.. things on. Tells me when someone mentions me. I've been drinking, shut up. My thought on the being able to go beyond the 5GB 'hard limit' was that you can swap memory with virtual machines. As for those live apps needing memory as they're running, how much memory does that tv app need for listings, or skype need to keep track of who's online? If it's all in idle state, or not used at all it isn't an issue.

There's also that thing where I was trying to think of best-case scenario with all of the information coming out and everyone just shitting all over anything that was within a 50 mile radius of the Xbox One.

@onan: Agreed, that isn't needed. But Microsoft is clearly taking a jab at a larger entertainment market than just games. And instead of having app developers learn a new language to program for the OS running games, they can run an OS with the same language that Windows 8 apps use (which I believe can also implement Windows Phone 8 apps, but don't quote me on that.)

Back to drinking. I have a cleaner draft of this written up that I never added to that... maybe tomorrow I'll soberly fix it.

Great article/thread going on here. I built and manage a small VMWare cluster... 6 hosts, about 100 VMs. I love the nitty gritty details about how it all works.

What you guys are talking about with memory limits, etc... VMWare and Hyper-V call it "ballooning".

In VMWare they accomplish ballooning with an application running inside the "guest", which is the virtual machine running on the hypervisor. This little application answers the hypervisor's request to it, to ask the guest VM for some memory. That memory is then actually available to other VMs running on the same box.

The hypervisor monitors the difference between allocated and active memory. You allocate 2GB for a VM, but it may only be using 200MB of that most of the time, so the rest of the memory can usually be used for ballooning.

Now... there's a slight performance hit when you balloon, but you can mitigate the performance hit with really fast RAM. Good thing these new consoles have that!

There are lots of other great benefits to VMs and hypervisors I can see coming to consoles. Like maybe I have 2 XBOX One's and I want to stop playing in one room, and resume playing in another. Not with a game save and the 'cloud' or anything like that. The hypervisor can actually move the game and it's active state over to the other console on my LAN. Fringe functionality maybe, but something like this could be possible. It's extremely usable for businesses and their applications for a variety of reasons.

Virtualization is great for servers.

Consoles are still unproven territory but I'm afraid that we are giving up the advantage some console games have from using low level programming.

Thanks for the explanation and I'm excited Microsoft is trying something new.

Sounds like Microsoft PR.

Thanks for sharing a good hypervisor