Brian Kardell: Okay. Hi, I'm Brian Kardell. I'm developer advocate at Igalia.
Eric Meyer: And I'm Eric Meyer, also a developer advocate at Igalia. And we have a special guest today. Guest, please introduce yourself.
Dhruv Mark Collins: Hey, I'm Dhruv Mark Collins, a developer at the graphics team here at Igalia. I've worked on Mesa 3D Turnip, the Vulkan driver powering the Steam Frame for the past three years.
Eric Meyer: Yeah. So that's what we're going to talk about today. A little while back as we recorded this, not too long ago, but a little while back, Valve, the video game company, surprised the industry by saying we're building three things, the controller, which is a little handheld controller, the Steam Machine, which is, I guess like a console, like a desktop dedicated gaming PC, and the Steam Frame, which is a VR headset.
Dhruv Mark Collins: Yeah. And I think they're not calling it the Steam Machine anymore, they're calling it the Gabe Cube, I believe.
Eric Meyer: Oh, are they? Okay.
Dhruv Mark Collins: Yeah.
Eric Meyer: That's awesome.
Brian Kardell: The first one is like a Switch, it's like a Nintendo Switch style.
Eric Meyer: I think it's just a handheld controller.
Dhruv Mark Collins: It's just a controller, yeah. What you're thinking of is probably the Steam Deck, which we've also done a bunch of work on. Yes. We've done a ton of work around that in our internal Valve project.
Brian Kardell: I'm curious about the controller, what is the ... I knew that there was this big announcement. And Aribo was centered on the little cube PC thing and the VR thing. And I didn't hear any talk about the controller. So just catch me up.
Eric Meyer: I mean, because it's a handheld controller like we get for an Xbox or a PlayStation or whatever. We didn't really do any work on it because there wasn't any work for us to do.
Dhruv Mark Collins: Yes, it's just a pure hardware thing. Basically, people really loved how the controller on the Steam Deck was. And so they thought, 'Hey, let's make this a dedicated controller.' And that's what we have for that.
Eric Meyer: Nice. Okay.
Brian Kardell: I guess that's why it made the least amount of noise because there's not a lot to it, I guess.
Dhruv Mark Collins: Yep. Not as flashy as the other things, just good old reliable.
Eric Meyer: But reliable is good in the space. So the Steam Frame, you said we here at Igalia did a lot of work on.
Dhruv Mark Collins: Yes, we've done a ton of work.
Eric Meyer: I know it's a VR headset but what-
Dhruv Mark Collins: What makes it special?
Eric Meyer: Yeah. What makes it special as a VR headset because there are not a ton of VR headsets, but there are others. So what's special about this one.
Brian Kardell: Is it tethered? Is it in the vein of the Oculus and the PICO full headset or is it more glasses-y like XREAL?
Dhruv Mark Collins: That's a good question actually. It's wireless. And this is actually a major change from what Valve did earlier. Valve's previous VR headset, the Valve Index was wired, you had to connect longer cable to your PC. And your PC would do all of the rendering and send over just a display signal over to the headset. That's what you see. This is something that Meta actually changed the landscape with, with the Meta Quest, where the headsets themselves would be able to render everything. They track the whole environment around them using a bunch of cameras. And this is essentially Valve's take on the concept. Having nailing down fully wireless, it can connect to your PC and stream things from your PC over WiFi. They have a whole thing with dongles just for this. And using some fancy stuff, especially around eye tracking, they've made it much more efficient than any past iteration of this. There's a technology they introduced called foveated streaming. So where you look at in the frame, it actually changes it. It tweaks the compression algorithms to focus on giving you a lot of bit rate on those areas, but not as much outside it. So a bunch of innovations that they've done here to ensure the wireless streaming from your PCN works great. But in addition to that, and this is a lot of what we've worked on, is getting the Steam Frame itself running games just on the headset alone, doesn't need anything externally. And this sets it apart from something like the Quest. On the Quest, you do have standalone games, but they're made for the Quest itself. You can't play any regular PC game on it. But with the Steam Frame, you actually can. You can run all your Steam library that you have that you could earlier play on something like the Steam Deck, you can just pick that up and download a game. A lot of games are compatible, it's not 100%, but you can download it and you can play it in VR. It just projects to a screen in front of you, a virtual screen.
Eric Meyer: Okay. So you can virtually, in the virtual environment, have as big a display quote unquote as you want, a virtual display to put up whatever the kids are playing?
Dhruv Mark Collins: That's right. You pretty much can.
Eric Meyer: So it's like you're playing at a desktop except virtually. So you can be lying on your couch and play Balatro or whatever.
Dhruv Mark Collins: Yeah, you can just, I don't know, be out and about and just be playing something. You have this huge screen in front of you, which might be a bit impractical if you're, I don't know, in a plane or some other form of transport and you just want to sit back and play something. The Deck existed for that, but that's a pretty small screen. Here, you can just have something, a huge screen in front of you, which you otherwise just wouldn't be able to have.
Brian Kardell: Yeah, this is similar in a way to the thing that you're describing, you can do this with the XREAL glasses except they're tethered. I just used them as an example, there are several other ones that are better or worse in other dimensions, it's just I think those are the most popular that people currently know. They had a lot of marketing, I think.
Dhruv Mark Collins: That's right.
Brian Kardell: And you can do the same thing on the Quest in terms of 2D versus really immersive stuff, you can watch Netflix on the Quest, which is amazing. You can do the same thing with the other glasses, they look more like Meta glasses, just like a pair of sunglasses. And I think with the Quest, maybe, I know there have been other attempts at doing this and probably will be with the Android XR, you'll be able to do pretty much any Android thing. So they would be similar though, they would be like a 2D projection in 3D space.
Dhruv Mark Collins: That's correct, yep.
Brian Kardell: So they also support really immersive things with this too, the Steam store does have things like Blade & Sorcery, which is a super fun hack and slash dungeon game.
Dhruv Mark Collins: All of the VR standalone titles are also playable.
Brian Kardell: I guess for me, that's what makes it extra fun is to put yourself in it, that's to me what makes it really worth putting on a helmet, to be immersed. Otherwise, if it's just a big screen, it's not the most comfortable form factor maybe.
Dhruv Mark Collins: Yeah, I think so, we're trying to enable both things, right?
Brian Kardell: Right, right, right. It's really cool that you can do both. And I was just wondering, do we use the same approaches and technologies for both or is the work that Igalia did more on one side or the other?
Dhruv Mark Collins: It's actually for both, because if you have a standalone VR game or a desktop game, both of them are going to use things like the GPU driver, sometimes also Fex, which is this translation layer that allows us to run X86 games on ARM.
Brian Kardell: Sorry, did you say Fex, like F-E-X?
Dhruv Mark Collins: Yes, Fex, yeah.
Brian Kardell: Okay. Can you explain that again?
Dhruv Mark Collins: So the important part about what changes going from the earlier Valve Index style headset to this is that with a standalone VR headset, you have a system like the chip that drives it, the CPU in there specifically, it's not a X86 CPU, right?
Eric Meyer: By X86, you mean Intel basically?
Dhruv Mark Collins: Yeah, Intel and later AMD, both of them. So these are the instructions, sot basically you could consider them the language that the CPU understands. And that's not the same with the Steam Frame. The Steam Frame uses a completely different instruction set called 64. And now if you try to run all of these games, which are compiled for understanding like written in X64 binary, and if you try to run them on an ARM CPU, it just doesn't know what that is, won't work at all. So you need to translate between them. And that's exactly what the Fex, F-E-X, project does. And we've been lucky enough to work on this. It's a project that's been running for a long time actually. I think it started around 2018. And we joined in somewhere last year on helping on fixing a bunch of bugs. Because, it's actually a bit funny, I was talking to Paulo Matos from the compilers team at Igalia about his experiences with this, and they've been fun. He played this game called Psychonauts. He actually has 28 hours on it. Except all 28 hours and the first three minutes because he was debugging this bug there. And he just had to keep running the game again and again to figure out what was wrong. Because how bugs work in this translation land is that you have an X86 instruction to do some kind of math, it could be slightly off on how ARM treats it. And if your ARM CPU does that math a bit differently, floating-point math or something, a number that's like 0.001 ends up being 0.0012, slight difference. But if your game's physics engine, it depends on one answer over the other, all of a sudden your game explodes. And those bugs are what he's been tracking down, crashes, a bunch of other things. But this is a core part of what enables a lot of being able to just pick up your existing Steam library, pick up a bunch of games from them, like you mentioned, Blade & Sorcery, that's an X86 game. And you can just, without the game developers needing to recompile for the new headset, you can just play the old game just fine.
Eric Meyer: Yeah. And that seems really smart to me. And almost like pointing a way forward. Because Valve, the Steam store, which Valve runs, from what I can tell, is basically the biggest game store in computer gaming.
Dhruv Mark Collins: Yes.
Eric Meyer: It's not even close. They're huge. And basically almost any game that comes out comes out on Steam, except for a very few, I guess, game developers who just refuse. But 99.9% of games are released into the Steam library. So Valve could have said to all these developers, 'Well, this is the CPU that we're using, and if you want to reach these users, then recompile your game.' But they didn't do that. They invested the work, they invested in the work of basically fixing what are open source projects, but making them more capable. So that they could instead say to the users, 'If you've got stuff in your library, it'll just work.'
Dhruv Mark Collins: Absolutely.
Eric Meyer: That seems really smart to me because it immediately, rather than having people who might want to get a Steam Frame asking, 'Okay, well, which of my games are compatible with it?' Just to be able to say to them basically all your games are compatible or 99.9% of your games will be compatible with it.
Dhruv Mark Collins: Yeah, anything that runs on hardware.
Eric Meyer: And that ability to translate between CPUs or instruction sets, that seems like a really smart thing for more than just Valve to do. It feels like this is something that the wider industry beyond gaming should maybe start aspiring to.
Dhruv Mark Collins: Yeah. And this is something that the wider industry has done to some extent. If you look at Apple, they worked on Rosetta too with their new MacBooks. Well, not quite new anymore. But when they introduced that, all of their existing software was on X86, but how do you run that whole old bunch of software on these new MacBooks? And that's where Rosetta 2 stepped in. It's a pretty good translator. Similarly, Microsoft did this for Windows on ARM. And they're shipping these Snapdragon laptops, and what they do is just, again, translate from X86 to ... And that allows you to run all these old X86 Windows applications. But until now, there was no way to do it on Linux in an open source way. All of the solutions were closed source. And I think that's the real innovation with what Valve is doing here. They're making it open. And Fex is really good at doing this in a performant way, especially I tested a lot on games, which might be somewhat less of a priority with the other ones.
Brian Kardell: I guess the Steam store has currently something like 240,000 games in it, a giant, giant catalog of things that you can get in there. I think that's great for game developers too, you have just a massive distribution channel that way.
Dhruv Mark Collins: Yeah, it's perfect, yeah. Because when you look at other VR headsets here, which have done similar things, like the Quest 3, everything that has to be run on the Quest, every game, has to be manually ported over by the developers. So this is novel for the standalone VR space. Every game that's on the Quest ... and this does have some advantages, if game developers tailor the game to your specific hardware, there is some amount of changes they can make to rendering and just working well on that grade of hardware to make it a much more polished experience for the end user. But there's the disadvantage of, if the game developer doesn't do that effort, you can't play it at all. And so what Valve allows here actually is, so if certain developers want to make that effort, they can, and then they can ship native games that are compiled with ARM in mind that are probably tested on the headset with this GPU and have all of these optimized rendering pipelines. But if you don't want to, and this is where you can just rely on Fex, Fex will translate things over, and so you can run these games which are initially meant for running on PCs on the Steam Frame without any ... Hopefully the experience might vary how good it can be from game to game. Some games are very, very demanding. And just because of pure physics reasons, you might not be able ... These games are made for running on dedicated GPUs, there's only so much power within a VR headset that's strapped to your face and has a limited thermal headroom and all of these other constraints because of that. But a good bit of stuff you can just run directly without the developers needing to step in and optimize their game specifically for the headset. It also makes optimizing the games much easier because if you have a game that you'd need to start from level zero of, okay, I need to get this running on the headset at all versus, okay, it already runs, what are the minimum tweaks I can do to make it run reasonably well? Even if you don't want to go down the whole path of recompiling it and everything for ARM. You can just maybe throw in a couple of tweaks.
Eric Meyer: Yeah. And from what I understood from talking to people within Igalia about this, one of the biggest challenges in some ways is that a lot of the errors that can happen here are not crash errors, they're rendering errors. Like you said, you were talking earlier about how if the physics engine depends on something the game just blows up or maybe crashes. But it can be more subtle than that. It can be just the way that, what are they called, shaders, rendering shaders are calculated so that ... the game is running, as far as the game can tell, everything's fine, but the visual output is maybe glitched or just completely indecipherable. And how do you test for that?
Dhruv Mark Collins: So with that actually, the physics error is a great example of a case. So in that case, the game is not crashing. If your physics engine is wonky all of a sudden, because the way it expects things to behave, the math to behave at a very specific level in the CPU, doesn't behave that way, then you're not seeing a crash. You're seeing your car float up into the sky in a driving game or something like that.
Brian Kardell: I hate it when that happens in real life.
Dhruv Mark Collins: Right. Yeah. And so you're just seeing all of these crazy things happen. And they're really hard to catch because you're not getting a crash, okay, here at this instruction specifically, this is where things go wrong. No. You have to spend those 29 hours to figure out where exactly in this whole ... all of these instructions, we're talking millions of instructions, has one been slightly mistranslated where the math operation it does gives you a slightly different answer and things just blow up because of that. And that's what makes this much harder to debug compared to a more controlled environment that you might see in a lot of other areas of tech.
Brian Kardell: How does that happen even? Is there a difference in ...
Dhruv Mark Collins: The math thing is generally around something like floating points.
Brian Kardell: I was just going to say, it's going to be around floating points.
Dhruv Mark Collins: Yeah. So ARM CPUs and X86 CPUs have some disagreements on how to do some of the very niche behaviors of rounding and other things. And so you can go the fast way of let me translate this X86 operation, that floating point operation to an ARM one, and it does mostly the same thing. In 90% of cases the answer will be the same on both. But there will be 10% of cases where the answer's going to be not what it's expecting, it'll be wrong in some generally small way. But those small ways can end up being big depending on what the software is trying to do, if it's just minor rounding behavior, if it doesn't strictly follow what the software expected it to.
Brian Kardell: So it could be, could it result in less than perfect rendering, like you get blurry edges, refractions aren't right?
Dhruv Mark Collins: There's two domains here. Fex operates on a CPU domain generally speaking. So this is around everything that the game logic is trying to do with like setting up things, where are things in the world and everything. And then there's the GPU domain where it's actually the CPU assembles a list of commands of like, here's how you render. And the GPU's job is to follow those commands and actually do the rendering. And so if there's an error in the way it has construed these commands, then the GPU might render something incorrectly, and that would be the CPU's fault. And that's one source of rendering bugs. Where the CPU itself is somehow giving the GPU wrong instructions. But the other end, and this is also common, and this is what we've tackled a lot in Mesa 3D Turnip, is when your GPU driver, which is responsible for running the GPU, it's basically the layer that the API that the CPU, the game itself deals with here for rendering would generally be something like DirectX, which we translate over using a DXVK into Vulkan. And Vulkan is where Mesa 3D Turnip is an implementation, it's a Vulkan driver, so it's an implementation of the Vulkan API essentially. And so if the Vulkan specification expects you to do a certain API operation a certain way, and you're just a slight bit off from that, you can get mis-rendering from that. Because in the game, the command might say to do a certain operation one way and your driver ends up doing it slightly differently because we didn't account for some edge case somewhere. And you end up with completely broken rendering because of that. So in the GPU domain that's how things break. Where the CPU is putting things together perfectly, but the GPU driver has some fault because there's a whole bunch of stuff involved there in terms of commands. You mentioned shaders earlier, shaders are basically just programs for the GPU to run. And these are defined in a Fex language called SpareV. It's like on the side of the Vulkan specification, both specifications are run by the Khronos Group, which we participate in. And it basically tells the driver that this is what the GPU program that you create should do. And there again, you have a bunch of things like math, and that math can be slightly wrong, and you just end up with all of these weird rendering bugs. Similarly, because these GPUs are massively parallel, you can have synchronization bugs very often. Where it turns out you're supposed to execute all these commands and the game maybe expected, according to the Vulkan API, it could assume a certain workload was done by now, but it turns out our driver doesn't handle it actually being done by then. And synchronizations often now you're reading invalid data for the next step. And all of these issues just pop up. And this is especially prominent on Qualcomm's proprietary driver earlier on. And if you run their driver on a bunch of Windows games especially, because it's just not tested against these things, they would have a lot of bugs. It's very hard to get a driver that implements the Vulkan specification perfectly, gets all of these VR edge cases completely correct. But that's what I've been working on along with the rest of the team at Igalia to just nail down the perfect Vulkan driver.
Eric Meyer: People often say, 'Well, there's a specification, right?' There's that gap between what people wrote down in a document and actual code.
Dhruv Mark Collins: Yeah. Some of this stuff gets very specific.
Brian Kardell: I have a couple of questions here. So one, is this like, what you're describing, like Hyrum's law, where they wrote the program with one thing in mind, and that thing had a bug, and when they did their testing they worked it out so that it worked perfectly on that.
Dhruv Mark Collins: Yes.
Brian Kardell: And so they have indirectly become reliant on a bug effectively.
Dhruv Mark Collins: A bug, yes.
Brian Kardell: Yeah. So yeah, I love Hyrum's law.
Dhruv Mark Collins: Yes. In a lot of cases, actually with X86 instructions too you see this, where they really depend on something, like that X86 specification actually, the floating point specification, especially earlier on, had a bunch of shortcuts of convenient answers that would be faster to compute, but not necessarily correct, my understanding is. And so all of this code that's built to run on X86 CPUs might want the wrong result in those cases because they've been designed to run on those CPUs. And so you have to be effectively wrong about those things intentionally to emulate them correctly, to ensure those applications run correctly.
Brian Kardell: So my other question is, you mentioned earlier that one of our colleagues, like you said, spent, I don't know, an inordinately long time, I can't remember what you said-
Dhruv Mark Collins: Yeah, 29 hours.
Brian Kardell: ... 29 hours or something like that on playing the first three minutes of a game over and over and over and over again. I'm just thinking about the things that you're saying and you're talking about a glitch in physics rendering, which I take would affect effectively calculation of your hit boxes and lots and lots of subtle things that would be really hard to notice. How do we even discover that there are those kinds of problems to debug them in the first place?
Dhruv Mark Collins: Just trying games out. You just try a game out. We just go through the library and someone reports it, 'Hey, this game's broken.' And guess who has to fix it? Then someone has to take a look and figure out, it's probably the CPU at fault, it's probably Fex. Or it's probably the GPU at fault, it's Turnip that's broken or some intermediate layer. And to be honest, it's somewhat of a painful process to investigate because you're dealing with this huge stack of things. And anywhere in that whole stack an issue could be that's causing this. And someone just has to look and then they figure it out or they push it onto someone else that like, 'Hey, this issue is actually in your part of the stack.'
Brian Kardell: I'm having difficulty wrapping my brain around this. I'm really sorry for ... it sounds too rudimentary, I guess, but I'm thinking about, I don't know, think about a game like Red Dead Redemption 2 or something like that, where it's like you might play that game for 40 hours before you get to the end. So there's an error in the hit box of a critically important end of game piece. Do you have to play 40 hours before you even get to the bug? Or is it more likely that if it's physics related you'll discover it much earlier because everything relies on the same engine?
Dhruv Mark Collins: Yes. In most cases, it's very unlikely that you only hit a bug in a very, very specific circumstance. It's not impossible though. In some cases, yes. You wouldn't have to play 40 hours, you can probably just get a save file, but it is that rudimentary. If the case is that specific, yeah, you would have to trace it down to running the game, 40 hours in, I don't know, work through all the cut scenes. And every time you want to attempt it again you have to do the whole thing, whole sequence again, get to that point, have it crash. Or the hit box is wrong, don't even have it crash, pause the game at that moment of execution and try to somehow introspect on what's wrong.
Brian Kardell: Wow. That must be incredibly frustrating.
Dhruv Mark Collins: It is.
Brian Kardell: Just thinking about last night, I was with my girlfriend, her daughter's birthday, and she was playing actually Red Dead Redemption 2, that's why it was fresh in my mind. And there's a particularly difficult bit in there. And she just played, again, just like you were talking about, the same three minutes a hundred times trying to make it happen and then, frustrating, go back and do it again. I can't imagine trying to debug a program that way would be really, really difficult because even-
Dhruv Mark Collins: It is. We've spent so many cycles on this. Actually, Paulo who previously spent all this time on Psychonauts has found a new game with another annoying issue, Crysis 2. The original was released somewhere in 2011 I think. And when you go to the main menu there and you specifically hit the resume game button, it's some kind of DRM, digital rights management, in the game gets triggered and tries to run this code which really, really breaks Fex. And there's this crash that he's gone through this whole sequence so many times by now. He spent multiple hours just trying to get there and have it stop at a moment where he can perfectly introspect, okay, this is why it's breaking. Yeah, it's honestly, we don't have a better way of doing this because things are not containerized as neatly as they are in, let's say, web land, you're running JS, it's all nice in this debug environment that's ... We just don't have access to that level of control over the game.
Eric Meyer: Yeah. This actually all reminds me of a story that I saw just recently about, interestingly enough, a Valve game where, and I'm not going to go through the whole thing, we can link to it in the show notes, but when they were compiling Half-Life 2 for the Oculus Rift, there was a game-breaking bug early in the game that wouldn't ... game breaking in the sense of you couldn't progress from that point. And so they went back to the original source code and compiled it. And it was there in the original code. And they're like, 'This bug traveled backwards in time. Millions of people have played this game. No one has ever had this problem.' The problem turned out to be that the original was compiled for X87 instruction sets, which had a lot of different precisions apparently. And when they were compiling for the Rift and also on new machines, that was being compiled to X86, which does not have this wide variety of precisions. So the problem, it was a floating point error was what it came down to.
Dhruv Mark Collins: Quick correction there actually. X87, it's a weird name, it's actually an extension to the X86 instruction set. So X86 games can use this X87 instruction set. And in fact, that is what Paulo dealt with on Psychonauts. It uses X87, and so it has these weird math bugs. X87 is actually a huge part of what Fex has had to deal with in terms of especially getting things to run fast. Because X87 is weird in no other CPU other than X86 servers implement these weird 80 bit floating points. And to do them, you have to do them in software, it's slow. You can cut corners with just treating them as 64 bit floats, but doesn't always work. So yeah, it's like a lot of these games have these intrinsic bugs that are just baked into them. And you just never find out because they just didn't run on anything that wasn't X86 GPUs.
Brian Kardell: It's interesting to me, well, there are different instruction sets and there are different hardware manufacturers, and it seems two very layman like why make your thing with ARM hardware if it's problematic for games. If you're building something for games, why not build it out of something-
Dhruv Mark Collins: Yeah, that's a great question actually. And the thing is that ARM excels at one thing that X86 is not necessarily caught up to, that's a lot of ARM SOCs, system on chips, have been in embedded devices. So things like phones, tablets, all of those pretty much at this point will run an ARM based chip. So they have a focus a lot of these. It's not intrinsically tied to the ARM instruction set as much as the ecosystem of the chips that ship with these. They've had a really tight focus on embedded devices that are very power conscious. You're running on a battery, it needs to be really efficient. And also, and this is actually the VR angle here, Qualcomm, which is the vendor behind the SOC that's in the Steam Frame, has these set of chips which are designed already for these VR workloads. They've been tested on that. And a huge part of that is being able to ... there's the battery end of it, but also things like being able to take in so many camera inputs. Because the headset being standalone now needs to figure out where it is relative to the rest of the room, needs to figure out where the controllers are relative to it by just looking at camera input and a bunch of sensors to enable all of that.
Brian Kardell: Eye tracking as well too, right?
Dhruv Mark Collins: Yes, eye tracking as well on top of that. And so all of these things, if you were making this on an X86 chip you would have to re-implement the whole world. But conveniently, the Meta Quest already started working on Snapdragon chips by Qualcomm. And so they have a lot of things that they can just reuse by building on this Qualcomm SOC. So it was far easier to deal with the limitation of, hey, can we get games running on CPUs? Then can we get all of this hardware stuff working with ... the VR specific hardware stuff working on X86 CPUs that are currently shipping? Even if they worked with something like AMD, I assume that's going to be such a monumental task that this just seemed like the much easier option.
Eric Meyer: Wow.
Brian Kardell: I just want to say that, obviously we're the maintainers of the Wolvic browser, which came from Firefox, it was originally Firefox reality, so we've been somehow involved in ... And actually Sergio, who is lead on that, also did the initial implementation for WebXR in WebKit itself. So we have this connection with VR and XR, and that's probably the part of this that I'm a little bit more familiar with. And we were talking about foveated rendering. There's lots of things that are really, really tricky to get right that are different than when you have just a desktop screen or a TV or some single screen. Where you're asking people to stick themselves into an environment and there are lots of things that can make that feel wrong or unnatural. And if you're not careful, there are way more incredible demands too like that. I think a lot of that foveated rendering is mainly about performance, right?
Dhruv Mark Collins: Yes.
Brian Kardell: You don't have to render a whole bunch of the thing as tightly as where your focus is actually centered because it's wasted, right?
Dhruv Mark Collins: Pretty much, yep.
Brian Kardell: And so I'm just wondering if you could speak to if we have done any of that or what are some of the things that are important in that space or that we might have worked on?
Dhruv Mark Collins: Yeah. So this is a really interesting topic actually of how is VR different as an environment to make things for compared to something like just your desktop. Because a lot of the things that we've done here, for example, you can use Fex to ... now there's ARM laptops. If you're running Linux on there, you can just use Fex and you can get games or X86 applications, games, anything you want working using that. So this benefits the ecosystem outside VR. But the things that are unique about VR here, I've encountered a good bunch while working on Turnip actually. So the development as a whole is a bit tougher because you constantly have to put a VR headset on and off. And so when it takes that long, 29 hours or whatever, I've spent just debugging sessions where I'm putting it on, off, on, off. And you can feel sick after a while of just going from VR back to the real world constantly, because that's an actual part of debugging. You just sometimes have to depend and look at the VR screen directly. But with this, with it encompassing the world around you, when you have a GPU bug that causes let's say minus data, and this exists in a desktop game, let's say you move your mouse a bit, but the mouse takes half a second of delay to move to that place. That's perfectly fine. Who minds that much, if it's occasional, whatever? Just no one's going to care that much. In VR, if you move your head and the actual screen moves half a second later, a lot of people are going to get motion sick from that. Because it effectively is your world when you're in VR. And I dealt with an issue regarding this where you saw a bunch of starters in certain scenarios where the GPU load was extremely high. And what we call the compositor for the VR environment, it would be at a lower priority. And so the game, if it used enough GPU resources, would actually delay the VR environment from updating a tiny bit. It could be half a second or something. Not an issue generally, but, again, when you're dealing with all of these environments where it just encompasses you fully, a tiny starter becomes a huge issue for you all of a sudden. Where, okay, this is going to make some people, can we ship this as a product, is this going to make some people nauseous because they were just trying to play this game and all of a sudden everything around them lags. Because when you're projecting things, especially in 2D games, you're projecting a screen in front of you, and you're in the VR environment, it also has pass through mode so you can see the world around you. And if you're seeing the world around you through your VR headset and the world around you starts lagging because of the game workload, that's not too great.
Brian Kardell: Yeah. I think it's interesting, there are a lot of things in software development, product development where we give users something new, and we're thinking about it in old ways. I think Eric, maybe there used to be this really common parallel where people would talk about how we thought the web was print at first, and basically every new medium is trying to apply the rules of the old medium. And I think that's definitely true with regard to the XR stuff. The little things that if you do what you think initially seems sensible it isn't somehow. So you think like, 'Oh, well, I want to render all this stuff the same.' But no, you don't really. And the screen that you were just talking about, this is something that I didn't really think of, but screens when we're watching movies and playing games and stuff like that, they're fixed in space. And they're usually a distance away from us. And so small movements of our head aren't really messing with the screen. And one of the things that I noticed is it's amazing to have this giant screen in front of you. But then if you turn your head, the screen goes with you. And even though that sounds like, sure, that's what we want, it's totally not. You want to be able to pin that in space. And maybe deal with the fact that your micro movements of your head need to be throttled to keep it from being jittery because that also can cause some motion sickness stuff. And I watched the development of these things on the Quest for a long time. And so this whole new environment is like, it's not new, but in the historical sense, it's still pretty new. It's full of really cool, interesting challenges. And I'm really excited that we have this Steam competitor and that Igalia is working really closely with them on this. Yeah, it seems really, really cool.
Dhruv Mark Collins: Yeah. I mean, as you mentioned, these challenges that we just didn't have, because a lot of this we were coming into with the perspective of running things on just like desktop games on flat screens, you're not running it inside a VR headset. And so we, even in something like Turnip, the GPU driver had to adjust things around that. And so in the case of the status that I mentioned, actually the default driver behavior is to just not care about that. It's not a factor that really matters to us. And in fact, a solution here actually slows down rendering because it might prioritize the game process. And generally you'd want the game to run the fastest, but all of a sudden now you introduce this new environment of VR, and maybe you want the game to not render the fastest if what you're sacrificing is the VR environment, like as you mentioned with head jitters and other things, you actually want to account for them. Because otherwise, yes, the screen will, if you turn around, the screen will just follow your face instead of being fixed in space, in your virtual space. And so to not have that dissonance in your head of like, what do you expect, how you expect reality to work, reality doesn't generally stutter, last I checked. And in this case it does all of a sudden, and that's an issue that you want to avoid. So it's all of these new limitations that we see popping up that are specific to this environment. Alongside a lot of the older limitations to just running games fast, that's definitely something that's been challenging too. But the new limitations have been particularly interesting to deal with. Especially in terms of run it fast on this device, which is mounted to your head, and I mean, your GPU inside your PC, it can hit 80 degrees Celsius, tough for something mounted to your face to do that.
Eric Meyer: Yikes.
Dhruv Mark Collins: So it has to not do that, and luckily it doesn't.
Brian Kardell: It's a really cool project. I'm excited to see it in action hopefully soon.
Dhruv Mark Collins: Yep. And very hard at work at getting the product to the finish line.
Brian Kardell: Cool.
Eric Meyer: Cool. Very cool. Dhruv, thank you so much for sharing all of this with us. I think for Brian and me both it's interesting because it's a little bit to one side of the kind of thing that we usually talk about, which is usually web. And this is not web. But at the same time, that whole idea of emulating other things is just very cool.
Dhruv Mark Collins: Yeah. In a way, I guess, it's trying to adopt the spirit of the web, of getting everything running everywhere. And this is something that Valve has pursued with something like Proton prior, which is a compatibility layer that allowed them to run Windows games on Linux. And that also just follows the philosophy of, okay, let us go to the games rather than the games coming to us. Let's not make the developers do all the effort of maintaining two different versions, one for Windows, one for Linux. Who knows how many other combinations, especially now, this is an additional combination, this is Linux+. And instead of them doing that, let's just make it. They can do it if they want to have a better experience, but they can also not. So on web, someone can do all the browser specific attributes in CSS, for example. They can get something perfect on one browser. And reasonably good enough on others. So it's up to the developers.
Brian Kardell: I was just wondering if it makes sense to mention that all of those other devices are running on Android open source based operating systems and Steam Frame is Linux, right?
Dhruv Mark Collins: Yeah. The Steam Frame uses Steam OS, which is based off Arch Linux. So that's different from every other standalone headset out there which just uses Android. I believe a huge part of the stack is open source. Valve did a lot of work, and especially we were involved in a bunch of that work to ensure that all of these components, which were earlier dependent on Qualcomm's proprietary drivers, Qualcomm has a proprietary Vulkan driver, which all the other headsets depend on, but it's not open source. But Mesa 3D Turnip has some other bits as well, which we've helped them get open source versions of. But yeah. So this actually has a side effect of allowing a bunch of different Qualcomm devices to run on this new open source stack that they've now created. And so something like Qualcomm laptops, for example, earlier on Linux at least, they got a limited amount of love. Now they can run this whole open stack which has just been created due to the Steam Frame.
Brian Kardell: I think also the Steam Deck and the new Cube thing are also Arch Linux.
Dhruv Mark Collins: Yeah, the Steam Machine, yeah, they're also Steam OS.
Brian Kardell: And yeah, so that's cool and interesting. I think we did another show on this, didn't we do a show, it's probably a couple of years ago now. I think it was called Steam powered ... We'll link to the episode.
Dhruv Mark Collins: Yeah. The Steam Deck and the Steam Machine or the Gabe Cube also do end up using Steam OS, which is Arch-based. And it also relies on Proton to ensure that Windows games run on it without being Windows-based.
Brian Kardell: They're full-fledged Linux, so I think you can run the desktop on them as well.
Dhruv Mark Collins: Yes, you can.
Brian Kardell: And it's an interesting idea of why did the technology get into your house in the first place? And it does seem to recognize this thing that gaming is a really big reason that people got home computers even in the first place. So it will be interesting to see the effect that it has on Linux actually.
Dhruv Mark Collins: Yes. It has had a pretty sizable effect actually I believe. With the Steam Deck, a lot of people who earlier wouldn't touch any desktop Linux now all of a sudden are on this platform where they got a device that just runs desktop Linux. And so a lot of people who would've never had that experience are now in the open source ecosystem. And earlier on they might not have seen the value in that. But just by having this device, which is so well done in terms of everything just working, they can get a taste of how things are on this side of the side of the fence.
Brian Kardell: Okay. Thanks.
Eric Meyer: Yeah. Thanks, Dhruv, this was really interesting.
Dhruv Mark Collins: All right. Yeah. Thank you. Thank you for having me.
Brian Kardell
Eric Meyer
Dhruv Mark Collins