OpenXR 1.1 is being released today, and so I’m digging into my archive of unpublished interviews to go back GDC 2018 when OpenXR working group chairman Nick Whiting presented some of the first information about the OpenXR specification, which wouldn’t have it’s initial release over a year later on July 29, 2019 (see episode #1384 for more on that). I’ll also be publishing an update on OpenXR 1.1 in episode #1385, and so these next three episodes should provide a pretty comprehensive look at the evolution of OpenXR over the past 6+ years. Also be sure to check out episode #507 from GDC 2017 where I chat with Joe Ludwig about the initial announcement of OpenXR.
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Ken Pai, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So OpenXR is releasing a 1.1 minor release version today on April 15th, 2024, and I decided to go back to some of my previous interviews that I've done about OpenXR that I haven't had a chance to publish yet, going back to GDC of 2018, when they had been meeting for a little over a year at that point, and they were releasing some of the first preliminary information about the OpenXR specification. And then the following year, in July of 2019, at SIGGRAPH, is when they released the 1.0 spec for OpenXR, and we're going to be releasing a new interview with OpenXR 1.1. And that'll be the third episode in the series. But I'm going to be starting with a conversation that I had with the OpenXR working group chair at the time, which was Nick Whiting, who at the time was also working at Epic Games. And then we'll be diving into some other progressions of the OpenXR spec, which actually has been one of the most successful open standards that Chronos Group has ever published so far. So we cover some of the very early information about the specification of trying to abstract out all the software development kits interfacing with XR devices and how to take all those components and make sure that across the industry, they're all doing something that's fairly standardized. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Nick Whiting happened on March 21st, 2018 at GDC in San Francisco, California. So with that, let's go ahead and dive right in.
[00:01:43.836] Nick Whiting: I'm Nick Whiting. I'm technical director of all the XR initiatives at Epic Games. So I work on everything from architecting on Unreal Engine 4 to the games that we produce. We released Robo Recall last year at GDC, in fact. And before that, we did a couple of technical demos, Showdown and Couch Knights and Bullet Train. So I'm kind of in charge of the engineering aspects of all those things. And I also happen to be the chair of the working group for OpenXR for Chronos. So heavily involved with that. We just had our first informational release here at GDC this year on Monday.
[00:02:13.016] Kent Bye: And I'm wondering if you could comment on OpenXR, because this is an initiative that started a number of years ago. And I first started coming to GDC 2015, and I talked to Neil Trevitt. And at that point, we weren't ready for standardization. And then the DK2 came out, then the Oculus One, and then the first Vive. And now we've got augmented reality there as well. And so we've had well over a year now of collaboration amongst all of the different augmented and virtual reality headset manufacturers saying, hey, we got all these SDKs. Why not find some sort of way to standardize them so that anybody who wants to have a peripheral doesn't have to reinvent the wheel and integrate with all these SDKs? And so there's been this process of doing that, theoretically. But what is it that you're reporting now as a check-in here at GDC as to where things are at?
[00:02:58.072] Nick Whiting: So, this was the first time when we actually released any concrete details about the specs. So, last GDC was kind of our, you know, coming out party for the OpenXR specification. We had a panel, but at that point it was still so early, we weren't ready to talk about any concrete details. We got a lot of people asking, like, how can you possibly standardize something that's developing so fast? You know, at that point, you know, AR was just starting to get going and VR was, you know, pretty mature, but there's still a lot of innovation. in the field, and people were kind of skeptical that you could do that, but we sat people down in a room, we got the dogs and cats to live peacefully with each other, and it's amazing how cooperative people were in the SPEC initiative, because they realized that there's a lot of systems that you don't need to reinvent the wheel for. Tracking gives you a pose at a predicted time, that's the same between all the SDKs that are out there. And then there's little bits and pieces of everybody's SDK that they drop into the OpenXR spec. So we've gotten everybody to settle on what does it look like for frame timing? How do we tell that we're beginning to render a frame and ending to render a frame and blocking the game state in order to hit the proper frame timings to lower latency? We got people to agree on that. The input system was contributed in large part and modeled off of Valve's controller mapping system that they have, where instead of binding directly to a key, the application expresses the actions that it can listen to and what types of data it wants. And that's really cool, because now if I have a new type of hardware device and I provide some sort of input, like a pedal or something... Or even a gesture or something. Or even a gesture, yeah, yeah. You can just recognize that and bind it. So it's, in a way, future-proofing things, because, as Joe Ludwig from Valve said, developer teams are ephemeral, but platforms are forever. And, you know, he works for a platform team, so he's a little bit biased. But I think the case there is true, right? Where it's very hard, after a product's been released for two or three years, to get the development team to go back and invest in adding a new hardware platform to it. But if they future-proof themselves by saying, these are the actions that I care about in the game, that makes it much easier for something to slot in to that ecosystem. So that's really what OpenXR is about, is allowing that expansion, kind of future-proofing, by figuring out the abstractions that applications can develop and give to the SDK that let everybody chime into the ecosystem, both from the hardware side and the application side.
[00:05:03.349] Kent Bye: Yeah, and you did a presentation here on, I think it was either Monday or Tuesday, and there was a lot of slides, and those slides go into great detail in terms of diving into the weeds of the specification and lots of graphs and everything, and I just had a chance to skim through it. One of the things that jumped out at me, though, was the design intentions that you had of, like, saying, you know, one of them was, for example, that You wanted to create something that is flexible and malleable to be able to build into the future. So you have to create some sort of modular architecture to be able to do that. But maybe you could give us an overview of some of those higher level intentions that you have for doing the OpenXR spec.
[00:05:36.795] Nick Whiting: Yeah, and I hope you appreciated all my fine programmer art in addition in there. There's some pretty sweet stick guys in there. Yeah, so at every level of the specification even down to the kind of function calls we try to plan in extensibility into there because one of the pillars that we set forth in OpenXR is that we don't want to try to predict what XR hardware is going to be like in two or three years. That's a foolish process, right? And we hope that we're surprised with it. So, you know, if it's from rendering to input to the frame timing stuff, everything has an extension that you can build into it. So if there's something that we haven't thought of that somebody wants to try, they can slot that in. There's kind of four levels of extensibility in the specification. The first one is the core, which includes things like frame timing and tracking that are fundamental across everything in the OpenXR system. And those are kind of the required minimal feature set. And on top of that, there's what we call KHR extensions, which are kind of ratified by the group and put out, and we think that a lot of people will implement. So things like graphics API support, platform support, a lot of the AR features are KHR extensions, because we feel that there's a significant percentage of the population that are going to want to implement that. But as you start to get into more esoteric applications or more specialized applications, there's also an EXT level where two or three companies might have some specialized piece of hardware or something that's applicable only to a certain subset. Thermal performance and some of the debugging stuff goes into the EXT level so that people that care about those sorts of things can feel free to innovate in there and hook into the platform wherever they want to, but it's not required for everybody to participate. Because one of the biggest dangers that we found early in OpenXR is trying to please too many people. At the same time, if your spec grows so massive, that makes it harder for anybody to implement it. And that means it's harder for applications to write to it, harder for hardware people to buy into it. So we really wanted to concentrate the core feature set on what is only truly applicable to everything that we plan to have in the ecosystem. And then even farther from that, there's a vendor extension where anybody can go off and write whatever extension they want that fits into the ecosystem. That doesn't have to be approved by anybody, but they can have support for that if they wanted to do something that was truly one-off. But by and large, what we wanted to make sure of in terms of extensibility is that we give as much flexibility to the runtime, which is the part that sits between the application and the device hardware, so that people can still innovate on things like reprojection or prediction timings and stuff like that, where the secret sauce for these companies really are. We hope that they'll be able to compete on new features now as opposed to having an API that just kind of reinvents the wheel for everybody and everybody's tracking is just slightly different.
[00:08:06.621] Kent Bye: And so as I try to think about all the different aspects of virtual reality or augmented reality, XR experience, there's a reflection of that in the specs. So imagine that there's something around the head tracking, the 60 degree of freedom, the three degree of freedom, the input controllers, the motion track controllers, the sound, haptics. And there may be other things with tracking other parts of the body, or if that may be abstracted into some general, if it's not just hand tracking, but all tracking of anything. So maybe you could walk through if those are all the major ones and how you sort of break it down to, I guess, the archetypal essential components of defining what the basis of a XR experience is.
[00:08:47.570] Nick Whiting: Yeah, so for the basis of the XR experience, basically we, you know, you kind of draw the circles of AR and VR and MR, and where they overlap is really where we define the core specifications. So, for tracking in particular, that's just, to us, another type of input. You know, we have something called semantic paths, which lets you address devices, kind of like you would address a file on a file system or a URL on a URL system. So you can go, you know, my device dot my tracker dot, you know, X position, Y position, Z position. And we define all that data in a way that any device and the device extension plugin system can feed that into the runtime and then that runtime can surface it to the application. But the place that makes that work with the application itself is it can define as many actions as it wants to know about. So if you want to go to like a human body tracking position, if my application says, you know, I can track a knee position for left knee and right knee and left elbow and right elbow, The runtime can say, hey, I know how to get that data because I have a Vive tracker or a Myo tracker or something like that plugged into it. That kind of future-proofs it so that the application can say, if you can provide me this data, I can use it. But if you can't provide me this data, I have a fallback or I just ignore it. So that's really where the extensibility and future-proofing comes in. We want people to be able to do all the crazy things like taking a Vicon tracking system and using it with a Oculus headset. And we hope that runtime manufacturers will be able to kind of synthesize all that data into one. And we specify how that gets relayed to the application so the application doesn't have to worry about it. It's just saying, okay, I got a set of positions that are in these different spaces. Here's what I want to do with them on my character or whatnot.
[00:10:15.245] Kent Bye: And so let's talk about, as you're defining the spec, how do you tell the difference between 6DOF and 3DOF, for example? When you say that there is an abstraction of tracking, does that mean that the hand-track controllers are the same as the head-tracking or the same as anything else that's tracking? Or does it break down into body parts so that there's a difference between the head-tracking of a virtual or augmented reality experience and the hand-tracking?
[00:10:36.414] Nick Whiting: So there's not really fundamental differences. There is one special case, which we call viewport poses, which is, you know, if you're rendering an HMD that has stereo, it's the pose of the left and the right eye associated with some projection parameters so that you can render your virtual camera to match what the tracking position is. So those come in in the same tracking specification, but they're tied very firmly with the rendering projection with it. But for 3DOF versus 6DOF, in those pose commands that I mentioned before where you can just say I care about the pose of X, Y, or Z on there, we have a device capabilities flag that says I can track position, I can track orientation, you can get a combination to suss out what kind of data is coming in. So the application can say, OK, I see that this is a rotation-only tracking. I know what to do with that if I have a 3DOF controller. Or if I have a 6DOF controller, I know I can actually trust that position to be accurate, and I don't have to put a head model or an arm model in there.
[00:11:28.602] Kent Bye: Well, can you describe the difference in the spec between the mobile and a full PC VR? What flags in the spec level would be able to tell the difference between whether or not that device is mobile or a PC?
[00:11:42.837] Nick Whiting: So the most obvious place is the kind of Android or other mobile platform support. That's implemented as KHR extensions in there, because we don't expect PC people to implement that stuff, just like we don't expect an Android device to implement a bunch of Windows and DirectX extensions.
[00:11:57.407] Kent Bye: So this would be at the extension level, then? So the core baseline of OpenXR would be just for mobile. Anything that's PC would be an extension?
[00:12:04.674] Nick Whiting: Well, no, sorry. Anything mobile and anything PC would be extensions, specifically. So the core spec handles a thing like tracking and providing reprojection matrices and providing a device marshaling layer for your input and stuff, which is the same across PC and mobile. You're obviously looking at different devices, but the fact that I can get a button press or I can get a pose at a given time is in the core level of the spec. And then something that's like, how do I draw to an Android window is in the Kronos extension layer on there.
[00:12:33.556] Kent Bye: So haptics is something that I think is probably the least well mature and evolved technology that I think, you know, in the next five to 10 years is going to probably have the most change and growth. And I'm just curious about the data structures that you send to haptic devices, if it's some sort of like matrix or like people that represent higher dimensional topologies. Because right now we have very simple on-off binary buzzers, but you're able to potentially have shapes or forms that are very sophisticated to be able to go over your entire body. And so what is it at the spec level that you've been able to actually handle haptics?
[00:13:06.297] Nick Whiting: So for haptics, one of the decisions that we made early on is they kind of go through the same XR action system that inputs do. So we define a bunch of parameters that you can send down to the runtime, and then it's up to it to interpret what to do with them. So at the base level, we have the kind of standard frequency amplitude and duration sort of haptics. But just like anything else, you can define your own XR action types. If you said, you know, I have some vector data. If you've seen those like tactical haptics has the, I believe they call them sheer haptics that actually move the skin on your hand to make it feel like you have something in there. You could provide a vector down saying this is the direction that something is pulling in my hand as an action system. And then the runtime can read that. And if it has a device that can make use of that data, just like the application can say which events it cares from. the runtime level can say, I can ingest that data and do something cool with the haptic system. So the idea is that the application kind of throws everything it knows over the fence and hope that anybody that's listening for it that knows how to interpret that data can do something useful with it. But if there's nothing, it can just get left out of the thing. So that's what kind of makes it scalable and extensible.
[00:14:11.705] Kent Bye: So what about the difference between AR and VR? Because when you have VR, you start to have things like being able to track a space and make a mapping of that space, but also perhaps bring in geodata to map in. And so you have these different layers of reality. And so from the spec level, how do you differentiate the difference between VR and AR?
[00:14:29.680] Nick Whiting: So there's a few different places in the core specification where it differentiates and that's mostly in the viewport configuration which says how many eyes do you want to render and what are their projection parameters. We have these things called viewport configurations that say I need this many renders. So for a pass-through AR device that was like ARKit or ARCore, It would have one viewport because you just have one device that you're looking through, whereas if you had a stereoscopic VR or AR headset like an Oculus or Vive or a HoloLens or something, you'd get two viewports, one for left eye, one for right eye. If you had something like a Star VR that has super wide field of view, they need two viewports per eye so that they can wrap it around your head a little bit better. and that would have two viewports per eye, and you'd have four total viewport configurations. So, for AR, basically the biggest difference at the core level, all the tracking is the same, you still get positions the same, but you only have one viewport that you need to render to. So, the applications say, hey, how many viewports do you support? I support one, because I'm a pass-through AR device, then go for that. And if it's a see-through AR device, like a HoloLens, then you get a left eye and a right eye with that. Things like ingesting meshing data and real-world sensor data for that would probably live at the KHR level for that. So not all VR devices can ingest meshing data, but some of them have stereoscopic cameras on the front that can give you a 3D topology of the room. So we would put that at the KHR level so that not all runtimes have to implement that, but ones that do have support for it in a well-defined way, so that if I'm ingesting camera data from a VR headset, it would be the same interface to me as an application. something coming from an AR headset that had meshing data.
[00:16:01.176] Kent Bye: So there's a lot of standalone VR headsets that are coming out, the Vive Focus, Oculus Go, Oculus Santa Cruz, and lots of phone-based AR. But specifically for the difference between the PC and the standalone, do you expect that these companies are going to have a single SDK that's able to go between the mobile and the PC, and they're both going to be implementing the OpenXR standards? Do you foresee that there may be some fragmentation between, like, hey, this is a mobile. We don't plan on interfacing with anything. We're going to sort of go off and do our own thing. Or do you see that the trends within these companies is to try to come up with one single SDK that spans both from mobile to PC so that they could have this seamless architecture between both of them?
[00:16:43.904] Nick Whiting: I think there's an important distinction between an SDK and a runtime, at least in the view of OpenXR. The SDK being the way that the application or the devices actually talk to the runtime in the middle, which is equivalent to your SteamVR runtime or your Oculus runtime. The way that an application can talk to an OpenXR runtime is agnostic to whatever platform it is. So, from the application's perspective, whether it's on a mobile phone or a PC, it's still talking to the exact same API. There may be more extensions that are enabled on a PC or a mobile device. Obviously, the Android extensions aren't going to work on PC and the Windows extensions aren't going to work on Android. But they're at least consistent across the ecosystem for everything else. We expect that in a case like that, somebody would probably have a desktop version of their runtime, which handled interfacing with a desktop class device that maybe has more methods of input or a slightly different interface with the rendering devices versus a mobile device. So there might be two separate inputs. But as far as the application is concerned, it doesn't care. It's talking to an OpenXR runtime, which speaks the same language, so to speak.
[00:17:44.312] Kent Bye: And I'm wondering if you could comment on if pretty much every major VR, AR headset is integrated in OpenXR. I know that's a tricky question because, for example, is ODG or Magic Leap or Microsoft? And there's obviously a ton of companies from China that have been coming up. And every year at CES, there's more and more. And so just trying to get a sense if this is something, and also PlayStation VR I should talk about because they're also in the ecosystem, but I don't know if there's much advantage for them to do that. From a software programming perspective, I guess it would presumably make it easier for software developers, if there was a standardization of this OpenXR, for them to be able to write something once and be able to have it deployed across all these platforms. And I'm just trying to figure out if there might be some standouts like PSVR or Magic Leap or anyone else.
[00:18:29.842] Nick Whiting: So, I can't comment as to anybody's position on whether or not they will end up implementing it, but if you look at the member companies on there, right, you know, you've got Oculus, you've got HTC, you've got Vive, you've got PlayStation, or Sony, I'm sorry, in the ecosystem actively contributing to the OpenXR specification, to the point where they're willing to put their logo on our page saying, hey, we are active members of OpenXR. Whether or not they implement it and in what time frame they do so is up to them to announce and handle, and a couple of them have. It's pretty publicly available information, so again, I can't speak for them, but if you Google it, you should be able to see what people's various commitments are. The advantage to it, though, is I'm still a believer that the kind of XR ecosystem right now is still a little bit nascent, so as an application developer, you want to have as large of an addressable market space as possible. I want to write my application and be able to sell it on as many platforms to get that maximum number of users that can buy my piece of software. And as a hardware manufacturer, I want to have access to the largest number of applications possible on my device so I can sell more hardware. But also for new hardware developers, one of their big challenges is if you have a specialized piece of peripheral or a piece of hardware, it's very hard to get an application developer, especially a small application developer, to actually hook it in. to their application. It costs them engineering time. A lot of these teams are still fairly small, and that's kind of reflective of the size of the industry at the moment, right? You can't have multiple hundreds-of-person development team and make it economically work for a long-term project. So a lot of these teams that are super successful are very small and very limited, and they can't necessarily throw an engineer at integrating an SDK. But if that SDK can slop into the OpenXR ecosystem and take advantage of the events that the application has presented, naturally, that all of a sudden makes much smaller runs of hardware actually a viable thing. It encourages experimentation and encourages us to try to find the next big thing. So I think overall it's a very healthy thing for the ecosystem to allow that.
[00:20:22.918] Kent Bye: Yeah, to me it's pretty remarkable, actually, to have so many companies collaborate on something like this, because the tendency could have been to have everything centralized and fragment, but the fact that there's this open XR standard means that it's, in a sense, creating a decentralization aspect of an open standard that levels the playing field for other peripherals to come in. Yeah, it's just for the OpenXR though, I guess moving forward, what are some of the biggest either challenges or open problems that have still yet to go forward as this industry or platform that is still kind of rapidly evolving?
[00:20:55.257] Nick Whiting: So as of last fall, we met in Chicago. We have face-to-face meetings every quarter where we can have kind of three days of very high bandwidth discussions. And that was the face-to-face where we said, OK, if we're going to ship this thing in any sort of reasonable time frame, we need to decide what our MVP is. And we kind of went through the entire list of things we could do and paired them down to the list of things that we viewed we had to do to make this a viable standard. for 1.0. So we've kind of gotten that and we're into the phase where hardware manufacturers are actually implementing runtimes that are compatible. I have seen pixels rendered through OpenXR on real-world devices. We've started a plug-in for the Unreal Engine compatible with OpenXR right now, so things are starting to come together. So where we're at in the process right now is we have a good kind of stab at the first version of the spec, but we need to suss out the bugs by actually going through and implementing against that spec. And it's turning up these little things, but the nice thing is nothing is super controversial. At the moment, everything is like, okay, this tweaked, how can we fix it to make it successful in the current spec? So the size of the problems are getting smaller, which is a good sign. The biggest one that's still to be tackled is how we do conformance testing, which is to say, how do we give our stamp of approval? What test do we have to run to say that you are a runtime that is actually OpenXR compliant? And a lot of the challenges from that come from, you know, with a graphics API, you can call a set of commands and then that's supposed to produce one image and you can just compare the output of that image to another one. But with VR and XR and AR, you put a device on a squishy meat bag and there's a lot of variability into them. It's not like we have a... lab with robots that are, you know, highly controlled and highly measured that we can put these devices through to verify the expected behavior under certain motion conditions is actually what comes out. So that's, I think, the biggest problem that's on our horizon. We'd like to get the spec out in what we call a provisional form, which is basically here's the specification, but we haven't gotten the conformance test that say you are certified to use, call yourself OpenXR compliant. yet, so that's I think the biggest thing at least on my mind is how we test these things and verify that the OpenXR standard is a mark of quality as opposed to something that just somebody could just slap their name on.
[00:22:57.444] Kent Bye: Is there any difference between LCD like virtual reality screens that we have today, AR and something that's on the distant horizon whether it's digital light fields, holographic displays or multifocal displays and if that changes anything for what data and what that means for the spec?
[00:23:13.587] Nick Whiting: We do take that into account somewhat with the spec in terms of extensions. So there's extensions for instance that let you pass a depth buffer or extra rendering parameters down because some display types might be able to take advantage of that depth information to do something intelligent or some displays might be able to take advantage of motion vector buffers to do reprojection more effectively. So that's a core part of the spec to kind of abstractly allow you to define this extra data that you send down to the display to help it do what it needs to do. So that's something that we leave to the extension land, that devices that need it can implement it. Devices that don't need it can choose to ignore that information, just like everything else.
[00:23:48.761] Kent Bye: Yeah, usually leading up to GDC, there's been a big new virtual reality demo from Epic Games. But this year, you've likely been working on the SDK for Magic Leap. So maybe you could just briefly share whatever you can about this new SDK from Magic Leap and the Lumen SDK.
[00:24:04.302] Nick Whiting: Yeah, so we were super happy on Monday to announce kind of day and date support with Unreal Engine 4 for Magic Leap and then today in our keynote we highlighted a few of the partners that were working with ILMxLAB, Wingnut, Shell Games and Framestore who are actually using it. So over the past few months kind of in dark secret rooms we've been working on getting Unreal Engine support for the device itself and for the SDK and then this is for our day-in-date release. We also released a binary version of UE4 so that people don't have to download from GitHub and compile if they don't want to if they want to try out the new Lumen SDK, which is the Magic Leap platform's SDK. So we're pretty excited because we think that lowering that barrier to entry will help get more people in there that just want to kind of mess around and see what the SDK looks like and what kind of feature set Magic Leap provides on their platform.
[00:24:48.207] Kent Bye: Yeah, I know that there's a lot of non-disclosure agreements with Magic Leap. And I think at GDC this year is the first time that we've started to see any sort of direct engagement with the developer community. There is a session that's going to be happening tomorrow and Thursday to talk more about the SDK. But I'm just wondering, what can you share with me about your phenomenological experience of going through the Magic Leap?
[00:25:08.123] Nick Whiting: Yeah, so it's actually taxing a lot of systems that we've just started to implement with ARKit and ARCore, which are obviously handheld pass-through APIs. But you know, in general, it's all kind of a spatial computing platform. You're ingesting information from the real world and then displaying it on a device. So, with Magic Leap, some of the stuff that we've been able to kind of buff up with the engine, we are adding eye tracking supporting, so there's a generic eye tracking. implementation in the engine that will eventually hopefully work with other companies like Tobii with their Vive headsets and their desktop-based trackers, but it'll obviously work on the Magic Leap platform. We're also ingesting spatial data, so if I'm looking around this room with the headset on, it'll start meshing the room, and then we run that through the Mixed Reality Mesh Component, which we affectionately call Mr. Mesh, MR Mesh, on there, and you can generate NavMesh data for it, you can generate physics collisions, so you can all of a sudden have, you know, throw a virtual ball and it bounces off and around the room and reacts as you expect it would. So it's really exciting for us to kind of have something that we can finally run that's ingesting that sort of data, and we can see what it's like for the developer and the designer experience to actually use that sort of data. Because you can give them a polygon suit, but then that's not necessarily something they can do. But if you give them a nav mesh, if you give them a physics constraint, all the stuff that they're used to working with, that makes it a little bit more accessible for everybody. But what does it feel like? You know, it's passed through augmented reality, and I think it's nice to have more and more people getting into that space. You know, we've had the HoloLens before, but having more competitors in that space will kind of push it, just like having Oculus and Vive and Sony and all the usual VR headset standards will push it. You know, it's really magical to Experience something changing in your world as opposed to having being placed in a different world I'm still a firm believer that VR is a certain kind of experience that plays really well when you want to be immersed in another world But I think there's also something magical about seeing your own world change a little bit So I think there's room for both and it is a kind of a magical experience in the same way that you know putting on a VR headset was a magical experience the first time around
[00:27:00.122] Kent Bye: Yeah, and just in terms of the HoloLens, it's like a 2D, you get this hologram effect, but I'm just, I talked to Paul Reynolds, who used to be in charge of the SDK, and he reported this story and this experience of a dragon coming and landing on his finger, and him, with his mind almost filling in the gaps in terms of feeling haptic feedback with something that wasn't actually there, but because our visual field dominates so much, it sort of does these weird crazy things with your mind. I'm just wondering if you had any similar experiences of that, of like having the visual fidelity match reality with the tracking that's so good that it starts to have almost like shadow or imaginary haptics or other things that fill in the gaps of your sensory experience.
[00:27:40.892] Nick Whiting: Yeah, there are a lot of those kind of moments. You know, your brain is very good as long as the input is consistent at filling in those gaps. You know, we're kind of pattern recognition machines and you can see that. You know, I kind of first played around with that in virtual reality where when you're holding a motion controller, it's not necessarily shaped like what's in the game that you're holding. But having something in there, your brain kind of maps one-to-one with that. Or when you use an avatar for the first time, and you can see your body in there, and your brain kind of maps, and you start to believe that your hands are the hands that are in the game, even if the motion isn't one-to-one. As long as it's close enough, your brain starts to fill in the gaps. With augmented reality that's pass-through augmented reality, it's the same sort of thing. I haven't had anything like a dragon landing on my finger, but it's amazing how much, as long as you get the rendering right, that your brain fills in. Pass-through devices can't render black, obviously, because they're only additive. But you'd be surprised when your brain expects to see black, like in somebody's pupil or something like that. Even if it's passed through and there's actually light behind it, your brain expects it to be black and just kind of makes it darker in and of itself. So there's a lot of those little kind of tricks that we're starting to notice, but it's still very early days. I mean, we're still learning stuff about VR, and I started on that about six years ago. So I expect it's going to be kind of another continual journey like that.
[00:28:50.229] Kent Bye: So, for you, what do you want to experience in VR?
[00:28:52.792] Nick Whiting: What do I want to experience overall in VR? Actually, there's a game that I've been waiting to play forever, and it finally came out right before GDC, but I was so busy that they hadn't been able to play it in MOS. Like, I love Zelda, and I love rodents, and so Mouse Zelda sounds really awesome to me, and the team that does it is super talented. I just love the world and the story building. The reason I love VR is because it's kind of a new narrative medium. And I think we've talked about this a little bit before is I love trying to find new ways and new grammars of storytelling with it. And I really feel that the attention to detail and the love that they've put into creating that mouse and making it super compelling in terms of animation and reacting to you and just all the little details that they put into it really is a narrative grammar in and of itself. And so I'm stupidly excited to play that after I can go catch up on my sleep after GDC.
[00:29:40.312] Kent Bye: Yeah, it's here in the showroom floor at the PlayStation booth. There's a MOS station there, so I'll have to try to check it out. And finally, what do you think is kind of the ultimate potential of virtual and augmented reality, and what it might be able to enable?
[00:29:54.739] Nick Whiting: Oh man, that's a deep question. I think, you know, I kind of touched on it before, but I think, you know, virtual reality and augmented reality ultimately serve two sorts of purposes. Virtual reality is great for immersing people in worlds and putting somebody in a narrative arc or, you know, a kind of created world and completely immersing them there. And you can tell a much different kind of story in there than you can in augmented reality. Augmented reality, I think, Ultimately, we'll find its success in kind of its utilitarian nature, kind of the same way that we have smartphones in our pockets that give us access to Google Maps and a lot of information. Augmented reality is super good at doing that sort of stuff. I think that augmented reality, though, is really going to come into its own when some of these other technologies are there to prop it up. You know, I'm most interested in applications that can take the sensor data that's coming in through augmented reality and then send it up to the cloud and actually do something intelligent. Label data, see where I am in my GPS and try to identify locations around it, pull that data in and tell me useful bits about the world or do meshing of the world and start to aggregate that data. and tell me something useful about it. I think we still need to improve our wireless networking capabilities a little bit with that, and I think we still need to improve our deep learning and AI applications with it. I think AI is really going to be fundamentally transformative once all those three things come together and we can leverage all the power of deep learning and sensor data aggregation and then start overlaying that with the real world, where I think VR's power is really, I want to go become immersed in a world and experience something really cool.
[00:31:25.677] Kent Bye: Awesome. And is there anything else that's left unsaid that you'd like to say?
[00:31:28.459] Nick Whiting: I can't think of anything. You're pretty thorough, as always. Awesome.
[00:31:33.844] Kent Bye: Well, thank you so much for joining me today on the podcast.
[00:31:35.845] Nick Whiting: Thank you. Always a pleasure.
[00:31:37.483] Kent Bye: So that was Nick Whiting. He at the time was working on the Unreal Engine at Epic Games, and he was also the OpenXR working group chair at the time, giving some of the very first information about OpenXR. So very interesting to go back in time and hear all the different stuff that was at the industry. This is before the release of Oculus Go and the Oculus Santa Cruz, which ended up being the Oculus Quest, had been announced, but wasn't released until May of 2019, the following year. Yeah, just all the different things that are happening with Microsoft and Magic Leap with augmented reality helped to really shape the OpenXR spec beyond just virtual reality and to really make it abstracted into XR. OpenXR was one of the first entities that actually adopted the XR as a term to describe both virtual reality and augmented reality. And so going through some of the updates over the years, I'll be jumping into 2019 in the next episode and then to here in 2024 with 1.1 release in the final episode. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
