The OpenXR 1.1 minor release comes out today, which moves some of the more commonly used extensions into core and it also marks a new Khronos Group commitment to a yearly release cycle. It’s been nearly five years since the initial release of OpenXR 1.0 on July 29, 2019, and Khronos Group President Neil Trevett tells me that it’s been one of the most successful open standards they’ve ever published, and I had a chance to catch up with Trevett and OpenXR working group chair Alfredo Muniz to talk about the range of conformant XR devices, engines, programs, what’s happening with Apple and OpenXR, next steps for where they plan to take the standard in the future, and how you can get more involved either through their OpenXR Discord channel, OpenXR Forums, or OpenXR GitHub Issue Tracker.
Also, here’s a link to the announcement slides for OpenXR 1.1, which have some really helpful overview information about how broadly OpenXR has been adopted, as well as a sneak peak at some of what is coming soon including “extending hand tracking, enhanced handling of spatial entities, expanded haptics support, controller render models (glTF), increased accessibility, and Metal (Mac OS) support.”
I posted a thread on X / Twitter that highlights some of these slides from the OpenXR 1.1 announcement page.
Here’s five previous interviews covering the evolution of OpenXR since 2015:
- My 2015 interview with Neil Trevett with some preliminary thoughts about an open standard for VR
- My 2016 interview with Neil Trevett announcing the formation of what would become OpenXR
- My 2017 GDC conversation with Joe Ludwig marking the announcement of OpenXR.
- My 2018 GDC conversation with OpenXR working group chair Nick Whiting with a first look at OpenXR.
- My 2019 SIGGRAPH conversation with Neil Trevett marking the official OpenXR 1.0 release
Also see some related topics within my interviews tagged with open standards.
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com. So today, on Monday, April 15th, 2024, is the release of the 1.1 version of the OpenXR specification. chance to sit down with the OpenXR Working Group Chair, Alfredo Munez, as well as with the President of the Kronos Group, Neil Trevitt, to talk about some of the accomplishments of the OpenXR specification, just from how much it's been able to be adopted across the XR industry, some of the intentions of having this runtime that is available on a broad range of different XR conformant devices, as well as in terms of the engine support and just talking generally around where things are at right now with the 1.1 release and how they're starting to bring some of the different extensions in OpenXR and promote them into the core of OpenXR and some of their plans for as they're moving forward in the future. So, that's what we're covering on today's episode of the Voices of VR podcast. So, this interview with Alfredo and Neil happened on Thursday, April 11th, 2024. So, with that, let's go ahead and dive right in.
[00:01:18.160] Alfredo Muniz: Hello everyone. My name is Alfredo Muñiz and I'm the current chairman for OpenXR. My main job is running a startup. We do full-body motion capture. And the working group, they really like to be open, but also fair. So they thought electing someone that was not a big runtime vendor, not an engine, but someone that can kind of see the big picture and not be biased to lead the group. So that's me.
[00:01:46.536] Neil Trevett: Cool. Yeah. How long have you been chair now, Alfredo?
[00:01:50.238] Alfredo Muniz: It is about a year and 10 months. But it's been a long year and 10 months.
[00:02:00.492] Neil Trevett: Yeah, that's cool. You're doing a good job. And my name's Neil Trevett. So my day job's at NVIDIA. I'm the president of the Kronos Group. I'm here as Alfredo's wingman today.
[00:02:11.016] Kent Bye: Great. Maybe you could each give a bit more context as to your background and your journey into working with VR.
[00:02:16.378] Alfredo Muniz: Yeah, sure thing. Where to begin? So my co-founder and I, we started a startup right after college, and it was actually making wearables for Parkinson's disease. And that was about eight years ago. So as you can imagine, it's been a journey to get to this point. Partway along that journey was going into government research, where we were detecting diseases on warfighters. COVID hit, we had to change our direction. And part of that shift was moving to VR, XR. We noticed that people were being more social with their games. And part of that was bringing your body into the virtual environment and Things like VRChat became really popular. We saw an opportunity there. We had a product that fit that market. And we got interested in approaching the market from the standardization point of view. So we looked at the Kronos Group and OpenXR. And we said, you know, if we join this, we do our thing, we standardize body tracking, then that'll make us essentially sell more of our product. So that's how it started. And it's turned into a lot more as I became chair. So I'm very happy to do what I do.
[00:03:37.236] Neil Trevett: Cool. Yeah, I think we've talked before, Kent. I've been involved in open standards to do with 3D since OpenGL 1.0, almost 30 years ago now, and working with various APIs, bringing it to the mobile and the web. And then, as we talked before, then one day Tim Sweeney calls them and says, I think we need a standard for XR. So we kicked off the OpenXR at Kronos. Thrilled to see the first new core release of OpenXR since 1.0. So it's a real milestone.
[00:04:10.996] Kent Bye: Nice. Maybe it's worth giving just a broad overview for OpenXR and the overall intent. And Neil, maybe you could start off with a little bit of the history since you've been at it since the very beginning of OpenXR. And so just give a bit more context to the evolution of OpenXR with the launch of 2019 and where we're at today.
[00:04:30.358] Neil Trevett: Sure, yeah. So when I got that fateful phone call from Tim, there were a good diversity of XR devices. It's grown a lot since then, but even back in the day, in 2016, 2017, you could see that multiple devices were beginning to ship. And all of the devices had proprietary APIs. That was a central problem that Tim Sweeney correctly identified, that if we didn't act quickly, this was going to turn into a real problem. So we established the working group, OpenXR Working Group, in 2017. And the group actually worked pretty quickly. Two years it took, give or take, monthly rounding, I'm not quite sure, but basically two years to get to the OpenXR 1.0 spec, which is not bad from a standing start, you know, a blank sheet of paper to a shipping complete specification in 24 months. That's actually pretty good. And I think we were lucky. There's always an element of chance as you create open standards, you have to be at the right place at the right time. And it's hard to predict the future, so you don't always hit that window. But I think OpenXR absolutely hit that window. I think it was widely recognized as a solution to this fragmentation problem. And, you know, any 1.0 spec is not going to solve the complete space, but it was enough. functionality that was solving real-world problems, that we were fortunate and we got very wide adoption. And till now, just a few years later, pretty well any XR device that's shipping will support the OpenXR API. It's one of the most convincing adoption curves out of any of the Khronos standards that we've done. And we're fortunate, and this is where Alfredo can speak more, we're fortunate we have many of the world-leading experts in XR in the working group figuring out how to best enable developers and how best to figure out what to do next. You know, the roadmap is a very interesting discussion that the working group is having right now.
[00:06:44.304] Alfredo Muniz: Kent, the way we like to talk about OpenXR is from the perspective of an application developer. So when you want to develop an app, there's all these different headsets. Before OpenXR, you would have to go into each headset's libraries. You know, for the MetaQuest, you would build your game, and it would only work on the MetaQuest because it's their code, their library. So if you wanted to use it on, you know, say a Microsoft, it just wouldn't work. You'd have to start over from scratch. which means you would need multiple teams to develop for each and every platform. That was the problem before OpenXR. So what we're trying to do is basically prevent that. The application developer, all they need to care about is making their app, and it'll just work the same way consistently across all these platforms.
[00:07:38.258] Kent Bye: Great. So this conversation will be airing on April 15th, 2024, where you're going to be announcing and launching the open XR 1.1. Maybe just give a brief overview for what's new from 1.0 to 1.1. Yeah.
[00:07:52.369] Alfredo Muniz: So we talked a lot about in the working group, how to do our messaging for 1.1. So we want to make it clear that it is a minor update. So in OpenXR, we have major, minor, and patch releases. And this class displays as a minor update, which we want to do every single year from now on. So to me, OpenXR 1.1, the biggest change is signifying to the community, the XR community, that we are now focusing on standardizing all these extensions that we built up over years of experimentation Just cutting that list of, um, I think we have like one 60 now at this point from all different vendors that do kind of the same thing and condensing that down. So it's going to be a multi-year journey, but we're starting. And I think we're starting on the right track with clarifying a lot of things that were unclear in the spec before we're consolidating a couple of eye tracking that we've done. and really just making it more easy for developers guiding them. There's multiple ways to do things. Well, there were in 1.0, but now with 1.1, this is the way that things are done.
[00:09:09.755] Neil Trevett: Yeah, I think as our predecessor, it's not like a huge lump of new functionality in one. There is some, though, we can talk about that. But I think it is significant that we've gone through this kind of long gestation period with OpenXR 1.0 being the core release. And it's been good. The flexibility of OpenXR has enabled multiple extensions to ship and use cases to be explored, new functionality to be tested. And that's been, I think, a really productive period, but that needs to now balance with consolidating those extensions. Because as Sergio says, in many cases, there's multiple ways to do things on different platforms. That's not the point of a portability standard, right? You want one way to do things across multiple platforms. And I think the fact now that the working group has set itself the path forward, again, as Alfredo says, on a roughly annual basis, we'll be bringing out new core specifications that gives us the opportunity to consolidate, to get better portability, you know, reduce fragmentation. But that's not to say that the experimentation will cease, but I think it'll be in better balance. There can be extensions to do the exploration, hand-in-hand with the regular consolidation to give the portability that the developers really need.
[00:10:31.487] Alfredo Muniz: it'll definitely be quicker. So we'll be experimenting much earlier on in the process of thinking of new functionality. So now we're working better as a group, where when we do come up with new ideas, we pitch it to the group before we release a vendor extension and we first try to reach consensus, at least on an initial thing, so that later on, we can bring it together much quicker, much easier.
[00:10:56.998] Kent Bye: Nice. And you sent over some slides. Maybe it's worth starting with the engines and the libraries, because I think most people who are going to be developing with OpenXR are going to mostly be interfacing with Unreal Engine, Unity, Godot, and there's a number of different other platforms that are listed on here. So it's been implemented in Unreal Engine since version 4.24. and Unreal Engine 5.0 also supports OpenXR. It's been Unity's OpenXR plugin available since 2020 LTS. Godot provides OpenXR support since March of 2023, that's in core 4.0 alpha 4. Autodesk vRed library, OpenXR supported since 2023.4. NVIDIA, Omniverse, and CloudXR platforms, and then WebXR that's in Chrome, Edge, and Firefox are using OpenXR as the default backend. And then the Calabro Mondando open source open XR implementation. There's a lightweight XR meta XR simulator to speed up unity XR development. And then there's a stereo kit from Autodesk open source mixed reality library for building HoloLens and VR applications. And so Yeah. Maybe you could just talk about the landscape of different game engines, integrating open XR and using it as a way that they're interfacing between the input of the device into how it's actually being executed within the programs.
[00:12:14.659] Alfredo Muniz: Sure. Sure thing. So you start off with the hardware just above the hardware is like the operating system. And those are still proprietary. You know, you go into a, an interface for these VR systems and they're going to be the company's interface. But that communicates with the hardware. However, to access that hardware, OpenXR serves as a translation layer on top of the operating system. So it's a way to access the hardware, but not all of it. We only expose what we want application developers to use. Sometimes for better or worse, application developers such as these engines and browsers and libraries here, they can only do so much with the hardware. and only things that are exposed through OpenXR. Even though there's obviously more to the hardware that can be done, OpenXR, in a way, kind of limits what you can do in exchange for the compatibility you get with all these different hardware systems. So yes, OpenXR is mainly for the engines, the browsers. They're the ones that will read the specification, understand it, and then implement it in their engine browser so that the application developer, the one that uses this library, can more easily write their code. It's a layer upon layer type of system.
[00:13:39.108] Neil Trevett: Yeah, I mean, I'd add one thing to that. I think, you know, in the early days, again, any 1.0 spec, yes, is making compromises to get that minimum viable product, 1.0 spec, out of the door. But a successful standard and OpenXR is, I think, by any measure, a successful standard, and it's going to get even more successful with the work of the Working Group, can actually prevent lowest common denominator, because the Working Group becomes an effective forum where hardware vendors who, without the working group, would be siloed in their own labs, they actually get to talk and collaborate as to how we can deliver a roadmap of functionality that, because we have multiple companies providing prospective resources and input, as the specification gains in adoption, we can actually speed up the deployment of new functionality. And I think you do see that, again, coming back to this seismic shift now. We're moving away from just doing the vendors doing individual extensions and then we have to consolidate to the working group actually figuring out what the roadmap should be and increasingly working together to deliver that in a timely way. So a successful standard is not a lowest common denominator. It can actually be an accelerant to having an aggressive roadmap, you know, shipping in a timely way into the industry.
[00:15:06.555] Alfredo Muniz: And we do have a few of these companies here in the working group. They help us because they give us the perspective from the application developers themselves. Before OpenXR, as I said, they were fragmented. You would have the program for each individual headset. Those vendors, they each had functions that they exposed, hardware that they exposed. And OpenXR 1.0 It was kind of lacking in that we took some of those things away. And a couple of these people that use Unreal and Unity, they were complaining. They were like, we switched to OpenXR and now I can't do this thing that I really need for my game. 1.1 addresses a couple of those, but we are still working on bridging the gap to make it more, I guess, filled so that people don't have to think about the past anymore and they could just think about OpenXR.
[00:16:01.390] Neil Trevett: Yeah, you have a job for life Alfredo if you want it. Keep that roadmap going.
[00:16:08.972] Kent Bye: Yeah, just a follow-on question about the support, because I know that OpenXR has the core, which you have the 1.0, now you have the 1.1, and then there's also extensions, and so for each of the support, does that mean that if it has OpenXR support, that essentially if you wanted to add in an extension, then it's automatically going to be implemented? Obviously, if there's no eye-tracking and you put in an extension for eye-tracking, you're not going to be able to use something, but I'm just wondering if there's other things on the hardware vendor side that have to happen in order to support specific extensions and maybe could talk about how that works between the default baseline of OpenXR versus how do you know whether or not a specific extension may or may not work?
[00:16:51.932] Alfredo Muniz: Yeah, great question. So first, I want to make it clear that OpenXR is built into the hardware on these devices. We call it the runtime. So whenever you want to interact with an XR system, it's through OpenXR. People may call it something else, but it always has to get translated back to OpenXR and then to the hardware libraries that are used on the runtime. To answer the question about extensions, so that's the whole point of OpenXR is to be able to bring extensions into core because OpenXR extensions are optional. And by optional, we mean not everyone has to implement them. If there's a cool feature that the meta has, but Microsoft doesn't, then too bad. It's only going to work through an extension on that one particular headset. However, by moving things to core, it is now required by everyone. So it is kind of limiting in that hardware is naturally different. So there are certain things that are just impossible with hardware. because they don't have the right sensors to actually be able to do that functionality. So we can't force people to implement extensions, but part of OpenXR is to minimize those extensions and bring as much as we can into core so that we can have a standardized set of functionality that's available everywhere.
[00:18:23.600] Neil Trevett: Yeah, that's right. I mean, for developers that are into 3D as well as VR, and it's some overlap, of course, it's a fairly familiar model. OpenGL and Vulkan and OpenGL ES and the other APIs have a very similar extension model. And exactly as Alfredo says, most of the extensions for most of the other APIs are also optional. So you get to interrogate your runtime to find out what extensions are available in that particular runtime on that particular piece of hardware before you can use them. And developers are used to that kind of discovery process, just like on GPUs. Now, if you have a GPU without ray tracing, you're probably not going to support the ray tracing extension. It's a similar kind of deal. But again, coming back to the larger effect of a successful standard, If there are enough members of the working group that are interested in a particular piece of functionality that's not pervasively shipped yet, if you can get a critical mass of the hardware vendors agreeing on an extension and deploying it, even if it's optional, it can be a positive force for, in the course of time, getting that functionality to be pervasive in the industry. Because the other folks who didn't have that functionality now have a much reduced barrier to commercial success because they say, oh, it's worth me investing in this piece of functionality and bringing that into my device or my runtime. Because there's already an API, and the developers are out there and know how to use it. So if I deploy it, it's not a maybe, am I going to get developer engagement? No, it's a well-trod path already, and I get immediate return on my investment.
[00:20:07.948] Alfredo Muniz: And then that company is able to take advantage of all the applications that have been built using that extension.
[00:20:16.104] Kent Bye: Okay, so it sounds like at a baseline at any of these companies and it's worth I guess probably mentioning some of the open XR conformant devices where we have like the Microsoft HoloLens mixed reality headsets with hand and eye tracking extensions that are available. Meta Rift S, Quest 3, Quest 2, Quest Pro. And it sounds like they've deprecated their own OpenXR API for Quest 1 was no longer going to be supported moving forward. I saw some developers saying that they were going to be adding in like the OpenXR APIs into their app so that if they pushed out the application, it would still have support for Quest 1. So I guess that's an option for Quest developers if they still wanted to do that. For HTC, there's the Vive Focus 3, Vive Cosmos, Vive XR Elite. Vive Wave Runtime, the Valve Index, a note here that says that Valve deprecated their own OpenVR APIs for OpenXR, the Vario headsets are all fully compliant with XR3, XR4, the Canon, Mreel, X1, Magic Leap 2, just announced recently that they're moving over to OpenXR, Xreel, Air 2, Air 2 Pro, Air 2 Ultra, Snapdragon Spaces from Qualcomm, their whole development platform is all based upon OpenXR, And then there's Pico's Neo3 and Pico4 from ByteDance. And then there's a couple of the spatial reality displays from Sony as well as from Acer. So yeah, I'd love to hear any comments on this broad range of different supported devices that it sounds like that they're going to be supporting OpenXR runtime on their devices and then potentially a number of the different extensions as well.
[00:21:49.058] Alfredo Muniz: Yeah. I want to highlight that one of the big things with OpenXR is back compatibility. So even though a lot of these native APIs, that's what we call them, have been deprecated by the hardware vendor, OpenXR, we want to make sure that any revision that we make, whether 2.0, 3.0, and beyond, is always compatible back to the beginning, just so that we're not losing people's work. It's very valuable, and it's very important to the group and our specification editor that we maintain that compatibility. So there are a number of different classes of devices in XR. From the runtime perspective, again, that translation layer that translates what you can access in hardware to the outside world. We mainly focus on the headsets that you wear, the VR, AR, MR. And recently, a new thing has been the spatial displays. So think of like having a screen in front of you, but instead of it being a flat screen, it's a 3D screen. So those also have OpenXR runtime components for them. The other thing are, we mentioned the Snapdragon Spaces by Qualcomm. Those are development kits. So if people wanted to, for example, make their own VR system, they would be able to use that knowing that it's compliant with OpenXR, so that it's possible when they build out and add their hardware, it'll be OpenXR compliant.
[00:23:24.125] Neil Trevett: Yeah, and this is kind of the heart of the adoption curve for OpenXR. And of course, all these devices are currently conformant to 1.0 because we don't have the conformance test quite yet, but they're very close. And of course, each company will hopefully upgrade to 1.1, but they'll do that at their own commercial pace and their own resources and roadmap. But we would hope certainly that all of these devices over time would strongly consider keeping up with the latest version. I think the interesting thing around, if you look at diverse devices that support OpenXR, it's quite diverse in terms of operating system form factor. We have see-through, we have pass-through, we have screens, we have wearables. And I think that OpenXR, the design has stood up well to being deployed on a bunch of different types of devices. We're not overly bound to one specific kind of device configuration, which is good.
[00:24:20.495] Kent Bye: Yeah, I think one notable thing that's missing on here is Apple and the Apple Vision Pro is not implementing OpenXR in any fashion. And so I know that there's been some people who have been implementing things like ALVR, which is like an interface to use like SteamVR with the Apple Vision Pro, and it's using the OpenXR, but yet on the hardware side, you're not able to necessarily use the full capability of OpenXR on the Apple Vision Pro. Even with Unity, they have their poly-spatial and AR foundation that is interfacing with the Apple Vision Pro hardware, but yet there's no direct integration into OpenXR. So I guess the dream of having the ability to write your application once and have it across all of the devices. Obviously there's going to be some devices like apple that decide not to implement it, but love to hear any comments or reflections on if there has been any discussions with apple or if they're pretty strongly set against any integrations with open XR. So yeah, I'd love to hear any reflections on that.
[00:25:22.807] Neil Trevett: So there's been no specific discussions with Apple. I mean, as always, Apple would be very welcome to adopt OpenXR if it suits their needs. I mean, the Vision Pro is an awesome device. It's interesting. Time will tell whether OpenXR adds value to Apple's business. From the Khronos point of view, it is interesting. We do have some examples of helping developers bridge that gap when the native API isn't available everywhere. The best example currently, though, is bringing Vulkan onto Apple platforms over Metal with the Molten VK translation layer, which actually works very well. I don't think there's been any serious consideration yet whether a molten XR would be feasible or desirable. But it is an interesting analogy, maybe, that maybe that kind of thing will happen in the future. But we'll see.
[00:26:19.210] Alfredo Muniz: Yeah, I agree with Neil. We'll have to wait and see. There is interest among the group because our goal is to make it easy on the developer, where they're able to just write that one app and it just works. But there's still a lot of steps if, and we don't even know if it's possible at this point.
[00:26:38.625] Kent Bye: Nice. Yeah. And Neil, I did come across a comment one of the representatives from Apple had made from the WebGPU meeting back in December 12th of 2019. Maciej Stachowiak said that Apple is not comfortable working under the Khronos IP framework because of a dispute between Apple Legal and Kronos, which is private. You can't talk about the nature of that dispute. And so, yeah, I just was wondering if there's anything more about that, that you could give any additional context on the sort of historical relationship between Apple and Kronos group.
[00:27:11.469] Neil Trevett: I've heard reference to those minutes. I wasn't in that meeting. But no, we have a good relationship with Apple. They're on the Khronos board and they're active in the WebGL working group at Khronos. So if there is a problem, I would ask the Apple people to come and we can get it sorted out. No, we're working closely with Apple. We have a good relationship.
[00:27:33.349] Kent Bye: OK, well, that's good to hear. And I know that with even the WebXR implementation that most of the Chromium-based browsers have decided to go with OpenXR as their interface, but you don't necessarily have to use OpenXR to interface through WebXR. So it sounds like that Apple is doing something completely different with their WebXR implementation with Safari. Is that correct?
[00:27:54.642] Neil Trevett: I actually don't have knowledge, but I suspect so. Yes, I expect so.
[00:27:59.885] Alfredo Muniz: Yeah, that's how it would work.
[00:28:01.743] Kent Bye: Okay. Well, maybe it's worth also digging into some of the new things that have been promoted from the realm of extensions into actually getting integrated into the 1.1 specification. And so maybe you could go through some of the new things that are going to be added into this new open XR 1.1.
[00:28:20.130] Alfredo Muniz: Sure thing. So a couple of things that we'd like to point out, and you'll see this in the press release also releases on Monday, April 15th. One of them is local floor. which is a pretty cool way to basically recenter the user space. Before, we had a confusing way to do it, where you wear the headset and you want to reset to a specific position. There were multiple ways for the app developer to do that. And now we made it very clear which one you should use, and we introduced a new one that's useful to that very specific situation. So we've actually written a blog post about it I think will also be released on the 15th. So be sure to check that out if you ran into issues with the local floor and how we defined it in OpenXR, because that'll help a lot. We added a new primary view configuration, which is stereo with foveated rendering. So this is a way for runtime vendors to standardize how they do foveated rendering, essentially. It wasn't available before, and now it is with 1.1. This used to be just a extension by Vario, but the group decided that this will be useful for everyone. So now we're all doing it.
[00:29:41.146] Neil Trevett: And that includes eye tracking optionally as well, right?
[00:29:44.007] Alfredo Muniz: I think. Yeah. Yeah. We have eye tracking as a different extension. We haven't yet standardized it. We haven't yet put it into core because we're still working out issues of privacy. Okay. Interesting. Yeah, different benders have different ways. Not everyone is comfortable with exposing certain parts of the eye.
[00:30:09.584] Kent Bye: I know that within Unity's poly-spatial, I don't think that there's other way to do like coveated rendering experiences, unless you're using Unity's poly-spatial plugin for the Apple Vision Pro as an example. So they have it locked down likely because of some privacy restrictions. But yeah, it sounds like because this is the 1.1 core and you want to have everyone use it. that there may be some difference in architecture that people have based upon how they're implementing eye tracking. But it sounds like you would at least have to have at least some information for foveated rendering, right? I mean, because you have to know what to be rendering. So I'm just wondering how you're able to, do you have to have the extension for eye tracking in order to use the core feature of foveated rendering then?
[00:30:55.650] Alfredo Muniz: So let me back up a bit and then go back to your question. So, The way that we work in the working group is, you know, we're not here sharing code on how internally things work. We don't do implementation details. We develop things internally. All that code is secret proprietary. OpenXR, we release a vendor extension, and it basically has the inputs and the outputs. So that's all that we're seeing as different companies. We don't actually know how it works internally, if that makes sense. In a way, and this is for IP reasons, right? Because we don't want to step on each other's toes or violate each other's IP. It could be that someone does eye tracking a lot better than someone else. And the reason for that is in the implementation details. You know, it could be as simple as that.
[00:31:50.203] Neil Trevett: And if I can make a, you're absolutely right, Alfredo, but let's be clear, that's a feature of the open standards, not a bug. And it's the core difference between an open source model and an open standard model. Any successful open standard is gonna have a level of abstraction. So exactly as Alfredo says, we're not specifying how you implement it, we're just specifying the commands and the results. And that enables multiple implementations. Sometimes you want open source. If everyone's just saying, oh, we'll just pile on and have one implementation of this thing, open source is great. But if you want to have healthy competition, lots of parallel innovation, Darwinian evolution, you want to have a whole set of devices like OpenXR has. You absolutely need to have an open standard. So it's very complementary to open source.
[00:32:41.872] Alfredo Muniz: Yeah, Ken, what was your original question?
[00:32:44.834] Kent Bye: So we were originally talking about the new stuff, then I asked around the eye tracking, how it seems like in order to have foveated rendering work, you would also need to have at least some extension for eye tracking. And so if one thing is in the core, but this other thing is not in the core as an extension, then how those things talk to each other, or if there's still haven't worked without having the extension. Cause if foveated rendering needs eye tracking, but it's eye tracking module may be separate. So just navigating two dynamics of the core functionality versus the extensions.
[00:33:18.922] Alfredo Muniz: Yeah, exactly. I'm not sure of the implementation details. You know, my company doesn't have a runtime, so this was not something that I've implemented personally. So I think that's about all I can say on that, unfortunately.
[00:33:33.577] Kent Bye: I know that in the blog post that you'd sent out, there is a lot of different things that you're listing in terms of, you know, things that have been promoted, or at least now are going to be made available in OpenXR 1.1. And I don't know if there's more things that you want to talk about in terms of some of those either new extensions that have been promoted or some of the additional functionality that you're highlighting with this 1.1 release.
[00:33:53.775] Alfredo Muniz: Yeah, we can talk about a little more. So another one. Someone made a YouTube video saying how when you hold a controller, all of our poses are, they just don't make sense. So we fixed it with this new one called a grip surface. When you're holding a controller, it's the point at which your hand touches the controller, specifically the center of your palm. So it's a new standard pose identifier. It's to resolve issues just like that, where people don't know which one to use for which situation. Now we've made it clear, this pose identifier for this, this one for that. OpenXR 1.1 is filled with a number of those where mainly we're clarifying for the application developer how best to do things given certain situations.
[00:34:46.676] Neil Trevett: Yeah, and again, it's not surprising, you know, a 1.0 spec or any of the standards often needs clarifications once people have tried to actually use it. And very often the clarifications and better non-normative explanations around the functionality is just as important as having the functionality itself, making sure that everyone understands how to use it consistently.
[00:35:07.334] Kent Bye: Yeah, I've definitely come across that and going from different XR programs where their pointer all of a sudden is now pointing in a direction that you're not expecting because it's got a different orientation. So we talked a little bit about the open floor, the stereo with coviated rendering, the grip surface. It's also a couple of other ones with XRUUID, which is implementing the IETF RFC4122, which is basically an open standard for common data type, unique identifier. Maybe you could give a bit more context for why that is going to be helpful.
[00:35:38.525] Alfredo Muniz: Yeah, so that's mainly helpful because there are certain components now, and we're also setting the framework for the future with this new common data type is what we're calling it. But being able to identify things uniquely, and when I mean things, I mean things in space that your VR system could pick up. such as objects, whether they're real or virtual. We like to abstract in OpenXR the hardware. So we don't want to say that you're holding a Vive controller. We just want to say you're holding a controller and it has these buttons. We want to extract away the hardware. And part of that is also this UUID, which is we don't want people to have to call back certain things in a different instance. So we don't want it hard mapped to this is controller number one, which internally I have mapped as a Vive controller. Instead, we want to give it a new UUID every time you start a session. That way people don't rely on these identifiers that they assign to be able to make their app less compatible with newer hardware.
[00:36:48.362] Kent Bye: Yeah. And I saw that there was other interaction profile improvements. And so because there are a number of different controllers and profiles that you have interaction profile paths and standard identifiers, standard component and output paths. So it sounds like all of that is around. If you want to have multiple different types of controllers that you want to integrate with it, then you'll be able to swap in between your controllers and have different default profiles. That's at least my understanding.
[00:37:16.319] Alfredo Muniz: Yeah, the interaction profile improvements. We just took everyone's controllers that were in OpenXR before as extensions, and now they're just there. So you no longer have to add in interaction profiles for common controllers. They'll just work.
[00:37:32.462] Kent Bye: Nice. And I see a number of different other fundamental improvements and open XR 1.1 that are providing developers with an extended set of universal tools, including the XR duration event polling to call idiom and new structures. Maybe you could just say a few words about some of those tool set improvements that you're also shipping.
[00:37:51.247] Alfredo Muniz: Yeah, so this was more for the runtime developers or runtime implementers. Essentially, as Neil was saying, when you release a new spec, there could be problems, errors, and it leads to inconsistencies between the runtimes, which means that if someone tries to run the same code on different runtime, it's not going to work the same way, which is what we absolutely want to avoid with OpenXR because we're saying it's a standard. There isn't much to say from the developer point of view. It just makes things easier for the runtimes to all behave the same.
[00:38:28.235] Kent Bye: And the last section in the updates talks about specification refinement, providing clear explanations of specification intent, including like the spaces, adding links to reference spaces, the rendering input. Maybe you can comment a little bit on the other refinements that are also coming in OpenXR 1.1.
[00:38:48.220] Alfredo Muniz: Yeah. So the whole spaces chapter in the OpenXR specification, we like to talk about it in terms of different chapters. So there's like input, action system, rendering, all sorts of different chapters. One of the ones that got the biggest facelift was spaces. And again, this was more for runtimes. We made it a little more efficient. So one of the extensions that we consolidated was XR locate spaces. Before, we had xr.locate.space singular. But when someone wanted to do an operation that required multiple spaces, such as if you're scanning your environment, you're picking up a lot of objects, each of those objects will have their own space. The problem is you would have to call this function n number of times for each object that you located. So xr.locate.spaces, you call it once and it gets all of them in one go. So it's increasing the efficiency of how we do space calculations in OpenXR. We really just made it clear on what is the role of the runtime, but also the app developer in terms of what they should be doing. The OpenXR specification focuses mostly on runtime developers. But there are certain sections of the spec that actually tell the application developer what they should do. We can't control what the application developer does. So in the specification, we have a lot of must. Runtime must do this. Runtime must do that. And those are things we can actually test. And every time a new patch release comes out, the headset vendors. You have to run a conformance test suite. And these are things that are tested automatically. If they fail, then we know that it's a problem with their code, and we can go back to, you know, you failed this test, you need to look at this paragraph in the specification and change that. But we cannot require that for application developers. They're able to do whatever they want. But the spaces section makes it clear what they should be doing.
[00:41:05.805] Neil Trevett: That's really interesting, just listening to Alfredo talk. I mean, again, any open standard is basically a contract between two entities, the runtime implementer and the application developer. So a good standard specification will try to make things clear to both people using that contract. But I think Alfredo's right. Typically, the Khronos standards tend to focus more on implementers. We're all trying to put more attention to educating the developers, too, so the contract is effective between the users and the implementers. It's a problem for the ages. It's always a thing.
[00:41:45.094] Alfredo Muniz: Part of that was we released OpenXR tutorials earlier, just like a month or two ago. It is geared towards application developers on, hey, these are best practices for how you should implement our specification from the app developer side. Because a lot of our resources are to the runtime, as Neil will see, which is good. But again, it's there's two sides.
[00:42:10.252] Kent Bye: Nice. Yeah. It was just, as I look at all the different announcements, it seems like it's a minor release, a lot of updates and quality of life types of issues, like all the things we're talking about, but also a lot of more mixed reality type of features that are getting promoted up from the extensions and have enough industry momentum to start to integrate things around the space. I don't know if that's an accurate summary.
[00:42:33.302] Alfredo Muniz: Yeah. And that's where we're headed with OpenXR. What we're talking about right now is a spatial reasoning. How do we take what's in our environment and how do we standardize that?
[00:42:45.190] Kent Bye: Great. And, uh, and finally, I'd love to hear from each of you what you think the ultimate potential of XR might be and what it might be able to enable.
[00:42:55.097] Alfredo Muniz: The ultimate potential as in like the killer app?
[00:42:58.905] Kent Bye: Just where you see, yeah, like the things that get you excited about the technology and it doesn't necessarily have to be like one single app, but what you see is some of the more exalted potentials for where this could all go.
[00:43:09.193] Alfredo Muniz: Uh, yeah. Um, Hmm. So I feel like every year my definition changes because every year I'm seeing things that are, that were impossible just, just a few days ago. So it's, um, a really exciting space to be in. I think the biggest thing for me is I can't wait for the point where it becomes invisible, where I'm able to wear this XR device. And I don't even know it's there because it's so light, it's so small. And it's like carrying around my phone, I guess. But without having to use my hands, I just talk, use my eyes. And I think that's what I see is where we're headed. And I hope where we can make it one day. is being able to just replace our cell phones with these things on our face.
[00:44:04.513] Neil Trevett: Actually, I was going to say something similar to that. If you're talking long-term, I think that that is the goal that lots of people have. Obviously, there's lots of awesome use cases for VR, but it does isolate you. And AR brings you into the real world, and you'll be able to use it all day, every day. And as Alfredo says, if it becomes invisible enough, you're not constantly thinking about using it. It can definitely transform. the way we interact with our technology in the long distant future. Eventually you can imagine AR in glasses like these could easily replace the mobile phone and then you mix in AI and you'll be smartly interacting with your agents in your headset. It is going to be a radically different way of using the technology that we have. And we haven't used the M-word yet, so I'm going to do it. That's kind of the metaverse-y vision, is XR becomes the user interface into artificial intelligence. So it's going to be a fascinating road.
[00:45:11.595] Alfredo Muniz: And I think of the early days of like YouTube, where a lot of the videos were just how-to videos. And I'm thinking, you have this thing on and it's able to identify what you're doing. And it can actually tell you how to do it. Walk you step-by-step using some AI where you may just be not trained to perform some maintenance on your car, but now all of a sudden you're a professional.
[00:45:38.911] Kent Bye: Is there anything else that's left and said that you'd like to say to the broader XR community?
[00:45:43.960] Neil Trevett: Thank you for the opportunity to talk about OpenXR 1.1. We hope the developers enjoy it out there. And we're really looking for feedback. Feedback is vital. So let us know the good stuff and the bad stuff. Alfredo and the working group, really listen to any feedback that we receive. It's a vital part of the evolutionary process.
[00:46:03.728] Alfredo Muniz: Yeah, drop us a note in the OpenXR Discord server. We're always looking at that. And yeah, as Neil says, feedback. good, and also bad. You know, any negative constructive feedback we will take to heart.
[00:46:19.494] Kent Bye: And the blog post announcement says provide OpenXR feedback to the OpenXR discord channel, the OpenXR forums, or the GitHub issue tracker. So each of those different places sounds like a good place to go. And I guess the roadmap is also going to be discussed where the roadmap being discussed.
[00:46:35.517] Alfredo Muniz: I'm not sure if we have it published yet.
[00:46:40.180] Neil Trevett: But future-looking topics typically get discussed on GitHub, because people can say, are we going to do this? And there's a discussion. We don't put draft specifications until they're ready, because it could be confusing if we change or backtrack. But general roadmap discussions, I think there's a bunch of interesting GitHub issues discussing various aspects. So that will probably be the best place. And then Discord, of course, a bit more freeform.
[00:47:06.533] Alfredo Muniz: Yeah, I think this is our first release in a long time. So there are still a couple of things we're working out. We do have a roadmap internally. In terms of sharing that with the public for review, I think that's a really good idea. And I think Neil will take that back with us and see if we can maybe start that off as a blog post or something.
[00:47:30.509] Kent Bye: Cool. Yeah. It was a, the last paragraph in your announcement was about the open XR roadmap and call for community feedback. So cool. Yeah. It sounds like as you're gearing up towards the announcement, then I'm sure that folks can refer to the blog post and get more information and join the discord and get more information from there. So. Awesome. Well, Neil and Alfredo, thanks so much for joining me today to give a little bit more update as to the latest and greatest from the Kronos group and the OpenXR 1.1 launch that's coming here on April 15th, 2024. And I guess from the vendor side, it's going to take time before they start to implement it, but. Really great to see how far this has come and how, Neil, I know you've talked about the closed versus open and how really the open model here is really on the forefront and leading above and beyond any of the proprietary solutions they're doing. And so it's really great to see how much buy-in you've had from the XR industry. And it's been a huge part of helping to grow XR and make it what it is today. So thanks again for all that you're doing with OpenXR and for joining me today to help break it all down.
[00:48:31.371] Neil Trevett: Great. It's good to see you again.
[00:48:33.419] Alfredo Muniz: With pleasure. Thanks so much, Ken.
[00:48:35.821] Kent Bye: So that was Alfredo Munez. He's the OpenXR Working Group Chair and also the founder of Zeed LLC, as well as Neil Shervat, who is the president of the Kronos Group. So I have a number of different takeaways about this interview is that, first of all, well, it's just really exciting to see how much OpenXR has been adopted across the XR industry and has made a huge impact for trying to standardize all the different SDKs and has been able to get pretty broad adoption on all these different devices that we talked about. Also across all the different game engines that there's different ways of interfacing with OpenXR. Probably the biggest missing device is Apple, who has decided to not integrate OpenXR into the core SDK of how their headset works. And I think that's Pretty typical from how you see open standards be developed is that there's usually a proprietary solution that's out there. Usually the proprietary solution comes first and then there's other standardizations that happen later, but this has been flipped where the industry has broad consensus around OpenXR and Apple has decided to be a little bit more of a closed walled garden. But as Neil said, there may be other ways of having some sort of interface where you could connect into some of the existing OpenXR specifications. And that's something that ALVR has already started to do with having application that could be on the Apple Vision Pro and start to use some of the different OpenXR interfaces, but it's way different than having native device support where you could add a whole variety of different extensions across these different devices. And so it's going to be pretty locked down in that respect. But overall, I think if you look at how much OpenXR has been adopted across the industry, then it's pretty impressive to see all the different both applications that have started to integrate it, as well as the game engines, as well as all the different conformant devices. So there'll be more information that's going to be released either within the slides or within their blog post. They say that what things are coming soon are like extending hand tracking, enhanced handling of spatial entities, expanded haptic support, controller window models with GLTF, increased accessibility, and metal macOS support. So like Neil said, it's been one of the most successful Kronos Group open standards that have come out so far, just in terms of the broad adoption curve across the XR industry. It's pretty impressive just to see all the different companies that are helping to develop this across all the different major headset manufacturers. And if you think about some of the different claims from Meta in terms of how much they've been supporting some of the different open standards Their very early support for OpenXR and integrated into some of their main ways of interfacing with the XR headsets is a real testament to these devices being a real computing platform that can interface with other computing platforms and not be this closed walled garden approach. So very exciting to see where they take this all in the future. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. If you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a list-supported podcast where I rely on donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
