Here’s my interview with Brian Vogelsang from June 1, 2023 at Augmented World Expo when he was still Qualcomm’s Senior Director of Product Management for Snapdragon Spaces and Developer Ecosystem. We cover a lot of Qualcomm’s announcements from AWE 2023, including the dual-render fusion features for tethering the touch-screen interfaces to AR experiences that could be seen on different head-mounted glasses form factors. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR podcast. Hello, my name is Kent Bye and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continuing my coverage of looking at AWE past and present, today's episode is with an interview I did a couple of years ago with, at the time, Brian Vogelsang was Qualcomm Senior Director of Product Management for Snapdragon Spaces and the developer ecosystem. And so we talk about a lot of different announcements that were happening at Qualcomm AWE in 2023 and a lot of it was on this kind of expansion out of using Snapdragon Spaces for the different OpenXR extensions and just generally talking around what's happening with the XR ecosystem because Qualcomm is really at the center of providing the chips for all these devices that are essentially everyone except for Apple or one way or another using these Qualcomm chips for XR devices. So Brian's actually with Android XR now, so he's with Google. But this is from a couple of years ago when he was still with Qualcomm. And I just had a chance to sit down with him and go over some of the different types of announcements that they're making, but also their strategy for building out this developer ecosystem and using OpenXR extensions with Snapdragon Spaces. And they were also doing a lot at the time with this kind of split between like the phone and also the headset. And so there's still this idea of, you know, I think at the time starting to think around like the external compute device and what if that were your phone and what if you started to use the affordances of the touchscreen as another extra display, but more than anything else, a touchpad to start to engage and interact with. So a lot of the different demos that were at AWE a couple of years ago, we're looking at this kind of new novel mechanic of using your phone to start to navigate different spatial interfaces for immersive technology. So a lot of people are saying, well, maybe the smart glasses and AR is going to replace our phones. I suspect we're still going to be carrying around the phones with us, especially because it's extra compute and technology. I just feel like we're not going to be getting rid of our phones anytime soon just because they're so useful and we're nowhere near to replacing all the utility. I think it is kind of like if you were going to work, you would usually want to use your main laptop and maybe you'll be able to get by by using some of your tablet or your phone. But depending on the work and the context, sometimes you just want to use the fully capable devices. And I feel like it's going to be a similar thing with the phone where we're more than likely still going to always have a phone, I suspect. But there will be some use cases where depending on what you're doing, you may not need the phone and you could just go with the smart glasses or AR alone. But I think it's probably more helpful to think around the spectrum of having the AR devices as an extension, an immersive thing that you can do with your phone from both a compute perspective, but also a human computer interaction perspective as well, which I think is a lot of what we're covering here within the context of this conversation. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Brian happened on Thursday, June 1st, 2023 at Augmented World Expo in Santa Clara, California. So with that, let's go ahead and dive right in.
[00:03:08.660] Brian Vogelsang: My name is Brian Vogelsang and I work in Qualcomm's XR business. My responsibility is for the Spaces, Snapdragon Spaces product and ecosystem. So I lead the team developing the SDKs and the products that developers use to create their experiences in Spaces. and then also the ecosystem team, the team working with developers.
[00:03:28.954] Kent Bye: Great, maybe you could give a bit more context for your background and your journey into working in this space.
[00:03:34.276] Brian Vogelsang: Yeah, so I've been working in XR for about five years, so I started in 2017, so a little bit longer. And, you know, prior to that I was doing Internet of Things, like, you know, IoT, and kind of got my first exposure to XR tech in 2016 or so, and just was really attracted to immersive computing in general. And so when I had an opportunity, Hugo said he was starting a business at Qualcomm focused exclusively on XR. I really jumped at the fact to participate in that and get into it pretty deeply.
[00:04:05.758] Kent Bye: So we're here at Augmented World Expo and there is a big keynote that Hugo Swart was giving yesterday on the main stage and announcing some new features of Snapdragon Spaces, including the dual render fusion. And so maybe we can start there with what is this new ability to use a 2D phone to interface with a spatial interface of like an augmented reality spatial representation of that same space?
[00:04:28.615] Brian Vogelsang: Yeah, so when we think about classes of augmented reality devices, you know, we sort of have a spectrum. So there's assisted reality type glasses, you know, you have your traditional like Vuzix or RealWear type devices that don't have much immersion, really used for work instruction or remote mentor type use cases. And then the other side of the spectrum, you have devices like a more all-in-one HoloLens or a Magic Leap 2, with much more immersive capabilities. And where we've really looked to approaching the market with Spaces was kind of with the smartphone and a pair of AR glasses tethered to it. So the glasses being a companion to the smartphone experience. And sort of where we started was how do we take the perception technology and enable developers to be able to use that. So these are tracking technology like positional tracking or hand tracking or spatial anchors, depth and 3D reconstruction and building a mesh in the real world and be able to have AR content interact with that. And so we built that capability into Spaces, but our initial approach was to use the phone more as a controller. So, you know, shoot a raycast out of it, have it be a 3DOF controller to interact with the content, almost as if you're familiar with early standalone 3DOF VR headsets, they had a 3DOF hardware controller that they came with. So we used the phone sort of like that as a companion to the Air experience. And with what we're calling dual render fusion, or this capability of Snapdragon Spaces that we just introduced at the show here, we now kind of flipped it around and said, well, why don't we take what's best about the phone an input device and a display and you know have that be a core part of the experience so instead of building a spatial application from scratch where every portion of that interaction and display happens in the real world why not start with a mobile application and enable just a little bit of spatial as much spatial as you need to solve you know the use case problem that you have so this allows By flipping it around, we no longer constrain the user experience to be, okay, the smartphone is just a controller with ArrayCast. Now it's a full mobile application, and we leave it up to the developer to be able to create the best input user experience for their particular use case, as well as using the phone as a supplemental display. So if they want to provide feedback on the display on the phone to complement, or really maybe the core experience is driven through the phone and the augmented experience in the real world, supplementary is only used for the content that's you know best served by being spatial so so this is kind of a new way of thinking about how the glasses and the smartphone work together it's only possible when you have this ability to you know have the same application running on the phone be able to render to two different things to two different displays the smartphone display and the real-world display at the same time and then also capture input both spatially, so you can put your hand out and use the glasses to use a distal interaction to control something with your hand, or use a gesture to grab something, or you could simply reposition that using the smartphone if the developer chooses to put controls in there to control that object. Maybe they want to use a multi-finger gesture to move the object around in 3D space. What we find is that it's just much more intuitive and natural for people who don't have to be taught how to use a spatial interaction using, let's say, their hands or a combination of the hand and the controller. They can now just use those familiar UI paradigms that they've been working in a smartphone with for the last 15 years. and more naturally do the experience. They can also do it faster in many cases. So you think about if you're going to build an AR application from scratch and you're going to do it entirely spatially, it has to have settings and things that configure it, or you might have to log in. And now if you have to bring up a virtual keyboard or other things and then type using your hands, not on the touchscreen, but in the real world, that could not be familiar for people. So just the idea of getting logged in to a device or some applications have multi-factor authentication. Now you'd have to do multiple steps. So just doing all those things on the smartphone where it's more natural. And then at the time you need to visualize something in 3D, you know, you put that into this secondary display. So it's kind of like, that's why we call it dual render fusion because we're rendering to both displays, taking input from both simultaneously, and letting the developer fuse them together in the experience that's the most natural for their particular application.
[00:09:01.758] Kent Bye: Yeah, I wanted to share some of my own experiences of actually having a chance to try out some of these demos because the first thing that was striking was that, you know, when I'm wearing some of these AR glasses, I'm looking at the augmented screen and I'm seeing some of the different virtual interfaces that are projected in on top of the physical world. But then I look down and it's almost like bifocals where I have like a clear vision of my phone and it's like such a familiar experience. interface of looking at my phone and seeing this correlation for how I can do like a pinch to be able to zoom in or to move around. And so much more in some ways, intuitive fusions of some of these interactions and UX that's more 2D, but having a result of a 3D UI. So having this translation from the 2D screen, when I was at IEEE 2015, I was talking to Rob Lindeman and he was talking about non-fatiguing interfaces, how When you're able to kind of rest your hand onto something, if you're sitting down and you have a tablet on your lap, you're able to potentially start to move around in a way that could be more comfortable and productive, or at least start to use the affordances of the touchscreen, but in the result, have more of a spatialized interaction of that. I see some of those really early indications for how to use the existing affordances of a touchscreen, but to start to have these interactions in a spatial environment. So it sounds like that's kind of the very beginnings of where this is going to go and to continue to expand out into novel use cases in that way.
[00:10:21.580] Brian Vogelsang: Yeah. It's really quite a simple concept, but maybe not obvious. And when implemented properly like we've done, I think really kind of feels just very natural. It feels like the way that people really have always wanted to do these kinds of things. Versus having to learn how to interact with a controller, perhaps, or try to teach someone, all right, this is how you use your hands to... raycast or move an object through a gesture or something. So it's not that we don't think those things are appropriate in many instances, but it's like present the user with the right way, the fastest way, the simplest way to do it at that time. And if you can do something much more quickly and intuitively on the phone, if the phone's there, use it. So I think also the other approach we're taking is that instead of saying, all right, you have to build your ground up app spatially and think about every use case from logging in to how you change settings to reset your server or whatever you may have to do in that application to make it a fully available complete solution just start with a mobile application that you may already have and now add a little bit of spatial to it and then over time introduce more and this allows Let's say a developer, in that case, I could use an example of an enterprise developer. Maybe they're building a training application. And maybe that already exists today on mobile. They already have it deployed and it's being used, fielded to users. And now you can introduce this little area of Spatial with the glasses to it. and then over time build upon that and add more and more spatial. So now they're sort of introduced to the glasses in a way that is a little bit more, it's slower and they can get used to some of these concepts. So we're not dropping them right into, okay, here's how you use a spatial computer interface right from the beginning.
[00:12:09.295] Kent Bye: And so a lot of these interfaces were on phones, and I didn't get a clear sense for what is behind those phones. Are some of these chips, like the XR2 Gen 1 or Gen 2, can you speak about the demos that you're showing here? Does it require the latest generation of the chips, or is this something that is more of a software side that you're able to do on all of the existing chips?
[00:12:30.518] Brian Vogelsang: Yeah. So the one thing specific to the dual render fusion capability of Snapdragon Spaces, that does require the smartphone. So Spaces runs on VRMR headsets. It also runs on standalone AR glasses. So at the show here, we announced together with a couple partners, DigiLens and TCL, Rayneo, they're building Spaces support into their standalone AR headsets. So in those cases, they don't use the smartphone. You know, their apps are running all in one on the standalone headset. AR headset. So they wouldn't have the dual render fusion capability, but for the, what we call a viewer, which is a pair of glasses that work in collaboration with a smartphone, let's say wired today, wireless is where we're headed, they can all take advantage of this capability. So it's really the smartphone chip that is running the experience. And so that's a non-XR chip. It's a Snapdragon 8 series. In the case of the devices that we have Today, like the Motorola smartphone that's tethered to the A3 glasses, which is what we're, there's a Lenovo A3 glass Motorola smartphone that we're doing a lot of the demonstrations on. This is also the same dev kit that's been available to the Snapdragon Spaces developer community for the last year. They can all take advantage of this instantly. So the glasses have a chip in them. In the case of The Lenovo A3 glass, it's an XR1, but that chip is really you doing more perception capabilities, so it's doing some of the positional tracking things. As we evolve to newer chips like the AR2, which is a recent processor we announced dedicated for AR glasses, that chip is taking on more of the perception workload and doing things like reprojection and hardware processing. But the developer, they don't really know about any of that stuff because it's obfuscated away. They just code in Unity, Air Foundation, with the Spaces plugin and SDK, or in Epic's Unreal Engine plugin on the smartphone, and the rest of it's sort of abstracted from them.
[00:14:25.014] Kent Bye: Yeah, I wanted to elaborate on that a little bit because I know that the Snapdragon Spaces is this SDK where you're interfacing with these chips like the XR2. What are the ways that you could interface with some of the CPU and the GPU? And sometimes there's neural rendering capabilities on some of these chips. I know I was talking to Nenea Reeves and she was saying that they were using some of the Spaces capability to do AI processing on some of their different applications. So I'd love to hear what are the different types of ways that the developers can get either lower level access to some of those capabilities of the chips or just if it's abstracted out that they don't have to worry about it and they're able to use the capabilities. So yeah, I'd love to get some clarity on some of the capabilities of what the Snapdragon Spaces is able to do with the chip sets that are available.
[00:15:10.131] Brian Vogelsang: Yeah, so a lot of what Spaces delivers to a developer is access to perception technologies, so it is the ability to create a 3D reconstruction of the room and have a spatial understanding of it mesh the real world so that the virtual objects can interact with the physical world in a way that behaves the laws of physics and that sort of thing. or tracking the hands or the user's position in that room, finding things like planes in the environment so that developers can understand, all right, where are the walls, semantic understanding of the real world, that sort of thing, or tracking hands or controllers. These are the things that Spaces provides primarily through the SDKs in Unity Now, the applications themselves, if we take, let's say, one of these dual-render fusion-based spaces apps that's running on a smartphone, it's a regular Android app, you know, it's a regular APK, and it can get access to the hardware capabilities that are distinct to a Snapdragon 8-tier premium smartphone. So that could include accelerants for neural processing or AI capabilities and other things. So if there's natural language processing or other things that you could use a DSP or you could use other parts of the hardware to accelerate, the developer has access to that. It's typically not done through spaces per se, although we are looking at where could we extend spaces to be able to provide more value to developers beyond, let's say, just the core perception technologies. But today it's really around perception. That said, we are really interested in how to help developers be able to explore the frontier of the latest in AR technology. And so if you have applications like Trip, we worked with them around things like eye tracking and how to use eye tracking in an experience to better drive a user experience or get feedback as to is the application working well for them. So those kinds of things are things we would look to do through Spaces as well.
[00:17:06.331] Kent Bye: Yeah, and the keynote talk that Hugo Swart gave yesterday on the main stage at AWE, he had it split into two major sections where he was talking about the hardware ecosystem, but also the software ecosystem. And the software ecosystem is where a lot of the innovations and partners that were shown a sizzle reel with lots of different companies using a variety of different use cases for some of these different AR applications, mostly, and then some XR stuff and VR applications. But yeah, I'd love to maybe first start with some of the different software applications for what are some of the exciting use cases and what's happening in the ecosystem that you see how people are using the Snapdragon Spaces.
[00:17:42.898] Brian Vogelsang: Yeah, yeah. So there's a whole bunch, and maybe a few. One is a company called X-Ray Glass. I'm not sure if your audience has heard of them, but they do live subtitling with the glasses, and they have an application that runs on the A3 on Spaces. It's actually one of the first dual-render fusion-enabled applications as well. So it's doing natural language processing, and as someone is speaking... they can subtitle what they're saying. So I can see the subtitles kind of floating in front of them. And the really amazing thing is they can translate different languages. So I could have someone speaking in Japanese and I'm seeing English subtitles. And so I think this is a really fascinating use case around how AR technology could be used for accessibility, how it could be used for communities that could take advantage of technology in new ways to help them. Or just basic, you know, I need to be able to speak with someone in a different language and use the glasses to kind of subtitle my real life. And so I think that's a really, really interesting one. We're working with a variety of companies in a bunch of different places, enterprise, consumer, VR and AR. On the AR side, if we kind of stick to that area, gaming is another really interesting one. There's one that we're showing a demo here of from Mark's Magnificent Puzzle Maze. I don't know if you saw the demonstration, but it's one of these puzzle games where you've got a marble run, and you actually have to move the marble through the maze and watch out for areas where you could potentially drop through the bottom of the maze. So it's a game of skill, and he's enabled this in AR as a Fusion application, so it was previously a mobile app. and it's got like 200 levels in it, it's like a fully realized game, but you use the accelerometer and the IMUs in the phone as the control, so you don't really even have to look down at the phone, you can simply walk around and look at this marble run and pan and tilt it to play it. So it was like a really interesting use of both the capabilities of the phone as a accelerometer and an IMU, and then the mechanic worked, applied really well to augmented reality. So that's another really interesting one.
[00:20:04.417] Kent Bye: Yeah, I did have a chance to play through that, and it is quite compelling to see the visual feedback of the tilt of this piece of wood where you have a maze that you're going through and use the 3DOF tilting of the phone to be able to actually navigate it. So it had a nice feel to it.
[00:20:20.768] Brian Vogelsang: And if we look at enterprise, I think we had about 24 different partners who have booths at AWE that are showing off different spaces, experiences. We have another 15 or so partners in the booth, the Snapdragon booth that are showing off things. And it's a whole spectrum enterprise applications. So people using it for like work instruction, guided work instruction, or 3D visualization, or collaboration in AR. We have demonstrations in mixed reality and virtual reality. We haven't talked as much about the VR and MR side, but there's a number of applications that are training and collaboration, you know, built around the Lenovo VRX. Lenovo are here showing off that headset, which is the first space is enabled. VR headset that's going to hit the market. I think they just announced this week that it's now shipping and available. So that provides both mixed and virtual reality. One of the things about Spaces is we can take applications that are built for optical see-through based glasses like the A3 and we can run those on a mixed reality headset. and vice versa. So you could build something for MR and now see it on an AR glass. And so this portability and ability to move between the realities is really interesting. I think as more and more mixed reality headsets hit the market, more developers are building for MR. Now they can take those same experiences and bring them to optical see-through glasses like the A3.
[00:21:42.284] Kent Bye: Yeah, I had a chance to see Chris Madsen from Engage XR, and David Whelan said that Engage XR is actually one of the first companies to integrate with the Snapdragon Spaces with more of a VR use case. So what does the Snapdragon Spaces give to VR in an application like Engage in a new Lenovo headset?
[00:21:59.338] Brian Vogelsang: Yeah, so if we look at Snapdragon Spaces, the core platform is all enabled through OpenXR, so all the perception technology we expose is done through industry standards. So Kronos, OpenXR, we extend it where necessary, so where the spec and standard is not, it doesn't cover, let's say, a new feature, we expand that capability, but we do it in an open standards way. So if you look at Chronos and OpenXR, there's been a lot of work happening with the VR community there. So everything we expose up through Unity or through Unreal Engine is through OpenXR. So developers can build applications and experiences that they've created for other virtual reality platforms and bring them over to Snapdragon Spaces. And then Spaces allows them to take advantage of newer capabilities that are starting to appear in these VR headsets around mixed reality. Let's say we want to turn the pass-through cameras on and we now want to start to have a spatial understanding of the environment, like detect the planes, get a depth map, be able to 3D reconstruct that environment and now introduce AR content into the real world, maybe have it interact with the real world using proper laws of physics and that sort of thing, those can all be done through an MR experience enabled in Spaces, very similar to what's being done in AR today. So I think that's one of the capabilities that Spaces affords is the ability to do things in VR and using OpenXR, bring VR applications over to a Spaces-based mixed reality device and then turn on those augmented reality, mixed reality features and leverage those as well.
[00:23:31.713] Kent Bye: So during Hugo's talk, he was saying that there was like 65 different XR devices that are out there that are using the latest XR2 chips. And there's also GoreTech and as well as Niantic who had some reference designs for some of the other hardware stuff. So I'd love to hear any reflections of what's happening in the hardware space because obviously there's a huge ecosystem of different XR devices and we'll have... some potentially new competitors with Apple and whatever they announce next week with XR. But Qualcomm and all these devices are working together in a sense of creating a whole ecosystem of all these different devices that are at the core, have a similar chipset that is in some ways sharing. There's a phrase that was at the beginning, which was that there's An XR ecosystem where the future is open. So I feel like there's a theme there where you have all these different hardware devices. So I'd love to hear some of those reflections of what's happening on the hardware side, as well as this openness on both the hardware and software.
[00:24:27.267] Brian Vogelsang: Yeah, so we're just really excited about the momentum we're seeing on the hardware side. So at the show here, we, of course, announced the TCL, the Rayneo AR glasses. DigiLens announced their standalone glasses support for Spaces. And then Oppo had announced a new VRMR headset that will be a dev kit for Spaces platform. And so these are our new headsets in addition to the VRX from Lenovo now being commercially shipping and the A3 that we've had in the market over the last year. We just see a much wider range of options now for developers to be able to build against and start to get more momentum and scale for the developer community, get more of these devices into the hands of enterprise users, consumer users, and start to continue to drive adoption. Really, I think at Qualcomm we want to see the VR market thrive, AR market thrive. We want to see end users getting benefit from this technology and see that expand, but it really starts with applications and experiences. And so it's really important, we believe, to be working with the developer community, get feedback from them, help create the right technology that can help the developers be successful in changing people's lives through these mediums. And then those developers need the OEMs to be able to get to market. They need the platforms that consumers can buy or enterprises can buy to realize their solutions. I'm just really excited about the diversity of hardware that's out there and the new entrants that we're seeing from folks like Oppo who are coming into the market. And we're just really excited to be working with them, powering them on their current generation of devices and looking towards the future.
[00:26:16.149] Kent Bye: Awesome. And finally, what do you think the ultimate potential of XR might be and what it might be able to enable?
[00:26:24.292] Brian Vogelsang: Wow, that's a big, that's a broad one. I think that ultimately, you know, the destination here, Steve Lucas on my team talked a little bit about this in his presentation yesterday. is standalone AR glasses. I mean, that's ultimately, I think, where we're going to see this technology reach real mass market scale and where we think the smartphone is going to be an important part of that with the glasses as companion. But one day we're going to be in the standalone AR glasses that we compute in every day. And maybe over time, Over the arc of the next five to ten years, we're going to use our phones less and less. We'll be using these glasses. So I think it excites me that we're moving to a new computing platform, that that computing platform is going to be spatial, immersive. It's going to be woven into our everyday lives. It's going to be, you know, hopefully much more intuitive and ambiently a part of our daily lives than our smartphone are and really get us out of looking at these screens and up and interacting with the world. So I'm just excited about spatial computing and the trajectory we're on. It's an exciting time to be working in this area.
[00:27:36.076] Kent Bye: Awesome. Well, thank you so much for joining me today to help unpack all the latest news from Qualcomm and excited to see where this all goes in the future. So thank you. Thanks, Ken. Thanks again for listening to this episode of the voices of your podcast. And if you enjoy the podcast and please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a, this is part of podcast. And so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voices of VR. Thanks for listening.
