#638: Updates on the Decentralized Metaverse: WebVR, A-Frame, & Supermedium VR

dmarcosThe open metaverse is going to be built on web standards like the WebXR Device API (formly WebVR), but the larger community of web developers has been waiting to fully commit to building immersive WebVR experiences until there’s universal support in all web browsers. The browsers that have implemented the WebVR 1.1 spec include Firefox Release 55, Oculus Browser, Samsung Internet, & Microsoft Edge. But Google Chrome and the WebVR developer community has been waiting for the official launch of what was being referred to as the WebVR 2.0 spec, but was recently renamed to WebXR Device API in December 2017, which is explained in more detail here.

I visited Mozilla’s offices last October to chat with A-Frame co-creator & co-maintainer Diego Marcos about the current state of WebVR. Marcos has since left Mozilla in order to work on the Supermedium WebVR browser, which creates a desktop VR browser designed for the Vive or Oculus to easily go in and out of viewing WebVR content as a seamless VR experience. Supermedium VR is a breath of fresh air to be able to seamlessly traverse amongst a curated set of WebVR proof of concepts, and the link traversal and security paradigms of WebXR are still open questions.


Mozilla also announced their Firefox Reality mixed reality browser this week, which is targeting the standalone VR headsets, primarily the HTC Vive Focus and the Oculus Go. It’ll also work for Daydream VR as well as Samsung Gear VR, but it’s going to be designing the immersive web browsing experience where there isn’t a context switch between the 2D screen and context switching into a VR HMD. Firefox Reality hasn’t implemented any WebVR features yet, and it’s currently a proof of concept for showing what browsing 2D web content in VR will look like. The increased resolution of these latest generation mobile VR headsets and upcoming standalone headsets makes reading text a lot easier than it was in previous iterations.

I’ve talked about Firefox Reality in the previous episodes of #350, #471, & #538 when it was still being referred to as the Servo experimental web browser built using the Rust programming language. Firefox Reality is currently the only open source, cross-platform mixed reality browser, and I’m curious to track the development more once they get more of the WebXR features implemented.

In my conversation with Marcos, I’m struck by how many open and unresolved issues still have to be resolved including link traversal, a security model that prevents spoofing sites, the portal mechanics of traversing multiple sites, and the potential of moving beyond a black box WebGL into what would be more like a 3D DOM elements but that has to deal with the additional privacy aspects of gaze and physical movement biometric data that having a 3D DOM would introduce.

It’s been a long journey to the official launch of WebVR, and here’s some of the previous conversations about WebVR since the beginning of the podcast in May 2014.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Support Voices of VR

Music: Fatality & Summer Trip

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So in the movie Ready Player One, the conceit is that there is this trillionaire who has designed this metaverse world that has all these puzzles that are embedded within it. And that if you solve all the puzzles, then you will then gain control and access over this closed walled garden version of the metaverse, as well as all the assets and money that he had accrued. So in actuality, I think that there's going to always be some combination of these closed world gardens that are able to provide a very high level of user experience in the open decentralized metaverse, what is essentially equivalent to the open web or the open internet right now. So when it comes to virtual reality, over the last four or five years of the resurgence of virtual reality, the open web has been lagging behind of what is possible in terms of a experience on a centralized system like either using Unity or Unreal Engine that has closer access to the dedicated hardware. Now actually over the last four years I've been covering WebVR and the rise of these decentralized systems and each year they get better and better and soon they're going to be on parity with having an experience that's run on native code. I think just the advances in WebGL as well as the Vulkan standard and just these different trends of Asm.js and the compiling down JavaScript, things are getting faster and faster. And Mozilla has been at the forefront of really pushing that edge. In fact, Vlad Vesovich was one of the original creators of WebGL and also started to really do a lot of the early work of WebVR before it was even named WebVR. And I had an interview with him and Tony Parisi back when I very first started this podcast, back in Silicon Valley Virtual Reality Conference in 2014. So I've been continuing to cover this story of the open web and its entrance into virtual reality. And if you look at the history of the web, there was a time when everything was a closed walled garden where you had like AOL, CompuServe, and CD-ROMs. Those were the essence of curated experiences. But at a certain point, it switched from people no longer seeing those networks were as valuable as the open web. And why is that? And what Philip Rosedale says is that you have to look at Metcalfe's Law, which is the value of a network is equal to the square of the total of the number of the nodes that are included in that network. So the more nodal connections and the more people that are involved in that, then the more valuable it becomes. And at the heart of it, what that means is that the open web is so valuable because you can interlink and connect between all these other different sites that are out there. So theoretically that would say that the open metaverse is going to have an exponential increase of value as more and more people create experiences for the open web but also link to each other. So just yesterday Mozilla announced Firefox Reality which is a open source web browser that's going to be on mostly the standalone headsets so specifically like Oculus Go and Vive Focus. But it's also going to be available in Daydream as well as on the Gear VR. But primarily it's going to be for the standalone headsets. So I actually had a chance to try out this browser and it's still very early. They don't even have the web VR that's hooked up yet. It's basically you can go in and download an APK. You can open up Daydream. You can go in and look at a browser and you can basically scroll through different web pages. So to start, it's just an ability to be able to browse on 2D web pages and that As time goes on, though, they're going to be integrating more and more of these WebVR capabilities. Right now in the WebVR community, there's this split between some of the people who believe that you should just go ahead and push out the 1.1 version of WebVR. And then there's other people who are waiting for what used to be called the 2.0 spec, but has now been rebranded as the WebXR device API. So it's basically like the WebXR 1.0. So the browsers that actually have the WebVR 1.1 implemented are the Firefox 55, the Oculus Browser, the Samsung Internet, as well as the Microsoft Edge. But Chrome is waiting for the initial kind of WebXR official release. And then at that point, we're going to start to have these major web browsers that have access to a lot of this WebVR content. So I actually visited Mozilla's offices back in October and talked to Diego Marcos about some of their latest efforts in WebVR as well as he's specifically the maintainer of A-Frame. He has since left Mozilla and now he's working on creating what is called the Super Medium VR browser. It's an application on the Vive where you can go and get a whole curated experience of all sorts of WebVR experiences that are out there. It really shows the best of in terms of like art and music and data visualization, architecture visualization, real estate, some games, but gaming isn't really the unique affordance of WebVR, I don't think. I mean, there will be games, but I think the advantage is that they're going to have all sorts of different experiments that are going to be able to be put out there on the web. And I think it's actually going to be designing for some of the unique affordances of VR, which is embodiment, as well as music and emotions and stories. So I think you're just going to see a lot more independent experimentation with the WebVR. And I'm really excited for that. I think if you actually go and check out a lot of the experiences on the Super Medium VR browser, you'll see a wide range of different experimentations of People just saying, hey, I'm going to experiment this one thing. I'm going to have all these different sounds drop around me. I'm going to play with specialized sound and see how that impacts your sense of embodiment and presence. And it'll be a short, quick experience that, you know, is just a few minutes. But I spent a good two and a half hours going through a lot of the experiences just to get a taste of them. And you can spend a lot longer really diving into each and every one of them. But right now, this is probably the last time that you'll ever have to be able to go and basically surf the entire web of WebVR. As time goes on, WebVR is going to get more and more popular. And I think at some point in 2018, it's just going to cross the chasm and it's just going to really explode once all of these browsers have access to be able to actually render out the WebVR. And it's an easy, seamless experience. So I'm going to dive into this interview with Diego Marcos, who's both a creator and maintainer of A-Frame, but now he's working on the Super Medium browser. And at the time of this interview, he was working at Mozilla, working on all their various virtual and augmented reality initiatives, and specifically working on A-Frame there under the auspices of Mozilla. So we'll be covering all that and more on today's episode of the Voices of VR podcast. So this interview with Diego happened on Thursday, October 19th, 2017 in San Francisco, California. So with that, let's go ahead and dive right in.

[00:07:08.422] Diego Marcos: So I'm Diego Marcos, and I'm one of the creators and maintainers of A-Frame. It's a framework for web VR content creation to onboard all web developers into the new medium.

[00:07:20.037] Kent Bye: Great, so I think the last time we talked was back in March or so, and since that time we've had the first version of Firefox that's launched, version 55, that has WebVR 1.1 built in. So now that we have the Firefox, there's the Samsung Internet, there's the Oculus browser that has some support for WebVR, and A-Frame seems to be one of the leading frameworks to be able to create WebVR experiences. So maybe you could tell me a bit about, like, what have you been seeing so far in terms of the types of sites and engagement from the community?

[00:07:54.412] Diego Marcos: So we've seen, like, all since before 2017, like, WebVR has been experimental. So what we've seen so far is a lot of experiments, like people kicking the tires and exploring the new medium. And we've seen, like, all sorts of experiences since, like, real estate, like 360 videos, games, escape the room experiences, like content creation tools to create content in VR. We've seen education applications. We've seen tourism applications to be able to go to places without moving from where you are. We've seen all sorts of genres so far, but mostly experimentation, and with the release of WebVR this year on Firefox 55, Oculus Browser, and Samsung Internet, we are expecting to see much more professional content, like production quality content out there. because right now we close the gap. So anyone with a Firefox release, standard installation, or the Oculus browser, they can just, with a single click of a button, they can consume content. And we expect this to motivate content creators to release more polished content because they can tap into an audience with almost zero friction. You just plug your headset, click a button, and you are in.

[00:09:16.319] Kent Bye: Yeah, I think, like, right now we're in this sort of transitory stage where, like, not every single browser is supporting WebVR. And so, like, I know Chrome is kind of waiting for the 2.0 version. We have, like, Safari that, up to this point, hasn't had any sort of implementation of any of the WebVR specs. Microsoft Edge also doesn't seem like they've deployed any of the WebVR. So, maybe you could tell me what that means when you have a site like Matterport or Sketchfab when they're trying to build in specific WebVR features. Is it a matter that if you happen to be using a browser that supports WebVR that you'd be able to kind of drop it into like a Cardboard or a Daydream or a Gear VR? and start to have that immersive experience. Or the thing that I've noticed is that usually when you start to drop these phones into these specific headsets, then they kind of go into the either default gear VR or daydream modes. And so it's like difficult sometimes to kind of jump into a web VR experience. And so you almost have to like have a headset at your computer, which the friction is super high. So maybe you could walk through that workflow for what people see right now on a site like Sketchfab or for Matterport that may have some features of web VR embedded with their website and what the user experience looks like right now.

[00:10:38.642] Diego Marcos: Yeah, so this is a big question. So the ecosystem right now of web VR and VR in general is kind of split into mobile versus desktop. And in mobile, you just have a single screen. And usually, you put the headset on in your holder, like your Gear VR or Daydream holder. And then you consume content in VR. In desktop, it's a bit different because you start normally browsing the web. in 2D mode and at some point you find some VR content and then you just click the enter VR button, you consume the content, you can take your headset off and you keep browsing in 2D mode. In mobile it's a bit different. Once you have engaged with VR, your headset is in the holder, then going back to 2D mode has more higher friction. So you would like to keep browsing in VR even if you go from a VR side to a 2D side you would still like keep browsing without taking the headset off and taking the phone off the holder and it's a bit more higher friction to transition from 2D to VR mode in the mobile space. So what people are doing, for example, Oculus is doing a really great job to integrate their browser with their home experience. So you can be consuming native content and quickly like jump into the home and start up a browser and consume 2D and also immersive content. Everything within VR, like the transition is more seamless. So we are seeing many of the platforms to integrate browsers as first class citizens in their home spaces. So this friction is reduced and you can just, even within a native app, you can just spin a browser and consume some web VR content within a native VR application and then go back to what you were doing before. So we are going to see like deeper and deeper integrations of these browsers with the respective platforms to reduce the friction to consume web VR content, which I think is very important. And it's going to be also important for like the main, so you mentioned like a Sketchfab, which they are doing like a great job like to So they adapt their experience to whatever the capabilities of the environment. So they have a great experience in 2D. They have a great experience in VR. But yeah, the browser has to be accessible and has to be easy to use once you are engaging with each of the platforms. So yeah, the applications, they have to account to adapt the software to the capabilities of the environments. But browsers need to help as well to reduce the friction.

[00:13:15.315] Kent Bye: Yeah, and one of the things that I've noticed where some of this immersive technology is making the inroads into a lot of consumer-facing is within the context of the real estate market. I know my parents had put up a house for sale, and they had somebody come in and do a Matterport scan of that. It took about four hours to scan everything. They had created an immersive 3D model of the house that they were selling, which meant that it was easier to kind of get a better sense of the space by navigating it, just even using the browser. And so you're able to kind of teleport around and to get a better spatial sense of the area. You have a better sense of the scale and where things are relative to each other. but yet this is something that I think is a use case where it degrades down to a 2D experience where it's just as good. And if you have a headset, then you get even more immersed and it has that little icon there. And so is that sort of the something that you see a trend in terms of like somebody may be implementing some features that has immersive content and if the browser enables it, then it maybe will show up that little icon in the lower right hand corner And if it doesn't, then it'll kind of degrade to the point where it'll do everything that that web browser can do in order to give a good 2D version of that. But if you have the capability to pop into a VR experience, then it'll show you that. And then if you have that gear, then you can kind of jump in and get the immersive version of it.

[00:14:45.992] Diego Marcos: Yeah, that's my personal opinion. It's not, it doesn't reflect any roadmaps or anything, but since the beginning, so first of all, to create good VR content, you have to acknowledge the constraints of the technology today to provide the best experience you can. Like we know, like the resolution of this place is not good enough to read text and you can, but it's not the best of experiences. And I'm always in favor, like to embrace VR when it makes sense and when it provides a better experience than alternatives. And the example I've been trying like to, we have like a couple of demos using A-Frame in the context of Wikipedia. Wikipedia for me is the perfect example. It represents like the perfect application for the web. And I talked to several people at Wikipedia trying to convince them that it's a good idea. The flow would go this way. You consume text content in the device that provides the best experience. It's going to be your 2D monitor, your 27 inches display on your desk. Or it's going to be your super retina high resolution display on your phone. But then, let's say that you go to the Tag Mahal Wikipedia entry. You read the text, you read the whole history about the Tag Mahal and then you want to be able to experience the Tag Mahal in person. And then this is a perfect opportunity for you if you have a headset, you just connect the headset and you have like a small button in a corner and then you click that button and then you are there standing in front of the Tag Mahal and you get like a different perspective of this piece of content in a much more powerful way than a 2D display can offer. So for me, an excellent use case for VR and WebVR in particular is this complementary use case. You still have your traditional content consumed in the traditional way, but if you have a headset, this website is going to give you something extra, some additional value that traditional input and output methods cannot provide. So if there's someone from Wikipedia listening there, just talk to us because I will be happy to do something with them.

[00:16:54.593] Kent Bye: Yeah, so in other words you keep reading the article within your browser because it's a better experience But yet to actually feel like you're going to these places to kind of hop into VR and get that spatial experience Of that sounds like what you're saying.

[00:17:08.306] Diego Marcos: Yeah. Yeah, exactly It's like playing with the constraints. Eventually, we'll want to consume all the digital content within a headset. When we have like sunglasses, like form factor, and you can wear them like 24 hours, and you will consume everything in the headset. But today, there's some constraint you have to acknowledge to keep the content honest. And you just use VR when it makes sense, and use other devices when it makes sense. And I think you can provide the best of the experience depending on the use case.

[00:17:41.340] Kent Bye: Yeah, I just had a chance to talk to Clay Bavor of Google and Google has come out with ARCore and this past summer also had an announcement from Apple that they released their ARKit and Google actually released a couple of experimental web browsers to be able to implement both ARCore and AR kit into these browsers so that you could start to have a immersive augmented reality experience using all of the features of these new SDK frameworks for augmented reality, but being able to use JavaScript and HTML to be able to actually generate these immersive experiences. And I think a number of people even can use A-Frame. So now have the possibility to, if you're a web developer, create an immersive experience and then bundle it within the context of these apps and then, you know, deliver it and give people access to that. So maybe from A-Frame's perspective, just curious to hear your comments on that in terms of the moving into starting to use the web development into these new augmented reality applications.

[00:18:47.271] Diego Marcos: Yeah. So WebAR is still very early days for WebAR. So to give a bit more context, we have two versions of the specs of the WebAR spec right now. The 1.1 that has shipped on Firefox 55, has shipped on Samsung Internet, has shipped on Oculus Browser, and has shipped on Microsoft Edge. So 2017 is going to be, remember, the year that WebVR shipped. And right now, we are working on the 2.0 version of the spec that is still mostly focused on VR use cases. And we expect to finalize the spec soon. And early next year, we will see the first implementations on different browsers. And for the AR context, both Google and Mozilla, they are exploring how we could extend those WebVR APIs to actually account for AR use cases. And right now, each of us, we are experimenting in different ways to have an idea of what we exactly want to do. And then we are starting to talk to consolidate those experiments, the things we learn from them, into an extension of the WebVR API to account for AR use cases. So right now, today, you can already play a little bit with what WebVR could look like. As you mentioned, Google released two applications, which is WebAR on ARCore, which is an Android app that you can install in your ARCore-enabled devices. It's like a proto-browser with WebVR 1.1 with AR extensions. So you can develop WebAR applications against these experimental API. And we're going to have something similar. So we're going to have our own applications, both for ARCore and ARKit, that implements our vision for AR APIs. So today, we announced, not released, but it gives a sneak peek of an A-Painter port for AR environments. So it lets you paint in the space. You have to leave your drawing on the table and move around, and using those kind of proto-browsers that have these AR extensions on iOS and Android. So it's still early days, but we are trying our best to let people play with it as soon as we can.

[00:21:10.422] Kent Bye: Yeah, and just talking to Brandon Jones from Google, you know, they're a lot more hesitant to launch into supporting WebVR. And I think they're waiting for the finalization of the 2.0 spec. And my sense is that they don't want to have to deal with the deprecated code and to not have to support something indefinitely and forever. So I think we're in this kind of nimblest time where it's kind of uncertain for how things will be archived for the long term. And it sounds like what you're saying with this web AR extensions that are going to be made available with some of the experimental web browser applications that you may be releasing here soon as well, is that this is kind of like still the spec for AR is kind of not really solidified yet. And so anything that is implemented is kind of subject to change. And so for people who are experimenting with it, it sounds like it's until we have a solid 2.0 spec, is that kind of the timeframe that people can kind of expect things to solidify a little bit more?

[00:22:11.078] Diego Marcos: Yeah, I think there's... So we want to ship WebVR as soon as we can and as fast as we can. And this is depending on finalizing 2.0. And yeah, like including right now AR into the conversation could delay the spec. So on the Firefox side of things, we release WebVR 1.1. So there's less pressure for us to finalize 2.0 because we have shipped WebVR. But for other browser vendors that they don't want to maintain two APIs, which is completely understandable, they are relying on finalizing 2.0 in order to release WebVR. So there's this tension. It's like, we would like to incorporate AR capabilities into WebVR 2.0, but this could delay the finalizing the spec. And right now it's like some people just want to focus like WebVR 2.0 on VR use cases so we can ship quick. And then once this ships, we can focus on, oh, how can we now improve WebVR 2.0 to account for AR scenarios in a way that it will be extending the existing APIs, but not breaking what is already there. It wouldn't be like WebVR 3.0, more like a WebVR 2.1. 2.0 with more stuff for AR use cases. So that's a little bit like, there's a lot of conversations. It's like, it's not like final decisions, but right now we're leaning like to, okay, let's nail down 2.0 for VR and then talk about AR afterwards. This is why within Google and within Mozilla, like the people interested in AR, they just don't want to be just waiting for 2.0 and then start talking about AR. They just want to play with it right now. And this is where we are releasing those proto-browsers with those experimental APIs to get a sense of what WebAR API would look like, get feedback from developers. And then once we are ready to sit down and spec out the API, we have a real sense of the needs for AR.

[00:24:12.566] Kent Bye: Yeah, one of the experiences that I've had going in and playing with WebVR was that it did feel like I would have to jump into an experience in order to go into another experience, kind of get out of VR and then go click on the browser and jump into another experience and then kind of go back into a VR experience. So I was still dreaming of the days where the link traversal is completely a solved problem so that I can just hop into WebVR and start kind of cruising around and discovering different sites, and then having these portal sites that help guide me into these different types of experiences. And so just this morning, you gave me a chance to check out some of the early demos of that work. And you said that you actually have to do some work on the browser side so that this is not just a framework consideration, but there's other issues that you have to kind of work out at the software level for the browser to be able to actually even implement this, I presume due to security reasons and whatnot. But just to kind of describe what I saw, it was almost like, you know, I'm in this world where there's like these little circles that are like, you know, if I'm in a ship, I'm almost like looking out these windows into these other worlds. And I can kind of peek my head into them, or I can push a button that would kind of teleport me into the link as a preview. So before I load down all the assets, and then I could click the button to be able to actually traverse into these different worlds. And so maybe you could kind of just give me a state of the union of link traversal, where it's at, if this is sort of some of the open questions in terms of the 2.0 spec for how this is going to work out.

[00:25:45.940] Diego Marcos: Yeah. So WebVR 1.0 already considers VR navigation. So the way the API works is this way. So in order for you to engage VR mode in a browser, you first need to account for a user gesture. A user gesture is like a signal from the user that we can trust that the user actually acknowledging that they are entering VR mode. This is important so not any random website takes over your headset and they can fool you and show you a website that is different than the one you really are. So it's a way for the user to acknowledge, yes, I'm on this website and I just want to engage VR mode now. But also in the web, navigation is one of the essential ingredients of the web, being able to navigate and browse interconnected pieces of content in a very low friction way. So we want to have that experience in VR as well. But that comes not without challenges. So right now, WebVR 1.1 allows for VR navigation, and both Oculus Browser and Firefox, they ship that feature, so you can actually traverse between websites, but it's far from perfect. For example, if you go from the website A to the website B, and the website B has an error, there's no way for you to know within the headset right now that that website has error. You're going to have a loading spinner, And you don't know, you can be waiting forever, right? That's one example. Another example is like, okay, I'm going to the website A to the website B, but can I trust that I'm going to the website B? It's like, once I navigate from one place to the other, the other place has complete control of the headset and they can fool me. It's like, I think it's the responsibility of the browser to give me a mechanism that I can trust so I can know where I'm at at any given time. So this part is not part of the spec, it's up to each browser to implement those mechanisms so that the user can trust. And this is what we are working on at the moment. We are thinking of ways of how to make in-VR navigation more secure. And so one of the things is like, there's certain situations like errors or navigation where the browser has to take over the headset and inform the user about what is happening. And those are the things that we are working today. And also it has to be what we call, I call the escape hatch. like a way for the user to always invoke the safe space or safe screen to be able to get out from content that you don't like to see anymore or to get out to get information about the place you are. Like the basic essential is to be able to see the URL since you don't have like the URL bar anymore present on the screen all the time. This have to be a way for the user to invoke that information at any given time. And it has to be a way that you know that a browser cannot fool. Like, there's several ways we are thinking we could implement that. It's like from reserving a specific button on the controller. So that button cannot be used by any content. So you know that when you are pressing this button, what you're going to see in the display is going to be this trusted space. But then there's another implications for that, because already today, the different VR platforms, they already have the concept of home space, and they already have one reserve button to go to those home spaces. If your WebVR browser also have another button that is reserved to go to this home, it could be confusing for the user. Oh, I'm in the browser home space? I'm in the SteamVR home space? And then it's one less button that applications can use. And like, yeah, we have to think if really reserving a button is the good way to do these kinds of things. Maybe it could be like a voice trigger. Like maybe with voice, you can trigger this home space. But if pages can also get like voice commands, how you make this command like trusted. I don't know, and even like visually, right? Like any website could fool the home space and make you believe that you're in the home space. How can you provide like a visual hint for the user to determine that this home space is the legitimate home space and not fooled by another website? We're thinking about different things. There's some personal information from the user that always is rendered on this home space that pages cannot know about. And then this personal information, you see that they're there. You know the website, they cannot access to that information. So it's a sign that this home space is the one that you want to see, the trusted one. There's a lot of implications. It's not like only ergonomy and usability, but it's also like privacy and security. And right now, the mechanism for link traversal ships on WebVR 1.1, but it's far from being the thing that you want to rely on for secure and private navigation.

[00:31:04.762] Kent Bye: Yeah, I imagine that each browser vendor has different security standards because, you know, whenever I open up a YouTube and I push like a full screen or something like that, it's like inevitably have this, like, are you sure you want to give this full screen? And it's like almost like, yes, I've done this a million times, just take over the screen already. But I think at the same time, there's certain security protocols to prevent some sort of hijacking of the screen than to be suddenly taken over by someone else without your consent. And so there's similar issues, it seems like, when you're going to VR, making sure that there's a full consent from the user to make sure that there's not some sort of hijacking of the browser that's taking over. And, um, you know, I guess that that's what I'm, what I hear you talking about all this, there's all these sort of implications for each browser has to navigate that. It sounds like.

[00:31:55.212] Diego Marcos: Yeah, for sure. Privacy is one of the main value propositions. For many people, it's what the security model would actually make the web succeed. So you can download any software on demand, it runs on your machine immediately, and it's secure. So we want to maintain those properties in VR as well. It's essential for the success of web VR. Yeah, but the challenges here is like in a 2D browser, there's always regions of the display that a website cannot control. You have a URL bar, you have like the browser UI, but all those things in VR disappear. So you need alternative methods to build trust that cannot rely on having like a reserved area of the screen all the time. So, yeah.

[00:32:49.286] Kent Bye: Yeah, and just we were also chatting talking about like canvas elements and how right now it seems like the way that web VR is being generated by creating this spatial immersion is through a library like 3JS, which is kind of creating this pixels, painting pixels into this 3D space and compiling it down to WebGL, which then at that point, you're losing a lot of the 3D DOM elements and so at this point there's JavaScript and HTML but there's no sort of equivalent CSS and cascading style sheets to be able to address each of those DOM elements in a 3D spatial space because at this point the DOM is kind of goes away and once you sort of compile it down to that WebGL and so what's happening with like Reimagining of the DOM to be able to go into a 3d space. Do you foresee that in the short term? we're going to continue to go down this path of using something like 3js and compiling down to WebGL or do you see a need that eventually at some point you're gonna have to like reimagine what the 2D model of the web was and then how that needs to evolve into a 3D DOM?

[00:34:02.062] Diego Marcos: Yeah, for sure. So the short answer, we want to use the DOM in VR. So we have all those libraries to do UIs, all this knowledge, all these technologies that people already know that we want to apply in the VR context to do 2D UIs. So this is going to happen eventually. And technically, it's not the hardest of the problems. But again, it's like security and privacy are higher in the priorities. And once you let applications to extract the pixels of a DOM element, you can infer a lot of user information from that. So there's concerns about how can we provide those DOM elements to be reused in a WebGL context without breaking the security and privacy contract that the browsers provide. So it's mostly figuring out how to make it in a secure manner than the technical issue. But yeah, for sure, we want to be able to reuse all those knowledge and all those technologies in a VR content. There's another school of thought that say, okay, at the end of the day, the DOM and CSS are technologies that were designed to make static documents, not to do UIs. And maybe this VR new medium is an opportunity to start with a blank page. If we have to imagine a technology to do UIs that are animated, that they are very responsive and very high-performance, what would be the technologies we would invent given those new constraints? And some people see it as an opportunity to leave the dumb behind and to provide maybe a better technology to do UIs. to the UIs, but there's a lot of knowledge, and there's a lot of people trained on those technologies, and being able to reuse that knowledge in VR will be very valuable as well. So, it will eventually happen. Actually, at SIGGRAPH, the Google folks at the last SIGGRAPH, they presented in the Chrono Sessions one prototype of how we could reuse, for example, iframes in VR, so you can actually display a YouTube video in a VR context. So there's people, very smart people, working on that issue as we speak. And it's going to happen, for sure. I don't know, eventually, which one will win, if we will be doing 2D UIs in VR using DOM and CSS, as we do today, or if there's going to be something else. I don't know. But I think we need both to compete with one another.

[00:36:30.737] Kent Bye: Yeah, I happened to pop into that Chrono session at SIGGRAPH where it was almost like this fabric floating around and then on top of that painting on a YouTube video, so able to kind of combine an element of a 2D video and just kind of start to paint all the pixels onto this fabric that's kind of floating in the wind, which I thought was a really great example of the merging of these two worlds. And you're also working on Servo here at Mozilla. There's a number of people who have taken this challenge of WebVR as kind of like a prototype to start to use some new computer programming languages to see if you could build a new browser from the ground up, it sounds like. So maybe you could tell me where Servo is at and what you're able to do with that that's different than some of your other projects of shipping a browser with Firefox.

[00:37:22.880] Diego Marcos: Yeah. So right now, we've been experimenting with Servo for a little bit over a year right now, and it is coming along pretty well. So right now we have support on Servo. If you download the source of Servo, compile for Android, you have like Gear VR and Daydream backends. So I would say that we can take the credit that it's the only open source browser engine today that offers both compatibility for Daydream and Gear VR. And as you said, we are building a browser engine from the ground up, and there's a lot of APIs. For example, we don't have WebRTC. The media stack to be able to play videos is still not complete. There's a lot of work to do there. So it's far from being a production-ready browser, but it's very promising what we are seeing so far. And the advantages of having like a browser built from the scratch is like you can rethink many parts of the architecture that are harder to change in traditional browsers, that they have a lot of users, they have a lot of people rely on them for so many things that if you want to change something, there's going to be a lot of pushback to make sure that you're not breaking anything that already works. So instead we have the luxury to be able to tweak and to optimize parts that traditional browsers would be much harder.

[00:38:45.840] Kent Bye: And so for you, what are some of the biggest problems that you're working on and trying to solve as we move forward and eventually reaching a 2.0 spec and what you're doing and working on to kind of help support the work and activity that people are able to do right now in the WebVR community?

[00:39:03.638] Diego Marcos: Yeah. So I spend 95% of my time on A-Frame. And A-Frame is like solving multiple problems at a time. So for example, it's like this kind of like transition from 1BR 1.0, 1.1 to 2.0, and maybe AR extensions. So what we want to say to people is use A-Frame, and we will take care of all that transition. So you just focus on the content and all these nasty details. It will just work for you. And on the other side of things, I just want to see higher and higher quality content with A-Frame. And to the point, like, I put someone in a headset, and they cannot tell if this is native or is the web. Like, we did, like, A-Painter or A-Blast. And for some people that are not VR experts, it's starting to get hard to say, oh, this is native or this is the web. But there's still a lot of places we have to improve. And that's my goal. There's a point like, OK, I put you in a headset, and this feels great. This is great VR. And then I tell you, this is a website. And yes, it blows your mind. Yeah.

[00:40:16.267] Kent Bye: So it sounds like that if people start to dive into WebVR and they use A-Frame, then A-Frame's kind of making the promise that just go make your content and then everything that we have implemented now will continue to work whenever we have this transition from 1.1 to 2.0. You don't have to worry about it. We'll take care of that. Is that true or are there things that are going to get deprecated in what is implemented now in A-Frame? And if not, then what are some other of the new features that are coming? So I guess the multi-part question in terms of like, if whatever is there in A-Frame is going to be there in that transition, and then what's going to be new or different when it actually 2.0 launches.

[00:40:55.384] Diego Marcos: Yeah, and when we are core, handling the transition between APIs should be seamless. You shouldn't be losing any features or anything, and your content should just work. And on A-Frame, we are also entering a stage of consolidation. With this A-Frame core, we are trying to make an API that is functional, but still is not too big, so we can optimize that API and make sure that we are going to keep it moving forward. and then let the ecosystem of components to fill out the gaps of the functionality that A-Frame Core doesn't provide. And then a lot of experimentation can happen. But we want to have like this kind of solid foundation that people can rely on to build their components. And yeah, the components, there's some components that will stop working if they rely on new features, but that's fine. So we want to enable people to experiment while keeping like a core that works, is performant, and you can rely on. So the answer is, we don't want to, you do notice if you're using WebVR 1.1, 2.0, or eventually AR, you just target A-Frame API and things just work. Yeah.

[00:42:09.387] Kent Bye: And for you, what do you want to experience in VR?

[00:42:13.370] Diego Marcos: What do I want to experience in VR? A lot of things. So what I want to experience in VR is both like a function of hardware and software. And we were like before this interview, we were like trying to play with so many demos and you realize there's so many frictions. Oh, like, SteamVR doesn't launch and I don't know why I have to restart the computer or now there's an update and things don't work. I just want to be able to experience VR and AR in a seamless way. So I put my headset in the morning It's on my head all day long, and I just don't think about it. It's like, oh, now it makes sense to use AR, or it makes sense to use VR. And I can just bring my email application, or right now I want to play a game, or right now I want to watch a movie. I just don't want to think about hardware and software anymore. It's just like VR and AR integrates in my life. I use it from the morning to the time I go to bed, and it just works. And there's so many. It's exciting, because I think I have worked for 10 years. There's so many friction points that we have to remove and improve that it says, I'm not going to be unemployed in a decade. So that's what I want to do. It's like, I want to play games. All those things that you can do today, but frictionless. I don't think about it. I just invoke those programs and capabilities with the gesture of my hand or with a voice command, and it just works.

[00:43:41.451] Kent Bye: Yeah, it's a good place to get to. Good goal. And finally, what do you think is kind of the ultimate potential of virtual and augmented reality and what it might be able to enable?

[00:43:54.006] Diego Marcos: Yeah. So for me, it's like when people ask, oh, why VR? Whatever. It's like, the main thing is like, I think we have to make 2D displays obsolete. We have to free digital content from those rectangles that we carry in our pockets or have on top of our desks. And yeah, your computing is everywhere. And it's the same, like reducing friction. I'm not using my phone anymore. I'm not using my desktop computer anymore. I'm just living. And I have this technology that assists me and helps me in different ways at different times. But I don't even think about it. So I just want this place, traditional rectangular this place to disappear from the world. And that's for me what AR and VR will enable.

[00:44:45.076] Kent Bye: Yeah. And we are also showing me, uh, Brett Victor has a lab here in Oakland, California, and that that is sort of a model for this immersive spatial computing. It's using a lot of projection mapped and, you know, kind of computer vision and augmented reality type of. experiences, but it's it sounds like it's much more immersive and being able to actually Grab pieces of paper and kind of interact with your computing environment, but you're kind of completely immersed within Turning your environment into a computer. It sounds like maybe you could just say a few words about that in terms of a vision of being able to collaborate with each other face-to-face but you know very embodied and visceral and being able to actually you know manipulate objects around but you know that experience of what it feels like to be completely immersed in your environment that turns into a computing environment.

[00:45:35.462] Diego Marcos: Yeah, so Dynamic Land is a Brecht Victor lab that he's running. To explain a little bit, people know about AR, right? Like projecting digital content in your physical space. You can project 3D models, have 2D UIs projected anywhere. So Dynamic Land, they do the same thing as AR can do, but in a different way. They use actual projectors. that project information in your table, on your walls. And the good thing is that it's like a shared environment. So each individual is not wearing their own headset. It's just like the information is projected in the space, and everybody has the same view of that information, which makes it very nice to collaborate. Once everybody has the same view, you can just play with that information around you in a collaborative way. And what they have is like, right now it's more like a programming environment. So those programs are represented by papers, and those programs are running in a computer somewhere. And those projectors, they have cameras that are able to see those papers. And when they see those papers, they run the program that is represented by that paper. And those programs can actually project things in the real world. And they have awareness of what other programs are nearby. One cool example is they have a book, a programming book, that you can turn the pages and you see different programs. And when the cameras and the projectors, they see that book, they project the result of those programs over the book. So you have a page teaching people about fractals, and then you see fractals actually being rendered on the book. So your book becomes magical. You have fractals coming out at the same time as you are learning about how they work. It's kind of magical. It's just like, yeah, it's like there's no screens. It's like the digital information integrates in your normal life and you have computers like helping you to understand better, to learn better, and yeah, and to collaborate better.

[00:47:38.843] Kent Bye: Yeah. Yeah, and to me that sounds like that's sort of the roadmap of where all of these immersive technologies are going, whether it's gonna be in an immersive environment in virtual reality, or phone-based AR has still that magical rectangle that you're looking for, but eventually as we get into the head-mounted displays, we'll be able to just have this kind of completely immersive environment. So instead of using something with projectors and projection mapping technology, then we'll be able to just have it all self-contained within the immersive computing platforms. So yeah, I look forward to actually having a direct experience of that. I've seen some demos of that, some AR sandbox that I know that Oliver Kralos has been working on for a number of years, some similar ideas to that. And also at the MIT, using Legos to do urban planning and being able to do visualizations and being able to measure the block sizes and then kind of project, okay, this is what happens with the kind of statistics of the demographics based upon, you know, the size of the buildings and this sort of urban planning setup that you're doing, you're actually able to visualize the impact of that over time. So yeah, it seems like it's on the same strand and all that. So yeah, is there anything else left unsaid that you'd like to say?

[00:48:51.502] Diego Marcos: No, I think we had a very nice conversation. Yeah, it was cool. It was fun. Awesome. Well, thank you so much. It was cool. Thank you. Thank you for having me.

[00:49:00.254] Kent Bye: So that was Diego Marcos. He's the co-creator and co-maintainer of A-Frame VR, as well as a co-founder of Super Medium VR. So I have a number of different takeaways about this interview is that first of all, after spending about two and a half hours in the super medium VR browser last night, exploring all the various experiments and web VR, it's really kind of the best of what's out there in web VR right now. And I highly recommend if you're curious about it to go just check it out and see what kind of experiments are out there. And the thing that's really inspiring is that you've got these applications out there that you can go and view source and start to copy and paste and start to remix. these different experiences and create your own experiences. That to me is the power and the beauty of the open web is that it just allows you to share your source and for you to be able to take some of these experiences and start to remix them in different ways. Now I think it's a little different within VR because anytime you look at code within a 2D web browser anything that's an image is copyrighted and Similarly, there's going to be 3d models by which that you know in order to get access to those, you know There are some copyright issues so you can't just completely lift some of these experiences But there's more and more things that are out there like Sketchfab has the download API now the poly has Creative Commons assets So be able to actually download some of these 3d assets to be able to use within your own experiences So more and more it's easier to be able to have a repository of these cultural artifacts that are licensed under Creative Commons You can start to use them So I expect to see a lot of experimentation, a lot of really creative experiments which are not commercial. They're just inherently artistic and weird and strange and small ephemeral experiences that you hear about and you have the experience over the course of a minute or three minutes and then you're done and you move on. But it's moving away from the app store model, which is that you have to hear about it and download it and pay for it. This is a whole other model, which is that you just experience it. You have the experience. And I think the funding mechanism to be able to support this type of artistic expression is the things that we kind of have to figure out. What are those mechanisms by which that people can actually sustain themselves by creating this open and free art, whether it's through cryptocurrency or microtransactions or donations or Patreon? or just other ways that you could either spread the word or just support these artists who are doing this. So the business model of all this has yet to be figured out, but I think the exciting thing is that if you watch a movie like Ready Player One, you see the downfalls of like an extreme centralization of power, control, and greed. And that the antidote to that is like the open standards and the open web and people figuring out ways to be able to create these decentralized systems that are able to not have these centralized choke points. It's not great that all of the options for you to be able to create an immersive experience and distribute it out to people means that you have to go through the terms of service and jumping through all the hoops that both Oculus, HTC Vive, Samsung, the Gear VR, as well as the Windows Mixed Reality, but mostly the SteamVR and Oculus are the two major outlets. And if you haven't been able to jump through all those hoops to be able to get your stuff out there, then, you know, it's just kind of limiting the amount of experimentation and types of experiences that people are even into, because those are very centered towards gaming. And I expect that there's going to be a lot of music experiences, a lot of embodiment, a lot of like just environmental exploration, a lot of exploration of mathematical structures, just a lot of weird stuff, but also like data visualization and look at space and architecture and geography and overlaying data on top of geography, geospatial visualization. Just there's so many things that I think the open web is going to be so much better for than we haven't seen a lot of those experiences yet. So a link traversal is a nightmare right now. Um, there's so many security issues. Also, you know, the fact that there's no 3d Dom yet means that everything's being compressed down into web GL. So they're painting pixels to basically create a 3d representation. It's not an actual spatial representation. If it was an actual spatial representation, you'd be able to target things with the Dom and be able to style things with something which would be kind of the equivalent of CSS, but would be perhaps shaders or other things that be able to. include different skins on top of these 3D objects. So right now that's all being compressed down into a 2D box of pixels, which basically means that it's a black box. You can't really address and be able to change things dynamically. there's lots of security implications for that because you know you may not want that to be there you want to have to actually really be sure what that content is but also once you're unveiled within vr and you go into another website it may be very easy to spoof that website so what are the mechanisms by which you you are for sure that you haven't somehow been hijacked and fooled into thinking that you're for example at your bank and you enter in your pin number when it's actually like a think bank that isn't actually your real bank and so Are there mechanisms of self-sovereign identity that you have as a private key that only you can be revealed and then maybe there's a public key dimension there that you are know for sure? All of this stuff has to be worked out for what exactly that looks like to be able to be ensured that there's not these security risks that are involved in WebVR. But overall, there's this tension between the centralization and decentralization. The advantages of the centralization is that they can move quickly and break things. They don't have to worry as much about backwards compatibility, which means that they're able to be on the cutting and bleeding edge of whatever the latest technology is. That's essentially what you get with Unity and Unreal Engine. However, if you want to maintain that experience over time, you have to continually update it to the technology is always changing. It's always being updated. You have to continue to create new binaries. And so there's just a kind of a maintenance nightmare that's involved there because there's not open standards that are backwards compatible. Open XR may start to solve some of that stuff, but yet at the same time, there's always going to be API changes and you're going to have to just keep up to date with. these breaking changes that are happening with the nature of these centralized solutions like Unity or Unreal Engine. The advantage of the open web is that for people who want to have that archivability, you basically create it once and then you don't have to worry about updating it and you can look at it in 5, 10, 20, or 30 years from now and it should theoretically still be readable because of these open standards. Now, because of those open standards, it actually takes a lot longer for it to launch because they want to make sure that once they launch it, they don't have to do this kind of backwards compatibility dance of maintaining something that was only intended as a stopgap from what was essentially designed during the DK1 days of not even having positional tracking. So then you add positional tracking to the head and then to the hands and so they basically needed to do a reboot and kind of re-architect things to not only be able to be closer up to speed with where the existing VR technology was at but also to be able to look into the future and see what augmented reality needs to be able to add these various different extensions on top of this WebXR standard so that it could not only do VR but AR as well. And that standardization process has taken a long, long time, probably longer than anybody had really expected. We've all been kind of waiting for the green light of Go to be switched and for the innovation to really start happening. It's been a challenge because there's already a limited amount of headsets that are out there and that if it's not easy for you to create something and have people to actually experience it, Then you can create these little experiments that a few handful of people can see, but until you can know you can have the audience and it's a seamless experience for them, then I think we've been just kind of waiting for that to happen. So hopefully we're going to see some more movement along that route this year, and then it'll be really providing a strong foundation for the decentralized open web. I have an interview that I did with Brandon Jones at, you know, Oculus Connect 4, as well as with the Samsung Internet team, as well as with Albin from Sketchfab. And we talk a lot about some of these open standards, and hopefully I'll be able to feature a lot of those other conversations here soon as well. Because I think that when you see dystopian sci-fi, like Ready Player One, where it's really showing the negative aspects of decentralized solutions, and we see the downfalls of centralization when we hear about the Cambridge Analytica breach when it comes to private data with Facebook, then there's just a risk of having these centralized systems that are providing a high level of user experience, but then there's these different trade-offs of that, which is that you lose the amount of freedom and power control that you have over your own destiny. and then decentralized companies tend to have that consolidation of power and they have more control over our lives. And so the decentralization is something that is allowing us to just have more freedom and to be able to express ourselves in ways that we want to. And there's all sorts of other open questions around governance and economies and how do you create a society where people are nice to each other and deal with harassment and trolling and all this stuff I think is questions that we have to figure out and I suspect that it's going to be some combination of technological solutions as well as just like a cultural evolution where we actually have to kind of grow up a little bit or find ways that we can architect the balance between public and private spaces so that we can have the opportunity to have these public, serendipitous interactions with people, but also make it safe enough so that if we want to actually have an intimate, close conversation, maybe we switch context into more private environments and private instances within virtual reality. So that's all I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a listener supporter podcast. I am trying to live into the decentralized dream of having complete authorial control over this content. And in order for me to continue to bring you this coverage, then I just need your support and $5 a month. is a great level to be able to support me. And if more and more people start to contribute at that level, I'd be able to continue what I'm doing, but also to grow and expand into other ventures. So you can donate today at patreon.com slash voices of VR. Thanks for listening.

More from this show