Vlad Vukićević is a technology innovator at Mozilla trying to figure out how virtual reality fits into the web. He’s the inventor of WebGL, and he’s been working with Unity & Unreal Engine 4 to be able to create plug-in free gaming and VR experiences on the web.
Vlad sees that we’re on the cusp of a VR revolution, and he wants to be sure the web is treated as a first-class citizen. He talks about JavaScript performance, asm.js, libraries like three.js, Unity’s IL2CPP, and the challenge of working with compiled libraries on the web.
We talk about the process of giving web developers the data and output capabilities that are necessary for VR on the web. And then talks about the implications of responsive web design when adding an additional dimension and abilities to provide a fully immersive experience.
Reddit discussion here.
TOPICS
- 0:00 – How VR fits into the web. WebGL and enabling high performance games on the web via Unity & UE4, and making sure the web is a first-class citizen in this upcoming VR revolution
- 0:57 – What is the importance of OpenGL and how did that lead to WebGL? Making sure the WebGl runs as well in mobile as it does on desktop PCs.
- 1:55 – Is three.js using WebGL in order to do the 3D rendering functionality that it provides? All of the browsers provide access to WebGL
- 2:28 – Is JavaScript fast enough to be equivalent to natively compiled code? asm.js A lot of graphically intense content comes from the graphics card, and Mozilla believes that JavaScript is up to the task
- 3:39 – What is asm.js? A subset of JavaScript design to be highly opimizable.
- 4:28 – The meaning behind the 1.4x times slower metric. The goal is to get as close to 1x parity as possible.
- 4:58 – Other cases in which asm.js can make the application faster?
- 5:26 – What is IL2CPP for Unity, and how does that enable Unity to get to the web
- 6:30 – Unity’s embedded player and moving to a plug-in free experience for the Web
- 7:21 – Moving away from walled gardens
- 7:38 – Custom binaries as being a blocker for putting VR content onto the web.
- 8:49 – Being aware of decisions for your VR experience that may prevent it from putting it on the web
- 9:37 – What’s the potential for the web and VR. Presenting content in VR via the web. Giving web developers to the data and output capabilities that are necessary for VR.
- 10:39 – The future of responsive web design after adding an additional dimension
- 11:42 – What makes you excited about virtual reality
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.975] Vladimir Vukicevic: I'm Vlad Vukicic. I'm currently a technology innovator at Mozilla, trying to figure out how VR fits into the web. Mozilla has done a lot of work with bringing 3D to the web with WebGL, and more recently we've done a lot of work in the game space with enabling high-performance games with engines like Unity, Unreal Engine, and others. And we feel that we're kind of on the cusp of a VR revolution, and we really want the web to be there as a first-class citizen, part of this new world. We think that the web is a fantastic way to distribute applications, to provide access to content, and we want VR to be there. People often talk about the metaverse and about what the next virtual reality is going to be like. We already have it in 2D, and it's called the web. So let's figure out how to actually get that extra VR aspect into it.
[00:00:58.094] Kent Bye: I see. And so maybe you could describe to me the importance of OpenGL, what OpenGL is, and how that led to WebGL.
[00:01:05.806] Vladimir Vukicevic: Sure. So OpenGL is a standard for doing 3D graphics. It has been around for many years. And when I first started looking at bringing 3D to the web, OpenGL was kind of the natural fit. It was an open standard. It was widely used in the industry. And conveniently, there was also a subset of it called OpenGL ES that was already optimized and designed for mobile devices. One of the key initial goals of WebGL was to make sure that it ran equally well on mobile as well as on desktop. It's taken about three years now since the release of WebGL 1.0, but now WebGL is pretty much everywhere. Microsoft finally added support for WebGL to Internet Explorer. I was pretty happy about that. And so now 3D content is accessible to pretty much anybody with a reasonably modern web browser, whether that's on desktop and on mobile. And OpenGL was kind of one of the key driving factors in making that possible.
[00:01:55.763] Kent Bye: And so is 3JS using something to be able to use OpenGL in order to get its 3D rendering?
[00:02:02.834] Vladimir Vukicevic: Yeah, so 3JS is built on top of WebGL. So all the browsers provide support for WebGL, which is kind of this open GL in the web API. And then libraries like 3JS make it easier to access this. They make it easier to think about in terms of primitives, in terms of meshes, actors, post-processing effects, instead of thinking about the actual low-level calls. So 3JS is using the swept ecology under the hood, but making it easier for developers to do these kinds of things.
[00:02:28.993] Kent Bye: Yeah, 3JS and their website has an Oculus plug-in, and that was actually what kind of helped introduce me to what was happening with that. And I guess one question that comes up in terms of JavaScript and the web is, can it be performant enough to be as good as natively compiled code? Is JavaScript up to the task for that?
[00:02:47.717] Vladimir Vukicevic: Absolutely. I think Mozilla has shown over the past year that you can get JavaScript through Asm.js running within 1.4x or so of native code performance. That's already compelling. That's basically equivalent to running last year's processor model instead of this year's, which is already well within kind of the target of of any game or application that you might actually want to ship. We've had tremendous success porting complex and high-end engines like Unreal Engine 4 and Unity to the web. They run at 60 frames a second, even at 4K resolutions and whatnot. Because it turns out that once you're doing a lot of this graphically intensive content, a lot of that power comes from your graphics card. So JavaScript kind of gets out of the way there. When you're talking about doing physics processing and all these other pieces, that's where kind of the raw computational power to things like Asm.js, which is a subset of JavaScript that can be highly optimized, really come into play.
[00:03:39.791] Kent Bye: And so yeah, maybe describe to me what is Asm.js?
[00:03:42.940] Vladimir Vukicevic: So Asm.js is a subset of JavaScript, meaning that it is just JavaScript. Any JavaScript interpreter or runtime can execute it, but it's designed to be highly optimizable, meaning that it takes away a lot of the things that involve a lot of extra work on the part of the JavaScript implementation. So it deals mainly in numbers, arrays, floating point values, loading and storing things, mathematical operations, the kind of very low-level things that you would expect from an assembly language for a CPU. And so it brings that to JavaScript. It has a rigorous structure so that a compiler can say, hey, this is just Asm.js. I recognize this. So I can generate really highly efficient code from that. And that's where we're seeing this 1.4x performance number come from, is from Asm.js.
[00:04:27.712] Kent Bye: I see. And when you say 1.4x, does that mean it's 1.4 times faster than what?
[00:04:31.975] Vladimir Vukicevic: 1.4 times slower. It's kind of an odd number to talk about. Like, I think it translates into 30 or 40% slower, however you do that reciprocal. Like, you're looking at better than twice as slow.
[00:04:44.101] Kent Bye: I see. Okay. I would expect it to be like 0.8 or like to do it the other way.
[00:04:48.167] Vladimir Vukicevic: Yeah. I, we've had that conversation actually many times how to actually represent that number, but kind of our goal is to get to as close to, you know, one X parody as possible.
[00:04:59.850] Kent Bye: When you first said 1.4x, I was like, oh, wow, it's faster.
[00:05:03.052] Vladimir Vukicevic: Unfortunately, it doesn't actually make it faster. So interestingly, there are actually some cases where some of the optimizations that go into play in generating Asm.js, and Asm.js can be compiled from C and C++. So that's kind of where the start comes into play. We've actually seen a few cases where once we do the optimizations, the generated code is actually faster than what some compilers generate natively. That was kind of always an odd outcome, but kind of cool to see.
[00:05:26.252] Kent Bye: OK, and I've seen announcements from Unity talking about this IL CPP, and maybe you could talk about what that is and how that fits into everything.
[00:05:35.299] Vladimir Vukicevic: Sure, IL 2 CPP is entirely Unity's product, so Mozilla was kind of not involved there, but that's what they're using to be able to port their .NET runtime, which generates IL, to C++, which then they will feed into our script and compiler, which will generate Asm.js out the other end.
[00:05:51.391] Kent Bye: I see. Yeah, I think that's one of my sort of frustrations in terms of looking at the field is that they provide C-sharp and they provide the Unity script or JavaScript, but there's no native C++. And so is this sort of their intermediary way of saying if you write in this new proprietary scripting, they can convert it to C++? And then I'm just trying to figure out, like, what is this thing? Is it just a proprietary thing or does it actually have legs outside of Unity?
[00:06:17.745] Vladimir Vukicevic: It is largely just kind of an internal part of how they get to the web. So, like, in Unity, you'll still be able to write in C Sharp or Unity Script or any of their languages. And then IL2CPP is just a piece that they will then use to actually be able to get over to the web. Okay.
[00:06:31.888] Kent Bye: And right now, when you do a Unity export, it's kind of like this embedded, almost like Flash player, you know, kind of just overtakes the screen. And, yeah, I'm just curious in terms of whether or not you see with these changes to moving to OpenGL, whether or not it's going to be a little bit more smooth or immersive in terms of integrating some of these experiences into the browser.
[00:06:51.521] Vladimir Vukicevic: So with the WebGL and HTML export that's coming in Unity 5, there will be a plugin-free experience for Unity content in the web. They're still going to support the plugin as well because they have a lot of legacy users of it, but you'll be able to take Unity content and it'll just be straight HTML and JavaScript. The actual output will likely be somewhat similar in that there will be kind of one large data package, but applications could do work to optimize that so they can take advantage of web features like streaming downloads and incremental asset delivery, things like that.
[00:07:21.403] Kent Bye: Wow, OK. So it sounds like they're going to be able to have a little bit more seamless, fully web experience instead of this kind of walled garden, I guess is what I'm hearing.
[00:07:30.747] Vladimir Vukicevic: Yeah, hopefully. I mean, Unity's been a great partner for us to work with. And I'm kind of looking forward to seeing what they do with the web stuff now that they have the core of it working.
[00:07:38.910] Kent Bye: I see. And one other issue that I've sort of come into in terms of loading in dynamic linked libraries into Unity, they say, oh, well, actually you can't even output it to the web because it's a security issue. And so do you know of ways that that could be gotten around? I mean, one thought is like, oh, well, do I have to sort of rewrite this DLL in a way that can be compiled down to then it can be exported to fully web standard code? Or what happens when you're dealing with these DLLs that may only be specific to like a Windows runtime or something like that?
[00:08:11.242] Vladimir Vukicevic: There's never going to be a way to actually take a binary DLL and bring that in or even get access to the general operating system functionality. But if you have some code that you've written as C and C++ that you're just using, that's doing some processing, maybe some custom physics or custom animation code, you should be able to just bring that through the whole path and actually end up with JavaScript at the end that can then work with the rest of the web compiled app. That's really up to Unity to enable that integration. There's nothing technically stopping that piece from happening. But again, that would just be purely for data processing. You couldn't go and talk to Windows. You couldn't go and talk to the Mac OS native APIs from that.
[00:08:49.281] Kent Bye: OK. So yeah, I see that as a potential roadblock in terms of web adoption, because the project I'm working on, I wouldn't be able to even output it. So I'm just trying to figure out, like, what's a good general workaround should people be aware of? Like, oh, hey, if you're using this Unity plug-in and it's using this binary DLL, it's going to prevent you from putting this on the web.
[00:09:07.564] Vladimir Vukicevic: Yeah, I guess that's true. If you are using kind of a purely binary DLL in there that you have no kind of control over, or perhaps you're integrating with some hardware or something that you actually need kind of raw device access, the web is going to be a challenge for you. I suspect that even in the Unity plugin, they couldn't do that either. So you're kind of stuck just doing kind of a standalone application if you're doing that. But I actually, I don't know how many games actually do use custom components kind of to that degree. So we'll see.
[00:09:36.870] Kent Bye: And what do you see as sort of the potential for the web and virtual reality put together?
[00:09:41.772] Vladimir Vukicevic: I think there's a lot of exciting potential there. Kind of the most obvious thing is we really want to make the web a place where people can experiment with VR. So regardless of what head-mounted display you have, what positional tracking you have, we want to provide a nice interface for that in web APIs to get you access to that data, to get you access to that output. That's one of the things that I'm kind of focusing on initially. That's probably going to include integration with some of the engines, but we also want to look at how do you integrate that with the web as it is today? How do you take HTML and CSS today and present it in such a way that it makes sense in VR? Does the concept of a VR browser make sense? What might the user experience look like in a VR browser world? There's a lot of kind of interesting areas to explore, but definitely kind of the first step is going to be just giving web developers access to the data and the output capabilities that are needed for VR and then kind of seeing what they come up with. That was successful for WebGL and I think it's gonna be successful for VR as well.
[00:10:36.667] Kent Bye: Yeah, it makes me think of the shift between making your website responsive by fitting both on PC browser but also a mobile browser. Now we're kind of moving the responsive design into the third dimension in some way. Is that what you kind of see? Depending on whatever input device you're seeing, it would be rendered out in either a fully immersive virtual reality, mobile, or a flat screen 2D PC screen.
[00:10:59.847] Vladimir Vukicevic: Yeah, absolutely. I think that's one of the most interesting pieces for us to try to figure out. What are the smallest set of primitives or changes needed to add to the web as it is today, to add to HTML, to add to CSS, that would enable people to do this kind of responsive design? You can go from small screen to large screen to VR. all from the same website. Like, what does a news website look like if all of a sudden you have access to a VR display? Is it still just trying to display text? Does it focus more on images? Does it actually bring in 3D content in there? I think there's a lot of interesting things to explore, and we want to figure out how to let people explore that within the context of HTML and CSS. So they don't have to actually become 3D programmers themselves, but you kind of bring in VR elements to existing websites.
[00:11:43.241] Kent Bye: And finally, what makes you the most excited about virtual reality?
[00:11:46.834] Vladimir Vukicevic: I'm excited because it seems like it's actually going to happen this time. Going around the Silicon Valley VR conference, there's a lot of excitement, there's a lot of people doing interesting things, and people are attacking the problem from many different angles. And so I think it's going to happen. And that's why I'm really excited to be working on the web aspect of it, because I think we're kind of on the cusp of a VR revolution, and I want to see it across all facets of the industry. Great.
[00:12:11.216] Kent Bye: Well, thanks so much.
[00:12:11.916] Vladimir Vukicevic: Yeah, thank you.
