Tony Parisi has a vision of the Metaverse that’s built on top of the open web with technologies like WebGL and three.js. Currently there are a lot of limitations and issues with latency that he believes will eventually be solved with browsers adding USB API support to be able to communicate directly to Virtual Reality HMDs like the Oculus Rift as well as all of the new different types of input devices for VR.
He talks about the Oculus Bridge by Instrument as a way to connect the Rift up to 3D content from a web browser.
Since the recording of this podcast at SVVRCon, Mozilla has also released a native app that serves as a VR API to that can communicate directly with a VR HMD. Here’s a good tutorial for getting started Mozilla’s VR API. Google’s Brandon Jones has a nice Quake 3 demo made with WebGL that you can use to test it out.
Tony recognizes that doing work for the open web is not an easy process, and there are a lot of challenges to it. He sees that there will be a lot of innovation and figuring out of the VR medium with proprietary tools like Unity. But he thinks that Unity’s popularity will be a short-term phenomena, and that even though there is a lot of warts on the open web’s approach at the moment that in the long-run it’ll be more popular and the chosen path for creating a vibrant and open Metaverse based upon the lessons of the Internet and open web.
Reddit discussion here.
TOPICS
- 0:00 – Intro. Founded Visi. Building VR apps. Part of VRML group to connect VR in an interconnected way.
- 0:40 – How do you see VR spreading on the web? WebGL and graphics cards enable making immersive environments with JavaScript. There will be new input devices, and sees that it will all eventually be supported by the open web.
- 1:36 – Existing plug-ins (at the time of SVVRCon). Oculus Bridge WebGL does stereo rendering for the browser. Desktop app that talks to the VR HMD. A bit of additional latency. Instant reality browser extension, but requires extension installation. Prefers downloading an app
- 3:45 – Uses open source toolkit three.js, so he didn’t have to write his own engine on top of WebGL. Most support behind it. Built a layer on top of three.js Visi Toolkit
- 4:57 – Used three.js plug-in with the Rift? Had to use the Oculus Bridge to connect it.
- 5:52 – Fragmentation into native apps with mobile and the future of the open web with all of these native apps? Believe in open technology, but it comes with a certain price. It gets harder with committees and group design. People go with tools like Unity to get the job done, but currently requires a plug-in and will eventually get compatibility with the open web. More people doing cross-platform HTML5 development now. Open web is the longer path, but perhaps be more successful in the long-run. Proprietary solutions will be popular at first, but has faith in the open web
- 9:33 – Unity vs. the Open Web. Unity’s popularity will only last a couple of years until the open web comes along
- 10:45 – Deal with latency issues with VR HMD. At some point browsers will adopt USB APIs to talk to VR HMDs
- 11:50 – Like VR because virtual worlds give people a more creative medium to build beautiful experiences
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:11.998] Tony Parisi: I'm Tony Parisi, founder of a San Francisco-based web agency called Vizi that creates WebGL applications, and now we're starting to build virtual reality applications for Oculus and other platforms. I've got a fairly long history in VR. About 20 years ago, I was one of a small group of people who created a technology called VRML, the Virtual Reality Modeling Language, which was a first attempt to create 3D content in a connected web environment.
[00:00:39.683] Kent Bye: I see. And so how do you foresee the virtual reality being integrated onto the web?
[00:00:44.845] Tony Parisi: So what's great about WebGL is now you've got hardware-accelerated 3D rendering. So whatever the GPU, the graphics processor, can handle on your phone or on your desktop computer is now accessible to be developed in JavaScript. So that really lowers the barrier for creating applications and content in 3D. 3D rendering is just a piece of virtual reality. It's the display part. There's a lot more to it than that. There's immersive hardware like Oculus and Game Face and Morpheus. There are inputs like the motion and game pads and Kinect and all new kinds of tracking devices that are being developed now with this new interest in virtual reality. So all those pieces will come together in my vision, in my view of the future, in an open web platform sort of way, where WebGL is the 3D rendering component of that.
[00:01:36.808] Kent Bye: I see. So maybe you could tell me a little bit about some of the existing plugins and extensions that are out there to be able to take WebGL content that's generated in the browser and to be able to experience it in the Rift.
[00:01:48.877] Tony Parisi: So I've actually been playing with one, so I can speak to that. It's called the Oculus Bridge. I don't have the GitHub URL handy in my brain, so I can't recite it right now, but if you Google Oculus Bridge WebGL, you'll find it on GitHub. And so what that does is it gives you a way to do the stereo rendering, format it for the Oculus, does it all into off-screen bitmaps, and then, as fast as it can, gets it onto the WebGL drawing surface in your browser, and will actually render pretty nicely in Oculus. And then there's a second piece to the Oculus Bridge which is a downloadable application that's a desktop app that is responsible for talking to the Oculus head tracking hardware. That part can't run in a browser without some kind of native code talking to it. The way the Oculus Bridge app works is it talks to the head tracking hardware and then sends information over a web socket. So it's a socket-based communication layer and then your web browser can talk web sockets and get the information. And it works pretty fast. It feels like there's a bit of latency. I don't know, people have said 10 milliseconds when I read about it online or whatever. I certainly haven't measured it in any technical way, but you can feel it a little bit. So that may not be the ideal situation, but it's the lightest, easiest way to get going with getting your WebGL content going in Oculus. There is another technology that I've forgotten what it's called. I think it's Instant Reality, but I may be getting the name wrong. that actually is a browser extension. So that'll talk to the USB port and will talk directly to the Oculus. So the good news about that is you have very little latency. I mean, it's like having a native application in terms of the head tracking. The issue is then you actually have to install an extension into your browser and a lot of consumers don't like to do that. So I've been sort of leaning toward the Oculus Bridge version because, yeah, even though you gotta go download that app to do the head tracking part, it's just an app, it's an install, it's sort of like getting your Leap Motion and installing the SDK for it. Once you've done that, it can talk to a web browser. So that's what I like so far.
[00:03:45.888] Kent Bye: And so, have you played with the 3JS at all?
[00:03:48.771] Tony Parisi: I'm a big 3JS fan. I've been using it for about three years, ever since I started working in WebGL. I'm not really a rendering jock. My personal skill set is I come in at a higher level than that. Animation, behaviors, interaction, and system support to do that well and fast. And so the idea of writing my own rendering system on top of WebGL was a bit daunting a few years ago when I got going on WebGL. I decided to not do that. I've written engines like that a few times in my life and I just didn't really want to do it. And I'd found out that there were, at the time, several open source toolkits that did it really well. And I looked at them all and 3.js seemed to have the most support. It was very popular. It was being used for a lot of Google demos. There were a lot of contributors on the GitHub project. So I decided to try that first, and I haven't been sorry. It's a really well-developed piece of software. It's open source. There's a lot of things that still need to be fixed in it, but it's got the most support behind it, most contributors, and a lot of functionality. So I use it in all my projects, and I've built a layer of software up on top of that called Visi. It's a Visi toolkit that I'm using in my business that uses 3JS but adds animation behaviors and other support.
[00:04:57.405] Kent Bye: Interesting, yeah. In looking at the 3GS website, I was in the process of writing an HTML5 Canvas web app, and I saw that there was a plugin for Oculus Rift, and that was actually kind of how the Rift and the 3D rendering kind of got under my radar, it was actually from that 3GS plugin. But I'm curious if you've used the 3GS plugin at all in terms of the Rift and what that gives you.
[00:05:18.253] Tony Parisi: Well, this Oculus Bridge project I was talking about has ready-to-go 3JS support. I wired that in. I had to fix a few issues with it to get a little bit technical. I found that the Oculus Bridge's 3JS rendering would only render from a single camera, so if you needed to do multiple cameras and multiple render passes, I had to do some hacking to do that. The camera itself couldn't be animated, so if you wanted to jump into a ship and start flying around, you couldn't do that, so I made some adjustments to that code. Thankfully it's open source, but it was already ready to go to render using 3.js.
[00:05:50.562] Kent Bye: I see, I see. With the web, there's been this trend of like, you started with the two-dimensional web, the web browser, then mobile comes along, and then you have all these third-party native web apps that are kind of creating all these little walled gardens of experiences and you know in terms of like the open web it's sort of splintered off in that way and I think with virtual reality has that same potential in terms of like creating these walled gardens of experiences that have to be downloaded and you know I'm just curious in terms of your perspective in terms of how the web could start to tie everything together in that way.
[00:06:25.084] Tony Parisi: So, I am a firm believer in open technologies and open standards and open web that comes with a certain price. Building technology that works interoperably, two browsers that render the content the same, the servers and clients talking to each other, that all gets a lot harder when you want to do it in an open environment and a level playing field. There are standards committees, there's a lot of potential for group design and bad stuff to happen. So a lot of people try to avoid it. If you're a developer today wanting to build something in 3D, it's very, very seductive to just go and use Unity because the tools are so good. You know it just works on all these platforms. That is, unless you want to build something on the web without a plug-in, then you're kind of stuck. They say they've solved that in the world of Unity by creating this WebGL output using Asm.js and Emscripten now, but they haven't shipped that yet. So for the moment, you build something in Unity, it won't run in your browser without a download. But it runs on Android, it runs on iOS, it runs from native Mac, PC, Linux, so that's great. So people tend to go with a tool like that so they can get the job done. Hey, I want to build a game. I'm going to use Unity. I don't want to learn a whole open stack full of open source stuff that half works. I don't want to get into a deal where maybe my WebGL doesn't render the same way in two different browsers. Now, with WebGL, that's not generally true. The implementations are very, very closely interoperable between at least Firefox and Chrome and Opera. Safari is pretty good, and now Internet Explorer is catching up. So not so true, but people get worried about that. So there's a convenient side to why the world fragmented with mobile, for example, back to mobile. But it's all coming back together. People are doing a lot more HTML5 mobile development after Facebook took a giant dump on it about two years ago and everyone was afraid it was never going to work. That's all come around again and people are doing cross-platform development in HTML5 for their mobile. The HTML5 standards have kind of caught up with mobile. There's a period pre-standardization where there's all this innovation and chaos. And the mobile world introduced that with all the accelerometer support, location APIs, and all the other things that were not in the web for a long time. That's all shaken out. The web's adopted a lot of those technologies. The browsers support them now. And now we can build something cross-platform in HTML that works really well. We're going to see that same kind of flying apart and then coming back together of 3D development, where there's going to be a lot of Unity stuff for the next couple years, especially for Oculus. Everyone's crazy about that. They want to build the next cool game and have a big hit, or they want to create the killer startup, and they don't want to create a tool in a whole stack. They're going to use Unity. On the other hand, there's people who are sort of more in my camp who will take the longer path to what turns out to be perhaps a bigger success over time, which is we're going to use the open technologies warts and all, and the world is going to catch up with us. At some point, a browser is going to have a USB interface, and then the Oculus head tracking will have no latency, and the other APIs that come into play, like with input sensing, motion sensing, and all that, will also probably all get adopted ultimately And then at that point, you know, you're back to a world of cross-platform development. But the proprietary technologies have pushed the envelope for us for a few years. That's just kind of the way life works.
[00:09:35.158] Kent Bye: Yeah, I personally had that HTML5 app and then tried to run it on my phone and it took a minute to load and that's sort of what led me to solutions like, hey, maybe I should write it once in Unity and forget about it and just have it work on my phone, which is where I want it to work. However, with Unity, you can't use, you know, binaries or DLLs and all this sort of Windows-specific things. A lot of the tools that are being developed and innovated are in these compiled binary packages that don't really transit to the web. So for me, I love the open web, but I fear that we're kind of getting into this zeitgeist of the unity that's sort of putting people in this mindset that is locking them out of the open web.
[00:10:14.280] Tony Parisi: I think it's awesome for Unity. I think we're going to see a lot of great content from this. I'm not concerned at all about this zeitgeist blowing everybody away, you know, sweeping everybody up and taking them away into a Unity-only world. I believe that's something that's just going to last for a few years and will coexist with other open web development styles of programming, by the way. But then ultimately, again, we'll turn into a curious sideshow to the much bigger metaverse that's going to happen over the next few years.
[00:10:44.964] Kent Bye: Yeah, and frankly, when I'm looking at the open web, I do fear that if I were to design it in the web, the latency, making people sick, literally, like not having good enough motion tracking, just make them feel nauseous. And how are you going to deal with that with your approach to going all in with the open web?
[00:11:04.062] Tony Parisi: So are we talking latency simply of the head tracking? So then going back to what I was saying a few minutes ago, I believe at some point the browsers, they haven't done this yet, but I believe they will adopt USB kind of APIs. Right now they're available in Chrome as extensions. So if you want a Chrome-only experience, it's not exactly open web, but it's browser-based. You can build something with USB extensions and it will talk directly to the hardware and that latency goes away. But I believe browsers will adopt that kind of a USB type of API over the years and that problem just goes away. You will be able to do it cross-platform. And in fact, it's arguable that those USB APIs could even possibly hit the market long before Oculus ships an actual production headset. So in which case it will be a complete non-issue by the time that actually happens.
[00:11:52.949] Kent Bye: Great. And finally, what do you see as the ultimate potential of virtual reality?
[00:11:59.348] Tony Parisi: This is a very personal point of view. I like virtual reality for its potential to give people a more creative medium. What's attracted me to virtual worlds, my early work in VRML, what went on with Second Life and other virtual world initiatives in the last decade, is the idea that people can express themselves artistically and creatively alone and then share it together in real time. Not so much puppet people waving at each other and typing in a text window, but actually building beautiful visual experiences and having those together. I don't think you need a Hollywood studio to build that for you. I think people can build those together with the right set of open tools.
[00:12:42.674] Kent Bye: Great. Well, thank you so much.
[00:12:44.416] Tony Parisi: You're very welcome, Kent. It was a pleasure.
