Magic Leap has been relatively quiet about the details of their digital light field technology after raising $542 million from Google, but they’re starting to be more public about what they’re doing. The most detailed information comes from a hands-on from the MIT Technology Review, and there was also a recent AMA with CEO Rony Abovitz.
I noticed that Paul Reynolds, a Senior Technical Director at Magic Leap, was following me on Twitter. I reached out to see if he’d be available for an interview for the Voices of VR podcast, and I was surprised to hear that he was interested.
Magic Leap is very supportive of what’s happening in the VR space because there’s a lot of innovation that’s good for the entire mixed reality ecosystem. They’re also very much interested in recruiting developers to create content for Magic Leap, and so I suspect that they see that the VR developers as being on the forefront of helping to create this new immersive world. Paul told me that they’re definitely interested in creating a healthy developer ecosystem where it’ll be possible to create Magic Leap content using the most popular 3rd party developer tools ranging from Unity to Unreal Engine.
Magic Leap is not quite ready to talk about specific dates or product plans right now, but are starting to release more information as they’re growing their company. Their presence at GDC will be more focused on recruitment because they see that people with a gaming background know how to create highly interactive, real-time 3D experiences.
The press refers to Magic Leap as an augmented reality start-up, but they’re cautious about labeling themselves as doing AR. They prefer calling it “cinematic reality” because they feel that digital light field is something quite different. Magic Leap’s technology actually has the capability to do fully immersive virtual reality, but the uncanny valley effects are leading them to focus on creating magical moments by embedding virtual objects into the real world.
There’s a lot of new information about what Magic Leap in my interview with Paul Reynolds, and I provided full transcript down below with some clean up.
Here’s the topics that we discussed:
PAUL REYNOLDS: I’m Paul Reynolds, senior technical director at Magic Leap. And at Magic Leap, we’re making a new way to interact to technology — a human-friendly way to interact with technology.
We’ve been in the news a lot lately talking about our ability to put virtual content into the real world. What we firmly believe is that the reality that we see is the function of the model-making function in our the brain. So basically, our brain is the highest-resolution renderer you’ll ever have. That render takes a lot of inputs including a light field. Everything our eyes do is they basically scan the light field to create signals for our brain to help it render what we see for us. We feel like that if we can create a digital light field that is similar to how we process the real world, we actually untap the power of the model-making function of the brain, basically tapping into perception.
So the interesting thing about being so natural in our output is that it’s an incredible engineering challenge. So we’re doing that as well as commercializing this and making a device and develop an ecosystem where we can create content for this light field technology.
My title is senior technical director. I’m in the content group, and at the intersection of all of these things between the engineering work we’re doing across many disciplines of engineering, the developer ecosystem, and the content development that we actually make the most of this technology with.
VOICES OF VR: So you’ve recently posted a slew of job different postings looking for “wizards” to come work there at Magic Leap with a wide range of hardware and software positions. So maybe you could go over some of the highlights of what type of positions you’re looking for.
PAUL: Yeah. Magic Leap is a really interesting place to work in, especially from an engineering perspective because we have incredibly complex and diverse engineering effort. So you’ll see job postings open for optics, opto-mechanical, embedded software, which is like firmware and drivers, mechanical engineering, perception and machine learning, all across the board. So much so we actually have literal rocket scientists that work in our group, and they’ve even said that the stuff we’re doing is more complicated than launching a rocket, which is kind of funny.
But that means that everything we do — because we’re putting virtual content in the real world, everything we do is real-time 3D. The philosophy behind the company with regards to that is that everything must be highly interactive and real-time 3D, and who’s the most experienced group of people? It’s video game developers.
My background is in video games. I’ve been in the video game industry for over 15 years now. We also believe that gaming helps push all of the APIs and technology that we have across the board because it’s got to be high-performance, interactive, and beautiful. And that’s a really hard thing. So you’ll see crazy engineering jobs across the board. And we’re just a scaling company in general. So we’re scaling quite a lot across the board this year. So it’s very diverse.
VOICES OF VR: Yeah, I even saw some like cloud engineers and also Unity developers. So would you say that Unity is your primary platform? Or are you also trying to integrate with Unreal Engine? Since it is speaking to a gaming audience, you’re explicitly calling out Unity.
PAUL: Yeah, I wouldn’t say that we have any particular primary platform. We basically want to support third-party development as much as possible. And as someone that comes from that world, I want to make sure that we put out a developer ecosystem that’s as friendly as possible to existing developers. So we’re looking to support any gaming engine across the board. Some engines may have nicer integrations than others just because they’re more popular. But our current architecture, we want to be accessible to anybody that has an existing game engine or investment in any kind of technology. We don’t want to be closed off to anything.
VOICES OF VR: There’s a mixed reality spectrum that ranges from fully-immersive virtual reality to a mix between augmented and virtual reality, and then to augmented reality, and then on down to sort of just normal, mundane reality with no augmentation at all. And so how do you see and understand this mixed reality spectrum ranging from virtual reality to augmented reality?
PAUL: Yeah so, we actually are really excited about all of the progression that’s happening in VR because that actually benefits us as well. I think anything AR gets unfairly judged as non-immersive, which I can firsthand testify that there can be extremely immersive experiences in an optical view-through experience.
I think that there will always be things that clearly better happen in a VR environment where you want to basically to take someone away to a completely different world. And just vice versa, they both have their strengths and tradeoffs. There’s going to be things that are much cooler that are integrated within the real world.
Some of the interesting things are, especially if you wan to think in terms of game development, there’s some tried-and-true game mechanics that have served us well over many years through the frame of a monitor that we’re figuring out even in VR versus an integrated context. Do those just port over? Or do we have to actually distill down what makes those things fun? And then how do we innovate those things in this new platform? So there are challenges.
And then there are some things that they just weren’t being done any justice on a 2D screen. Some of these games and game mechanics should exist in this new kind of platform.
So I think that there’s tradeoffs in both. And I think there’s strengths in both. And like I said, all of the technology innovation and all of the user interaction exploration that’s happening right now is beneficial to the entire market, as far as I’m concerned. So I’m very excited about all of these new things that are happening.
VOICES OF VR: Where would you classify Magic Leap on that spectrum? I mean, I know that it’s been labeled as augmented reality, but then you’ve also said that it’s something different like “Cinematic Reality.” Would you classify Magic Leap as augmented reality?
PAUL: Well so, from a purist’s standpoint our light field is being integrated into the real world light field. So we feel like we’re actually fulfilling the dream of what — these past years of what AR has been. We’re very excited about that. I think that we’re actually implementing it. So yeah, I — I mean this is virtual content in your real world. You know, if I had to pick one line of that, we’re definitely, on the overall view of integrating into the real world, we’re definitely on that side of things.
But we also feel like that’s the harder problem to solve. It’s really hard from a performance, and fidelity, and technology standpoint. It’s a harder problem to have content solidly located in the real-world light field.
What we feel like is that that could make for a really awesome VR experience too. You know, that same technology.
That’s why we we’ve been trying to define this whole new thing because we think it’s a whole new thing. I think it’s inspired by the visions and the dreams of all these — of what has happened in the past.
— KentBye (@kentbye) April 2, 2014
VOICES OF VR: At the IEEE VR conference last year, they had a slide of the spectrum between Augmented Reality to Virtual Reality. And what they said is that as you get closer to the end of Virtual Reality it’s more about cultivating the sense of presence into another world. Whereas when you’re in augmented reality, you have a presence of being in the [real] world, and your body is there and so you don’t have to recreate your virtual body. And so, would you say that Magic Leap has the capability of doing fully immersive, virtual reality then?
PAUL: Yeah. I mean, I think we definitely — By understanding more about the perception system and by presenting a natural light field, we certainly have that ability.
Our focus is putting virtual content in the real world.
We do talk about presence though. So the interesting thing is that’s kind of an overloaded term because for us. When we talk about presence, one of our huge strengths is that the virtual content feels like it’s there. As opposed to feeling like you’re somewhere else.
Our presence is more like, this thing, you have a sense of it’s mass, scale, it’s location relative to you. That’s presence in our –and it’s really strong especially in the near-range, the closer to you. We’re typically, especially in video game type of stuff, the closer things get to you, they start to break down.
One of our strengths is that it’s a really powerful moment when you see virtual content really close, and it feels even more that it’s there instead of getting more blurry or lower resolution.
So yeah, we do talk about presence, but in a totally different way. So we have that ability, you know, immersing even further is something we do.
VOICES OF VR: Right, so would it be able to like over take your entire visual field? I guess that’s the difference between augmented realities where you still see reality, and virtual reality you’re taking it into a completely different world. So would it be able to just completely overtake you’re visual system and have you believe that you’re in another world?
PAUL: I mean from a technical perspective, it certainly has the ability to. From a what-we’re-trying-to-accomplish perspective, I don’t think that’s what we want to do.
Because what we found is you get a much more magical experience when this thing appears in your real world.
Basically, our perception system is so tuned for the real world. You kind of hit an uncanny valley the more you try to overtake that. We’ve happened upon begin able to generate a light field that feels very natural and it’s perceived as natural, and we get a lot of benefits — our brain being able to fill out a ton of detail when you give it the right signals.
So our focus is definitely on brining magic into your real world. It’s just where our strength is. But our technology capability, we could certainly do that. But I just don’t think it’s something we really even consider right now.
VOICES OF VR: Yeah, I see. And that makes sense. I’ve found that the most immersive virtual reality experiences are actually the ones that are like low-poly. Because there is a lot of ability to project, and your minds accepts it more.
When it’s more closer to reality, it starts to reject those little differences between not being quite there. So it sounds like you’re able to to do that by taking that object and putting it within our reality.
And so with this cinematic experience, I’m wondering what you guys use internally, if it’s not presence — like within VR, it’s like putting you into another world and being “present there.” What is for you the ultimate yardstick for creating a successful cinematic reality experience?
PAUL: So for us, it really is about creating a magic moment. A successful moment is when someone causally accepts that virtual object is really in there in the room with them or in their space.
So for example, if we were to put a virtual dragon into the room. For the person to be talking about the dragon as if it’s there instead of how it’s rendered or they technical details of it. Just the causal observer is talking about, “I want the dragon to do this.” And all the things we want to be able to do.
To us, that’s when you’re no longer fighting disbelieve. People are just delighted. It’s no longer you trying to explain to them what this could be. They already get it, and they just accept it. And then it’s more talking about all of the other things they want to happen.
So the big bar is when people not consider this technology. It’s just this magic that’s in their world. We don’t talk bout photons and pixels. We talk about this object, the things I can with it now that I have this magical ability to put whatever I want in the room.
So for us, that is the crossover moment that kind of ties into the presence thing as well like when you feel this thing is in there with you.
VOICES OF VR: And with Google Glass, that was something where people were wearing it out in public, it had a camera, and it had the capability to broadcast the lives of other people onto the web without their full consent. And I think that we saw that there was a lot of backlash to that because it was something that kind of violated the privacy of other people.
Virtual reality seems to avoid that by being more of a private experience, but with AR it’s something that you’re wearing out in the world. So I’m wondering if you guys that see if AR needs to be more of a private experience at first before you take it out into the wide world? Or if you see that it kind of needs to demonstrate value more before it’s released onto the streets of mainstream society?
PAUL: Right. So I’m actually really proud that internally we’re extremely sensitive to these privacy concerns. It’s part of the why we’re being so careful on a lot of ways.
So from my perspective, there’s two major privacy concerns. There’s the user’s first-person data, and then there’s the data of the people around them.
I don’t think we can control where our users go. So I think that we just have to assume they’ll be in public places. So given that, I think what we have to be sure to do is that any sensing of the world that we do that we do it in an ethical way. That we don’t take more information than we need. And that any data that we do have for that user is transparent and accessible to that user, and we give them some control of it. It’s basically their data. That’s how we feel about it.
So to make virtual content even more integrated into the real world. You have to be able to know more about the real world in terms of sensing and context.
So the line between, “Is it worth it to take this much more potentially private data to make the experience more magical?” So towing that line. And giving the user access and control to it. I think’s a better approach as opposed to trying to restrict usage.
We do feel like we’re basically reinventing the way humans and technology interact, and we want that to be an all-day, wherever-they-are basis. I think it’s better now to be respectful, and aware, and ethical with data than to ignore it. And just try to make the coolest possible stuff, and then have to retrofit privacy and things like that. I’m really happy that we’re conscientious of that.
In addition to the fashion issue of people wearing these things in public. We want to make something people aren’t afraid of, they’re excited about. And they want to wear whatever we make proudly, and give them a reason content and experience wise to do that.
PAUL: I think, honestly, the most exciting thing is that UI as we know it will probably go away. I think the graphical user interface and the mouse and the keyboard — it was a huge step towards making technology accessible to humans. And the same with touch screens. I think mobile touch screens made these things even more accessible.
But there is a limit to a 2D, flat plane interaction. Touch screens gave this a second wind with multi-touch gestures and such. But I feel like we’ve kind of stretched the limits of — we’re basically starting to limit how much technology we can utilize by how we interact with it.
So I really like the idea that the best UI is not the one that’s in your face, and that the user interaction only interacts when you’re ready for it. And so one thing we’re pretty — in our design-thought — We don’t want to be this Time Square, heads-up display, head-locked display in your face all the time. Really if you’re not using it, it’s not in your face.
That’s where the magic part comes in and permeates through beyond just the experiences we create. But the interaction of the technology where when you call upon it and when you need it, then it’s there. And when you need it, it goes away and it’s a natural interaction.
So to me I think that is going to be the most exciting thing. Because as someone who has grown up through the boon of the modern web and search, I have pretty good Google Fu. If I need to research or learn something, I can drill down pretty quickly.
But it kind of goes back to the mom test we talked about this week. My mom, if she had a more intuitive way of accessing this power of search in a more natural way, I think it’s going to accelerate everything. It’s literally world changing the way that we can access technology in a totally different way.
That’s the most exciting to me, which is probably more at a higher level from a user interaction standpoint than actual design.
So we’re doing of tons of exploration work with that because it does change everything when you can arbitrarily stick a virtual object in the real world, it opens up so many possibilities.
VOICES OF VR: VR has had a long history of being inspired by science fiction, and one of the things that I find really interesting is that Magic Leap has actually hired on the author of Snow Crash, Neal Stephenson as your chief futurist. Maybe you could comment of on how Magic Leap is using the visionary skills of a fiction, sci-fi writer and moving into non-fiction & how he’s been helping to shape these technologies that he’s been dreaming about for over two decades.
PAUL: Yeah. So I was fortunate enough to be a part of Neal coming into the company. He’s very practical and very involved.
So one of the first things that you’ll notice about Neal speaking with him is that he’s a very — which is very obvious when I say it out lout — but he’s very thoughtful with his words. So when you have a conversation with him, he will pause and think, and then he will speak.
One of the first dinners we had was, Rony, our CEO, basically saying, “You are the metaverse guy. So there’s all types of way we could work together. The first of which is you’ve already envisioned this future. You could be our,” — I think Rony used the term — “philosophical spirit guide on a lot of things.”
And so Neal paused as he does. And he was like, “I think I’m more interested in making things.” I think the term he used was that “I don’t want to be a head on a stick. I actually want to create things in this.”
He’s definitely the chief futurist, but he’s extremely hands-on. He has a whole list of things he wants to create with this technology. And I think the next day we were bringing him and showing him stuff, and he actually had one of our engineers to take a part a thing for him to look at the electronics inside because he was very interested in how it worked.
So I think Neal is guiding us by actually making things. As opposed to us to us like running things by him.
But without a doubt, his writing has influenced us heavily. And it is pretty surreal at some times to be talking to him a bout these things. But he’s very practical.
VOICES OF VR: You mentioned the metaverse, and he’s famous for coining that term. But that sort of implies this interconnection of completely, fully-immersive virtual spaces, kind of like the Internet. It seems like in some ways augmented reality would be bound to more the physical geography. Do you feel like AR would be able to provide like a small window into the metaverse? Maybe you could comment a little bit how you see augmented reality fitting into the metaverse? Or if you feel like it’s going to be a sole artifact of virtual reality?
PAUL: I think it’s up for debate. One thing we concentrate a lot on — To really put virtual content in the real world, the most magic things happen when that virtual content respects the constraints of the virtual world. For example, if I did have that dragon flying around the room. If that dragon did not interpenetrate or fly through anything that’s real.
But it’s also not a limited constraint so that we can take advantage of the fact that we’re expanding beyond the physical world. An example I would give is that if you and I are in the same room together and we’re sharing a cinematic reality experience, like we’re playing a virtual game together on a table top. We’re sharing that same room together physically. But then imagine that it’s a four-player game, and there’s two other people playing with us. And one person is in Seattle and one person is in New York.
So we’re sharing the same physical space, you and I, but the four of us are sharing this kind of virtual thing. And they’re in their respective physical places. So who’s to say who’s in the actual real place or not? Which starts to get pretty meta.
And to me that’s how I see — I think the metaverse idea, I think a lot of people think about it kind of like how you articulated it, like this alternate dimension that you have to jump in to.
But I think you could just as easily consider it this layer on top of our real world where it’s all relative to our personal perspective if you’re in the real spot or not.
So no, I don’t think it’s limited to a VR experience at all. If anything, I think we could actually bring this kind of metaverse to the physical world.
VOICES OF VR: The last question I wanted to ask is — What do you see as the ultimate potential for cinematic reality?
PAUL: Kind of like what I mentioned in the UI thought that — Access to the power of technology has been limited to really a select few that have the ability to interact with it in the current forms that we have to interact with it.
So I think the ultimate potential is presenting the power of information in search, and just technology in general that has benefited a lot of people in my generation’s lives. That being a much more natural and instinctual interaction that we all have.
So people that are power users now actually have even more powerful. And that people that are really out of touch will actually have access and it makes sense to them and it doesn’t require this explanation and bring-up to get them access to it.
And related to that, the fact that we’ve been working through screens for the past hundred years. You know, I think that’s going to weaken greatly. That putting virtual content into our world instead of us diving into those.
And then that continues to evolve, we kind of — I don’t know about you, but I have these kind of different interaction modes where I might be on my phone for this, but I type a longer e-mail so I’ll go to my computer. Or I’ll want to watch a movie so I’ll go to my couch in the living room. We all have these kind of modal interactions with technology. I think that’s going to blur much, much more.
I think you’ll no longer feel burdened by having to use one particular technology for one particular thing you want to do. I think that’s really the world-changing technology for some, because of my access to the information on the Internet and my ability to drill down through it.
So I think if we can give me more access and give more people access in general then that’s pretty huge. I think that’s the ultimate potential of this.
VOICES OF VR: Awesome. Well, thank you so much for joining me today.
PAUL: Yep. Thank you.
Reddit discussion is here.
Theme music: “Fatality” by Tigoolio