In this podcast episode, I share some of my take aways from Magic Leap’s LEAPcon as well as talk with the creators behind the Sigur Ros interactive music experience called Tónandi, which is the best interactive AR experience that I’ve had so far. I talk with the creative & tech leads & Co-Directors Mike Tucker & Steve Mangiat as well as with Rebecca Barkin, who is a Senior Director, Magic Leap Studios & Senior Director of Strategy & Partners at Magic Leap.
My bottom-line take away is there’s a tension between the idealism of the potential of head-mounted AR and the pragmatic reality of how to sustainably get there. Magic Leap is choosing to empower artists to push the potential of what’s possible in terms or storytelling and cultivating an emotional connection, which is symbolically reflected in their first experience from Magic Leap Studios being a cutting-edge, interactive ambient music experience like Tónandi, which translates into “sound spirits.”
There’s been a lot of skepticism around Magic Leap’s marketing hype, but the story has had to be ahead of where the technology is because it takes time to produce a consumer product of something that doesn’t exist yet. Will the technology be able to live up to the potential of the stories that Magic Leap is telling? I’m convinced that spatial reality will be ubiquitous by either 2025 or 2045, and that it’s just a matter of time before their vision comes to pass. How we get there is another question.
But the developers I’ve talked with are overall genuinely excited about the potential, and they tell me that Magic Leap is the best AR HMD on the market today. No one knows all of the best practices for designing mixed reality experience yet and there’s a lot to figure out, which excites the pioneering developers to help figure it out.
On the pramgamtic reality side, the price is a barrier for nearly every application beyond enterprise apps, architecture, medical, or high-end immersive entertainment. But the phone-based AR ecosystem with the iOS ARkit & Android AR Core is providing a more robust and viable ecosystem for many developers to bootstrap themselves with viable immersive agency contracts that allows for a progressive enhancement from phone-based AR to Magic Leap consumer to Location-Based Entertainment.
I have many more take aways and impressions that I share in today’s podcast episode as well as in the Twitter threads down below.
LISTEN TO THIS EPISODE OF oTHE VOICES OF VR PODCAST
Here is a Twitter thread with my reporting from Magic Leap’s LEAPcon
THREAD:
My AR journey at @magicleap's #LEAPCon begins.
Today is demo day for me + connecting with AR developers — Come say "Hi!"
Wednesday morning is the keynote.
I'll periodically update this thread with updates and insights over the next couple of days. pic.twitter.com/1QRLuBJxJU— Kent Bye (Voices of VR) (@kentbye) October 9, 2018
Here is a video made by JJ Castillo that captures the vibe and excitement of LEAPcon.
https://twitter.com/JJCastilloVR/status/1051909202338357249?s=19
Behind the scenes of Tónandi | Sigur Rós in collaboration with Magic Leap Studios
https://www.youtube.com/watch?v=4CPKeleqQHo
Trailer for Tónandi
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to The Voices of VR Podcast. So last week I had a chance to go to the Magic Leap LeapCon, which the experience felt like the very first Silicon Valley virtual reality conference that happened way back in May 19th and 20th of 2014. That's where I started the Voices of VR podcast. And I have to say that the energy that was there was palpable. There was a lot of excitement from the developers who were wanting to get their hands on to the technology to start to innovate and push forward what is possible with head-mounted augmented reality. Now there is this tension though between the idealism of what's possible versus the pragmatism of what's practical. I think this has been the story of Magic Leap for the last five years is that there's been a big grand vision of what is possible with this new spatial computing, cinematic realities, and augmented reality on the head out in the world. And this is a vision that when Magic Leap started, this didn't exist yet. And so their marketing and selling of that vision has always had to be further ahead to what the engineering pragmatic reality was. And so now that the first headset has been out, we're able to match what some of the marketing hype versus the pragmatic reality and there's still a tension there and I would say that this is a tension that exists in every single company where you build a product and you have to sell the story of what that product is and there's always going to be a slight mismatch and perhaps even the marketing being slightly ahead of what the vision of what you've had time to actually build is. Ideally you have like perfect synchrony between those two but In the case of Magic Leap, I think they've gotten themselves in a bit of trouble from having a little bit too far and too aggressive of marketing that has not necessarily matched with what the pragmatic reality is. But that said, from talking to the creators and the developers, which is what I take a sampling from what the thoughts and the sentiment is within the community, And some of the biggest creators that I know from the virtual reality scene are super excited about this medium. It's the most solid augmented reality platform that's out there. It's designed. It's comfortable. It's ergonomic. It looks a lot cooler than the HoloLens. There's eye tracking. There's a lot of things that are going to be coming to it. The only thing is that it's super expensive, and so it's not going to be very practical to buy one. Although it's $2,300, you know, if you were to buy a VR headset with a PC and a computer, you're talking about the same price range of a couple thousand dollars. And I think over time everything is going to come down in price, but right now there's this tension between the idealism of what's possible and the pragmatism of what's realistic. And there's a lot of different things of my experience of Magic Leap that I hope to dive into, but I wanted to start my coverage of Magic Leap with Tanandi, which Rebecca Barkan, she's an executive producer on Tanandi. She's also the senior director of Magic Leap Studios and also the senior director of strategy and partners at Magic Leap. And there was a promotional video that came out about Tenandi and she said that Tenandi as an experience was a statement of who they are, that they wanted to tie themselves to artists and creators and be able to show that this is a technology that is able to make an emotional connection to people. And I had a chance to experience Tenandi. One of the Magic Leap employees, when they were visiting Portland, stopped by my house and gave me an opportunity to do Tenandi. And I have to say that it's one of the best AR experiences that I've had so far. Something about being able to viscerally reach out and touch the environment with your hands gives you this sense of embodiment and this synchrony between your actions and agency in the music and it creates this kind of synesthetic effect of the music and the visuals being connected but it was in the context of my living room and so it's somewhere I was already familiar and it's just like these beautiful visuals and this perfect balance between curiosity of not knowing quite what to expect but it not being so chaotic is that I don't I don't understand any of it. And so there's this process of exploration and discovery and awe and wonder, and it's just an absolutely beautiful experience. And doing this interview with both Rebecca Barkan and the two creative and tech leads of the project, both Mike Tucker and Steve Mongeon. Like I have even a deeper appreciation of what they were able to do because they're really trying to take the medium and do this full exploration of interactive audio and really pushing forward what it means to have a music experience in a spatial reality. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Rebecca, Mike, and Steve happened on Tuesday, October 9th, 2018 at the Magic Leap conference in Los Angeles, California. So with that, let's go ahead and dive right in.
[00:04:40.217] Rebecca Barkin: So I'm Rebecca Barkin and I work on the Magic Leap Studios team. I served as executive producer on the Tenandi project and also senior director of our strategy and partnerships for Studio on the Whole.
[00:04:52.025] Mike Tucker: I'm Mike Tucker. I'm a creative and tech lead at Magic Leap and was a co-director on Tenandi.
[00:04:59.030] Steve Mangiat: My name's Steve Monjott. I was also creative and tech lead on Tenandi.
[00:05:02.853] Kent Bye: Great, so maybe you could give me a bit of a backstory for how this specific project came about. Because I had a chance to experience it and I felt like it was a really great experience that showed the potential of interacting in a spatial environment with your hands and this interactive music video featuring Sugar Roast. So maybe you could tell me a bit about how this project came about.
[00:05:22.832] Rebecca Barkin: Sure, yeah. I mean, these guys were here before me, but you know, Roni in short, he's a huge lover of music. And so long, you know, early in the days of Magically, he kind of started these relationships with artists and he always really wanted the platform to be used by artists and creators and to be easy to use and to make really emotional experiences. So one of the first bands in the door was Sigur Rós. And so these guys can probably speak to the early days of concepting.
[00:05:50.663] Steve Mangiat: Yeah, I mean, at the time this project started, there really wasn't anything else out there like this, you know, an AR, fully immersive, where you can interact with your hands and other inputs. And I think Roni really saw kind of into the future, because this is going back five years.
[00:06:05.078] Mike Tucker: Yeah, at the time we only had what was called the Beast, which was this massive prototype, and it was immovable, so you could not get a sense of what spatial design would become, but they were able to sort of think into the future and came up with this idea of music spirits, and that's what Tanandi literally means in Icelandic. And so the idea is you have these music spirits inhabit your space, and together they form this like musical soundscape and adapt to your environment.
[00:06:32.789] Rebecca Barkin: When I joined, these guys have been kicking around a prototype for how you actually interact with the sound. Early on, we had been using a controller to interact with the sound. And one of the decisions we made when we kind of took a pause from prototyping and said, let's make this thing official and let's actually put a real scope around it and start to dig into what the timelines look like. One of the first things we did was evaluate it creatively from a very sort of critical perspective. And what we found was that the controller was sort of acting as a little bit of a barrier to the kind of interaction we wanted people to be able to have with sound, to stretch it and really manipulate it. We wanted them to be able to feel it.
[00:07:08.938] Steve Mangiat: Yeah, because this is one of the first experiences too, we want it to be intuitive, so there's no tutorial or UI when you jump in, and anyone can be able to go in there and just start exploring with their hands. I think that was another advantage of using your hands versus the controller, which has buttons to learn things like that.
[00:07:26.239] Mike Tucker: That was really the core of the design experience. We want people to go in there and really make discovery and the... Yeah, discovery and exploration.
[00:07:35.632] Steve Mangiat: We tell people like, hey, just explore with your hands like you're a kid again. And usually, you know, people pick up on that pretty quickly.
[00:07:42.990] Rebecca Barkin: They were really cautious, they didn't want to make a music video. And this is no knock on VR, but in the early days of VR when we were embracing artists and creators, it was like, how do we put someone at a concert? But then you don't have that exchange of energy, right, with the people next to you. The heart was in the right place, but we wanted to make something that allowed you to sort of understand the brand and the music and the sound that you love so much and have a much more intimate relationship with it and to be able to compose. But I think one of the really interesting things that these guys worked really closely with, Sarah Hopper, who's the art director for the band, creative director for the band, and Paul Corley, who's their musical director, was like, how do you pair visuals with sounds and give people the right amount of compositional tools without it becoming kind of intimidating and uncomfortable?
[00:08:29.795] Kent Bye: Yeah, the thing that I was really struck with was how interactive and participatory it was, even the audio reactive nature of it. And Jason Riggs, one of the things he told me is that the future of music is going to be immersive and interactive. And so thinking about, well, what does that look like in terms of how to create this openness to allow the user to participate in the agency and have it be a part of the composition? So maybe you could talk a bit about how that is structured in this piece.
[00:08:59.087] Steve Mangiat: Well, just to point out the audio-reactive component of the visual, that's really important, because you're selling an illusion that an object actually exists in space, and when the visual matches up with the sound, it becomes much more emotional and responsive. Like the opening, there's this electricity, and when you put out your hand, it warps the visual, but also the audio, and it sounds like you would expect, it sounds like electricity, and people really react to that. In terms of composing, It was a really hard problem, to be honest. It took a lot of discussion. We knew we didn't want to make a music video, something that just played and you sit on the couch and it's very passive. But also we didn't want to make an instrument, because then there's a danger of not really sounding like Sigur Rós or authentic. So in the end it became kind of like a world designed by Sigur Rós that you can step in and you can explore and choose to interact but also there's things that are outside of your control just like in the environment and It gives it that feeling that it's like a world designed by them
[00:09:58.037] Mike Tucker: Yeah, there's a lot of mass going on behind the scenes to make all the sounds work well together. We have over 1,200 sounds from the bands. Many of those are new. And we literally went through it as a database and labeled what key they're in and what sort of tempos they could go well together with. So we have this pretty amazing underlying structure where we can basically, we call them recipes, where you can toss in all these different elements and essentially compose a new piece of music and make it completely interactive on the fly.
[00:10:26.998] Steve Mangiat: Right, and also it's non-linear and the amount of time you spend it can vary. You can just be in there for two minutes versus 20 minutes. And the band said, like, I remember Jonsi saying, I do not ever want to hear a loop because then it sounds bad. So the advantage of having all these sounds, we put them into different banks that then randomly weave together. So you can be in there for 20 minutes and not ever hear a loop.
[00:10:48.918] Rebecca Barkin: I mean, it was an interesting challenge because it was a really abstract brief. And we wanted people to be able to play with music and do it believably. And I think that sort of concept of synesthesia that we kind of used to say, like, have we gotten the interaction right? Like, that was so critical. And we spent so much time trying to figure out different ways to cue a sense of coexistence. And so there's some subtle moments, like with the bulbs in the garden scene where they know where you are and when you kind of approach they just sort of gently extend upward and then turn to look at you as kind of a cue to come and play with them and those were some of the ones that we had to push pretty hard to get vocals in there which was another challenge too because you wanted to sound like them and Jonsi's voice is so signature but again the band wanted to make sure that it wasn't like a traditional expectation of what their music should sound like. So that balance of how do you communicate their sound without lyrics and without the same sort of bass lines and that type of thing was a really hard challenge.
[00:11:53.321] Kent Bye: Well, the more I learn about what you were able to accomplish, the more impressed I am in terms of what it took to architect and put this together. Because it reminds me of something like Facade, where it's an interactive narrative where you go through a run that may be taking 15 or 20 minutes, but you may only go through 5% or 10% of all of the potential interactions. And it sounds similar that there's a whole database of sounds and chord progressions that maybe are reactive to your agency. The chord progression thing is interesting to me because as I study about music theory, that's something that's not insignificant. So it sounds like you have different recipes for chords that would at least sound good together rather than just do a mashup and random. But that it also sounds like any one person going through this would have a completely potentially different experience of that. So yeah, I'm just curious to hear how you were able to take something like as abstract as music theory in this way and try to have this sense that it is still in the alignment or brand of Sugar Roast, but have this kind of remix component to it.
[00:12:56.547] Steve Mangiat: Yeah, I think, well, using their sounds is really important. So at one point we tried to make a in-engine synth and it just didn't really feel like their sounds because they spent a lot of time. I think the band itself is, there's not really intricate chord progressions in their songs. Are you familiar with Yeah, so it is about the sound and the texture, I would say. And so if you use their sounds, if there is a tonal element and you can change the pitch so that does match maybe a sound from a different recording session or a different song, with their music, it did kind of work. But I think it depends on the band. There might be a band that uses chords in a much more intricate way where it would be more difficult to weave together, I think, from different sessions.
[00:13:45.333] Rebecca Barkin: helped with that too, right, from their perspective in terms of establishing parameters. I remember you guys always talking about the box of like, can we put these sounds together and what type of, how far can someone push that? Because we didn't want it to become uncomfortable at any point.
[00:13:59.442] Steve Mangiat: Yeah. And Paul also helped weaving together lots of field recordings, which really helped add to that texture. And it makes you feel like you're in a space that they would be familiar with.
[00:14:11.621] Rebecca Barkin: how many different types of sound, there's like glaciers.
[00:14:16.327] Steve Mangiat: There's sounds from Iceland which really make you feel like this is a Sigur Rós experience.
[00:14:21.689] Kent Bye: Well, maybe you could expand a bit on the visuals as well, because I know that some of the generative art pieces that you've done before that I had a chance to see during the Mobile Game Jam back in 2015, but there's this abstraction of the visuals that are also matching the music, or maybe some sort of symbolic representation of that. So maybe you could talk about that creative process of giving a space and geometry to this ambient experience.
[00:14:46.684] Mike Tucker: Yeah, so I think one challenge with making a really sort of believable but also interactive element is that it needs to be dynamic, right? We're a mixed reality medium where the design of the space needs to adapt wherever you're playing it, whether you're in a small bedroom or a large living room. these different systems need to work in all these cases. And the same goes to the way they reflect the music that you're hearing. So in a lot of the elements we designed, they're all essentially generative. So we're actually dynamically generating meshes and having them react to things like the music, but also your touch. So we have this pretty elaborate grass system. So as you run your hand through the grass, it'll sort of bend very naturally and dynamically. And pretty much over a half of the elements within this project are using a graphics technique called compute shaders. And that allows us to really push the GPU that's on the Magic Leap system to these untapped resources. And we're actually going to be doing a talk about that. So if you're interested in hearing more about the graphics, listen to that tech talk that we're going to be doing.
[00:15:53.617] Steve Mangiat: And also the Compute Shaders are driving particles a lot too, which makes it feel more dynamic, because you can get more complex motion as your hand passes through. There's like fluid sims and things like that, which I think is really important to make it feel alive.
[00:16:08.959] Kent Bye: And Rebecca, I'm just curious to hear from your perspective of some of the either design choices or aspects that you were involved with in the project as this project was coming about, what you learned about the augmented reality medium and other insights or trade-offs that you're making.
[00:16:22.102] Rebecca Barkin: Yeah, one of the most interesting things for me was, first of all, I'm a huge fan of the band. I mean, I actually walked down the aisle to their music. So I came in like, if anyone should be able to connect with this, it's me. Like, I should be able to get this. And I don't come from a traditional gaming background and actually Most of this team doesn't, and we take our design inspirations from a lot of different places. Music is one that's really in our hearts, but I would say that the most challenging thing was when we were trying to nail down like a story or some sort of scope for the project. Because it's nonlinear and it's not like reward based, you know, we had to find a way to do that. And we had to find a way to move people through the experience. And so one of the things we did early on was like, we're kind of racking our brains about how do we get to a place where we understand what we're even trying to prototype and make. And we started with just an emotional journey. And I actually found that to be a really interesting way to approach this medium on the whole, because it's really hard if you just say what you want to make. If you fall in love with a concept before you start prototyping it, you're bound to be disappointed to some degree with an early medium. I think we found our own sort of production process and that was to inspire. We had emotional targets. And so then the review process became a lot easier, right? Because when you go in, it's like, this is supposed to make me feel a little aggravated. Does it make me feel a little aggravated? Like the red world? Do I feel like I can compose? There were times where we went too far in one direction where you could compose too much. And there were times where we went too far in the other direction where it was like, You know, what are we doing in here? So I think that it's been humbling for me to understand how critical the prototyping process is. And these guys are champions at it. So it's been really good for me to be able to say like, OK, you have to surrender a little bit of control to the process. But if we can just establish the emotional target we want to hit, then maybe we found a different way to approach making things in this medium.
[00:18:24.847] Steve Mangiat: Yeah, prototyping and get on the device as quickly as possible is the best way to figure out what's going to work and what's not. It could look nice on paper, but it's going to feel totally different on the device. And we are lucky to have been prototyping on Magic Leap for years now, since the early days, before the PUQ existed.
[00:18:44.483] Kent Bye: Well, I just published a couple of interviews with Yelena Rochesky talking about the hierarchy of being, and she said something very similar in the sense of how you're starting from a human-centered design perspective, starting with the emotions that you're trying to evoke, and very similar to what Robin Honecke does as well in her game design, was trying to start with the experience that you want to give, and then you work backwards. And so maybe you could talk about how you were able to do that translation of starting with an emotion and then what type of interactions or behaviors or context and environment that would do this kind of magical translation through the human experience of taking all the sensory input and to try to evoke those emotions and what you learn to that process.
[00:19:23.882] Steve Mangiat: I think we mentioned that discovery was a big part of it. So the idea that you can see a visual, maybe on the other side of the room, you approach it, reach out your hand and see what happens. And that's a reward. So there is some rewards in there.
[00:19:38.780] Rebecca Barkin: There was a time when we were talking with Paul and Sarah, we were like, how does the band make music? Maybe we can use that as part of what informs our process. What is their process like? It was actually very similar to just prototyping. They sit down, they have no set, we're going to hit this target. they play until something strikes a chord and then they try to build on it. And so we knew they were a good partner in that sense right off the bat, but some of the things like sticky lines is a great example. I don't know if you've done the experience, so you might remember where you can like pull up that it's like these lines that what it's your synthesizer to, right? You can like pull up with it, right? And originally we had a hard time with that one. There were times where it was kind of on the floor. There were times where it wasn't as effective, but we knew we wanted it to be, it was a place where we could get a little edge out of the visual and the sound. And that was something we were missing for a little while. So that moment where you could stretch it and really pull it up and it felt like it's stuck to your hand and Was a really powerful moment.
[00:20:40.673] Steve Mangiat: So we looked for things like that Yeah, that moment too is using owns his guitar And so we can change the volume depending on the speed of your hand too because he plays guitar with a bow So we want to let the user feel like you're Yonzi in that moment, which I think was a good starting point, too Is interesting to Mike if you want to comment on that because we felt pretty strongly there shouldn't be any onboarding
[00:21:05.241] Mike Tucker: Yeah, absolutely. We have an interesting opportunity with Magic Leap to sort of break down the barriers of mediums that we've seen in the past. So, typically music has been sort of kept in its own corner and video games is in another area. And here we had an opportunity to really try and blend those things together and we wanted to avoid some of the common pitfalls of video game design and that it can really sort of eliminate a lot of the possible audience. And here we wanted to create an experience that you could put your grandma in and have her, you know, enjoy it.
[00:21:39.552] Rebecca Barkin: definitely the challenge of intuitiveness. And so that's why that light seed moment, I'm using like our jargon, which I know other people don't call it that, but that light seed moment at the beginning was a great cue for us because it was like, can we get someone to reach out and touch that thing instinctively? Like without telling them like, Put it away put it like a model of the hand there like reach out and touch this like can we get them to just do it and so when we saw people demo it and We saw them just like routinely go like this after like a few minutes and reach out in front of them Like we knew okay that that's the direction we want to go in but then it makes it hard like you have to give cues and Some sense of sentient co-presence, you know
[00:22:20.220] Steve Mangiat: Yeah, there still are some subtle moments in Tenandi that I think it might take a few playthroughs to really at least notice everything that we put in there. And I think that's a good thing because, you know, there's a reason to come back to it.
[00:22:34.470] Kent Bye: Great. And finally, what do you each think is the ultimate potential of augmented reality and what it might be able to enable?
[00:22:43.376] Steve Mangiat: That's a big question.
[00:22:45.748] Rebecca Barkin: I think we, I mean, I think it's definitely just in general where things are going for sure. And we feel really strongly that we don't exactly know yet what it is that's going to really move people. That's what we're trying to figure out. And so you're kind of asking the world for a little bit of patience and to say, like, go with us on this ride. Like, I know Tenendi might not be what you expected, but we were trying to learn something about what moves people. And I think ultimately one of its strengths for sure is that you're not isolated anymore. You don't have to close out the outside world. You can really still see and experience the world and layer onto it something that makes it richer and enhances the human experience versus sort of taking away from it or isolating it.
[00:23:32.118] Mike Tucker: Yeah, this is really the first time that we've seen the computing world bridge over to the physical world in a true sense of that word. And as all these different sensors come online, you're able to build more interesting content that helps bridge that gap. And the limits are really endless with the kind of applications. So I'm sure we'll see a lot of surprises in the next couple of years.
[00:23:53.552] Steve Mangiat: Yeah, we want to see more weird art on Magic Leap. Like Rebecca was saying, we don't really know, and it's just going to be a learning process. And I think for music, you know, possibilities are endless. Like with all these sensors on the device, eventually the music can be aware of what you're doing, your emotions even. And just the idea that music can be aware of you is really interesting. We're only scratching the surface.
[00:24:15.172] Rebecca Barkin: two things, right? Like, there's a practical appeal, which is really critical. And we feel like we'll get there. As industries invest and people invest, we'll get there to the practical appeal. But you have to also sort of prove out that people can have an emotional connection to the things that they see there. So those two paths have to be pursued to get it where it needs to be.
[00:24:33.197] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say? No? OK. Great. Awesome. Well, thank you so much for joining me on the podcast. So thank you.
[00:24:39.739] Rebecca Barkin: Yeah, yeah. Thank you so much. It's been great. We all follow. So we were happy, more than happy to do it.
[00:24:44.596] Mike Tucker: Thanks, Ken.
[00:24:45.417] Kent Bye: Yeah, thanks a lot. So that was Rebecca Barkin. She's a senior director of Magic Leap Studios, as well as a senior director of strategy and partners at Magic Leap, as well as the creative and tech leads, Mike Tucker and Steve Mangent. So I have a number of different takeaways about this interview is that, first of all, If you have a chance to see Tenandi, I highly recommend it because it's really a beautiful experience. And after learning about all that they did to create this dynamic and interactive music experience where they're trying to put at the same level the music and the visuals and the spatial audio and the interaction, I really think that it's the interactions that really sells this experience because, you know, I've seen a lot of virtual reality experiences, but something that's very unique about the augmented reality headset is to be able to actually get your hands within the experience when you see your hands it's like the perfect embodiment because it is your body you don't need to fool yourself by doing any sort of mediation through avatars or trying to track elbows and virtual reality it's just hard to do like accurate tracking of hands and when you actually have your fingers in the experience and you're able to grasp and do the normal affordances of how you would use your hands to be able to interact with an experience You're able to find that you have this one-to-one interaction and you're able to manipulate this digital world. And it's probably one of the more profound turning points when I was like, aha, this is a key aspect of what makes the augmented reality such a compelling medium is to actually have your hands within the experience and being able to feel like you're manipulating things. Because, you know, one thing that the HoloLens does is that it has this like very simple gesture. You're just like pinching and that you don't get the sense of being able to actually grab stuff and it being able to respond to your hands in any sort of meaningful way. And just to be able to. take your hands out and to brush things and side to side they did a lot of really interesting integrations for being able to both detect your hands and to create an experience that gave you that satisfaction of having that agency and there's one thing that I learned from talking to leap motion which is the the depth sensor that you put on top of virtual reality headset. And they've been working a lot on these hand interactions and that when you have that hand interactions and you feel like your will and your agencies having the environment respond to you, it just really sells the experience. And I think that that is going to be one of the most compelling aspects of this medium. When I'm talking to other creators and developers at Magic Leap, LeapCon, someone like Insomniac, that was the thing that they decided to design SeedLink purely around that process of you using your hands within the experience. having a controller gives a level of abstraction. And one of the things that I found is that there's a lot of little different buttons that are on the Magic Leap controller. There's the back button, there's a bumper, there's a trigger, and there's like a touchpad. And there's no sort of universal way to say what is what and what you can do with it. Basically, every new experience you go into, you have to decide what the interaction design pattern they're using with those controllers and those buttons. And it sounds like that when they did Tenanti, they were like, you know what, let's just like abandon all of those abstractions and just go purely with your hands and be able to interact with the world. And I think as things go forward and they start to match the level of computer vision to be able to do hand manipulations to something as matching something like Leap Motion, I really think that from what I've seen that's out there, Leap Motion is really kind of the best of class to be able to really sell those hand interactions. I mean, The demos that I saw at GDC were just amazing, and I really look forward to where they're taking their North Star. But they're starting simple at Magic Leap, and one of the things that the developers told me is that over time, they're going to be sending more and more over-the-air updates in terms of just updating the software, and that some of the developers are like, you know, once we get some of these new updates, it's going to feel like the Magic Leap 2, even though it's still the first developer kit, there's still a lot of things they can update and upgrade over time, and that there's going to be new avatars and other systems as well. I was able to see a lot of different things at the Magic Leap conference. And again, I would say that the fundamental tension that I felt while being there was this tension between the idealism of what's possible to inspire the weird, crazy art and storytellers to jump on and start experimenting with the platform versus the pragmatism of like, is this going to be viable? Now, certainly it's going to be viable for certain contexts, especially in enterprise applications, but in some ways they didn't necessarily design the experience to have enterprise applications. And the reason why I say that is that they made the design choice to make something that was very custom and bespoke and really centered on to like one person experience. there's a special fitting that you have to get there's a custom nose piece and a custom thing that goes on your head to figure out what your IPD is and based upon your IPD there's like a large and a small size and then on top of that they basically designed it so it has to get super close to your eyes and there's no way for people who have glasses to really use it it just really doesn't work at all with glasses I mean I have glasses And I try to use it and I have to like hold it. And sometimes it like loses the eye tracking and I get the entire experience starts jittering. It's just a much worse experience. And so it's really designed in an ableist way where it's like, if you have glasses, we're sorry, but it's not going to work. And so their solution is to have you buy like extra glasses inserts to put into the magic leap. But the problem with that I see is that that's great. If you're the only person that's going to be using it and that no one ever is going to be using it, that has a different eye prescriptions than you, then. If they are, then you're going to have to basically buy all the other prescription inserts to be able to customize that for the person. So there's a number of those types of decisions that just make it feel like, well, it's really kind of designed for an individual to buy and to use at home as a consumer, but yet When it comes to like the enterprise use cases or location based entertainment There's a lot of like emotional labor and friction that they're gonna have to go through in order to either Abandon this customized bespoke nose pieces and headrest which to me I would have loved to try all of them just to see like what the real difference was and is it really worth going through all the hassle and trouble of doing those custom fittings before you can even see an experience and And then on top of that, if they could have designed it in a way that I could have worn it with my glasses on without having to worry about some sort of prescription that is going to be slightly different, it's going to make me see the world in a way that's not necessarily as comfortable as my glasses. And it's just kind of a pain for me as a journalist who I'm doing demos all the time. There's different trade-offs that they went with that I think I'm curious to see how that plays out. Maybe as an individual, it's a better experience overall, but is that worth the hassle and the friction of being able to show the device to people for the first time? And if you're planning on using this for location-based entertainment, then there's just going to be a lot of friction for actually getting people into the headsets. So I ended up doing about 12 hours of interviews with over 20 people. I saw nine or 10 demos while I was there. And so I'm going to be exploring all these different issues of Magic Leap and my big takeaways. But what I would say is that there's some developers that I really trust that are genuinely really excited about this as a platform and as a medium. You know, for me, there's a lot of things I have questions around different design decisions they made around the actual product. But also there's this like deeper experience of this is a company that has had some of the most intense secrecy of any company we've seen in quite a while. And they still have quite a bit of that culture of secrecy that's embedded into their DNA. They are not a public company. I think the difference that is different between what they're doing versus what any other company is doing that's out there is that most of the other companies that are releasing these types of headsets are already like publicly traded on the stock market and like magically business startup. And so they have this like close minded, like they've got to protect their intellectual property so that nobody swoops in and takes it from them. But At the same time, this culture of secrecy has prevented them from being able to do the things that they could have been doing, in my opinion, that could have been developing a developer ecosystem. Because they've created all this hype and marketing around everything they've been doing, there's been a bit of this skepticism that what they're doing is just complete vaporware. I don't think it's vaporware. I think it's a good product once you have like all the right key ingredients for it. And, um, it's, it's actually doing a lot of things that you can't do with any other platform. And that like, if you believe that there's enough trends within the industry that is like leading us towards this spatial computing revolution, you can look at what Apple is doing with like their. ARKit and their iPhone X and like they're on the roadmap to eventually release their own classes. Google has both the ARCore, they have like all of this Google Daydream and VR and investing in all these different virtual reality experiences. I mean like Google is moving towards spatial computing through the first stops of ambient computing and then on into the voice interactions. I just see that they're going that way. Facebook obviously is investing very heavily in a lot of the virtual reality spaces and adding more 3D features just through Facebook within the last couple of weeks. And also they're doing all sorts of stuff with Instagram and facial filters. And Microsoft has a HoloLens and the Windows Mixed Reality, Windows Holographic operating system that is going to be running on all of this. So many things that the operating system, the protocol layer from GLTF and the open standards that have been coming out And then Snap is, you know, another publicly traded company that's doing a lot of like augmented reality stuff that is interfacing with the public and being able to do augmentation of your identity. So when you look at like the overall ecosystem of where tech is going, like this is obviously where everything is going to be headed. And to me, it's just a matter of time before this is going to cross the chasm and reach into the major mainstream. I'd say by 2025, I expect both VR and AR to be just mass ubiquitous technologies that are just widely distributed and probably like where cell phones are today based upon where we have been since like 2007. I mean, it's like 11 years since the iPhone came out, which was, you know, there's other smartphones before that, but that was really like a turning point that really just catalyzed the overall ecosystem. So just imagine once the Apple comes out with their augmented reality glasses or whatever it is that they have along with their watch and their, AirPods and every company that's out there that's looking at the future is seeing that embodiment and spatial computing is like where everything is going. So I think Magic Leap is playing a key role here and they are, if anything, they are just inspiring and giving a platform for these artists to be able to create and innovate. And a lot of the artists and creators that I know from Sundance and places that they're on the festival circuit, they're really the ones that have been pushing the edge of what's possible. All of them that I talked to at the Magic Leap Con were there because they are super excited about the augmented reality medium. they see that the augmented reality is actually a little bit more robust ecosystem when it comes to phone-based AR and ARKit and ARCore. There's going to be billions of phones that are out there soon, if not already, that have the capability to be able to take some of these augmented reality experiences that you're creating, maybe for the magic leap, but there's going to be a graceful degradation to be able to have a similar windowed effect through phone-based AR to be able to have a similar experience. And so when it comes to brands and marketing and advertising to be able to do these different campaigns, you're going to want to be able to do a spatial capture of a celebrity and be able to put them into your room and take a selfie of them with yourself and that's just going to be a thing that is already happening and it's going to get more and more people excited about the technology what the potentials are and there's just like this revolution of immersion and experiential technology so that we want to be immersed into these experiences and like the Magic Leap is certainly playing into that and they're starting and really empowering the artists and the creators and the makers. And then the other side is that there's also pragmatic applications for this as well in terms of like architecture, engineering, design, certainly retail with Wayfair. And I guess some of the question is like, is it practical enough for them to be able to expect their audience to buy stuff? Well, the audience isn't going to buy stuff unless there's like compelling, emotionally connecting content that's out there. And I think that is part of Magic Leap's strategy is to let the artists and indie developers and the creators and the makers really push the medium forward. So while I was in LA, I had a chance to go to a couple other events, two art shows, I went to a number of location-based entertainment locations, and I also had a chance to drop by IndieCade just to see what was happening there. IndieCade is a little bit of a leading indicator for what the future of play looks like, and if you look at a lot of the trends at IndieCade, there's a lot of Spatial games, a lot of full embodiment, a lot of like role-playing action. There's games that are using these multimodal inputs of using your voice and sound as a game with one hand clapping. And there's more and more of these principles from immersive theater and interactive storytelling and role-playing that are starting to come into gaming as well. And so I just see that as things go forward, this is just where the interaction and play is going. I had a chance to talk to Sam Roberts about some of the trends he's seeing, and he's reflecting that as well. And there's a couple last things I want to say about my experience at Magic Leap. One is that the Mica demo with the AI virtual character was absolutely mind-blowing in terms of just how you're able to have this intimacy within the character that is going to be able to respond to your facial interactions and gestures with their eye gaze and it's a it's not photorealistic it's something that's a little stylized and you know it actually worked really quite well because the interaction again going back to the interaction if there's a virtual character that's interacting with you you don't need to have it be photorealistic it needs to be at least good enough so that as you're interacting with it it's responding to you and so when there's things that respond to you within an augmented reality environment that just takes the level of presence to the next level because you have the social presence and you have your active presence where you're actively moving around and that if it responds to you whatever it is whether it's a virtual avatar or virtual human or whatever it is in the environment if it responds to you then it just makes you feel like it's real And so the Micah demo was like this little immersive theater piece. And I'm really excited to see where they take this, this AI driven character that just was super compelling. And it was the best virtual human interaction that I've ever had within any virtual augmented reality experience. And so look forward to see where they take Micah. And the last thing that I wanted to say is just that There was some friction there that I had with Magic Leap where at so many different times they tried to stop me from doing what I do, which is talk to developers. They didn't want me there early. They only wanted me there on the first day, which by the way, they said it was on October 9th and 10th, but like everybody showed up on October 9th and like, okay, what's happening? And they're like, oh, nothing's happening. It's just a demo day for press. And then there's a lot of developers that were there ready to attend their first day of the conference and it actually wasn't starting. So there's actually a lot of developers that were there milling about. And part of my process is to walk around and talk to people. And there was a number of different press relations and everything that just didn't want me there talking to developers, which is the basis of how I do most of my coverage. And so I had to really convince them to get me to even stay. And then in the opening night party, when I was trying to go meet and talk to a lot of people that I know, people that I don't know just get a sense of what's happening. They didn't let me into the party because they had made some decision that they weren't going to allow press, which to me is absolutely just ridiculous. That was the first conference I've been to with over 60 conferences over the last four and a half years that the press was like denied from going into hanging out with the core and the heart of the community. And I was just And since I was outraged, I was like, are you kidding me? Like, this is like my job is to like connect to developers and to see what's happening in the community. And that I just felt cut off and rejected and exiled from being able to actually do that. And so I think that overall there's this culture of secrecy that if I'm sort of embodying this openness and transparency, like Magic Leap has a serious issue with like being open and transparent because it's like, like, what are they afraid of? They've been in this stealth mode for so long, but there's still this like culture of secrecy that is embedded within the fabric of their DNA as a company. And I think that there's this tension between like the pragmatic makers or the engineers who are making this versus the culture of secrecy and proprietary nature. And there's a lot of things that magically be saying that they want to create this decentralized AR cloud? Well, that's not going to happen when they have this extreme culture of secrecy that they're still trying to get out of. To really have a viable ecosystem in a decentralized way requires a level of transparency and openness and really kind of a different financial structure, to be honest, with a foundation. If you look at a lot of blockchain technologies that are successful, they have these nonprofit foundations that are in charge of running it. And like Magic Leap is a startup still. And so I have a hard time believing some of their vision of what they want to create with this, what they call the Magicverse, which is essentially like the AR cloud from anybody else that's not a part of the ecosystem within Magic Leap. They're coming up with a branded name of the Magicverse because they want to inspire people with what it means to be able to overlay different layers of reality. But again, this is like an example of the tension between the marketing and the story of what it is versus like the pragmatic reality and what is the things that you need to do to actually collaborate with a bunch of people who are already building the AR cloud. If you really want it to be decentralized, then how are you going to interface with what else is already actually happening in this space? So Magic Leap is a new company, they're still figuring a lot of stuff out. At the end of the day, I trust what the developers are going to be doing, what the creatives, what the storytellers, what the makers are going to do, and I'm just excited to see what they're able to create from that. I think there's still a lot of skepticism within the general tech press community and I don't necessarily know if there's one single grand narrative about what is happening with Magic Leap and where it's going to go in the future. I think I'm going to be looking to the independent developers. Like I said, I did over 20 interviews with different people over the two days of Magic LeapCon, and I think that is a good sampling from a range of people who were all in, they're going to do everything within Magic Leap and just push what's possible forward. But also, none of them are doing it full time. A lot of people are working on other projects. Some people are looking to maybe enterprise training applications. Some people are waiting and seeing until it's going to be economically viable for them personally. But there's other people that are just the artists and creators and the storytellers and they're super excited to be able to start to innovate on the platform. And I expect that when I go to Sundance and Tribeca and South by Southwest and Venice Film Festival like all these big premier festivals next year are going to be a little bit of a leading indication as to what's going to be happening with the medium as it evolves and unfolds because I think you're going to see a lot of experimentation there. And I hope to see more experimentation on the WebVR, WebAR. You know, unfortunately, the Helio web browser doesn't launch with a lot of like the standardized WebVR integrations. They said they're going to be launching it later this year, but they decided to launch it with something that was kind of like their own custom thing in order for you to get a web experience on there you had to sort of use their thing rather than what is standardized within the web development community that was a little disappointing to me to learn that but hopefully as they get more and more WebXR integrations it'll be easier for you to create whatever you want on your website and to really push it out there. Because if there's anything that Magic Leap as an underdog has the potential to do, it's to empower the indies, empower the artists, and empower the web community to be able to innovate on this platform and to create an alternative to a lot of the walled gardens that are out there. Because I think we're going to start to see a little bit of a stasis in terms of what's happening with innovation. on these virtual reality platforms, because it's kind of getting commercialized to the point where doing things that are pragmatic and going to make money. And the real innovation, I think, is going to be happening by the artists. And I hope to see a lot more VR and AR both on the web, as well as on these new platforms. So that's all that I have for today and yeah I just wanted to thank you for listening to the Voices of VR podcast and I am an independent journalist in community media and I need your support to help me to continue to do this type of coverage and if you enjoy this type of independent journalism then I do need your support. I don't actually own Magically, I'm not currently making enough on my podcast to even afford one. And so I would love to increase the Patreon just to make it to the point where I'm actually sustainable as an independent journalist, but also to be able to buy this technology to keep up with what's happening with the medium as it unfolds. So you can donate today at patreon.com slash Voices of VR. Thanks for listening.
