John Gaeta first got into virtual reality when he was approached by the Wachowski siblings to be the visual effects supervisor on The Matrix. Now he’s Executive Creative Director of Industrial Light & Magic’s Experience LAB (ILMxLAB) where he’s designing and enabling platforms that will lead to the Matrix — “minus the machine dominance” of course. ILMxLAB has a decade of story to explore within the Star Wars Universe using the latest virtual and augmented reality technologies. I had a chance to catch up with John at Sundance where he talked about collaborating with a dream team of storytellers, technologists, and theme park designers to create immersive entertainment experiences. We talk about the evolution of passive and interactive storytelling, crafting tribal social VR experiences, and the importance of including humanity and emotion within a medium that can otherwise feel isolating.
LISTEN TO THE VOICES OF VR PODCAST
Here’s a summary of the work that ILMxLAB has been doing, and it features a lot of footage of their CAVE-like Holo-Cinema system that can be used to preview virtual environments and create interactive experiences:
ILMxLAB’s Rob Bredow was at Oculus Connect 2 where he presented a graphic that a showed a spectrum of immersive experiences across the two axes of Pre-Rendered vs. Real-Time and Passive vs. Interactive:
The range of storytelling opportunities in #VR by @ilmxlab @rbredow #oc2 pic.twitter.com/zsLjr1xhPE
— tipatat (@tipatat) September 23, 2015
This shows that ILMxLAB’s perspective that there will be a range of interactivity within narrative experiences that are perspective, then a hybrid of plates and CG or backgrounds will be added, and then move to multi-point navigation within a CG environment, and finally there will be navigable movies with parallel stories rendered in CG. On the real-time end of the spectrum, it starts with real-time shorts, and then dynamic characters will be added, progressing to touch navigation and casual interactions for a single user, and then finally to 3D navigation with complex scenes that are modifiable by multiple users.
In the end, John Gaeta is the really interested in adding social elements to ILMxLAB experiences with real-time human interactions, and so he’s been quietly visiting many social VR platforms and has given a talk in ConVRge complete with a lightsaber dance party. ILMxLAB has been collaborating with Disney Imagineers in order to help create experiences that are meant to be done with a small tribe of friends and family. While John is impressed with the technical wizardry that can be demonstrated in VR experiences, he believes that interactions with other humans and stories with human emotion are going to be key for the virtual reality medium and immersive storytelling to really take off. In the meantime, ILMxLAB will be continuing to experiment with augmented and virtual reality and is interested in collaborating with content producers who have unique ideas for how to best use these new immersive platforms.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is John Gaeta. I am the creative director of ILMxLAB, which is an immersive entertainment division that Lucasfilm has spun up recently in the last year. We are looking at a whole variety of platforms where we can change the way stories are being told or at least experienced. We are a hybrid of folks that are at Lucasfilms. For example, we count amongst us the Lucasfilm Story Group, the people who are developing all of the scripts and all of the content that you're seeing in the features. It's ILM, Industrial Light and Magic, that is, and it's Skywalker Sound. And more specifically, inside of those groups, they have various types of research and development pods. So we combine different groups of talent, brilliant engineer talent, and we focus on proofs of concept, prototypes, and now we are endeavoring to pilot some future content. So myself, how did I get into VR? I met two Polish kids who were interested in the subject. from a literary point of view, right? So we tried to form concepts that appeared in the movie, The Matrix, in the following two films. So from that point to today, I've been very lucky to not only have been exposed to it intellectually, but folks who were like-minded and really wanted, you know, people have been after this for decades, as we all know, so I've been lucky enough to meet a lot of folks through time that were experimenting So there's a lot of converging paths. So here I am back again. I knew that Star Wars, for example, as Lucasfilm looked to essentially do a decade more of story, you know, starting with that as the foundational piece of information that they wanted to tell stories across another. more than several films, you know, so that's a very, very rare situation where there's a thread line and you can actually create maps of where things are going, where things have come from, you know, all sorts of things when you're talking about that much development. So that's super fascinating. And then if you want to take these platforms seriously, They're not really just there for gimmicks, you know. Eventually we're going to want to have some sort of takeaway, some reward, something deeper to take from it. And so we are really thinking hard about how our stories are expanded, how they intersect. the cinema, how they change potentially your opinion or your view of what you thought you were seeing, and allowing the stories that you might see in the cinema as context going in, which is actually quite important because it actually empowers anybody to feel like they have some knowledge already walking into what must seem like a rather foreign experience that VR is. As everyday folks start to do it for the first time, it'll be strange for them, right? And so we think giving them something that has, you know, context, that's familiar, that's something that is desirable, just on its own, that universe, will get them started and then lead them through what might begin as something rather sublime. deeper into something that could be more sophisticated for those who feel like going that far. Some folks may not. Some folks may be like, you know, the most that they want is just to step inside the story that they saw. And I think that's a mass medium right there. You know, I think that the most people will do that just to start. But then they'll be the more ambitious, you know, and certainly anyone who plays games and all that feels a little bit more Interested in affecting the world or having the world react back to them. They'll go in further. So anyway, all that stuff is out there Yeah, and that's what we're doing. And in particular, what you're doing right now is you're standing on the third floor of the New Frontiers installation, the 10th anniversary over at Sundance. And we are presenting an installation called Holo Cinema, H-O-L-O, which is not VR. It is a form of immersive projection. It shows you a moment in cinema. But rather than wearing VR glasses, you're actually wearing track stereo glasses. And it's all real-time rendered, which means that you can walk around the content. It's fully 4D, like VR, except you're in it and you can see yourself and your friends and all that. So, it's about as close an approximation of what augmented reality enabled glasses to come might give us. And so, we are... endeavoring to work out what we think would be compelling in both form and or content, even this early. You know, it's like this could be a couple few years before we really start getting to access it, you know, the public and all. But we're thinking about it. And again, this all all stems back from the fact that we're allowed to at Lucasfilm think about stories in literally five to 10 year chunks, which is which is pretty rare. Yeah, at the Oculus Connect 2, you're presenting there and showing kind of a matrix of interactivity versus immersion. And maybe you could talk a little bit about the different types of experiences that you foresee creating that either have completely passive and interactive. And it seemed to be a fleshed out matrix in the way that you guys think about it. Right, so I do think about it in terms, it's like a layer cake. You know, on top of course is the story itself, right? So, foundationally we want to know why we're someplace and what might be happening at that time. Or at the very least understand the destination. So in a movie that would be the setting, you know, for the story scenes and all that. But it is a destination and in truth if it really existed it would be a persistent place. So the first layer is the story. The second layer that we're doing is cinema. We believe cinema is solid. We believe in master storytelling. We don't think that's going to go anywhere. We think that people are driven and compelled emotionally in ways that other platforms cannot yet do with performance, you know, and just humans in the content. So the cinema piece is very important. So then from there though, and what we were sort of suggesting with what we were showing at Oculus Connect, was that you could then step through the film as if there was a portal. You can step into it and the first thing that you could do is you could walk around the beautiful sculpture that that scene was, right? You could orbit it, you could take any perspective on the scene that you just saw. Upon further investigation, you found that you could navigate around the setting, the destination. You could actually discover other scenes that were going on. So any place where characters may have converged onto the editorial timeline. So this character appeared halfway through the cut. Well, they came from somewhere and there was something going on with them in parallel to the scene we were looking at somewhere else. And so we were really looking into that a lot, right? Like not endless multi-tracks, but just the right amount, right? Where we can sort of build out very important sort of converging pieces. And so once you can explore and navigate, be it in VR, be it in some other medium you can navigate. We were in some of our experiments, we were doing things with mobile devices enabled with sensors that were powered by modified game engines being broadcast over the cloud, so it wasn't even on-device. But at any rate, so there you are, you're discovering other scenes and then you realize you can walk around the place and the place is actually alive and dynamic and you can see other things, you can go into rooms, suddenly you're doing the room thing. You could literally choose to tap on something that you may have seen in the background. So one example we have was that like, you know, in a scene in which the droids are running away from pursuers, we see X-Wing's power over the top of them. in this later navigable form where you're able to literally tap upon the X-wings, go up there, follow them, go inside, actually operate them, shoot them. So there is this thread line that we've created and it's all casual interactivity, it's experiential. As opposed to, for us, it's not competitive at this point, it's experiential. So we've started experimenting on a very, very casual, interactive level. A thread line from passive cinema, passive storytelling, portaling, navigating, like bullet time. and then exploring, and then playing, right? And of course, if you're in there with friends, then you're talking about multi-user social experiences. So the destination can serve all those things, including the base from which you crafted your original cinema from. So that's what we were showing at Oculus. I'm not sure if folks kind of gathered it, you know, entirely. I mean, think about, like, yes, this isn't something you're going to be able to do with a whole movie, but you could definitely envision special portals, you know, a certain number, and then, yeah, with what appears to be short but very, very deep experience that could be created from that. Yeah, it makes me think of Sleep No More, which is like immersive theater where you have like 21 coinciding parallel stories represented by the actors running from room to room and you follow them. And it seems like to me like immersive theater, potentially like theme park attractions, theater in general, has a lot to teach virtual reality. So I'm wondering if what other kind of things are you looking for for inspiration when you're trying to figure out like how to tell stories within VR? That is a good question. I mean, yeah, you're right about theme parks in a particular sense that, when done very well, and we work with the Imagineers, and so we actually, you know, we feel like students, right? And learning, you know, we're talking decades and decades. of very, very skillful guiding of people through, you know, a series of pre-experiences that are getting them in the right headspace, you know, tonally. And then when the experience occurs, they're very cognizant of the fact that you're often there with others, right? You're there with a clan-sized group, whether that's friends or family. It's very interesting because it's true. We're often, right, in these groups of... We prefer socially to be in groups of like three, four, five, right? Or families like that, right? So it's really important to understand that it's not... The future isn't... The future isn't us, like, passed out in the corner, isolated, singularly, like that terrific VR film that we saw online recently. But it's not that. That is not the future. The future is figuring out how to allow us to have these experiences in the kind of clan clusters that we prefer, right? To be social. Because at that point, if you're able to be with people you want to be with, then it becomes about yourselves. And you can be doing things as long as you care to. As long as you care to be with your friends and family. That's a learning that I don't know if people sort of gather that off the top, that they think, oh, theme park, they're making rides. It's about ride flow. It's about people flow. But it's not really all about that. So there's that. I can't even remember what your question was. Just looking to other mediums, it seems like, obviously, film and TV have some to teach VR. But actually, it seems to me like Sleep No More, an immersive theater experience like that, is something that's way closer to designing a VR experience than film or 2D mediums. Well, look, if you're going to bring up sleep no more, I mean, it's true that there is, you know, people are enjoying that, people are liking that, but again, the reason why it's actually a powerful experience for folks is because, you know, it's stories being told by humans to humans who are left there to see the emotions direct, you know, and all sorts of things. not actually separated by 100 feet and a stage, right? So that is interesting. And I do think that VR does need to understand that it's going to not amount to anything until it figures out how to ingest emotional attributes. It's fun to be in, like, fantasy worlds with droids and or, you know, crazy creatures and, like, AI thingies walking around. That's totally fun. I'm into that too. But the thing that transfixes humans are more humans, right? They'll never go out of business. People will always prefer, in the end, to follow the career of a particular actor because they are touched by them or, you know, there's something about that, right? So that's pretty much a fundamental need. The folks need to sort of pivot from just turning a game engine into a camera of sorts, right? They need to pivot into figuring out how to train that camera volumetrically. on something more human. There's other ways to get human and of course you can use capture and all that but really things that are expressing, understanding that. I mean there are ways you can sort of like channel expression. I'm sure some of the folks that listen to your show would know what I mean by that but really We do have to figure that out and quite frankly, again, what is really the potential growth of a platform if it feels isolating? So if it doesn't get social soon, it's going to go slower than folks want it to. There's a lot of artistry, there's a lot of brilliant science and minds being applied to this because it's just a cool frontier. You know, we're all into it, right? But folks need to be really cognizant of the fact that if it doesn't start pivoting towards an emotional place. So I say all that. So the thing that I also am not really that enamored with, I mean, I'm, you know, my friends pioneered volumetric cinematography and moving through image-based constructs, not being stuck at a fixed point. You know, like 360 video at this point to me is like fine for the moment but extremely suffocating. So people are going to need to start figuring out how to as well ingest reality or they need to start exploring mixed reality much more seriously and consider that potential. I gotta go, unfortunately, but it was nice to talk to you.
[00:15:04.435] John Gaeta: Okay, great.
[00:15:04.875] Kent Bye: Well, thank you. Cheerio.
[00:15:06.875] John Gaeta: And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.
