I interviewed What If…? – An Immersive Story director Dave Bushore at Venice Immersive 2024. See more context in the rough transcript below. What If…? also picked up an Emmy for Outstanding Innovation In Emerging Media Programming as awarded by a juried portion of the Emerging Media Programming Peer Group of the Academy of Television Arts & Sciences.
Also, Bushore announced that he was leaving Marvel after returning home from Venice Immersive 2024. The Hollywood Reporter reported on how Bushore was really inspired by Venice Immersive and will be likely focusing on the frontiers of immersive storytelling in his next venture.
“Being invited to Venice was mind-blowing,” Bushore tells The Hollywood Reporter. “Being there and seeing the embracing of this new type of content, and the people coming through doing the experience, filmmakers and ambassadors, that was a really cool thing.”
And it’s exactly what this self-described “story nerd” wants to do more of.
Bushore is not joining a new company, but instead is bounding into parts unknown. He seeks to take his experience in the intersection of technology and new ways audiences participate in their favorite brands. His goal is to create exciting, emotionally poignant, groundbreaking experiences that continue to push the boundaries of media and entertainment.
My conversation with Bushore happened in the middle of his journey at Venice Immersive, and you can tell that his excitement about the potentials were palpable.
Here’s their artist’s statement:
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. The podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So I'm continuing on my series of looking at different immersive stories from Venice Immersive 2024 and 2021. Today's episode, we'll be looking at what if an immersive story, which is a part of the best of selection, actually covered this previously back in episode 1391. When it first came out back in May of 2024, I had a chance to talk to one of the producers and technical art creators in that episode. And At Venice Immersive, I had a chance to sit down with the director and executive producer, Dave Beshore, who has a lot more context for the creative inspiration and his own perspectives on, as a self-described story nerd, on the future of immersive storytelling and immersive entertainment. So after Venice Immersive, then What If was awarded with a special jury prize award for Outstanding Innovation in Emerging Media Programming, which is a part of the Emerging Media Programming Peer Group program. So a special Emmy recognition for the innovative work of mixed reality and immersive storytelling. And yeah, very well deserving there. And Dave also announced that he's leaving Marvel and going off into pursue other immersive storytelling ventures. So good. A little bit of a sneak peek for where his mind is at as he was seeing all the different immersive stories there at Venice Immersive. And yeah, very much looking forward to see where he takes it all in the future. But yeah, Yeah, we kind of do a recap of what if, numbers of stories, some of the different processes that they went about to create this story. And yeah, just working with the Apple Vision Pro, Unreal Engine, and the intersection of these big IP, like Marvel, and then how to start to translate some of these stories and to allow you to become a character and to have these kind of mashup of different aspects of the intellectual property. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Dave happened on Saturday, August 31st, 2024. So with that, let's go ahead and dive right in.
[00:02:13.186] Dave Bushore: My name is Dave Boucher. I am the director and executive producer of What If an Immersive Story. I've been working in the immersive media sphere for about six or seven years. I've produced and directed a few projects. The first big project we did was Avengers Damage Control. That was also a partnership with ILM through VFX. Disney in the Void, internally, which was an LBE location-based experience, which I'm sure you're very well aware of. And a couple other experimental projects, one we did for the parks, which was just sort of a fun thing. Eternals-based experience, because I used to work on all the park stuff, it was trying to figure out where those crossovers were. But yeah, I've been kind of playing in this space for a few years now, and this one was the one where it was like, okay, this is the confluence of all of those, and trying to push this forward in terms of what's possible.
[00:02:56.968] Kent Bye: Great. Maybe give a bit more context as to your background and your journey into this space.
[00:03:01.429] Dave Bushore: Yeah. So I've been at Marvel Studios for 15 years. I've done various things there from digital content, marketing, to all the behind the scenes for the films, the home entertainment platform, production and storytelling that was on the Blu-rays and the DVDs, working with the filmmakers and all of that over the years. I did some marketing for a while. I helped build the franchise team inside of Marvel Studios. So kind of all brand, like in terms of like everything that's of, about, next to, related to the films and the big IPs that we're putting out, you know, that kind of brand management style and learning a lot about how those things interact inside the company. I've had various jobs over the year, but always in a creative role, always some sort of a creative oversight or creative production role. I've always been someone inside the company that's always sort of tried to find those fun cracks to play in right um have done specialized content did a tv special one of the first things i did was the this was a quite a while ago but the tv special before avengers one came out or post avengers one i can't remember actually that was quite so long ago but i've done a bunch of different stuff in all all learnings in terms of being there for 15 years great and maybe give a bit more context for how xr started to come onto your radar in your professional life yeah totally it's interesting i think honestly it was something that was just born out of i'm a big kid in general i love movies i've loved movies my entire life i love comic books i love storytelling in general i'm an addict for anything story right i love talking to people i'll talk to people at the coffee shop i'll talk to people on the boat on the way over here you know xyz i just love getting to that and so About 10 years ago, I got access to one of the first Oculus headsets, right? And got into that world and growing up watching shows like Star Trek or Star Wars or, you know, anything sci-fi that was sort of about these things that were possible. The old back and forth of just technology influenced storytelling to storytelling influenced technology. getting to go inside one of the early oculus headsets and when everything was sort of you know really really nascent at the time right seeing that world for the first time going whoa this is actually what i would love to have done when i was a kid all the things they were talking about you know in terms of those world building platforms and then it kind of just went from there i got really excited about it seeing the opportunity to kind of take storytelling in a whole new direction, right? As I think so many people have when they've gotten into those types of products. And really just trying to find a way to do that, right? Pitched a bunch of ideas internally at Marvel, you know, met with executives, always throwing things around. Obviously those opportunities kind of started to come as we did some earlier of the experiences I mentioned before, kind of test cases. But it really was just being a kid in a candy store, just kind of like seeing the opportunity and just trying to do cool stuff.
[00:05:52.975] Kent Bye: And is, what if, is this the first time that you've had a chance to direct a piece or are there other immersive projects or films that you were able to step in as director?
[00:06:01.774] Dave Bushore: Nothing on the film side or anything like that. It was always sort of creatively behind the scenes working with the filmmakers or the marketing teams or so on and so forth. Like I said before, I've produced some special content in conjunction with those teams for things like Comic-Con or like I said, little small TV specials or the Blu-ray special features or things like that. And that was all really fun. In terms of XR content, it was really the stepping stones of... seeing what ILM X Lab at the time was doing and then now ILM Immersive is seeing the community that they were building inside of Disney and kind of having conversations with them and and Vicki Dobbs who's led that team and hearing what they were thinking and obviously the story and history at ILM that I'm very well aware of and a huge fan of and getting to go up there and live like oh my god these are all my favorite film franchises, these are all my favorite characters and then people are in the basement like actually like gluing things together like they do up there and going I want to play in that sandbox. How do I do that? Right. And it was really just simply just kind of asking. Right. I think everybody who's in this world is great because you kind of don't have to ask for permission. Right. You don't really have to ask permission to do anything creative. But the idea of no one's watching, let's go glue some stuff together and see what it is. So then it was just sort of stepping stones. Right. The big opportunity, like I said, was. doing some test cases. We were working with the Disney Studio Technology team, which is an amazing team. And they are really great partners to creatives internally inside of Disney and the company because they are out there trying to find those new things, right? And then showcasing them to the executives and showcasing them to people like me. And I'm going, oh man, When I was working on parks, you know, things, it was great partnership because I'd see something that Disney Studio Technology was showcasing to the executive teams across the company and then being able to go back to the parks team and be able to say, have you guys seen this? We should think about this. And then they have their own R&D team. So inside of Disney, it was really just sort of that great confluence that they have in terms of storytelling, creativity, R&D, and pushing things forward. As I got into those things and just saying, hey, can I play in this sandbox? Doing those test cases, right? Avengers Damage Control, some earlier ones. So I was the director and executive producer on Damage Control. I was the creative director of sorts in terms of from a Marvel IP perspective on the Eternals experience we partnered with. vendors on that, that had amazing teams that we worked with. We worked with the filmmakers on that experience and then some other test cases. But for What If specifically, this is sort of, yeah, my first big, big, big project where I led it all the way through creatively, right? Obviously still working with the teams inside of Marvel. Brad Wunderbaum, who hired me originally 15 years ago, who's the EP on the show and oversees all the streaming series for Marvel. And Brian Andrews, who's the co-creator, but really the director for all the episodes of What If, in terms of working with those guys to say, hey, we're going to try this thing out. Are you cool with it? And they're like, yeah, this is awesome. And we showed them what we wanted to do. So all that was just a big confluence of opportunity and partnership.
[00:08:54.769] Kent Bye: And so how did this specific Apple Vision Pro version of What If, an immersive experience, how did it first begin? What was the catalyzing moment or the inciting moment or the turning point for it to come to be? What was that moment?
[00:09:09.139] Dave Bushore: Someone, it's like any great inciting incident, right? There's an explosion. No, I'm joking. Someone robs a bank. It was actually, again, like it was sort of that opportunity that was presented to us. We knew from some conversations that there may be an opportunity to do something with Apple Vision Pro. We didn't know really what it was going to be because it was a while, not that long ago, but... they hadn't announced, they hadn't done any of that. But when Bob got on stage with Tim a little over a year ago, and that big announcement at WWDC, we had kind of known to some degree that we were going to be doing something and that we should know how it was going to go. But that announcement from Bob and Disney saying we're going to create content for this, and then given the opportunity to go do it, right? So that was sort of the auspice of it, right? It was sort of slow going at first and then figuring out how that was gonna go. Obviously there's a lot of, it's a big machine in terms of the people involved and the companies involved between Apple and Disney and so on and so forth. It was also very natural, right? A lot of people wanted to do this, and it was exciting because it's new. And so we just kind of took that jumping off point from that WWDC. We were obviously kind of developing in terms of story, in terms of opportunity. With Marvel, it's great because the characters live in your world. They're about your world. You can be with them. So we had a ton of concepts early on. We didn't know exactly which way we were going to go.
[00:10:27.802] Kent Bye: but then once that solidified then we were like we knew what if was the right one boom we're off to the races yeah when this uh experience first came out in may i was not familiar with the what if series so i like binged all the series and i played through the experience just to get a little bit more context to this world and you know my understanding at least is that the what if universe is sort of like okay we're gonna have this multiversal potentiality of possible alternative timelines of stories. And so you can have main characters, but you have like, oh, what if this happened? And then this character goes this now in this other arc. And so it feels like kind of like a low stakes playground for you to work with in terms of narratively and creatively where you can experiment with stories without it, you know, feeling like it's a part of a canonical part of a larger show that's going to always be referenced. It's kind of like a speculative area that you can explore and kind of explore stories in a way. So I'm wondering like what was the decision point to kind of focus it on that specific IP to then use the mixed reality to explore some of these different alternative timelines or potential stories?
[00:11:35.393] Dave Bushore: I wouldn't say it was easy or speculative. There was nothing easy or speculative about it. There was a lot of speculation about how we would try to pull it off. And I think from a Marvel perspective,
[00:11:49.557] Kent Bye: I just mean from speculative, I mean what's possible and what's actual. And then just from the quantum metaphors, there's the possibility space and then what's actualized. And so in that possibility space, there's more speculative futures that are kind of like these alternative branches that you can go down. That's what I meant. Yeah.
[00:12:06.408] Dave Bushore: No, and exactly that word, the speculative possibilities, right? So in the idea of when we jumped into it, it was we had some fun ideas. One of them was initially Miss Minutes, specifically who is in the experience in a really cool way. Obviously, you've done it. And she's a great character. So we were kind of starting there. But then we knew we needed to go bigger, but was kind of thinking around Loki season two. We didn't know if the timings were going to work out. We wanted to do something connected because it's Marvel. Right. But it kind of organically took place where we were looking at what the technology offered, the resolution, the quality, but still kind of working in the dark. Right. We didn't have full access to anything. You know, we were working with with our partners and trying to figure those things out. But trying to get to what exactly what you said is the speculative possibility. Right. So as The Watcher says, it's a prism of endless possibilities inside the multiverse. Right. That phrase right there is a great entry point for any story for you to be the central protagonist, right? Because when you come in, you don't know what's going to happen. And that hook is awesome for us creatively, right? Because we can say we know we want to be what if, we know it's going to look amazing. It can be all ages, right? We can write a story that feels grounded and elevated to adult audience, but also feels inviting and fun to younger audiences. So from that standpoint, perfect, right? Great. We know they're continuing on that show, right? And we know, to your point, we can have fun. This is about having fun, and what if is fun, right? So if we're going to do something, right? We knew we weren't going to be a quote-unquote art installation, right? As great as those are, and I love those types of experiences. We knew we were going to want to be broad. We knew we were going to want to be fun. And we knew we wanted to get people invited in in a way that was, huh, I want to check this out, and I get to be a part of something. And the multiverse is... You know, as a storytelling device, I personally think it's spectacular. Right. Like I know people have opinions around multiverse, but I think it doesn't have to be about that. Right. It's about how do we put you in the headset? And that was the perfect opportunity. And I think that's the thing. Right. Is one of the things I think XR is starting to transition out of slowly as we're sitting here in Venice and there's this amazing opportunity. installations right here that have all this amazing different type of content in the most storied film festival in the world right where we're seeing iconic films debut and new ips and different things along with all of the immersive stuff is just that which is like now it's being legitimized and getting the audience to go i want to participate in this and so when we were picking ips and different things of course big ips great right because Iron Mans of the world and so on and so forth are great because people know them. They're very recognizable. But they also come with a massive expectation of, I better get to do this. And we still run into that with What If. People are like, well, why am I not able to walk around the entire open world? Or why am I not able to blank? Well, those are all by design, because first and foremost, it was a narrative adventure, and we wanted the story to really matter, and we wanted it to be meaningful, and we wanted people to come out with a little bit of emotionality in terms of, we know we're not telling a story that's about really changing the world, but the idea that choice matters, perspective matters, and when you have more information on what's actually going on in the multiverse in this one instance, Your decisions matter. You get to snap your fingers at the end of the experience and have an impact on those timelines. What you choose directly impacts what those other characters do in their timelines because of the example you've set. And that's a great bar to always try to aim for, right? Narratively, right? What am I going through and how am I understanding the story and what other characters are going through? Because we've always been passive in those things. We have to put two plus blank equals five, right? Well, whatever is in between two plus and five, you putting it together yourself is the fun part, right? And doing it in this type of technology is the really fun part because when you're sitting on your couch and watching a movie, which I love to do, you're trying to figure out the subtext, but you are the subtext in this one, right? So all of that just added up to why What If was like perfect opportunity.
[00:16:02.652] Kent Bye: I've been doing interviews in the XR space for over 10 years now, and one of the things that I've seen with newly launched hardware is if you have early access, you have to deal with a lot of the bugs, and you end up being the front lines of QA, and then on top of that, you're using the Unreal Engine, which is, again, like a fork of the main line, so you're doing your own innovations to even push the limits of what even the game engine can do, so you have... When I think about the evolution of these projects, you have the technology, what it's able to enable. You have the artists who want to be able to use that technology. Then they create something that then is trying to express that through a story or an experience. And then it's distributed to the audience who can then experience it and then feed it back. But ahead of it being launched, then you're kind of in a bubble where you're figuring out stuff on your own. trying to push it to the limits and see what you can do or not do. In a three-DOF headset without any input controls, you can do immersive 360 video, but this is all about how do you start to integrate hand interactions, mixed reality, and tell a story in an immersive way that makes it feel like you're embodied and more immersed. And so maybe just talk through a little bit of that. How do you start to then take the affordances of the different interactions that are available and then feed that back into a story that feels like it's meaningful to push forward a narrative and a story that you want to tell?
[00:17:25.034] Dave Bushore: Yeah, I mean, I think all the things you hit on, right, they're all part of some of the parts, right? So like any piece of storytelling could be a book, be a podcast, could be a movie, could be anything, right? How you build it is up to you, what pieces you put together. We had a great number of pieces, right? We had the same challenges that you brought up, which is being first, right? The first guy through the door is gonna get the bloodiest, right? Because there's a door, right? You gotta open it for everybody else. I'm not saying we did that in any stretch of the imagination, but being able to work on a piece of hardware and in software that we didn't really know anything about and we're learning on the fly. And then, you know, as those updates come and as they got closer to launch and we were working through like, okay, well, there's a bug for this now, how do we solve that bug And we're going, can we solve it? And we're hoping we can solve it. And the clock is ticking. Right. And we're still trying to improve the story as you're figuring out how the camera works while you're shooting the footage. Right. But that's a fun process, too. Right. I think there's something very like indie film about it. Right. And that's what I like about it. But specifically to sort of go into that a little bit differently is I think all of those pieces, right, from we knew it would do hand gestures. We created 10 custom hand gestures for the experience. And that's a big shout out to the design and development team at ILM that figured out how to do that in a really cool, authentic way. And authenticity off of that is incredible. being able to and have the opportunity to work with ILM inside of the Unreal Engine in a very specific way, which I'm not going to go into because it's not my specialty, right? But knowing what it's capable of is now we don't have to convince you that the world is real, right? So a lot of experiences specifically, and for me it was about creating presence with the characters and presence with the environment because I think that's like anything right you want to create a sense of i'm in a different place how does it feel and to your question earlier about the expectation of marvel fans or the expectation of marvel right some of the locations we picked are extremely iconic right some of them we had never seen before and are yet iconic so being able to go to titan right what does that feel like designing a courtroom on titan what does that feel like right how do they operate here and again and we're playing in the what-if world so we get to have fun with those things but Soul World specifically right like we'd seen that in the comics I had loved reading about Soul World in the comics when I was a kid and then seeing it in Avengers Infinity War and Endgame right in very specific ways and then taking that and going okay well how do we and with our writers David Dong and Phil McCarty who did a great job of sort of interpreting what I wanted to accomplish and what What If was about, we looked at very specific comics early on, specifically one, though, where the Silver Surfer gets the Infinity Stones. It's a great cover, and it's about what would he do, right? And so we were like, man, that's a great idea for making you the protagonist, right? That's an iconic character who's given a massive amount of power. And so how do we create presence with the hand gestures, making you feel that you actually have the mystic arts, making you feel that you're in these iconic locations, but doing the What If to it and spinning it and making it different, right? and then taking an iconic story with Wanda and Vision and giving you the possibility to combine those things and to give you the power to give them their happy ending, right? Spoilers. So if anyone wants spoilers, you can cut this part out and then they can tune in later. But the idea of actually being able to combine all of those elements to tell that story where people are not questioning whether they're there, they know and they believe they're in an episode of What If, which is the goal all along, to have an episode that sits in between season one and season two with season three coming. and it feel authentic, a version of intra-story living, right, where you're actually a part of an interactive episode of a show, and not questioning it, and believing that the wish fulfillment of actually being able to do the mystic arts in a really cool way, combined with Infinity Stones, which by the way, that right there, to answer your question earlier about like the challenges or the things that Marvel comes with, which is sort of like expectation and sort of the fun of it is it's all very fun but the expectations are extremely high from the fans and we heard that right we combined the mystic arts with infinity stones Right? Like, with. Not next to or adjacent. So those two competing narrative functions of what those things do and trying to find a way to thread those two together in a realistic way that's like, well, why are you doing this? And being able to kind of pull on those what-if threads of, well, Agamotto, right? He's the original Sorcerer Supreme and the guy who sort of like, in the comics, created the mystic arts and so on and so forth. an original story where maybe he created a spell just in case the Infinity Stones ever were, you know, blank that you could control them in case. But that's the fun of what if, right? So again, just to kind of wrap up, that is a long winded way to say all of those pieces where they're narratively driven in terms of the kind of gestures we wanted, fitting that into the story and then the team being able to build them, the worlds that we could build and why we wanted to go there and the characters we wanted to be in the middle, right? That was the true opportunity of like smashing all of this together.
[00:22:28.101] Kent Bye: And as I do lots of different interviews and folks in the XR industry, there's people are coming from filmmaking. There's people coming from architecture, game design and like human computer interaction and web design. And so each of these design disciplines have different practices. And I've noticed like a trend of like, say, architecture and film tends to have like a little bit more of a linearized process of like pre-production, production, post-production processes. Whereas game design and web design tends to be more rapid iterations, where you're doing constant game testing in order to even develop what the core mechanics are that are interesting. Because you can't know until you actually put it into the process of someone going through the experience. And so these are two conflicting design patterns of rapid iteration versus pre-planning and everything. And so in terms of the story, usually with the script development, you have most of it locked down. And there's obviously things that change when you're shooting it. But when you're entering in this other highly iterative rapid iteration design pattern into that process of the script, how much did the interaction design feedback into the story as you were developing in it? And if you felt like you had like the bare bones of what you wanted to do with the story in place and that it was more of like nuancing or how much those abilities and interactions were actually driving the story into different directions.
[00:23:44.628] Dave Bushore: It was both at the same time, right? So I come from a film background, graduating film school, you know, and then doing a post-grad film degree and then working in the studio system, right? And understanding linear production and how it works, right? But also being able to experiment early on in production with some of the experiences that we had built with ILM and also seeing how they had done things like Vader Immortal and so on and so forth. It was It was a learning curve, but this was the first one where it was both at the same time. So we obviously had ideas. We had a rough outline. We brought the writers in. We worked up a treatment, took that to Marvel, right, internally to Brad and the team and to Brian and everybody and kind of pitched it. And everyone was like, this is awesome. We love it. Okay, go for it, right? And then it was just sort of off to the races. And so then it was figuring out, So we were going to go off and write the script, but we knew, and specifically this writing process was different, but I knew that's what was going to happen because working on damage control, what I learned on that project was we knew we had the opportunity to do these very specific cool scenes. And I am a ardent believer of the story is the most important thing and making that the thing at the forefront. And what I learned from Damage Control was we tried to do that, but we were constantly fighting against the idea of, well, we have this really cool scene and we want to keep it, right? And that's the kill your darlings thing that everyone talks about and everyone knows about. And if you're not willing to do it, then you're going to mess up the story, right? And so I learned on that one that as hard as it is, you do have to do that. And I knew that from working on tons of other stuff, right? But with What If, very early on, I fought really hard to make sure that once we knew we were going, that we were able to go off and think about the script after getting the outline approved and go, okay. But also having to, and this is where the second part comes in that you're talking about in terms of linear production meets sort of massive iteration from a gameplay perspective, because we knew we had to do that. We didn't have... three or five years to develop this, right? We knew we had a general date in mind. We didn't know the actual release date, but we knew generally where we wanted to be from a production standpoint is trying to think through without knowing how do you write a linear story that you want people to be able to passively engage in because this type of technology in the Apple vision pro specifically this opportunity is There's always the I don't like games people who come into experiences or people that love games So how do you balance that right and trying to find that way and so early on it was that conversation? Okay This is the story we want to tell writing that down and then going back and looking at each scene and knowing where we were in the production process talking with the development team and talking with the teams that were developing the gestures and what they were learning and how that was going to work and going, OK, how do we overlap the iteration inside of the script? And it was extremely hard, but a testament to, I think, the sort of thing I've sort of coined and pulling from one of my favorite movies that we've talked about lately is the never give up, never surrender mantra of Galaxy Quest is sort of just always pushing those two together. is just this idea of doing that for the first time was really really really hard but we had commitments on the writing side to constantly go oh hey we're going to change this gesture right or this thing doesn't work anymore and we literally had to rewrite some gestures or some things to make it work in the os or make it work in the headset and that's not a negative thing it's just the way it goes right And then the development team taking the new idea and going, OK, now how do we make that work? So it was very much a back and forth. But I think that's the way this is going to be for a while, as people learn, especially this new piece of hardware. But it was fun. Like I said, I think there is a little bit of pain involved. But when you're developing the camera as you're shooting the film, there's going to be those things that happen. It was very much something that I wanted to do from the beginning creatively, to know that I had the flexibility, A, to push the story so I knew where I could push, right? Because we knew we had to do the animation. You can't change the animation at some point, right? The characters have to move and be there. But then knowing, asking my producer, Sharif, okay, how late can I change this? How late can I change that? When do I have to stop with this? And then understanding that creatively. And as we were learning, getting better and better and better understanding of where you can push and pull from a production standpoint. And I think that's the beauty of what great filmmakers do. I'm not putting myself in some class. I'm saying what you see great filmmakers do is... They know the process so well and they're able to take advantage of it. And so for me, that was the learning curve of going, well, I understand linear production and I understand XR production, but we hadn't done this one before. And so it was learning the cracks. And now I think I know where the cracks are a little bit better. I think we all do, especially the ILM side. So it was fun. It was fun.
[00:28:29.520] Kent Bye: Yeah, it's really fascinating to hear that process because it is a bit of a blending of those and having a producer that's trying to put in some limits and constraints and deadlines and to kind of match the linear deadline nature to something that is also iterative in a way that wants to always kind of be changing and evolving.
[00:28:48.780] Dave Bushore: way, right? I mean, I think it's ILM Marvel. I mean, everybody wants to push the creative as far as they can. There are limits. I don't think anybody creatively wants limits, otherwise they wouldn't be in a creative industry, right? But I think it's a testament to the ILM team and all of the people that help bring this to life from from the Disney side, from the Marvel side, from the ILM side, like we had people coming out of the woodwork to say, I'll help with that, right? And I mean, that's a testament to, I think what this type of technology is, is really like getting people excited about it is when they see it and they, oh, I can help you make that better. I'll do it this weekend. We had that on this project. It was amazing. People that had big responsibilities from visual effects side of things that would go, hey, are you guys done with this thing? And I'd be like, why? And they're like, well, I can see something we can do here. And it was really, really, really awesome to be a part of a project where it was a grassroots team effort through and through, and everybody poured their heart and soul into it. And that's not... say it was special in that way, but it was special because I think we didn't know what we were doing and everybody just went every day with like full fledged passion to figure it out.
[00:30:00.438] Kent Bye: Yeah, one of the key affordances that I've noticed with Apple Vision Pro is how good stereoscopic 3D movies look. And VR technology has been out for over a decade, but I think the screen resolution combined with partnering with Disney and big companies that had libraries of 3D content and then wanting to have the apps on there, but also have just how magical it is to have this story behind stereoscopic 3D content that has always been there but the form factor for people to see it have been really imperfect with 3D glasses in a theater like seeing it in the VR headset or this spatial computing headset feels like its final form for where it's really meant to shine and it feels like the stereoscopic cinematics that you have and what if is able to really lean into the magic of that of being able to set the context to set the scene and then then you get embodied into the world but at what point did you discover that the 3d stereoscopic cinematics was compelling enough to dedicate so much time to integrating those different segments into this project
[00:31:06.135] Dave Bushore: It was definitely around when we were having conversations with Pat Conron, who's the visual lead on this and has been at ILM for almost 30 years and trying to figure out how far can we push it, right? So like I said before, presence, right? And so telling a story is having the characters be believable and like, you know, someone was going through here in the Venice space the other day and we've heard it from other people, but When we were developing it, we were like, I would be constantly asking Pat, like, can we put a shadow here? Can we do this? And like shadows, you don't, like when you think about 3D, right? To your point about the glasses that we've all worn for years in the theaters, right? There's a barrier to entry that you put on at the door, right? Are there amazing 3D movies? 100%. And I love 3D movies. But in the idea that... when you put on the headset and once you get in there, there's no more barrier for entry. So talking with the visual development team and the visual effects team with Pat and them is going, how far can we push the shadows? How far can we push the lighting specifically in scenes? And then looking at the stereoscopic cinematics you were talking about and from a narrative function, having those work in sympathy and in concert to tell that story. So again, as soon as the characters come and you believe that they're really there, that it's what if in your living room, giving you the opportunity to quote unquote take a break was one of the ideas because what I've found is at least the experiences that I've done at home and worked on with people is we're at the very early stages even though VR technology has been around for a decade plus, right? Really it's hit its stride in the last 10 years. We're still cognitively at the very early stages of people understanding in their head what does this feel like, right? So we really took that into heart and I really wanted to make sure that again, testament to my producer Sharif and to the team, as we were talking through all of these, they were really adamant about being able to have some passive, right? And I was like, that's the right, that's a good note, right? So that we would have this big overwhelming beginning, right? Where we're showing you something in VR, then we're throwing the logo over your head and people always react to the Marvel Studios logo because you've seen it for years in front of the biggest movies, right? And you know what it's like, but you've never seen it fly over your head. And you get that visceral like, oh, wow, right? and then being able to just pace those things out. And so those cinematics can be very strategically to, hey, we're going to tell you a story, and the first one you see actually is Wong. So Wong comes through a portal into your living room, walks in, right? And he's like, ugh, this guy, I can't believe, or this person. But then he starts telling you a story and he brings up, you know, a cinematic that tells you the backstory. So we set the stage early to teach you, hey, this is how we're going to give you backstory, this is how we're going to set that up so that when we get to the penultimate ending in Soul World, right before the big battle, and you're learning the backstories of these characters you've been with, you have a very good understanding of how to digest really heartfelt emotional content, right? But we start low, we start the bar low, small shard, small amount of content, easily digestible, 10 seconds, right? Then, okay, here we go, next adventure, and you're off and running, first big cinematic on a big, amazing, multiversal shard that you've seen from the show, which was an awesome conversation as we were developing it, just how to get that looking right. And then getting that stereoscopic to look right, one of the very early conversations was parallax, right? Being able, knowing that people can move in their space with the Apple Vision Pro, like not wanting them to run around, but the idea that we knew they could and would probably want to to some degree, is being able to look around the corner. That's the fun part of this type of content, right? It can be flat, but if I look through the looking glass, what's on the other side, how far can I see? That was a really fun conversation, and we didn't even push it as far as we thought we could. That was more of a production thing, right? How many eggs are we gonna put in each basket? But to your point about the cinematics, there was a lot of strategy and creativity that went into planning those and writing those and trying to set up the audience to understand the mechanism for how we're gonna bring the story to life and so that you feel a part of it, right?
[00:34:53.732] Kent Bye: Yeah, I felt like really set the broader context in the scene for each time where because a lot of times it's a problem where you're thrown into a world and you're like, OK, where am I? What's happening? And then I feel like that 3D stereoscopic cinematic gives you a great mechanism to use the language of film to quickly set the context for this so that when you are embedded into a place, then you kind of have a little bit more idea for what's happening.
[00:35:16.567] Dave Bushore: Think about those big action movies that you go see in the theaters, right? And they need to get you from point A to point B and fill in the context. There's a quick 30-second montage of news footage, right? And so you're used to that language. You're used to that news update, right? And so we just thought of it as a big screen, right? Like really cool multiversal glass shard screen. Like we knew people would go, oh, right, I watch. And then how do we put the watcher in that space now? And what's really fun is you do the experience and you see that footage. If you look at the watcher, he's watching. If you move in the room, he follows you around, but he's ever-present, just the way he is in the show, that omnipresence. So you're layering what is a storytelling device and him bringing that to life for you, right up to you, but then also being in 3D space in the back of your living room, and you've got layer on layer on layer. And that's the fun of what that device can do.
[00:36:07.691] Kent Bye: Yeah, I know one other big component, technologically, at least in this project, is that it's actually using mixed reality beyond just simple VR. You're actually taking the world around you and having overlays in a way that I know that Unreal Engine is something that even people that have access to existing versions, that there was some magic wizardry that happened to happen to even get that to work. But in terms of telling the story, how could you start to use the affordances of the mixed reality component to then bring these characters into your home. Because sometimes when I do VR, I don't want to see my messy area. And other times it's like, particularly aware of like, okay, am I going to have a murder scene in this location in my house where Now, every time I come in here, I'm going to think about this murder here. So there's like this, like I'm a little bit more cautious or hesitant to the, what am I bringing into my space? Cause it's going to be actually impacting my memory. But I feel like this is enough of a suspension of disbelief that you're able to go into this world that I didn't have that hesitation, but it's still like a part of the affordance of like working with people's physical spaces and having a deeper connection because it feels like they're having these friends over, but they're not friends. They're like these characters from another universe. And so I'm wondering how that process of the affordances of that mixed reality started to play into how you started to think about what you could do with that.
[00:37:29.468] Dave Bushore: I think it was, and it's interesting, I totally agree with you in terms of the memory aspect. I think there's a lot of stories to be told specifically around the word memory with people. Because I think the idea of having really impactful anything, any story, any experience creates a memory, right? And so that you said that is exactly what I think is the power of any story, right? You come away with something. I think it's interesting, your point, and I'm just kind of fueling off of what you're saying because it excites me, is I think it's an interesting conversation around what people bring into their spaces. It's the same conversation around what people bring into their homes in terms of content for their kids or so on and so forth. And I think those are conversations. But I think what's very interesting is, is will people in the future have dedicated rooms for these things, right? Where it's like, this is where I do these things because that's where they happen. I think that's a great, like people have screening rooms, right? Like I have a big screen in my house and that's where I watch movies. Right? I think that's where this is all headed to some degree. But then sort of to answer your question around bringing the characters in is one of the very early things I really wanted to push for was specific scenes where with Wong, his entrance and him feeling overpowering and cool because he comes through a portal. So the Watcher appears because he is in the multiverse, he is omnipresent, he can come and go as he wishes because he has control over where he can go, right? If he wants to go somewhere, he can go there, right? And Wong is different because Wong is from very specific places, right? And he uses portals, right? Which everybody loves and people have seen him in movies and in comic books and they always are like, I want to do that, I want to do that. And so we set that up very specifically, right? Wong being our mentor character, and being the character that's a little apprehensive about this whole thing, being like, you're gonna do what with this person? You're gonna, what? Is that whole idea of him feeling powerful, cool, and fun, and sort of the guy that's gonna rib you a little bit, and then when we get closer to the end of the story, gonna be your buddy. is as he enters through a portal and then it closes behind you, the very next time when he finally agrees to train you is then he throws a portal over you. And we're building up this sort of like, wow, that character did two different things. He entered my space, he took my space and turned it into another space by taking me somewhere. You're still in the same room, right? But that idea of how does that interplay work between the characters, right? And that adding gravitas or adding blocking right because so much of what we wanted to do which is theater in the round really right because most of our experience we have you facing forward to a degree facing in a 270 sort of theater space you can look around it's 360 of course we built it for that there's all kinds of easter eggs in there for exactly that reason and there are fights in the experience that take place in 360. But that idea of blocking, right? How do you move characters? How do you make them feel powerful, cool? And then elevating Wong into the third act where he comes to you for help, right? So when he enters the first time, you need him. And when he comes back at the end, he needs you. And when he enters your space, he walks right up to you. And you're watching the same, like you were saying earlier, cinematic shard. One of the things I really was specific about and wanted to do was have him stand right next to you. That you're getting the information the same time as he's getting it. So when he's like, we have to find the time stone, right? It's been stolen. And he's looking through all of the shards and pulls up a very specific timeline because he thinks he knows where it's at and then he's right. And then that content is playing out and you're watching it in real time because he's looking through the multiverse with you. That is really fun to get to play with right now. How many characters can you put on that stage? How do you play with that? What kind of blocking do you have? What role do you play in the blocking? How do we get you to look where we want you to look? And that was something we did with Wong in Soul World, where we kept elevating where he was in the space. So when you get into Soul World, as you probably can attest to to some degree, is it's very existential. It's very ethereal. You don't know which way is up or down, maybe down because we did the amazing water. It's like a liminal space. Yeah, exactly. And you sort of like, when you saw it in the movie, we went, okay, well, we know what we want to design it as because that's what people are going to reference. Because in the comic books, it's very liminal, very ethereal, right? And we played on that with the dialogue with Wong at the end. But the idea of being when you get into Soul World, beginning the first 30 seconds, we don't give you any content. No characters show up. It's just very ethereal music done by Laura and the team to like really create that existentialism. And then we start slowly building on the story is specifically keeping you then in that 270. Once we get you in there, we're going to let you go, wow, I don't know which way is up. But then we're going to set you, right? We're going to block the character so that we can keep your attention span. But then we're going to surprise you. So when Vision is like, well, it's been really hard to find a way out of Soul World. And then off to your right behind you, you hear Wong say, impossible? And you have to, oh, whoa, Wong's here. Amazing. Wait, I was just looking over there, but he wasn't there a second ago. Or was I looking over there? Was I looking to the left? And then, wait, oh, wow, cool. That type of ability to change your space and then change the space that's around your space, playing without blocking is the fun part.
[00:42:36.721] Kent Bye: Yeah, one more kind of theoretical story question for you, then we can start to wrap up. So I think one of the ultimate potentials of this medium where it's going is like this live action role play where you're making choices and those choices have stakes where it actually impacts the unfolding of what the story is. And I remember this conversation I had with Eric Darnell of Baobab Studios back in 2016. And he said, look, there's this dialectical tension between storytelling practice and game design, where the storytelling practice is a time-based medium. As a director, you're controlling very precisely the timing of events that you can control the emotional pacing of the story as it's unfolding. And as soon as you introduce agency into that equation, it disrupts that normal building and releasing of that tension where it then becomes more of a matter of a question that the audience is asking themselves is what can I do with my control of my agency at this moment rather than receiving whatever stories that's coming in. So it becomes more of a yang exertion of energy rather than a yin reception of a story that's coming in. So that's like the yin and the yang of that receiving versus giving. And it tends to be like The fundamental paradox of interactive storytelling is that you either end up on one end of the spectrum. In this piece, it happens to be on a little bit more of a linear narrative with light interactivity, but the interactivity is not having any sort of real consequence into the outcome of the story. And so it makes me think of a piece like Facade by Michael Mateus and Andrew Stern, where It's a much shorter experience, 10 to 15 minutes, but there's much more variance for all the different branches you could go through. And so it's more of a replayable game that you explore the possibility space of those actions rather than a story that's being told. It's just like a completely different paradigm where they're trying to look at what's happened at that moment and deliver you the best story for whatever actions you've taken up to that point. So all that's to say is this question around the role of interactivity and agency and how you foresee that continuing to evolve, if the stuff that you personally are working on or Marvel or Disney is still going to be biased towards that story-driven stuff, or is there going to be more experimentations for the ability to explore different levels of agency and how that impacts the story as it's unfolding?
[00:44:55.894] Dave Bushore: I think it comes down to what you're doing and why. So when I say what you're doing, are you telling a big, long, linear story that you want people to follow the linear story? And to your point, we tried to do a little bit of everything, because we all are story nerds on the team. We all are gamers on the team. We are all XR fans. We've all played in this space for a long time, and we all know and there was no end of ideas. It was how much opportunity we have to do them. So we tried to figure out ways to balance it as much as possible to give, again, a general audience coming into an experience like this from Marvel Studios, ILM, that's inviting and fun and for all ages, a little bit of all the things we love. So to your point, that piece that you were talking about is branching narrative was something we talked about, but we knew from a production standpoint we did not have the time or the resources to do true branching narrative. As much as the multiverse provides that opportunity, it was bringing you branches of the multiverse and letting you see what's going on and then putting you in those scenes to have some agency right so not exactly inverting it but sort of right and just trying to tell that a different way and making you feel like you're part of a bigger three-part thing that's taking place in three specific locations and we we had fun with the marvel comics idea of the old adage you know earth 616 right from the comics is the home base of all the marvel stories and then there's all these other addresses where earth 719 or whatever right we did that right so what's fun is at the beginning of the experience it says earth present day So you're like, oh, I'm here. This is the timeline I'm in. Present day is whatever day it is, right? We didn't want to get into the malfeasance of trying to figure out what timeline you're in. But it was fun. And then the idea of, OK, well, I'm told the story that's happening in this timeline. And when you get to that timeline, you transverse through the shard into that world. It says, timeline blank, blank, blank. So while I'm on a journey, it's branched, there's these different stories, and I have a little agency inside of here. I have a little game play, right? Because it was Pick Your Poison, right? You can, from a production standpoint, and being the first one to do it this big on Apple Vision Pro, not knowing certain things, it was a question of how far we really courageously, and there was a lot of courage to do any of it, push certain things, right? But we knew in the end, the choice moment was a narrative function of the Infinity Stones, and we knew from a fan perspective that just being able to land that plane, right? Being able to build the narrative function of the mystic arts to where you get to give Wanda and Vision their happy ending at the end, right? Using the Infinity Stones once you've earned all six, right? You do that amazing thing, right? Which is so cool narratively and for comic book fans or people who are fans of those characters or just a fan of the story when they're in it and they're like, wow, that's really special. I did something cool for those people. And then the next thing up is, well, now you have all the stones, and the Watcher warns you, like, I cannot intervene because that's not what he does. This is up to you, right? Now trying to give that agency back. So pairing the emotionality, pairing the agency, pairing the interplay of those things with gameplay and fun interactions that are iconic, right? That's Marvel. Right. That's what we wanted to do. And so we pulled that off. That was our goal. But I really think it comes down to the creator. Right. And what they want to do. I've got a bunch of ideas for how to push those boundaries even further. And I think not just IP based stuff, but I think what you see here in Venice and how the people that have created some of these experiences are playing with. Is that interactive or is that story? Right? Because the type of story that they're telling, right? Like, I think in terms of the emotionality is what strings are you pulling on on the audience? Are you asking them to delve into how they feel as a human being? Are you asking them to participate in a story that's maybe bigger than it's about the world or the multiverse, right? And then it's just about how you pull on those things. And I think that's the fun part of creating anything, right? It's just how are you using the technology? How do you want to do it? And then what do you want the people to feel when they come out, right? Like, I think that's the fun part. Like, Not to go into it, but like, yeah, I have like so many ideas in terms of like, I think there's a massive opportunity here to do what I would call a generic experience where everybody comes out with the same ending, but everybody comes out feeling something different. because it's individualized. That's the power of this technology as we all know who are fans of it, is it's like an unbelievable empathy machine because it's your world, right? You're just there with you. Now, in the future you'll be there with other people, but you're still only having your experience. And that was like the whole auspice for the what if conversation I was having with the writers is, Everything about life and storytelling is perspective, right? You only have your perspective on the world. You can hear other people's perspectives. But I think when you get perspective, which we have, one of the chapters when you go to Soul World is literally called that, right? Where you learn the backstories. And so how do you open those lenses up? And that's what everyone's doing here, right? They're pushing that lens of storytelling just the same way cinema's done for years.
[00:49:53.604] Kent Bye: And finally, what do you think the ultimate potential of this type of immersive storytelling might be and when it might be able to enable?
[00:50:00.970] Dave Bushore: I really think it's the opportunity. Cinema for me, when I was a kid, and I think for a lot of people growing up, for my dad, I would always watch movies with my dad. And then when I would go to the movies, it was the working man's art form in terms of you could go to places that you couldn't go to maybe in the real world for $8. You could go to the cinema, you could watch a movie about Lawrence of Arabia, and you were like, wow, that was amazing, and I felt like I was there, and I didn't know anything about that, or I learned something, or now I want to learn about that, or whatever. And I think that's still there. I think that's still part of cinema, obviously, learning things in the storytelling and learning about different situations people are in or adventures or whatever. But I think really the biggest opportunity is that cross-media opportunity to do both at the same time, to have a show or a movie that has parts and pieces that live in sort of the immersive world. Do you have to do both to participate? No. Can you do one or the other? Yes. If you do both, do you get a bigger story? Why not, right? But I think it's also the physical spaces. I think there's so much opportunity to push on physical spaces. As we all know with AR technologies, it continues to develop and we're a long ways away from this. But I think what Vision Pro shows us is I don't have to wear a backpack anymore, the quality is immense, and obviously there's other companies doing, we're just talking about this Vision Pro project, is how do I overlay things onto the real world and be able to go to places to have completely new experiences I really think like, we could talk about this for an hour because that's the power of the medium as you know, but even just what I was saying before about when you were saying bringing things into your space, do people have dedicated screening rooms for these types of things and special products that they've bought that go in there? 100%. Because there's people developing all of those things that are going to pair with devices or do certain things. As those things develop and as those things, you're just going to see an explosion, I think, at some point. Once the tipping point happens truly from people feeling like, oh, I don't know if that's for me, to I'm missing out on being a part of that, the floodgates will open.
[00:52:06.316] Kent Bye: Awesome. Any final thoughts or anything left unsaid that you'd like to say to the broader immersive community?
[00:52:11.428] Dave Bushore: No, keep making stuff like everybody. Keep making stuff, make big stuff, make small stuff, make stuff that means something to you, because that's the most important thing, because someone else will find it and go, that means something to me because we're all humans. We're all going through stuff. We're all experiencing the world in our own way, but we're all experiencing the world together. Right. So I think just keep pushing and go big, right? Like whatever it is, tell your story, tell your story big, you know, try to get the resources to get behind it and push it as far as you can. I know so many people are doing doing that work and we were so amazingly like, had the amazing opportunity of being inside of Disney and Marvel to have resources at our disposal, right? But a lot of people don't have that in this community and they are working really hard to get them and I see it here talking with people, right? And so many amazing, cool things where I saw, I'm gonna butcher his name, but it's the So Mara. Do you know the exact name? I'm really bad with names.
[00:53:05.768] Kent Bye: Which, the name of the piece?
[00:53:06.708] Dave Bushore: So Meta, this, Giampaolo?
[00:53:09.719] Kent Bye: Oh, yeah, the Nightmara?
[00:53:11.101] Dave Bushore: Yes, Nightmara, yeah, yeah, yeah. I'm really bad with names. Yeah, Gianpaolo, yeah. Gianpaolo, yeah. So I was talking with him yesterday, and what I said to him was, doing that experience, it reminded me, I said, right when I came out, I said it to the docent, the girl who was running the experience, she's probably, I don't know, 20-something, right? And I said, she goes, what'd you think? And I go, that's The Simpsons, version one. Because when I watched it growing up with The Simpsons and growing up with those types of shows, what were they doing? Pushing the boundaries of what story could be in terms of animation for adults and for audiences at home. And the animation was so rudimentary and new, but groundbreaking at the same time because they didn't care. They were like, this is what we're going to do. And when I saw his piece, I went, that's an episode, that's a thing that I'd tune in for because the way he structured it, and the way he did cut points, is I know I'm not actively participating, but I'm actively participating in the story because he's putting me in positions where I don't know what's gonna happen. And the way he moves things in the space, is just calling that one out specifically. It just blew my mind because it's so original. It's so specific to what it's trying to do. Do that. Everybody who wants to tell a story, do the thing that's in your head. Keep going because we need so much more of that in this. And everybody's doing it, so let's just keep going.
[00:54:21.345] Kent Bye: Awesome. Well, after listening to you for the last hour, I feel super hyped and psyched up for the future of this whole medium.
[00:54:27.367] Dave Bushore: What's that? That's the one thing I'm really good at.
[00:54:30.126] Kent Bye: So, yeah, I can feel the passion coming through around as a story nerd myself and also listening to you to think about the possibilities and what you were able to do and just hear a little bit more about your process. It was a real pleasure just to hear about all the different considerations that you did to bring this project together. And, yeah, I think it shouldn't be underestimated about how much innovation had to happen in order to even bring this project to completion in 2020. as early into the product release cycle as it was so and as a project of this scope and scale so i'm really excited to see where y'all take it in the future so thanks again for joining me on the podcast to help break it all down of course thanks that was it was great to be on and great to finally meet in person and yeah this is awesome thank you Thanks again for listening to these episodes from Venice Immersive 2024. And yeah, I am a crowdfunded independent journalist. And so if you enjoy this coverage and find it valuable, then please do consider joining my Patreon at patreon.com slash voices of VR. Thanks for listening.
