ILM Immersive & Marvel Studios is releasing What If…?: An Immersive Story exclusively on Apple Vision Pro tomorrow, and I had a chance to take an early look and speak with producer and executive producer Shereif Fattouh as well as technical art director Indira Guerrieri about the process of creating this experience. It’s mostly an immersive story with light interactions, but you can’t actually die and so it’s more about immersing yourself into these worlds with first-person embodiment and exploring around 10 different gesture-based interactions in nearly 50 different interactive moments across the 40 to 45-minute experience. There’s also some disembodied moments where you’re watching 3D movie cinematics on shards of glass floating in the environment, and so they’re also really leaning into how good 3D movies look on the Apple Vision Pro within this experience.
Overall, it’s blending lots of elements together that feels like something new and a new genre of immersive storytelling that’s building upon existing IP, and giving new opportunities to engage with these character’s stories as well as their worlds.
This is also the first Unreal Engine experience releasing on the Apple Vision Pro store, and so Fattouh told me that ILM Immersive has been making private modifications to get things working. There are features like mixed reality passthrough that are not available yet in Unreal Engine 5.4 yet. Agile Lens’ Alex Coulombe told me, “The Epic Games XR team is small so I would I err on the side of assuming no major Vision Pro updates coming soon from Epic. To have the current features working it still requires a source build of Unreal Engine 5.4 (the Epic Games Launcher doesn’t work) and current features include fully immersive mode, hand tracking / gesture recognition, forward and deferred rendering.” So hopefully we’ll get more on this soon from Epic Games, but at least we know it’s possible and we’ll start to see more Unreal Engine experiences on the Apple Vision Pro soon.
Be sure to check out the transcript below for more information, and also don’t miss my episode featuring my talk on an Elemental Theory of Presence, Experiential Design, and Immersive Storytelling, as well as my previous 480+ episodes on the topic of immersive storytelling over the past decade including previous coverage on Cycles, Myth: A Frozen Tale, Star Wars: Tales from the Galaxy’s Edge, and a conversation with John Gaeta about ILMxLab’s The Holo-Cinema from Sundance New Frontier 2016.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:11.863] Kent Bye: Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So in today's episode, we're going to be diving into a new experience that's coming out exclusively on the Apple Vision Pro from Marvel Studios and ILM Immersive. That's the Industrial Light and Magic Immersive from Disney, and the piece is called What If? An Immersive Story. So it's an interactive Disney Plus original story that's being released on Thursday, May 30th, and will be offered as a free app for a limited time. Just to clarify, it's called a Disney Plus original story, but it's not actually going to be on Disney Plus. It's only going to be a standalone application and exclusively on the Apple Vision Pro. So I had a chance to talk to an executive producer and technical art director about this piece to dive into a lot of what they were able to create. And it's very much an immersive story with some interactive components and in some ways forging in a new genre of immersive storytelling that's really heavily leaning upon the cinematic tradition. This piece is based upon the What If series that's on Disney+, and there's two seasons of it. And I actually watched through just so I had a little bit better sense of some of the characters that they might be focusing in on, but also some of the rules of the world that you're going into, since you are in this multiversal realities where What If as a series really dives into like, what if this happened instead of this happened? And then in some sort of parallel universe, this is how this character's story plays out. So that's kind of the essence of the world. And what are the rules of that world? There's a few things that if you watch the TV series, you'll get a little bit more out of this immersive experience. However, this stands within its own right. It doesn't necessarily lean upon the existing storylines within the TV series. And I'd highly recommend folks to go check it out and watch it. It's going to be free for a limited amount of time on the Apple Vision Pro if you have access to it. Before you listen to this, we're going to be talking about it. And I think it's always better just to have your own direct embodied experience and you get a little bit more out of it once you hear us talk about it as well. And ILM Immersive has done a number of different experiences that if you haven't checked out, you should definitely check out. There's the Star Wars Tales from Galaxy Edge that's on the Quest. There's also Vader Immortal that's also on the Quest. There is also a number of different pieces that I covered previously, like Cycles premiered at SIGGRAPH 2018, although that was more from Disney proper. Then there's also Myth of Frozen Tale that actually happened to see it back at Sundance 2019. And it's actually also released on the Quest and also on Disney Plus as a 2D version of as well as john gaeta from what used to be called ilm xlab it's now called ilm immersive they had something called the hollow cinema that was showing at sundance 2016. And they also did a number of location-based experience collaborations with The Void, back with Star Wars Secrets of the Empire, as well as with Marvel Studios, Avengers, Damage Control. And then they also worked with Alejandro Iñárritu and Legend Entertainment for Karna'i Arena. So they've been in the immersive storytelling business for a while. And so this piece is starting to kind of pull in a lot of what I think is like elements of watching a 3D movie. So there's like disembodied third person perspective. You're watching these different cutscenes, but then you kind of have these first person embodied interactions that allow you to be immersed into the world and also have these light interactions that allow you to embody a character as an apprentice who's starting to learn how to operate some of these different Infinity Stones as you're collecting them throughout the course of the story. And also, one other point that's worth mentioning before we dive in is that this piece was actually built on the Unreal Engine. And I checked with Alex Coulombe, because I know he's been trying to get the Unreal Engine 5.4 to work on Apple Vision Pro. And he said that, I can say with 100% certainty that this will be the first Apple Vision Pro app using Unreal Engine that's on the store. So in this interview, Sharif told me that they've been making their own modifications to the engine in order to get it to work. So that's number coming on today's episode of Voices of VR podcast. So this interview with Sharif and Indira happened on Tuesday, May 28th, 2024. So with that, let's go ahead and dive right in.
[00:04:12.898] Sherief Fattouh: Hi, I'm Sherief Fattouh. I'm executive producer here at ILM Immersive and produced and executive produced this experience.
[00:04:21.943] Indira Guerrieri: And I'm Indira Guerrieri. I'm a senior technical artist at ILM Immersive.
[00:04:28.676] Kent Bye: Maybe each of you could give a bit more context as to your background and your journey into working with VR.
[00:04:34.332] Indira Guerrieri: Well, in brief, I used to work at ILM for a long time in the Star Wars episode one, two days. And then since then, I've done a lot of sort of looked at on creatures. I moved into some briefly into the startup space for AI beings. And then I found myself back at ILM, but in VR, which is very enjoyable. I've been having a great time doing that. That's in brief my journey.
[00:05:04.445] Sherief Fattouh: For me, I actually started in traditional AAA games. I was working at Electronic Arts for almost 15 years, worked on a lot of different titles at EA Sports, Dead Space franchise, all iterations of that, except the new remake, which is fantastic as well. And I worked in mobile briefly at Zynga after EA, and then I've been at ILM Immersive, previously XLab was what it was called, for almost seven years now, and really focusing on kind of immersive storytelling in VR, AR, et cetera. So that has been my background.
[00:05:41.003] Kent Bye: Okay. So we're here talking today about the immersive project that's going to be on the Apple Vision Pro called What If, and maybe you could just give a bit more context as to where did this project begin?
[00:05:53.447] Sherief Fattouh: yeah so it was a really fun concept you know we were kind of presented an opportunity you know disney and apple and you know have a great relationship with one another and so when it got finally you know word that this device was going to be released um and that was going to be this amazing you know kind of mixed reality device we were kind of given a chance to think about what creatively we would want to make in the space that's authentic to our mission you know at ilm and marvel and you know all of the internal Disney groups of like creatively, what are the kind of stories that we could tell with having a device that is going to be able to do the kind of pass through and all the advancements and the technology with the hand detection, et cetera. A really exciting development that we'd all been kind of thinking about, especially with an ILM immersive, but across the company of what could we do once the technology reaches the point where we can seamlessly create experiences across both mixed reality in your living room going into virtual reality and all those amazing, amazing technological advances. So once we got an understanding of what the capabilities were, we really started to brainstorm what kind of authentic story could we tell. And then working with Dave Boucher, who was our director at Marvel Studios, it really was this really natural fit to go into the What If universe because obviously from its namesake, it really is exploring all of these different possibilities of what could be done with these iconic characters and kind of having that freedom to tell a story that with characters that you know and love, but different variations of them, you know, just provided that freedom that was really great to tell like an authentic new story and really have it fit inside of the universe that was just so perfectly structured to do so without having to be too concerned of like where you are on the timeline and the rest of the kind of normal franchise, you know, zeitgeist, right? It is very curated to a great way, but it's like that IP, you know, specifically with What If is very, uh, It's really about having fun. You know, the showrunners, we got to work with them very closely and Brian Andrews and Brad Winterbaum. And, you know, really the premise of that being the explore these different corners and possibilities, you know, just like the comic books did. I used to read a lot of those and it was just a really fun opportunity to do a showcase experience for what we could do.
[00:08:13.422] Kent Bye: Okay. And yeah, because it is mostly a linear narrative in a way that there are some choices there at the end, and there's also some interactions. And so I'm wondering if you could give a sense of, as you're developing this project, if you're looking at like a script first, or if you're also diving into the interactions to see how the interactions would play in and just walk through that. And then we'll sort of get into the other technical art and design side, but maybe just level, like, where did you begin with putting together the story?
[00:08:42.932] Sherief Fattouh: Yeah, I mean, again, it always starts, you know, with the experiences that we build, we always are, you know, story first, narrative first, you know, what is the authentic story that we want to tell here? I mean, I think Marvel has been so great with working with us of wanting to tell, you know, even though it is in an animated style, and it's also on this kind of device that, you know, we're not really making a game, it is an immersive story. And so the themes that we had here of redemption and possibilities and perspectives, you know, is pretty much a standard creative process of really there's a technological component that is required in any kind of software experience where we were getting an understanding of like what is capable, what we're capable of delivering, what's not capable, right? Like it's not all magic. Like there are technical constraints that occur with any technology. So really the creative process was starting with getting an understanding of what is possible that we can do within the framework of the technology. But then it became a very traditional script development of, okay, now we understand these prototypes and tests that we've run. And now we're giving it back to the writers and the key creatives at Marvel to think about what story they want to tell there. And then we work, though, very closely hand in hand with everything is very thoughtful in the experience we try to be of how is the interactive going to kind of further the storyline? Like it's not just for the sake of, oh, this is cool. We can do it. Let's just wedge it in. It was very much thoughtfully of. How can this enhance the story that we're trying to tell? Why does it matter to the characters? Why does it matter in this fiction? And I think the writing team did such a fantastic job of listening to all the feedback from our interactive teams of like, OK, well, we can do this, but not that. This is something that's on the table. You know, there was a lot of exploration there. We all kind of share this love for the content and have folks here that we work with on development team are all hardcore Marvel fans and some of them like Indira and others. Well, I don't want to speak. I don't know for sure if you've worked on a Marvel show before, but a lot of folks worked on visual effects for the films, so they're intimately familiar with that creative process on that end of it. So it was a very natural progression working with them, with the filmmakers to really understand what story once we give them the context of what you can do and then translating it into what we should do and doing story development and storyboarding and concept art and all the awesome parts that get us to our final product.
[00:11:09.511] Kent Bye: And Indira, I had a chance to watch the two seasons of What If before I actually watched the VR experience because I wasn't familiar with the show. And so when I was watching the show, I had a hard time discerning whether or not it was all done in 2D or if it was made in 3D. And so I'm wondering, you know, at a baseline, if there was existing assets from the show that you start with when you start to work with the art side. I'd love to hear where you came in and what needed to be done in order to start to do this translation into a fully immersive piece.
[00:11:40.714] Indira Guerrieri: Yeah, it was interesting because it's a D&D show, so a lot of it is composited. There are various pieces that are added, flame bark and others. There was a look that was necessary for the show that we wanted to... emulate as much as possible in a 3D world, which was challenging because you're breaking into a different dimension. So we had this amazing artwork. A lot of the sort of acid textures were already done from the seasons. So we had to basically take all of these assets and make sure that somehow they lived together. in the space, in a 3D space, which actually brought in all sorts of conversations, I'm sure you remember, with others about how were we gonna keep the 2D aspect or how were we gonna add on to that. And we ended up with a sort of combination of being really true to the beautiful artwork, the beautiful work that was done, and adding the spatial dimension by making everything a little bit more realistic in some elements of the environments. The characters stayed very 2D, but there were cheats in order to achieve dimensionality in the toon shading. So, yeah, it was a back and forth trying to design for that 3D experience and keep faithful to the original.
[00:13:12.246] Kent Bye: Okay. And I saw in the credits that at the very end, there was Unreal Engine that was credited. And then earlier in the credits, I saw that there was Engine Engineers. Is this something that you use in Unreal Engine or is this something that you created your own type of system to be able to work with Apple Vision Probe?
[00:13:29.528] Sherief Fattouh: No, we did end up using the Unreal Engine. So we obviously, in any project, there's a lot of modifications that you want to do to achieve whatever you're trying to attempt with that. But we did end up using Unreal.
[00:13:42.244] Kent Bye: OK. Is this the first Unreal project that's on Apple Vision Pro?
[00:13:47.272] Sherief Fattouh: I don't know exactly. I think so, but I wouldn't be able to say exactly. You know, I've been able to see the other experiences, so there's some really fun ones, but I'm not really sure what engines they use.
[00:13:59.263] Kent Bye: Okay. Yeah, I know Alex Coulomb was trying to get it working with Unreal Engine, so it's great to see that the experience looked great. And, you know, for me, when I watch this piece, the thing that comes to mind is that it's jumping back and forth between, like, a first-person and body perspective, but also, like, a third-person omniscient perspective where you're disembodied and you're watching these 2D cinematics. And it seems like the IP of What If is very much leaning into this multiverse of possibilities and it allows you to kind of look through these broken shards of glass to have these, like what feels like you're watching a 3D movie, but through a specialized object that you're in the scene with. So it felt like the affordances of the 3D movies that are great on the Apple Vision Pro, but also these scenes where you're more a first person character. So I'd love to hear any elaboration on this affordance of jumping back and forth, this first person and third person perspective.
[00:14:53.564] Sherief Fattouh: Yeah, I mean, I'll let Indira speak to some of the specifics potentially of the 3D conversion process of like, you know, getting the linear video, like what you're seeing in the shards, like kind of traditionally built, you know, flying bar studios who actually do the animation for the show did those components. And it was awesome to be able to kind of, you know, as you mentioned, the Apple Vision Pro is really great at editing. 3D spatial videos. So we were trying to add a sense of depth. We experimented with a lot of different approaches. We bespoke all 3D inside of the shards, or what we're calling a 2 and 1 half D approach. But this kind of stereo converted one really started to resonate. We saw some of the other examples, early examples of what that looks like to create depth. It was really cool. to explore, but I'll turn it over to Indira to speak. There's a lot of really fun explorations on reflections and other things like that to try to bring it to life.
[00:15:46.485] Indira Guerrieri: Yeah, I think like Sharif says, there's a lot of experimentation in trying to marry the 3D content that we had. Movies that were made by flying bark that had to be converted into format appropriate for the Vision Pro. So there was a lot of that, plus they're on shards. So the shards have, there was a whole component of animating the rotation so that you would seamlessly move into the 3D spaces with certain tricks about how we want it. There was a lot of back and forth about how do we want to do this? How does it feel? We were talking with one of the designers today, Joe, and how do we want to make this transition last year? There was a lot of discussion about that because you want it to be organic and not jarring. We tried to, for example, I'm thinking of with the transition to court where you have Thanos holding, we tried to sort of marry the 3D content with what is happening in the footage, which I think we did a pretty huge job getting that transition worked out. Yeah.
[00:16:56.072] Kent Bye: Yeah, I think for anyone who has Apple Vision Pro, watching 3D movies ends up being one of the most compelling things. And I feel like this piece is like really leaning into that importance, but also allowing you to be embodied in the first person perspective. And when I went through it, I played through one scene like three different times. But the first time I did it, I was using the shield. And the second time I went through it, I was like, I'm going to see if I can die. And I like... Got hit a bunch of times and I didn't seem like I could die. There was no hit counter. And so, you know, as someone who's coming from a gaming background, maybe you could describe the philosophy of interaction when it comes to like, it's not quite a game because you can't really die. So you're not like playing it over, but it's still an interactive narrative. So how do you start to conceive of what is the worth of this type of interaction that if you do nothing, it'll still progress. But if you do something, then you feel even more immersed into the world.
[00:17:47.625] Sherief Fattouh: Yeah, I mean, it really is the heart of these kind of explorations that we're doing. I think, you know, the medium is so new, you know, there's always the kind of classic, you know, this is even going into traditional games, you know, especially when you're doing like immersive or cinematic type content for single player, you know, narrative based games, the balance of how much is going to be interactive, the level of agency that you have in the interactivity and the amount of how you're going to convey the story to the player in those cases what we refer to as the audience here because it is you know we're not really making a game it is a it is a new form we we'd like to think of it as you know it really is an immersive story so it is totally subjective of course you know like how much do we want we of course thought about should we allow you to quote unquote die should we ask you to restart if you don't do it And the more we thought about it, it was it was just really the thought process was we're looking at this as as if you were watching a show right on the What If series, like we really wanted to think about it as a narrative experience that we're asking you to have an agency and there are different outcomes depending on what you do. You may not have noticed, but even in that kind of Thanos moment, there's like the lines he delivers are different, whether, you know, if you actually block or if you don't block like, you know, it's just a minor thing, but his dialog will be a little different to you and we just wanted to really think about accessibility and and more of a lean back experience for really pushing the immersion and the narrative structure of the story forward and it can be kind of jarring when you have to replay a certain point or you're asking a skill based component to it there's nothing wrong with that it's it's gonna be a lot of fun there's there's challenge i'm playing final fantasy hard mode right now it's brutal But it's like, I love the challenge of trying to, you know, redo these really hard moments and it's satisfying when you actually are able to succeed. But that wasn't kind of our goal in this experience. It was really driving the story and the immersion first and foremost.
[00:19:48.840] Kent Bye: And Indira, when you think about your relationship to the interactive components when you're doing the artwork, I know that there's certainly some shields and other artwork that is very holographic that you have with both pulling in the stones, but also the shield that's in front of you. Maybe you could just talk about some of the considerations of once you start to have more in the VR space, not only the optimization, but also the look and feel of of how things look from the first-person perspective and just maybe talk about some of those specific considerations that you had to go through when you're creating the look and feel of the piece.
[00:20:22.964] Sherief Fattouh: I'm laughing a little because the sorting issues were a lot of fun trying to debug. There was a lot of the composition. We've logged them out of our mind.
[00:20:34.884] Indira Guerrieri: Yeah, our head of effects, Micah, could probably talk at length about this because there was so much interaction with elements that, you know, explosions and the bracer, the shield, all of these things need to feel like they're part of the environment. So consistent design, the glow and the fact that things can interact together, like, for example, when you put the shield and you're asked to, you know, shoot the my shield and then that does a reaction on the, it bounces off and it hits Wanda. Those kinds of things were fun to integrate. There was a lot of collaboration between the tech artists effects with the explosions so that everything felt integrated into the same project. look and feel as each other. And like Sharif mentions, there were quite a few transparency issues as there are always on mobile devices. Although much more, I think we could do a lot more once we figured it out and lots of other devices, I would think. I mean, we have a lot more flexibility and adjusting how we wanted our layers to be arranged. I'm just thinking about how all the various challenges involved in the integration. Yeah.
[00:21:58.059] Kent Bye: And one of the things that I noticed in playing through the experience was that, you know, a lot of times you'll have a moment where, okay, now it's time to do something and then it'll have something jump in very quickly. Like it's very much holding your hand to make sure that you're doing the right actions. And then other times there were moments where I was trying to do a gesture and nothing was happening. I was like, okay, am I supposed to be doing something here or not? And so I feel like there's this fundamental tension of, okay, is the viewer supposed to be doing something or not? And if they are, then if you're doing the guidance at every moment, then it just feels like you're puppeteering something that it doesn't feel like as much of an agency. It's sort of this weird tension where you could have different skill levels where you're if people are comfortable with not knowing and being willing to play around, there could be like no guidance. But sometimes I did find the guidance very helpful because when I did try to do something, it wasn't happening. So it's a tension. I think it's an inherent part of all immersive narratives, but I'd love to hear some of your process of trying to find that middle ground for what seems to be more geared towards beginners and defaulting towards the guidance rather than having people feel completely lost where they're supposed to do something, but they don't know what to do.
[00:23:11.348] Sherief Fattouh: Yeah, I mean, that was one of the key, you know, on the design standpoint. And just as we're going through the interactions, you know, it's obviously being very thoughtful. Many of us from the gaming background understand that level of tutorialization, you know, the handholding versus the agency, you know, the satisfaction I was mentioning before of like the fun of exploring the do it. I think our philosophy on this one in particular was just it was new even to us as developers of the amount of gestures that we have and unique bespoke gestures. I think there was over 10, you know, with using the infinity stones that unlike a lot of other experiences, like there probably isn't a bespoke, like Dr. Strange spell of like, okay, you know, you rotate the hand, do this. And then you clap. It's like, you kind of really do need that. You know, we created in that fiction, the ghost hands is what we call them. But that's in fiction of like the hand of Agamotto is showing you how to cast these spells. But we were very intentional of like every interactive moment in the experience. And I think at last count, there was over 48 interactive moments. We were thoughtful of like, how long do we want it to... What are we going to show you what to do? How long before we show it to you again? And then there is a concept, as you mentioned, you can't really die. So it will time out and just progress. But then how long... in each of those moments. And that was great working with our director, Dave, and just from a pacing standpoint, again, as a story being, you know, the tent pole and the narrative of, okay, at what point does it become, should we just continue to progress, give folks an opportunity to do it? Or, you know, some of the moments have very long timeouts, for example, like the time stone moment, for example, like, which is not a spoiler, but like, there's a really important choice that you have to be given at that moment. So that timeout is actually significantly longer than, for example, like you have to block a thing and you need to bring up your shield. Like we didn't feel that needed to be overly drawn out, but we want to make it the player, the audience member to feel, you know, to think about this choice that's being presented to them. You know, when Wong warns you, it's like, you shouldn't be using the stone like this, but Hela's begging you to use the stone. It's like, what are you going to do? that we don't want to rush you through. So like we let that play itself out, even at the risk that you may not know what to do exactly. Like we hopefully, you know, we show you, but some folks might miss it because there's a lot of emotional reaction happening and you have to make concessions at some point. And, you know, we always kind of deferred to, you know, Emotionality, pacing, really trying to create a human moment. And sometimes that comes a little bit at the expense of like what would be most intuitive or, you know, like, oh, it'd be more fun to explore it. It's very subjective. It's just our intent was go more on the cinematic side. It doesn't mean it's always the right one, but it is, you know, kind of what our thought process is there.
[00:25:59.011] Kent Bye: So I know previously there's been a number of different immersive VR projects that what was previously ILMLxLAB, now ILM Immersive has done on different VR projects that are released on the Oculus Quest or the MetaQuest. And now there was also previous projects that I know that have interviewed like folks working on things like cycles and other pieces that have shown at different festivals like SIGGRAPH. And so this piece, the what-if piece, is going to be released as a standalone app. And I'm curious if there's other distribution options that you're having, because not everyone has an Apple Vision Pro. Is this something that might be available for a location-based experience in the context of one of the theme parks? Or if there's other ways that you're thinking about how this is starting to feed into a broader pipeline for pushing forward what's possible with this type of virtual reality, augmented reality type of mixed reality storytelling, immersive storytelling?
[00:26:50.225] Sherief Fattouh: Yeah, I mean, it's a great question. I think right now, you know, we're definitely focused on the launch, you know, and the Apple Vision Pro, you know, the experience was designed for the ground up from it. But I think that there's always opportunities, you know, we're always looking at how can our experiences reach, you know, as many fans as possible and what are the opportunities that can be presented there. But right now we're, you know, focused on the, on the Vision Pro.
[00:27:13.564] Kent Bye: Okay. And Indira, is there any other takeaway that you have from after working on this project? Where do you want to see things go in the future?
[00:27:22.133] Indira Guerrieri: Well, I really enjoyed working with the Apple Vision Pro because of the range of colors and high dynamic range exposure, all sorts of things that I'm used to in visual effects that were like, oh, they're available to me here. I don't have to compromise my sense of taste or, well, not that other games do, but there is always a mobile gameplay that gets in the way of sometimes of things looking as beautiful as you want them to. We're trying to match. gorgeous artwork and we wanted to do it justice. And I feel that we got a lot of the way there with some compromises, like you always have to make on a mobile platform, but less than really, I thought that we would have to make. And that, to me, makes the immersive experience more enjoyable because it's something beautiful to look at. So you can stop the game and maybe you don't, you know, as you're saying, there is, no, I got killed. I got to restart. All the adrenaline you have with playing a game is not necessarily there. And so you can sort of, like you're in the void or you're in soul world and you look around and you go, oh my gosh, this is really pretty. So to me, that was really enjoyable. And I hope to be able to do more of that for sure. Yeah, I really enjoyed that.
[00:28:43.444] Kent Bye: Yeah, that's definitely one of my favorite scenes was the soul world and that whole look and feel of that space was really quite beautiful. And to me, the Apple Vision Pro is more higher fidelity than the Quest, but maybe on par or maybe sometimes lower than what you see with PC VR. And so you're still able to really push the level of fidelity. And so having, I think, more toon shaded flat look of the piece, even though there are textures and maybe there's one scene with When you're in training where it did feel like a little bit of a mixed of the aesthetics where there was a little bit more like reflections and look like more a little bit photo real and some of that with like- Oh, you mean the one of the choices?
[00:29:20.813] Indira Guerrieri: That was completely intentional. It was supposed to be like you were in a room. So that was actually a little cameo of another look. Yeah. When Miss Minutes is on the table and she's typing away, that was a slightly different look because it was meant to be more realistic. Glad you noticed.
[00:29:42.344] Kent Bye: Yeah. And I was also referring to when you're getting in the training scene. The void.
[00:29:47.548] Indira Guerrieri: Oh, the void, yes.
[00:29:48.569] Kent Bye: The void, yeah. There's some like reflections or at least the shimmering on the environment that I don't see in other places. It felt like a little bit more of a blend between like a photo reel.
[00:29:58.697] Indira Guerrieri: Yeah.
[00:29:59.198] Kent Bye: With the tune shaded aesthetic.
[00:30:01.089] Indira Guerrieri: Yeah, that was also intentional. We were trying to, well, let's see, do things feel right when they're completely toon shaded in a 3D world? And we felt, remember Sharif, we had these discussions. No, we need to add more of a feel of being in a 3D world while still respecting the toon look, which was, so it was a sort of a hybrid choice.
[00:30:24.606] Sherief Fattouh: Yeah, in that particular location, because it was original, you know, kind of all new, you know, it was also a little more exploration. Like we had seen the collector's emporium in the shows and different things. And it's like being true to like what that is, is always so important to us. But like we were able to play a little bit, just even in fiction, you know, it's like that was like a destroyed version of New York. I forgot the timeline number. I don't have it on me right now. I'm surprised it's burned in my head. But it's like, you know, that was a destroyed version of New York. And that's what's so fun with this fiction is that it can be a different look and feel. I mean, even in the multiverse of madness, right? One of my favorite scenes in the film was, you know, when Dr. Strange and America Chavez are flying through all and they're like breaking through reality. And you see like a toon shaded version and a crystal version and like a watercolor version. And, you know, it was like, just such a cool shot exploring all these different visual styles. And, you know, we obviously are focused pretty much on the toon shaded look. But even I think you called out, it's like when you see the shield, we did that intentionally of like the kind of spells are a little bit more photo real or like what you see from the cinematic thing than the toon shaded because that was with the vision pro you actually see your real hands and so that was also a conscious choice of like we're gonna make this shield look you know that hot orange sizzling look to it is more of the cinematic style but when you're in the vr world like you see a comp like that part looks that way but the environment you're in that kind of animated style like that was intentional because it's like you're coming from the real world in your living room But going into this shard, you know, it's nuanced. I'm sure most people won't pick it up, but it was thoughtful of like playing with that'll be more on the photoreal side. And as Indira said, like, you know, one of the endings, that was also a different look. And like that table and the computers, like they were kind of more on the CG trying to be, I wouldn't call it photoreal, but different, you know, more realistic look than the toon shaded. So we're always pushing that and exploring. We kind of just have fun with it, to be honest.
[00:32:24.631] Indira Guerrieri: And the Vision Pro gives you the environment to light with to a certain extent. We wanted to use that. We were in a toon world. We wanted to use that. So we wanted to try it out. And we have so many more things that we want to try out, but I guess you have to call it an end at some point.
[00:32:43.296] Kent Bye: So, yeah. Yeah, just a real technical achievement to even put it all together in a way that is very coherent and also flows as an immersive story. And yeah, it's a real, I think, pushing at the edge of what... There is an immersive storytelling that has been at the festivals for a number of years that I've been covering since 2016. And this feels like a new more of a cinematic genre that is pulling from the cinematic tradition, but also having these interactive components. And so... Yeah, I feel like it's kind of a niche that, with the Vision Pro, is starting to see these new forms that I haven't seen yet.
[00:33:16.405] Sherief Fattouh: And that's really been our goal. There are some amazing experiences, I think, in VR. And even we're starting to see some in AR, traditional genres. And I think the fun part is exploring Is there a space for, you know, kind of more of a passive narrative thing that is really more of an immersive story that could really tie directly into original series and content that really is like, we kind of looked at it as like an experiment of a new type of experience because, you know, it's hard to label just as a game or just as, you know, even in traditional VR you have. Now we're getting kind of a lexicon of established genres and types and some of them are amazing, right? And it's no knock, it's just more of, This is very much like I kind of look back on some of the days in AAA gaming and is there a place for narrative-based content? Does everything have to have a multiplayer component? Does everything have to have that? It's a very healthy debate to have, but I think You know, our philosophy is always the more diversity that you have, there's different kinds of experiences. As long as people enjoy them, I think offering that to audiences is a plus of just having diversity of content, just as anything in like the entertainment space of there's different movie genres and different games as well. And hopefully this could be, you know, continue to grow. And we see a lot of fun explorations of what these kind of experiences can be.
[00:34:37.728] Kent Bye: Yeah. And I think the other component for me is that you're building upon a set of IP and characters that have this rich history and backstory that people are familiar with and having, I guess, a narrative reward as here is how this story plays out. And I don't think that other stories necessarily were able to lean into that because it doesn't have as much meaning if you don't have that whole history of the whole Marvel cinematic universe that people can lean upon for you to do this type of story. Yeah, and I guess as we wrap up, I'd love to hear what each of you think the ultimate potential of immersive storytelling might be and what it might be able to enable.
[00:35:16.719] Indira Guerrieri: I think to me, relatively new to the medium of immersive storytelling, it's delightful to be able to not necessarily have to play a game in order to get into spaces like this. You know, a lot of times you're sort of forced into the game mode because that's all that's available out there. While here, I mean, the Vision Pro does have an aspect very much like the iPhone before it of building sort of, oh, I'm doing something while I'm experiencing this, while I'm watching a movie. I think it was Dave who was mentioning it. I thought, oh, that's a good idea. You're playing this experience and you go, oh, I forgot about this character. Let me look him up.
[00:36:00.174] Sherief Fattouh: another window there's still all that interaction that we get to do in this space is new to me certainly and i think it will be quite delightful yeah i mean for me i i really think that we're scratching the surface of this my hope is that like this experience and similar ones just kind of inspire audiences but also other developers of like what kind of new types of content can be out there that it can be things that really change the way that we consume entertainment. I think that when immersive is such a big word that can encompass so many different things, but I'm excited to see where it goes. We have ideas, we have tons of ideas of what we would do more and in different IPs, of course, and different stories that we could tell, but I'm personally excited to just see What are things that hadn't been thought about? I mean, I think even in the aspirational pieces we've seen, you know, at that Apple Vision Pro launch of like, I'm a big sports fan and I'm like, how can I change the way that I'm viewing like a live sporting event? You know, like those kind of things, you know, just really that are not necessarily traditional, but I think could end up becoming just very magical, right? Like, it feels like we're in this kind of era of magic and wonder and excitement. To me, it's just really cool to explore it and not be bound by, well, it's this formula and it has to be this and it has to be that. You know, it's kind of really opening the doors with technology to see how can I really engage in content that I love whether it's through stories, whether it's through music or, you know, other forms that are like, what can make that more immersive and feel like I'm living it? And I think that I'm actually more just as excited. I love for the audiences to enjoy this experience, but I'm really excited of like, could it spark ideas or, you know, other thoughts of everyone of what can this space do that has not been done? Because I think we're just starting as a whole in the medium. That's my hope. So...
[00:38:02.461] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:38:07.564] Sherief Fattouh: Just thank you for caring about this kind of content. Just really, I would love to invite healthy discussions and hopefully more content that is pushing boundaries, whatever form that takes. That would be exciting to see. I'm always excited to new news and other stuff is awesome to push these boundaries.
[00:38:25.655] Indira Guerrieri: Yeah, go check it out.
[00:38:31.590] Kent Bye: Awesome. Yeah. So I guess it's going to be released as a standalone free app on the Apple Vision Pro. It's going to be released as free at the beginning and then at some point that may change. But yeah, I'd highly recommend folks go check it out and get to see another, I think, a new kind of emerging genre that is really leaning heavily into the cinematic tradition and leaning into a lot of the stereoscopic 3D affordances and some switching between the third person and first person in this piece. And yeah, just excited to see where you continue to push forward with this type of immersive storytelling, embodiment and interaction and where you take it in the future. So thanks again for joining me here on the podcast.
[00:39:07.899] Indira Guerrieri: Thank you.
[00:39:08.440] Sherief Fattouh: Thank you for having us.
[00:39:10.825] Indira Guerrieri: Thank you.
[00:39:12.150] Kent Bye: So that was Sherief Fattouh. He's an executive producer at ILM Immersive, and he produced and executive produced the What If experience. And then Indira Guerrieri, she's a senior technical artist at ILM Immersive and was also the technical art director on the What If and Immersive story. So I have a number of different takeaways about this interview is that, first of all, well i think if you're looking for a game this is not a game like experience it's much more an immersive story that is geared towards having you be immersed and interact with these different gestures there are like 48 different points that he said at last count where you are actually interacting with 10 different gestures. And there is a little bit more in beginner mode. And so there's AR overlays that are helping to tutorialize throughout the entirety of the experience that help to remind you how to actually do these 10 different gestures. So it's very much geared towards the audience that may not have any type of gaming or immersive experience, but also just trying to use the affordances of the Apple Vision Pro. to transport you into these environments to allow you to have these light interaction moments that are not like really super critical like they said there's no way to die and there's maybe slight variations in the dialogue that are happening throughout the course experience that have some different changes there are a couple of choices that you're making i'm not going to really dive into that here i'll leave that for folks to really explore but there are these moments where you take these different branches to explore the different potentialities. Because like I said at the top, the What If series is really oriented around multiversal realities. And there's these multiple versions of the same character in these different parallel realities. So if something shifts and changes, then you can see how that same character would react. And so it's a really useful thing to use in terms of interactive immersive storytelling, because there's not a lot of stakes in terms of experimenting with the canonical storylines of the Marvel Cinematic Universe, because there are these many different splinters and threads of these same characters in these different situations. And yeah, like I said, the What If series really starts to explore different aspects of it. Like there's some aspects of the series that I'd recommend watching the TV series, just because there are some moments of like an absolute point in time where you kind of understand what that means in the context of the TV series, but in the context of this immersive experience, it's explained and you kind of understand it but you don't really understand the full context of why that might be a narrative plot point that's interesting to really dive into so i definitely recommend checking out the tv series and the tv series feels like it's very isolated where these characters and stories are not really tying together but in the end they do tie things together but this immersive experience is completely separate of that where they're pulling in some of these different characters like the watcher is something that's unique to the series and You know, who's the watcher or what is his powers? What are the rules of this world or what can and cannot do? So those I think are helpful to have a little bit more context and story before watching it. And yeah, like I said at the top, Alex Coulombe said that he could say with 100% certainty that this is the first Apple Vision Pro app that's going to be on the store using the Unreal Engine. And Sharif also told me in the context of this conversation that they have been making some modifications. And I asked Alex what he expects to see coming forward, if we might see an announcement at WWDC or anything. And he said that the Epic Games XR team is small. And so Alex said that he would err on the side of assuming that no major Vision Pro updates are coming soon from Epic. To have the current features working, it still requires a source build of Unreal Engine 5.4. The Epic Games launcher doesn't work. And current features include fully immersive mode, hand tracking, gesture recognition, forward and deferred rendering. And there were elements of pass-through in this experience, and so there are aspects of whatever private branch that IOM Immersive is maintaining that is doing features that are not even available on the public version. So hopefully we'll see either some contributions of some of that so that everybody has access, or I can totally understand why IOM Immersive may keep some of those features private, you know, open sourcing stuff may be difficult sometimes, but at least for now, it's theoretically possible to do some of these things, but it may not be coming for everyone until we get more insights from Epic Games. Hopefully we'll have a little bit more announcements from WWDC, but that's still yet to be seen where that goes. But yeah, overall, this experience is very much rooted in the cinematic tradition. So it's much more about you Flipping back and forth between this disembodied, omniscient perspective where you're watching these shards of glass with stereoscopic 3D movies. They call it like a 2.5D because it is just projected onto a 2D plane and it's not technically 3D because it's just using the stereoscopic effects of what you typically see in a 3D movie. And so they're blending these shards and then the shards of glass kind of flip over and you kind of go into the immersive 3D version. And so there's a lot of trying to match the aesthetics of what it looks like from the 2D animated version with stereoscopic effects with being fully immersed into these different modes with 3D. Tune shadings with a lot of flat shading, not a lot of like reflections. Although in the void space, they do have it like more of a photorealistic blending because you have more of your body in some of those different experiences and they wanted to make it so that you felt like you were having this seamless integration between the. realistic aspects of what they call the void which is your space to be able to train how to use all these different magical spells and you have the kind of more photorealistic shields when i move my hand back and forth really quickly the 30 hertz update rate for hand tracking has a little bit of a lag between these mixed reality components and so when you're like moving around quickly That kind of delay just breaks presence for me. And I think part of what they did from the design perspective was they don't actually require you to move around quickly at all. You just hold up in a fist and it has this shield that completely blocks your body. So it's again, it's another super easy mode. Hopefully this will be something that might be changed in the future with Apple to have like a higher refresh rate for hand tracking, because having just 30 Hertz really breaks presence when you're kind of moving around quickly. But like I said, this experience isn't really requiring you to move around quickly. So it's only if you're like me, an XR nerd trying to break the technology or see the limits of it. And there are a number of different gestures that are spread throughout here. And when you do do the gestures in a way that you don't have to be tutorialized, it does feel really good to actually have it work. There were a number of times when I tried to do gestures or was doing things and there wasn't these interactive components. And so it's always this sort of tricky thing where you're having agency or interactivity, but If you don't have any affordance to receive that, then you can find yourself like trying to do something and it doesn't react. And so it's very much biased towards this more passive immersive story and then only interacting when you're really asked to do something specific. But this kind of tutorialization where you have this overlay of the hands moving around, I think about this as in terms of You know, the future of these types of gesture interactions where you can have like an augmented reality overlay within the context of your world to follow it. And that type of training of the users is something that I expect to see a lot more when you have these kind of more hand gesture based interactions. incidentally the job simulator and vacation summer from alchemy labs just recently launched on apple vision pro haven't had a chance to dive in and check it out yet but that's also something where you probably have a lot more rich hand interactions that they've been innovating so alchemy labs definitely check out to see in terms of like agency and gameplay definitely see what they've been doing with their series that i believe is like built within the context of unity And I think one of the other things that is unique and different about this experience is that it is leaning upon the Marvel Cinematic Universe. And so you have all these different characters that have their quality of their being, their relationships. And so because they are leveraging this existing IP, they have more freedom to start to play around with these characters and how they're interacting with each other in a way that means a little bit more if you actually are familiar with these characters and their whole backstories. Yeah. how they've been depicted in other IP and other movies, other TV shows and whatnot. So the more that you've watched all of the Marvel Cinematic Universe, I think the more hidden Easter eggs or other aspects that you'll have more appreciation for. And as I've been using the Apple Vision Pro, the 3D movies is probably one of the most compelling things that you can see on the Apple Vision Pro, especially in the context of immersive entertainment and something that you really can't see on any other headset right now, just because of the high level of fidelity and just the support from all these big players like Disney Plus and Apple has different 3D videos. But just to use that type of affordance of a 3D movie and how good it looks already, but to put that in the context of these immersive environments and context switching between the mixed reality and virtual reality. One technical note is that anytime you go from like a mixed reality pass-through into the fully immersive VR in the context of Apple Vision Pro, then actually ask for your consent. It's just more of a warning to make sure that you're not bumping into anything and it doesn't just throw you into it and all of a sudden you are running into stuff. There's no chaperone system. in the context of Apple Vision Pro. So there's nothing that you can set up to say, okay, don't bother me with this question every time I open a new app. But there's a number of different options where you can say, don't ask me again, no, or okay. If you say okay, that means that when you close the app and open it again, it'll pop up again and it'll ask you again, given the context. If you say don't show me again, it'll basically never show it to you again. So if you click OK, there's some moments where you're going back and forth between like the fully immersive mode and the mixed reality mode and going back into the immersive mode. And then it'll pop up in the middle experience that same question again. And that's just because there's no way to bypass that. And I think there's also no way to be in fully immersive mode and to blend in some of the world around you into that experience. It's either one or the other. And that's something that Apple has decided to do. Either it's like fully immersive or you have some sort of being able to share the world around you. But it just ends up creating this friction where it's more on their side to make these different architectural decisions or to change the dialogue because it's really confusing just to have those three options of like, don't show me again, no, or okay. I just would read it quickly and just click okay just to consent, not realizing that actually the proper answer in this context is don't show me again. Otherwise, it's going to be popping up a lot throughout the course of the experience. So that's like one early thing that I'm sure all this will get sorted out. Or if not, then going between the mixed reality and VR mode will always have this kind of like annoying pop-up dialogue that you have to click through in order to get through. But overall, I feel like this is forging into like this new genre of immersive storytelling where it's really leaning upon the cinematic tradition and really providing some kind of narrative rewards that felt quite satisfying and also quite satisfying to be immersed, fully immersed into some of these different scenes and, uh, see these characters around me discussing things and you know the fact that they're really pushing forward and building something like this on an engine it's a sign of things to come to see more high fidelity experiences that other people will be able to create like i said we'll still have to wait until those are broadly available for the rest of the community but it's theoretically possible to get some of this stuff working So this is being released on Thursday, May 30th. It's going to be free for a certain amount of time. They haven't said for how long. They said in the press release that it's the first ever interactive Disney Plus original story, but it's not actually going to be available on Disney Plus. You can only get it on the Apple Vision Pro. So it's an Apple Vision Pro exclusive and it's going to be a self-contained application moving forward. And at some point they're going to be charging, but they haven't announced. when they're going to be doing that or how much it's going to cost so definitely check it out if you have access to the apple vision pro or if you know someone who has it go check it out definitely worth seeing some of their take of where they want to take this type of immersive storytelling in the future so that's all i have for today and i just wanted to thank you for listening to the voices of vr podcast and if you enjoyed the podcast and please do spread the word tell your friends and consider becoming a member of the patreon this is a this is part of podcast and so i do rely upon donations from people like yourself in order to continue to bring this coverage So you could become a member and donate today at patreon.com slash voices of VR. Thanks for listening.
