I interviewed Gargoyle Doyle director Ethan Shaftel at Venice Immersive 2023. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR podcast. It's a podcast that looks at immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my series of looking at different experiences from Venice Immersive 2023, this is episode number 24 of 35, as well as the fourth of 10 of looking at the context of ideas, adventure, So, this is a piece called Gargoyle Doyle by Yifan Sheftal, which is a character-driven buddy comedy animation piece. This is also the first episode of seven of doing a deep dive into animation. So, I'm going to be doing a little bit of a mini-series within a series, looking at some of the different grammar of immersive storytelling that's really being innovated and pioneered within the context of animation of VR. So we'll be diving into some of those ideas as well. But this piece takes place in the context of a church, outside of a church, and it's really exploring these character relationships of these two characters. Doyle, who's a gargoyle, and Chet, who's kind of like a rain gutter. It's kind of exploring different aspects of their friendship and their relationship as it unfolds over time. So lots of really interesting insights from some of the different underlying aspects of the grammar and storytelling from Ethan that he's diving into. It goes through kind of the series and evolution from a lot of the other animation pieces that he's worked on over the years as well. A lot of them have also showed up on the film festival circuit, both at Sundance and Tribeca and Venice. And yeah, the center of gravity of this piece, I think, is emotional presence. It's very much a character-driven story, very well written. One of the stronger scripts, I think, this year at Venice Immersive. And yeah, we dive into Ethan's process that he uses that it's very much a iterative process where he's doing this kind of gray boxing, these pre-visualization animatics that he's doing within the context of 360 videos that's showing in VR. Also, this piece has a lot of embodied and environmental presence. It's actually a mixed reality piece. At the beginning, you're in a MetaQuest Pro and you have a video pass-through augmented reality in the context of the MetaQuest Pro. but at some point you sit down and you watch the virtual reality experience but you're in this kind of museum context and you have this kind of mixed reality experience that gives you this sense of this environmental presence but also this embodiment as you're walking around the space and pieces also i think throughout the course of the piece exploring the context of the environment of just of this church and being able to focus on one place over many hundreds of years and the different dynamics of how those changes are feeding into the characters and the story. And there's some light interactivity that happens when you're in the museum context and being able to push buttons to essentially kind of progress the experience forward. But I think it serves the purpose of getting you more grounded into this broader environmental context. So that's what we're covering on today's episode of the Ways of the VR podcast. So this interview with Ethan happened on Sunday, September 3rd, 2023 at Venice Immersive in Venice, Italy. So with that, let's go ahead and dive right in.
[00:03:03.817] Ethan Shaftel: My name is Ethan Shaftell and I make VR narrative, primarily animation and comedy. And I've been doing that since about 2016, 2017 is when I really started to go all in on it.
[00:03:16.207] Kent Bye: If you could give a bit more context as to your background and your journey into working with VR.
[00:03:21.272] Ethan Shaftel: Yeah, you know, my background from when I was really young has always been like, I want to make movies and I made home videos and I made all sorts of, you know, cinematic shorts for really my whole childhood and then transitioned into going to film school. And so that's kind of like the language that I was speaking, but during that time, when I was a kid, I was really impacted by, like, seeing games like Myst. Because I was never a gamer, but when I saw Myst and the tonal aspects of it and the exploration of a world, it really showed me the kind of work that I wanted to do involved world building and presence in some fashion. So even at film school, even though it was sort of traditional film school education, I've always been interested in interactivity in a non-game way in narrative. So that well predates VR for me. And so I did experiments and prototypes, you know, starting in being quite young and being in film school of interactive narrative, and they never went anywhere. You know what I mean? This was before New Frontiers. There wasn't places to exhibit it. The codec would change on the computers, and then I wouldn't be able to even show what I had made. So it never kind of led to anything professionally, but I was always interested in it. But I went into traditional film and I was an editor for a long time, but always this interactive narrative and what the affordances of interactivity could do was in my kind of DNA and in my worldview. So coming out of 2016 and 2017, right in that era, to me what VR represented most was an opportunity to kind of define a medium. And the first things I did was prototype pieces that I had maybe written 10 years before, but put them in the headset. And then suddenly I wasn't getting the same types of questions and confusion about what it was supposed to be. The question of like, is it a game or is it a movie, didn't come up because it's like, oh, it's VR and VR is whatever we make. And so the whole like ability to have experiences and have narratives without the baggage of a pre-existing medium is really what I thought was exciting.
[00:05:20.656] Kent Bye: Yeah, and so maybe you could talk about what were some of the initial projects that really got you into actually learning about VR by making VR.
[00:05:29.185] Ethan Shaftel: Yeah. Initially, I was very interested in doing live action. So at that point, people were shooting 360 video. They were stitching together, building together rigs of multiple cameras. There was a lot of difficulty in shooting. There was a lot of difficulty in stitching. So there was a lot of just many obstacles to doing live action material. But what I was doing with a small team of pre-existing collaborators outside of VR was putting together prototypes and pitches for live-action pieces, but using animation as just the prototype, as the blocking, you know, the spatial exploration. So we made a piece called You Die at the End, you know, which was very cool, but it was intended to be a live-action experience, and we just couldn't get the money for it, and the obstacles were so great at that point, and we wanted to do 360 video, we wanted to do stereoscopic the obstacles were so big at that point and the cost was so great that it just was not feasible to explore that project as live-action and Then I was looking at you know, I was going to film festivals I was seeing what was out there for VR and I basically was like our animatics our prototypes We've polished them up a little bit more we can start getting these into festivals as is like we don't need to be putting everything on live-action capture and So that mindset led to Extravaganza, which was a piece of mine that was made in 2017 and got into Tribeca that year. You know, it's more than a prototype, it's more than an animatic, but that's when the affordances of animation really became clear to me. And I had had somewhat of a motion graphics background, so not really hard animation, but I was already in that world and I knew other motion graphics artists and animators. So the team could be assembled to put together VR narrative 360 video, but do it in animation format and then start to do things like visitor movement and visitor scale that is impossible in 360 video and then quickly became, you know, for me the most important ingredient in VR.
[00:07:19.915] Kent Bye: Yeah, remind me again what Extravaganza was about because I think I was at Tribeca 2017 and I'm sure I saw it but I'm just trying to remember.
[00:07:26.602] Ethan Shaftel: So Extravaganza, you know, you, the visitor, are a puppet, an embodied puppet, like a wooden puppet, as if VR was invented in this previous era where everything's like Little levers and pulleys and you get pulled down into the ground and your costume changes and there was a live-action component Which is the headset that you're in as this puppet maquette, you know with other little players playing out These little dramas is put on the face. You can see out the headset and see Paul Scheer and Josh Gemberling so two funny comedians and known TV actors and film actors are Discussing the VR prototype and then Paul Scheer puts it onto his face and then that element is live-action You know, we had his footage Molded onto a shape of his face But really it's all the animation is where the story takes place him watching you and commenting on the action Is sort of what unfolds so I was still trying to look for including a human face at that point I was really like obsessed like we need a human face in these projects, but that piece basically showed that we have a lot more flexibility and in using animation and animated assets, and we can do better stereoscopy, you can do better presence, and you can start to make directorial decisions about the scale of the visitor relative to the world and relative to the other characters or the human element. That scale can say something about the characters and can say something about the story and can help determine psychological point of view, which is where I think we start getting to cinema. It starts to be a directorial tool that I felt like was lacking you know, when it comes to, okay, let's put a 360 camera somewhere and film, oh, but we can't get too close to the wall, or we can't get too close to the ground, and then you start to stage it in a way that's dependent on the technology, and you're not able to, you know, especially at that point, you're not able to make decisions, kind of strong, opinionated decisions about why you're shooting it in this way, and what that says about the character.
[00:09:20.946] Kent Bye: Yeah, and so what was the next piece that you created after that?
[00:09:24.492] Ethan Shaftel: So out of extravaganza, I mean, going to Tribeca that year was hugely, was game-changing for me. I met, you know, everybody in the industry at that time. I met people who became collaborators, you know, for long-term collaborators. So it really was awesome. Like it really was a, I really felt like, okay, now I'm part of this medium that's evolving and I'm seeing what's being done. So that was huge. And what ultimately came from that was a couple of 360 animation projects. that were also comedy. And so that was unusual about Extravaganza 2, as I was looking around and there weren't many things that were being done that were humorous. So there weren't many things being done that were animated at least in that way. So I got approached by a company called Shadow Machine Google were partnering to do comedy 360 and so like I was on the shortlist just by default because of extravaganza And so I ended up directing two pieces for them one called kaiju confidential and one called space buddies that were really really fun to do that were supported by a decent budget and a great animation team and and I could test out my workflow that we've created for Extravaganza, and the workflow really is making these animatics, making many iterations of an in-headset animatic, tweaking and changing, you know, using placeholder voiceover for many versions, showing people, getting feedback, changing little things about scale and position and the types of little tweaks, you know, where shadows fall in the room to get attention, all the types of things that you can't do when you're shooting footage and you're locked with what you have. or that you'd be hard-pressed to plan for outside of an animatic. So it's really the testing and iteration. So that is where I really feel like I polished that workflow that, you know, I still do now. And then I guess the other aspect of that... Well, yeah, I think it was really the workflow. The idea that you make these animatics and then you remake it with high-quality assets and you re-record it with professional comedians really got locked in at that point.
[00:11:21.527] Kent Bye: I think that was Kaiju Confidential, was that at Sundance that year?
[00:11:24.761] Ethan Shaftel: Yeah, so that was 2019. Kaiju was at Sundance. Space Waves was at Tribeca. And yeah, and I always wanted to go to Sundance. It was great to be back at Tribeca again. So it really felt like I was making some headway, just developing the grammar of the type of pieces that I wanted to do. And then the other big thing that came out of those two pieces was I started to utilize a specific type of visitor movement. So it's camera movement, but in VR, you're really moving the person. that had very specific rules and structure to how I used it. At that point I was using it in 360 video and I couldn't have articulated sort of why it was working or what wasn't. I could only arrive at it through iteration. You know, it was only the animatic iteration and constant feedback on what was working and what people liked or what people didn't even notice did I arrive at what I call zips now. These kind of fast, specific movements towards or away your object of attention that if everything's working, the audience isn't really even aware of. And many of those zips also change scale of the visitor relative to the character. Kaiju Confidential used that a lot, and I've used that ever since. I mean, it's in Gargoyle Doll here.
[00:12:31.126] Kent Bye: Yeah, and there's another one that was like featuring a genie. What was that one called?
[00:12:34.238] Ethan Shaftel: Yeah, so what came out of Kaiju, really, was conversations with a company called Veer, who is a financier and distribution company. And they really wanted, they're like, OK, can we get the rights to Kaiju? I was like, well, Google has them. I don't have any control over that. And they're like, well, do you want to make another thing for us? I was like, that sounds great. But I'm really interested in sick stuff. Because one thing I realize is that the zips don't work if I don't know where people are looking. 360 video is a dumb medium from the standpoint of it unfolds in one way. You have to be planning for every eventuality of visitor attention. And what I wanted to know is, OK, a cue happens over to your left. I want to wait until the person sees. I want the punchline to happen as soon as they look to their left. I want the lightning crash out the window as soon as they look, not before, not after. And to do that, you need to be in a game engine. So I basically pitched them on, OK, I'll make you a 360 video, but we're going to do it in Unity. and we'll output the 360 video at the end, and then as a bonus, we'll have a 6DOF version." And they, you know, bought that, and that became Ajax How Powerful, which, for me, the exciting part was, okay, now I have this type of technique that is working, and I could start to articulate what was working about it, and turn it up to 11, make the scale aspect much more apparent, and make the storytelling and what it's doing for the story and the character and why I'm using it more impactful. And then couple that technique with the ability to know where people are looking, so I can wait for them, I can have things loop until they look, I can protect against them experiencing these movements incorrectly. and only with that in a game engine can I ensure that. So that was really where I feel like it came together.
[00:14:16.880] Kent Bye: Yeah, and I have a blurred spatial context for the festival that showed up because it was a lot of Museum of Other Realities during the pandemic. And I remember seeing it, but I don't remember what festival it was at. Where did it premiere?
[00:14:26.982] Ethan Shaftel: Yeah, well, that was, you know, the big disappointment of COVID among other disappointments, but really the lockdown was most sad for me because the intended, you know, rollout and launch got canceled and then canceled again and then pushed further down the road with festivals. So the initial intention, we were going to premiere at Tribeca. And so we were all watching, obviously, as Sundance then didn't cancel, but then South By did. And it was like, well, Tribeca, you know, how big a deal is this that's unfolding around us with COVID? And it turned out to be a huge deal. So Tribeca rightfully got canceled at that point. And though they did do a virtual version several months later, if you remember, they were supposed to be, you know, April 2020. At that point, that wasn't planned at cancellation. That didn't come about until a little bit later. So we ended up premiering at Venice in 2020 for their VR Expanded, I think they were calling it. which was great and it was very special what they were able to do online, but it was a real bitter pill to not be, you know, where we are now having this interview on the island and be able to exhibit for my colleagues and my peers in the way that I was, you know, imagining and hoping for. And instead it was, it was remote. I was in my Quest 2 in my office, you know, at weird times of day, but yet still managing to meet people and strike up conversations in VR chat. So it was definitely successful for what it was.
[00:15:45.136] Kent Bye: Yeah, and I know last year at Venice 2022, you were here with a prototype of Gargoyle Doyle, and now you're here with the premiere of it. So yeah, I'd love to hear about how this project came about.
[00:15:56.111] Ethan Shaftel: Yeah, so the production bridge, we were part of the production bridge and doing pitches a year ago. And that was a really great experience. So at that point, we'd already been working on it for about a year. So, you know, a year ago, 2022, I brought an animatic prototype that was pretty robust. And that we were screening and getting feedback on as part of those pitches. But yeah, I'd probably been writing for a year and a half before that. So that's two and a half years from now, maybe even three. And then making these versions of the animatic for a full year. So by the time we brought it to Venice last year, I think I'd estimated that 150 people had been in the headset to see this prototype already. And we were on draft eight or nine at that point of really significant changes narratively, blocking-wise, theme, I mean, everything. is up for grabs, you know, in this kind of iteration process. So with Gargoyle Doyle, I think that really the origin of the project was looking ahead, seeing how passthrough was working for sort of admin functions on the quest, imagining headsets that could fluidly move between VR and AR, and trying to think ahead to the type of piece that I wanted to make for that headset. And initially, like, I really failed at it because I, you know, there was a lot of buzz. I mean, there still is, but there's continual buzz about AR. And the type of pieces that I make don't seem to map onto AR very well at all initially. Like you can't change scale if your own space is being augmented with objects. You have a concrete relationship to the walls in your living room. You're not gonna shrink. You can't change position. So none of the techniques that I built up seem to work. You know, I was looking at HoloLens and like Magic Leap and saying like, Well, I'm never going to make anything cool for that. And I was trying. I was like writing ideas and it just, they fell so flat. So I finally gave myself permission. I was like, you know what? I love VR. I want to shrink. I want to grow. I'll just write something for VR. So I started working on this story called Gargoyle Doyle as a VR piece only. and giving up entirely on the idea of an AR and mixed reality piece. And it was only in, you know, after, I don't know, months of writing this and trying to think about spatially what attracted me to this idea of a gargoyle, and why I thought this was unique spatially, and what I could do with it, that I came across this idea narratively where I was like, well, we're gonna end this piece. Like, this piece has to end in a museum. Like, that's act three. The cathedral is going to be destroyed. I knew where the shape of this was going. So where does the statue wake up? What's heaven? What's the combination of his hopes and dreams, but coupled with the disappointments of his life? Well, they end up somewhere else. They're preserved in some fashion, even though they've lost the cathedral. So as soon as I realized that there was going to be a museum, and I was writing scenes like, oh, it'll be in VR, I realized, well, that's it. The museum is your home. And in fact, we'll put exposition in the museum first. So the experience of Gargoyle Doil as it exists now and from that point was, It's a mixed reality environment where you learn some backstory, you get some exposition, but it mostly just gets you into the chair. You don't want to belabor the point in this museum. You want to get into the story of Doyle. But then it sets the scene for when you come out and there's going to be surprises there. The characters will have come with you into this environment, and we can tell the last act of the character story and their reunion can be with you. And then I started to think about, OK, well, that means that your agency is different at these different layers of the experience. And what does that mean? And how do we differentiate it? Okay, well, we should be standing. You're very aware of your body when you're standing, and less so when you're seated. You watch cinema when you're seated. You play tennis when you're standing. So having these two layers and having this interplay, but really tied together and using it narratively, then I got really excited about it. And then I was like, okay, this is gonna work really well. And that led to bringing a prototype to the production bridge.
[00:19:46.111] Kent Bye: Yeah, I was really quite impressed of how well it does work and especially when you have a physical installation where you're able to very tightly control the correspondence between the blending of the virtual and the physical, having pedestals that are have nothing on them, but when you're in this augmented reality, you have this overlaying of that pedestal with a completely virtual object. And so you're able to have this persistent augmented reality context where you're going around and looking at the exhibition, pushing these little buttons for the narrative. And then when it comes down to watching the bulk of the character story, you're sitting down and watching it. I guess before we dive into the heart of the piece Do you imagine in terms of distribution that this would be in an installation? context or do you imagine that people would be seeing this in their homes because you know both the MediQuest Pro and the Apple Vision Pro there are thousands of dollars and or at least a thousand dollars for the MediQuest Pro and then $3,500 for Apple Vision Pro So there's not necessarily like consumer scale and that you know, we have the quest 3 that's coming out So that's going to be much better So you have this on the MetaQuest Pro, but potentially portable over to the Quest 3. So yeah, love to hear what you're thinking in terms of, because there's a whole other design problem of like figuring out what the space is for someone and then setting up like little guardians around what the walls are and then trying to adapt the narrative so that it fits into whatever space people might have.
[00:21:08.572] Ethan Shaftel: Yeah, it's definitely been on our minds from the beginning. I will say that thinking about and putting together this project, we learned a lot from Ajax How Powerful. Because with Ajax, we ultimately saw a return. It's a profitable project, we were able to post-festival season, see what it could do in the marketplace in China, see in LBE locations, use telecoms licensing, how that all adds up in revenue for an independent project. And so it was that process and reverse engineering a little bit, look at what type of project, how much, what's the budget for a project that's gonna come out in multiple ways and what do those channels look like and how does our one narrative fit all of them? It's been on our minds from the beginning. So I will say that the simplest version of this technologically, not necessarily in any other way, but the simplest technologically is a controlled space like this, right? There's no setup that the visitor themselves doesn't have to launch the app. They don't have to do any room setup. They don't do anything at all. They're handed a headset. They've passed over their ticket. They've handed a headset and the room just works. That's the easiest to develop in Unity. It's the most manual on our side, but the easiest for the visitor. And so I think there's a place for that, for sure. Festivals are one. And LBEs, such as they exist, it could be a good place for it. Now, how robust those locations are is an open question, but that we can imagine very easily. So there's been quite a lot of development time allocated to what's the real version that most people, whoever see this project, what are they going to see? And we have a couple good answers for that. And one is a seated-only version that can either use pass-through or not, but it's kind of an inverse footprint of what we see here. What you see here, the visitor rather, moves through this space and ends up kind of sitting in the middle, looking out around at everything, and including the central diorama. The inverted version is you're on your couch. You have a coffee table in front of you, or any kind of table, or any kind of surface, or you don't need anything at all. But in front of you, in the middle of your room, which is the open space in most people's rooms, or a table space, that becomes the central diorama. And around that is arrayed these other exhibits. and the controls for the exhibits and moving through the exposition and the narrative you see is either driven by buttons in front of you, like when you go to Versailles or something, there's a little panel with outlines of the different things, you know, oh, it's a portrait of this person or that's a cabinet from whatever. You can imagine a panel where you're prompting the narration and the exposition, or it's actually led by the guidance of the narrator on the PA system, and you're not even moving that forward. And you're still going in and out in scale into this diorama and then out to the outer space, and you still have the character resolution take place in this outer space, but even without, on a headset that doesn't have any pass-through, you're looking at a virtual museum we've designed. You know, it's just another layer of the VR experience where those scenes take place. So that conception of it and feeling like, okay, and I mean, that informed how we wrote the script to record all the other assets from our stars so that we had a really robust set of guidance and instruction for all the different types of versions of this that need different amounts of instruction or leading by somebody like the narrator. So we have that and those are developed
[00:24:30.870] Kent Bye: You know, but it remains we've seen how we exactly slot in each planned version with the distribution options that appear Yeah, it's really helpful just because you know, you're on the frontiers of this new mixed reality format And so the exhibition or the distribution of that is still like a big open question So that's really helpful to hear a bit more context on that
[00:24:49.226] Ethan Shaftel: Yeah. And I do want to mention one other thing, which is while we're here, we've gone deep into designing a better throughput version, because one of the frustrating things about our experience so far is we just can't get many people through the experience. It's like 12 people a day or something because. of how long the slots are, and it's one person in this big space, and it's six by six meters. So I think on the other side of deploying it to LBEs or even other festivals, wanting to get more people in comfortably, we've had some really clever solutions to do it with a low technology, not networked headsets, not faulty, we need robust Wi-Fi kind of situations, but clever blocking, clever staggering, so that people can be entering the shared space of the museum, and of course you're normally in a museum with other people, you know, looking at exhibits, and people are maybe already seated at locations where there's dioramas, but only in their versions, using clever layout, clever blocking, and different builds for different headsets and chairs, so that we get people, you know, someone's entering every six minutes, and there's four people in there, and we have 100 people instead of, you know, 12 in a day. So that's really what we're working on for that whole leg of distribution. Because I said, I mean, even just for festivals or for paying customers at commercial locations,
[00:26:01.168] Kent Bye: It's a really good experience to use the space and to have the bigger footprint that's gonna be different from in your living room Yeah, that makes a lot of sense It reminds me of what the Felix and Paul did with the Phi Center with the infinite having like a big giant space But they kind of like are directing people to walk through different ways and they can go up to 100 or 150 people per hour with a big giant space, but yeah certainly it's a frustration here at Venice when the tickets are only 12 per day and it's very hot tickets so. But yeah, let's talk a little bit about the actual characters and experience of Gargoyle Doyle. I really appreciated this experience and I really loved Ajax, All Powerful as well and so, and I've seen all the other pieces and I think each, like you said, the comedy genre within VR, there's not a lot of them that are out there and I've had a chance to to see your series of different experiences and they're really best of class in terms of pushing forward what's possible in comedy and genre. But I feel like this piece is, you know, it has comedic elements, but it really actually feels more of a character-driven experience where you have the relationship of these two primary characters. And so, yeah, maybe could give a bit more context of who we're going to be joined with here on the experience of Gargoyle Doyle.
[00:27:07.027] Ethan Shaftel: Yeah, for sure. You know, coming out of the experience of writing Ajax and seeing it and I was so happy with the script and it felt tight and it felt purposeful and it felt like that was something that was, you know, spatially relevant and there was a reason for it to be VR. You know, it was a win in my mind for the script. And then moving into, okay, what are we gonna write next? What's the next experience? And what are my sort of aspirations in the medium? It felt like there was an opportunity to say more, you know, and to say more about about life and about consciousness and I've been thinking a lot about finitude just in general and our finite time on earth and then how determined our lives are no matter you could be the most fearless explorer and meet every but talk to everybody at every party and go every country you possibly could even with that adventurous mindset You're going to just scratch the surface of the possible people you might have met. And so many of us are not that fearless. There's conversations we never struck up. And we feel grateful for the ones that we did that led to a friendship or led to a relationship, because it almost didn't happen. What if I didn't say hello that one time? So that was really on my mind. And I had this title. I had this title that was Gargoyle Doyle. just because it rhymes, you know, and who this character was. And the real, like, thing that struck out is that we've seen gargoyles in popular culture, we've seen animated pieces, but mostly they're, you know, walking around, they're flying around, they're not limited, you know, but real grotesques and gargoyles are Cut off at the waist, you know that they're stuck on a wall I mean, of course, they're not conscious but what if they were then that seems such like an exaggeration of what I was thinking about anyways of this like limited, you know being stuck or just like Determined by your location or the people that happen to be near you or the ones that you end up spending your whole life and like all the value of your life comes from you know, the people in the town where you were born or that the job you decided to take. It's just, what a great exaggeration of that to have some gargoyle on a wall and his whole life is the one guy next to him, you know, or the couple of other gargoyles and the street across from them that they're watching for their whole existence. So that was really the root of the character of Doyle. And then, you know, when you think about heightening your writing and you have an idea, you're trying to fit it into something that means something. It's kind of like, okay, well, He shouldn't want to be there. So something must have happened. So what are the options? Well, he was supposed to be somewhere important and he ends up somewhere that to him feels ignored. No one can see him. It's not his purpose. And then, you know, what kind of opposite character should he interact with? someone who is very happy wherever he is, adaptable in any circumstance, always finding the bright side about, well, if, you know, if that's what I am, I'm going to be the best possible one. So that kind of buddy comedy is what drives it all. And then everything else was sort of an excuse to take that, you know, any plot point that happens was just to ratchet up the tension between the two of them. Chet and Doyle so you know Doyle is this gargoyle who believes he had a different life he believes he ought to have a different life than he has and he was going to have a different life than he has and he's a failure because of that and Chet is you know by nature of just the demands of the narrative someone who you know I had to find a character that would be not what he was supposed to be but but living in the moment to the extent possible every second. And then what ultimately, I have to say that one of the things that was great about coming out of the Venice Production Bridge last year was not only did we get partnerships and our additional financing, I just got some really good editorial feedback from multiple sources, Arte, different people who are just used to giving. good criticism on films and read a lot of scripts and you know see a lot of movies or see a lot of VR and I was really pushed and inspired to take a harder look at what I think is like the third character in the movie which is like the humans the town around this cathedral and especially what opinions Doyle and Chet have about people that would be opposite. And that's, you know, a lot of the writing and rerunning that's happened for the last year has gone into, okay, we've got the buddy comedy working, and we've got these characters broadly working, but what do they think of the history of 800 years from their perspective? And trying to make that both valid, you know, have their opinions be, yeah, I could see imagining that. And so Doyle looks at humanity and sees, just a string of failures, just misery. They're always doing something wrong and he's not wrong about it necessarily. And Chet sees the same string of failures and maybe stupidity or closed-mindedness and he sees resilience. He sees like, but you know, they're just Trying always there, you know, he sees resilience. He sees optimism and that's their takeaways and then both of them have it part right and Doyle his character journey is First being sort of pulled out of his trauma by Chet takes 800 years But Chet pulls him out and he starts to enjoy the moment, you know, and then when even Chet's advice fails him, how do you approach the future? How do you approach uncertainty? And Chet doesn't have an answer for that. His answer is live in the moment. Then it's humanity that teaches him the lesson of sort of what to do next.
[00:32:19.567] Kent Bye: Beautiful. Well, that's a great summary of the arc of the story. And I guess a question around the rewrites that happened over the last year. When you were here last year at the Venice Production Bridge, were you showing people the full script as a demo, as animatics, like the full arc of the story? Or was it just a section of it?
[00:32:36.660] Ethan Shaftel: Yeah, so it was the full thing. I can't remember how long it was. It was shorter than this version, but it was slow. Like people said it was too long. I mean, frequently the note, it's too long actually means it's paced poorly. So one nice thing that's come out of it is we have a longer version, but people don't complain about that anymore because now it's actually kind of working. But yeah, so that was a full animatic. And the animatic, the VR scenes were 360 video. And the AR scenes were the same code that we're using. There's actually no difference between our prototype and the final. It's that we just keep adding to it and refining it, and fixing lots of problems, of course. the same basic code base we'd already created. It's just that when you went into VR, we were triggering a 360 video, and that's purely for workflow reasons. That's because my animatic process is very traditional animation. You get still frames, which could be a storyboard or it could be a render, so I was getting 360 renders from an animator, and then I'm editing them to audio. Placeholder, sound effects, my own voice and my co-writer's voice are the only two voices in the prototype up until we recorded Jason Isaacson Haley Joel Osment and Tania Miller, it was still our voices only. But yeah, for me, the animatic has to be complete, because you can't have any hand-waving and say like, oh, it's going to be better here. You have to go all in as a creator and be like, I want this to make someone laugh. I want this to make somebody cry. I want them to be amazed by this. And so you're working as hard as you can, as if it's the final version, even though it's not, and kind of not taking the out that there's something unfinished about it. even though, of course, it's unfinished and it's placeholder, I'm still just trying to get the laugh or trying to get the emotion. So playing it again and again with people watching people in the headset, talking to them afterwards, you know, it's hard emotionally because it's the worst version it's ever going to be. Even when it's the perfect script, it's much worse when it's me instead of Jason Isaacs. You know, it's much worse when it's no facial expressions. That was a version where the placeholder assets don't move. I mean, there's no animation, there's camera movement, there's visitor motion. And there's hard cuts to changes and assets, but there was no design. There's no artwork. It's mostly a radio play with VR visuals.
[00:34:40.739] Kent Bye: Gray boxing.
[00:34:41.540] Ethan Shaftel: Yeah, exactly. It's yeah, it's exactly a gray box, but it's one that I still hope makes you laugh. But that's such a hard fought laugh that then you know, okay, great. I can actually take this script and record it and it's going to work.
[00:34:53.030] Kent Bye: So as you were refining it from last year's version to this year's version, was there a lot of the arc of the broader context of the cathedral? Because there's a lot of like change in the environment over those 800 years around the humans. Was that an additional thing of how to figure out how to bring in more aspects of the humans?
[00:35:10.985] Ethan Shaftel: Yeah, we always had this section, this sort of sequence that I call the history montage, where they go through a couple times in history with Chet trying to prove his point and Doyle responding with very good, you know, critique of why he doesn't think that proves the point. We always had this notion, because part of the spatial concept was, well, if they can't move, we're going to change the world around them in time. We're going to use time as our vector here, and they're going to have shared flashbacks, which is something you sort of see in cartoons. You know, a thought bubble appears, and both Tom and Jerry are thinking about the same thing. It was like, wouldn't it be cool if our flashbacks are everything around the cathedral changes, but them? And then they're talking live in the flashback. They're not remembering them speaking then. They're remembering what's on the street. They're in the present. And I just hadn't seen that sort of spatially done. And I had an intuition that it was going to work pretty seamlessly, even though it was sort of awkward to describe. So we had that section, but it was different, more generic, cheaper gags. It was more like, OK, what's something funny that happens in the Middle Ages? And it didn't add up to much. And honestly, that's what people suggested cutting, because it did drag. That was the part that didn't move the characters forward, and we were treading water, which was very good feedback. But I'm glad we arrived at where we did, which was maybe it can be one of the best parts of the movie instead of one of the worst parts of the movie. And we get there not only by making it more entertaining, but to get into it for a reason and be exploring something. So we have the curiosity already, but then more importantly, pay that off. Why are we bothering to hear their opinions about humanity? Well, because it has a narrative reason in Act 3. And so that was a lot of versions, it was a lot of writing, and it was a lot of differences. I mean, I went back to history books and reading Noah Yuval Harari and, you know, these pieces about the broad strokes of human history to try to figure out, okay, what might I think if I was a gargoyle and saw this happen, sort of broadly European history, and then come up with two real strong points of view about it.
[00:37:10.170] Kent Bye: Yeah, it seems like the big heart of this experience is this dialectic between these opposite temperaments of one more melancholic and pessimistic one more sanguine or just happy jovial perspectives and and yeah, just also the perspectives on their outlook on life, optimism versus pessimism. And so there's a lot of ways that you can play with that in terms of the drama of the characters, but also the comedic elements. And so love to hear how you conceive of this dialectic of these two polar opposites and how you start to play with that both in creating like a dramatic arc of the characters, but also to play with comedically.
[00:37:47.302] Ethan Shaftel: Yeah, I think, you know, it's one of the nice things about feeling like you can make many iterations is that it takes a little bit of the pressure off from trying to justify all choices because the justification can be, you know, let's try it and then reverse engineer what's working based on reaction. So it was a little bit of a guess and check or a, you know, write a version that seems funny, has an opinion and then share and get some more people in the headset and come over to my house and I got, Referrals from writers friends and you know people that hadn't seen it or had never seen a version to come over and watch it For me also the real danger was okay, even if I get excited about my sort of point of view on this This isn't a video essay. This isn't a paper that I'm writing with this like this is what I believe about history It's just one portion of this movie. That's actually about change, it's about characters dealing with change, and how to deal with change, and what happens when you feel broken, because, I mean, being broken is just change that you don't want, you know, it's something that you regret, and come back from that. So, it was a little bit of a balance, and I got, I mean, and again, at every stage, even when we recorded scripts, I had alternate lines, even from our lead actors, and was putting together radio play versions, audio-only versions, and sharing it with 10, 12 people, and still up to the very end, I mean, I don't know, I think it's some people's least favorite part of the movie still. I happen to think that it's starting to work, but we're taking a look at it even from the feedback here of like, okay, do we need to tighten? Do we need to explore? Do we need to expand? It's not always cutting that's the right choice. But yeah, keeping it in mind that actually it plays a narrative role that is specific and you have to be able to support that narrative role sort of more than anything else about the sequence in and of itself.
[00:39:37.891] Kent Bye: Yeah, a technical question around the zip cuts where you had mentioned in the Ajax All-Powerful that looking at a specific place to then queue, in this piece that's showing here with Gargoyle Doyle at Venice Immersive 2023, are you having, like, you have to look at a specific location and then a zip cut happens, or is it more of an automated thing?
[00:39:57.840] Ethan Shaftel: Yeah, so that's exactly it. So at different times, and it always precedes a visitor movement, you know, a camera movement that is going to reframe and sometimes resize the visitor. There's a target and that target is on what either already is or will be your object of attention. So I think of it primarily as a collaboration. It's like camera movement in a film, but it's a collaboration between both the director and the visitor themselves. The visitor is responsible for pans. So I am able to look anywhere I want, follow any audio or visual cue. Someone looks over my shoulder and I turn around to see what they're looking at. That's my responsibility in the headset. And I'm never going to have something imposed on me. I'm never going to have to move my head faster or slower. I'm never going to have the world change orientation on me in any axis. I can be totally secure in the kind of orientation of the universe as a whole. But as soon as I, you know, there's a moment in Gargoyle Duel where Chet's trying to pretend that they have a high-status spot and he says, you know, the sunrise over the majestic, and he kind of looks off and she says, sheep, and Doyle looks and Chet looks off to the side. And it's not at all where you're looking right then. The visitor turns their head. More than 90 degrees so kind of a large movement in the headset to see that sure enough and and of course at that point You know hard left in the earphones. You're hearing the sheep sounds from the street below so Everyone reacts in the same way, you know, the characters looking this way the sheep are sounding from the street You're in a spinny chair. Anyways, you turn over your left and as soon as your eyes hit the sheep And it doesn't matter how fast or slow you make that head movement. And if you don't look for a few seconds, the sheep just keep buying down there and Doyle and Chet keep blinking like nothing pauses, but they're waiting for you. As soon as you see the sheep, you move back like the camera, which is you. moves into an over-the-shoulder shot of Doyle. And we've just ended one visual sequence with its own kind of logic, and we're starting a new one. Now we have this other view. We're now looking at Doyle in the foreground and Chet over his shoulder, and we're set up for the next sequence, and you've changed size slightly. You've shrunk. You used to be Doyle-sized, you know, and it's at a subconscious level, but you're roughly the size of Doyle during his complaints, and then you become roughly the size of Chet now that you're looking over Doyle's shoulder at this vista, which is Disappointing to Doyle but Chet's trying to make the most of it and then the movie continues and you're unaware that there was any weight delay It's not about your agency at all. You're not making choices. You're not controlling the narrative in any way But the narrative is adapting itself even in a very small way just to your pace in your movement and so that's what we did in Ajax a ton and it's what we were doing but without the interactive gaze trigger and in Kaiju and Space Buddies. So there was the movement, but I couldn't wait for the audience the way that with the Sixth Off projects that I've done, I can. And that's really the building block of, for me, of this language. And it does a couple things that helps make the immersive piece more cinematic in like the true way. Cinematic in the way like as cinema operates, which One, it divorces you from your position in space in any kind of real serious way. Because of course in VR we have a concrete relationship to the room around us, but you're moving and changing so frequently that that starts to be less important than Doyle's point of view. So it imposes a little bit of directorial and pace control, like an editor in film, allows you to put emphasis on certain things and change the scene and refresh the point of view regularly, you know, for narrative reasons, opening up the next scene and have a sense of movement and cinematic movement and progression through a scene.
[00:43:42.615] Kent Bye: Yeah, and so earlier you had said that, you know, moving from AR to VR, and that you said something along the lines of, like, oh, then you can, like, go into the VR mode in a 360 video. When you said the 360 video, did you mean that there's a literal 360 video stereoscopic render, or did you mean that, I'm just trying to figure out, because it felt like it was fixed off, but I wasn't testing it per se, but what did you mean by that sort of video frame, if it was 360 video or more of the VR mode?
[00:44:09.457] Ethan Shaftel: May have been talking about the prototype because in the prototype for just workflow and efficiency and speed of iteration Our VR scenes were all 360 video and this allows the animatic process to happen more fluidly This piece here that you saw it's six off from beginning to end and that was the intention Okay.
[00:44:27.055] Kent Bye: Okay. So you were talking about the prototype in that thing?
[00:44:29.278] Ethan Shaftel: Yeah, I utilize 360 video in the development process because then I can work as an editor I can sit in Premiere with frames of animation and tweak and change and write a line and then record it in my closet and then put a new joke in and then change the timing and Which is the animatic workflow, you know editing together storyboards, but with time once you start adding time You know, the difference between a storyboard and animatic is time and sound. And as soon as you're working in time, you have pace, you're making a movie. As soon as it's time's involved, you're making a movie instead of a description of a movie. And so I try to live as much as possible in an editorial environment, like cutting animatics, in the writing and development process as possible.
[00:45:11.300] Kent Bye: Okay, okay. So that makes sense. So then you just render out a 360 video and then you're able to, as people are watching it, get the sense of it. But it all has the same timing that once you get the final version, then you've got a little bit of a template. With film, you have storyboards and visualization, but this is just sort of like another previs technique for VR.
[00:45:29.229] Ethan Shaftel: Yeah, exactly. It's just a previs that I feel like pays some good dividends because it lowers the obstacles to iteration. And then once we're starting to work, so we already have a project in, like, when we're rendering out those frames, we already have a nascent full 6DOF project in development, but we're not going through the iteration process of like, okay, here's now an audio track, now animator, retime everything and all movement to my new edit of the dialogue. That friction would mean, well, let's just do four versions. Without that friction, we do 65 versions. And then what comes out at the end is that much better.
[00:46:03.063] Kent Bye: Gotcha. Great. And yeah, what's next for Gargoyle Doyle then?
[00:46:07.808] Ethan Shaftel: Yeah, well, I think there's a lot of interesting things. You know, we already talked about the different types of distribution channels and how that played a role in our planning. So we're at the point now where, you know, we made it to the premiere. We got the version that's screened here. We want to take the animation further. We want to react just like any, you know, all the movies playing at this festival. Maybe not all of them, but there's movies and VR where little tweaks and cuts and changes of the plan, you know, are going to be made after the festival and the same with us. So there's more animation to do. There's some sequences that I'm interested in doing that were more visual and not storytelling that we sort of cut for time. You know, we had these time-lapse sequences that we were working on where it's not just a flash into the flashback. It's the world is being, you know, it looks like time-lapse photography. The world's being constructed and destructed around them. So those are pieces that we're going to do. And then we have all the development needed to actually deploy the version for the home. you know, the menu system which doesn't exist right now where you're making the choices about and setting the room, you know, so that plus the development of the multi-person version. So there's plenty more to do that's, you know, already planned. So that's, yeah, that's in broad strokes what's next for Gargoyle Doyle. And I do want to just bring back to one thing because I want to be able to mention my partners and the people on this journey with me just about how Like it's really special being here at Venice this year. And it was really important to me to be here and screen it here and to share with the audience after coming from the disappointment of the COVID years. And then especially after being in the production bridge last year, the production bridge really did change like the vector of the project creatively, but also business-wise. We picked up several new co-producers and partners at that time, which is who I want to mention. Amalek's Film came on board, which is an Austrian company. We got art residency from Spheric Art Residency in Tulum, and Roth Productions supported it. And, you know, there's a lot that happened for us that wouldn't have happened without being in the Production Bridge. So I also just want to thank Lovettus Production Bridge for that opportunity.
[00:48:09.760] Kent Bye: Yeah, it's really a great opportunity for a lot of people to, you know, being here on the VR island, it can be difficult to get to, but once you're here, you have all the films and experiences, but also a lot of funders and folks that are invited to come pitch different projects, 14 projects that are selected to pitch, but also a lot of other projects that are coming up. So, yeah, just a real great vibe for the XR industry in general. And yeah, I guess as we're starting to wrap up, I'm curious what you think the ultimate potential of virtual reality, augmented reality, mixed reality, and immersive storytelling might be, and what it might be able to enable.
[00:48:43.770] Ethan Shaftel: Yeah. I don't know how well I can predict the future, but I do know that there's so many things that VR does well out of the box. Things with interaction, things where you play a central role in your movement, whether it's a game or whether it's an active exploration. the experiences that put you at the center of it and your presence and your immersion, that's what impressed us all and why most of us fell in love with VR as soon as we put on a headset. And I have no problem with any of those things. I think where my interests lie tend to be in the stories where it's not about me, it's really about the protagonist and that's the cinema experience. It's the novel experience. So where I see VR moving is I, you know, the idea that you can put something on your face or, you know, in some way replace your location, your sight, your sound completely, you're no longer in one room, you're in another, or you're bringing new things and people into your room that weren't there before, like that's not going away. And so, you know, the technology and the hardware is going to Determine how we use it, but it's you know here to stay But then as a part of it what I try to like keep in mind is I think that our experience of third-person Protagonist based stories where it doesn't matter my background and it doesn't matter my choices and it's not about my agency it's about that unique character and what they went through and the emotional catharsis I get when their triumphs or tribulations happen on the screen or on the page, that's something that's really important and obviously important. And I think that that type of storytelling is going to play a role in this medium no matter what, even with the stuff that's a little more obviously cool about VR.
[00:50:24.483] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:50:29.506] Ethan Shaftel: No, just that I'm just happy to be here and just really excited about the future.
[00:50:34.675] Kent Bye: Awesome. Well, Ethan, really enjoyed the Gargoyle Doyle. It's one of the few character-driven experiences. Sometimes you don't see a lot of VR with a lot of characters, especially as so engaged in the two interactions of a character. And I think this is a great exploration of that as a full arc of the story that will... Save for people to see the full story for themselves. But yeah, also the use of augmented reality the AR mixed reality stuff I think worked actually quite well as well in terms of like give me the sense of this space into how How that fed into the story as well that it wasn't just a gimmick But it was really actually serving a larger purpose for the story that you're telling and yeah Just really enjoyed hearing a little bit more about your journey and process of creating it And yeah, thanks again for for joining me here to help break it all down. So, thank you.
[00:51:18.169] Ethan Shaftel: Thank you so much
[00:51:19.537] Kent Bye: Thanks for listening to this interview from Fitness Immersive 2023. You can go check out the Critics Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot of digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
