I interviewed The Tent creator Rory Mitchell remotely ahead of the SXSW XR Experience 2024. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So I'm about to dive into a 20 episode series, diving into a number of different immersive stories that were being shown at South by Southwest 2024. I didn't have a chance to attend SXSW physically this year, but I was able to see around two-thirds of all the different programs and then do around 18 interviews with creators, and I'll be also featuring some producers that I've done previous interviews with as well in the course of this series. So I want to start off with a piece called The Tent, which is an iPad augmented reality piece that explores the bounds of photogrammetry to tell a modern-day fairy tale about the moral and political implications of the unhoused crisis. It's mostly theatrically staged, but in the opening there's a cinematically inspired volumetric tracking shot that's pretty epic and spectacular within the context of an iPad. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Rory happened on Friday, March 1st, 2024. So with that, let's go ahead and dive right in.
[00:01:22.738] Rory Mitchell: My name is Rory Mitchell. I'm an XR director and producer. I started making VR with 360 video about 10 years ago. And for the last few years, I've been focused on using volumetric video and photogrammetry to tell mixed reality stories really centered on actor-driven humanist content.
[00:01:49.426] Kent Bye: Okay. Maybe you could give a bit more context as to your background and your journey into this space.
[00:01:54.268] Rory Mitchell: Yes. My background is as a theater director and documentary filmmaker. And I've been really fascinated with VR since I was a kid in the first coming of VR. Back in the nineties, I remember going in and experiencing Dactyl Nightmare at the Sherman Oaks Galleria. And the headset was very heavy. But then in 2014, a few friends of mine from the black box theater and documentary world, Christina Heller and Ian Forrester out here, had started making VR with some taped together GoPros. And they invited me over and they put me in a headset and that was my first taste of modern VR. And because I had had enough filmmaking experience that I was like, oh, I think I could figure out how to slap some GoPros together and start making some stuff, really to bring like the documentary practice and theater practice into a headset. That was really what we were focused on. So I got together with a few buddies, Dan Lichtblau and Josh Polin, and we started our first VR company, Beard and Glasses VR. And we just started making as much 360 video as we could. And yeah, it's still so fascinated by the medium.
[00:03:10.422] Kent Bye: Yeah, so maybe you could just run through some of the previous projects that you've had a chance to make in the context of 360 video.
[00:03:16.484] Rory Mitchell: Yeah. One of the first projects we did was called The Visigoths, which was written by my friend Martha Marion, who's also written The Tent that is going to be at South By. And The Visigoths came from an original idea I'd been kind of playing around with that we've all seen a lot of, but the idea of a young couple documenting their lives with a 360 video camera. and what happens down the line when they're trapped in their memories in a VR headset after the breakup. So it really put you, it was an intimate romantic drama that put you steps away from a young couple falling in and out of love. And we worked on it, rehearsed it like an immersive theater show. and worked on the script with the actors and kept developing it and then shot the whole thing. And we were just so fascinated with the idea of being able to turn invisible and step into the lives of others to witness their intimacies and put yourself feet away from them. But the entire time we were working on that, like that one has never been distributed. Only handfuls of folks have seen it because it was strong sexual content and nudity. And as you know, there's no age lock that allows for more adults. You know, it's a real, we were trying to capture the feeling of being in a relationship with someone. And you're the third wheel, the ghost in that relationship with them. And so there's inevitably a lot of nudity in a relationship. So that's never really been seen by anybody. And then after that, we spent three years making a documentary about Defend Boyle Heights. which was a group of radical anti-gentrification protesters who were using Black Bloc style tactics to protest art galleries and coffee shops that were opening up. in the predominantly Hispanic area of Boyle Heights. That was really powerful because it allowed us to place you on the front lines of the conflict at the protests between the so-called gentrifiers and the protesters, and then also to put you inside the art galleries and in the homes of the activists. And there was a story that's told by the environments of each of those people that goes beyond anything else. So 360 video in that case, when you're sitting in a three-story white-walled art gallery converted out of a warehouse, and then you're in the humble, you know, small studio apartment of the activists, you know, there was a story that came through that belied whatever anybody had to say in the actual belied words.
[00:06:06.971] Kent Bye: Right. And it looks like that you started the Mercantile agency like in 2020, and maybe you can just talk about that pivot and get into more photogrammetry or when that came onto your radar and production pipeline to start to do this type of both spatial capture of environments with photogrammetry, but also volumetric capture.
[00:06:27.089] Rory Mitchell: Yeah, absolutely. The pivot to the Mercantile was, I'd done a lot of nonprofit advocacy work. and pivoting to the mercantile was moving into more of marketing work, marketing content, marketing videos. I got married, I had a baby. On the first day of her life, I used Scaniverse and I scanned her and I just kind of texted her to people and was like, here, drop this baby on your table. And it was so much fun. And people were so amazed to get my little one-day-old baby on their tabletop. And I was like, ah, I wonder if I could kind of get animator so it looks like she's breathing a little bit or doing a little bit. something more, and then I realized that I had to finally take the plunge into volumetric video. I mean, you know, I'd been aware of a lot of wonderful projects that people had done with volumetric video early on. You know, I still preferred the fidelity of 360 video, but, you know, I knew that there was stuff that we were going to be able to do with volumetric video that I really just wanted to figure out. You know, we really just, since we started Making VR, I always dreamed of being able to crack the shell of 360 video and be able to get as close to the actors as possible and to really put the viewer in that place between two actors really doing their thing, right? Like the feeling of being with incredible actors and what the invisible thing that's transferred between human beings was what finally got me excited. So we just really started like cobbling together a volumetric video system using Intel RealSense cameras and that didn't work very well. You know, you couldn't get Azure Connects for a long time and then I finally managed to get a hold of some Azure Connects and found a company called Soar that was kind of making something similar to DepthKits work using Azure Connects and a server and working with my friend and DP Sebastian Amelin who's really just shot a ton of 360 video over years and he's been around like so he and I built our little indie studio here in downtown Los Angeles in the fashion district and just started trying to bring people in using Arcturus's post-production pipeline to kind of clean the holograms and then get them ready for playback. Oh, and then the photogrammetry, like it realized like early on the photogrammetry was super cool. And even if volumetric video could be a little hinky at times, photogrammetry I knew was, you know, was just gorgeous. And so a little less of a barrier of entry to the photogrammetry stuff. So I was able to start just playing around with photogrammetry, capture of environments, capture of objects. And we ended up doing just a ton of photogrammetry for this project. Sebastian and I were outside Martha's house. We'd just get there at like five o'clock in the morning and start shooting for a little while. you know, before the sun would come up. And we ended up going back to the house four or five times because we would, we scanned the house, we scanned the interior, and then we scanned the outside. And I flew the drone all around it. And then, you know, you get like a little bit of the houses next door and we're like, oh, it'd be really cool if for like an opening shot, we could kind of float down the street and then find our lead on the front porch. And we're like, okay, well, we'll go back. And then we like, so then we like scanned a little bit more of the neighborhood. And then we were like, ah, well, we're kind of getting just a little bit of that really interesting KFC on the corner over here. Maybe we should go back and just, okay, just one more time. So then we went back and we got, then we're like, okay, now we're like, eventually we ended up with like six square blocks worth of East Hollywood that we scanned after four or five trips so that we could have our kind of opening cinematic shot floating down Western Boulevard. in East Hollywood. And I think it's kind of fascinating because we're a US-UK co-production with my friend, the producer Lou Doi, who's just incredible, gifted dancer and XR creator out of London. And, you know, she's brought a team with her, including our narrator, Greg Moss, who used to be an actor and is now a writer over there. And he lends his English narration to the story. But they were really fascinated with this slice of Los Angeles that no one really expects to see in East Hollywood, that I think that everyone, you know, just associates the classic glitz and glam of L.A., and so to kind of be in a more working-class neighborhood in Los Angeles, off the beaten path, and to have a story told about that neighborhood, I think, was at least very interesting for our team in the U.K. to experience it.
[00:11:33.347] Kent Bye: Yeah, well, that helps set a lot of the context for the tent. And there's certainly a lot of really amazing photogrammetry with that establishing shot like you were talking about and quite a bit of volumetric capture as well. But maybe we could take a step back and you could describe to me, like, where did the idea for the tent start? Was it coming from wanting to tell the story or was it also like having the technology and starting with, you know, you knew you wanted to do something with the photogrammetry. Yeah, maybe just go back to the inciting moment for The Tent and how this project came about.
[00:12:06.798] Rory Mitchell: Yeah, absolutely. So Martha Marion, the writer of The Visigoths and The Tent, you know, during the former administration, Martha, I kind of describe Martha as a light worker, really. You know, she's an immersive theater artist and a poet. And she's studying social work now, but she is really a light worker, like she brings light to people. And so she was getting very active in immigrants' rights during the former administration. And she had also had a number of experiences, direct experiences with the unhoused community in her neighborhood and attempting to bring them in to help them, to invite them in to cook dinner, invite them in to play the piano and sing and to try helping people. And she was interrogating her own activism and the complications that come when we make a decision to try and help people and the interpersonal social dynamics of activism and really just how complicated it is. And so Martha had been thinking about that, and she started writing a script, and she pitched it to me, and I immediately loved it. And we were originally gonna make it as a 360 video, and then that wasn't cool anymore, so we were just gonna make it as a short film, a traditional framed short film, just so more people could see it. And then we put it down at the beginning of the pandemic because it was a little too close to home, and then a little way through, I was starting to build the volumetric studio, and I said, oh, I think because this story is really just one woman and a voice in a tent, the technical limitations of the volumetric capture studio that we have would allow us to tell this story volumetrically, so let's see if we can do that.
[00:14:18.992] Kent Bye: Yeah, and it's delivered on iPad Pro that I had a chance to see it. I guess it's also compatible with iPhone, but I watched it in the comforts of my home since I'm not going to be at South by Southwest this year. So I had a chance to check it out. And because I'm holding this iPad Pro, first of all, I had to find a table to kind of set the context and then. I found myself like holding the iPad and then almost becoming the cinematographer where I'm getting the different shots and I'm able to like move around but I'm also holding like an iPad and so it's like very heavy so I ended up like a lot of times just resting my hands on the table because it felt like there wasn't lot of dynamic action that was happening that I needed to be agile to pan around and to look to see where the action was and so in terms of the dynamic motion of the piece it was pretty static and you know there was moments where I was able to kind of move around and figure out where I really wanted to watch the piece but because it was a little bit more distant from what I would say would normally be like a immersive 360 video where I might be inside and be completely surrounded and the location where you might put me might put me a little bit closer to the actor and I might be able to see a little bit more of The resolution of the fly metric capture. I felt like it was fairly low res in terms of the fidelity But it didn't need to be higher res and I felt like it was the appropriate resolution for the context of watching it on an iPad screen where the actor is actually like fairly far away a lot of times because I I didn't know what was going to happen in the tent and so I was often having the tent in the foreground and it didn't give me much benefit to get too close to the actor and so I felt like I was more of a medium to long shots a lot of the times and maybe some close-ups but because it was fairly low resolution I didn't find myself wanting to get too close and I didn't want to also miss anything. And so, yeah, I'd love to hear the process of walking through like the use of the spatial medium and turning the audience into one of the crew of the center photographer rather than just watching it on a 2D screen. I'm actually kind of moving my body around to get aspects of the story. So I'd love to hear you elaborate on how you start to think about that.
[00:16:28.322] Rory Mitchell: Yeah, absolutely. I mean, I really wish I could have seen you watching it. Like, it gives me, like, great joy to watch people interact with this thing. You know, I think with a lot of folks who don't, like, you know, you are one of the most well-versed XR people around. And so you knew what you could do with it from the jump. It's interesting watching both like lay people as well as even some VR folks start to watch it and they don't move at first. And then at some point their arms start to get a little tired and their arm moves or they readjust and they see that the whole world moves with it. And you see them discover that it's like, oh wait, this is a fully 3D thing that I can move around in. That there is almost an element of teaching people how to view this as it's happening. There were a lot of choices that we made throughout the process that we discovered, you know, really the very first iteration we had really took up the entire room and you were basically on the front lawn and you would like walk 20 feet to like get up to the porch. And we were thrilled. We were like, oh, it works. This is incredible. I can't believe it works. This is great. Yeah, there was a lot of very, like, just very fundamental, like, technically trying to get things to work and play back and even at the very low fidelity that it is because we used this indie capture setup. We made a lot of decisions at various points to restrict the action to a tabletop, At one point, we had wanted to blend diorama and human scale, but the fidelity of the volumetric capture didn't really allow for that. Keeping her small on the table worked a lot better in the food montage. She's supposed to be peering out the blinds, watching to see what happens. And you see her peer out the blinds later on, so you get that. But we spent probably a week playing with that, trying to make it work. And we were like, oh yeah, no, we just got to get rid of that. This is not working. So I think we made a lot of, it was a very organic, artistic process and really looking at it as an experiment in a lot of ways as far as like what can we actually do that can maintain someone's interest for 22 minutes.
[00:19:04.571] Kent Bye: Yeah, I think the fatiguing nature is one of the biggest limitations in terms of it's not that comfortable to hold it out there. And so resting my arms on the table was a big help. The other thing that happened to me while I was watching this piece was the opening shot. I see this big city. And so my first inclination is to start to kind of walk around and to kind of zoom in and look at everything. And I'm like looking at it. And then, so there's a little glowing orb that's directing your attention towards different spots, but then the whole world starts to move and pan around. And I was in the middle of also moving around. And so I didn't notice that the world was moving. And then I lost the little white dot. I went back and watched the beginning again, where I was like, okay, I'm just going to stand still and see what happens. And. it was like more of a cinematic moving through a space so there's a little bit of a language of cinema where you're doing like this tracking shot whereas i was thinking of the language of any other type of spatial medium with i can be the one who does the tracking shot so as you talk about the evolution for how that shot came about i can totally see how it was uh kind of a thing that continued to expand out into what it became, which was a really epic tracking shot. But I did find the kind of language of what I normally expect with that type of spatial model versus more of a cinematic experience. And there was at least the orb that I was moving around, but the world was moving around. I kind of lost track of it for a bit and I had to go back and the second time from the beginning just to kind of understand each of those beats that I had missed because I was almost being punished by exploring around and by moving my body too much in a way that was architected in a way for you to not be moving around.
[00:20:51.495] Rory Mitchell: So you're 100% correct with all of that. I love that you mentioned that. Yes, we didn't intend to punish you with it. But I totally understand what you're saying. And I think that I've watched people do that exact thing. And I'm kind of like, oh, you're going to get lost in the parking lot. Or you're going to get lost in a tree. Or I see where you, you know, now you've got branches all around you. You got lost in a tree. It's OK. We had about a week where It was going to be interactive, where it was going to be entirely spatial. That world was not going to move. We were going to encourage you to walk as you got to the laundromat. You were going to hear what he had to say about the laundromat with a location trigger, and then you would walk over to the... We were, for a moment, we spent about literally a week being like, OK, well, look, we should do it more like this. We're going to change the whole thing. It's going to be, I think, this balance between interactive and I want to say passive because it's never going to be passive because, you know, you're still moving through it. But that just what you're talking about, that dichotomy between like more control, fully interactive versus on rails, cinematic experience and what ended up Making the decision for us was the cadence of the storytelling and the narration and wanting to weave that spell of that level, that sort of immersion and the rhythms of the piece starting to come through with the music and all of that. And we'd love to do something interactive like you described with this tool set, with these pipelines, but to plan it out more in advance to have it. And this is actually, this has been an ongoing discussion, showing this to other VR developers and folks who are like, oh, we want to like, Like, just this question of, like, gamifying stuff versus sitting in a more traditional sit-back storytelling experience. And I had one actor, and I think that this speaks to this, what's interactive, what's not interactive. I had a good friend of mine watch it who's an actor, and he said, oh, this is really, like, surprisingly exhilarating to be freed from the directorial point of view. as though I've just walked into this, as though I'm just eavesdropping on this woman's life, and that he, like you, could kind of walk wherever he wanted to and has broken out of that shot, reverse shot, wide shot, master shot, like this language that we have so built into our heads at this point for how stories are supposed to be told.
[00:23:42.392] Kent Bye: Yeah, this week, I just did an interview with Lisa Masseri, who's an anthropologist who has a book about the land of the unreal, where she spent a year in 2018, doing anthropological research about the immersive storytelling communities within LA. And she focused in on this phrase of the unreal as a way of describing you know, there's a lot of facades within the context of LA. Are you really going to have something that's the full-fledged experience or something that's just a front where it's giving the image of something and show she was using facade as a metaphor for VR as you're going in into these alternative worlds, but yet there's a facade nature where you're at the surface As I was watching this experience, I was thinking about L.A. and also the unreal, the land of fantasy and dreams of L.A., but also this being set in L.A., but also my own experience of having this unrealness, because obviously it's a scripted world and it's kind of absurdist in a certain way, but it's also talking about topics that are very serious in a way that has a little bit of a snarky tone or some satire. I didn't get the sense like, oh, this is like a straight up empathy VR piece. Although there are moments there at the end that slip into that framing in some ways. But I'd love to hear you elaborate on this balance of like addressing these real issues, especially with your background and documentary and as a storyteller, but also having the tone that you chose to take in the tent.
[00:25:12.522] Rory Mitchell: Yeah, I think that's a fascinating question that I think gets to the heart of the piece. It's a fairy tale. It's a modern fairy tale. It's not so much about the woman in the tent as it is about the woman in the house. And the unhoused crisis in Los Angeles has gone on my entire life, for generations at this point. And I think there are a lot of unkind people who do not have much empathy for the suffering of the people who have found themselves in crisis. I think that there are also a lot of people who are doing their best. Some of the best and brightest that we have in the city are devoting every day of their lives to alleviating that suffering. There are people who are working. There are tons of organizations. People are doing wonderful work. But somehow, for some reason, it does not seem like we can solve this. Can it be solved? And Maybe it's too easy for people to point the fingers at the people in the tent and say, oh, you did this, you did that. I can dismiss what's going on here. And maybe we might want to look at ourselves. That this is, I think, that for Martha and I, this is self-indicting, I think. And I myself have done a lot of work for non-profits, and we need to get messages out, we need to raise awareness, we need to do a lot of stuff, and it becomes, there's a whole non-profit industrial complex that kind of arises around these things. I mean, I think a lot of this piece was informed by my work with, on the last documentary, Diverse and Subversive, with Defend Boyle Heights, and a lot of their analysis comes down to looking at the nonprofit industrial complex and the way in which people are able to assuage their egos by aligning with a cause. But even if your motivations might be muddy, you can still end up doing the right thing and helping people in the end. which I think is what happens at the end of this piece, which is even though she is conflicted and the woman in the house, Anna, goes through, is kind of called to the floor, called to account by the woman in the tent, that in the end, she can come to a new recognition of the role that she plays. in the suffering of the unhoused and that it is a satire around these kind of interpersonal dynamics of activism and what it means when you try to help people and questions of power, questions of privilege, questions of all of this stuff that is frankly easier to not get into if you just want to make a nice VR piece.
[00:28:19.145] Kent Bye: Yeah, I think that this is a dynamic where as I was watching it, after having just read this anthropological critique of the so called empathy machine of VR, deconstructing it, and how, at the end of the day, it's also like trying to address some of these issues from a structural perspective. But yet, when it gets at that big, large structural scale becomes much more of a political issue rather than a moral issue moral in the sense where the actions of an individual can make some difference in individuals life, but in terms of the overall issue, it becomes such a larger scale becomes much more structural. And so then it's a question that I think of is like, okay, what's the structural dynamics of this as an issue? And is that something that we can even tell a story about in a way that I think in the description you said you're using like 2,500 years of tradition from theater and storytelling that you're telling the story, but yet at the same time, some of the other dynamics of this larger story are kind of in the background and the structural issues are those larger political issues that become much more difficult to kind of spatialize or to condense down into either a story like this or to, you know, figure out how to connect those dots in some ways.
[00:29:29.779] Rory Mitchell: Absolutely. It all comes down to just two people in dialogue with each other, that we are all somehow like co-creating this reality that we're living in. And the more transparent and frank we can all be about our experiences, of our lived experiences, then that seems to be maybe one of the only, you know, kind of all we can do, I think.
[00:30:00.513] Kent Bye: Yeah, I'm reminded of an interview I did with a philosopher named Lewis Gordon, where he was making the differentiation between a moral action and a political action, or a moral action is something that you are accounted for that you have ethics that you're living into. And a political action is actions that are taken that after many generations, those actions are basically invisible. Like when you take a political act, then who catalyzed that act ends up being much more of a collective story rather than an individual moral story. So I feel like there's a dimension of this story that is taking some of these political dynamics, but it translated into an individual moral actor. It seems like a piece that's really kind of wrestling with this tension between the morality and the political dimensions of these issues.
[00:30:44.784] Rory Mitchell: It absolutely is. That's that is 100 percent. We spent a long time, Martha and I, just kind of talking through the voice of the woman in the tent and not wanting to do a trauma plot, not wanting to say like, oh, this is, oh, this is what happened. Here's it. And now, as soon as you have a reason for why the woman is in the tent, then people can write it off. Right? They can say like, oh, like, oh, I understand why she's in there. We also knew that you would never see the woman in the tent, which I know that some viewers are not thrilled about. People would really like to see the woman in the tent. And I understand that because that is, that's data that we would like to have.
[00:31:29.266] Kent Bye: I did try to take the iPad and go into the tent, but everything disappeared. So I did try to see.
[00:31:36.012] Rory Mitchell: Yes, we did put a blinding white light inside the tent for anyone who wanted to take advantage of the affordance of spatial entertainment to just go look in the tent for themselves. You know, we knew you were never gonna see the, oh, spoiler alert. We knew you were never gonna see the woman in the tent because again, it's very easy for us to dismiss suffering. We are looking for reasons to dismiss what we see all the time. We spend a lot of time just kind of talking about the woman in the tent and wanting to make her as smart and educated in the language of social justice, if not more so than Anna was, and to be able to call her to account.
[00:32:23.062] Kent Bye: that we have been diving deep into this piece and it is on some of these higher end iPad pros. So what, what is your plan for this piece in terms of distribution or, you know, obviously it's going to be making its world premiere at South by Southwest, but I know that a lot of folks take it onto the festival circuit, but do you have plans at some point to make it available for folks to see on the app store?
[00:32:47.315] Rory Mitchell: Absolutely, I mean I think that is one of the reasons we built it for mobile first is you know after 10 years of trying to put people in headsets We realized that We need to make spatial entertainment more accessible to people, that there are people out there that would like to experience the magic of spatial entertainment, but they're not gamers, a headset is not appealing to them, all of the many well-catalogued drawbacks of VR headsets, and that what we're saying by building for mobile is that there's maybe almost too much focus on the headset itself, the physicality of the headset, the specific experience of being in a headset, and less so on what we're putting into the headset and what we can do with the content that we put into the headset. And so I think by taking this, because this is fundamentally an XR piece, and we're using the tools that people are using to create VR, but we're taking it out of the VR headset and putting it into an iPad, and I think that that you know, spark some interesting conversations like all of the stuff that you've pointed out here, which is like, what is this balance between being on tracks versus being able to explore freely? Where is the flow of storytelling? What is the balance between the user and the storyteller within this? And so we, you know, I mean, we're going to put this into a headset at some point. Just because holding an iPad is it gets tiring so we will put it into headset at some point But the goal is to release this on the App Store to find a way to optimize it a little bit further So we'll play on a greater variety of devices because I believe there is an audience for this new form of content that's beyond the rectangle that whatever this spatial entertainment thing is going to be like People are ready for it. Like, we all recognize that the rectangle is dying in some way. Like, you know, it's not going away. The rectangle is going to be here. But the stuff that's going to come is going to be a far more fluid and exciting form of storytelling than we've experienced before. From, you know, the difference from radio to television, the difference from still pictures to motion pictures. Like, a new thing is coming. And I'm so tremendously excited to see what we're all going to make and how it's going to change us in the process.
[00:35:08.827] Kent Bye: Yeah, and with the Apple Vision Pro coming out on February 2nd, it's been out for almost a month now, and this is an iPad app. And have you thought about whether or not you would want to try to make this into an Apple Vision Pro experience?
[00:35:23.358] Rory Mitchell: Oh yeah, absolutely. I am curious about the high resolution of the Apple Vision Pro and how that's going to play well with our low resolution volumetric capture. That is something that has occurred to me, but I don't think it matters. We would love to put it into an Apple Vision Pro. I think that would be a wonderful experience. We'd love to put it into a MetaQuest 3. I mean, I think that like this would be very cool to just not have to hold your arms up for 22 minutes at a time. The direction that we're going with this is to be able to have higher fidelity volumetric capture. I mean, volumetric capture is a really like I'm tremendously excited about what volumetric capture allows us to do. I think one of the things that I feel VR has been missing for the last 10 years is just humans, is just physical humans, and we have avatars. I love live VR theater. Incredibly powerful. I love the work that the ferryman folks are doing and live VR theater is wonderful You know stuff that metahumans are doing Alex Columb with his live theater stuff is wonderful you know one of the reasons we did 360 was because we just wanted like people and faces and this and I think that as volumetric video matures as a tool and We're going to be able to bring more of the traditions of film and television and the sophisticated storytelling that we rely upon, you know, our uniquely human affordances. You know, this is what we're talking about with like 2,500 years of theater tradition and all of this is one person standing in front of a of an audience and saying, here, I'm Sophocles, and I'm going to tell you this story. And then Dionysus steps out of the chorus, and he says, hey, I'm Dionysus. And then he starts walking along. But because it's humans and because we have mirror neurons, whatever you want to call them, whatever it is that allows us to, quote unquote, vibe with each other, you know, is what I think volumetric video is going to allow us to do. And really, like, again, going back to the Greeks, like, the next thing that we want to do, we'll just have three actors in it. And that's going to be incredible. I think once we can get three volumetric actors in a scene together, and I think that also gets to, like, what you were talking about in terms of, you know, fairly static scenes. You know, I think once we put you in between two intense actors doing their thing, and you're like right in between them. Now it's like, oh, it's a little too intense. I'm going to back up and move away from them. But they're in your room. They're on your table. They're doing this thing. It is an immersive theater show in your living room. It's so exciting.
[00:38:14.427] Kent Bye: Yeah. One technical question in terms of like, as I was playing it, it was split across two apps. And is that the way it's also going to be shown at South by Southwest is like a two part or like, why is it too big to fit into one app?
[00:38:28.618] Rory Mitchell: Yes. Yes. It's too big to fit into one app. And we also thought we would give people a little break from holding it up. There's actually a at South by people will be watching part one. on some podiums and then they will go into a little AstroTurf front yard and they will go into a tent to watch the second part. which I think was interesting considering the ending of the whole piece, which I don't want to mention, but I think that it kind of is in dialogue with itself with that. But yeah, we are definitely pushing up against the limits of the hardware and the volumetric capture technology and the ability to deliver these big photogrammetry models at you know, at scale. Our 3D artist, Logan Davis, did incredible work just, like, simplifying and re-topographing and texturing. And then Miller Klitzner, our developer, who was just here the entire time, who's really working like an editor, an art director. You know, everything is happening. I'm very, like, you know, having worked in the entertainment industry for some years in physical production, like, I'm really fascinated with the systems that are going to arise to creating this sort of content at scale. and what it's going to look like when we're producing 13 episodes of volumetric show, you know. I've spent some time thinking about what those crews might look like, what that production might look like. We got to do it. It's coming.
[00:40:04.030] Kent Bye: Yeah, we did want to share one experience that I had and just the discussions I was having with Lisa Masseri as an anthropologist, we were talking about the ethics of the empathy machine and the sense of who are you embodying? Are you a ghost? Are you a character? And so I felt like the way that in the story, there's a witness to the person who is unhoused. And in some ways, there's a nice switch where when it goes from the third person omniscient perspective to more of a first person perspective, it's more of like an embodiment of that witness. And so it's almost like empathy inception where you're empathizing with somebody who is empathizing rather than trying to embody the person who is the unhoused. So I feel like that was a kind of a nice sleight of hand in some ways where it felt like I was still a witness in a way that I could embody myself, but it could also embody the character who was also witnessing. So anyway, I felt like the handling of that, I thought in terms of the ethics of empathy projects was an interesting twist that I hadn't seen before.
[00:41:07.797] Rory Mitchell: Oh, you just put that so well. I need to listen back to this so I can write it down, so I can speak to people like that. Because that is such a, it's really, it's a mysterious process. And that is wonderful to hear, because it's just trying to twist it a little bit, right?
[00:41:28.223] Kent Bye: Yeah, because there is an opportunity within VR pieces to embody a character. And there's also an opportunity in 360 video to just be an invisible ghost witness. And I feel like there's also other methods there. I think that you're exploring in this piece that it hit me in a way that would have been different had you made some other choices. So we'll leave it at that.
[00:41:50.163] Rory Mitchell: I love it. I love it. That thrills me.
[00:41:53.355] Kent Bye: Great. Well, as we start to wrap up here, I'm curious what you think the ultimate potential of virtual reality and immersive storytelling might be and what it might be able to enable.
[00:42:04.017] Rory Mitchell: I think we're standing at the edge of what we all sense in our bones is coming, and we catch glimpses of it. We're the blind men in the room with the elephant, and we maybe see a piece, a VR piece that gives us a glimpse of what this is going to lead to. I don't think that nothing is going to lead to utopia. It will clearly be a mixed bag of what we have to deal with, but I do believe at least the potential for spatial narratives to get closer to the quick of our lived reality. I mean, I loved in your interview with Paul Raphael, just him referring to the abstraction of the cinematic language, you know, and I've been trying to find a way That language is perfect for it because I think that, you know, lived reality versus the rectangle, it's a lot. It's really like, you know, the rectangle is, for instance, the rectangle both blocks out the rest of the world and refers it to just this one little slice of it, but it also distorts it. I think the, I'm not phrasing this well. I knew this question was coming, but I didn't quite, there's so many things I wanted to say at once, but I think that, here's an example, right? An awkward moment in a spatial narrative may be more intense than a stabbing in a rectangular narrative. that an awkward pause between two people who are sitting across the table from you may do more to us inside than seeing a much bigger, dramatic thing in the rectangle, and the rectangle has kind of hyped needs to hype stuff up in order to get that emotional reaction out of us. So there is some way in which a spatial narrative might help to regulate our internal emotionality with lived experience without having this exacerbator of the rectangle. And since we learn how to live through the rectangle, maybe we can learn how to live differently or a little I don't want to say quieter, but somehow some way closer to what our actual lived reality is with this stuff. Like not an empathy machine, like that's not it, but there is something fundamental in being able to get closer and being able to recreate what our actual lived experience is. And just the recognition that going back to like the dude in the cave who knew exactly the speed to like wave the torch in front of the deer to make them look like they were drinking from the stream in the cave paintings, to the kineticoscope, to the zoetrope, to the clobbity-doobity-bobby. You know, we've invented so many different technologies to tell our stories. And the rectangle, I think a lot of people take the rectangle for granted. that it is the be-all and the end-all of humanity's ability to tell our stories. And spatial narratives, I think, will maybe recalibrate that storytelling instinct.
[00:45:26.311] Kent Bye: Yeah, definitely have a blend of those different techniques in this piece in terms of the cinematic language, the spatial language, and the theatrical tradition. I think you're leaning quite heavily on as well. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:45:42.701] Rory Mitchell: I am just so inspired by the immersive community. I'm so honored that you've interviewed me for this piece. It means a lot to me. The team that has supported this project and has made this, this was a labor of love by, you know, from all of us on it. Maddie Wager, our lead actor, a trained clown. They're given incredible performance. I can't wait for people to see it. Jet Eveleth, who is also another trained clown. We have a lot of clowns in Los Angeles these days, and they're remarkable performers. Jet is the woman in the tent and just stepped in and delivered an incredible performance. Greg Moss, our narrator, who's the Englishman who allowed me to kind of distance ourselves from Los Angeles by putting the words in this classic fairy tale English narration, there was a distancing, there's a Verfremdungseffekt with the Englishman's narration. And then just Miller Klitzner, our developer, and Veronica Flint, who helped consult with the dev, and Lou Doi, our producer, and Pete Davison, our other producer, and my good friend Dave Lang, who composed 18 minutes of music in the course of two weeks. Everybody is just so, you know, this was a labor of love, really. It's just, I'm excited to share it with people, and it means so much that you wanted to talk about it. And you are such an astute observer of the scene for so many years, Kent, and you tie everyone together in such a way. You learn so much from everybody. It's just, I was so excited to hear your thoughts on this thing. And you did not disappoint.
[00:47:27.397] Kent Bye: Awesome. Well, thanks. Yeah, I feel like each of these projects that are showing at these festivals, like South by Southwest, I think are the frontiers of trying to mix and mash all these different traditions together. And I feel like The way that you're they're weaving in this really beautiful cinematic opening shot of like volumetric photogrammetry I thought is something I certainly haven't seen much before and you know all these other aspects of bringing in the theatrical tradition into the spatial contracts and volumetric and I do think that holding up a iPad does get pretty fatiguing but like you said bringing into the headset or figuring other you know it's a 22 minute piece and so it ends up being like a ergonomically not as optimized for where it's going to be like maybe in five or ten years when maybe we're wearing headsets that are mixed reality or something. Apple Vision Pro has to figure out their own ergonomics. That's a separate conversation, but. But, uh, but yeah, I feel like at the end of the day, you're, you're kind of experimenting with some of these ethical and moral and political questions in a way that is provocative in a way that is trying to facilitate conversations like we've had here, where it's not the end all and be all, but it's a, a catalyst to kind of dive deeper into these issues. And, uh, yeah, just really appreciate the way that these stories are able to do that. So thanks again for taking the time to share a little bit more about your journey and process and a little bit more context about the tent.
[00:48:47.901] Rory Mitchell: Thank you, Kent. Really, really honored to be here. Thank you so much.
[00:48:52.406] Kent Bye: So thanks again for listening to this interview. This is usually where I would share some additional takeaways, but I've started to do a little bit more real time takeaways at the end of my conversations with folks to give some of my impressions. And I think as time goes on, I'm going to figure out how to use XR technologies within the context of the VoicesofVR.com website itself to do these type of spatial visualizations. So I'm putting a lot of my energy on thinking about that a lot more right now. But if you do want a little bit more in-depth conversations around some of these different ideas around immersive storytelling, I highly recommend a talk that I gave on YouTube. You can search for StoryCon Keynote, can't buy. I did a whole primer on presence, immersive storytelling, and experiential design. So, that's all that I have for today, and I just want to thank you all for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.