A Machine for Viewing was a series of three film essays that were shown to someone in virtual reality, whose first-person perspective was then projected onto the Egyptian movie theater screen at Sundance with a crowd who was watching the film essays through the eyes of the one person in virtual reality. There were embodied interactions that the person in VR would perform, and so the audience member became a performer who’s interactions would slightly modulate what the rest of the audience would get to see. So the embodied interactions of the viewer could shift the aspect ratio of the film, which was a film essay about the evolution of aspect ratios throughout the history of film.
There are many layers of Inception-like meta-analysis of using VR to watch someone watch a film essay in VR about the evolution of communications mediums while watching how the new VR medium can simultaneously modulate the individual and collective experience of said media. A Machine for Viewing was a collaborative project between VR artist and VRTOV creative director Oscar Raby, educator, filmmaker, and “virtual reality tourist” Richard Misek, and documentary filmmaker and “guest/visitor to VR” Charlie Shackleton. I had a chance to talk with the three lead artists just after their final screening at Sundance 2020 to explore what film can learn from VR, and what VR can learn from film.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So continuing on in my series of looking at some of the XR experiences at Sundance 2020, specifically the immersive storytelling innovations, the technical innovations, as well as the experiential design process. So today's interview is with a piece called A Machine for Viewing, which actually took place as part of the New Frontier double feature in the Egyptian theater on Monday and Tuesday night. I had a chance to see The Viewing on Tuesday night. So A Machine for Viewing was three film essays done by Oscar Raby, Richard Misak, and Charlie Shackleton. So they created these film essays, but they were viewed by having somebody in virtual reality. So they're in VR watching the film essay in a virtual theater and that was being projected onto the screen. So you're watching the film essay through someone's eyes who was in VR in a theater watching it. So it was sort of like these layers of inception. But they were trying to explore different aspects of embodiment and really explore this intersection between what are the affordances of the virtual reality medium and what are the affordances of the film medium and how are these two going to come together. And they did that by exploring different aspects of film history, unpacking the parallels between film and the medium of virtual reality. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Oskar, Richard and Charlie happened on Tuesday, January 28th, 2020 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:01:42.148] Oscar Raby: Hello, I'm Oskar Ravy, creative director of Virtual, a virtual reality studio based in Melbourne, Australia.
[00:01:48.412] Richard Misek: I'm Richard Misek. I'm an educator and filmmaker and currently a creative virtual reality tourist.
[00:01:56.796] Charlie Shackleton: Hi, I'm Charlie Shackleton. I'm a documentary filmmaker and also a guest-slash-visitor to the world of VR.
[00:02:05.446] Kent Bye: Cool, so why don't you each give a bit more context as to your background and your journey into working with immersive technologies?
[00:02:14.042] Oscar Raby: I've been working with virtual reality for the past five, six years now. And I still don't know what I'm doing. So I've teamed up with people that speak other languages, which is not an unusual thing for me to find. And finding what we can do with these machines and these toys that we play with, mainly trying to find what is that place that allows us to gather around and talk about these things that we talk about.
[00:02:42.520] Richard Misek: Well, I mean, my experience of virtual reality is much shorter. It's just for two years of this project. But my longer background of how this project came about was that I make video essays and essay films. And the project started as a hypothetical question. It's like, well, what would a virtual reality video essay be? Like, what if you could do more than just watch images cut together in new ways, but actually engage in a different way with pre-existing images and with cinema? And so that was the kind of a springboard for, in one way or another, for each of our episodes.
[00:03:15.602] Kent Bye: And so your background is also in film?
[00:03:18.504] Richard Misek: My background is absolutely in film. I teach film. I write about film. I make films. I think about film. I talk about film. I'm entirely immersed in film. So I come to virtual reality very much from a film perspective and remain from a film perspective even when I'm working with Oscar.
[00:03:37.084] Charlie Shackleton: Yeah, likewise, I come from film and I think primarily in that language and it was Richard who approached me a couple of years ago and told me about this project, A Machine for Viewing, that would contain a number of video essays in virtual reality, all of which came to be in some way about that relationship between virtual reality and cinema. And so I took a bit of coaxing, I think, move further away from cinema and further towards VR and hopefully wind up somewhere in the middle. But I am still very much a newbie.
[00:04:12.140] Kent Bye: And I'm wondering if you could give a bit more context as to this specific project and the evolution and how each of you got involved.
[00:04:20.054] Richard Misek: Well the project in fact started, the previous project I was involved with was one about video essays and it involved bringing together academics and artists and filmmakers to share languages and then eventually to do some joint work. And so I was like I think four or five days at the Whitechapel Gallery in 2016 where people got together and talked that and Charlie was one of the people I met there And then about a year later, I thought I'd apply for a bit more money and I had the idea of like, where can I take it further? And VR was having its real peak moment there. And so I just wrote a very bad application, but I put VR into it very much, you know, just VR everything and the money came. It was amazing how things, you know, and now it's AI. And I'm sure, you know, people are doing the same with just, you know, jumping onto the AI bandwagon and it's, you know, it's not a bad thing to do. self-confessedly jumped onto the VR bandwagon, and then the project just developed from there. And it sort of took the form of a conversation, really, a kind of quite a... I mean, we worked two years on it, but we were all doing other things, of course, at the same time. So it was quite a... in some ways, quite a leisurely conversation that took quite a lot of time to unfold between Oscar, Charlie, and me. And especially as Oscar is based in Australia, there was a lot of missed Skype calls, you know, initially. Did it sound like this was a problem? No, no, not at all. And so, yeah, it was really like, I'm really happy that there's a product and tonight was the first time, you know, that it actually all came together into a product. But it's, you know, for me, what's been most interesting about it is the process and the kind of sharing of languages and the bridging of gaps and the kind of exploring, but not knowing, you know, so it's like we wanted to do something with virtual reality that was the necessary thing, you know, that is what the money was for and that is what we had to deliver. But then, you know, it somehow evolved into being like having a bit of a mixed reality live element to it and I can't really remember how that happened. It was just a kind of meandering conversation which ended up there. And somehow this kind of path of moving between VR and cinema and liveness and juggling them all was what the project ended up as being. So a little bit much more kind of freestyle in a way and much less specifically focused on delivering three VR video essays, which is how it started.
[00:06:39.655] Kent Bye: Yeah, and the piece itself, it does feel like it's these three essays that you could potentially watch in a 2D screen without being in VR because they are very cinematic and you're going through a lot of the film history and commenting about the evolution of the language of cinema. And then you have people in the audience in VR watching in a movie theater, watching the screen while people that are actually in a theater are watching the person watching the VR on the big screen. So you have this kind of meta viewing of these video essays. And so maybe you could talk a bit about like that decision to have this inception many layers of watching your video essays through the eyes of someone in VR watching the essay.
[00:07:21.531] Richard Misek: Yeah, I mean, there's a real danger in just going, you know, too many layers of meta. But I think this was just inherently meta, this project. And in fact, Charlie's and my episodes were very directly based on the works that we'd done in the first project, the video essay project. And Charlie used a lot of the same clips, and I actually used my final video that I made and put it into the VR space. and so the layers are sort of somehow like a part of that sense of like it's within the work but it also is the work and it's like the work isn't a VR installation though it will at some point in the future take the form of a standalone where people just sit in it and you know do the three episodes. It's not really a performance though you know though we did it live in a theatre today Interesting that you say, you know, it could almost be a flat video because that's sort of a third iteration of it because it will just be a series of online videos maybe in a few months time, you know, once we've kind of, it reaches its next stage. And I really like the idea that it's a little bit of all of these things but not entirely all of them and like if you just see the performance, you get maybe 80% of it, and if you just will watch it online you'll get maybe 70% of it. If you experience it, maybe you get 75% of it. But no one form is it. It sort of straddles them a little bit. And each of them gives a little something more. So even just, if it becomes a flat video online, there'll be something about that that will provide something that the live experience didn't provide, even though it'll just be online. So it's nice just to slip between these things.
[00:08:51.887] Charlie Shackleton: Like certainly for me the pleasure of seeing all of these things coexist is not that it advocates any kind of hierarchy or any kind of right way. Like obviously all of the individual episodes are sort of interrogating this question of is there a right way to consume images or film or media in all of these different new emerging spaces. And so for me, it's sort of almost as stimulating when something doesn't land in a performance context, but does land in a VR space, or makes sense 2D, but once you put it in 3D, loses something. And it feels like you're having this conversation as you consume the episodes, especially in the unpredictable environment of a cinema with 300 people in it. And in fact, at the previous performance, one of the audience members in the Q&A said something I think really kind of profound, which was that even when things went wrong, the dialogue among the audience kind of became the show because they were having all the same conversations that we've been having for two years, which is, What is cinema? What is the essence of it? What is VR? How do those two things talk to each other? When do those two things fail to communicate? Which is just as likely, obviously. So yeah, certainly, I think, I mean, I'm glad to hear that you had as much trouble explaining it to people as I do, because it is very complicated, obviously, and there's all of these layers on top of each other. But I think that complication and those contradictions are a part of it, hopefully.
[00:10:19.738] Oscar Raby: I want to add that the thing that it is, it's a multi-part, multi-headed, multi-dimensional thing that doesn't want to be tied into any one field or label. And I think that if we can claim a success, I would claim that one. that it wants to be and it can be many other versions of itself in many other occasions depending on the circumstances. When we presented this work in Amsterdam it was part of a conference so the setting, the ritual around it, the building was adding a different layer to what we experience here in Sundance in a very traditional and beautiful cinema you enter the theater and you're already in that spirit and you see around we look at the world surrounding us right now and their past shows from ages ago and all the users and all of us as users of this tool that we created get embedded into that frame of mind into that spirit of here is a new way of entertainment of experiencing narrative content of experiencing people sitting down to take in some sort of image making ritual So when I think of what we have now, I don't think of three pieces of content that could be reproduced again as flatties, as video. But I think of the platform we created and how that tool is the main creation of the project. The way of doing things and the way of meeting in the middle, coming from different perspectives and traditions.
[00:11:45.241] Kent Bye: Yeah, I actually was at the IDFA doc lab in Amsterdam and saw the first chapter, but didn't see all three chapters because at the initial screening, we just saw the first chapter and then I had another screening at the Dome that I wanted to go to. But you're right in the sense that it was more of bleachers rather than like a cinema where you would go see a movie. And so I could definitely see the difference of how the environment that you see invokes the different aspects of cinema. And for each of the chapters, there's different levels of embodiment that you're taking from the audience member who is in VR and the first chapter you are having them do this mechanic of giving a square to be able to change the aspect ratio of the screen and then the second one actually stand up and start to move their body around a lot more and so you have this performative element of them being in the front row everybody in the theater can see them with a spotlight on and they're moving around and they're seeing like a projection of their embodied movements projected onto the screen. And in the third version, it's from the perspective of the booth and you cut away and you see a image of like a webcam image of them moving around, kind of deconstructing what they're doing with the different levels of the either track controllers or how they're moving around. you could actually see the webcam footage mirrored in the background. So it was like another meta level. So I felt like each chapter, there was different levels of which that you're trying to use the affordances of VR, but make it like a bit of a performative aspect of their embodied interactions with the piece be a part of the overall performance. And so seeing how the affordances of VR could be translated into a 2D representation as well.
[00:13:29.180] Oscar Raby: We arrived in the conversation the collective conversation of VR. We arrived at the factor or the The thing that cinema could take from VR. That's the thing that VR could bring to the cinema conversation Was that now we can look everywhere. So it's a camera based understanding. It's a visually based visual centric understanding of what we are can do It's just we can look around everywhere. However, there's another layer that VR can provide to cinema, which is real time. Things that are happening at the same time as they are being done, at the same time as the decisions that make that image be that image. are realized is something that we're just scratching the surface of with this project. But I think it has massive potential in terms of determining what is a thing that belongs with the tools that we already use, and it's just the next door way of doing things.
[00:14:25.350] Charlie Shackleton: To tie those last two things together, I think it's that leap and that way of thinking that is so anathema to cinema, obviously, that makes it feel all the more radical when you put it in a cinema, especially a cinema with a history and a cinema that... We're in the Egyptian theatre at Sundance. Every stock image I've ever seen of Sundance for the last 20 years is a picture of the outside of this theatre. And it was funny, because even when people were filing in at the beginning, and I begin the piece sitting among the audience next to the person who's going to participate in my episode. And we're surrounded by all this kit, and there's sensors up in front of us. And people are obviously intrigued, and they're coming over. But then when they would start sitting down in the seats around us, they started asking us, like, is this wrong? Am I going to be able to see it from here? Where am I meant to look? Is this a good seat? Is this a bad seat? Am I looking at the screen? Am I looking at you? And I was saying to him, you know, neither, both, either, or, but that idea is so foreign to the experience of cinema going that it right from the off, I think, puts people in this completely different headspace.
[00:15:33.810] Richard Misek: I think the one thing that has most excited me in the project is just a tiny little thing, it's a detail, but somehow it's given me so much pleasure, and it's been the live-cutting element of it. And in my episode it just happened a few times, but I was able to press a button and switch to another camera perspective. and the person in the headset was still going through the first-person POV experience but what was being fed out to the screen was another camera angle which in a way of course is cinematic you know it's just covering the scene but it's so disruptive you know it's just so like oh my god like the the sense of taboo that I was breaking when I was pressing that button and it's like you know maybe I'll cut now, maybe I'll cut now, okay, I'll cut now, you know, and it's somehow, especially if somebody who comes from an editing background where every cut, you know, you have this sort of ideology of like Walter Murch, where there can only be one right place to cut, you know, and your whole body feels it, and you know it, and then it's set in gold, and then suddenly, to turn that on its head, and just to, you know, to kind of, for cutting, just to be able to happen a bit earlier, a bit later, anytime you like, really, And somehow it looks cinematic, but the process of it is just the most uncinematic thing I've ever done. And I really loved it.
[00:16:49.810] Kent Bye: I think it was when in the third episode when it was from the perspective of the booth and the person who was on screen He wasn't reacting to seeing himself. And so I was like, oh this must be like I was wondering, you know when you were cutting away how much was their first-person perspective and how much that was being shown on the screen that they couldn't see but it sounds like then the second and third episode you have different elements like that and But there was a question at the end that I thought was very striking. It was someone had asked about what does VR mean for the philosophy of cinema? It's very deep questions one that I've actually been thinking quite a lot about and so I'm just curious to pose that to you each of you and Then I'll share some of my thoughts as well
[00:17:30.513] Oscar Raby: In creating this project we thought of many titles and there was always the shadow of the machine. Machine as in something that is not metallic, that is not plastic, that is not electronic, but there is a complex thought made into form, a complex system of thought made into form. When we think about the roles and the elements and the ingredients that become part of the machinery of cinema, it's not just the screen, it's not just the pixels, it's not the texture and all the things that we've mentioned in our narratives, the proximity and the framing and the aspect ratios. but it's also the things that we do with it, the things that we talk about, the rituals that we surround us with when we talk about cinema, when we go to the cinema, when we make cinema. And in my episode, which takes us to the projection booth, it's not so much the space of the booth, but the people that do their work there, and the machines that do their work with them. in the conjunction of those things, because that is somehow a scenic dog of what happens in the auditorium, in the cinema room. People are somehow colliding, getting closer to the screen to make something happen, which we don't have access to. That's their own personal experience. We really don't have access to it. We try to make conventions to talk about it, but we have no idea of what happens to each one of them. By the same token, we have no idea what sort of film the technician is looking at, at the back. when they're taking care of sound levels, when they're switching on and off the lights, when they're looking at the projector, and that is the film for them, not the thing that's happening on the screen, but the projector is the film experience. There's a bigger machinery that expands beyond the frame and has nothing to do with the visual frame, it's the frame of the film.
[00:19:24.065] Richard Misek: Yeah, I mean, my sense of what VR gives to the philosophy of cinema is, again, an outsider's sense, just having come to it recently, but a sense of the two as being quite separate. And it feels to me sometimes like looking at festivals, it's like sometimes VR is given a place, not just VR, but immersive and interactive. anything that's not linear flat cinema you know is sometimes given a place at the table by more forward-looking festivals but often not more often it's not and even when it is it's always a separate program and it's programmed by separate people and there's this sort of sense of like you know I mean it's great that Sundance embraces it and you know the kind of a you know the new frontier program is just one for most forward-looking emerging media programs of the world but at the same time it's still somehow that imbrication doesn't quite happen and it's like New Frontier people go to the Ray for six hours a day and all the film people go and watch five movies a day and it feels that cinema really needs that shaking right now because it's quite a, in many fields at least, I feel it's quite a conservative medium And for all the amazing technological changes that have happened in the last 25 years, cinema has absorbed them in its economic models of how it's produced, but it hasn't changed cinema. And most of what appears on screen now, even though it's on Netflix and the platforms have changed, could have been made 20 years ago, 30 years ago. And it feels like cinema really needs some kind of an injection of something else. And maybe this is just a kind of a first step in that, but it's very necessary. So I think VR puts the mirror up to cinema in a way which perhaps hasn't quite yet helped it along enough, but maybe will do so more in the future.
[00:21:08.084] Charlie Shackleton: Yeah, I'd echo much of that. Although I'd add that New Frontier is perhaps the only strand of its kind that does actually play films as well, like linear 2D films.
[00:21:19.227] Kent Bye: Well, there's Tribeca in Venice, as well as South by Southwest that all have VR programs as well.
[00:21:25.801] Charlie Shackleton: Oh, no, well, you correct me if I'm wrong. I specifically meant that the Strand itself that is New Frontier that plays VR also includes film in its program.
[00:21:35.263] Kent Bye: Oh, yeah. So the new, yeah, yeah. So most of the emergent program, some of the, it's like the doc lab has video installations, but you're right in the sense that most of the emergent programs of the other festivals are mostly just the emergent technology. But yeah, there are 2D films that are included into New Frontier. And I think that is fairly unique. Yeah.
[00:21:52.423] Charlie Shackleton: Which I really like and I think that fluidity is really healthy and indeed the ones that end up playing there seem to be the ones that are maybe exceptions to the rule that Richard is talking about. I remember they played Camera Person in New Frontier a few years ago, which played everywhere else around the world very successfully as a linear documentary. It was released in cinemas. It was released in the Criterion Collection. But seeing it in the context of New Frontier, I think it's actually a really interesting frame. And I think, yeah, a kind of indication of where a healthy relationship between all of these forms might go. Certainly on a personal level, this whole experience has changed the way I think about any new project in that, you know, I have experience as a journalist and as a writer, documentary filmmaker, and now obviously having done a VR project. And it certainly changed the way that I would go into any conceptual idea or subject or topic that I wanted to cover with a sort of default position of like, how would this be a film? How would this be a documentary film? Would it be short? Would it be a feature? Whatever. Now it feels like the way I'm envisioning new work as much more of a blank canvas and you're adding whatever elements can help tell that story, whether it's video, audio, interactive, anything else. And yeah, I think that fluidity feels very exciting to me.
[00:23:09.888] Kent Bye: Yeah, and one of the reasons why I like to get more context as people's background and what orientation they're coming from as they come to VR is because VR as a medium seems like it is integrating all these aspects of interaction, of video games, of the human-computer interaction, of computers and the World Wide Web, in terms of when you make choices, what are the affordances that you use to take action. And then there's aspects of cinema and the cinematic storytelling of cinema, but then there's theater and other aspects of embodiment and architecture, and it's really fusing together all these different disciplines in a way that I mean, cinema is probably the closest in terms of incorporating all these, but most scripts are written, they're produced, they're post-production. It's a very waterfall approach, like building a building. Whereas game design is very iterative, like you have to design something quickly, you have to user test it, you're failing fast, and you have to really see if those interactions are actually working. you're trying to find which blend of all these different elements of the storytelling affordances of each of these different mediums and I feel like VR is like trying to fuse all those together and so I feel like looking at what storytelling is in this new medium I think is going to help eventually come up with this larger perspective that then is going to be fed back into these other mediums to see how you could start to boil down to the essence of, you know, the human experience or the essence of a story or context or character and how that can be expressed within these immersive mediums.
[00:24:38.096] Richard Misek: What do you think VR is giving to cinema right now?
[00:24:42.399] Kent Bye: What is VR giving to cinema? Well, in this piece that we just watched, who is the, what was the name of the performer? Miwa Maitrek. Yeah, Mia was performance, and so that in some sense was a 2D representation that was very cinematic, but it was projection mapped, but she was using a live embodied performance. And so there's a certain quality of a live performance that is different than something that's pre-recorded. And something that's coordinated in that way feels like somewhat magical. And so I feel like there's that aspect. And I think the other aspect is going to be world building and being able to see how, you know, the book of distance is a great example. That's here this year that allows you to dive deep into a certain context, but you're embodied into that context. And so you do these different performative actions with the principles of embodied cognition. Once you have the capability to be in. an immersive experience and have some actions, then you put it into your body in a new way. And so I feel like there's going to be certain aspects of, you know, you can think of the polarity between yin and yang, where cinema is very yin, where the authorial control of the piece of cinema is fixed, in the sense that the director has complete control of what you're seeing on the screen. But an interactive game is much more putting more agency and control into the user. And so they're able to make choices and take action and express their agency. And so it's moving away from that linearity and thinking it more as a world that you're exploring that maybe you're able to pick up different aspects of that story. I feel like that there's going to be certain ways in which the four instances of film are going to be great to tell certain stories, but maybe people don't have enough context or they're not engaged enough. And so I feel like they're probably going to be working together in certain ways. Like there's a experience here called Haifa, which is about mycelium and it's very poetic. It's didactic in the sense that it's trying to explain each part of the fungi and the mycelium as you go through the life cycle of a mushroom. And you're embodying that. And you know, after I went to that experience, I was like, oh, is this really what happens? And I was talking to the creator. She's like, yeah, that's what's happened. And you know, I could see watching a film about mycelium. And then I would watch that film and have those embodied experiences in my body that would have me much more engaged. And I think another example is like traveling while black, where you can feel like you're completely immersed within a context where you can enter it in as a ghost. And you are not necessarily going to feel like if you were actually there, and you're around a circle of African American Americans, and you're not African American, and whenever you're in a situation like that, you don't have the shared experiences, then it changes the context. So I feel like you're going to be able to enter into specific context within virtual reality and then be able to feel like you're able to experience different aspects of the human experience that you wouldn't normally have access to. So I think those are some of the initial things, but I think aspects of world building and context and character development Quibbly, I think, is another aspect where it's going to be short form. I think there's going to be interesting ways in which that is going to explore character and context in a way that is also going to be different than the normal. You're going to have more non-linearity that's going to be possible with it. more aspects of agency with that as well. You know, Bandersnatch is another example from Netflix, where you're able to make some choices and see that agency, but there's a spectrum between authorial control and generativity, where the generativity is like the most extreme, is like an open world, where you lose a lot of that narrative tension, but as you go from authorial control to that generativity, then you are on the spectrum of authorial control, you have complete control, whereas as soon as you give up agency, you're giving your agency as a director away to the audience. And so at what point on that spectrum do you want to live? And I feel like the future of cinema, if it continues to have different aspects of agency, you know, maybe it's going to be considered a game, maybe it's VR, you know, maybe it's going to be a whole other new genre. I don't know. I feel like, you know, this aspect of allowing the audience to participate is going to be a part of what shapes the future.
[00:28:50.836] Oscar Raby: Can we talk a bit about agency and control? Because I think there is a range, a spectrum there, where we tend to put cinema in one end and interactive media, where games is foremost example, most salient example, as separate things. And the closer you get to the tools, The more I get to see that we have the illusion of control because we think of the author that's controlling everything because they have a vision that's very clear and they're going to tell everyone from the DOP to the technicians to the gaffers to the actors. distributors exactly what they want to get as a cinematic statement then they come to the realization of being in the studio and having to ask the actors and having to ask the cinematographers this is what i want but what i want is not what the tool wants what the tool gives us is what the tool decides to give us There's an illusion of control there that goes from cinematic approaches, even if we can rely on the timeline and the frame, still the tool is what has the most certainty in all of that conversation. More than the director, the producers, more than the EP, more than the funders, is the tool. What it can do when it breaks and how it breaks is the final author of the piece. And that goes from cinema all the way to games.
[00:30:15.096] Richard Misek: Yeah, I absolutely accept your analysis of there being a spectrum and where do you find yourself in that spectrum, but I would also say that there are multiple spectrums and that's one of many spectrums. Another spectrum you could say would be between high budget and low budget, artisanal versus studio produced, and you could keep going forever. And I kind of feel that with that spectrum, obviously it is the irreconcilable. Somehow there's this sense of storytelling and interactivity as being irreconcilable. But I wonder if in 10 years or 20 years' time we're talking about this, we'll be thinking of this as a spectrum at all. There'll be some other spectrum we're talking about. And, you know, like when your friend Oscar, the guy who made GTA, like the way he was, we met him a couple of days ago, and the way he was talking about his project he was working on now, it was like, he was like a director of a big budget Hollywood movie, would talk about a Hollywood movie, exactly like you were talking about dealing with actors and script writers and things. And suddenly that distance between interactive and narrative just immediately collapsed. And so, you know, I think somehow it's not just that spectrum.
[00:31:28.548] Kent Bye: Yeah, well, I agree. And I'm a big fan of Gödel's incompleteness theorem, which essentially says that for any consistent set of logic is going to be one of two things. It's going to be consistent or complete, and you can only choose one. So if it's consistent, it's incomplete. If it's complete, then it's inconsistent and has paradoxes. I take a pluralistic approach of trying to have many different taxonomies that are kind of split up into the quality and the context and the character and you know I think there's going to be an infinite number of budget and different ways in which that you can have different spectrums but I find them helpful in the sense of you know seeing a lot of immersive experiences and then trying to see how I categorize them in my mind, in my own body, and then having conversations with different creators because there's different trade-offs that you have as a creator as to decisions you have to make in creating an immersive piece. And so that's at least how I've been trying to think about things. But I wanted to ask a little bit about the cinematic language of film, because you're obviously pulling in from a whole wide history of film history, film theory, and, you know, looking to the past of ideas that come up, and then they die out, and then they get reborn, like the square aspect ratio that then is an idea from over 50 years ago, then gets reborn in Instagram. And so You have even just the form of cinema that could have different aspect ratios making different affordances. But I'm curious, what can film teach VR in terms of different lessons we can take from the whole hundred plus years of cinematic history?
[00:32:57.690] Charlie Shackleton: Yeah, I mean, I think, like, a theme of my episode is certainly a warning from the past, from the early days of cinema to now, the early days of VR, not to mistake what emerges as orthodoxy for some sort of, like, preordained truth, you know? That, like, we now look at these things that have become absolutely fundamental to cinema, the widescreen cinema screen, surround sound, all of these various aspects that are now common to almost all major films. And they all emerged for a variety of like pragmatic or preferential reasons that are no better than the alternative. It's just that history turned out that way. You know, we could all be watching screens that are 10 feet taller than they are wide, but that's not how history went. And I think what my piece tries to look at is how people tried to stop that tide of like orthodoxy establishing itself so that the form could still be an experimental one. And VR to me still now feels like an experimental form where people are taking it in various different directions. But as it becomes more and more established, it's inevitable that there will become dominant preferences and ideologies and orthodoxies that people start to replicate. And then equipment will be built that fits those orthodoxies and then they become even more reinforced. So I hope that VR can learn from cinema's mistakes in that way and remain something that can push at its own limits and create new possibilities rather than becoming more and more alike.
[00:34:42.284] Richard Misek: I think one thing that cinema can offer VR is, how to put it, temporal sophistication. That's not quite the right word, but an ability to represent, a very fluid ability to represent temporality. which currently VR doesn't quite have. You know, VR is essentially spatial, like when you talk about world building, when you talk about interaction, immersion, you know, these are all kind of spatial metaphors, basically. And VR is very much now anchored in space. And you see this, you know, right on the basic level in the interface of, you know, the game engine of Unity. where you land in it, and even when you just put on an Oculus headset, you land in it, and it's like a Cartesian space with a horizon. And then that's replicated again in the Unity game engine, where it's just a 3D space, and that's the starting point. And somehow, it's too simple to say cinema is time and VR is space, but there's a kind of a... I think what VR has that cinema doesn't is this sense, like you said, of placing you in environments and allowing you to interact with them. this environmental engagement, but what cinema, I think, can still offer to VR is some answers about how to condense time, expand time, move between temporalities, juggle temporalities, fragment temporalities, and have a kind of temporal complexity, which I think in part, it's currently down to the technology of VR that doesn't allow it, but I think in part, also in large part, it's just that that sense of possibility hasn't quite evolved. And so, you know, I'm thinking like in future new softwares that can allow VR to be used where there's an easier control of time, where you're not stuck in one space. And then if you want to go to another space, you have to end that particular build and start another build.
[00:36:35.224] Oscar Raby: The sophistication starts from the tool. Absolutely right. When you open Premiere, you see the timeline and you can rely on that. That's a blank project that already has a timeline established for you as a convention. Not so much as a tool itself, but as a convention in the language. You open Unity and the first thing you see is a 3D space, right? Only recently, Unity started having a timeline as an available tool that didn't require any kind of further rearrangement of the project. So yeah, the things that we use to create these projects and these experiences are the things that get embedded by default in the language that we create and that we refine and that we twist and create nuances on. If anything, you're absolutely right. We just need more tools, more diverse tools, more hybrid tools to create the variations. And hopefully those tools can come from cinema because the language of cinema has created such sophistication and nuances that we don't have yet in VR. However, the pace that we grow that language in VR is so vast because we also are able to bring other disciplines. So it can only grow, it can only get better.
[00:37:47.418] Kent Bye: One other quick note that I'll make in terms of what's already happening with VR and cinema is the virtual production that's happening in Hollywood where you have a lot of like Steven Spielberg in Ready Player One where he would actually go into a scene that was built in like a real-time game engine and then be able to cinematically walk around and then be able to have them in real-time move the objects around so that the pre-visualization is now moving into real-time game engine but also having these LED walls in the background so that you can start to have it modeled in a real-time game engine and tracking the camera, you can start to mimic the parallax effects that you would get and you don't have to actually go on set. So actually a lot of the Mandalorian was shot with these types of virtual production tools where you're able to build the worlds out and then kind of mimic you actually being there. So I feel like there's going to continue to be that strand in terms of just the logistics of producing. But just to start to kind of wrap up things here, I'm curious what each of you want to experience in VR. Like what type of experiences, now that you've created this experience coming from a film background, is there any specific experiences that you would like to have?
[00:38:53.671] Oscar Raby: Absolutely. I want to see VR being used to create something else. Not VR as the end product, but VR as the means to empower the roles of production into having different affordances. So I imagine a TV production set in which the camera operators and the editor and the sound recordist and the sound designer are all live performing their duties with some sort of VR interface. to create a different product, which could be like telly, or it could be a film, or it could be a theatre piece, but somehow VR and body, the haptic and body interfaces that VR provides are the tools that we start using to create these experiences.
[00:39:36.233] Richard Misek: Yeah, I would also like that mixing of tools and technologies and to experience it, you know, kind of like we a little bit tried to create today and like we experienced perhaps much more effectively with Miwa's amazing work. That sort of pre-recorded liveness and, you know, 3D and 2D all kind of mixing together and, you know, the Ready Player One example, it's like I would so much rather have been on set experiencing what they were doing with that technology than actually watching the film, you know.
[00:40:06.379] Charlie Shackleton: Yeah, I think in my admittedly relatively limited experience of what's out there in VR, a frustrating sense I often have when I do VR experiences is the feeling that the experience has been born of someone wanting to make a VR project and looking for a subject, and maybe indeed looking for a subject that Well, OK. Maybe finding a subject that is actually really suited to VR and that couldn't exist in another form. But still, I always have that sense that it's happened that way around. And I'd love to start seeing more work where someone starts from a position of, I want to address this subject, tell this story, create this mood, whatever that may be. And therefore, it demands this form and that the form is following function to that extent.
[00:40:57.325] Kent Bye: Great. And finally, what do you each think is the ultimate potential of immersive technologies and immersive storytelling and what it might be able to enable?
[00:41:09.200] Oscar Raby: I would really love to see the cultural response, the language response, the artistic response to the need that we somehow felt around 2000, that everything has to be connected, that we should be connected, and connection is the ultimate goal for every single living entity. I would love to just have that time, that bubble, that time that you get when you're reading a book at a park, and no one will bother you because they know that you have something holding in your hand. You're holding something in your hand that says, I want to be on my own. The first times, the first approaches, the first times that I put the headset on people, that we've all done as VR makers, and you see them go into that bubble of their own experience, and as much as we can get through the monitor, it's never the thing that they're experiencing. I would love that to be more and more and more and be socially acceptable, right? That it doesn't have to compete with the social anything. It's just I'm having a time with myself on my own. Thank you very much.
[00:42:19.263] Richard Misek: I think I would just very shortly and simply say that I'm no visionary. I don't spend long hours thinking about this question, what the future of technology, what technology in the future will offer us. But I would like immersive somehow to give me a heightened sense of my physical presence in the world and allow me to enjoy my everyday non-technological existence more.
[00:42:45.291] Charlie Shackleton: Yeah, I don't know if I can beat those two answers, except to say that, yeah, I'd like it to. I still feel like there are myriad creative ideas and expressions that don't yet have a vehicle for the telling. And inevitably, it will, wherever this technology goes, lead to the opening of some of those doors.
[00:43:09.321] Kent Bye: Awesome. Great. Well, thank you so much for joining me today on the podcast. So thank you. Thank you. Thank you. Thank you. So that was Oscar Raby. He's the creative director of Virtual. Been working in VR for the last five to six years. Richard Misek, he's an educator and filmmaker and a creative virtual reality tourist. And then Charlie Shackleton, he's a documentary filmmaker and a guest slash visitor to VR. So I have a number of different takeaways about this interview, is that first of all, well, it was really striking to hear from both Richard and Charlie, who come from a film background, call themselves guests or visitors or tourists to VR, where they got a grant to be able to make this experience, but yet they're not necessarily interested in pursuing virtual reality as a medium in its own right. They had fun as an exploration, but their center of gravity is within film. And so they created these film essays and had this aspect of exploring embodiment and live performance. And, you know, there's Walter Murch, who talks a lot about editing when you're making an edit you have to really feel it with your whole body and it's like all about this instinctual aspect of when to put the cut and so one of the things that Richard said is that as he was doing this VR version on the screen he's seeing it on the big screen but he's making the decision live in the moment to do this live cut and so in some ways he's getting that same feeling of when to cut but in a live performance. And so I think that's one aspect of VR that you start to give a little bit more of those live components where it's a little bit less certain as to exactly how things are going to unfold. In this case, they have different embodied interactions that the person in the audience is doing. And so they're either using their hands to create a square and change the size of the aspect ratio. So in the first piece by Charlie Shackleton, he's really exploring aspect ratio and talking about how we have wider than taller. But you know, there was a film theorist that 50 years ago, it was really advocating for this one by one aspect ratio, which really never really took off until like something like Instagram, where You have the form factor of the phone and you're able to get equal horizontal and vertical in that as a format. And then have people that were talking about that over 50 years ago as how this would be a great idea. But yet it kind of took up to this moment to have the right distribution platform to actually have that aspect ratio really take off in any significant way. so it was just interesting to hear looking at this historical evolution of film and to talk about these specific things and also to suss out what is it that film is giving to vr and what is vr giving to film one of the things that richard said is that there's this temporal sophistication so this Fluid ability to be able to represent spatiality and so you have different things of montage and cuts and edits There's a whole language that's evolved within film that is able to explore different aspects of temporality And so being able to draw upon that well-established language I think is just something that I've already started to see in happen within virtual reality. And actually the thing to add even more is as you move your body, you're able to change the experience of as you're moving through space. So things like super hot, where you're actually changing the pace at which things are unfolding based upon how you're moving. And we've seen a lot within the film of things like time-lapse and other things like that. And I expect to eventually see a lot more of that within virtual reality as well, to have full spatialized time-lapses. We can actually see the full life of a bush, for example, or to go through the different seasonal cycles of plants as they go through their different phases. So I expect to see more of that at some point to be able to explore different aspects of temporality. But there was this moment when they're asking me, you know, what I thought about what VR is giving to film. And there are different aspects of agency and generative control. where I see in one extreme I said that you know you have complete authorial control and then the other extreme you have the generativity and Oscar his response to that was that actually how much authorial control do film directors actually have when they have to work with producers and limit budgets and you have to look at different aspects of the cinematography and actors and the writers and you know There's all these different phases that has to have a whole group of people to be able to collaborate to be able to produce it So how much control are you actually having on top of all the different? Technological limitations that you have where the technological limitations actually end up driving a lot of the different creative decisions as well and so I think he was just questioning that assertion of complete authorial control and And, you know, I think that's completely right in the sense of producing the piece of content. And I think that's true for any content, whether it's film or interactive media or whatever it is, you still have those different types of limitations of just needing to cooperate with a lot of different people and different constraints of communication and technological limitations. I guess the distinction that I was trying to make in saying that in terms of authorial control is that when you watch a film, whatever you do as a viewer, whatever agency that you do is not going to have any sort of feedback mechanism into what you're watching. So you have a film and it's released and everybody around the world sees pretty much the same thing. Unless you have something like Bandersnatch, which is an interactive film that you could start have different variations. On the other extreme of video game, if you have this open world, then it's really like each person's experiences of that is going to be modulated through the different choices and the agency that they decide to express within that experience. And so you get this highly variable experience sets based upon the sets of decisions that they're making. So I guess that's the differentiation that I would make between agency and authorial control is in a film, whatever gets delivered to the movie theater is like the final say of the collective of all the authored voices that are trying to create that piece of media. And then the video game, it has a lot more of that interaction where there is this room to be able to have these explored interactions and the experience that each individual has is going to be different. So Anyway, I just wanted to kind of unpack that a little bit, because as I was talking to them as critical theorists, they're breaking down different aspects of my critical theory of just the way I'm looking at it. And I think one of the things that Richard said is like, well, you can look at any number of different axes and aspects, whether it's budget, low budget, high budget, and you see all these different decisions. My main point is that there's different equivalence classes and different decisions that as creators, you have to make. And depending on those different decisions, there's going to be sort of sets of different things that you're making decisions around that is going to change the experience that people have within whatever media that you're creating. I think that's true for film, for video games, as well as immersive virtual reality experiences, for theater, for web experiences and websites. As designers, you have these different decisions that you have to make. And I think part of the challenge within virtual reality is trying to map out that whole landscape of all the decisions that you have to make and how those different decisions affect someone's direct phenomenological experience as they're going through whatever that you're creating. So I think that's sort of the task of trying to create this experiential design framework to be able to make these different types of decisions. So just a couple of other thoughts. One of the things that I think was striking to me was hearing them talk about how people in the audience, as they were sitting behind Charlie and you know, they have a virtual reality headset and gear there and they're in a movie theater. And so they're thinking to themselves, like, is this a good place for me to sit? Is this a bad place? You know, like where's going to be the optimal space for me to see this experience? Because I had kind of a similar experience, which is, you know, I sat right behind. people that were going to be in virtual reality doing it. And I was like, is this going to be the best place for me to sit to see? And so there's something about the new frontier where these different types of experiences makes you question like your whole ritual that you usually go through when you go into a theater. Usually when you go into a theater, there's just an algorithm that you do like, okay, I'm just going to go and I'm going to sit in this place. I'm going to have the optimal view experience because you know, you've gone to a movie so many different times, but when the new frontier, it's going to be something that you've never quite seen before. And so knowing where to sit and have the best view and when there's things that are in the audience it kind of starts to disrupt that a little bit and I think that was part of the purpose of trying to have people to reflect upon their own moviegoing experience by watching people who are in VR be in a theater watching these different film essays. So I actually look forward to having these videos up online because I think that was a lot of the meat of what they were talking about. And just from a film theory perspective to kind of look at these different evolutions and for Oscar rabies piece, you know, he's up in the film projector. And so for somebody who is the projectionist watching a film, their experience of the film has a lot more to do with managing the technology and making sure things go right. You know, It's a lot different than being in the audience and just being able to sit back and relax and enjoy the show for the projectionists there. It's like high stakes drama of trying to make sure that, you know, back in the day when they actually have real to real films and a lot of mechanics that have to be managed at this point, it's now just digital projectors and a lot less of that. needing to ensure that things are going correctly but still from this old lineage of film and the projectionist and being in that projection booth and talking about a lot of those different dynamics while having somebody in this room scale experience handling the different tracked objects. So at some point I think they're going to be releasing videos of some of these essays exploring some of these different concepts and hopefully when those come out be able to watch those as well to get more of a sense of this experience but I really enjoyed the different intellectual explorations and trying to think about the essence of the medium and comparing it to virtual reality. I look forward to seeing what comes next of this project and for some of these different film essays that they showed to be able to get out there and for people to watch as well. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, leave a review on iTunes. You know, anytime you listen to something that you enjoy, tweet it out, send out a tweet and share it with your friends or just in private messages, just connect to somebody else and have them listen and have a conversation about it. And if you enjoy the podcast and you want to see more, then please do consider becoming a member of the Patreon. This is a list of support a podcast. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voice of VR. Thanks for listening.
