#1292: Pioneering the VR Essay with “Shadowtime” Critiquing Sci-Fi Dystopic Aspirations of VR

I interviewed Shadowtime co-directors Sister Sylvester (Kathryn Hamilton) and Deniz Tortum at Venice Immersive 2023. See more context in the rough transcript below.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Rough Transcript

[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my series of looking at different experiences from Venice Immersive 2023, this is episode number 22 of 35, and number 2 of 10 of looking at the contextual domain of ideas and adventure. So this piece is called Shadow Time by Sister Sylvester, a.k.a. Katherine Hamilton, and Dennis Tortum. So this is a VR essay, one of the first VR essays that I've seen that is using the medium of virtual reality to talk about the medium of virtual reality. This is a piece that actually learned a lot more about listening to the creators talk about the piece. And I highly recommend if you do have a chance to see the piece to go see it, because we will be unpacking lots of different nuances of the piece. And there's certainly going to be certain elements of spoilers to unpack different aspects of what the piece is doing. That's kind of true for each of the different experiences and certainly for some of these experiences they may not be available to see and I think there's still a lot of provocative ideas that are worth exploring as well as the process of experience design. So it's always kind of a tricky balance of trying to comprehensively cover what's happening in the medium of virtual reality, especially at the edges of innovation and pieces like this. especially because they're not always easy to see. So with all that in mind, we're going to be doing a deep dive into both the history of computer graphics, a lot of philosophical ideas, and a little bit of a critical take of the medium of virtual reality. So there's a piece that I'd recommend folks check out called Our Arc. I'll put a link in the description. That's a film that both Dennis and Sylvester did previous to this experience. where they dive into unpacking different aspects of Elon Musk and the simulation theory and they kind of have some twists in that piece where there's some similar twists within shadow time that they continue to speak from a voice that isn't always necessarily their authorial voice. They're kind of like Doing these interesting little twists where they're doing this reverse psychology brainwash propaganda type of stuff of arguing for certain things that they don't necessarily fully believe themselves and so part of this conversation for me was decoding what was being said in the experience in versus what their deeper beliefs and convictions of the authors themselves because They're taking quite a critical look at the medium of virtuality overall generally but also being at the same time quite compelled to the power of the medium and And so there's a catalyzing article that's worth checking out. It's from Wired Magazine by Wagner James Au from February 25th, 2016. It's called VR Will Make Life Better or Just Be an Opiate for the Masses. And so this is an article where Wagner James Au actually gets some quotes from both John Carmack as well as Palmer Luckey where they're talking about different potentials of the virtual reality medium. And so Palmer Luckey says, everyone wants to have a happy life, but it's going to be impossible to give everyone everything they want. continues on to say, virtual reality can make it so anyone anywhere can have these experiences. I'm not sure if some of these different experiences in the article were things that Palmer is saying or things that Wagner is saying, because Wagner says, but VR can provide billions of people with virtual versions of everything the wealthy take for granted. Turning the Louvre sailing the Sun dappled coast of California or simply sitting in a meadow beneath a clear blue sky free of smog and pollution So it's a little unclear as to what are all the different things that Palmer is saying versus what things that are being added in from Wagner but nonetheless, I think the gist of this article is that there's a deeper critique of trying to push back from a Some of these ideas that are from dystopic science fiction and kind of being exalted as a virtue of the medium of virtual reality itself and so critiquing the larger Economic and political and cultural context around some of these media and taking a little bit more of a critical look so in shadow time you're kind of taking through the history of computer graphics and there's kind of an evolution of a seeing this box as a metaphor of the very first virtual object of the box and kind of expanding that out and to Able to go into all these different worlds and then slowly over time the experience kind of gets taken over By a little bit more of a sinister force and it kind of ends of this real creepy uncanny experience with you and a lot of different virtual avatars that are kind of Looking at you in a way that it's just trying to amp up the uncanny valley. Let's say So this is a piece that's in the center of gravity of ideas and very much exploring these philosophical concepts and taking very much a critical take but also a deep dive into history in a lot of ways. It's taking you through a lot of the history of computer graphics and the center of gravity of presence is very much into this mental presence of exploring different dimensions of these ideas and these concepts and taking these critical takes and you kind of have to really pay attention closely to piece together all the different puzzle pieces of the narrative and then actually through the conversation to get a lot of additional context for what the creators were intending with some of the different aspects of their piece. There's a lot of different dimensions of interactivity as you're kind of engaging with these different objects to kind of progress through the experience, but also having this deep sense of embodied presence as you have these hand tracking and also using the environment around you to really tell different aspects of the story as well. It's also worth mentioning that there's a quite exquisite camera obscure installation that they had where as you walk into this experience you see a inverted projection of the water that's just outside of this island of the VR island there at Venice and really quite beautiful installation that you get to see as a part of like sitting down in this room So there's a component there that's also a camera obscura that was very much also inspiring different aspects of their work as well that we kind of got into this Descartes dualism that they unpack more in the course of this conversation. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Dennis and Catherine, aka Sister Sylvester, happened on Sunday, September 3rd, 2023 at Venice Immersive in Venice, Italy. So with that, let's go ahead and Dive right in!

[00:06:15.351] Deniz Tortum: Studying at MIT and also was a researcher at the Open Documentary Lab and that's when I started researching VR and practicing VR and that's when my journey started and that's also kind of like around the time where Katherine and I met as well.

[00:06:33.648] Sister Sylvester: Hi, I'm Catherine. I go by Sister Sylvester. My background is in performance and I actually started writing about VR, not making VR, and I was writing about how much I disliked VR. I was living in Istanbul at the time, working with a friend. We had a commission to make a performance piece, but the EU wouldn't give the friend a visa, so we started looking into ways of kind of remote presence and obviously VR I guess it was around 2014 and VR, you know 360 video was really blowing up online, but we noticed that the kind of like 360 video around refugee content was quadrupling overnight every night and it was really kind of weird stuff that was it was like every organization in the world decided to parachute in and and make these videos that would be super unethical if they were in any other medium. But somehow because it was 360 video and there weren't any rules, these really weird things were being made. So I started writing about what I was seeing around that, and in the course of it began researching more what was going on in VR. And a mutual friend told me that I had to look at the work of one Dennis Torton, who was doing really interesting things, but also kind of approaching it in a critical way. And so I interviewed Dennis for that article and started to look at his work and read his thesis that he had written at MIT. And it was one of the first moments that I was like, oh, wait, maybe I'm not just feeling completely critical of this medium. Maybe there's actually some interesting stuff here that, like a way to approach it that I am kind of excited about. So, yeah.

[00:08:05.647] Kent Bye: And I'd love if each of you could give a bit more context as to your background and your journey into working with the medium of VR.

[00:08:13.179] Deniz Tortum: Yeah, my background is in film and not necessarily narrative film but also experimental film and non-fiction, especially experimental forms of non-fiction. And I think kind of like my first Step into VR was through interactive media and also kind of art games and I guess like right around like 2013-14 when making films I was also kind of looking into art games and especially there was this creative duo studio called Tale of Tales. They did amazing games like Graveyard and I thought their work was very cinematic but also pushing this visual language in ways that almost experimental cinema did in the 60s and 70s. And from there on I got more into media theory, interactive art, and that's how I found myself. MIT Open Documentary Lab and when I was there it was kind of like the VR hype was starting and I started researching that and all the festivals started programming VR, there were a lot of talks I think Nani de la Peña's piece at Sundance was like the year before so I guess 2014, 15, 16 there were all these first VR exhibitions in the festivals And kind of following this cultural moment, I also started writing on VR and I think I had two things in mind. One is that what is specific to the medium of VR, especially when it comes to its kind of narrative core and also its formal qualities. And I think in my research and in my thesis I took, in a way, a very modernist approach, kind of similar to what Rudolf Arnheim did for cinema in the 20s and 30s of looking at VR and being like, okay, what is very, very specific to this medium? What can this medium do? that any other medium cannot do and what is the unique affordances of this medium and I kind of started writing my thesis through that but also on the other hand there was like also the cultural moment which was that I think there were several things happening One was the main discourse around VR back then was that VR is this empathy machine and it was kind of clear that this was a, you know, kind of almost like a marketing discourse and its depth didn't really go that deep and kind of like started trying to, you know, unearth that a bit and also try to see what other cultural ideas there is in VR and I think one of the, also the, maybe the founding encounters of Shadow Time is also this interview from 2015. It's an interview that was published in the Wired magazine with Palmer Luckey. And in that interview, the interviewer asks Palmer Luckey, saying, what do you think VR is good for? What do you think VR can do? And Palmer Luckey, at this point, he's young, naive, but also kind of like he puts almost his thoughts on this interview very transparently and also I think with him there is, Carmack was there with Palmer Luckey as well, but Palmer Luckey in this interview says that you know the world has a lot of problems, it's a messed up place, there's a lot of inequalities, there's a lot of things that is not right but we can solve all of these. But if we perfect VR, we can give everyone the lives of the rich. We can provide everyone with a lot of experiences. And if you live a happy life in VR, you are living a happy life, period. And this interview actually, you know, it was kind of a shocking interview because he takes kind of dystopian ideas, which was like presented in Matrix 15 years ago, and which was like kind of the cultural moment, like even 15, 20 years ago, and then kind of represented as utopian ideas and I don't think many people had trouble with this that much and we kind of like started thinking through that of like okay what is what are the core ideas that in VR that might be distracting us or that might be giving us like false promises false futures and how can we talk about this how can we explore these further Yeah, so kind of I think my thesis then did it in a more formal slash narrative way of how can we actually break the presence of VR and how can we imagine it as a more like self-reflective medium where like also there's more critical agency for the user And then afterwards, I did two VR pieces. One is called September 1955. It's a historical reenactment of the Istanbul pogrom. It's a piece I did with Çağrı Zaman and Nil Tuzcu. And I did another 360 video called Flood Plane, which was at Venice Immersive in 2018.

[00:13:07.892] Kent Bye: A bit more context. It's your background and journey into VR.

[00:13:13.467] Sister Sylvester: Yeah, so my background, like I said, is in performance and I had been making a lot of work that was kind of lecture performance-y, video essay kind of influenced and a lot of it had been about technology. I mean, I think that the kind of empathy machine narrative was really like the biggest provocation for me in like wanting to write about how much I hated what was going on in VR and then became this point of discussion that we had in common where we were like, what else can it be? What else can we do? And also why is no one kind of rigorously attacking this completely false notion of what this thing is? Yeah, this kind of like marketing spiel around what this medium is that seems to, you know, just so much writing that can, like we've been through that, we've gone there, like Susan Sontag has happened, you know what I mean? We don't need to go back to thinking that this thing can put you in someone's shoes and that that somehow creates a moral response or that that somehow is like an ethical thing to do. But yeah, I've been making live performance work, but not usually characters or actors or anything like that. So weirdly this, in VR, this is the first time that I've ever done a piece with a fictional character speaking fictional words, despite the fact that I've always worked in like theatre and live performance. A lot of the work that I do in performance is playing with different technologies, maybe super lo-fi or like weird kind of tinkering type technologies, putting things together in strange ways. And what else to say? Like the encounter with VR, like I said before, was really through, like I was making live performance, these kinds of performative essays, but I was also writing essays and articles. And it began with this attempt to critique the medium. And then it became exciting to try and think of what a kind of video essay in VR would be, what a way of critiquing the medium from within would be. And then the other thing that also, I think, drew me to it and this kind of created this love-hate relationship with it is that My dad actually worked on simulation training programs in the 90s. So he was a soldier in the British military and he was working with the group that when the Americans created their simulation training program first and then the British wanted a version of it and so he was kind of like looking at things seeing what happened and when I was researching the article I was also researching militaristic roots of it and what You know, we have a lot of Ivan Sutherland in our piece. Looking at that very first, the article that Sutherland writes in 1966, where he kind of anticipates a lot of the computer-human interfaces, ways of relating, but he also has this really, really strange paragraph at the end. where he talks about the ideal display being a kind of mathematical wonderland where the laws of the world can be suspended and he talks about in an ideal situation there would be a chair, handcuffs and a gun and the chair would be solid enough to sit in, the handcuffs would be confining and the bullet would be fatal. and he said something like, this would truly be the wonderland in which Alice walked. And I just thought this like, at the end of this technical paper, this super cryptic, weird paragraph that harks back to the kind of militaristic roots of the technology, but also is infused with all this like weird 1960s thing and like the space that he was imagining was this kind of noir-ish interrogation room. Yeah, I don't know, there was something in that that was so strange and kind of enticing to start to pick apart. So yeah, I think like Dennis said, that cultural moment where you had this idea that, you know, these dystopian ideas of if we all live a good life in a headset, we're living a good life, it doesn't matter, there's no difference. There was that, there was Zuckerberg talking to the UN and those super weird photos of all these like dudes in suits at the UN with headsets on, while Zuckerberg wanders among them. And then there was Elon Musk saying that there's the one billion chance that we live in base reality and all these things were kind of entering the dialogue, entering cultural conversation at the same time and kind of became the things that we were trying to pull apart in this piece. But yeah, it's the first time I'm making VR. I think that was your question.

[00:17:20.803] Kent Bye: Yeah, I think that was actually one of the Gear VR launches at Samsung that had everybody that was watching the Gear VR and he walks in and sort of a super creepy iconic photo of, it was kind of like a surprise that he was there. And so it's just the idea that he's sneaking in without them noticing him. So it was kind of like this iconic photo that has come up for a lot of times.

[00:17:39.952] Sister Sylvester: It's super creepy, right? And yet it was being presented, like, there's a little bit of tongue-in-cheekness and critique around it, but, like, these super dystopian ideas were being presented as if they were real solutions to things that were going on in the world at the time. And I think that was also really fascinating to us. Like, what happened? When was this switch? We all know that these come from dystopian novels, but suddenly they're being presented as, like, exciting solutions to the problems of the world.

[00:18:05.370] Kent Bye: Yeah, and so you had said that you got a recommendation to check out some of Dan's work. Dan talked about some of the projects that he created. And so maybe you could talk about you meeting each other. And then you sent me last night, Dan, the piece called Our Arc. And so what kind of led to your collaboration?

[00:18:21.037] Deniz Tortum: Yeah, so I think the first time we met was when Catherine was writing the article called Voyeur Reality, which was published in the New Inquiry. And afterwards, I think a year or so afterwards, we then started talking about making a VR piece, kind of like a self-reflective VR essay in VR and looking at the history of it, but looking at also the affordances of it, the kind of narrative formations of it and one of the things we were also interested in is that like is there a way to kind of like present VR kind of like the potentials of VR within the VR and from there on I think our research led us to let us almost like a bit away from VR I think because one thing is that I don't think VR or virtual reality is just virtual reality it's a combination of technologies kind of visual embodied sonic languages and different media. So, you know, video games, live simulations, VR or films that use kind of these logics or, you know, the apps like Snapchat filters, TikTok filters. I think they all exist. in the same kind of computational culture universe. And I think there are also films like Tenet, which seems like instead of a plot, it has a game mechanic at the center of it. Or there's the new Harmony Korean film, which is currently premiering it. The Venice Film Festival, Agra Drift, which is, you know, rather than a film, it is more like a Grand Theft Auto cutscene that is shot in real life. So kind of all these things are emerging into this one kind of like common cultural language. and we were very much interested in that cultural moment and our research led us to kind of like invention of virtual reality but also the invention of virtual reality is also the invention of computer graphics because Ivan Sutherland invents this device called sketchpad and he's the first to draw lines on a computer and like draw shapes and he draws a cube and then the next logical step is how can you see the cube not on a screen but in 3d space as a volume and then that leads him to invent the virtual reality headset so kind of the History of computer graphics and history of virtual reality go really hand in hand. What does this kind of technological progress change in our culture, in our thinking? How does it affect our conceptualization of the world itself? These were the questions I think we were dealing with at first, especially with the film project we did, Our Arc. And that piece is basically, I think, founded on two ideas. One, or maybe just one idea, or that is a continuation of all this. But there is also, as Catherine mentioned, this interview with Elon Musk. And in this interview, he says that there's one in a billion chance that we are in Bayes' reality. And he voices this kind of idea, which is I think popular around more of the tech spheres, the idea of simulation hypothesis that the world itself or the universe itself is a computation and there is like kind of like we are in ancestor simulations or there's like a base reality that we are not in. I mean I think it's of course an interesting idea but the problem we had with it is what is the ethics of the world's richest man saying this and what does it mean for him to say this and for us like it is a really like act of like deferral of responsibility like what happens when you say the world is not real the world is a simulation then nothing matters and especially saying this in a time of planetary urgency, in the middle of climate crisis. What do you do with this? And kind of like related to this, we were also kind of looking at the history of computation, virtual reality, computer graphics. And you know, since the 60s, we are building these virtual worlds, computer generated worlds, we are populating them. but also at the same time we are destroying the physical world and kind of like these happen in kind of like this reverse graph to each other and what do we do with that as well and what does it mean to build these virtual worlds when there's you know massive species extinction there's massive heating and kind of our question was that can these be related in any way and I think that's the question we are dealing with and kind of like the question is like can this computational culture like is it creating a new cultural philosophical culture for us that is actually distracting us from the planetary urgencies or if these things are creating like blind spots in our thinking that blocks certain interiorities or certain ways of like forming collectives So I think those questions lie at the heart of our work and our research both in our arc but also in Shadowtime

[00:23:35.428] Sister Sylvester: We just add to that like another thing that we were looking at the connection between you know what Musk was saying about not being in base reality and I remember Dennis brought into one of the meetings the UN climate report where it was something like 90 80% of the technologies mentioned in it to mitigate or to prevent climate crisis were either didn't yet exist as technologies or were not scalable. So we were also looking at in relationship to like what does this kind of computational thinking like lead us to this techno utopianism where we're somehow relying on these technologies and then for like that in our arc, we follow these scientists at UMass Amherst who are trying to create 3D models of all animals on Earth, beginning with the ones most likely to go extinct. And there was something so beautifully kind of poetic and quixotic about this attempt to create this total archive, which is, you know, in many ways, completely fucking useless, because, like, my background is also, I'm an amateur microbiologist in my spare time. And obviously, there's such a back and forth between computing and biology, just in terms of language as well, like language gets borrowed back and forth, but each discipline is kind of outdated by the time the other discipline borrows its language. And so in that moment, I was also seeing this like, weird sense where these scientists were scanning an organism but they were just scanning the shell of the organism whereas in biology obviously all of the discussion is around the ecosystem and also the microbiome like the idea of looking at something as simply an individual is so outdated as a notion and yet the kind of computer side of it was still there it was still thinking of these animals in terms of like a single organism and what the appearance of them is and what the shell of them is almost So I remember also when we first encountered that archive, just spending a ton of time playing with the models and looking at them from the inside out and trying to think what it meant to, at this moment in history, preserve the shells of these organisms. as if that was somehow going to be a bulwark like we say in the film against their loss and just how kind of futile and poetic in a way it is and how much it kind of represents the ways in which we approach the climate crisis.

[00:25:48.039] Deniz Tortum: I think just I want to add one thing about digital life

[00:25:52.385] Kent Bye: Is that the company that was scanning things?

[00:25:54.266] Deniz Tortum: Yeah, in our arc, the film starts with Digital Life, which is a company that's scanning old animals. But they also see it as a kind of a necrophauna project, that when in the future, if we bring these animals back from the dead with their DNA, then we would have exact dimensions of them. But as Catherine said, it's kind of these animals don't live by themselves. They live in an ecosystem and they live in, you know, like other living organisms in certain conditions. There is a weird poetic defeatism in that as well. That is our hope in a way. And it is similar to the unscalable or unproven technologies of the UN climate report etc so we are putting our hope into paths that are not proven yet and I think we are like there's a lot of wishful thinking that all of this will work and I truly hope it works but we should be probably preparing and trying harder

[00:27:03.399] Sister Sylvester: Yeah, and also putting our hopes into things that we don't know will work, but also just seem, like, philosophically wrong, like the wrong way to approach things. And I don't know, it just ties into this broader thing that you were talking about, the Silicon Valley thing, also the George Church thing, the de-extinction programs, that, like, there's... You don't know who George Church is? Okay, this is another conversation. Oh, George Church from Harvard? Yeah. Okay, yeah. I work with students at university. I do a bio-art class and I had them this year looking at George Church's experiments and creating these kind of like speculative design responses to it and it was just so much further there. it's just like the ways of thinking and this I think connects to what we're talking about because we talked a lot about the philosophical stakes of these different technologies as well and what does it mean that these individuals like Elon Musk and like George Church come up with these grand schemes to save the planet through these means that are completely non-proven and yet kind of tap into a way of thinking that is so deeply individualistic and so kind of I mean, they're caricatures of themselves. So, yeah, like how do we get to that way of thinking? What is it in our kind of computational society that leads to these kinds of solutions? And one of the questions we started to ask in Shadow Time as well was what, and also in NAROC, was what ways of thinking will the headset lead to? Will simulation lead to? How will that change the way that we conceive of the relationship between ourselves our minds our bodies in the world? Once we can see the world kind of as this separate containable simulated thing like it leads to these ideas of base reality because we can see that Computer graphics are almost indistinguishable from this world. Then we think okay. So this world could also have been made So yeah

[00:28:50.609] Deniz Tortum: I can add one thing to that which is so kind of like in that Elon Musk interview he then backs his opinion in a very simple way but saying that you know look like 40 years ago we had Pong and now we have computer games that look almost like indistinguishable from reality and let's think like what would we have like in 50 years in 100 years and in the future there probably won't be any way to distinguish between reality and simulated reality, then how do we know that this thing we are living in is real anyway? So, kind of like that idea of these technologies do change how we conceptualize reality, how we think about it, And just to give another example, one book that really inspired us is this book by art historian Jonathan Crary. It's called The Techniques of the Observer. And in that book, one chapter of the book, he looks at the camera obscura. And he says that in 16th century, camera obscura has been a very popular tool for writers, thinkers of the era. And this is, I think, a bit of a poetic argument and probably like a bit of a stretch. I mean, there isn't a way to prove it, but his argument goes that one of the people who really loved being in Camera Obscura is René Descartes. And as you spend time in the camera obscura, you're in this dark room, and the world outside is projected in here, and it's upside down. But you see the world, but you're not in the world with your body. You're just kind of like, almost like you're in your mind, and your body is something different from your mind. So kind of he argues that like the whole Descartesian dualism is conceptualized through the technology of camera obscura and like the spatial and visual sensual affordances it brings and from that we were very very inspired with that and then thinking through that we were thinking if camera obscura did that then what would virtual reality do to us to our conceptualization of the world.

[00:31:02.120] Kent Bye: And you have a camera obscura installation as a part of your exhibition of Shadowtime here, which I think is very appropriate. So, yeah, well, I guess there's a lot to unpack in terms of Shadowtime. I don't know if we'll have time to unpack all of it.

[00:31:15.534] Sister Sylvester: You turn us on and we're just like...

[00:31:18.758] Kent Bye: So our arc is very much in the spirit of a film essay that's about virtual reality. And then Shadowtime is very much like a, I would say, one of the first virtual reality essays about virtual reality that I've seen, where it's really using the medium of VR to interrogate VR itself. And so love to hear about your process of having all of these concerns and critiques and questions, both about the implications of the technology, but also how to use the technology to provoke these deeper questions is to actually give people an embodied experience of these deeper interrogations that you're doing. So I'd love to hear about your process of starting to, you know, going back to the history of computer graphics, starting with Ivan Sutherland, University of Utah, you're in the lab, and then you have the sort of Damocles come down, you see the cube, and then you progress through a series of different scenes that are trying to explore this relationship between the Descartes dualism kind of model of both the virtual experiences, but also the physical experiences. So yeah, I'd love to hear a little bit about this construction of this deeper argument of using VR to explore and critique VR.

[00:32:23.734] Sister Sylvester: I think, I mean, one of the really great discoveries I think as we were developing it was that the cube could become our central metaphor for the piece. So the cube is the first virtual object and of course the sort of Damocles was actually kind of AR more than VR. You saw this cube in the space of the person. So that also became a really, I mean we play with a pass-through in the piece where you see the physical space that you're in. But the fact that that first moment involved seeing this virtual object and the physical space with the virtual object superimposed on it in three dimensions, I think that became a really great imagistic origin or place for our research. So in the piece, the cube, the one interaction that you have really is interacting with this cube. And the cube expands sometimes, the cube is the world and the cube is also something you manipulate. And the idea that the cube contains everything, the cube contains all possibilities of that very first cube, we started to try and think of it as this is the potential. These are all of the potential simulations, these are all of the potentials of this new world contained in this one cube. So within the piece your interaction becomes using this cube to kind of cat's cradle new worlds into being and kind of guide yourself through the world and I don't know just finding that one interaction became a way for us to have like this central metaphor for the piece.

[00:33:52.438] Deniz Tortum: Yeah, and I think one thing about both R-Ark and Shadowtime is they're both essay, like one is an essay film, one is an essay VR, if we hope it becomes a genre. But I think different than most of the other essay films and essayistic work, the voice is not our voice. Both pieces are a form of dark propaganda. Almost our arc is kind of the voice is talking, channeling Elon Musk, and in Shadowtime the voice and the character is channeling Palmer Luckey. and that also leaves the audience in a bit of an uncomfortable place or at least that's our hope to kind of like leave this experience with rather than answers but with provocations and maybe and hopefully with some sort of anger And I think that's related to our thinking around climate communication, which is a very, very hard thing to talk about the climate crisis because it is such a hard and taxing subject. And I think we are really good at shutting our minds off when it comes to the climate crisis. That's why we were thinking that we need to just reinvent new methods. There are many emotions around climate change, but a lot of people focus on hope, which is more the activist language, or fear, which is more the tumour language. Even though I don't like that term, I think that there's a mix between fear and grief, and I think we mix grief for fear, But then there's also all these other emotions that we want to play with, and one is anger. And then the others are like, what is, let's say, climate nihilism? What is climate excitement? What is climate hedonism? There's all these new questions, new emotions. emerging in this planetary era. And how do we explore it? How do we talk about it? And how do we reinvent ways of talking about it? And I guess that's one way we were trying to do in both pieces.

[00:36:11.817] Sister Sylvester: So in our arc, it starts very much as an essay. You think it's an essay, and then there's a shift at the end where we just shift to first person in the narrative. So it suddenly starts saying, we, and you understand that this person, that speaker, is trying to implicate you somehow in this. It's no longer just a history, it is a kind of propaganda. It's like a recruitment video for simulation theory. And it's been interesting having it travel, because some people get it and then we get some angry people, which is what we wanted. But we get people who really think, like we had reviews on Letterboxd being like, these people are creating propaganda for Elon Musk, don't listen to them, this is dangerous, which was really fun. And I think actually maybe more productive, like I don't know if it's more productive, but you know, so many pieces you come out and you sign a petition and like what's that gonna so there was something that we wanted to test out which is like a kind of provocation and then in this piece as well we worked really hard to try and find that place of ambiguity in the end and i don't know that it always works for everyone i mean we've had someone come out of it and say um you know i loved it i loved it but i think you're too optimistic and we're like wow did you watch the same piece And then we had someone come out yesterday who was actually a VC, which was interesting, who was incredibly angry at us. A venture capitalist who came out and was like, again said, this is dangerous, this needs a warning on it. He said, you're weaponizing mindfulness techniques. And you're going to brainwash people into accepting everything you're saying, which is not obviously all right. We don't want to brainwash people. But trying to also understand, not coming maybe from a moral place, but trying to understand what the seduction is of this. Because there is something incredibly seductive about simulation theory, about being able to say, oh, it doesn't fucking matter that everything's disappearing. We can move to this kind of shiny other space of creativity rather than desperate attempts to preserve. We can make new. We can start over. The seduction of being able to start over And that was something we really wanted to explore as well.

[00:38:12.234] Deniz Tortum: I would like to add one thing, which is regarding what you were saying about brainwashing. I think there's the aspects of brainwashing in both pieces, in Shadow Time and in Our Arc, but in a way, in an imperfect way. It has its holes and it gives many clues to the viewer that the kind of propaganda being presented in these pieces have a lot of arguments that's not working. And I think in that, that's why I think Shadow Time is an essayistic work because it is the self-reflective mode. It is trying to provide the viewers with tools to understand how VR can manipulate them and how to almost build some sort of intellectual cognitive defense mechanisms to actually interact with these VR pieces and keep a distance but also appreciate it. So I think that was an attempt that we tried as well.

[00:39:09.796] Sister Sylvester: It's training to resist brainwashing.

[00:39:14.497] Kent Bye: So I actually had a chance to see this piece twice, once at an exhibition that was happening at Onyx Studios during Trampaca around three months ago, and now here again with the final form and its full installation at Venice Immersive 2023. So I have like a dual, like some things are repeating and some things are different, but the thing that's coming up and being the striking moment is like this duality theme of like there's the realm of I don't know if you call, do you call it the real or do you call it the physical or do you call it the virtual? What do you, how do you create this dialectic? Okay, so you can't physical.

[00:39:45.698] Sister Sylvester: I know physical and virtual. Yeah. Okay.

[00:39:48.985] Kent Bye: Because I know that there is a dialectic sometimes to lean towards the real to contrast it to the virtual. And I'd be curious to hear your thoughts on Chalmers and his Reality Plus, where he's arguing that the virtual reality is a genuine reality. But yeah.

[00:40:00.633] Sister Sylvester: Before that even, I think there's something that we talk about and that I think is important with the virtual real thing, which is that all of these technologies of simulation very much have their roots in the physical world. So even though we talk about like cloud computing, no, it's like giant servers that are sucking up energy. the physical impact of the virtual was something that was really important to us. I think saying like the virtual is also very real in its effects on the world. So I think having it be physical and virtual rather than real and virtual is important. Yeah.

[00:40:30.871] Deniz Tortum: Yeah, and I think around that question, and you asked about Reality Plus, I haven't read it. I've only heard from friends and very briefly about that book, so I don't think I'm in a position to elaborate or comment on it. But I think in relation to that, maybe one thing is we are not necessarily interested in this ontological discussion around reality. We are maybe more interested in the ethics of having that discussion when there is a kind of a very physical emergency and kind of we are like asking if those ontological questions and those philosophical discussions Will they provide us with tools to, I mean, for lack of a word, like to survive or like kind of to keep the planet habitable? And, you know, the question of like whether we need to do it or not is like another completely different question. But I think kind of like there, this is the ethics of, you know, what questions would give us the right tools at this point? So yeah, I think that's a question that we are struggling with and we can't answer. But one kind of thing we think is that those ontological discussions around the nature of reality seems like they're acting as kind of distractions.

[00:41:58.837] Kent Bye: The reason why I ask that is because you have a quote that's on the outside that's saying, one realm is the realm of the heart, or one realm is the realm of the body, or I forget.

[00:42:07.379] Sister Sylvester: You're in two worlds at the same time. Your body is in one world, but your heart is in the other world. Did I get it right? Yeah.

[00:42:14.682] Kent Bye: That is an ontological argument, though.

[00:42:16.406] Sister Sylvester: Yeah, our character is making it.

[00:42:17.867] Deniz Tortum: Yeah, that's not us. That's the character almost trying to recruit you in the piece into kind of like believing that the virtual world is like a better world and like it's kind of this is where you should invest your life in. So it's kind of not our argument but the character's.

[00:42:34.099] Kent Bye: I was confusing the film essay as you as the author.

[00:42:36.141] Sister Sylvester: This is the brainwashing training. This is what you got to resist. Okay.

[00:42:41.720] Kent Bye: Okay, so what do you actually believe then? Because you said you weren't making the argument, it felt like you were making the argument in the piece, but that's the voice that you're speaking, so where do you actually stand? Or you say it's more of the ethics and it's not... The reason why I'm confused is because it seemed like such a central part of the piece that was sort of in there. So yeah, but then you're saying you're not saying it, but I hear it, so... Alma is saying it, the character. Elon?

[00:43:02.793] Sister Sylvester: Yeah, Elon. Alma or Elon is saying it, yeah.

[00:43:06.336] Deniz Tortum: What do we believe in? I don't know. I don't have my personal answer about the nature of reality. It doesn't matter that much to me. I almost think that's the wrong question. It's definitely not the wrong question, but it is not the urgent question.

[00:43:25.503] Sister Sylvester: we're asking in this piece. I mean I think Alma, the character, says that it's part of her technique to entice, to seduce you into the world of simulation, to seduce you into giving up on the physical world. She's trying to convince you that, you know, she's Elon-ing you, she's trying to convince you that the simulated world can hold you like the physical world can.

[00:43:47.272] Kent Bye: Part of the reason why I'm focusing on that is because I think from my own personal perspective VR has a potential to bring about some fundamental paradigm shifts that can bring about new metaphysical ideas and for me I'm very drawn to Alfred North Whitehead's process philosophy which is very much of trying to think about the world in a relational context and that order to really have humanity survive that the key thing is to have a shift in consciousness so that we are actually In more right relationship to the world around us and that you know, it's not going to be a technological solution It's going to be more of a cultural impulse that's going to drive the technology drive the laws and drive the behaviors and that the big potential that I see is that there is this shift that can happen that does see the shift into I see it as the virtual versus real type of framing creates this duality. And Chalmers is making the argument in Reality Plus that virtual reality is a genuine reality because we can have just as meaningful experiences. As an example, you have a social encounter in a VR. Do you come out and say, well, I had a virtual fake encounter with this other person? It's kind of like when you talk to someone on the phone, you don't say, I had a virtual fake conversation with this person. You say that you had a real conversation because it's actually about the relationship between there. And that as we go into these immersive experiences, it creates these aspects of our agency, of our emotions, of our relationships, of our mind, and also of our body and environment that is creating new relational dynamics where I think that's why I'm pushing that.

[00:45:10.841] Sister Sylvester: I would just say that when I was first writing that article and I read Dana's thesis, this was what was really exciting to me because it was proposing this other way of thinking about VR as a space where the laws of physics and the relationship between action and meaning could be changed and that could lead you to literally forge new neural pathways where you understand things in a different way. It's kind of like what you call an embodied metaphor.

[00:45:36.964] Deniz Tortum: Yeah, so basically kind of the idea there, my thesis was called Embodied Montage and the idea there is that it kind of like draws on ideas from embodied cognition that kind of language and thinking is very much tied to our bodily affordances and how our body is in relation with the world itself. There was a book I was pulling ideas from called Metaphors We Live By and very like simplistically kind of like the argument goes in this way that they say that the metaphors we use, the words we use are very much in relation to our physical affordances and you know there's a why we say I'm up when we feel energetic and I'm down when we feel sad. It's very much related to gravity in our body. When we're energetic, we stand up, it's a new day and all that. When you're tired, sad or depressed, it's kind of like the act of lying down there. So the body affects the ways of thinking and kind of forming thoughts, concepts. So the idea was there in kind of like the proposal in my thesis was that if we can change the bodily affordances, both how our body acts within itself and also our body's relationship with the world, Can we then use this as almost like a practice ground to think new concepts, think new ideas, and can we use VR as this platform to reach new ideas? So yeah that was that I think I wanted to say one thing about the reality plus idea of like kind of virtual reality being also providing as happy and satisfactory life as the physical world. I don't think I have a problem with that idea by itself but I think when we are thinking about these ideas we should also be aware how these ideas are being weaponized. And I think Palmer Luckey's interview, for example, or like kind of that Facebook's whole quest of like kind of like providing this life and like presenting it as like a happy life is a huge, you know, like a monetizable financial act of control. And just being aware of like what ideas are getting weaponized by kind of the power holders is this constant act we should keep in mind, I think.

[00:48:01.700] Sister Sylvester: And I think also, yeah, of course an encounter in VR is also a real encounter, and connected to that, the impact of these technologies on the world is also very, very real. So I think the idea that Zuckerberg and Pamelaki saying, give everyone a headset, one of those quotes was literally like, only the rich will have access to a smog-free sky, but everyone can have blue skies in the headset. So, like I said, it's not that VR is part of the world and the real encounters and the meaningful encounters you can have in VR. It's the way in which the proposal is that the underclass is placated by this technology in order to prevent the need to solve the problems. the world and to allow access to the physical world only for those very very you know 0.1% and everyone else is placated with this technology. So I think that's where our critique is not in saying that the virtual isn't real.

[00:49:03.066] Kent Bye: We talked about that very first scene and then you move into a series of 360 panoramic shots that are stereoscopic that then have, as you interact with it, you have this kind of modulation and then eventually you end up with a scene with a lot of bodies being mirrored across. You just turn your head and you have this whole crowd of people. And so in the conversation I'm talking to you I sort of understand that you're making an argument but you're also doing it in somebody else's voice and sort of a brainwashing reverse psychology kind of argument where you're making the argument that you're kind of arguing against in some ways.

[00:49:33.053] Sister Sylvester: It's a provocation.

[00:49:35.952] Kent Bye: So you're providing that provocation. So Marnie, if you could tie how the interaction of going through these different photogrammetry scenes that are spatialized that then have these AI style transfer as a second phase of that, and then the final scene. I'd love to have you tie together what you're trying to say in the story and how that relates back to the visuals that you're seeing. And so just so that you get a clear arc of how the entire experience is unfolding.

[00:50:00.568] Deniz Tortum: So in the beginning we start with just like our hands forming and a cube forming in front of us. Ivan Sutherland's cube and the hands are the first hands that Catmull did at Sutherland's lab in 72. So kind of like you're with these like first virtual objects, a cube and your hands. and the piece starts there almost like at the kind of like beginning point of virtual reality and then we find ourselves with Alma who's kind of this recruiter and then she talks to us and then afterwards we find ourselves in the lab of Ivan Sutherland and there's the cube and there Alma says that look at this cube and kind of imagine all the past and all the future contained in that cube And in that scene we were thinking that how would it have felt in 1968 for the people who wore this headset, looked at this very simple cube and what type of futures they saw in there. And we think they saw kind of a clear future in a way and they saw a lot of things. And we were trying to trace back to that moment of how would it have felt to see just that cube and imagine a future. And from there on the next scene is basically imagining that future, like you pull the cube and with every pull you're in a new environment, kind of a scant real environment and you go through these environments with every pull and this cube is able to generate many different environments, many realities and the future itself. And then slowly the cube grows up and you lose control of this and then becomes this kind of AI generated sequence of like the world that starts creating itself and you lose agency there. You're in this just like these worlds that is constantly creating itself and almost like you get lost in that scene and lose agency. and in the last scene you find yourself again in this big hall and there's a cube and there's like all the other people kind of like these ghostly figures appearing and there Alma is like doing her last trick of brainwashing you and kind of talking to you about you know, like kind of why, why this world, why the physical world is not the only world we have and why there's like other worlds and why this is also a comforting idea and kind of like I think that comfort is also what is very seductive even to us and we are kind of playing with that and trying to push the brainstorm to the last moment and ending the piece there.

[00:52:40.300] Sister Sylvester: And Alma does use the word real. She says, if this world was real and if this world was all we had. So she is making that argument. But yeah, I mean, exactly what you just said, but the hands become human agency. The cube is like all of this potential, but at a certain point you lose control. And it is both kind of overwhelming, but also seductive. but also you lose track, like there's no meaningful narrative in those AI worlds, like all of human history is kind of mushed together, so like a Buddhist statue actually becomes Jesus, becomes like we have like Islamic tiles on an old cathedral, and then I think like past and future and narratives become so confused in that. I don't know, there's like a kind of a sense that nothing has any meaning anymore because everything is everything. And so hopefully that propels you into the last scene with a sense of both being seduced by these images because they're beautiful, but there's also something that's lost in just such a mass of tangled webs of image.

[00:53:40.713] Kent Bye: Yeah, I think I had missed in both our arc and shadow time that there was like a voice that was not your own because it starts in the our arc is a very traditional film essay and then I Missed the twist at the end because I was watching it last night a very long days of Venice But also here in shadow time I didn't get that there was a sort of a narrator that was different than your voice as well because I don't see her I don't see her embodied and She's in the scene, I don't remember seeing anybody in the scene

[00:54:05.452] Sister Sylvester: You seen the new one or you just saw the old one?

[00:54:07.053] Kent Bye: I saw both.

[00:54:08.234] Sister Sylvester: Yeah, you sit. Remember there's a woman who appears right in front of you. Oh, that's right.

[00:54:11.697] Kent Bye: Okay. Yeah. Okay. Okay. I remember. I don't know. Was she in the first version? She was. Okay. Okay. So, okay. All right. Okay. I remember that now. Okay. So there's, there's little things that I didn't realize that the film essay part was like a character's voice rather than the author's voice.

[00:54:26.056] Deniz Tortum: But just to add to that, I think we really want that ambiguity. It's because I think both pieces work better when that voice has that tension of like, oh, are they being for real? Or are they not? And are they critical or not? It's not an obviously critical piece. And I think this is one of the few interviews we are giving about both these pieces, because our intentions in a way take away from the experiences.

[00:55:01.060] Sister Sylvester: We shouldn't have told you any of this, you have to erase it all. But I do think it's also part of us, like there's part of me that I think that's why you believe it, because it is. Wouldn't you say?

[00:55:11.348] Deniz Tortum: Like it's part of what... There's that tension that we are both critical but also very seduced and kind of, you know, we go back and forth and we ourselves like don't really like figure it out like how to deal with this or like we don't have an answer of like what is the right thing to do. It is just kind of like, yeah.

[00:55:32.946] Sister Sylvester: I see the appeal. I'm not, but I see how seductive that solution is. And I see that kind of, I understand it. So yeah, it's definitely part of us.

[00:55:46.513] Kent Bye: It reminds me of the Don't Look Up movie where there's a whole strand in that story where there's technology that's going to save everything and then the technology doesn't do anything. But yeah, I guess as we start to wrap up, I'd love to hear each of you tell me what either the ultimate potential of virtual reality might be or what you could see as the negative aspect of that, what the dystopic potential of the technology might be. You can choose.

[00:56:10.167] Sister Sylvester: I would say that the piece says that. I don't want to say anymore. I think we've given too much away already. I don't want to give any more away. I think that, yeah, I think we purposefully tried to end the piece on a really ambiguous note. And I kind of want to leave it with that ambiguity.

[00:56:26.102] Deniz Tortum: I must just say more. Sorry, sorry, Catherine. I just want to say that when we think about virtual reality, when we think about these particular technologies, the promise of virtual reality or the future of virtual reality is not separate from all the other futures. It's not separate from the future of the world. So I think probably the important thing is to think of all these things together. Because I think we tend to think of for example virtual reality and I know you don't can't but like with these technologies we tend to think about these technologies as like separate things which has its own developments progresses and we also think kind of this development and progress has no end but it does and how do we imagine it's with the world itself these futures all together. And I think what you were saying about is virtual reality, can virtual reality be a tool for shaping a new consciousness or shaping kind of a new worldview, is a question we are very interested in as well. But we are also just thinking that like we should be aware what else is at stake. Like that's not the only question we are dealing with.

[00:57:42.740] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?

[00:57:47.491] Deniz Tortum: Just thanks a lot. And this interview has been great. I don't know. I had a lot of fun.

[00:57:54.216] Sister Sylvester: Thank you, this is great. And also thank you for seeing everything. You go around and you see everything. It's incredible.

[00:57:59.541] Deniz Tortum: I also want to add that I was really impressed by everything you wrote and I was even impressed that you saw everything. I don't know anyone who is this committed and doing such hard work and it's really important and special. So thank you.

[00:58:16.754] Kent Bye: Yeah, well, I think it's important to see the work because I think the reflections are really a deep metaphor for what's happening in the industry of where things are going. So I really appreciate the more critical lens that you're addressing all these issues. And thanks for taking the time to look at it through that critical lens and to take the time to share some of your thoughts. So thank you.

[00:58:34.114] Deniz Tortum: Thank you. Thank you.

[00:58:36.805] Kent Bye: Thanks for listening to this interview from Fitness Immersive 2023. You can go check out the Critics' Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot of digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listen-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.

More from this show