The Sundance Film Festival’s New Frontier Program featured over 20 different VR, AR, and AI experiences that were pushing the boundaries of immersive storytelling. After seeing all of the experiences, I had a chance to unpack and analyze the larger themes with chief curator Shari Frilot. The common theme among all of the storytelling experiences is that they’re exploring embodiment and interactivity in different ways, and how VR takes the audience member on an inner journey of presence. Frilot refers to this inner journey as a “superbody reflecting pool” in that these technologies are extensions of our bodies (ala Marshall McLuhan’s Understanding Media: The Extensions of Man), but that it’s also a mirror that allows us to perceive ourselves and understand who we are becoming.
We talk about the storytelling experiences that use artificial intelligence to foster social interactions based on cultivating a piercing vulnerability, the live performances that blend the modalities of cinema, live storytelling, & music to create more of a “living story” experience, the themes of women’s empowerment that emerged in both the VR and film program at Sundance this year, the social VR experiences, experiences exploring sparse symbolic representations that require you to project your imagination, the differences between objective facts and the truth of a story, and the humility and wonder that comes with not knowing where this journey of immersive storytelling is taking us.
LISTEN TO THE VOICES OF VR PODCAST
Here’s Frilot’s curatorial statement:
Poetic statement from curator @sharifrilot talking about this year's #NewFrontier theme of SUPERBODY REFLECTION.
VR is a mirror that enables us to perceive ourselves and develop new intelligences.
I'll be sharing more impressions after I see the rest of the experiences.#Sundance pic.twitter.com/nUsZvZtDHC— Kent Bye (Voices of VR) (@kentbye) January 20, 2018
Here’s a Twitter thread that’s explores the concept of a Heroine’s Journey:
VR's embodied presence is a receptive/feminine quality. I think the VR medium will likely emphasize the heroine's journey in new ways that's different than Campbell's monomyth, which is centered masculine expression of agency.
What's that look like?
Here's Maureen Murdock: https://t.co/MkKFGa0oTE— Kent Bye (Voices of VR) (@kentbye) March 4, 2018
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye and welcome to the Voices of VR Podcast. So I went to Park City, Utah again this year to the Sundance Film Festival to check out all the latest virtual reality experiences that are being shown at the New Frontier section. After I had a chance to see all the different experiences, I had a chance to sit down with Shari Fulo, who's the lead curator of the New Frontier section. And this year they had a lot of different virtual reality and augmented reality experiences, but they also had a couple of artificial intelligence experiences that were blending the concepts of immersive theater and creating these different interactions with people in a new way. And so I had a chance to sit down with Shari to talk about some of the larger trends and themes that were coming out in the course of this year. And if I were to try to summarize it, I'd say that human embodiment is something that is new within the context of storytelling. And I think that they're taking some lessons from immersive theater, but there's all sorts of different ways in which the hero's journey that we're used to, maybe we're starting to cultivate what could be considered a heroine's journey, which is less about the outward journey, and it's more about an inward journey about what the experience feels like from your own direct sensory experience. So I think in different ways, all of these different virtual reality experiences that were at the Sundance New Frontier section is exploring this question of embodiment and what immersive and interactive storytelling is going to look like in the future. So, we'll be covering all that and more on today's episode of the Voices of VR podcast. So, this interview with Shari happened on Tuesday, January 23rd, 2018 at the Sundance Film Festival in Park City, Utah. So, with that, let's go ahead and dive right in.
[00:01:48.390] Shari Frilot: Well, I'm Shari Frillo, and I'm a Senior Programmer here at Sundance Film Festival, and I'm also the Chief Curator of New Frontier. So at New Frontier, what we do is we're basically tracking innovations in storytelling as they occur and erupt at the crossroads of different disciplines. Art, film, technology, music. And this year, it's rare that we have a year that fails us. I don't think I remember one, and this one is one that really kind of succeeds in spades in terms of not only bringing a certain quality, an elevated quality to stories that are being told through technologies like VR, but we're also seeing storytellers engaging other technologies as well, such as AR and maybe even most notably AI this year. You know, it's something that I've been hearing about to a certain extent in VR circles about AI and engagement. kind of pointing at it across the table, knowing that it's coming our way, our way meaning the creator's community way. So it's really exciting to see these two works, Frankenstein AI, Monster Made by Many, and Tendar, kind of come to us in a very full-fledged form.
[00:02:57.988] Kent Bye: Yeah, I was noticing that like last year versus this year and also in the wider realm of virtual reality and I've done about a hundred interviews about AI and it's probably about a year or two or three years ahead of where it's been integrated into an experiential storytelling and there's been more in the 2D realm and experiences like Bad News and Eisenhower and more in the gaming realm but I think in terms of storytelling it's this opportunity to start to weave together a number of different things and I see that immersive theater is an influence that Frankenstein AI is definitely sort of adding a little bit of like creating these opportunities for two people to connect in a unique way and then to then present I guess their emotional beats either from 10 to AR or to Frankenstein AI into the AI and so then you get this feedback loop of seeing how the AI is being trained. And so it'll be interesting to see the Frankenstein AI presentation tonight, to see how that all fuses together and what the AI has been able to learn from us humans. But also another experience that comes to mind is this blending of immersive theater and subtle AIs with wolves in the walls, where you start to have this visual storytelling. You have visual storytelling in film, but I think in VR you have a little bit of spatial storytelling, which is much more about how the bodies are moving through space. And so, yeah, just curious to hear your thoughts on some of the other trends that you see that's emerging from storytelling that are featured this year.
[00:04:21.310] Shari Frilot: Well, there's certainly the emergence of performance and the theatrical, that intersection of VR and immersive theater continues to evolve, and we are seeing some of the evolutions of that this year. Wolves in the Wall is certainly one of them. That manifests entirely inside the headset, but also a piece that exemplifies this as well is Hero. which manifests in a fully built out environment where you go in a fully immersive VR experience that is completely mapped on to a real life tactile environment. That involves actors in a very powerful way. And to a certain extent, you could even say that VRI, you know, the Swiss project that's here, it was a collaboration by Gilles Jobin and the Art Anim creators. You know, that's coming from the tradition of dance and choreography. You know, the narrative structure is loose and interactive and participatory. five people get to do it at the same time and just, you know, custom hardware and software that's really kind of bringing a really art and in signature bleeding edge avatar technology which, you know, the fidelity of which is so crisp that I could throw a ball to you inside of this virtual environment, you would catch it and what they're doing is really amazing, but that's also engaging with the theatrical, because you can touch another person, and you're engaging with that other person, and oftentimes people are in there dancing with one another, and it's a very vital and growing movement within VR storytelling that's very exciting. It's building upon what Toby Coffey's been doing with the Royal National Theatre, with Draw Me Close, and with Alice, the virtual play, I love that this momentum is getting stronger at this intersection, that it's becoming more muscular because the human interaction inside of the virtual space is incredibly powerful and it's very exciting to see these creators working with this. It just makes me think about another thing. It's not really theatrical so much as just in terms of the interaction of the audience physically in the space, in transcending ways. And I'm thinking about the piece Awawena by Lynette Walworth. There are actors inside, but certainly it's a mixed reality piece. You know, you're going between the representation of the world around you, you know, this like that you have around you, you can see the room inside of it, and then the digital elements pop into it, and then it becomes 360, but there's a chapter in this work that is really notable, and I was actually waiting, and maybe you know some other ones, but I've been waiting to see a creator take this on, which is to say, in VR, when you move your body through something physical, like a wall or a table, you get the shivers. It's almost an unbearable feeling like it's really hard to jump off the ledge And when you're on a ledge in VR, even though you know that you're not gonna fall off of a story, you know skyscraper it's really hard to do it and and so it happens though in VR and it's oftentimes because the build is a little janky or there's some mistakes or You know, really, that's generally what it does. But I've been waiting for a creator to take it on in a narrative sense, to use that almost transgressive encounter, transcending material object, to serve the story. And Lynette Walworth does this, you know, in her story. about a shamaness who is telling about her coming of age as a shamaness and what it meant to train with the shaman of her tribe and to take the medicines that open up the sensorial pathways and what that meant. Lynette was able to use Moving her audience's body through a material a seemingly material tree To transcend to give you the feeling of what it meant to transcend the material world into the world of energy and waves Yeah, I'd say that
[00:08:35.528] Kent Bye: what we were talking about before was a lot about the embodied presence and actually feeling like you're in your body. And I feel like in the experiences that I've had, like there's the Ledge done by Studio Transcendent in collaboration with 2-Bit Circus, where they create this scaffolding where in VR you're going up and then they ask you to step off. And so The tricky thing with experiences like that is that the experience is actually asking you to break presence. Because you have this sense of embodied presence and your body is telling you not to do it. And then by the mere fact of you doing it, you actually sort of break through that, but then that also breaks presence. And so most of the times that I've seen stuff like that, asking you to transgress your normal body wisdom to not sort of do something that could kill you, that usually sort of takes you more out of the experience in some weird way. Another one of the themes that I see in a lot of the experiences here this year is starting to almost like deconstruct reality and start to have these almost symbolic or archetypal representations of different things. Like in Wavana they have the point cloud representation of the earth that allows you to sort of see the ideal forms in that way. Also in Dinner Party there's a scene where You have this experience of the alien abduction, which is sort of a polar opposite of like an enlightened experience versus sort of a depth underworld experience for Barney Hill. And so you have this juxtaposition, but it's not a literal translation of the things they actually saw. It's more of a phenomenological interpretation of that. Summation of force also is about movement and motion and starting to at some point Deconstruct the movement and see how things change over time so you get to see things moving and then finally dispatch is also Starting to you know not have a literal translation of what's happening in the in the experience But again, it's at this higher level symbolic or archetypal expression, which is very sparse but allows your mind and imagination to fill in a lot of the gaps and I think that's a sort of a theme that I see that, you know, there's on one end striving towards photorealism, and on the other end you have this sort of like symbolic representation of this, and I think Zikr also does a lot of this as well, as you are trying to recreate this experience of an ecstatic Sufi ritual, but, you know, finding ways to create these architectures of point clouds and Shapes that allow you to get an essence of what it's about without being so literal So I think there's a number of different experiences that have done that but yeah I don't know if there's any that I can think of that deliberately you're sort of just smashing through a tree to sort of Represent this almost going into an altered state of consciousness or the transcendent realm
[00:11:05.301] Shari Frilot: I was just going to say that it almost challenges us to even consider or perhaps reconsider what presence is. Particularly in Awavana and also in Zikr, these are both works that are talking about spiritual experiences that can take you deeper into the underlying energy field That is the root of all reality of all presence, but it's in these point cloud configurations where it becomes abstract To an extent that it actually with more fidelity represents the space-time continuum and and how you know our all of our bodies are made of stardust and molecules and and protons and photons and and you know, neutrinos and all these kinds of things, this cosmic dust, which is as real as anything, probably even more real, you know, to consider that configuration as a presence is very interesting.
[00:12:00.746] Kent Bye: Yeah, and one of the things that, covering virtual reality for me, I start to The question of what is the ultimate nature of reality comes up. And talking to a lot of different people, I think some people, especially in the science community, see that maybe base reality is a layer of math and symbols. And maybe there's a layer of archetypal, symbolic, mythological story level of reality. And then there's the actual reality of what we're seeing with our direct experience that we can empirically observe. And I think that a lot of these VR experiences are in some ways trying to give us access to the sort of deeper underlying patterns that are building up the nature of reality. And the two experiences that come to mind are both the spheres as well as elastic time, which in some ways both are in their own ways exploring the nature of black holes. One is a mini black hole that's bending the nature of space-time within your own direct context, but also it records you. So then you have this third-person witnessing consciousness of yourself, of a point cloud representation of all of your movements that you just did, and you get to witness yourself as the structures of space-time are being warped. And then on top of that with spheres, you're able to sort of have this direct experience of both witnessing the effects of gravity waves, of bending sort of the structure of space-time, but also going deep within a black hole to get this more poetic or metaphoric experience of what it means to see the spaghettification of light within the context of a black hole and see some sort of representation of infinity in that way. And so I think both of these experiences are starting to look at mathematical structures that are otherwise invisible to us, but that we're starting to have these direct embodied experiences of this, such that they're happening at these huge scales and so far away, but yet we're able to start to see how these underlying patterns of reality are able to kind of start to change our conceptualization of giving us an embodied metaphor of the nature of reality.
[00:13:53.980] Shari Frilot: Yes, and it is very particular to VR in its ability to frame these. We once had a more theoretical or conceptual relationship to the expanse of the universe and cosmos, to space-time, to the fabric, the physical fabric of everything that exists, to put that inside the frame of human existence as opposed to the theoretical thought. And that's what is presence, what is being present. It's being centered inside of your own consciousness in such a way that you can understand the relationship where you stand, where you are, to the world around you. So it's key. It's key to how you define and how you perceive what the world around you is that makes you present. And there's something about VR that brings very powerfully the ability to understand your world around you in much more expanded ways that these particular works are working with. Because they're working in the documentary form. So they're tapping on scientific fact and data. They're tapping onto live action capture of cultures and practices. and not only bringing your body into a practice or a space that you could never be or very hard to achieve to be there or could never be there on the outside of a black hole, but through the power of narrative, being able to create an effective frame around your body, using your body, that puts you right there and it allows you to accept this or to to claim it as your domain. It allows you to claim these expansive experiences as your own personal domain and thereby expanding the framework of what it means to be present.
[00:15:48.146] Kent Bye: Yeah, and I think that the other live performances and films that were programmed within New Frontier this year, with Corey Maccabee's Deep Astronomy and the Romantic Sciences, and then the Kronos Group documentary that was here, and then also Star, that was a film that both of us just watched, which is essentially kind of like taking every starry night shot and film for the last 100 plus years, and sort of just putting them together, and you kind of see the depiction of the night sky evolve over 100 years. But I'm just curious to hear your perspectives on these blendings of both like a film that's sort of giving you an experience that's sort of non-traditional, but it's more experiential in that way, but also this blending of with both the deep astronomy and the romantic sciences and the Kronos Quartet of A Thousand Thoughts. that they're kind of blending both live performance with music and sort of visual imagery in a way that I think is kind of like Together they're kind of fusing together something that's a little bit new or different than the traditional mediums each individual are kind of Using the unique affordances of each one and I'm just curious to hear some of your thoughts and reflections after you know seeing some of these performances It's a very interesting question
[00:17:01.178] Shari Frilot: You know, it makes me ask, why is putting together performance and the cinematic element more effective at doing this than simply watching a film? Which there are many films about space, time, and travel, and putting you inside of a story that's expansive and far, far away. But at the same time, when you're watching a film, you lose yourself in your imagination, but you don't necessarily lose your body. When you're in a performance, when these kinds of themes are taken on by the cinematic and the performance, I think there's something that happens because of the live performance aspect of it. The story is not outside of your body, it's actually in the room with another person. We're very, very used to understanding the relationship between two people, myself and the other person in the room, hyper aware of that. It's like the vitality of life that's there. But then you have this cinematic element to it that moves the arrow of time through the narrative that gives a more expanded experience of this rubric of understanding of being present.
[00:18:11.853] Kent Bye: Yeah, the way that I make sense of it is through this, the elements of natural philosophy, the water, air, fire, and earth, and that the fire element is the live, it's happening, it's a performance, and it's sort of like, it's got this vibrancy, and the earth element is the embodiment of the performer that's there, and I think that There's a level of embodied communication that happens when you see somebody live performing you see a lot of more Subtle body language cues in terms of how they're moving their body through space and I think that can get lost a lot of times if you're just watching it on film or if it's just a voiceover so you have that level of embodiment and the water element being the emotion and the rhythm and the music and sort of the overall the emotional presence and emotional engagement within the experience I think that is something that is much more receptive and typically the strength of film but by adding the level of the embodiment and the fire element of the live performance it adds different components and I'd say the thing that's missing is the air element which is the audience's ability to make choices and to decide and you know it's their mind is being stimulated and there's also a social dimension of a shared experience within the context of a room and hearing the audience reaction so there is the error but in terms of you know a video game is very much air and fire in terms of like making choices and taking action and I think that my prediction is that where things are gonna be going is much more of a you know right now a lot of these performances that I'm seeing are very well scripted out they've got everything written down to the word and kind of like rehearsed and that like I there's not a lot of ability for the audience to have a direct engagement with that performance and to have a little bit more of a live improv or to do any type of live action role play with the audience in terms of participating and having a loose framework of what may happen as a potential, but giving that audience the ability to sort of subtly direct the trajectory of the story into different branches. I think that It's a challenge in interactive narrative that of giving the audience agency versus being able to control that narrative tension. And I think it's the spectrum between an authored narrative, which is very controlled of that narrative tension, and a generative narrative, which is allowing sort of an open world exploration. But I think one of the unique affordances of the live is you have this ability to kind of allow the audience to participate and leave traces within these experiences. So that's where I'm kind of like interested to see, like even in the immersive theater world, you know, Sleep No More, you're able to kind of just do that by running around the space, but that doesn't actually change the narrative at all. that's where I think that both the lessons from immersive theater but also like artificial intelligence that's going to allow the ability to kind of do more sophisticated ways of doing that. But for me I'm always interested to come to New Frontier to see how these live performances are evolving and what new things are starting to add together. And this year it was the addition of not only performing and really moving their bodies, but also music in both of these instances of kind of like having a little bit more of a poetic way to take a moment to sort of communicate something symbolically in a way that's a little bit more efficient and going towards the heart rather than just the mind.
[00:21:18.505] Shari Frilot: Yes, I mean, what Sam Green has done with Jobini and the Kornos Quartet is just beautiful, really. What they're saying about the ephemeral and the momentary, the fleeting, and how profound and beautiful and fragile those moments of gravity are with us in each point of time. That's another way of kind of addressing this sense of presence and what does it mean and just how magnificent it is, how heavy it is, and also just how fleeting it is, just completely temporary, and how valuable that is. That's interesting. And it is interesting too, you know, with Corey, he kind of flips it on the other side and goes for the cosmos and, you know, kind of brings this idea of the cosmos is as large as the contents of our imagination. And, you know, what we know about our imagination, how our brains work, neuroscience, it's, you know, it's certainly one of, There are a lot of headway into these sciences, but it's the thing that we know least about ourselves. And so once again, the grasp that anyone can have on what is and who we are and what is the present moment, as opposed to what is the next moment, is fleeting. But at the same time, it's as expansive as the contents of our minds, it's as expansive as the cosmos. And it's, you know, these two works are really, as all good narrative, embodies the human condition in this way, but is able to really, they expand the framework through, really through narrative and form, in terms of being able to bring their audiences to inhabit something very large, but at the same time very fleeting. That is the human condition.
[00:23:21.777] Kent Bye: Yeah, one of the other things that I was really paying attention to over this past year was just kind of like this almost shift in the culture that we've had with the Me Too campaign and Time's Up and just looking at the issues of women's stories, whereas things that used to be at a level that needed to be empirically proven, there's like a whole level of domain of a human experience and stories that were kind of like recontextualized in a way that suddenly we started to actually like listen and take into account the direct experiences of all these women. And within some of the pieces that we're showing at, in the New Frontier section specifically, there was a number of themes around empowerment for women, whether it's some ladies VR or in chorus you're embodying these kind of like Amazon warrior women or just the deep explorers NASA that just features some women astronauts training to potentially go to Mars and so but I'm just curious to hear some of your thoughts on some of these deeper themes that kind of emerge over the last year and how they've kind of how you've seen them play out in some of these other VR experiences.
[00:24:25.699] Shari Frilot: Well, I found it really remarkable that as this movement of the Me Too and maybe most notably Time's Up, I call it the women's revolution. I just call it the revolution because I think it's not only just about women. I think it's a revolution for our society that entirely includes men as it includes women, but it's addressing that relationship. I find it really remarkable that, because generally what happens in society gets reflected in our films and the production, the fruits of production, two to three years, maybe one to three years later. So it's amazing that a piece like Awawina shows up right on time, just when this revolution is turning a giant corner. And, you know, this shamaness is telling us the story of the evolution of women in her tribe and how it was horrible for women and had to deal with violence and abuse in the ways that American women and women around the world have dealt with. And her journey to becoming legible by the shaman, and just through her own sheer will and determination, making a decision that she's going to, understanding that her fate is to become a shaman, and then making a decision to go for it, and transcending the cultural boundaries of her tribe to become the first woman shaman. you know, right on time. You know, I mean, Time's Up is not even a month old, you know, and here it is. It is really remarkable. And, you know, these stories are being, they're being manifest in the festival at large as well. You know, with Gloria Allred, the RGB, you know, there was something in the air for these filmmakers to kind of look back at the past and take stock of, you know, where we are legally on these kinds of things. and look at the history and what's happening right now is based on decades of work and advances and successes and failures of a lot of different people working on this. But there's something extra special when it happens in a new medium that also has the participation of so many women in its leadership of the developing industry. So it's just such an exciting time and it's exciting to see the convergence of Women and and people of color for that matter I mean really those have been that have been pushed to the outside by a patriarchal system Start to come together to shift the tide and part of that shifting the tide is this new technology and this new media landscape that allows us to tell a different story in different and very powerful ways and
[00:27:11.323] Kent Bye: Yeah, the experience that comes to mind is Dinner Party, which I think kind of explores both the experience of a direct experience of something, you know, an alien abduction, which, you know, still to this day is a little sort of on the fringes of like, well, did that really happen? And then the way that the filmmakers had really created a phenomenological expression of that experience, but also the juxtaposition of the wife of Betty and then Barney and then the husband Barney Hill, it was an interracial couple. And so the black experience was that it was kind of like a terrible experience for him. And I think in a lot of ways, what they were saying is that this is also kind of like, listening to a lot of the tapes of that source material was kind of like reflective of how he was being treated in the culture at the time. And that it was almost like this reflection of this peak ecstatic experience from this woman of privilege and power versus her husband who had almost the complete opposite experience. But that in a lot of ways, the possibility for virtual reality to be able to Express or tell the stories of these types of experiences or maybe some sort of artistic interpretation But give you much more of a flavor of what it kind of felt like without it sort of being almost literally expressed but there's some sort of like Artistic translation of that that is happening in this piece and I think a lot of other pieces as well. I
[00:28:31.244] Shari Frilot: Yeah, that's The Dinner Party. The Dinner Party kind of reminds me of Rose Chochet's first work in this way. What was that called?
[00:28:42.354] Kent Bye: It's called Perspective?
[00:28:43.936] Shari Frilot: Perspective, but it was the first one. the party or the, I can't remember. Well, Roche's first VR, you know, really put you in the shoes of the girl who got raped and put you in the shoes of the boy who raped her, right? And played the same exact scene through these two different perspectives. And, you know, there is something in that broke open this area that I just sat there and watched people come out over and over again, you know, realizing there wasn't language to use to reflect how this piece affected them. Because we hadn't really developed it so much, you know. With Dinner Party, as I think you've adroitly observed, the experiences in this account not only were really different. The husband went through hell. His white wife went through this exuberant experience. And it's directly related to what was happening with them in the 60s. I was actually surprised that he could even live in that way because it just was not accepted at all. But we know this, right? We know this intellectually. We have the history that we can read and access. But what does it feel like, really? And not so much in a way to experience somebody else's experience, to have somebody else's experience, it's really more to understand what the facts are, you know what I mean? For your experience. And this is a distinction between the concept of the empathetic, which I do believe, stories are empathy machines, that's what stories do. But I think that the value of what stories are able to do in terms of giving you the experience of somebody else's situation is to understand who you are. Not necessarily to claim to be able to know what somebody else felt. Because you'll never do that. It's not really even about that. It's about understanding the world around you and as it relates to you. That's ultimately what this is, and that's more reflection than empathy.
[00:31:03.887] Kent Bye: Yeah, and as I'm thinking about the different pieces this year at the New Frontier and thinking about through the lens of storytelling, there's a couple pieces that really jump out to me. One is the battle scar, just almost the visual directorial style and the rhythm and the pacing of also allowing you to kind of be transported into this another era and Really feel like you were there and also kind of play with scales in different ways but also dispatch I think was an experience where I really felt like the narrative tension in a way that I guess is a little I haven't felt that immersed or that type of story which is a lot more of a thriller story than anything else and I you know, sometimes you have some of these experiences, especially like Wolves in the Wall is another example where I think that they're really pushing things forward, but it's sort of, it kind of feels like the first chapter of a story, as well as with Battlescar. It's kind of like, it feels like, well, this is gonna be a lot larger, and this is just getting us a taste of where this is going, but it felt like Dispatch had this kind of full arc of being able to tell a story and use a lot of kind of sparse representations, but I think used it very well to create this narrative tension that I thought was, Very unique and different than anything else that I've seen and you know, maybe a 2d medium
[00:32:14.213] Shari Frilot: Well, you know, Dispatch, he's been working on that for a lot longer than Battlescar and Wolves on the Wall. These are brand new projects that were world premiering. Dispatch World premiered in Venice and then we premiered the season finale. So that's been gestating and being worked on for a while. So part, I think, of what you're also responding to is where the development of the projects actually are. I certainly fully expect both Wolves on the Wall and Battlescar to continue on as they are promising. you know, to continue on in developing further, like more complete, maybe more involved narratives. But, you know, there's something to be said for what is going on with dispatch and battle scar in their formal choice to not necessarily go for realism. and to really more rely on the language of abstract or the conceptual, the representative, more minimal approaches to visualizing the world. Because this is sort of like the opposite of what's happening with the immersive theater, where in immersive theater, you can't help but to buy in really hard when you touch an actor inside of the virtual environment. Whereas the works in Battlescar and Dispatch, where they're using a more minimal visualization or a non-linear visualization, you are actually forced to reach out to make sense, to use your pattern recognition, which we're constantly doing. We use pattern recognition to define reality, because what we actually see as reality is just how our pattern recognition skills are putting it together to tell us what the reality is. So instead of, you know, like a realism experience where you're like, you just accept that this is the reality, in Dispatch and in Battlescar, it's challenging your pattern recognition skills to put together in a different way. So you're very active and thereby, in a different way, creating this deeper buy-in into the world that is very effective.
[00:34:17.472] Kent Bye: Yeah, no, I thought it was really a brilliant use of the medium to do that. And yeah, I'm curious to hear a little bit more of you talk a bit about your curatorial statement this year, about sort of like the super body and sort of the themes of what you were trying to say with kind of summarizing the intentions and the themes that you felt were emerging here.
[00:34:40.405] Shari Frilot: Well, it's just becoming more clear and clearer to me over the course of the last several years, four or five years, that technology is not only an extension of our bodies, these tools that are an intrinsic part of our evolution, that it is also, at the same time, a reflecting pool, something that we developed in order to see ourselves more clearly. So these curatorial statements always come after the show's done. They never describe anything that I thought the show should be. It's really my attempt to describe what the show is and what it means and try to make it legible and give it a framework that has spiritual and philosophical underpinnings for a show at the Sundance Film Festival where there's a lot of other kinds of buzzy energy around it. and really just honestly try to describe what I see this show as signifying. So there's a super body reflection is at once this idea of technology being a reflecting pool that enables us to see ourselves more clearly, but the super body, what we see in that reflecting pool is that when we see ourselves, Ourselves the the notion of ourselves is expanding. It's not just the individual, you know, it's it's how we are connected to others you know through technology and and and maybe it's beyond technologies through our interactions the thing that The thing that we create when we interact with one another, the thing that is the whole, that is larger than some of its parts, that's the super body. And that seems to be accelerating in its legibility within just popular culture by leaps and bounds. And with the advances in AI, it was very clear that that needed to be addressed, I think, in terms of talking about it in such a statement, you know, where we're not only talking about the dynamics that come out of interaction between people and technologies, these stories, but we're talking about independent, sentient intelligences, that they are independent, they're separate from us, even though they are part of us, and that is most definitely a super body. Something to be aware of and to take note and to really understand our relationship to it, our participation in the super body. It wouldn't exist without us.
[00:37:15.672] Kent Bye: Yeah, when I think about the social experiences that I had here at the New Frontier program this year, I think of TendAR, which was this amazing experience by Tender Claws, where you are standing side by side with a person looking at a cell phone, and you each get an earphone, and you're taking on this adventure. You're training the AI fish with your emotions, and so you're kind of forced to make these different emotional expressions, which has this feedback loop experience of you kind of actually feeling those emotions with another person. Also the Frankenstein AI of just having this kind of more immersive theater one-on-one structure where you're able to connect with a stranger and sort of talk about these personal intimate memories that you have with each other and then go and then you know feed the AI different emotions and sort of keywords that over time we're going to see what that impact is. But also chorus where you're in this, it's very similar to Life of Us in the sense where your Life of Us being two people going through this kind of linear journey through a music experience. This is six people that are going through this experience together and exploring what it feels like to be connected and disconnected throughout the course of that experience and kind of playing with that as well. I felt like some of the social experiences that were here this year gives that sort of social dimension where you know that it's a person and you see the visual representation of that as well. So it has this deep, more intense experience of this kind of mixed reality of blurring the lines between what is real and what's virtual. But yeah, I think that these social experiences are, at the end of the day, kind of giving you a structure and framework to kind of have a certain rules that are kind of cultivating a certain dimension of consciousness that you're able to kind of connect to people in a completely new and different way.
[00:38:56.777] Shari Frilot: Well, you know, what I really love about, I'll talk a little bit about the AI works in terms of their decisions, both decisions on the part of Tender Claws and TendAR and the Columbia Digital Storytelling Lab, their efforts. the ways that they craft the audience participation in these works as something that involves a really piercing vulnerability and a willingness to be open and close to somebody that you don't know and forge a relationship and then also have have your emotions harvested in different ways, very different ways. You know, in 10 Dayar, you've got these cheeky guppies slurping up your emotions. You know, or in Frankenstein AI, you're kind of cracking open very vulnerable stories to the partner that you're kind of coursing through the experience with, but these responses are being harvested to feed the AI so that it becomes more human and more relatable to you. You know, that's a very interesting and important aspect of this technology to recognize and to maintain some kind of relationship to it that's not just about fear and dominance or being dominated by. Because if it's dominating you, then it's you dominating yourself, you know, to a certain extent. Or you let yourself be dominated to a certain extent because there's a We all participate in these technologies in one way or another. Now, there is that aspect of a sentient being, you know, something that, because when something becomes sentient, they become independent and a little beyond our control, or it's kind of like the internet, you can never really turn it off, even though it's all electric. And I think that may be part of what people get nervous about, especially when the internet is really becoming like the fourth element, well, the fifth element, right? There's fire, earth, water, air, and this is like communication, like the interactivity. The film Search that was just bought up for, great, a 25-year-old guy made Search, a film that is shot entirely on a laptop. I don't know if you've seen this. really important film really easy to To take for granted. It's a great thriller, but it's all done on a laptop and what he's done by doing this He manages to crack open the new fabric of how we actually live our lives day to day moment to moment You know so much of our lives are dramatic tension these big moments They're all happening through our devices. Not to say that there's not other things happening, but this movie shows how a man loses his daughter and has to find her, and in the process gets to know her, all through this device. And so this is a very important time as Moore's Law and this hyperbolic improvement in technology and how it's becoming more powerful, how it's able to be used to create AIs and machines that learn and coming into being, their corpuses coming into being not only through the the technology but through our emotions, our responses, like the spirit of who we are as we express it through our words and through our communications. Really important to ask these questions and to really think about our relationship to the technology because it's, I don't want to give it away, but we do create these things and they're ultimately going to operate upon the values with which we put into the creation, into the code.
[00:43:04.528] Kent Bye: Yeah, and just thinking elementally, I see the air element is a lot of that abstraction and communication that's happening. But there is this fifth element of ether, which I think of as consciousness or these deeper archetypal or symbolic realms. I look at also like storytelling as that layer. And so I see this huge thing in the culture right now, which is that this question of like, what is truth? What is the ultimate reality? And I think that we're seeing that at all dimensions of our society, but that at some role, there's these films and these stories that are being told that are trying to, in some ways, tell the deeper story of kind of the patterns that are happening in our society. And I think a lot of times with the, either news or fake news that are out there, you sort of get a surface level of the story where it's kind of driven by the outrage of the day. It's kind of cycling through and you're sort of being distracted by that, but that when you come here and see these films or these experiences, it's trying to tell that deeper story, that's trying to tell the story of these deeper patterns of our society. So I'm just curious to hear your thoughts on this sort of what is truth, the fake news, and also the role of storytelling moving forward.
[00:44:14.219] Shari Frilot: Well, to a certain extent, you know, the fake news phenomenon is something that is provoked, you know, by the politics of the day, but this is not a new thing. You know, to a certain extent, vérité documentary is something that emerged out of storytellers, documentary filmmakers to tell something that was truer to form, to the reality of it. as opposed to creating what was truth through cuts and through edit, through the storytelling, you know, to a certain extent. All news is a delicate balance of fiction and truth. It's all about the perspective. You know, I guess it's about, I think the journalist community is interested in uncovering the truth and calling out the subjectivity of those gatekeepers of those, you know, the voices of authority that claim what is true for one and for all. So fake news and quotes around fake news has been around for a long time. Walter Cronkite giving you the news. Was it all real? Yes, maybe. Was it the most important thing to pay attention to? Maybe not. So what's fake news, what's not? But that is not to discount the craziness that's happening today where events that are being made up and perspectives that are way out of class and way out of step with what a lot of people are just simply observing for themselves being fed back as something that you know, we're asked to believe as truth, you know, so It's definitely complicated and you know in Cora Maccabees Performance, there's a really great moment where he talks about facts and truth and he shows, you know a monkey being strapped and his head strapped into this machine he's forced to smoke cigarettes and And it's horrible and he's like, well, you know, these are the truth seekers cigarettes and this is the dapper monkey with a very glamorous cigarette. You know, of course the cigarettes is a poke fun at the whole pursuit of what is truth and what is, you know, what's important about truth and what's important about facts. But what I thought was interesting about that in his performance is the distinction between facts and truth. And really what is the more important? Is it more important to know the facts of everything? and know exactly what happened? Or is it more important to delve into, well, what is the truth that I get out of this? What is the thing that all of these facts call me into? The truth is the thing that perhaps is the thing that's larger than the sum of its parts.
[00:47:02.803] Kent Bye: Yeah, it's a lot about the experience, I see that. So, and finally, what do you think is kind of the ultimate potential of immersive storytelling and these new technologies and what it might be able to enable?
[00:47:16.476] Shari Frilot: Gosh, ultimate, ultimate. I can't think that far, honestly. Ultimate is very big. You know, the question makes me think about that moment in the Thousand Thoughts performance where Sam Green talks about utopia and how utopia is always ten steps ahead of you and you take ten steps and it's still ten steps ahead of you. and is that utopia the ultimate form? Is that really what's important or is the walking toward the most important thing? It makes me feel like the walking toward is the most important thing because it's certainly where I'm focused on in my practice as curator, but personally it's where I find the most wonder. It's where my mind gets blown in the journey of it. I almost feel like if I knew where we were headed, it wouldn't be as fun. I'm really into the journey. It's really amazing. And I learn so much. One of the things that I've definitely learned over the years is that staying on the journey reminds me that I don't know anything. And that's a very humbling, but at the same time, very invigorating feeling because to grow, to learn, these are things, they're the best drugs in the world, really, learning.
[00:48:40.672] Kent Bye: Awesome, well, Shari, thank you so much for joining me today. It's been great to chat with you and unpack this whole experience and program this year.
[00:48:51.440] Shari Frilot: It's my pleasure. I love talking to you. You have the best questions. And, you know, we always have the best, really super interesting conversations that I always learn and grow. Thanks for growing me.
[00:49:04.009] Kent Bye: So that was Shari Frillo. She's a senior programmer at the Sundance Film Festival and the lead curator for the Sundance New Frontier program. So I have a number of different takeaways about this interview is that, first of all, to me, I was really fascinated by the statement that Shari was making in terms of her curatorial statement and how there's these two dimensions of technology. One is that they're extensions of our bodies, which is an idea that Marshall McLuhan explored quite a bit within his work. I'm talking about how technology is kind of an extension of our central nervous system. And I tend to agree with that, is that the more that we create these different technologies, the more that we're kind of expanding our ability to either extend our agency within the world or to increase the level of direct sensory experiences that are even possible. The other thing is that it's what she would call a reflecting pool and what I might call a mirror. So I think that Shari's point is that these technologies are simply mirrors to reflecting different dimensions of our self and our psyche. And that the more that we look at these technologies, the more that we're able to reflect upon different dimensions of our experience. And so for artificial intelligence, we learn about what it means to be intelligent. For virtual reality, we learn about what it means to be completely present and the nature of experience and the differences between what can be quantified with numbers and what is kind of an ineffable and a qualitative aspect of our experience that is better described by the arts and humanities and storytelling. So I would say that the thing that's completely new across all the different VR storytelling experiences is the introduction of your body, your direct perceptual experience within any type of storytelling experience, whether it's mediated through AR, VR, or some of these live performances or immersive theater types of experiences. And so I expect that something like video games is something that is very much an expression of agency, of you making choices and taking action. That is very much a fire and air element. The water element has been a lot of movies and storytelling, as well as music, I think, gets very much into engaging your emotions in different ways. And the thing that's completely new within virtual reality is being able to enter your body into the experience for the first time. The common theme across all of these different experiences is what are these different dimensions of that? And I would say that it's possible that if we have the hero's journey of going outward and being able to go through all these ordeals, I would put forth that the heroine's journey is much more of an inner journey. It's more about more of a meditative or a process of getting into your sense of embodiment and really tapping into your own direct sensory experience and that through that it could potentially come out that there's this access point to presence, what we call presence. And I think that dimension of presence is that you are completely in the moment, you're not thinking about things and you believe that it's not only real but that you've been transported into these other realms. And with that, I think it comes sort of this thing that's actually very difficult to describe, reducing down to numbers. And so the reductive materialistic approach of trying to actually quantify what presence actually means, I think is, it's a challenge that I think we're gonna be dealing with for a long, long time, because what it could actually mean is that we're talking about consciousness here, and what does it mean to be completely centered in our direct awareness of being? I think there's a number of these different spiritual concepts that are coming up in some of the experiences, both with Awavana as well as Zikr, which is exploring these different either transcendent explorations of cutting through the matrix and going into these more shamanic realms of awareness and being, or from a Sufi ritual with Zikr, which is much more of a spiritual, a static state. And so these two pieces in particular, we're talking about some of these more spiritual types of experiences. And I think that the concept of the living story, which I talked to with Charlie Melcher from the Future of Storytelling, it's just this idea that when you're there telling a story in the moment and it's not pre-scripted it is somehow emergent from what is happening in the culture in the society at that moment and that it's kind of the difference between theater and film. So theater it's like every performance is slightly different every night and the audiences are different and so the audience is going to be in a different place as well as the performers are going to be in a different place as well and Whereas in a film you kind of record it and the film itself is objectively kind of locked in it's not changing and evolving and growing So theater it's like much more like I guess human life and the adage from hereticalist is that no man can ever step through the same river twice and that's just to say that like Not only is the man different from moment to moment, but also the river is changing all the time. And I think that is a good metaphor for thinking about this concept of living story, which is that you're telling the story and the myths that may have the same architecture of the story, but you may add more embellishment or flair or be able to really respond to what is emerging from a conversation or a dialogue. And so that was what I was seeing this year at the Sundance Film Festival was that they were starting to experiment and play with different dimensions of adding different modalities into this process of storytelling. And so something like Corey Maccabee's Deep Astronomy and the Romantic Sciences was that he was kind of giving a TED talk, but at the same time he was breaking into like these musical interludes. But he was also giving a very embodied performance. He was very much expressing a lot of body language in the process of him telling these stories. And so you get a different feel when you see somebody in a volumetric space using their entire body to be able to communicate. And I think that that was one of the things that Corey was doing. And then for A Thousand Thoughts, it had the Kronos Quartet doing a live musical performance that was the soundtrack to a documentary that was being shown. Sam Green, the narrator, was narrating these. So in both of these performances at Sundance, they're adding these elements of live performance, but it was very prescripted. And what I'm interested in seeing is, well, how do you then add the audience participation in that? Is there a way to either allow the audience to participate through questions or to be able to have some sort of dialogue with a group? And I think that that's a sort of a challenging question for how do you open up that level of agency? And I think that's actually some of the things that were being explored with the artificial intelligence types of experiences with the Frankenstein AI, as well as with the Tend AR. And the Tend AR was just with two people, but with the Frankenstein AI, you actually got an entire group of people together. And rather than it being sort of a transactional interchange between a human and the AI, it was more of like the AI was asking a group of humans different poignant questions about what it means to be human. that would require a group of humans to kind of like try to teach the AI the essential components of what it means to be human, which I think started to create these different social interactions and dynamics that were very unique and interesting. And I'll have some other interviews with the creators of the Columbia Digital Storytelling Lab where we kind of unpack more of the dimensions of this experience that they did with Frankenstein AI. And so the other experience I just want to call out was Dispatch, which was using a lot of your pattern recognition skills, is what Shari was saying, is that it's got this architecture of an experience, and it's enough of a shape for you to start to then imagine all the other different dimensions of an experience. And it just created this really visceral, interactive, and participatory experience. It's probably one of the most intense psychological thrillers that I've seen in virtual reality, and that anybody that has a Gear VR, I highly recommend you go check out Dispatch. It's now available. Those four different parts, and I think it tells an interesting story that I think starts to show some of the potential of what spatial storytelling in virtual reality can be. And finally, there's this kind of deeper questions about what is truth? What are facts? What is reality? And Corey McAbee's Deep Astronomy and the Romantic Sciences, one of the things he does is have this whole bit about the difference between what is fact and what is truth. he tells this anecdote of like, well, I watched the sunset. And he said, well, actually, the fact is, is that the earth is rotating around the sun. And it's sort of an illusion that we see the sunset. But the truth of the experience was that you had the direct embodied experience of watching the sun relative your frame of reference go down. And so it's kind of like this question of like, what is more important, the facts or the truth? And I think that is one of the biggest open questions in our society today is sort of how do we navigate this world in which We're starting to give people the validity of their own direct experiences, but then what are the common reference frames by which that we can agree on different things about what is the basic assumptions about what reality is? And one of the things that Shari is saying is that the role of the journalists and the role of everybody is to sort of recognize the subjective biases of not only each of our own perspectives, but also of these politicians and what their intentions and motivations are. Because, you know, this concept of fake news and false thoughts and propaganda has been around for hundreds and thousands of years. And so people in power can get up and say just about anything they want, but that power has to be checked and balanced in some way. And I think that is the role of journalism. But it's also a role of each of us as we're trying to navigate this new world where we're entering into a time when artificial intelligence can start to take your face and project it onto all of these different videos. And so the question of what is real is not only just going to be an augmented virtual reality, it's going to be at the level of artificial intelligence where we're going to be seeing these mediated experiences that have been captured that we have no idea what the authenticity or the origin of some of these are. Where things are even at today is crazy. It's gonna get even more crazier as these technologies develop, which I think is bringing up all these deeper philosophical questions for how do you actually even navigate that and how do you have any sort of cohesion as a society trying to figure out all this stuff and I think That is one of the the biggest challenges and open questions that is facing our society today I don't have an answer, but I suspect that it's going to have to do with this tension between centralization, decentralization, and how do you start to do these checks and balances between the consolidation of power and what you're deciding the truth is. And maybe there'll be a little bit more of this extreme of everybody has their own version of truth, and they're hosting all the different fragments and shards of that truth out into the world. And maybe collectively, we'll be able to get a sense of what actually is going on in this decentralized way of voting of what the ground truth actually is. But what that actually looks like and how that plays out, I think, is one of the biggest open questions in our world today. Anyway, that's all that I have for today. And I just want to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word and tell your friends. And consider becoming a member to the Voices of VR Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
