Play episode

#637: The Yang and the Yin of Immersive Storytelling with Oculus’ Yelena Rachitsky

April 3, 2018by kentbye

yelena-rachitskyThe future of VR storytelling will be immersive and interactive. Yelena Rachitsky is an executive producer of experiences at Oculus, and she’s been inspired by how interactive narratives have allowed her to feel like a participant who is more engaged, more present, and more alive. The fundamental challenge of interactive narratives is how to balance the giving and receiving of making choices and taking action vs receiving a narrative and being emotionally engaged and having an embodied experience of immersion and presence. Balancing the active and passive dimensions is the underlying tension of the yang and yin of any experience.

The boundaries between what is a game and what is an immersive story will continue to be blurred, but Rachitsky looks at the center of gravity of an experience. Are you centered in your embodied experience and emotional engagement of a story (yin)? Or are you centered in your head of thinking about the strategy of your next action in achieving a goal in a game (yang)?

She’s recommends that experiential designers start with more yin aspects of an experience including the feeling, the colors, the space, and the visceral sensory experience of a story that you’re primarily telling directly to someone’s body. She’s also been finding a lot of inspiration and innovation of the future of storytelling from immersive theater where actors are able to use their body language to communicate unconsciously with the audience and use their bodies moving through space in order to drive specific behaviors. The Oculus-produced Wolves in the Walls used immersive theater actors from the production Then She Fell in order to do the motion capture, and to help tell the spatial story using the body language of an embodied character in the story.

I had a chance to catch up with Rachitsky at Sundance this year where Oculus had five different experiences including Dispatch, Masters of the Sun, Space Explorers, Spheres, & Wolves in the Walls. Rachitsky has been a key person in helping to discover immersive storytellers and supporting projects that push the edge of innovation when it comes to the future of interactive storytelling. She says that the biggest open question that is driving her journey into immersive storytelling is “How can you be passive and active at the same time?”

LISTEN TO THE VOICES OF VR PODCAST

Rachitsky says that immersive storytelling isn’t about the beginning, middle, or end, but rather it is about cultivating an experience that you have, and it’s about the story that you tell yourself after you take the headset off. This matches some of the depth psychological perspectives on immersive storytelling that John Bucher shared in his Storytelling for Virtual Reality book where VR storytelling could be used as a technological as a vehicle for inner reflection and contemplation.

I suspect that the focus on embodiment and the audience’s direct experience is part of a larger trend towards a new forms of storytelling that transcend the Yang Archetypal journey of Joseph Campbell’s Hero’s Journey, and VR and AR are more about a more receptive Yin Archetypal Journey that I would say is more non-linear, cyclical, embodied, sensory, centered in your own experience, environmental, nurturing, receptive, cooperative, community-driven, worldbuilding, depth psychological, connective, transcendent, esoteric, & alchemical.

The exact patterns and underlying structures of this more yin archetypal journey are still be explored in VR stories, but there’s likely a lot of inspiration that might come from kishōtenketsu literary structures found in classic Chinese, Korean and Japanese narratives that focus more on conflict-free stories of cooperation, collaboration, and revealing holistic interconnections of how the totality is greater than the sum of all of the individual parts.

I’ve recorded nearly 100 interviews on the future of immersive storytelling now (here’s a list of the Top 50 from 2016), and a consistent theme has been this underlying tension of giving and receiving where there is a striving for a balance of the active and the passive. I find that the concepts of the yang and the yin from Chinese philosophy and the four elements from natural philosphy provide compelling metaphors to talk about this underlying tension.

Using metaphors from natural philosophy, the fire element (active presence) and air element (mental & social presence) are yang expressions of exerting energy outward while the water element (emotional presence) and earth element (embodied & environmental presence) are more yin expressions of receiving energy internally. My keynote at the Immersive Technology Conference elaborates on how these play out in the more yang communications mediums like videos games and more yin communications mediums of film and VR.

Video games focus on outward yang expressions of making choices and taking action while film focuses on inward yin expressions of receiving an emotionally-engaging story. VR introduces the body and direct embodied sensory experience, but it’s possible that this focus on embodiment and presence helps to create new expressions of yin archetypal stories that have otherwise been impossible to tell.

Most of my recent conversations about VR storytelling from Sundance 2018 & the Immersive Design Summit have been focused on this emerging yin archetypal journey of how embodiment & presence are revealing these new structures of immersive storytelling.

The concept of a “Living Story” from the future of storytelling’s Charlie Melcher is very similar to what The VOID’s Camille Cellucci calls “Story-Living,” which is about “creating spaces and worlds where people have a chance to live out their own stories within a framework that we design.” The Ready Player One movie did not include some of the “story-living” live action role playing scenes that were included within the novel, but Ernest Cline was definitely attuned to the trends towards immersive narratives when his novel came out in 2011, which is the year that the Punchdrunk immersive theater production Sleep No More opened up in New York City.

Whether it’s a living story or story-living, both involve becoming an active participant and character within the story that’s unfolding. AI is going to play a huge role in helping to resolve some of this tension between authorial control of the story and creating generative possibility spaces, and it’s something that I’m starting to explore in the Voices of AI podcast with interviews with AI storytelling pioneer Michael Mateas, AI social simulator designer & improv actor Ben Samuel, and AI researcher/indie game developer Kristin Siu. Oculus’ Rachitsky is looking forward to integrating more and more AI technologies within future VR storytelling experiences, and she’s even experimenting with using live actors randomly appearing within some future VR experiences that she’s working on.

I expect that the underlying tension between giving and receiving, active and passive, and the yang and the yin to continue to be explored through a variety of different immersive storytelling experiences. While Ready Player One explores a typical Yang Archetypal Journey in the style of Campbell’s monomyth, these types of active gaming and mental puzzle-solving experiences may look great on a film screen, but they’re not always compelling VR experiences that amplify the unique affordances of immersion and presence in VR.

I predict that immersive storytellers will continue to define and explore new storytelling structures that I expect will initially be focusing these more Yin Archetypal Journey of immersion and presence. There will continue to be a fusion of traditional storytelling techniques from cinema, but it’s possible that VR stories need to completely detach from the paradigms of storytelling that tend to focus on conflict, drama, and outward journeys.

It’s possible that the Kishōtenketsu story structures from Eastern cultures might work well in VR as they focus on more cooperative and conflict-free stories that focus on the Gestalt of interconnectivity. It’s also likely that if there does turn out to be a fundamental Yin Archetypal Journey structure that’s different than the Campbell’s monomyth that it’s likely that these stories have been ignored and overlooked, and that it’s possible that the mediums of VR and AR have been needed in order to provide people with an embodied, direct experience of these types of stories.

Eventually we’ll be able to find a perfect balance of the yang and the yin in immersive stories, but perhaps before we get this perfect balance then we’ll need focus on these Yin Archetypal Journey of immersion and presence. Once we open our minds about what the optimal structures for embodied stories that center us in our experiences, then I expect more of a seamless integration of live-action role play, gaming elements, social interactions, and collaborative stories.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Support Voices of VR

Music: Fatality & Summer Trip

Share this
FacebookXRedditEmail

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So the future of storytelling and virtuality is going to be immersive and interactive. The basic challenge is how do you allow someone to participate and be fully engaged with an experience, but also tell a story where they're actually just receiving the larger thrust of a narrative that you're trying to tell them. So this balance between interactivity and storytelling is something that video games have been dealing with for a long time. And I think in immersive virtual reality experiences, you're now adding your body into that experience. And so it's much more about cultivating a sense of presence and immersion, as well as balancing interactivity and agency, as well as a receiving of a story. So you're either coming from this question from one or two different backgrounds. Either you're coming from gaming, where you're focusing on that interactivity, or you're coming from film and storytelling, where you're focusing on the visual and more cinematic aspects of storytelling. So there's this blending that is happening that I find at the film festivals. I'm seeing this confluence of people who are coming from these different backgrounds and trying to, at the end of the day, tell a good story. And I think one of the biggest proponents of supporting these types of stories has been Oculus. And Jelena Rochesky is somebody who is an executive producer of experiences at Oculus. And she is somebody who I see at all the different film festivals and as somebody who is a few steps ahead of basically anyone else in the industry. I mean, if you look at the different types of experiences that are at these film festivals and who is supporting them. Oculus has by and far been the biggest supporter of these types of immersive experiences than anyone else in the industry. I think what Yelena has been able to do is just absolutely amazing. That said, I have a lot of really complicated views on Facebook and Oculus. I just want to say that there's been a lot of things that have come up within the last month or so with Facebook and privacy. This is an issue that I've been sort of addressing in different ways. And just generally, I have a very complicated relationship with Facebook. I have connections with individuals and it's often those connections with those individuals that allows me to have access to these types of conversations. But generally, like GDC just happened and there was no access to anybody from Oculus to be able to talk about any of these issues. And so I feel sometimes a little shut out of being able to actually engage in conversation with a lot of these deeper issues. But that aside, I think this conversation with Yelena is an opportunity where I was able to just really get a sense of her story of where she's coming into the industry and some of the biggest open questions and challenges she's looking at, which is this challenge between interactivity and storytelling. So that's what we'll be covering on today's episode of the Voices of VR podcast. So this interview with Yelena happened on Saturday, January 20th, 2018 at the SumDance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.

[00:03:09.739] Yelena Rachitsky: Hi, Kent. Well, first of all, I'm happy to be on this and also just want to say that I love everything that you're doing and your dedication to expanding this field and making sure that it all gets documented. So thank you for that. Thank you. I'm Yelena Ruchitskaya, I'm an executive producer of experiences over at Oculus. And I've been, over the past couple of years or so, helping to oversee our slate of content around VR entertainment. So, pushing forward, what is storytelling being in VR? Thinking a lot about interactive storytelling. We've launched over 20, maybe 30 projects in the past couple of years or so. And so, really continuing to think about how do we elevate the craft of VR?

[00:03:52.879] Kent Bye: Yeah, and maybe you could tell me a bit more about your background coming into this VR space and sort of, you know, your pathway into storytelling and also interactive storytelling.

[00:04:03.119] Yelena Rachitsky: Yeah, yeah, so I got into this because I was at, I started in documentaries. I worked at Participant Media for about four years. I helped when there was an inconvenient truth, during a time of fooding, of waiting for Superman, and I saw the power that stories can have to change. So many people changed the way that they ate after fooding, and people learned about the educational system after waiting for Superman. and it was incredibly impactful and it was powerful to be there during that time because I really felt like that was like the golden age of documentaries. But there was something that never quite sat right with me and it was even connected to Sundance too because I would come to Sundance and I felt like a lot of people were having the same kind of conversations about how a film got made, how to get actors, kind of how the distribution worked. And it felt like we were just having the same conversation year after year. And I've always just had this personality of trying to understand what else is there. This kind of like seeking of what else are we doing that are continuing to make these experiences more powerful. Then I remember started seeing interactive storytelling. So National Film Board of Canada, for instance, where in Canada they've been putting public funding towards the arts for decades now. And just to experiment, like raw experimentation of innovation and storytelling. So, for instance, Welcome to Pine Point by The Goggles was an interactive web-based project that I saw, and I was like, this is interesting. And it was this interesting blend of interactivity and story where the interactivity was very minimal, but I still felt like I had some agency in it. And then I started discovering New Frontier. I remember I walked into it one day at Sundance, and it was just this haven from the craziness. And I just sat there on some, I think, beanbag chair with these crazy projections around me. And I was like, something just felt right about that space and something felt right about that community. And I started thinking a lot about how technology and storytelling were coming together. And how we, as an audience, can feel more involved. There is something powerful about sitting and watching a movie. People know how to create great stories in movies. But feeling like a participant, like I have some level of importance in this space, to me made me feel more engaged and made me feel more present. it kind of made me feel a little more alive. And so I started seeking that feeling of aliveness. And I remember I talked to my boss and I said, when I was quitting after four years and I didn't have another job, because there was nothing out there that really worked in this field that I was exploring. And I just said, I just want to follow the feeling that feels right. I think back on that. And I'm just like, did I really say that? But then that went on that journey of really exploring what there was. And there was New Frontier, Tribeca was doing stuff, the National Film Board of Canada was doing stuff. People were exploring projection mapping of changing spaces, which I find to be really powerful. People were hacking Kinect sensors. Web-based interactive stuff was also kind of interesting, though was a little bit challenging. And that led me to Sundance New Frontier, which I think really kind of launched that space. Kamal Sinclair, who now runs the Story Lab, was on maternity leave, and they were looking for someone to replace her. And serendipitously, that happened to be me. And my job is basically to understand that landscape. And it was to see who are these people that are just making these things all over the place. and to help find them, to bring them to the story lab. And then I went over to work with Shari of helping to find projects for the festival, and programming the panels, and programming the conversations, and then that led me to future of storytelling over in New York, where it was that continuation of how is the digital age changing the way we tell stories. in a very broad way, the psychology of it, the marketing of it, the technology of it, and then I would say with VR, it was still when I was working at Sundance, and Oculus had just launched their Kickstarter campaign, and we saw it, because my job was just to understand and to search these things. Also, before that, I was also a producer, I produced films, and we reached out, and me and a producer went over to a small Irvine office in Oculus, and they had one demo station in a corner, and I think the first thing we saw was VR cinema, where you're in a movie theater by yourself, and you're watching a film. And I think, and then there was one other project. And even though there was nothing fully there that fit that Sundance space, we, we like thought, this could be interesting, and we programmed it. So it was that first year that Oculus was at Sundance, and I remember I was also programming a panel called, The Future of Storytelling, we even had Nate Mitchell on it that year. And that was the first time that we had a technology there, apart from all the things we had before, like the projection mapping, the Kinect sensors, and the performances, where storytellers and traditional storytellers at Sundance came there and saw something and you can see that there was like a spark in their eye, that they finally saw there's a technology that fits what I do. there's something there. And they didn't know quite what it was yet, but that's when it became clear that this is something that really can fit into both gaming and storytelling and all of that. So after I went to Future of Storytelling, an opportunity with Oculus came. And to me, VR felt like it was the culmination of all these different art forms and storytelling forms that I've seen before, but more accessible, can reach a wider audience, because it's not all location-based. and the possibilities were pretty endless with it, so the accessibility, the creation of it, and then that really just jumped into me diving into it and exploring what all of these possibilities are.

[00:09:29.583] Kent Bye: Yeah, and I'm curious from your perspective, because it feels like at Oculus, there's obviously a lot of games where you're expressing your agency, making choices and taking action. And then in the story, it's perhaps a little bit more about the environment that you're in or receiving a story and the narrative. But there's a fine line there in between, like, what is a story and what is a game? And I guess this is probably an issue that you are constantly kind of rubbing up against. as you're kind of focusing on the storytelling aspect, I'm just curious like how you kind of make sense of that, of like kind of separating what's a game and what's more about this exploration of story.

[00:10:06.593] Yelena Rachitsky: Yeah, I think about this all the time, and I think one of my ultimate goals, and it's been one of my ultimate goals for the past six or seven years, is to find that perfect blend of interactivity and story. I think VR has been an interesting space because it's an industry that's interdisciplinary, so everyone comes from a different industry with their own vocabulary of what they've used. So to a game industry, they've used the word game in a specific way. To the non-game industry, it means something else. Even the word producer means something different in different industries. The way I've been seeing it is that, because a big focus of me right now is pushing forward interactivity within story, but I think of what we do of being more emotionally led. So even though there's interactivity, you're more emotionally compelled and motivated to do something by the experience allowing you and encouraging you to do that. Whereas with gaming, oftentimes it's more strategically led. And they blur a lot. And I think the word game has been expanding so much. There's narrative games, there's explorative games, there's like the Gone Home style, there's the Vanishing of Ethan Carter style, and those start being kind of in the middle. But I think of us as really focusing on the emotional quality of it. So the interactivity isn't the main focus, it's the experience that you have is the main focus.

[00:11:25.402] Kent Bye: Yeah, and I feel like that there's a huge component of both the environment, so having the environment tell a story, but also embodiment. That's something that's completely new within virtual reality. And when I think about that, I think about something like Sleep No More or Then She Fell, immersive theater experiences that are telling a story, but they're not actually using any words. And I think that's probably one of my biggest complaints of seeing a story within VR is that they'll kind of write a script and narrate it and then sort of throw the visuals on top of it. I understand why that's required, but if you look at something like Then She Fell or Sleep No More, they're not relying upon that. It's almost like they are relying upon a new type of embodied and visual storytelling that involves people moving in a certain way, and that there's that movement of bodies through space. that starts to go beyond what you can do with just narrating it. And so I'm just curious to hear some of your thoughts, because I know that immersive theaters, you know, there's just recently the Immersive Design Summit, which a lot of the, you know, majority of people that were there were from more of the immersive theater space, but that I kind of see this immersive theater as the vanguard and the frontier of what storytelling and virtual reality is going to eventually look like. you know, aside from all the locomotion and, you know, haptic sensory limitations with the virtualized version of that, but that in terms of how they're structuring and telling stories and sort of multiple storylines and allowing a narrative to happen, but to be able to tell that story in an embodied way.

[00:12:50.975] Yelena Rachitsky: Yeah, well, first of all, immersive theater has been my passion, and it continues to be. And my goal in VR is how do you create that? In the Immersive Design Summit, I gave a talk, and I talked about that concept of sparkly eyes, of what's the thing that connects all these immersive experience creators, that they want everyone to leave with that sparkle in their eye, that just, like, what just happened? It was an experience that they felt so present to that moment. And that's what I continue to think about. How can we emulate that? How can we make someone feel so present? And then when they take off the headset, the world, looks and feels a little bit more different, and the world feels a little more magical. So we think about that all the time. So in your question about embodiment, what's interesting is all the projects that we have at Sundance, Dispatch, Masters of the Sun, Space Explorers, Spheres, and Wolves on the Walls, the role of the audience is completely different in each one of them. But they all work in their unique way, which is different than when VR first started and people asked the question of, well, you have to have a reason of why you're there and who are you. And now we've learned you don't actually need a reason, it just needs to work. But I definitely agree with you, but I think it all depends on what the concept of what you're trying to do is. What is your goal when you're trying to create a VR experience? And the way that I, when I talk to a lot of creators, the thing I ask is, what is the feeling that you want to create? And then how do you use these tools of VR to push that feeling forward? Narration is part of it, but the colors, the space. that visceral experience that you get is them telling a story to your body, which is just as important as the story being told kind of cerebrally in your understanding with it. So for instance with spheres, even though there's narration and Jessica Chastain is giving narration, the most powerful feeling for me is what's going on? is that I'm actually becoming the star, and then I'm the star that's going through this black hole, and I'm seeing myself getting spaghettified, which I heard is actually a scientific technical term, and then I become this black hole. So even though there was a narration, the memory that I have from it is the feeling that my body had. And similar with wolves, you catch the story of what's going on, but it's also this connection that I have with Lucy, the character, and the way that they gently guide me. So I'm so excited about with wolves is that they worked with third rail. And Third Rail is a company that did Then She Fell, one of my favorite immersive theater shows. And they used one of the main actors from Third Rail to do the motion capture and thought a lot about the subtle movements that the actress and the actors have in immersive theater to guide you without even realizing that you're being guided. So for instance, when Lucy hands you the camera, she does it when she's turned around a little bit and she puts her hand back and just kind of gives it to you. So it's not as if she's forcing you to have it, she's allowing you to make your own decision. And Third Rail works a lot with body movements. So there was the workshop that they had and they said they lead a lot with a sternum. So in an order for an actor to get you to go from one place to another, they actually use their sternum. They make it very tight and strong and they move their sternum. And somehow you're compelled to also go in that direction that they go in. And also if you noticed with wolves, the lighting was incredibly important in guiding you to a space where it's spotlighting certain places in the set. And so you know you wanna go there. and you're guided there. So it's this cross between the interactivity, but also the illusion of agency that a lot of immersive theaters do really, really well. So even though a lot of these do have the narration, and a lot of them, you know, more than others, for instance, Dispatch is an auditory story. It's kind of, I think of it in a sense, more like a podcast, and then you're using your imagination, your mind of imagination to see the visuals, but it's not the main part of it. But with that said, like the power of voice and spatial sound to add to that bodily experience is highly important. But I do agree that you can't rely on that narration because people also have a hard time processing narration in VR more so than they do in regular film because you're more overwhelmed with the space. There's more to process. There's more to take in. So you really have to simplify what you're saying and doing.

[00:16:56.382] Kent Bye: And when I talked to Fran Pichetta a couple years ago here at Sundance when she was showing 6x9, you know, one thing she said is that she had actually done a lot of audio tours and that those audio tours were kind of designed to allow you to become more present into the space. And so they were using the second person. So like you are going to experience this or you're going to see this rather than. having somebody narrate something like my experience was, you know, when I was in solitary confinement was about this. And it was taking you out of your own direct experience of it. And so the audio allowing you to be more connected as to what is happening in the scene. And I think Dispatch does a great job of that because you see these almost like symbolic or archetypal representations that are very low fidelity. But with the audio, your brain can really kind of fill in a lot of those gaps and you have sort of a rough architecture of the space of what's happening. But it's sparse enough that it allows you to really participate with your imagination as to what's happening. And your brain is actually doing a lot to kind of recognize the patterns that are happening here and just playing with colors as well, that they were doing that. But, but the audio having a sense of, of being directly connected to what is happening to you. And I think if you kind of think of it as a guided tour, then it's just a challenge when you're trying to. tell the story and there's no visual storytelling that's happening at the same time, then that's the thing where my brain kind of like wants to see a direct connection between what's being said and what I'm seeing. And it makes me think of like the different stuff that Punchdrunk has been doing in terms of like moving outside of just doing experiences where people are in an immersive theater and they give them audio tours and they're out and they're kind of blurring the reality of what's real, who's an actor, who's not. and having these different interactions, but it's just sort of like they're able to kind of create this layer of story on top of like their reality and that they're doing that through the audio and the podcast. So I do think that the audio is gonna be a huge part. It's just my frustration sometimes is kind of like somebody who may be coming from a documentary or film background and sort of relying upon writing out all the script without sort of a consideration of what is the visual connection and the storytelling of that.

[00:19:05.452] Yelena Rachitsky: Yeah, no, I think you're completely doing it wrong if you don't have consideration of the storytelling. So the spatial part of it, when you think about Dear Angelica, for instance, even though it's not interactive, they played so much with the constriction and expansion of space to push your emotional journey through the experience and also scale as a huge portion of making you either feel big or small. And that feeling of expansion and contraction changes your experience that you had with the story. And so much about VR storytelling is not the thing that is the beginning, middle, end that's told to you. It's the experience that you had yourself. And to me, it's the story you tell yourself after. So it's incredibly important. Yeah, and with that interactivity and storytelling blend, it's also such a challenge and we're still at the very beginnings of trying to figure out how do you, because you can't be passive and active really at the same time. So it's hard to do while you listen. So if you give someone the ability to move while you're trying to give them information, it's too much but if you stop someone and you have a time of narration and someone can't do anything and then they're able to move in or interact and they have full freedom and then their freedom gets taken away, that's an uncomfortable feeling for people. It's like you just gave me this freedom, I was able to do all these things and now you're making me just like stop here and listen. So it's really trying to find what's that blend, what's that fluidity? where it all kind of goes into each other. Like, for instance, with our Blade Runner Memory Lab project, we used a volumetric capture through Microsoft, and it looks really beautiful, and you're able to get a live actor component, and we added an interactivity layer where you feel definitely part of it. But there was a big challenge in giving you this information of this Blade Runner story while giving you a task to do. And at times, even though it succeeded in so many ways, and it looks really gorgeous, there are these moments where the passive-active, passive-active doesn't necessarily become fluid. And then on the other side of it, with Coco VR, I don't know if you've had a chance to watch that. It's our project that we did with Pixar, and it's the first project we did that has had social co-presence. It's all about being highly interactive. You know, we thought a lot about what can we do that Pixar doesn't do and decided we wanted to make it all about the audience and all about the audience's journey and experience and that social component just added so much to it and that's personally something that I'm super excited about and we're going to be doing a lot more of. But the story is pretty minimal So you you have an intro with Miguel taking you in and then it's fully interactive and those little points where you even have a story component in there Because it's so highly interactive people even had a hard time waiting for a few seconds for that story to come in I see this getting a lot better once we start getting a lot better and at AI characters and having these intuitive interactions that people react to you. So, for instance, a great example is something like First Contact, which is basically a demo tutorial piece for learning how to use the touch controllers. But the way that the robot emotes with you and encourages you to have this interactivity, but but still plays along with you is very fluid. It's a very fluid experience. It creates an emotional response. There's not just passive active, passive active. It kind of comes a little bit in the middle, but it's just a start of that. So a lot of what we have on our slate right now that we're working on is focusing on that. It's focusing on that perfect blend between the two. It's gonna take a little while to really get to that perfect space, but I'm so excited about uncovering more learnings that we can continue applying to more and more projects.

[00:22:46.820] Kent Bye: It's really interesting that there's, at Sundance here, there's the motto is the story lives in you. So it's really talking about that, how so much of the story is, you know, you projecting your own experiences onto that. And I think that in VR, you know, it's like the question is, well, how do you make it sparse enough so that you can actually have that ability to participate in that way? I do see that there's a spectrum between authored narrative and generative narrative, where at one extreme it's completely authored and it's on the rails and you have no ability to really make much decisions. Some decisions that you do make are kind of flavoring the experience perhaps, or maybe it's pacing yourself through the experience, but it's not really the choices that you're making isn't really necessarily changing the outcome. And then you get more and more into the place of like branching narratives. And then maybe that's sort of in the middle and then choose your own adventure. But then the other extreme, it's sort of like, much more like a conversation. And so it might be at the essence of maybe like a depth psychological, like therapy session, where it's more about you learning about the nature of your own story, rather than the story that's being told to you but it's more about high agency interaction where you're able to really fully participate and I think that's where in the immersive theater scene I see that a lot of the immersive theater is very authored like there's not a lot of experiences that I've seen where you're making choices where it's sort of you know having some actual impact into narrative and I think that as you're going through a one-on-one experience within VR with artificial intelligence you have much more leeway to be able to do in both natural language processing with AI but also like deeper AI to be able to do like constraints and planning to be able to say okay well we're gonna recreate this and because you did this it's going to now change everything that follows from that and so I Eisen or is another experience from more of a 2d but the AI within that context is starting to get into this like You know deep simulation where you're actually making choices and those choices are actually changing the whole plot that's being presented to you So I'd imagine that in immersive theater that's gonna be very difficult to be like Choices being made and then now let's sort of you know How do we communicate with all the actors like this is where we're at and this is how we're flowing I think in the virtualized world, we're going to see a lot more of being able to take that agency to that next level. But at the same time, I see that there's something magical with working with live actors and, you know, if there's going to be Wizard of Oz or immersive theater, NVR kind of blending together. But you lose something with that interactivity when it's sort of just a AI that doesn't have common sense or ability to kind of react dynamically to whatever you're doing.

[00:25:14.574] Yelena Rachitsky: So we're actually starting to experiment with bringing live actors into VR, kind of like a sleep no more experience where you won't know if it's an NPC or if it's a live person, and it could be a live person and it takes you on that special experience. And it might not be every time you try the experience. It might be sometimes. But something that's kind of that blend of journey and sleep no more. I'm super excited about that project. I'm looking forward to telling you more when I can. But thinking about that a lot and that blend between the two, it's going to take a long time for any non-player character to be natural like a human. Because just human tendencies are so subtle and so specific. And we as humans recognize very quickly what's not human. So I'm not thinking that's going to come Very quickly to give us the answer to the interactive storytelling unless you do it in a fairly subtle approach But incorporating live actors alongside non-player characters and also creating social experiences I think can get us there in unique ways because you still can create these unique connections with other people that are humans but it doesn't rely on pushing that storytelling forward and And I think about those two differences between that open world generative where anything can happen versus the controlled where you have this agency but there is still a direction that you're going into. And I don't have an answer of whether one's better than the other. Just think any story that you tell that's good and specific and to that feeling that you want to accomplish in someone There's different ways of approaching it. So just it just really depends on what your goal is in that I personally when you have a full open world and you can do anything and it's I branching narrative, it's really hard for it to not be distracting. It's really hard for you to make too many choices and it takes you out of that experience a little bit. It brings it into your head versus your body when you know you're making choices and you have to be conscious of the choice-making. So I want to be careful about that because I think a lot about these experiences that we make as being more body-oriented and So I really like what Wolves is doing with the subtle interactivity. So there's a lot more going on behind the hood than you see at first. And Lucy reacts to just where you choose to listen. She reacts to when you grab something from her. She reacts when you don't do something to her to really approach more natural human interactivity that feels more intuitive versus blatant interactivity where you know you do things and I think that helps you respect the space a little bit more and it helps just feel more natural and connected. It's a little bit risky because people won't recognize the complexity of it, but I'm excited to see that team continue pushing in that direction and see how that turns out because then it just feels more natural, like that intuition of interactivity where you don't even notice why you do some things, like why do you brush something aside when something's in front of you and you realize, oh, you know, I just did that and that felt so real. And that could also make you more present in a space than you realize, because your body is just kind of there. And when you're fully present in the spaces, I think you can get more close to trying to create that sparkly eye that we do in live theater.

[00:28:40.479] Kent Bye: Great. And finally, what do you think is kind of the ultimate potential of virtual reality and immersive storytelling and what it might be able to enable?

[00:28:51.316] Yelena Rachitsky: That's a big question, Kent. So what's the first one?

[00:28:56.562] Kent Bye: Just the ultimate potential of VR and immersive.

[00:28:59.631] Yelena Rachitsky: I think it's different for every person. I think VR is, the future of it is being this seamless computing platform where you're able to do anything and hop from experience to experience and have it be a seamless part of your life that helps you be more productive, helps you be more connective, and also helps you be entertained. I specifically focus more on the experiences that you have emotional connections to, so that's my focus. But once it also just becomes a more seamless experience, hopping into those entertainment experiences is also just gonna be less friction, which I'm really excited about. But as far as that space, my constant goal is that space between interactivity and story. How do you just create a compelling, beautiful piece of content that you remember as a momentous experience that you had? Like if you had a trip that you took and it was something that you remembered for your life, that was a moment that shifted something, that was a moment that moved something in me, that was a moment that I connected in a different way, To me, I'm excited about bringing VR experiences that could do that.

[00:29:59.160] Kent Bye: Awesome. Well, Yelena, thank you so much for joining me today.

[00:30:03.923] Yelena Rachitsky: Thank you, Kent. And yeah, congratulations on just getting so much of this down, I think, in a few years. We're going to see where VR goes. But I think that documentation that you have over how many podcasts?

[00:30:15.771] Kent Bye: It's over 800 interviews that I've done over since May of 2014. Yeah.

[00:30:20.273] Yelena Rachitsky: So it's going to be important in the history books. But it's been great talking to you. Thank you.

[00:30:26.467] Kent Bye: Thanks. So that was Jelena Richesky. She's an executive producer of experiences at Oculus. So I have a number of different takeaways about this interview is that, first of all, there's one question that I think encapsulates the entire thrust of the challenge of immersive storytelling. And that question that she asked is, how can you be passive and active at the same time? This is the essence of how do you participate in your agency expression as well as at the same time that you're actively listening and receiving. I think a good model to think about that is when you have a conversation. And when you're really engaged in a conversation, you're both listening to somebody, but you're at the same time thinking about your next response and being able to actually participate in a conversation. But doing both of those at the same time where you're actually really centering and emotionally and empathetically listening, even that is difficult in conversations. Sometimes people are so centered in their own experience that they can't hear anything other than how any experience makes them feel. So it's this challenge of giving and receiving, the passive and the active, the yang and the yang, being able to receive a story but also at the same time participate. And so I think this is the thrust of the challenge of immersive storytelling. So with that, I think there's a couple of metaphors to think about the tension between these two. One is video games, which is all about making choices and taking action. That's the air element and the fire element, and those are both more yang or rising active energy. And then there's the receptivity aspect, which is the water and earth element, which is the emotional engagement as well as the embodiment. So if you notice that in terms of storytelling, what you're saying is that she's very centered in the emotional center of gravity of the experience. Like, what is the story that's being told and how is that getting you emotionally engaged? And as you're in a virtual reality experience, are you thinking more about the agency that you're going to be expressing in the strategy that you're taking? Or are you more centered about really empathetically tuning in and listening with your body, and she said there's actually a difference between whether or not you're listening with your head or listening with your body. When you're listening to your head, it's a mental abstraction, you're making choices, you're trying to see how you as an individual agent can take action within that world versus you receiving with your body what is happening within that experience. And I think This tension between the young and the young, the passive and the active, and being able to express your agency and your activity versus receiving it is the heart of what makes this immersive storytelling so interesting to explore. Because this is, as human beings, that is what makes the essence of an experience. You can go to a movie and have a bit of an experience, but the thing that makes it an experience is you able to fully participate in that experience. It can certainly happen when you go to a movie and you just get totally emotionally engaged, but it's often because you have something happening in your life that you're projecting into that experience. And so there's always this interaction of your inner self that's being put into the narrative. What Elena said is that a lot of times in an immersive story, it's much more about what happens after you take the headset off, what that sensory experience that you had and what stories that you're telling. And so I think that is the most important part is like immersive storytelling is about centering you in your embodied experience and being able to really emotionally engage you. And there's certain dimensions of the agency and interactivity that are certainly a part of it, but it starts to get more into a game and more into the strategy. And so this is the essence of, I think as time goes on, we're going to be seeing more and more of a blurred line between what's a game and what's a story. I think these are just going to seamlessly kind of blend together into like these live action role play, immersive theater slash interactive games and puzzles that you're exploring. So I think one of the things that Yelena said was that she gave a talk at the Immersive Design Summit and she was talking about this sparkly eye feeling that people have after they have one of these experiences. And I think what she meant by that is that you have this sense of awe and wonder of being able to have your mind opened up to new possibilities as you have some of these immersive storytelling experiences. at the essence is that you have an experience that changes the way that the world looks and feels. It just feels different and you've been able to have your mind open up to these new possibilities and you have this magical and enchanted feeling that is changing your perspective. And I think that is the essence of the concept of a living story or story living, where you're actually like participating within this context of the story, where you're actually in the quality of the moment of the time and that you're able to interact and engage with that story. And that in some ways, if that story is resonant enough, it's able to really tune into what's happening in your life. And it becomes more of a transformational experience in that way. I think that's the essence of eventually where we're all going is that we're not going to want to just have stories. We're going to have like deep, meaningful stories that are attuned to us and able to have transformational experiences. Now, to be able to do that every night, I think it's, you know, sometimes people wait a lifetime to be able to have those types of experiences. But I think where this is going is that it's more of an attenuation of being able to create a high level archetypal story that is a little bit of the universal story such that these human dynamics that we all experience in some way, and what are those experiences that are common to the human experience of what it means to be a human. And then there's going to be different variations of stories that only certain individuals are going to be able to go through. But if the story is universal enough, it's going to be able to resonate with you in some deep way. And so the fact that Oculus is starting to collaborate with all these immersive storytellers means that people like from Then She Fell and Sleep No More, these immersive actors have trained themselves to use their bodies to move through space and to use body language to be able to communicate unconsciously with other human beings. And I think That's a trend that I was starting to really see at Sundance this year was this blending of the immersive theater background and those insights of embodied storytelling and how can you use your body positioned in space to be able to communicate things in a way that is just unconscious. And there's things that they talked about that I hadn't even thought about going through the experience wolves and walls where they had different behaviors that they were encouraging, but it was all told through that body language. And I have some interviews with the creators of wolves in the walls, where they talk a little bit more about the process of immersive theater and how they worked with these actors to be able to do that. And some of the different variations that they had to do in terms of doing that type of immersive storytelling. But overall, what Yelena said is that she got just hooked into this feeling like a participant, feeling more engaged, feeling more present, and feeling more alive. And that's the future of immersive storytelling, is that interactivity, that participation, that embodiment, that full engagement of all of your emotions. And I think AI, interactive characters, immersive theater, live action role play, these are all things that are out there in terms of existing protocols for what people are doing to be able to achieve this perfect balance. And I think more and more, there's going to be an infusion of these principles into the immersive technology. We're going to start to see more immersive actors within experiences that you know are not AI. And it's sort of living into the dream of Diamond Age, which is these people who are actors who are living into these different immersive experiences that you're having in virtual reality. So that's all that I have for today. And yeah, this is the time where I give a pitch for the Patreon. And, you know, if you enjoy this podcast and enjoy these different insights, I really do need your support to be able to continue to provide this as a service to the community. The whole industry is going to take time to really get to a point where I think it's really sustaining people as creators. But at the same time, it's still possible to push forward what is possible within virtual reality as a medium. And I think that's what I've been just trying to do on the podcast is really just trace this evolution of this medium and what's new, what's different, what is unknown, what's not known, what are the experiments that you should be thinking about, and just talking to leaders in the field like Elena to be able to share some of her wisdoms and insights and things that she's thinking about and working on in the future. So if you enjoy that as a service to the community, then please do consider becoming a member of the Patreon and supporting this effort to continue to document and share all of this knowledge with the larger community. You can do that by going to patreon.com and becoming a member. $5 a month is a great amount that if people were just to give $5 a month and support what I'm doing here, that would allow me to start to kick up and expand both into AI and to do more video, do more live streaming, expand out into math. I mean, I have so much that I wanna get out there, but I think that having more support from these Patreons would allow me to continue to grow and expand and to continue to educate and spread the word about These deeper concepts and revelations of what all this immersive technology means and how it can help us live into the highest potential of what it means to be human. So donate today at patreon.com slash Voices of VR. Thanks for listening.

Tagged as:

More from this show

Menu

#637: The Yang and the Yin of Immersive Storytelling with Oculus’ Yelena Rachitsky