I had a chance to talk about storytelling in VR with three of the co-founders of Oculus Story Studio during Oculus Connect 3. Saschka Unseld, Maxwell Planck, and Edward Saatchi were showing off a preview of their third VR experience Dear Angelica as well as their immersive storytelling tool of Quill, which enabled them to create a VR narrative experience entirely within VR.
LISTEN TO THE VOICES OF VR PODCAST
They all emphasized to me that it’s still very early days of figuring out the unique affordances of virtual reality as a storytelling medium, and that Oculus Story Studio is still doing quite a bit of experimentation. They were in agreement in believing that it’s likely going to take a long time to figure out what narrative in VR looks like, and that it could be another generation before VR finds it’s true form.
While I agree that VR storytelling is still very much within a Wild West phase of development, at the same time I do believe that there have been a lot of solid lessons learned about VR as a storytelling medium that I’ve covered on The Voices of VR podcast. At the bottom of this post is a Top 50 List of Voices of VR interviews about storytelling in VR where the list is broken up into the following seven categories: the language of VR storytelling, interactive storytelling, multiple perspectives and empathy in storytelling, social storytelling, world building & environmental storytelling, plausibility & presence in narrative, and audio.
Some of the key discoveries that Oculus Story Studio made with Dear Angelica are first of all that changing scale as an effective way to evoke different emotional reactions. They also discovered that stopping and scrubbing through time was a very compelling experience that allowed audience members to have more control over their pacing through an experience. They also developed a unique “Quillustration” aesthetic that is like a lucid dream that’s trying to mimic how memory works. Perhaps having tools to create VR stories within VR will provide new narrative devices for how stories will be told in VR.
Saschka defined the essential components of a story in VR as simply having a beginning, middle, and end, and this broadens the scope of what could be classified as a narrative within a VR experience. Edward says that it often feels like they have the “dead hand of cinema” hovering over whatever VR storytellers do within a VR experience. The target VR demographic right now is so familiar with the film and video game mediums that they are bringing a whole set of expectations that impacts how they consume and receive VR narrative experiences.
Saschka was also really cautious and skeptical about creating stories that have branching narratives with multiple endings. He interprets multiple storylines as a sign that the author may not know what he/she wants to say, and this blocks his process of cultivating a personal connection with the content creator.
We also had a wide-ranging discussion about narrative vs interactivity, and the balance between creating authored stories versus balancing the amount of control a user has within the context of their sandbox of interactivity. Oculus Story Studio is made up of a lot of filmmaking gamers and so they cited a number of 2D narrative games as inspiration including Stanley Parable, Papers Please, Tacoma, Virginia, Gone Home, LMNO, and Façade. In the end, they imagine that VR experiences will be like the Holodeck in that it’s social, it’s a game, but it’s a movie.
We’re still quite a ways away from having a widespread consensus on where VR storytelling is going, and Oculus Story Studio will continue to try to find that sweet spot between authored narrative and the sandbox of interactivity.
Top 50 Voices of VR Interviews on VR Storytelling
THE LANGUAGE OF VR STORYTELLING
- The Four Different Types of Stories in VR (292)
- The Language of Cinematic VR with Google’s Jessica Brillhart (291)
- Storytelling in VR: Ambiguity and Implication in 1st Person Narratives (339)
- Pushing the Language of Cinematic VR Forward with ‘Sonar’ (296)
- “Pearl” is an Emotionally Powerful Story about Selfless Service (415)
- Ted Schilowitz on Bringing VR & Interactive Storytelling to Hollywood (439)
- What Broadway Theater Can Teach VR Video Production (380)
- Oculus Story Studio’s Quill: An Immersive Storytelling Tool (467)
- Storytelling in Virtual & Mixed Reality with SPACES (374)
- John Gaeta on ILMxLAB & Immersive Storytelling (294)
INTERACTIVE STORYTELLING
- AI and the Future of Interactive Drama (293)
- Storytelling in VR & the Tradeoffs of Empathy and Interactivity (290)
- Using Code as a Canvas for Living Stories (411)
- Sequenced & the Challenge of Interactive VR Narratives (396)
- Interactive Storytelling Triggered by Gaze, Kevin Cornish (349)
- “Luna”: A Deep Game, Narrative Puzzler about Recovering From Grief & Trauma (438)
- Cracking the Narrative Code of VR with the Interactive Documentary Genre (407)
MULTIPLE PERSPECTIVES & EMPATHY in STORYTELLING
- Rose Troche on the Vulnerability of a 1st-Person Perspective (286)
- Situational Knowledges in VR Narrative: The Role of Place & Perspective (408)
- Nonny de la Peña on Immersive Journalism, Empathy, & VR storytelling (6)
- Building Empathy with a 360-degree Video about a Sexual Assault from Two Perspectives (242)
- Nonny de la Pena on Empathy in VR (298)
- Empathizing with a War-Torn Family in ‘Giant’ (342)
SOCIAL STORYTELLING
- Group Explorations of User-Generated Worlds with VRChat (318)
- What Dungeons & Dragons Can Teach Storytelling in VR (441)
- Telling Stories with Improv Acting in ‘Mindshow’ (420)
- Wizard of Oz Narratives: Puppeting Virtual Characters with Improv Acting (409)
WORLDBUILDING & ENVIRONMENTAL STORYTELLING
- Alex McDowell on World Building in Storytelling (309)
- Building Storyworlds with Lawnmower Man’s Brett Leonard (406)
- Explore the Psychological Impacts of Solitary Confinement in ‘6×9’ (287)
- Embedding a Story within a Place with ‘Obduction’ (432)
- Denny Unger on the Future of Non-Linear Storytelling (462)
- The Principle of Embodied Cognition as connected to the Environment (Episodes: 412, 469, 375, & 73)
- Designing Google Earth VR: The Overview Effect & Finding Common Ground (475)
- Walk Through a Vincent van Gogh Painting with ‘The Night Cafe’ (259)
- Walking On a Virtual Tightrope Across the World Trade Centers (345)
- Using Magic to Create Astonishment with The VOID (299)
- Beyond Room-Scale: Exploring Infinite Worlds with THE VOID (284)
PLAUSIBIILTY AND PRESENCE IN NARRATIVE
- Rob Morgan on Narrative Design in VR & escaping the uncanny valley by implementing interactive social behaviors in NPCs (125)
- ‘Rick & Morty Simulator’: Making Narratives More Plausible through Interruption (433)
- Betty Mohler on Social Interactions in VR, Uncanny Valley Expectations, & Locomotion in VR (129)
- Richard Skarbez on Immersion & Coherence being the two key components of Presence (130)
- Mel Slater on VR Presence, Virtual Body Ownership, & the Time Travel Illusion (183)
- Technolust’s Cloudstep VR Locomotion & Adding Social Behavior Scripts to NPCs (237)
- Ross Mead on designing social behaviors & body language for virtual human avatars (56)
- Job Simulator and the Magic of Hand Presence (315)
- VR Time Perception Insights from Filmmaking & Cognitive Science (379) + Time Dilation (363)
AUDIO
- Audio Objects for Narrative 360 VR with Dolby Atmos (398)
- OSSIC & 3D Audio as the Next Frontier of Immersion (399)
- Rod Haxton on VisiSonics’ RealSpace 3D audio licensed to Oculus & their Audio Panoramic Camera (124)
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR Podcast. So, as humans, we're storytelling creatures, and with every communications medium, there turns out to be unique ways that we can tell our stories. So with virtual reality, it's a brand new communication medium that we're still trying to figure out what are the unique affordances of this new medium and how can we tell stories with it? And so right now, we're really still in the Wild West of storytelling in VR. And we've had a couple of years to really start to explore it within the context of consumer virtual reality. So back at Sundance of 2015, Oculus launched the Oculus Story Studio with their first experience called Lost, which was their first foray into trying to see how do you tell stories within VR. And so on today's episode, I'm going to be talking to three of the co-founders of Oculus Story Studio with Shaska Unsell, Maxwell Plank, and Edward Saatchi. And so we'll be talking about that tension between authored narrative versus this interactive sandbox where you're able to really express your agency. And how do you balance that within the context of telling a story within virtual reality? So we'll be talking about that as well as some of their lessons learned from the first three experiences that they've created so far. And at the end, I'll be giving some highlights from some of the top 50 interviews that I've done about storytelling in VR, as well as kind of extracting some of the bigger themes that I see emerging that help us kind of identify some of those unique affordances of the VR medium. So that's what we'll be covering on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by the Voices of VR Patreon campaign. Over the last two and a half years, I've done over 500 interviews in the VR community, with over 10% of them focusing specifically on storytelling in VR. Your support with the Patreon campaign can help ensure that I can continue to travel around the country to see the latest VR experiences and talk to some of the leading creators and seeing what's actually happening on the front lines of creating this new virtual reality medium. Your support ensures that I can continue to do this type of coverage, but also maintain a certain amount of editorial independence as well as freedom to be able to dive deep into some of these deeper issues and patterns that I see emerging within the VR community. So, donate today at patreon.com slash voicesofvr. So, this interview with Maxwell, Edward, and Saskia happened at Oculus Connect 3 that was happening in San Jose, California from October 5th to 7th. So, with that, let's go ahead and dive right in.
[00:02:49.460] Maxwell Planck: I'm Maxwell Plank. I'm the technical founder of Oculus Story Studio.
[00:02:53.742] Edward Saatchi: Hey, I'm Edward Saatchi, business co-founder of Story Studio.
[00:02:58.244] Saschka Unseld: Saskia Anselden. I'm the creative director of Story Studio.
[00:03:02.012] Kent Bye: Great. So let's talk about storytelling in VR, because you have narrative and interactivity. You have presence of actually being embodied, but then you also have being able to tell a story. So maybe you could tell me a little bit about how you guys have tackled this problem and some of the big insights that you've gotten so far about storytelling in VR.
[00:03:23.182] Saschka Unseld: I mean, I think the biggest insights are that we still are experimenting a lot. I think the biggest thing that came in the last two years was that initially we thought there's certain rules, like don't move the viewer and whatever stuff, and then over the last two years we're like, no, actually you can move people, actually you can do cuts, actually you can do all these things. So I think not rushing to judgment on things you can do and can't do is probably the biggest thing we learned in the last two years. In D'Angelica, one of the big breakthroughs for us was, and that was mostly because we were directly creating it in Krill, was using scale to change your relationship to a character. There's moments when the main character, Jessica, lies in her bed and is writing a letter. We have her tiny as miniature, and that gives you a very different kind of emotional connection to her. And there's other moments when she's life-sized, and there's parts of her memories that are larger than life. And kind of playing around with that scale in a narrative is really, really interesting and something that you can't really do in movies because you just think it's a wide shot and a close-up, but you perceive the world still as existing in the same scale. And using scale to tell a narrative is something really, really fascinating.
[00:04:30.645] Maxwell Planck: I also love to hear about how when we went after Dear Angelica, we were looking for creating an imagination space. What would it be like to be inside a girl's thoughts and dreams and how you thought of illustration and how that inspired this completely new look, which was inspiring for me. It's very different from Lost and Henry. I'd love for you to talk about it.
[00:04:50.581] Saschka Unseld: Yeah, it was actually like, thinking back, it was actually two Oculus Connects ago, like the very first Oculus Connect and there was a moment where we were kind of sitting in one of the hotel rooms in a big group with Kamau from Sundance and kind of the guys from Kite and Lightning and Werner Herzog as well. And I remember saying like I wonder how something would look like when you're in a space and there's a singular line being drawn in front of you and then a second one and suddenly out of these random abstract lines form comes out of and something is created in front of your eyes out of nothing and it's an illustration. So the inception of D'Angelica goes like ages ago, but we had no idea if it's going to work or not. Or if you could tell a story in something that is purely illustrative and that takes place in someone's mind. And then with the Oldsbrook coming on board as the illustrator and the artist on it, and we're creating Quill for her, we enabled her to do that. And I think the interesting thing became that suddenly it all becomes more like a lucid dream and like memory and maybe that's a bit more how VR works. That's more the kind of narrative devices you use to tell stories and a painterly approach just feels like really natural to that.
[00:06:02.715] Kent Bye: Yeah, one of the things that Eric Darnell said is he made the differentiation between film and VR by saying that film is like telling the story from a singular perspective, whereas VR is kind of like giving somebody an experience and they're able to kind of generate their own stories from that. And I think that's been a little bit of the challenge of giving people an experience, but then within that experience, then how do you embed the narrative within that? So I'm curious to hear some of your thoughts on that.
[00:06:30.255] Edward Saatchi: Yeah, I was just thinking yesterday someone was saying we still haven't seen a character in VR that's as compelling as a Pixar character or we still haven't seen a character that kind of grabs people and you want to revisit for years. And I was just thinking that the dead hand of cinema is over a lot of what we do in virtual reality. So people judge some of the virtual reality movies that we make, and we do as well, based on what's good for a movie. Just like back in the 1900s, maybe they judged a movie based on, well, this isn't how you do it in a play, this sucks, this isn't right. And just taking the example of a character, maybe it's not necessarily taking those lessons from movies or from literature to create an amazing character in virtual reality, just as an example again. But AI, level design, gameplay programming, right? Like creating a compelling character in VR may be a completely different sort of quality set to what we experience in movies. Do they connect with you? Do they talk to you? Do they interact with you? So I think the lesson I've taken is what's going to make a good story in virtual reality is completely different to what would have made a good story in movies. And that we have to almost create a new way of judging these works.
[00:07:49.185] Saschka Unseld: I think it's important to unpack the term narrative or story. Because what do we mean when we say narrative or when we say story? And I think that it's most basic, it's that something has a beginning and a middle and an end. And any experience I have has a beginning and middle and end. Like you watch something for two minutes, there's going to be a beginning, there's going to be a middle, there's going to be an end. Like you cannot not have it. So the question then becomes, how do you shape them? Like, we talked with Lost a lot about that. What is the in moment? What is the call to story moment? Like, what's your journey through the time you spend in the VR experience? Because by default, it will be a story. Will it feel like something that comes to an end? That is the big question. And I think there's a million different ways to do that. Movies do that in one way. Poetry does it in a completely different way. But that's the only thing, shaping someone's experience through a beginning, middle, and end, and being aware of it. And then you have a narrative. But it can look however you want it to look.
[00:08:38.690] Maxwell Planck: One thing that I'm really proud of with Dear Angelica is we were trying something new. With each piece we're making, we are exploring this new medium. And that is why I'm proud that if you look at Lost Henry and now Dear Angelica, Dear Angelica looks a lot different than what we were doing there. Because we need to experiment, we need to try new things. And we need to find out how to shape that beginning, middle, end. And I think Dear Angelic is really novel, but we will keep doing this. I think it's going to take a long time before we figure out what narrative in VR looks like. And so we need to keep innovating with not only technology, but cool new creative ideas.
[00:09:15.141] Kent Bye: Yeah, for me, when I did a talk with Devin Dolan from Cinec Media, and he started to categorize the four different types of storytelling on two different axes. One was either you're a ghost within the experience, or you're a character, or you have some sort of impact within the experience, or you don't. So, I think most experiences, whether it's 360 video or all film, you're a passive ghost, and you have no interactivity or agency. And I think the ultimate is, I think, if you have full interactivity and agency, and I think that's where a lot of the artificial intelligence and being able to be fully embodied and have an embodied sense of presence and feeling like you're actually physically co-present with other characters that you're interacting with. You know, Henry felt like it was beautifully rendered, but I didn't feel like I was embodied as a character. I still kind of felt like that ghost, and I think there was a blog post about the Patrick Swayze effect. To me, I kind of thought of that as the plausibility illusion. Like, I didn't feel like I was plausibly there because I didn't have any agency within the experience. And so I think finding those balances between, you know, how do you create that sense of presence and agency within an experience, but yet still have a narrative unfold. And Eric Darnell feels that they're kind of fundamentally in conflict, that agency and empathy are at conflict with each other. And I think that's a little bit what you guys have been struggling with in some ways.
[00:10:32.964] Saschka Unseld: I don't think if they're in conflict with each other. Actually, I also think the bigger problem is that we come into watching something with our past experience of what something should be. Is it a movie or is it a game? That is how the binary way how we see the world. When you see kids see something, they don't give a shit. They're like, this is great, this is fun, while we're like, oh, I wish I could do this or I wish I could do this because I know this from games or I know this from movies. So we kind of try to fit these things into our existing worldview that we were trained for however many years we're on this earth to see the world as. And I think it's also about just letting go of that and like, did you have a good time? Yes, it was great. You can have different kinds of experiences. I think some can be fully interactive and some can be fully passive. Like there's some evenings where I don't want to interact with something, I still want a story. And it's already interactive that I can look around. There was one interesting thing on D'Angelica that we discovered just in the last few months and it came out of a tool we needed for production which was we needed to be able to pause the experience at any time because I needed to give notes and feedback on something and if it just kept running it was kind of difficult because then I needed to re-watch the whole thing till the next minute where I wanted to give notes. So we added a time scrub feature but also a pause feature and the interesting thing was that when you press pause, the images were still alive. So the lines move and we have these things where there's detail drawn in when you walk closer to something. And out of some technical luck, these functionalities were alive when I pressed pause. So suddenly I could pause at any time but still discover things. And that broke open the door to the idea of, hey, wait a minute, maybe it doesn't necessarily need to be a fixed linear story, but it also doesn't need to be branching. It can be more like I go through it at completely my own pace. If I want to discover more things, I can discover more things. If I don't want to, I can just watch it. And I would call that a way of interactivity as well. It's not picking up an object. which we know from the world, but it's a time interactivity that we don't know from the world. So discovering those things I find way more interesting than trying to put existing views of experiences and of games onto what VR could be.
[00:12:43.463] Edward Saatchi: Yeah, I completely agree with Sashko. And there is a generational element to it that like maybe the three of us all grew up playing video games and watching movies and waiting for that holy grail moment where they might come together in some way. And when I watch The Next Generation, as I often do nightly, Or even Voyager. Deep Space Nine has the holosuites, but they don't do as good holodeck episodes. It is the best of the shows. It's social. It's a game. It's a movie. They don't care. All of these lines that are so important to us. and people get so worked up about, are just going to completely fall away and blur. And we, as we've gone through it, I think it's more of a function of the technology than anything else. I don't think it's a creative barrier. There is something queasy about being told that you're a character when you can't actually do very much. It draws attention to itself as an illusion. But just for us, we went from a complete passive thing in Lost, you're just a camera. Maybe the hand acknowledges you a tiny bit, the robot acknowledges you, but you're basically a camera. Henry, the character's looking at you, connecting with you. Dear Angelica, sort of similar connection. In our next project, which we can't talk about specifically, but we're exploring having a character really connect with you much more than Henry. Interact with you, talk to you like, It's more that you're the audience to a monologue in a play as the analogy that Sashka uses, but we're very strongly exploring this and we don't see any line between games and film that isn't being placed there by technology. Creatively, we do want to have experiences that are story, but we're also there, I think anyway.
[00:14:31.758] Saschka Unseld: And I do think like ultimately we are sadly the dinosaurs, like we grew up with film and games, so I think in 50 years it won't be us who revolutionizes that stuff. I think the thing that Jelena Brzezinski announced today in the keynote that working with existing colleges and institutions to kind of help them teach the kids that rise up up now to work in VIs ultimately was going to completely revolutionize everything because they don't have these preconceptions that we have that we constantly like, is this a movie? Is this a game? Does this need, do I need to see my body? Do I not need to see my body? Keep going back to like my mom, the first computer she got was an Atari ST and I still have the manual. and it explains how to click on something, and what moving a mouse means, and that then the cursor moves. And I remembered in these earlier computer models, actually it wasn't a mouse cursor, it was a hand, because everyone felt like we need to represent our hand. And I was like, no, we don't need our hand in there, a cursor is much more functional. So I'm thinking if we're in VR, yes, we can replicate our body, but isn't it even more interesting if we have something completely different? Like why think in body and in computer gaming and in kind of movie terms, the interesting things with games is that a controller allows you to augment what you are and not replicate exactly what you are.
[00:15:45.913] Kent Bye: Yeah, I think that for me in doing over 500 interviews now in the VR space and trying to see different patterns, I've kind of identified what I see as four different types of distinct presence. One is a social presence when you're with other people and you get the sense that you're actually in the same space with them. One is actually embodied presence, and that's where you actually feel like your body is in the experience. And that's something that Mel Slater would call the virtual body ownership illusion, where you actually have the plausibility that you have agency within the experience. And I think that with the touch controllers, you start to get that sense that you actually have a body that's actually in the experience. But then you have active presence, when you're actually doing some sort of activity within the experience, where you don't necessarily need to have a body, but you just have some sort of tool or expression of your agency within the experience. And then the final is like the emotional presence, where you actually feel like your emotions are engaged. And I think all these different levels, especially the story and the narrative, are engaging the emotions in a big way. But I think the big two open questions are, I think social is going to be there, and maybe in sort of group experiences of like, you know, you go through a tour guide of something and there's like a Dungeons and Dragons type of experience but the embodiment with the active presence I think are the big things that are the hardest to integrate within the story. We talked a little bit about expressing your interactivity and there's local agency which means that you can kind of look around in a 360 experience, you can pause it. But then there's a global agency where all your micro-interactions are somehow accumulating and actually impacting the outcome of the story, where you may be having an interaction with a non-player character and those dialogues may be sending you on a completely different branch or a different ending based upon those interactions, where you're doing dynamic, artificial intelligent type of interactions. So I think, to me, it sounds like you're starting to already look into this exploration of the balance between local and global agency and storytelling moving forward.
[00:17:40.818] Saschka Unseld: With that I'd always be kind of, maybe I'm a dinosaur and maybe I'm conservative with this, but I'm not interested in multiple ending stories. I think stories for me are interesting. I watch a movie or I read a book because of a specific artist's voice. I don't want a generated story. I want a human connection with a person who created something. And for me, that person, yes, could have created 20 endings, but I want that person's point of view. And I want that person's point of view and what that person thinks is the right ending for that kind of narrative. Because for me, it's a human connection with the creator. So the more I'm removed from that, the less I'm interested in a story.
[00:18:22.297] Maxwell Planck: Yeah, I think that is the hard part. You mentioned there's a conflict between providing agency and telling a story. I love being told a story well. I love a director doing their craft and kind of sitting back and letting that happen. And so when you're giving more agency, you need to give it in such a way where the director still has control of, you're still the audience. I'm still telling you a story here, but I'm giving you these moments of touch. so that you create more of an empathy with this story. So it's more of a magic trick. It's this illusion of you feel like you're more connected and have more agency, but you also have the confidence that a director is taking me somewhere. And I think that's why sometimes it's a conflict, but I think there's a magic to it. There's an art that we're trying to figure out.
[00:19:01.810] Edward Saatchi: When Sasha was saying that, I was just thinking of Ken Levine. And I think we're all probably in alignment that so long as the ending is curated by the creator, then it feels like a properly curated experience. But I think Ken Levine's voice comes through massively, even though there is quite a lot of meaningful choice in Bioshock or global agency, I guess, as you'd put it. So I don't know, there are games that offer multiple paths, where I feel a singular organizing intelligence is behind it. But I think you're probably right that it needs a single ending, the multiple endings thing might... I agree with you, but I actually think Ken's a good magician.
[00:19:40.826] Maxwell Planck: Yeah, you're right.
[00:19:42.067] Saschka Unseld: You do feel enough agency that you have influence, but actually you don't.
[00:19:46.331] Edward Saatchi: Yeah, you are sort of on the rails in Bioshock Infinite, essentially.
[00:19:49.633] Saschka Unseld: In something like Stanley Parable, the whole point is to give you multiple stuff and have you question it. So if that is your underlying story, then it makes sense again. If you want to confront the player or the viewer or the visitor with that kind of delusion, then it makes sense.
[00:20:04.595] Edward Saatchi: It just shows, and I think this is something that's great for StoryStudio that I'm really proud of, is that the three of us are gamers. And we just listed a few games that have way more sophisticated storytelling than 90% of the movies that I see as a movie lover. I'm not just talking about blockbusters, like 90% of the movies that one sees. So that respect for narrative games, whether it's Tacoma, or Virginia, or Gone Home, or Diresta. That, for us, is really, really motivating. And bringing that into VR movies, I think, is going to be, for next year, we hope to inspire teams that they should be looking at gameplay programmers. They should be hiring people who've done AI in games. They should be looking at level designers and bringing those people into VR movies, not just folks from Pixar and DreamWorks. I think that's where the industry, hopefully, is going to go.
[00:20:57.380] Kent Bye: Are you guys familiar with Facade at all, Andrew Stern's? So in that I think there's like five different endings and you're kind of interacting with the two characters and he's got a drama manager that whether you're going down these different stories there's kind of a natural flow of how that story unfolds in terms of the beats that are made but there's kind of like a random mix of 2,000 dialogue pairs so that when you go through the experience you maybe only experience 15 to 20 percent of those dialogues but to me that feels like an experience that's able to still have a story but still have some branching so I'm not... Yeah, I think it was interesting and fun.
[00:21:33.611] Saschka Unseld: It's an interesting experience. But for me, it's maybe even more at the base. For me, it's like if I watch something or see something, what does that storyteller have to say? I think anyone who tells a story needs to have something to say. Otherwise, I'm like, why are you doing this? And unless that comes across, I don't care or I don't get why something is being made. So that doesn't actually mean that you can't have multiple endings. But what needs to come across is like, what is it that you as a storyteller want to say? And often multiple endings, like in multiple choice books, I feel like it never comes across what the person actually wants to say.
[00:22:09.028] Maxwell Planck: One game I can think of that had multiple endings that I felt like a point was made was Papers, Please. That had multiple endings and my ending felt like mine, but it was an arc.
[00:22:17.978] Saschka Unseld: But you still got what the point of the story was and what the person wanted to say.
[00:22:21.721] Maxwell Planck: It's just hard. You're right. You could have a storyteller who's like, there's 20 endings and I've crafted them all to be really great. It's just really hard to do that. There's always going to be one that's your best.
[00:22:30.887] Saschka Unseld: It makes sense for some stories. Some stories are inherently about having multiple endings, but that doesn't mean that all stories need to have that.
[00:22:38.812] Edward Saatchi: Something that we're exploring made us take a look at LMNO. if you've heard of that game. So Doug Church worked with Spielberg, Doug Church who did many amazing games from the past, worked with Spielberg to create a few hour game that's like a movie and a game. And it was largely based on an interactive character that would kind of connect with you and talk to you. And what we learned and saw was that you can get a lot of meaningful choice just based on how a character is reacting to you, right? Not necessarily on what doesn't actually generally happen that often in life, like these two paths that are massively important. It's more about these human moments and feeling as though you've had an emotional impact on a character, which very rarely happens in video games outside of KOTOR or something like that. So, taking a look at that, I think, as I've been listening to you, Sashka, in the office, the environmental storytelling stuff, right, like Sleep No More, I feel like is less important to Sashka than the character, right? And taking ideas from games about characters and how a main character can respond to you meaningfully, yeah, you seem much less interested in the multiple choice and much more interested in maybe that their character is catering to you and your actions.
[00:23:53.270] Saschka Unseld: I don't know if that's right. I just think often multiple choice and multiple endings is a cop-out of deciding what you want to say. And you see that in a lot of student films where they're like, they want to keep the end ambiguous. I'm like, no, you just don't know what you want to say. And often multiple paths and multiple endings are kind of an easy cop-out of not making a point. And that's why I often just find them very lacking. of, again, a connection between me and the artist, because I think ultimately any kind of art is a connection between me as the consumer, or me as the viewer, me as the visitor, and the artist's creation.
[00:24:28.348] Kent Bye: Yeah, and I think this gets back to the conflict between expressing your agency within an experience and the story that's unfolding. If I look at something like Rick and Morty's experience, that's an experience where it's highly interruptible. So it's the job simulator developers that have created this system to be able to interrupt the dialogue at any moment. And you have to be able to interrupt somebody in order to make it feel like it's plausible. If you throw a shoe in somebody's face, then you want to have them react to whatever you just did. And then maybe if they just ignore you and keep pushing on through the narrative, then it's going to feel like it's not real. You're going to be taken out in that.
[00:25:01.246] Maxwell Planck: It was your choice as a designer to give them a shoe to throw. So if you do that, you have to design for it. So sometimes you just have to make smart constraint choices in what agency you give them needs to be controlled. So if you gave them a shoe and they can be interrupted, that's part of the experience and it should be polished.
[00:25:16.536] Saschka Unseld: I think it's actually a sliding, there's no black and white separation between something that I would say is more on the narrative side and then there's something on the opposite end which is basically a sandbox for you just play around with. And I think that grey zone in the middle is interesting. I do think at one point it becomes more of a sandbox which I tell my story and just have fun with and I just experiment and build weird stuff. But that's purely me creating something versus someone telling me something which is I want to know that artist thing So even if they build something in Minecraft the interesting Minecraft things I'd visit is an artist's voice and Minecraft itself becomes a sandbox You might want to talk about Abzu because I thought what you were saying about that swimming game was absolutely fascinating You were talking about like I think creating a someone getting used to a mechanic Yeah I mean I think that's what's interesting with games is that like the strongest games like Abzu or like The Inside they have a super simple mechanic and then kind of revert your expectation of that mechanic and I think that is for me a narrative because the beginning is you establish a mechanic and the middle is you revert your expectations of the mechanic and the end is the outcome. So I think, again, freeing ourselves from, like, save-the-cat movie structure stuff. A narrative just means beginning, middle, end, and it can be anything.
[00:26:39.433] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:26:47.258] Maxwell Planck: I'm going to go to the next generation. I would be proud if we're referred to like the Lumiere brothers early in film. We are not the Orson Welles, right? The next generation needs to grow up on this and not be hung up on our worldview. So I think that we have a long way to go until this medium finds itself. I think the journey is going to be exciting. I mean, early day film, people paid for Nickelodeons, they paid for vaudeville acts or plays, and those were fascinating, and I think VR has that momentum. But it is going to evolve over many years, and I think a generation before it really finds its true form.
[00:27:21.345] Edward Saatchi: You know, we talk a little bit sometimes about getting to California. We are obviously in California already. But, you know, that all of the studios, we all sort of start on the East Coast in New York, and each state on the way is a big exploration. you know with Lost we just wanted to prove could you tell a story, with Henry the very beginnings of an interactive character, with Dear Angelica quite a few things have fallen out of that, comics, this painted movie and quill and a drama, but you know we want to explore AI, we want to explore social, and I think California to me means you know, what is that product? So that story studio almost is making movies, but building one singular product, which is what we now call a VR movie, but won't in the future. And what is that product that someone would pay for, right? What is a VR movie that someone is going to say, you know what, I want to spend $10 on that. I want to spend $20 on that. Cause that's the biggest contribution we'll ever make is once this actually becomes a viable industry, which I'm sure it will.
[00:28:23.825] Saschka Unseld: The future of VR was the question, right? Or what's the purpose of VR?
[00:28:26.389] Kent Bye: The ultimate potential of virtual reality, yeah.
[00:28:28.712] Saschka Unseld: The ultimate potential of virtual reality. I don't know. The only stuff that comes to my mind are weirdly hyperbolic terms that I would then hate myself for saying. I don't know. I think I like how it challenges my brain to work in it and to create in it and how it makes me move out of my comfort zone to create in it and not fall back onto low. I know how the movies are made. I like the challenge of it. I think that's why I'm in it and the potential of it. I don't know. I think everything. I think you can do everything with it. I think there's something there that I'm trying to formulate. Whatever it is I'm searching for, whatever it is, I think it's there. But I think in hindsight we'll be able to put it to words.
[00:29:14.178] Maxwell Planck: The thing that's exciting for me is that there's so many problems to solve and so much territory in front of us that I could do this for the rest of my career. So I don't know what the ultimate potential is, but that's what's exciting about this is that we've got such a long way to go that I'm an engineer who loves to solve problems. And I got a lot in front of me.
[00:29:30.048] Edward Saatchi: Just to talk about Dear Angelica for two seconds, Max always talks about made in VR, right? That we shouldn't maybe be using tools that were created to make games or tools that were created to make movies, right? This is a completely new art form. And I think that Dear Angelica is the first movie made almost completely in virtual reality. So that's just a hint at how early we are, that nothing yet has even been made with tools designed for this medium, right? We're still just pathetically putting together old tools for old mediums, or not old, but past, not past, other mediums.
[00:30:08.679] Saschka Unseld: Yeah and it's interesting like so during the Angelica we made Quill just for Wesley so she could paint in VR in her style and now we started to work with more illustrators just to make sure the tool doesn't have a style of its own so that it's always the illustrators or the artist style. But we also started working with comic book artists because we're adding functionality so that you can build the Angelica like stories just with Quill. And I find it really interesting to see their learning curve, because it took us really long to get here. But if I look at those people, for a week, they kind of imitate what they used to do in comics and in flat illustrations. And then they already evolve into breaking out of that boundary. And by the third week, they actually throw their first two weeks away and really break into it. But seeing that happen within three weeks, because they can so instantaneously create in VR, is great to see because it really took us way too long to kind of slowly shed the shackles of our past. So I think that Made in VR and Creating in VR kind of accelerates that process so much. So seeing what those comic guys and girls are creating is amazing within the short learning span that they have.
[00:31:16.219] Kent Bye: Awesome. Well, thank you so much. Thank you.
[00:31:19.002] Saschka Unseld: It was great. Thanks. Thanks, Kent. Love your podcast.
[00:31:23.108] Kent Bye: So that was three of the co-founders of Oculus Story Studio with Maxwell Plank, Edward Saatchi, and Shostka Unseld. So let me just go through some of the big takeaways from this interview, and then I'm going to unpack some of my larger thoughts about storytelling and virtual reality. So first of all, I think Sashka is trying to put forth a pretty simple framework for how you think about and define a story. A story is something that has a beginning, middle, and end. And that's it. As long as you're able to accomplish that sense of an arc that has a beginning, middle, and end, then you could reasonably call that a story. The other big thing that I took away is that as creators of VR narrative experiences at the Oculus Story Studio, the thing that Edward said is that it kind of feels like they have the dead hand of cinema hovering over them. And the way that Sashka described it is that anybody that's going in and experiencing some of these VR experiences are bringing a lot of predetermined expectations based upon their experiences of either gaming as well as in film. From Sashko's perspective, he is really quite skeptical of stories with multiple endings. Not that it's completely impossible, but he just sees that anything that is ambiguous as an ending or as multiple endings for him just kind of dilutes the message of what the creator is actually trying to say. For him, it's a process of looking at the story and being able to connect with the content creator and have a clear message of what is trying to be communicated within the narrative. So the way that I kind of interpret that is that it seems like on the spectrum of authored narrative versus highly dynamic interactive experiences, it seems like they're actually more leaning towards the passive ghost type of experiences, where you may feel like you're present, but you're kind of a ghost. You don't really have any real true global agency within the story that's unfolding. and that any interaction that you may have may be of small consequence, where you may be able to make eye contact with the character and have a connection, but that type of local agency isn't going to drive any sort of different outcome. And it's going to kind of feel like you're on rails. And I think that is one dimension of VR and storytelling. And in my episode number 292, talking with Devin Dolan, talking about the four different types of storytelling in VR, That is one of the quadrants where you are a passive ghost with no impact. On the other extreme of that spectrum, you are a character and you do have impact. I think that's where it starts to get really interesting into thinking about some of these other deeper questions about how much agency you're able to actually exert within the experience. So I think this is one of the unique affordances of the VR medium that we haven't been able to really have before. We've had interactive games, but we haven't been able to actually cultivate that sense of embodied presence. And I think that sense of presence within an experience changes things in so many different ways. So there's actually a couple of statements from this interview that I both on the surface agree with, but also disagree with. And I'll just read those now. So first of all, it was said that I think it's going to take a long time to figure out what narrative in VR looks like, as well as I think we have a long way to go before this medium finds itself. So I think that overall, these statements are both true, and I think it is a process that is unfolding. At the same time, I would argue that we actually have figured out quite a lot of it if you look at the entire canon of different experiences that have been created out there by the entire virtual reality community. At this point, Oculus Studio has essentially done like three experiences that they've completed and are working on the fourth, and they've done lots of exploration. But I actually think if you look at the wider VR community, there's been a lot of exploration. And over the last couple of years, I've done over 500 interviews. really exploring the medium of virtual reality across all the different applications, but specifically about 10% of those have been specifically focusing in on storytelling in VR. And so I made a top 50 list of all the things that I've learned so far about storytelling in VR, and I wanted to just kind of cluster those into seven different categories and try to extrapolate a little bit of what I see so far happening in the realm of storytelling in VR. First of all, there's the category of the basic language and grammar of virtual reality storytelling. And this is going into different details about like editing and cuts and how do you sequence stories. And, you know, there's examples like Pearl, for example, is an amazing experience where they are able to include lots of different edits. So there's lots to be learned from experiences like Sonar. And what can Broadway teach VR in terms of looking at it from more of a theater perspective rather than film? And again, this interview with Devon Dolan, I think, gives a great overview of the four different types of storytelling in VR as to whether or not you're a ghost or character and whether or not you have impact or not. The next major theme that I think has been emerging is exploring the limits of interactivity in storytelling. You know, I think that at the extreme of being able to have highly dynamic interactive conversations, I think actually is going to require at some point either artificial intelligence once it gets good enough. But in the short term, we've already started to see some experiences where people have started to use things like improv actors, as well as a living story. So looking at what kind of inspiration we can take from immersive theater, but also like interaction based upon gaze and triggering things. And this fundamental trade-off between empathy and interactivity, as Eric Darnau talked about in episode 290. The next major cluster of themes that I think have been emerging within storytelling in VR is this concept of multiple perspectives and empathy in storytelling. So Rose Trochet and perspective talks about telling a story from five different perspectives and it's the same story but being able to be situated in different location and from different perspectives, you actually get a more robust experience of the entire story from everybody's perspective. And that just naturally starts to cultivate a sense of empathy of being able to actually step into somebody else's shoes and be able to see the world through their eyes. And so I've also done some interviews with Donny de la Pena talking about this exploration of empathy and storytelling. and how having that sense of embodied presence in the VR has some unique qualities of being able to trigger these feelings of empathy that go beyond the capabilities of any other medium. The next big thing that I see coming up that's unique to VR is social storytelling. So, for example, going into a VR chat and walking around looking at a location, but doing that within the context of a group and having a social experience with other people where you're able to dynamically interact with them at the same time And so that the story that's emerging is kind of the collective story that is being told and you're able to experience it by also telling your story and being able to have conversations with other people in the context of being in these tour guide exploration context of these different realms of the metaverse. This also includes insights from Dungeons & Dragons and what can we learn from collaborative storytelling from D&D, but also this Wizard of Oz concept where you have improv actors that are in a show and being able to actually interact with them in real time. And other experiences that do that in an asynchronous way with Mindshow VR, where you're able to actually act out different stories with different characters. The next big theme that I think has been emerging is this concept of world building and environmental storytelling. So we've been having environmental storytelling with Myst and other experiences like that, but I think with Abduction, as well as the sense of being embodied within a place within a VR experience, it starts to do something different to our bodies. Just this principle of embodied cognition says that we think beyond just our minds, that our bodies are involved in that thinking process, but also our environment. So what's it mean when you're able to actually put somebody into this synthetically created environment? You're able to actually evoke different memories from them. but also the objects within that environment represent memories and stories. And so you're able to actually have a exploration of narrative by going around and looking at the objects and being able to discover more about your world. And so we talked about that with Alex McDowell and Denny Unger and the future of the gallery and just other types of experiences where you're put into a physical location and a lot of the story is emerging from actually being immersed within that place. The next theme of topics that I think have been emerging within storytelling in VR is really understanding these concepts of plausibility and presence. And I think that the first time that I heard of this was from Rob Morgan back in episode 125 when he was saying, you know, in the Assembly, when you're in a VR experience and you're interacting with these NPCs, these concepts of body language and eye contact and being able to interrupt your NPCs and if you walk away from them, they should stop talking, These are all types of things that when you actually have an embodied presence, these are the types of social cues that you come to expect in order to believe that these characters are plausible. And when they don't meet that expectation, then they start to fall into the uncanny valley. So learning about the concepts of the uncanny valley and plausibility and the research from Mel Slater, talking about the different components of plausibility, and body language, all these things I think are key components to actually creating those experiences where you want to have some sort of highly dynamic interaction with other characters within the experience. And that by researching what people have been talking about with plausibility and presence can give a lot of insight for how to actually cultivate that and find that sweet spot of being able to be present with a character that's telling you a story, but also make it feel like you're able to actually interact with it and it's plausible. I think one of the best experiences that I've seen so far that's been able to pull that off has been the Rick and Morty simulator that I talked about with Alex Schwartz in episode 433. And finally the last theme that I just wanted to pull out is just the importance of audio in terms of storytelling in VR and all the different dimensions of spatialized audio and just learning more about that because right now at this point it's the visuals and the audio that are the two major things in terms of being able to tell a story. Now, there's other haptics of being able to actually do beyond room scale and 4D experiences and installation with different things like the void. But I think at this point, for the most of the storytelling in VR, it's mostly focusing on the visuals and the audio. And nailing those two can actually give a pretty compelling narrative experience at this point. So I just wanted to kind of highlight and extrapolate some of the larger themes that I've seen based upon the top 50 interviews that I've done so far about really investigating storytelling in VR and kind of use this podcast as a launching point for people who are starting to get into storytelling in VR and people that are trying to figure out all the different nuances of the VR medium and what they can do in order to really push the limits of what's possible beyond just these passive 360 videos that are out there and really start to use the VR medium to its full extent, I think, is what some of the explorations that Oculus Story Studio is already starting to look at in terms of how do you create a connection to the character and How are you able to allow that character to still tell the story but yet you feel intimately connected to it and you're able to have that sense of embodied presence with them. So that's all that I have for today. I just wanted to thank you again for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do tell your friends, spread the word and become a donor to the Patreon. You know, just a few dollars a month. It really does make a huge difference to ensure that I can continue to bring you this type of coverage. So you can go to patreon.com slash Voices of VR. Thanks for listening.
