Catherine Rehwinkel is working on creating a conceptual framework that enables storytellers to storyboard linear stories in VR. She’s a filmmaker who recently finished a master’s degree computational & systems thinking at NYU. She’s been inspired by Donna Haraway’s “Situated Knowledges” feminist theories that take into account how location and place impact our perspective on events, and she’s intrigued by VR stories that grow and evolve when you watch them from different locations.
LISTEN TO THE VOICES OF VR PODCAST
One way that I understand the importance of location in VR storytelling is looking at Rose Troche’s Perspective series where she explores how a narrative story can change if you watch the same events through different character’s eyes. Rose concluded that the first-person perspective is extremely vulnerable, and in order to get a more complete picture of an event then it’s helpful to take into account many different perspectives from different people.
Similarly, Donna Haraway doesn’t believe that we can have a truly objective, passive, or omniscient scientific observations that are independent of our subjectivity. Instead, Haraway’s concept of “situated knowledges” is calling for us to think of people as a lot more messy and complicated creatures who are full of contradictions. Situated knowledges can be described by thinking about subjects who become “complex contraptions made of biological vision and personal will, the scientific gaze is dissolved into a network of contested observations, and objects become Coyote-Frankensteins, produced and yet much more in control than the traditional modest witness would care to admit.”
In order to get a more complete picture of any topic, then you have to triangulate between many different complex and contradictory subjective perspectives (incidentally, this is part of the philosophy and intention driving the Voices of VR podcast). Catherine Rehwinkel believes that VR is particularly well-suited medium for simulating the multitude of different perspectives through the simple mechanism of changing your location as you watch and re-watch a series of linear events take place. And if the narrative is constructed well enough, then your understanding of the story could continue to evolve and grow as you watch from many different vantage points. Catherine believes in this vision for storytelling in VR, and is in the process of building conceptual tools for storytellers and VR designers to storyboard, architect, and prototype these types of experiences inspired by the concept of situated knowledges.
You can learn more about Catherine’s Narrative VR Framework from her recent VRSalon.org talk about Google NYC.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to The Voices of VR Podcast. On today's episode, I have Katherine Raewinkle, who's a filmmaker that's been working on a conceptual framework in order to help create linear stories within virtual reality. So Katherine's taking a step back and trying to think of some higher level metaphors and analogies in order to describe the differences between the film medium and virtual reality. And so she's trying to come up with a framework for virtual reality designers to be able to start to world build and experiment with changing different perspectives and see how that is able to help define the story that they're creating. Catherine's taking a lot of inspiration from Donna Hathaway's situated knowledges and how multiple perspectives are really needed to get the full picture. So we'll be taking a really high level deep dive into narrative in VR, and some of the tools that are needed in order to really start to do 3D storyboards. So that's what we'll be covering on today's episode of the Voices of VR podcast. But first, a quick word from our sponsors. Today's episode is brought to you by The Virtual Reality Company. VRC is at the intersection of technology and entertainment, creating interactive storytelling experiences. The thing that's unique about VRC is that they have strategic partnerships with companies like Dbox, which is a haptic chair that takes immersion and presence to the next level. So they're making these digital out-of-home experiences for movies, studios, and original content. For more information, check out thevrcompany.com. Today's episode is also brought to you by The VR Society, which is a new organization made up of major Hollywood studios. The intention is to do consumer research, content production seminars, as well as give away awards to VR professionals. They're going to be hosting a big conference in the fall in Los Angeles to share ideas, experiences, and challenges with other VR professionals. To get more information, check out thevrsociety.com. So this interview with Catherine happened on Saturday, July 16th, right after a VR AR meetup in New York City. So with that, let's go ahead and dive right in.
[00:02:18.652] Catherine Rehwinkel: Hi, I'm Catherine Rehwinkle, and I'm a filmmaker, and I recently just completed two and a half years of studying computationalism and systems thinking for non-technical people, or outside of the strictures of science and fields and subfields. So I'm right now building a conceptual framework for VR, storytelling in VR, for linear stories. And by linear stories I mean event flows. I'm not interested in branching narratives. That's not how reality works in my mind. Events follow each other and that's what has happened. So in looking at historical events for the purpose of telling a story, Film for a century has been based on one perspective. Accommodating one perspective, one observer's perspective to watch a string of linear events and that's what you get. And in VR there's opportunity to basically create a space. you design a space, then you design the elements that are involved in events. And so you plot that out when you create the space. So here's where the difference between film and VR, what VR can do lies, is perspective. So if you think about the film frame as a state, a physical, spatial state, and you think about time as change, in that space, spatial change. How then can you take each state, each frame, and divorce the viewer's perspective, which in film is linked very directly to the frame, right? You have a certain perspective on a per frame basis, and those things are one in the same. But in VR, you can have, with photogrammetry and 3D environments, you can have any perspective you want on that state or frame. So how do you design a space with linear events happening that the viewer can take any perspective and do that in a way that is not too expensive? So, the way I view film is a car ride, basically. You're on a road trip and then you're the passenger and you have a small window to look out onto the world. And you're on a road and the director is the one determining the speed. Sometimes you're blindfolded. That's editing, you know? You're cutting out time and space with editing. But, you know, virtual reality is a form of reality, a representation of reality. How can virtual reality be reality with value added? And another notion to keep in mind is if you're looking at a story as an object, you can go as high up above to look at it as possible and see all of those states, all the states of that environment and the elements contained within it at once, what would that look like? Imagine a whole story as an object, a crystalline object that you can turn around. And they say you can't see all of one object, all of something at once. So how do you cheat that and view something all at once? You have a frame rate for the world. I call that the inertial frame rate, to borrow a physics term. And then you have the frame rate of the observer or the observer's vehicle, the camera. That's the non-inertial frame rate. So you can view an environment or a story event in one moment, in one state, and you can turn it around and look at it from any perspective you want in one instant. But your camera, your observer camera, obviously that has its own frame rate. So you're looking at something all at once by having a camera that its frame rate is divorced from the frame rate of the environment.
[00:06:41.160] Kent Bye: So I'm sort of getting these higher-level analogies and metaphors and I really like the car metaphor because you get this sense of when you're watching a movie it is like being in a passenger seat and the director is the driver and in some sense the way that he's driving through the arc of the story is his perspective and his editorial decision but yet VR experience is less like being in a car and more of like just being on a stage where maybe there's a theater actors that are around you and there may be things that are happening all around you but you can look in 360 degrees and rather than moving through this trajectory of a linear perspective, you're kind of looking at actors around you in the stage. As well as when you talk about taking this snapshot and looking down on these objects, what I think of is, like, I get this image of, like, taking a photograph, and that photograph kind of represents a frame. But yet, in VR, it's more of, like, making a 3D model clay sculpture of the scene. You could pick up and look around. So, to me, it feels much more like architecture, designing spaces, than set design. So that's the initial thoughts I have of what you're saying. Trying to use these high-level computational thinking metaphors or analogies to be able to give people an understanding of the difference between these two different mediums. And so, what then, once you have this 3D model clay, what's the innovation beyond doing gaze-based control or anything else? What kind of breakthroughs are you able to have in terms of thinking about linear storytelling by using these analogies?
[00:08:10.027] Catherine Rehwinkel: So to go back to the car metaphor for a second, you know, you're talking about being on a theater stage. To expand the car metaphor, think about it as a submarine, right? And as you said, it's like designing a space or the architecture of a story event space. So you're a submarine with a periscope moving around and you're designing the ocean. As the VR designer, you design the ocean. Perhaps some stories, if we were to take this room here and we wanted to tell a story in one minute of all of the conversations happening right now leading up to some dramatic event. You have an ocean-like design. We just have the room. I can place the camera here between us. I can place the camera over there between those people. And I can have, say, nine cameras that are running concurrently, recording the whole space for one minute. But if I have nine cameras, that's nine minutes of content, if the viewer cares to view each of those subspaces. And so, if you have an ocean-like space that you've designed, in this case, that would be a good design metaphor. However, if you want to explore the story of a family saga that takes place over 700 years, That's a long time to ask the viewer to make choices about where they're choosing to observe events from the story. So, in that case, you may design a river rather than an ocean. Which brings us a little closer to the strictly controlled car metaphor of film. You're driving on a road and you're only seeing out the window what the director wants you to see. However, in this case, you have a river of space that's following some events. You're constrained because you're not going into the spaces that are not relevant to the story over time. However, you're still giving the viewer some control over their perspective and you can liken that to a camera operator.
[00:10:20.231] Kent Bye: So in this case with the river metaphor you're trying to say there's a flow of the arc of the story but yet they're like in a kayak where they can look all around and maybe you kind of want them to look forward in some ways because that's where the main action is but they could be looking in other directions. But you're sort of moving through different scenes in some ways like rather than just being one scene you're kind of following the flow of a full arc of a story but that could span 700 years or In a lot of ways, film is like a science of studying time. It's time compression. It's like artists' way of taking and compressing time. You get this time compression. And through that time compression, you get this experience of a story. And I think that drama is like life with all the boring bits taken out. It's what Andrew Stern told me. It's like the drama in the story is like taking a bit out all the parts. So there is a bit of a still constructing time. So that's sort of, as you were saying, I have those thoughts as well.
[00:11:11.930] Catherine Rehwinkel: Well, I think when I was actually talking to Amelia Winger-Bearskin about this the other day, time as a concept is just an attribute that you can use to describe events. Time is only the rate of spatial change. You know, that's what defines time is like the movement of the planets or whatever. So when you're talking about time-based media or film as a compression of time, you're talking about simply like film's reliance on time as a way to describe the events that are happening in a space. you know, as they happen in reality. So because we can actually have, you know, with 3D or photogrammetry, we can create all of the spaces that a story takes place in and then have them exist concurrently with the areas where events are happening, which may be in different spaces. you now simply have space. And so in the framework that I'm working on, as you scrub through space, you're scrubbing through different spatial scenes, like where scenes happen. You're also scrubbing through time. So if you move in space, you have to stop time in order to see events that are happening at once. You know, events that are happening simultaneously, now you can accommodate that. In traditional film, you can't accommodate simultaneous events. You have to do a lot of setup and really make people understand that this is a flashback or whatever.
[00:12:51.330] Kent Bye: Yeah, the thing that makes me think of is Sleep No More which you literally have a warehouse with like a hundred rooms and like 21 actors and each actor in some sense is their own self-contained story as they're going through the space and they're moving between room to room and interacting with other characters and so you could sit in one spot and see whatever characters come into those rooms or you could follow characters but And in some ways, that's an example of a film where you're a ghost, but you have no impact on the story. So you have local agency, but you don't really have any global agency. You're not really impacting the story at all. But still, you're making a choice of many different rooms to go to, just like the example here of all these different conversations. You can think of that in the same way. If you put the camera in the middle, imagine you looking in a different direction. You could only hear that conversation. That, in some sense, would be also making a choice. and parallel events happening and giving you the choice and the agency to decide what to look and pay attention to.
[00:13:51.180] Catherine Rehwinkel: Right. And in this case, the viewer is not asked to make choices. You, as the designer, and I use the term designer instead of director, as the designer, you're choosing for them what choices they have. And it's not about making a choice, because they're not sacrificing. They're not, at any given moment, if they're missing out on seeing something, they can go right back to the thing that they've met. They can watch something, and then they want to know, hmm, what's happening at the exact same time? Stop time, or go back in time, move, and see that event from another perspective. And by event, I just mean the state of the entire story event space.
[00:14:35.125] Kent Bye: So there was an experience at Sundance this year that was called Perspective. And the idea was that it was a police shooting with two cops and two African-American adolescents. And so everybody in the story messes up in some way. and one of the kids gets shot but they have four rounds that you watch it and you're able to see intercut like one iteration you see a sample of four the next one you see another sample of four and then so you get a sense of the story but each time you watch one of those iterations you get a new perspective from the different viewpoints so in some ways Grosstrache, the director, says there's a certain amount of vulnerability of the first-person perspective, like seeing things from one way. But the thing that I think VR has to do is you have the capability to actually see the same story unfold from these multiple perspectives. In this case, it happened to be from the first-person perspective of the camera. But it sounds like a little bit what you're saying is that you could watch a scene from one viewpoint, but yet if you stop time and go back in time and watch it from a different viewpoint, you may actually have a different interpretation or more information.
[00:15:41.005] Catherine Rehwinkel: That's right. And, you know, something that has been my motivation for developing this is the idea that for the last 100 years, there's been, you know, the industrial revolution has caused what we are now calling the Anthropocene. And that's tied with the scientific revolution. That's the idea that looking at some data, proving it, and then building on that, in a kind of a tunnel-like or probe-like perspective on reality. That's how you create predictive control over understanding events, phenomena. And filmmaking is no different, traditionally. You're watching a string of events from a certain perspective, and that gives you a certain level of certainty about those events. Because you are seeing those events from that perspective, you know that that is how you are meant to view the events. And I think now we need pluralistic perspectives. We need to be able to view things that are happening around us from different concepts of embodiment. We need to be able to make choices about how we view events based on our specific situation. You know, Donna Haraway talks about situated knowledges. in science like the importance of being able to acknowledge that where you're viewing phenomena from impacts your understanding of the phenomena and then in turn impacts the phenomena and I think that VR in general holds the promise of opening people's minds to understanding that there is no one way to view events and If you can change the idea that there's not just one way to see reality, reality becomes emergent if you can view reality from different perspectives.
[00:17:46.210] Kent Bye: So what I get from that, what you're saying in some ways is that you talked about earlier this linear trajectory of the story, but then you said there is this mechanism of being able to stop time, move to a different perspective, perhaps even rewind a little bit. And so within a film, you wouldn't really necessarily get anything new or different if you rewind and watch the scene over. You would just see the exact same thing you saw. In VR, what you're saying is that you could stop time, even though it's the same linear story, you're kind of giving people an option to be able to kind of teleport around and get a sense of being able to see the full vision. And that by those different perspectives, you have new situated knowledge that you're able to see something different. So what would be a concrete example of a scene or a story that you would get something new or different from a scene that unfolded and you change your position in the room, what would that give you?
[00:18:36.930] Catherine Rehwinkel: Okay, so you're not changing, as you said, you're not changing the events. That's a linear string of events that do not change. However, you're changing, as an interaction, you're changing your own understanding of the events and what you take away. So I can have a story space and I can choose to view it from any perspective that I want. And that means that there are multitudes of stories contained within one story. You can make as many frames of the story that there are multiplied by as many perspectives that are possible. That's how many different stories you can create. So, you know, I can have a hundred thousand different versions of a story. And that's as concrete, I think, as I can describe. Like, if there's a scene that takes place and it takes place in, like, maybe the bride's mother's bedroom, and it takes place in the wedding chapel. And at the beginning of the day that this story takes place, most of the action is taking place in the mother-in-law's bedroom. and then the rest of the story is taking place in the chapel. However, I can choose to stay in the mother-in-law's bedroom for the duration of those events. And I can just come away with a new story. It's a new edit. It's new camera angles and those angles cut together.
[00:20:10.448] Kent Bye: Yeah, the thing that I think of that's sort of a concrete example that I saw in a film perspective is that in this experience there's the two adolescents, one of them like steals an apple and they get caught by the police and there's this confrontation and then one of the police goes away and they go around and you know one kid ends up on the ground and the other kid is sort of like starts throwing things at the cop. So this is like, as you watch each of the different perspectives, you sort of see different things. And the first versions, I didn't see that the kid who wasn't on the ground started throwing things at the cop. And so then the other cop sort of sees him throwing things at the cop, which from their perspective is like use of force. They have certain protocol, but they're allowed to use a certain amount of force proportional to how much force is being exerted. Well, the first three times I saw the story, I didn't see the kid throw anything, and so it was like completely unprovoked. And then when I watched the last version, then I was like, oh, well, that officer didn't see everything that led up to that. He was missing that context. He's just coming in, seeing things being thrown at, which, you know, is a big thing you're not supposed to do. And so then he ends up getting shot. The point being is that I didn't see that piece of information that kind of changed my perspective. It literally was like, oh, well, I guess it's not so clear cut. You know, the director, Rose Trochet, said every single one of the characters makes a mistake, and it's ambiguous. Each one of them have things that they didn't do quite right, and you can't fully empathize with that until you see each of their perspectives to be able to see how none of them had perfect information.
[00:21:47.277] Catherine Rehwinkel: Yeah, I mean, it's ambiguous if you're viewing it from one perspective. But now, as you know, since you saw it from all three perspectives, you know what the truth is. But none of those are viewable at one time. So, depending on what you're observing, there is one perspective, and then those events can be ambiguous. But because you can have all of those perspectives, it's no longer ambiguous. You know what happened.
[00:22:15.998] Kent Bye: Yeah, and I think the thing that makes me think of is that we talk a lot about VR teaching empathy, and I think that being able to maintain multiple ontologies, multiple viewpoints, multiple worldviews, multiple perspectives at the same moment, I think that going through these types of experiences sort of gave me just a sense of like, oh, well, I want to hear both sides of the story. wanting to actually empathize and really understand what may have been happening in the context for what everybody was having to go through and I think that to me is like the potential of VR is to start to build that type of empathy from that multiple perspective rather than the singular perspective.
[00:22:53.092] Catherine Rehwinkel: Yes, I agree with you completely. And empathy, I'm just riffing right now, but you can think of empathy as probabilistic understanding of someone's situation in a series of events. And if film has, over the last century, taught audiences about experiences that are beyond what their lifetimes could contain in lived experience, then perhaps now VR can teach people a new way to view the probabilities of the way events unfold for other individuals and other environments. And VR, I mean, this is getting pretty idealistic, but VR and this thinking of situated knowledges and multiple perspectives and being able to contain all knowledge of an event in storytelling, perhaps can re-educate modern society about the nature of reality. That there are many things that, depending on where you are, where you're standing on the sidewalk, you see something but your mind creates a model and you know that it's likely that if I were standing over there by that trash can across the street, I would be able to see something else that I can't see.
[00:24:14.157] Kent Bye: Yeah, it's like you're just putting yourself into other people's shoes, literally, like, as you're walking around, and I think it kind of changes your perspective. And so, like, talking about the specific toolset that you're building, it sounds like you are trying to develop some sort of conceptual framework for content creators, storytellers, filmmakers kind of moving into VR, but also just VR experience makers. to have some sort of way to construct these scenes in a way that would accommodate a variation of the story depending on your perspective. And so maybe you could tell me about that. What does that look like?
[00:24:48.054] Catherine Rehwinkel: Sure. So the other day, I spoke about this for the first time. And the phrase I used was a physics engine for viewing a screenplay before it's produced, shot, and edited. A screenplay is a really nice basis for VR storytelling already because it tells us only what we can see and hear and a spec screenplay, not a production screenplay, doesn't specify shots. It just tells you the events that are happening and whether you can see them or hear them. And it has a certain perspective, similarly to the way a book, a novel might have a perspective. Certain styles of novel might have a very kind of realistic perspective that doesn't go into the character's thoughts and it doesn't have flashbacks. It's just telling you what you're seeing.
[00:25:47.771] Kent Bye: So you describe it as a screenplay, but I kind of think of the visual representation of a film as a storyboard. What does a 3D storyboard look like?
[00:25:56.712] Catherine Rehwinkel: A 3D storyboard is a set with characters blocked out in it. Now you have a camera that you can move around for any given state or frame, any time slice of the events and characters involved, and you can position your camera any way you want at any time in your understanding of the events as the viewer and as the writer. Let me rewind for a second. The framework that I'm building is to create a common language between developers, people who are developing this technology, and writers, creators, designers. There needs to be a common language for what is possible. Just as in traditional film, you have very established industry standards. You know what's possible, you know what it costs. And that is what screenwriters use. Professional screenwriters use this knowledge when they're writing a screenplay. It informs what they can write. And in creating a computationally based framework that accommodates narrative. I want to provide writers and designers with the knowledge of what is possible computationally.
[00:27:19.538] Kent Bye: You know, there was a talk by Alex McDowell at the Unity Vision Summit, and he talks about this concept of world building. And he talked about working on Minority Report, which, before there was even a script, they brought together all these futurists to really dream up what the actual set looked like, to create the world. And as they create the world, then they're able to understand, like, oh, well, in this world, there's, like, elevators that go up. And so then they can have a scene that happens on that elevator. but they never would have been able to create that scene on the elevator without the context of that world. So it sounds like through the process of creating that world and doing a little bit of world building and having some sort of cross-disciplinary way of describing and visualizing and understanding that world, perhaps even experiencing it within a VR experience, then they're able to take that as a basis to be able to actually do the construction of the story.
[00:28:09.360] Catherine Rehwinkel: Yes, and I think that's where developing a tool that allows a writer to write as they move a camera around to explore these individual states of the world that you've built and to view the events that are happening in them from multiple perspectives. So the writer is armed with the capability of putting themselves in the viewer's shoes. What is the viewer interested in looking at right now? What am I interested in looking at? It's similar to if you're at a natural history museum and you look at a diorama. You're going to move around in order to, you know, look more closely at this frozen moment because Already you know that there are interesting details to learn from when you're looking at a diorama. That's why the diorama is there. So it's simultaneously being the viewer as you're writing and using the convincing optical physics that traditional film has pioneered for us. In order to move a story, a narrative, along, you know, there's a lot to take from traditional film, and our eyes can only see 120 degrees, or whatever, of field of view. So, what are you placing together? What information are you placing in the frame?
[00:29:39.490] Kent Bye: Yeah, it sounds like that because historically in film a screenwriter could completely construct the vision of the world and have a very tight control over that, what that looked like from that perspective. Sometimes they would start perhaps a graphic novel or comic and then kind of have the whole pipeline of the video production come in and flesh out the script and have the director and there's this well-defined process but yet In VR it's not as well defined and you could actually have a bit more of a feedback loop is what I'm hearing is that you actually kind of build out the world a little bit more and the scenes and the blocking and you're able to kind of like place yourself in that embodiment of that world and kind of get that vibe and then from that perspective you're able to perhaps be inspired to go back and write specific aspects of the script.
[00:30:23.291] Catherine Rehwinkel: That's right. And then this gives the writer the ability to actually place a camera in the 3D world and visualize where that camera is from a meta-camera, a camera that can see what is being shown, you know, similarly to an outline, a writing outline, but it's a camera outline. And, you know, writing is perspective. And I think this is something I wanted to mention for a little while in our conversation, the novel. There's the big debate of should I see the movie first or should I read the book first? I always recommend seeing the movie first because the book is always going to give you more detail into a world that you love.
[00:31:05.653] Kent Bye: I've heard the opposite. A lot of people say you want to not be biased by someone else's perspective, be able to construct your own vision of it, and then you're able to perhaps appreciate someone else's perspective a little bit more without unduly biased or influenced by someone else's perspective.
[00:31:20.976] Catherine Rehwinkel: I mean, I've heard that too, but that viewpoint is prioritizing casting as an important way to understand a story. It's prioritizing location, art direction. It's prioritizing relatively superficial elements as things that might bias your vision of the story. A novel is always going to be able to contain, through words and the writer's perspective, visual perspective, many more pieces of information to help you model that world and that string of events than a movie ever can for the reasons that we've been discussing. So I may be, you know, alone in that viewpoint, but that's been an inspiration for developing this conceptual framework. This is allowing the writer to write, like, the screenplay version of a novel. and allow the viewer to pour over each page of that story in the form of frozen moments that the viewer can reposition their embodied perspective. And, you know, to talk about embodiment for a second, in a traditional film, embodiment is already accounted for in the production. In the creation of the story, embodiment is where the camera is shooting the action, and the action is informed by where the camera is shooting it from. In this new framework, that embodiment is also accounted for by the design, the writing from the very beginning. What events are you able to see from any angle? And those events better be important.
[00:33:03.219] Kent Bye: So what's next for this project moving forward?
[00:33:06.310] Catherine Rehwinkel: Well, so I have two projects that I'm kind of building as use cases for this because I think you need to tell a compelling story in order to prove to people that this is a valuable way to take in stories. So one story is a reincarnation story. It's the story of a bandwidth of energy that propagates through different life forms. Perhaps it's 50 mice. in one generation, one person in the next generation, or a thousand androids in one generation. And it's about using space to communicate the ideas of time. So as you transition through space, you're also moving through time. And you can also see from high up, you can also see how this force, this life force moves through all the different iterations of reincarnation. And then another story that I'm working on is about an artificial intelligence who saves humanity by taking away humanity's ability to use language. so that humans are restricted to the experience of communicating with their embodied relationship to each other. What I'm trying to do with these stories is actually use the concepts, the abstract concepts in this framework, as the material for the stories.
[00:34:42.598] Kent Bye: Yeah, it reminds me that Sleep No More is a completely non-verbal experience. It's all through interpretive dance, and so they're using their embodiment to tell the story, and so I think there's something that's really smart of that. A couple things. One, it's, first of all, it's universal. You don't need to know English to be able to experience it. You can come from any cultural background and still get the sense of what they're doing by the relationship of their bodies and kind of emotional expressiveness relative to each other. But also, by not using language, it sort of puts us more into the right brain and less into the left brain. So it just creates more of this Kairos time of being out of time and out of the Kronos aspect of the regimented aspect of time. So it's a little bit more like a dream state, a lucid dream. So that was sort of my personal experience of that. But yeah, it's interesting to hear the AI threads as I'm just coming back from 60 interviews on the Voices of AI podcast that I'm about to kick off. So yeah, you'd mentioned machine learning. Is there a sort of interest in machine learning and AI for the story? Or are you also kind of incorporating this into the actual VR technology as well?
[00:35:47.408] Catherine Rehwinkel: Yes, it's perhaps too much to go into now, but my thesis at the program I just graduated from was a way to analyze film and to use the propositional relationship of what is described in the screenplay over time and space. What can you see and what can you hear? And to connect those semantic terms to frames, specific video frames and audio sample frames, and you already know what is in the frame because you're connecting it to what's in the screenplay. And I created a system for doing that relatively precisely. And then you also have a lot of paperwork that the film industry creates. You have makeup artist notes, you have prop master notes, you have craft service budgets, you have script supervisor notes. You have a lot of information about everything that is contained in the frame visually and everything that was, all the energy that was put into creating that frame. And then once you take that information, you also have audience reaction. With a streaming service like Netflix and so on, you can actually create tests to determine how successful these moments are. And then when you can teach a system what elements create the highest level of success, what elements are most convincing to audience members, you create a whole propositional calculus that can teach a, albeit complex system, what is successful. And then you can create a model, a historical model, that can accommodate predictive tasks. Give me a new screenplay. what are the most successful camera angles, focal lengths. Whenever a little boy in a red shirt is running by in the background out of focus, you know, if that's the heuristic that always ends up in a successful audience takeaway, then, you know, that's what the system learns. Who knows?
[00:38:09.205] Kent Bye: Interesting, yeah. So just to kind of summarize from my understanding is that given streaming services like Netflix they're able to determine when people are watching or stopping and so they have very fixed data in terms of usage patterns but yet to be able to correlate that to the semantic meaning of the script and also the visual content you could do some sort of natural language processing through the script to be able to find all these components to be able to then correlate those visual components that are described to the actual film through like some sort of convolutional neural network to be able to actually see what's happening in the screen to do this automatic correlation between the script and the video and then to combine that with the viewing usage patterns to then somehow drive future decisions about content is what I'm here.
[00:38:54.100] Catherine Rehwinkel: That's right. Every frame is a state unto itself, and it has many attributes. And you can cut those attributes and group those attributes however you want. And then those become inputs for a convolutional neural network. Wow. Interesting.
[00:39:10.113] Kent Bye: Yeah. Yeah. So. For people who want to dive more into the AI component, we can switch over to the voices of AI. I'll be sort of diving into the nitty gritty. But finally, just to kind of wrap things up here, wondering what you think the ultimate potential of virtual reality is and what it might be able to enable.
[00:39:27.873] Catherine Rehwinkel: I think that the potential of virtual reality is to give people a new understanding of reality as they create a mental model about it through stories that they learn about life from. And also as you collect data about how people observe a world, observe event flows, You can create a model to help an artificial intelligence create a model of how you move through the world and collect information and predict what might happen next, what's probably going to happen next in this story. So story becomes a template for mind. And with virtual reality, you are not constrained by traditional film storytelling. And film has had the benefit of being the most convincing because it involves optical physics. And optical physics contain a lot of information about things at a great distance from us or a near distance. And if you want to create a robust AI, you need to take in a lot of stories and a lot of different perspectives. And that's what VR can give us.
[00:40:45.168] Kent Bye: Well, thank you so much.
[00:40:46.709] Catherine Rehwinkel: Thank you very much, Ben.
[00:40:48.610] Kent Bye: So that was Catherine Rae Winkle. She's a filmmaker who's been coming up with some conceptual frameworks in order to help VR designers create linear VR stories. So there's a lot of kind of higher level abstracted thinking that's happening in this podcast. And I wanted to just kind of highlight some of the analogies and metaphors and takeaways that I took from it. So first of all, if you think about the difference between film and VR, you can think of a filmmaker as somebody who's driving a car. There's a very specific frame that's in front of you and they're really controlling what you're able to see. Essentially they have control of what you're looking at and it's kind of their perspective in a lot of ways. And so thinking about what is a metaphor or analogy to think about VR design, then what Catherine says is that it's more like you're within a submarine and you have control of the periscope and you're able to kind of look around the scene from your own perspective. But from a VR design perspective, you're kind of designing the ocean that that submarine is in. So it's a little bit different way of thinking about kind of constructing the environments. Now, I think the other big point is to emphasize how someone's individual perspective is flawed and incomplete, in that anything that you're looking at a story, you're really only going to get the partial perspective of what's happening in that story. And so this was first really made clear to me in Roche's perspective, where it's told from four different perspectives. And as you go through and watch the same story four different times, intercut from those four different perspectives, you start to see the different nuances and can start to empathize from other people's perspective. So I think reality is much like this. This is the point that Rose Trichet was trying to make in episode 286 when I had a chance to talk to her at Sundance, is that there's a vulnerability of the first-person perspective. And it's also a theme that has been echoed by Alex McDowell, who I interviewed back in episode 309, where he was talking about back before we had the written word and lots of printed books, that stories used to be transmitted through oral traditions of people sharing stories with each other. And a lot of their subjective modifications would be added into the story. And so in a lot of ways, the oral histories that were passed down were imbued with a lot of subjectivity and multiple perspectives in that way. And so once things started to be written down and published in books, I kind of started to move to this singular perspective mindset. And especially with films and novels, it's all pervasive, like singular perspective. And I think that VR presents this opportunity to move beyond that singular perspective. Taking that into account, I think that you can start to understand this idea of situated knowledges from Donna Haraway, which comes from feminist theory and essentially says a little bit of the same thing, which is essentially that from any singular perspective, it's going to be complicated and incomplete. And so just in reading a little bit about situated knowledges online, rather than kind of advocating from a singular omniscient God perspective, Haraway is proposing a split and contradictory observer. She's saying she's arguing for politics and epistemologies of location, positioning, and situating. So in other words, kind of where you're located in a scene and within the context of world is really impacting what you're seeing. So she's saying, I am arguing from the view from a body, always complex, contradictory, structuring, and structured body, versus the view from above, from nowhere, from simplicity. And only the God trick is forbidden. So essentially, I think what she's saying there is that knowledge is produced from like these many different facets of complex and embodied observers within specific locations of space and time. So it's not based on this omniscient positioning, but more on contradictions from embedded positions. So if you kind of take that academic language and break it down, you're basically saying that in order to get the truth of what's happening, you have to see things through many different eyes and many different worldviews, many different ontologies and perspectives. try to triangulate from all those different contradictions what the kind of universal truth might be, if that's even achievable. But I think that this is kind of the strength of storytelling within VR, is that you start to really provide a medium where situated knowledges can really be explored. And I think that's, in essence, what Catherine is trying to create, is this framework for people to start to do these 3D storyboards for people to start to experiment with what kind of stories where these situated knowledges will start to really change the outcome of what people are seeing. And I think Rose Troche with Perspective, both from last year and this year, there's two series that they did with Specular Theory and Morris May, starts to really explore some of these stories where situated knowledge has kind of changed the outcome of how you see the story. If you only see from one perspective, you're not going to get the full story. And so it's really calling for radical empathy to be able to deal with the contradictions and complexities of all of our biases and worldviews and preconceived notions And perhaps something like virtual reality won't really be able to overcome a lot of those preconceived notions that we have, but perhaps there may be different ways of really stepping into different perspectives within stories, within VR. And I think this is something that's very unique to the medium of virtual reality and something that Katherine Raewinkle is trying to really explore. So, those were the big takeaways that I had from this podcast. Lots of deep, conceptual, philosophical thinking there, but I'm glad that you stuck through the end, and I think it's a podcast that's worth listening to again. And again, if you want to listen to some other thinking about this, I'd recommend listening to Rose Troche in episode 286, as well as Alex McDowell in episode 309. So I just wanted to thank you for listening to the Voices of VR podcast. If you enjoy this series on storytelling this week, then please tell all of your storytelling friends and start to do a deep dive into narrative and VR. And I'll be continuing with a few more episodes this week. And please do consider becoming a contributor to my Patreon at patreon.com slash Voices of VR.
