The MIT Open Documentary Lab held it’s first official event back on Tuesday, March 20, 2012 a day-long summit called “The New Arts of Documentary.” Part of the MIT Open Doc Lab’s mission is to bring “storytellers, technologists, and scholars together to explore new documentary forms with a particular focus on collaborative, interactive, and immersive storytelling.”
William Uricchio is the Founder & Principal Investigator of the MIT Open Doc Lab, and he’s got a background in painting, philosophy, and a Ph.D. in film where he’s studied new communications mediums in the phase before they’ve crystallized into commonly accepted standards for the affordances, language, and uses for storytelling. So he’s particularly interested in tracking the evolution of immersive storytelling potential within virtual reality.
I had a chance to catch up with Uricchio at the IDFA DocLab 2019 where we talk about a number of the open questions that the MIT Open Doc Lab is currently investigating, a little bit about how virtual reality fits relative to other mediums from a Comparative Media Studies perspective, and some of the challenges he sees for virtual reality. There hasn’t been a consensus about what language to use from and which theoretical scopes work best when talking about virtual reality experiences as it’s pulling insights from film, video games, theater, immersive theater, literature, web design, magic, audio tours, performance art, and psychogeography.
We have a brief debate as to whether 360 video should be considered VR, and there are a number of other points of disagreement that I elaborate on in my takeaways at the end of the podcast interview related to whether or not the immersive technology stack is stable enough to build critical frameworks on top of.
Other things worth calling out here in the show notes is the Ph.D. paper by Deniz Tortum on Embodied Montage in Virtual Reality. Uricchio also mentioned the film semiotics scholar Christian Metz who has done a lot of pioneering work in showing how film can be seen as a language. There’s still a lot of work being done in trying to determine how much of the cinematic vocabulary is present or absent within VR, and overall Uricchio is also really excited about the many potentials of telling data stories with AR that are overlaid a specific geographic context.
Overall, the MIT Open DocLab is doing a lot of great work in this space, and there’s also quite a bit of overlap with the work they’re doing and the work I’m doing here with the Voices of VR podcast. They’ve been publishing quite a bit of articles and information on their Immerse News Medium hub covering what’s happening in the immersive storytelling space with XR and AI. There’s a great interactive white paper called Moments of Innovation that helps to contextualize the history and evolution of different communication modalities. They’re keeping a database of innovative interactive and immersive documentaries at Docubase. The Co-Creation Studio is a new initiative to look at the collaborative production of documentary content. Finally, there’s a page with all of the MIT Open Doc Lab’s research since it’s beginning in 2012.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So continuing on my series of looking at different narrative innovations coming out of the IDFA DocLab in Amsterdam, today's conversation is with William Uricchio. He's a professor at MIT, and he runs the OpenDocLab, or the Open Documentary Lab. So the OpenDocLab was something that was started back in 2012. There was the New Arts of Documentary Summit that happened on Tuesday, March 20th, 2012, which was where the lab formally got underway in that day-long summit. So OpenDocLab has been looking at this intersection of the form of documentary, so the creative treatment of actuality, And this emergence of all these new media technologies and how there's new ways of telling stories about what's happening in the world using all of the new affordances of all the new media technologies, which that's part of the challenge is to try to figure out what is the language that you're going to use to make sense of it. What kind of critical frameworks are you going to be pulling from? And so that's a lot of what he's looking at there at the OpenDocLab. This conversation was just to give a brief overview of the work that he's doing, some of the questions that he's asking. For his perspective, he's looking at it in the context of film and documentary theory, critical theory, literary theory, all sorts of other critical frameworks. And he's very familiar with the evolution of different communications mediums. And so he really likes to see new communications mediums and how they unfold until there's a certain point where it crystallizes and things just make sense as to the affordances of the medium and how to actually use it using the technology. And that he studied that previously in the evolution of new media. And so with virtual reality, he's diving in and doing a similar kind of treatment as we're in the midst of this unfolding process of it still forming and trying to get their head wrapped around it and to make sense of it. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with William happened on Sunday, November 24th, 2019 at the IDFA DocLab in Amsterdam, Netherlands. So with that, let's go ahead and dive right in.
[00:02:13.583] William Uricchio: My name is William Uricchio, a professor at MIT, where I run a thing called the Open Documentary Lab. And what the lab does is basically look at new technologies, emerging technologies, and how they fit or what their potentials are in the bigger remit of documentaries.
[00:02:31.440] Kent Bye: Great, so maybe you can give me a bit more context as to your background and your journey into what you're doing now.
[00:02:37.883] William Uricchio: Yeah, so my academic background is schizophrenic, painting, and philosophy. That's where it all began. The way I paid for a lot of my education was working in film. Started sweeping floors, started to assemble ends, became an editor, and actually started to make pretty good money as an editor. Did philosophy through university. But when it came time for graduate school, I thought, you know, I'll do philosophy, but I also want to do film. I want to know more about this medium I'm working in. So I went to New York University. I applied both for the theory program and the production program. And I quickly realized production was kind of doing more of what I was already doing commercially and being paid pretty well for. So I took the theory route. And I loved it. It was really fun. Did a lot in especially film history, especially the very first years. I was interested in the moment when the medium stabilized. And that turned out to be kind of the bigger pattern in my academic life. I specialize in beginnings. So beginnings of the telephone, beginnings of photography, first couple of years where a lot of different technological possibilities are there, but at a certain moment it sort of standardizes. One or two standards pop in and it becomes taken for granted. And I'm really interested in how from all the kind of possibilities taken for grantedness emerges. That's a magical moment for me. So I've done it with the book, with telegraphy, telephony, photography, and last but not least, you know, virtual reality and augmented reality. AI is a little murkier. That's a hard one to pin down. So with that in mind, I'm interested also in documentary. I wrote my dissertation on very early city representation, first 20 years. Actually a part of documentary history that's not been well explored. It's usually written off as naive somehow, as people not knowing what to do with the camera. Just they point it and they press the button. And to me, it's sort of the culmination of 19th century representational systems. So, with that kind of stuff in mind, what's happening now has really been inspiring to me. Like, I look at what's happening today through the same eyes that I looked at what was happening 130 or 140 years ago. A new technology is there, people are trying to make sense of it and make sense of the world around them, and they find really innovative ways to do it. But ways that are in sync with bigger cultural norms. So, if you go back to 1896 or 1897 with these early city films, that people call naive. In fact, what they do is that you could look at physics or you could look at sociology, the models of that day. These films are consistent. They're a slice of reality, a chunk of reality. No editing, no analysis, just an idea of a kind of whole. A Newtonian physics, if you will. Time and space are united. At that moment, historically, Einstein was doing his work on relativity. That would hit film later. That would hit film in the later 20s with people like Ruthman, where you get this analytic editing and kind of evocative or, yeah, kind of the evocation of the feel of a day. But that wasn't happening much earlier. So I'm interested in kind of locating these technical practices, these stylistic practices against the bigger cultural dynamics. Anyway, today all that boils down to meaning we take a really close look at at new technologies, at their implications for things like storytelling or not storytelling, something more like story finding, how they enable new kinds of people to speak, not just professional cameramen and not just big organizations, big companies like the BBC or NFB. Actually, all of us are carrying around, pretty much all of us, including in the third world, developing world, are carrying around high-def video cameras. All of us can record the world around us. All of us can share that. So what does that mean? And how should we think about the power of these new voices? How can we coalesce them? And how can we, especially in a moment of like rampant fake news, how can we do this in a way that has a kind of epistemological order to it?
[00:06:30.356] Kent Bye: Well, because you have an interest in philosophy, what branches were you looking at? Epistemology or aesthetics? Or what were you really focused on in terms of the philosophical perspectives?
[00:06:39.948] William Uricchio: So yeah, the answer to that question is not a good one. I was doing symbolic logic. And had I known then that computers would be where they are today, I would have probably stuck with it. I did it right through the end. I didn't write the dissertation because I already had a job teaching film at that point, and I had my PhD in film. But no, I was doing symbolic, and I loved it because it was It's a really profound ordering system. It's profoundly satisfying because it's a kind of world where everything works. I'm dyslexic, so I never can add numbers and usually get the same sum, but I'm really good with mathematical principles. So, symbolic really spoke to that. You don't have to do numbers. You just do the formulas. So, no, it was pretty schizophrenic. It was that, not... I mean, I did aesthetics. Obviously, I did some work in aesthetics, and that related to my film work. But the real stuff I found satisfying in philosophy was symbolic.
[00:07:29.770] Kent Bye: Well, my background in training is in electrical engineering. And then I was always also doing film at the same time, so I have this kind of dialectic between art and science as well. And I've worked in startups and open source, and then did early days of video blogging and podcasting back in 2005. So I started seriously doing podcasting for the last 10 years, doing lots of probably over 2,000 interviews with different people, specifically in VR, 1,400 interviews now. So being able to track the evolution of the medium is something I've also been interested in, but also trying to figure out the underlying structure of experiential design. And what I've been thinking about a lot lately is like language, philosophy of language, but also the cinematic storytelling aspects of language, but how VR is kind of pulling together aspects of gaming and theater and film and the internet and other aspects of user experience that we've had from website design. So all those things are kind of fusing together. So for you, like what are the things that you're trying to figure out? Are you also trying to figure out like what is the frameworks for experiential design or how do we make sense of how to tell stories with pulling in all these different aspects of agency and narrative and how those come together?
[00:08:46.842] William Uricchio: So I'm in the very same space, I would say, conceptually. And I guess as an academic, I tend to stand back and look at the force fields. And right now, I mean, you just did a great description of the force fields. On one side, I think our cultural default in terms of notions of narrative comes from our tradition in literature and film. And both of those have the common ground of being fixed platforms. You can't really change the page order of a book with ease. You can tear out some pages and move it around, but there's a sequence, and there's a sequence in film. And I think we've all pretty much grown up, and certainly our academic notions of narrative are predicated on very particular notion of narrative. And I'm wondering, I guess for me the question is, so what happens when you have the potential, some kind of interactivity as a platform, an interactive platform rather than a fixed platform? Does that change or alter the potentials for what constitutes narrative? Do we by default just back in our older notion of narrative into these new spaces, or are there other kind of narrative possibilities? Gamers have been really interesting in this regard because probably more than anyone else, they've been the ones to come up with the notion of... It's always been there, but they really hyper-articulated the notion of experiential narrative. Narrative is a process in our brain. Narrative is a way that we connect the dots, rather than narrative is something that a teller tells us and fixes in cemented order on the pages of a book or a strip of celluloid. So, in that regard, yeah, I've been looking a lot at this kind of tension, this force field between on one side the film and video people who want to tell a story and our cultural default. Most older people expect there to be a story versus the folks coming from interactive theater, immersive theater, from games, from magic. Magic is a really interesting space where you direct attention without really directing it. Where in that domain, VR becomes more of an environment in which you can plant narrative elements that people can discover at will. And the analogy that I like to use for this is something like a theme park, like a Disneyland or something, which is perfectly legible and coherent to the punters. No one has a hard time figuring out what's there. And yet, it's really interactive. There are thousands, millions of potential stories to be discovered there. Storylines are planted. If you really are desperate to figure it out, it's there. But you can kind of construct your own with those characters. Psychogeography that the situation has developed where you map a city based on your memory of experience. That's another really cool way to think about this. So I'm deeply fascinated. I'm not against story in the old sense, just to throw in another analogy. Come to a new city. You can jump on the tour bus. You have a linear trip through a city with a bunch of other people. You all hear the same story. And if it's a well-informed guide, it's brilliant. You understand the city. You understand the lay of the place. But wandering is a very different beast. Wandering is where you kind of invent your own story. You stop in stores if you want to, you have a coffee or you check things out. You're very attentive to street names and number sequences so you can find your way back. It's a radically different experience. Together they're brilliant. If you have an overview and then wander, it really is a terrific way to see. So I see that as kind of a parallel to the world of film and television, the linear fixed story form versus what interactive forms are capable of, which is letting you wander, discover, make sense of it. It's not an either or. I like having them both in my experiential world, but my work is focused on the interactive side.
[00:12:12.640] Kent Bye: Well, since you have a background in symbolic logic, are you familiar with Alfred North Whitehead's process philosophy? Whitehead with Bertrand Russell wrote the Principia Mathematica and then he went off into philosophy but started to look at a metaphysics that's more about process and unfolding processes rather than sort of substance metaphysics. And so he's really concerned about the relationships of how things are unfolding. I think because he tried to prove all of mathematics into this kind of symbolic logic and then Gödel came up and eventually kind of, with his incompleteness theorems, upended that. I think he was also looking at what was happening in quantum physics and seeing like, well, maybe the fundamental base of all of reality is not these physical tangible things, but also like these actual occurrences or these processes that are unfolding and thinking more in terms of relationships rather than physical stuff. And for me, I feel like a white head and his process philosophy is providing like a whole new philosophical grounding for what's happening within the experiential realm.
[00:13:07.206] William Uricchio: I think you're right. And I think in a funny way, it's like the pragmatic Anglo-Saxon alternative to what Heidegger is doing with this notion of being, right? Where he just, rather than cogito ergo sum, he's all about I am, therefore I think. And trying to put his hands on this kind of ephemeral state of being and becoming, but in that kind of murkier Teutonic tradition versus our sort of more applied Anglo-Saxon. So yeah, I think those are really relevant discourses. And alas, I don't think we have a language that's well-developed to capture that in this experiential domain of VR. So to just go back to the earlier point, what language set do you use? The language of film to articulate this stuff? Or the language of experience design? Or the language of... Do we try to come up with a new language from somewhere between Whitehead and Heidegger? Or do we take the experiential language of psychologists and go into the immersion and empathy stuff? There are a lot of grids that are slicing through this domain right now that provide analytic entryway, but they don't talk well to one another. And that's, I think, the challenge of VR. VR right now faces two fundamental challenges. It faces three. Let me say that differently. It faces three. One is the decline of investment right now. At least as far as the numbers I track, it's been not doing so well right now. VR winter might be too strong a term, but it's not looking like spring anymore. Second is the kind of quick takeover of the big corporations and therefore incompatibilities. I mean, there's kind of a battle for whose system is going to have dominance. So that's kind of screwing things up in terms of, I think, consumer adaption. They're waiting to see who the winner is. But third, and I think most fundamental, is the kind of conceptual dilemma of what's the language to talk about this through? What is this really? Is this an entertainment system? What is it exactly? And you can see that in the kinds of language, the kind of analytic grids that are set on this. And at least where I sit in the academy, I just listed them. You can kind of shift around those different perspectives, but they don't really add up. What typically happens in most earlier media forms is that at a certain point, like about now, you start to see something that's intrinsically about the medium start to emerge. Will that happen today? I don't know. We're seeing a kind of collapse in our political world just because of the fragmentation of our media channels. I'm not sure if we're in a new time right now where there's so many channels that there's consensus is hard to come by. If this were 30 years ago, I would have predicted like five years we would have a language and we'd all sort of simmer down and there would be a normative version of what this is. Now, I don't know if that's going to be the case.
[00:15:38.625] Kent Bye: Well, what Gödel was saying with his incompleteness is that if you have any consistent system, it's going to be incomplete. And if you have any complete system, it's going to be inconsistent. So I've adapted more of a pluralistic approach of saying, well, in order to make decisions, you have to then start to think about equivalence classes. And Stephen Ellis of NASA had told Mel Slater, who told me that any good theory of presence is going to have good equivalence classes between what are the trade-offs. So for me, what I see the trade-offs of experiential design are, in essence, the quality of the experience. So whether you're exerting your energy outward or receiving either mental and social presence, so making choices or taking action of expressing your agency. So much more like a video game in the agency perspective or like a book or reading the internet or communicating with your friends using the abstractions of language. And then you have the more receptive aspects of like your sense of embodied presence of your body and your identity and your avatar representation. in your sense of all of your sensory experiences, and that's your sense of your body, and then the emotional attenuation through the music and things like the film medium that is building the consonants and dissonance cycles to give you this experience of time, but also your sense of emotional presence. So the active presence, the mental and social presence, the embodied presence, and the emotional presence. That's the quality of the experience, like the degrees of presence that you have. But then there's the context. And the context is when I ask people, what is the ultimate potential of VR? They usually answer into one of the domains of human experience, whether that's entertainment or medicine or going on dates or dealing with death and grieving, religion, higher education, your career. hanging out with friends, dealing with people who have accessibility issues, the expression of your identity, different economic exchange of value, communication, early education, and then connecting to your home and family. So for me, those are all the different aspects of the context. And there's going to be specific experiences where in AR, you're able to take your existing grounding of your existing context and add new context on top of it, where in VR, you can be in any context and go into another context. So I think there's sort of a context difference between AR and VR. And then you have the character of the experience, which I feel like is these, what is the essence of your will, your beauty, truth, goodness, justice, the aspects of character development. There's usually a change over the course of a story, but there's gonna be both a character of the personalities of the people, the NPCs of the story, but also the character of the area that you're in, the deeper environment that you're in. So you can have character of that environment. And then finally, I'd say there's the story and how that's changing and unfolding over time. So when I start to go through experiences, I'm paying attention to the quality experience, the context, and the character, and the story that's unfolding, and then be in dialogue with the creator, and then talk about their process, and then sort of analyze their design process for how they created this. But that it's very similar to ethics, I'd say, in that You know, ethics, there's no perfect answer. There's all these trade-offs, but there's no perfect trade-off. And so in the philosophy of technology, they talk a lot about how is technology an art or a science? Can you use the philosophy of science with technology? It doesn't seem like you can because it's more of a creative act. It's more about design theory and engineering. So you have trade-offs, and so you have all these imperfect trade-offs that are creating this experience. And so it feels like taking these sort of more design approach and for me I've been trying to find like what is the way that I talk about these experience and for me I tend to be a little bit more of that phenomenological approach of looking at the quality context character in the story.
[00:19:11.135] William Uricchio: So I guess for me one of the dilemmas with VR is the nature of the quality because I think it correlates to the amount of experience you have with it and I can be bowled over by something and about the third time I experience it, it's like I'm looking at the cracks, I'm looking at where the scotch tape is and I'm seeing more what doesn't work about it than what did the first time I saw it. And with the kind of non-stop evolution of the headsets, with the shifting pixel rate, with all that stuff, because it hasn't hard-baked around a norm, I think the qualitative for me is always on shifting ground. And I suspect it's that way with the public as well, and the problem with its slow uptake. So let's just start with a basic linguistic problem here. If you experience 360 video or an HTC Vive, those are radically different experiences, yet given the same taxonomic category. Oh, this is VR, some would say. If you looked at Oculus two years ago and look at it now, you're looking at, again, two different kinds of experience. I mean, a lot of similarities, but qualitatively there are distinctions. This makes the whole notion of qualitative assessment tricky because given the claim of virtual reality, the sad name it has, I mean, so, okay, there is a claim there. It might appear to be sort of real at first, and then the more exposure you have, the less it works. Anyone who works in the business knows that, right? You're seeing what's constructed about it. So relatively few have been the projects that exploit those glitches. That's where the action might actually be, rather than trying to replicate a world that we can immerse ourselves in and let all the other stuff happen that we just talked about, those kind of applications. That's the problem, because it is not consolidated around a norm. Cinema is not a great medium, but at least we accept it. Like, that's what it is. It's 35mm. Okay, IMAX 70, whatever, that's pushing it a bit. But by and large, we accept those constraints and then work within it. With VR, there's always the promise of something better around the corner. And that's destabilizing in terms of any kind of qualitative assessment. At least when I look at it, it's always, what's wrong and what's next? As opposed to, let me take this as a given and then move on to next order questioning.
[00:21:18.873] Kent Bye: So I'm focusing a lot lately on Sundance, Tribeca, South by Southwest, and Venice, seeing all the experiences there and talking to as many of the creators as I can. And I feel like what's happening at events like this that we're here at the doc lab is that there's a bunch of experiences, a small group of people get to see those, and then they'll be able to talk about their experience with each other. So I feel like there's this cultivation of figuring out how to even talk about your experience. And so I think of a metaphor of like wine tasting. And if you look at the neuroscience of wine tasting, you can see how if you don't drink wine, you can just drink wine and you can't really actually talk about it. But if you drink wine enough, then the thing about wine is that there's many flavors at the same time. So you have to cultivate your palate of being able to talk about the different nuances of that flavor. And that once you get to the point of cultivating that palate, then you have language that describes that. Then when you drink wine, you actually have arguably a richer experience of that wine. And I feel like the same thing is happening with immersive technologies, is that we are learning how to notice and pay attention and to speak about the nuances of our experience. and that we're still at the very beginning, but from what I've seen in the community, that we go through an experience and then you start to evaluate different aspects of, okay, what was my experience? What was the artist intending? What's the gap there? Did I have the experience that they were trying to create with them trying to modulate my experience and my consciousness through this mediated experience? But there's the language part that I think is interesting because it's like being able to pay attention to what's happening inside of us and then be able to talk about it to other people and then to be able to share our individual experience to see what us as an individual experience versus what the collective are experiencing. And I feel like that is a bit of knowing, like, well, maybe you just had a bad morning, and maybe it was more you than the experience, and maybe everybody else is loving this. And so for me, I've been kind of going through that calibration process, but also just being in conversation with people and starting to develop the language. So I don't necessarily think it's something inherent to the technology. I think it's more about the cultural normative practices of people being able to experience it and talk about it and be able to develop the critical frameworks to be able to understand it, but also As we go through life, we have to pay attention to different aspects of experience and know how to talk about it. So I feel like being able to control the experience, we're now able to then have a standard that we can have people go into an experience and then have that direct phenomenological experience, but then be able to talk to other people where, in essence, we have the same experience, where that had never been really possible before.
[00:23:47.261] William Uricchio: So basically, I agree. I think that's the process, right? I think people convening, talking, it starts with, you know, small cohorts and spreads outwards. I don't see a lot of good criticism out there yet. That is an important step in terms of making this a broader discourse. But I still think there is a technological problem with VR, which is its instability. It keeps changing. The rules of the game keeps changing, like what the technology is. Is 360 VR or not? To me, that's, I think it's not. I mean, I'm happy to argue that, but for a lot of people it is. How do I develop a coherent critical discourse for stuff that's 360, it's a fixed world, minimal interactivity, except what I can do with my head, versus something, a real-time capture where I can move around, I can alter the world. Those are fundamentally different, experientially fundamentally different. To still lump them under the same category, to me, is kind of problematic. And if I would go back to the very heyday of photography and look at what Daguerre and Fox Talbot are doing, those are radically different notions of what photography is as a medium. I mean, they're both the same in some ways, you're capturing the world, but one is inherently replicable, the other is not. One is built to sort of hold the pristineness of the moment, the other is made to replicate it. I mean, they have very different, they point in very different directions. And it takes a little while for that discourse to settle, like, oh, here's what the medium is. That's the moment I'm still waiting for here. And to me, the technological, what I would point to is what I would call instability is limiting, I think, our advance as a field. And you just see it here in terms of what's installed and how it's installed. It's, you know, okay, we sit here at Info where everything's under a kind of broad umbrella of immersive, and that's fine. But if the word VR starts to be invoked, then it gets a little bit slippery. And we know, we know that there are two big evolutionary steps awaiting us still, like more on social. The social is pretty under, there's some interesting experiments, but it's not really with us yet. And we know that the eye trackers are coming, foveated rendering is coming, which will really, should really change both the resolution power, but with it responsiveness and also data tracking, all that dark side that'll come with it. But it should enable, I think, a quantum leap experientially. But it's in the labs, but it's not here in our world. So I think it's still nascent in that sense, and that's a problem that's inhibiting us from developing a kind of coherent critical discourse.
[00:26:03.594] Kent Bye: Yeah, at most film festivals, they do separate out if they do have awards, they will have an interactive award and then like a narrative or 360 video award because I do think that even the curators of these festivals see that there is a big difference between the two and they shouldn't be judged on the same merits. However, I would say that 360 video is still VR and the argument that I would give is that, you know, Mel Slater, he talks about different elements of presence being the place illusion and the plausibility illusion. So the place illusion is that you're in another place and that the plausibility illusion is that it's real and you're able to kind of interrogate and interact with the environment. You have less plausibility illusion when you don't have the agency to be able to interact with the world. But I do think that you still are able to get a pretty good sense of place illusion. So you're using technology that is trying to address different elements of your sensory experience. And so having just a spatial experience of an environment still feels significantly different than a normal 2D frame. Like if you're not going to have 360 video in the ontological class of VR, I think it's different enough from just a video. because there's more similarities for how you do experiential design and the production and the storytelling within a 360 video. There's more analogs for the storytelling affordances within a 360 video. The big difference is whether or not you have the depth and you have 360 video volumetric capture as well. So then once you have like a volumetric capture, something that's in the middle, then that's somewhere in between, now you have the depth and maybe then you're able to then sprinkle on different elements of agency. So I don't know, I feel like even in film, you have the narrative and the documentary. I feel like the 360 video is more analogous to like the documentary genre of like at Sundance, there's the narrative competition and documentary. I feel like 360 video is more like the documentary of VR and that eventually we'll get to volumetric but just because you can't have agency doesn't mean it's not VR.
[00:27:59.691] William Uricchio: OK, I mean, but I think that's where we are. We're at a point where we have our views and we can we can each have our kind of definitional constructs. But that's the problem. They don't necessarily align. And we can agree to disagree, but it doesn't solve the problem. So I don't know. I think it's going to shake down in the long haul. I'd be surprised in 10 years if we would be having this conversation and if there hasn't been some kind of more broad rearticulation of what these forms are. And I can only point to those other moments in things, in media like photography, where something similar is there. It's kind of related, but kind of different. And at a certain moment, that just gets articulated in terms of what's the norm. And it's not all of those. It's only one of those. So we'll see. We'll have to see where this takes us.
[00:28:40.417] Kent Bye: I think talking to people in the video game world, they would agree with you. They often, people will say that 360 video is not VR. So it's an ongoing debate. But one thing that I've been interested in is looking at film and learning from film and looking at the storytelling affordances of editing or lighting and framing and different ways of assessing like the different things you can do to modulate narrative tension. And then how to talk about the qualitative aspects for how that either builds and releases narrative tension or gives you a different qualitative aspect. And I don't know if that's been formalized in a way to have like a very strict taxonomy of all the different things that you can do within film and to see how many of those can then be analogs into VR and then how there may be new fundamental primitives within the storytelling language. And I'm not sure if that's happened within film to be able to kind of isolate all the primitives that you can start to manipulate.
[00:29:33.553] William Uricchio: So I would point to someone like Christian Metz. Okay, it's under the guise of semiotics, but he's really in that spirit of like Jeannette, who really parses every possible way in which a text can be constructed and framed. And Metz did the same thing with, I think, a pretty comprehensive analysis of like all the things that make it work. To me, the driver in film, the thing that most film historians, why the documentaries I love, those early ones, those early city films, are detested by most film historians is because they don't have cuts, for the most part. Or they have one or two cuts, but basically pretty few. They don't use editing analytically. And when analytic editing enters the scene, it's like, ah, the birth of cinema. This is our tool. We've gone from living photography into something called cinema. And it happens through montage or through editing. And that's a place for VR. I think the more we buy into the reality conceit, the trickier cutting can be. And there are lots of ways to play with it. And I would point to Dennis Tordham, a gifted maker, but also a guy who thought a lot about it. He wrote a thesis in our program at MIT. And he uses the notion of embodied montage, where actually it's the movements of the user that enable that kind of reconstruction of the world, even its temporalities. But film will always have the advantage. I mean, again, because it's part and parcel of how we take the medium to be. that it can really radically manipulate time and use that for incredible emotional advantage in ways that VR risks nausea, confusion, epistemological vertigo, I don't know what, but it's a lot tougher in VR. And VR has other affordances. I think that key part of the cinematic vocabulary, and for some people, defining part of the cinematic vocabulary is just absent or pretty minimal in VR.
[00:31:22.194] Kent Bye: So for you, what are some of the either biggest open questions you're trying to answer or open problems you're trying to solve?
[00:31:28.941] William Uricchio: So I guess what I'm trying to understand right now are the tectonics if you will probably something you really have a great sense of much better than I Tectonics of the field in terms of like where it's heading. Are we heading more for a kind of game or immersive theater? Notion of story or are we going to persist with trying to kind of? Thinking of new tricks to direct the viewers attention here because you need this point for the plot to develop I find that a misuse of the medium frankly not to say I haven't seen good stuff that does it but So I'm very curious about like how I'm watching to see what will happen I think because of where the money is and where the weight of the industry is and where the Expectations of the demographic that's buying this stuff is there's a lot of pressure to tell compelling stories and to try to evolve a storytelling vocabulary versus letting people explore but you know what the gamers are in ascendance and so i'm watching that i don't know verdicts out for me on which way it's going to go probably both ways doesn't have to be one or the other but that's a huge question for me second i'm really waiting to see what eye trackers do and foveated rendering That'll take a huge load off the computers, it should make the rendering much crisper, which is great because right now it's pretty pixely. And third, I think the whole idea of having, like Kareem Ben-Khalifa's The Enemy, where you have eye contact with the characters, that makes a hell of a difference. Versus Karnat Yorina, a project that probably costs just as much or more, but where you feel like you're in Madame Tussauds, a kind of moving Madame Tussauds. or you're Dante with Virgil wandering around hell and no one notices you. I think that's a fundamental missing link and I'm waiting for that to enter the real world domain as opposed to the labs that we inhabit.
[00:33:04.625] Kent Bye: Have you heard of or seen the Under Presents yet that just came out last week that uses immersive theater actors? It's something I'd recommend checking out because it is starting to blend both storytelling, immersive theater, and games in a way that's sort of a trifecta. And I feel like there's a lot of really good reactions. But I saw it at Sundance and had an early preview and it just came out within the last week or so. definitely check that out. But just to wrap things up here, I'm just curious about what you think the ultimate potential of all these immersive technologies are, and immersive storytelling, and what they might be able to enable.
[00:33:37.255] William Uricchio: So the one I'm watching with greatest interest is AR, and I'm watching it because I think the future, for example, of newspapers or Television news archive all of them have archives or morgues packed with stuff that sits there and molders doesn't get used unless There's a reason to yank it up for a story I'd love to see that stuff geolocated where we can layer our world with a durational dimension Where on a building we can see stuff that happened there who lived their stories. I think we live in a world that We spend a lot of our time sort of passing through a world from one point to another point, ignoring what's in between, and I'd love to see us load that space in between with information that we can access and better understand the world we inhabit. So for Stories of Gentrification, there are a lot of cool projects this year on gentrification, but how cool would it be to just see price histories of buildings when you walk down the street? Where you inhabit the world, you can come to a coherent analysis based on what's around you in the world. What I love also with AR is that it's in the world, it enables conversation, it's technologically relatively, versions of it are pretty low-hanging fruit for most folks. VR is getting, in some ways, ever more esoteric. What I hope for in VR is for people to kind of forget the reality as we know it and start to explore non-Cartesian coordinates. It's a little ironic or sad or whatever to me when I look at Unity and Unreal engines and the camera position in there that articulates, that reinforces a notion of perspective and a perceptual regime that we've had from the 15th century. Hey, we're entering a new space. This should be the medium that enables us to explore that, not simply trapping us back into the old. Will that happen? Will some new engine designer allow us to break the camera or add new dimensions? We'll have to see.
[00:35:19.227] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:35:22.848] William Uricchio: No, just a big thank you. It's been a great community to hang out with, learn lots from it. So no, thanks.
[00:35:28.109] Kent Bye: Awesome. Well, thank you so much for all the work that you're doing in the field. And I look forward to both the research projects that you're doing, I think, immersed on Medium for a lot of the different results in both VR and AI that you're working on. So yeah, just thanks for sitting down to talk to me.
[00:35:42.033] William Uricchio: And thanks for your work. Geez, this is the compendium of thoughts on VR. So thank you.
[00:35:46.659] Kent Bye: So that was William Uricchio. He's a professor at MIT and he's created the Open Documentary Lab. He's looking at the evolution of this new media, this cross-section of interactive media and new forms of storytelling. So I have a number of different takeaways about this interview is that first of all, Well, I really appreciate the historical context and the theoretical frameworks that William is starting to pull in from, you know, media theory and documentary theory, literary criticism, lots of different aspects of comparative media studies and the work that he's doing. And because he's coming from that center of gravity, there's a whole other area of gaming, interactive gaming, theater. other different types of critical frameworks that are being looked at to be able to understand and have the language. So there's a number of things that he's saying is that like right now, there's still a fusion of trying to figure out what type of ways to kind of fuse all these things together. And there's things that I disagree with. Just as an example, he says, well, he thinks it's too early to start to try to come up with any sort of robust critical framework, because the technology is still changing so quickly that we can't have any strong ground to stand on. And I can see how that is definitely true in terms of just over the last five years, we've gone from the DK one to like the Oculus Quest. And so that's a huge amount of change and shift. And, you know, what's going to happen with eye tracking, biometric sensors, brain control interfaces, you know, we're still at the very, very early days of this. And I think there is quite a lot of shifts and changes that are going to come. However, I guess I disagree with that point because the way that I'm trying to look at this media is through this more phenomenological lens of looking at the quality, context, character, and story, that those things seem to be pretty set. I mean, your own experience of things are going to be a certain amount of the quality of presence that you're going to have. The context of being able to flip into a completely different context, I feel, is able to be explored in completely novel ways with both 360 video as well as in other virtual reality experiences. the character of an experience. I see a lot of experimentation at if a doc lab that is focusing on more character aspects rather than the story. So it's looking at your character aspects of your boundaries, your limitations, and putting you into these different contexts to be able to start to play with those boundaries a little bit. Things that start to be more about the character of the experience rather than the story of the experience. And then the overall story is how it unfolds and changes over time. And so whether it's you're watching somebody else's story unfold or whether or not you are the protagonist and you are trying to go through a situation where your character is catalyzed into some sort of growth or change, then you become the participant within this unfolding story. So for me, I feel like that there's enough of a critical framework to be able to look at that as a foundation and to be able to pull in all of these different aspects of looking at through the theater scope, looking at through the gaming scope, looking at through media scope, looking through the internet and artificial intelligence and data and what happens on the web and user experience. And so I feel like there's certain ways that at least the way that I think about it, at least that it can start to make sense of it. And I feel like it's at a solid enough ground that we can start to explore the phenomenological aspects of what's it mean to have a brain control interface and start to have Certain aspects of being able to read your mind and read your thoughts and so what's it mean to be able to? Have an experience that's able to read your thoughts That's a certain amount of mental and social presence that you know You're able to input and how is that going to be? Integrated into different aspects of agency and be able to control different aspects in the world Now, even though we don't have brain control interfaces that are out in the market right now, to be able to actually do that, it can start to look at it theoretically like, well, okay, once it does come, then we can start to imagine what kind of narrative conceits we can start to do once we have that and plug it in. And because I've been covering this field so closely for the last five and a half years, I feel like I've got a good enough sense of where the technological roadmap is going to be able to start to map out what is the embodied experience of those different types of media and what is the phenomenological experience of that for how you get this sense of embodied presence. And I think you can also look to see what's happening with a lot of the research from people like Mel Slater looking at things like plausibility illusion and a place illusion and start to break those down into the fundamental component parts. So the plausibility illusion could be different aspects of your mental and social presence and your agency and being able to make mental models of the world that make sense, and then to start to interrogate those worlds. And if those worlds start to break down in different ways, then the plausibility illusion is broken, because, you know, it doesn't match your expectations, or there's something that, you know, kind of breaks that sense of plausibility illusion. And the place illusion is like some combination of your sense of embodied presence with the larger context of the world that you're in. So you actually feel like you're actually in that context. and so i guess given that as a framework i would continue to argue that 360 video is indeed a part of the virtual reality medium the previous interview that i had done right before this one was with kai messenberg and he's talking about how there's the definition of a medium and he's using this definition from Bernard Miesch, which is basically two components. It's being able to distribute content and then the actual content that you're distributing. And so you have the actual content and then modes of distributing it. So, and if that's the definition of a medium, then I feel like the 360 video is indeed a part of the virtual reality medium. You can look at an example like notes on blindness. there's a version of Notes on Blindness that's a 360 video and then there's a version of Notes on Blindness that is a spatialized volumetric six degrees of freedom and it's distributed through a computer. So in one version you have all of the affordances of parallax, you have being able to move your body around in that space and move around in a way that maybe gives you a deeper sense of embodied presence and convincing you that these abstractions are actually there. Now, when you start to reduce that down to a 360 video, you lose a lot of that parallax effects. You don't have the six degrees of freedom. You have less of an ability of feeling like that you're still embedded within that scene of that context. But I would say that the character of that story, the context of that story, as well as the story itself are still pretty consistent between the 360 video version and the version where you actually have the full volumetric. And so then it becomes a matter of like, well, why would you define one, just the 360 video as to not VR and the other version of Notes on Blindness as VR, if the only difference is sort of like the six degrees of freedom versus three degrees of freedom, being able to have motion parallax as you're moving around. You know, is that just a matter of using photogrammetry or using stereoscopic effects? Agency is not really necessarily an issue on either one of those. And so you're not actually interacting in any way. But for me, I feel like as a medium, the 360 video is just as valid as anything that's like spatialized into whether it's using volumetric capture or it's using a full being able to walk around in a scene. It gives you a different sense of embodied presence, but I would still argue for Looking at these as both virtual reality, especially because you're watching it on a virtual reality headset, and it's a markedly different experience than if you watch it on, say, a YouTube video that has a spatialized way to kind of move around. That's a lot different than actually being embedded within that scene. Once you're embedded in that scene, it gives you that deep sense of context. And as you move around, it's trying to match all of your sensory motor contingencies. and then I point to different experiences like traveling while black, vr time, the waiting room, common ground and black bag, which was at Venice and which had a whole bunch of different narrative innovations when it comes to like the basic fundamental storytelling grammar of virtual reality. I still think that There's lots of different innovations that can happen even in the 360 video you start to have different ways of spatialized storytelling So I feel like there's more things similar to where spatialized storytelling is going to go in the future That the 360 medium is just as valid to be able to explore some of those things and I would point to experiences like black bag Which was showing at Venice this year so Sorry to go down that big rabbit hole, but there's something about like really defending 360 video It's like a part of the virtual reality medium, you know, there's been a lot of people within the virtual reality community you know, there was a big sale that happened and the headset that got the most sales with the oculus go and then there was like in Hamilton from Upload vr wrote a whole article and editorial like saying how this is terrible for vr Road to vr is ben lang who's also saying, you know, this is terrible It's like really polluting what real true vr actually is and so You kind of have this inherent bias of people who are coming more from the video game world or saying that spatialized six degrees of freedom and interactivity is like the fundamental parts of what it means and defines to be a part of virtual reality and And just to take a step back, William is saying that like, this is a part of where VR is at right now is that there's disagreements with, you know, the fundamental definitions about what VR even is and what constitutes a legitimate expression of a virtual reality medium. And I think it's a useful debate because it starts to get into the nuances of trying to actually be clear about what the definition is and if it's a matter of motion parallax or being able to have agency or move around or, you know, what is it about the spatialized experience that makes something VR and not VR? Is the oculus go not vr and then everything that's in the oculus quest somehow vr If it's something that you're using what's essentially the same amount of technology of putting something on your face That's occluding the world outside of you and as you move around it's trying to match all your sensory motor contingencies that for me Is if you look to something like the iEEE vr you can start to look at you know How they start to define different elements of like what actually makes it a virtual reality Medium. And I think being able to actually tie those things down, you know, I think William's point is that, well, that's going to continue to change and evolve. And things are so unsettled that we really can't make any firm ground. And I guess I just, you know, saying, okay, I think there's enough that's out there. I think it's pretty subtle based upon what I've been seeing. Sure, there's going to still continue to be innovations that come, but they're pretty well defined in terms of what those technological roadmaps are. And there's going to be the expression of what the phenomenal or qualitative aspect of those technologies, once they get out there, once you're able to actually integrate that in. But I've seen enough of the early prototypes that I can start to see, okay, there's ways that you can start to come up with some conceptual frameworks around that. So anyway, I wanted to just say, you know, a couple other things about this conversation is that we just really appreciated the pointing out to different theorists, you know, with with William and his background in philosophy, pointing to people like Christian Metz, who really was trying to, you know, look at the semiotics of film and look at film as a language. And, you know, that led me to the Wikipedia article on film semiotics to start to see like all of the other work that's already been done, looking and trying to define and come up with some different conceptual frameworks around the cinematic language of film. Because, you know, I think there's a lot of things to be learned into virtual reality that are just going to be ported over from film. And so there's just been a lot of, you know, over 100 years of different analysts and critics starting to look at how to define and think about that. And the thing I love about the IDFA doc lab is that it brings together all these people that are coming from that academic background that have studied these different aspects of media theory, comparative literature, and critical theory, literary theory and film theory, all these different ways of trying to come up with these variety of different conceptual frameworks. And I think the exciting thing right now is that it's an opportunity to kind of disrupt all those existing frameworks or to come up with a meta framework that helps make sense of a lot of these other conceptual frameworks that are out there. And I think that's a big thing that drives me. It's trying to kind of piece this puzzle together based upon what I'm seeing and to try to give some experiential design frameworks to creators out there to be able to help navigate this whole world. So Christian Metz and the Film Scene Mediatic sort of, it sent me on this whole rabbit hole of looking at Charles Deleuze, who's also a philosopher who did quite a bit of work of trying to mash up the different realms of philosophy with film. And Deleuze also went into Bergson and Bergson has this whole ideas about time and duration that, you know, was a big influence on him and also into the evolution of the more phenomenological traditions of the continental traditions of philosophy. So I feel like that this is an area where trying to connect the dots between all the sort of deeper philosophical grounding and the theoretical frameworks that it's just an opportunity to start to fuse all these different things together. And that's Something that I just had a great time talking to William about these different things and I think there's legitimate Debates in terms of what is the linguistic grounding there certainly isn't any consensus at all And so I'm just glad to see that the MIT open doc lab is starting to look at this seriously I mean they've been around since 2012 way before I was doing a deep dive into this since I started in 2014 and So they've been looking at it, but they're starting to try to pull together both a number of different writers. They have a whole page on Medium called Immersed. If you go to the MIT OpenDocLab, there's all sorts of really great stuff that they have that they're working on. So definitely go there and check out to see all the other projects that they're working on there at the MIT OpenDocLab. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast, and if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from listeners like yourself in order to continue to bring you this coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
