The Columbia University School of the Arts’ Digital Storytelling Lab has been conducting interactive storytelling experiments to create guidelines, rules, and structures for facilitating emergent and interactive stories through an open-ended, collaborative process. Interactive storytelling requires surrendering of complete control over authorship of the narrative, and it’s open question for how to best structure an experience that allows for full immersion, creative expression, and a balance of diverse participation.
I had a chance to talk with ColumbiaDSL co-founder and director Lance Weiler about their Sherlock Holmes & the Internet of Things project, the major principles that guide their experiential design process, how they’re using collaborative storytelling to examine and shape solutions to larger policy and ethical issues in a playful way.
LISTEN TO THE VOICES OF VR PODCAST
Columbia DSL’s immersive and interactive experience called Frankenstein AI will be showing at Sundance New Frontier later in January.
This is a listener supported podcast, considering making a donation to the Voices of VR Podcast Patreon
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So VRChat continues to cross the chasm of bringing social VR experiences into the mainstream where there's like over 9,000 concurrent users that are using VRChat at any given time. And in VRChat, they are doing all sorts of different role-playing experiences. You're able to embody these different characters and start to play out different improv interactions with people. And then you're also able to create your own worlds. And so you have this combination of world-building as well as embodying these different characters. the question becomes, well, where do you go from here? Where is this all going? And I think that creating the structures for emergent storytelling is one of the biggest open questions for what is some of the unique affordances of virtual reality as a medium. It's kind of like you're combining the best aspects of visual storytelling from film, but also allowing the user and participant to express their agency and to participate. The more that someone is able to participate, the more agency expression they have, the more engaged and the more immersed they are into the medium. And so the future of interactive storytelling is going to be moving away from the authored stories and more into the generative stories. And so in today's episode, I talked to Lance Weiler, who thinks quite a bit about what are the structures that you form in order to have these emergent storytelling properties. He's the co-founder and director of the Columbia University School of the Arts Digital Storytelling Lab. And we talk about some of the collaborative storytelling experiments and frameworks that they've generated in order to have these emergent stories. So that's what we'll be covering on today's episode of Voices of VR podcast. So this interview with Lance happened at Columbia University in New York City on Monday, September 11th, 2017. So, with that, let's go ahead and dive right in.
[00:02:07.313] Lance Weiler: My name is Lance Weiler. I am a founding member and director of the Columbia University School of the Arts Digital Storytelling Lab, and we explore new forms and functions of storytelling. I'm a storyteller by trade. I've been a writer-director for over 20 years, working in film, television, and games. I've been here at Columbia now for about six years.
[00:02:29.331] Kent Bye: Great. And I know that you've been doing these kind of collaborative storytelling type of experiments. Maybe you could talk a bit about some of these structures that you've been able to create and what that actually looks like.
[00:02:40.512] Lance Weiler: Sure. I think maybe about a decade or so ago, I got very interested in this idea of those formerly known as the audience. You know, feeling like with the rapid commoditization of technology, more and more people were obviously starting to create. And I thought it would be very interesting to challenge that top-down notion of what a story was. So in the work that I've been doing, a lot of the focus has been in this one side of the work is this idea of a kind of emergent collaboration. What does that look like? How do you build systems that allow people to be able to co-create? What are the conditions that are necessary to do so? and how can you take what can be a very chaotic process and use it as a way to try to rapidly accelerate the development of something. So one example, we do three types of things at the Digital Storytelling Lab. One is we build programs, the other is we design prototypes and run prototypes, and the last is that we build labs, external labs and things along those lines. One of our first prototypes was a project entitled Sherlock Holmes and the Internet of Things. The idea of kind of taking the literature of Arthur Conan Doyle which he was inspired by Charles Dickens to serialize his work and when he was serializing his work he was writing about new emergent technology at the time, writing about ballistics and fingerprinting and bloodstains and things along those lines. But when the work was serialized, local law enforcement was reading it and saying, wow, these are really good ideas. We should actually apply them. And that idea of where, all of a sudden, a fictional world could influence forensic science was really fascinating, at least the acceleration of it or the uptake on it. And so with that project, we grew it to about 2,600 collaborators in 60 countries. And since it's over its two and a half year lifespan, that's been about 160 self-organized events all over the world. And people kind of took it and started forking it in all kinds of really exciting ways, kind of like the way you fork software. So we built like a framework, you know, almost like a creative system that allowed them to run with it. in a variety of different ways, and there's four kind of guiding principles that we use at the lab. One is this idea of a trace. We found that the more that somebody can see little pieces of themselves in a story, the more likely it is to not only spread, but to raise the engagement level of the participants. The second is this idea of granting agency because a lot of our work is about individual and group collaboration. We realized that ideal group size for this particular project and for a number of the others that we work on is between four to six participants. If it was too small, dominant personalities would take over and other people would just start to follow and kind of disengage. If it was too large, it would slip into almost like a consensus kind of vortex where no decisions could be made. And so we realized if we broke it out and we allow people to kind of go off and do things as an individual and then come back into a group with whatever they had done and go back and forth in that manner, It greatly increased the agency that they felt that they had over what was being designed or being produced. The third is the notion of a thematic frame. Having Sherlock Holmes in The Internet of Things as a thematic frame was a great example because immediately out of the gate, whether you've read an Arthur Conan Doyle story or You've just stumbled upon like any of the episodic or serialized shows that literally lift those pairings of those two detectives. You've encountered the tropes or you know the tropes of a mystery. So if you saw a taped out body, you would know there was some sort of a crime. So immediately people had a collaborative language that they could use and a shared language. The last is this fascinating idea of serendipity management. This notion that a lot of digital work is overwrought with explanation and concern that somebody won't use it properly or experience it properly. and it doesn't leave enough room for the imagination or to ignite the imagination of the participant. And so we really work to build kind of almost blank space within the experiences, places where people can just randomly bump into each other, so creative sparks will flow from that. And so we use those as kind of design principles and we're constantly expanding upon that. But in terms of this emergent collaborative work, Those four have been very powerful for us.
[00:07:22.584] Kent Bye: Yeah, and I had a chance to go through this world-building experience a couple months ago, and what was so striking to me was that it seemed like it was blending a lot of influences from different trajectories, whether it was like story games and being able to kind of break down the process of telling a story collaboratively by, you know, having people make small decisions that kind of collectively add up to something. But there's also a certain amount of world building that's happening here, which is, I guess, before the story's actually being told, you're kind of just brainstorming and generating a collaborative process so that people, again, can collaborate with each other in small ways and then take what is a constraint that's provided to them and then be able to connect the dots from there. but also this idea of immersive theater of actually being a participant within this experience but they're not just passive recipients of the story but they're actually have the global agency in order to actually help direct the trajectory of the story and I think that that level of the trace I think is really powerful to start to see how something that you may have had a short idea see that get borne out into a full story which reminds me of improv when Actors kind of take prompts from the audience and then they're able to integrate that within the performance And so just curious to hear from you other influences or just some reflections on that
[00:08:36.730] Lance Weiler: Well I think definitely improv is an influence. In some ways comedic improv is fascinating or just improv in general where they're taking audience suggestions and weaving it in. I think a wonderful example of those design principles I mentioned was I had seen Ben Folds do a presentation at the Kennedy Center and he came out on stage and he had like something like 10 minutes to compose a piece and so everybody in the audience started shouting out you know like what chord is it gonna be in, you know, and they're like, A minor, you know, and everybody's going crazy, and then they select it, and then people are shouting out, like, phrases that are meant to be used within it, and then somebody hands him the program for the night, and he settles on this wacky phrase about something that seems impossible, right? And then he kind of sits down, and he starts to play something incredibly recognizable, you know so everybody laughs you know so he loosens it up and then at that point he starts going out to the different parts of the orchestra and they start feeding off of him and then you know it becomes this electric thing where when he finishes and he completes it you know a little bit over 10 minutes the place goes nuts you know they go crazy they feel like they've been there right there with him you know had this high stakes you know are we going to do it in this time allotment and what was really interesting was they used that as a way to announce that he was going to be the artistic director for the year and they could have come out and just said it but the fact that they did it it got everybody in the whole place was collaborating with him and so I think that that notion of what does it take to build a space that is conducive to that, and I think a lot of what we work to do first is we do a lot of things analog, a lot of things with paper, we rapidly prototype. And the goal is to really hone the human experience. Once we hone that or get a better sense of it, then we start to bring emergent technology into it and we start to explore. So, for instance, Sherlock Holmes and the Internet of Things has an AI-enabled rotary phone that acts as a docent in one of the experiences, right? How I was in Docent when you went through that experience guiding everybody through, the rotary phone takes on like three different personas. One is Scotland Yard, the other is the killer, and then the other is a voice from beyond, you know, because Arthur Conan Doyle was into clairvoyance. So that idea of how can you be playful, but I think the key to it is finding something that allows for some type of a human touch. So we're fascinated by how do we make engaging human experiences, and then we find the technology. So in a lot of ways, I guess we're story agnostic. We don't really care if there's emerging technology in it, if it's VR, AR. if it's AI, if it's IOT, or any other acronym that you want to throw at it, but just this idea that it's about that experience, and if we're doing that right, then we'll find the right platform or technology to tell it, or maybe we won't use any technology, it just depends.
[00:11:39.388] Kent Bye: Also reminds me of Dungeons and Dragons where there's a dungeon master who is in some ways the drama manager of the storyteller, but also is able to say no at certain times. You know, he can have people roll the dice, but he can also say no, that's not going to happen. And so in some ways the dungeon master is managing that overall arc of that story. And so what it sounds like is having that docent within these experiences and artificial intelligence that you're able to add these characters that are able to maybe add some more structure or to guide the narrative along the way such that they could either give instructions for what to do next or to help create some sort of twist or narrative tension within that process of telling the story.
[00:12:17.727] Lance Weiler: One thing that's interesting is we're working on a new project that one of the focuses of the lab is to embody research here at the university and we're collaborating with SAFE Lab which is out of the School of Social Work and what we're doing with SAFE Lab is they do a lot of research using natural language processing and they do research looking at the escalation of violence through social media. and they take data from Twitter and then they run it through NLP to try to see if they can determine patterns and then figure out ways that they can help deploy local intervention through organizations. Most of their focus happens to be in gang-related violence in the south side of Chicago, some parts of Brooklyn and in Harlem. and we got together with them, looked at what they were doing, and said, well, what if we came together and we created a de-escalation room? What if we created a creative engine that took a little bit of what the form of an escape room is, and what if we married that with some of the practices from Theatre of the Oppressed? And with Theatre of the Oppressed, for anybody who isn't familiar with it, you know, it's a process where you use performance with a group of people who are from a community, who are within that community and and they role-play and they tell stories and they perform those stories and Constantly iterate on them to try to find a truth in order to create solutions for themselves So it's not about like parachuting in with a solution It's like these wonderful ways that solutions can come from that community. And so with the de-escalation room we're exploring that like how do you build a creative engine and that then can be dropped into a number of digital scholars labs in different cities. So then youth are able to start to not only learn de-escalation techniques and digital literacy, but they're also able to do it in a way where they have a voice and they're able to do it in a way that they feel is important. And so I think that that's fascinating in terms of thinking about these ideas of shared narrative. What does it mean to create a shared narrative? How do you go about doing it? And how can story be used as an innovation tool, communication tool, mobilization, learning, entertainment? So a lot of our focus here at Columbia is on that. we come out of the School of the Arts and there's a rich tradition for story, you know, whether it be through the theater program, the film program, the creative writing program. So we're really focused on story and so a lot of the things that we're doing at the Digital Storytelling Lab are looking at how can we be better storytellers in the 21st century and I think the biggest thing around that is realizing that it is not just a one to many anymore, it's many to many.
[00:15:06.779] Kent Bye: Yeah, and when I had an interview with Jason Riggs of AASIC, one of the things he said is that the future of audio is immersive and interactive. And I think that's generalizable to almost every media, is that it's going to be more and more immersive and more and more interactive. And so when you look at storytelling as a craft, it seems like it's a time-based medium, and usually it's been very authored, and there's a certain philosophy around it that goes back thousands of years. We have video games that are more interactive narrative mechanisms, and immersive theater is becoming much more popular. But I'm just curious, from your perspective, it seems like we're moving from just storytelling into this process of experiential design, where it's an interactive experience rather than just a passive receiving of the story. So I'm just curious how you think about that, engaging the audience, and what kind of metaphors you use to make sense of that.
[00:15:56.102] Lance Weiler: Well, I think emergence is probably the biggest one. I think the one thing is you have to, it's interesting because it's almost like you're planting flags in the distance and you're trying to figure out how you can get there. And then all of a sudden you have a whole swarm of people that are all moving in all different directions and they're all pulling this mob in like different ways and you're like no no we need to go over there you know but then you're you're like oh well if I try to just control that it becomes counterproductive. How can I better create paths or allow them to create the roadway to get there? together, you know. And so I think that finding that is a real, it's something that we're constantly experimenting with. We do monthly meetups, for instance, at the Film Society of Lincoln Center. We use that as a social space where we can just test out, you know, methodologies, you know, shape pedagogy and just play and experiment. So I think in a lot of ways having those types of spaces where it's drawing from a diversity of perspectives and disciplines is kind of what I feel like the key to these new emergent forms. And in some ways they're not so new. Certain aspects of them, you know, go back a long time, whether it's oral traditions around you know, embellishing a story in the way that they were passed, or it's things that were found within theater in terms of the workshopping of theater, or the idea of improv, you know, like, we're just kind of picking, you know, almost like, oh, that's interesting, let's try that, or whoa, what are they doing over there with speculative Design and what they're creating objects for the future. Oh, let's try something like that And let's see what that looks like if we mash it up with these other things So it's kind of almost like this crazy big stew, you know, sometimes it doesn't taste very good and other times it tastes delicious And and so what are some of the biggest questions that are driving your work forward? I think some of the biggest questions are, how can we make it truly inclusive? And I think some of the other elements are, how can we give more purpose to what it is? Can we use these things to help shape policy? Can we use these types of techniques and when I say things I mean these emergent spaces where we're allowing people to come together and co-create and what feels playful and feels like a narrative then starts to lead towards these potential threads that could be used for you know problem solving or policy shaping or further education or if somebody wanted to use them for entertainment. So I think some of the things that I'm fascinated by is, I'm really fascinated by systems thinking. You know, this idea of how all these pieces can kind of come together and how do we build creative engines that allow people to do it. I've been doing immersive work for a long time, you know, not even knowing what it was called and often it felt like it was very bespoke, always recreating the wheel over and over and over again. And so I think some of the work that I'm trying to do here is I wanted to build a space at the lab that allowed us to be truly innovative, that allowed us to experiment, that wasn't dictated by market trends or bottom lines, that allowed us to really push at what these new forms and functions could be. So when I think about that, I'm constantly challenging myself to think, how can I make that inclusive? How can we make it more mutually valuable for the people who come into it? and how can we use story as a way to not only connect us but to shape these shared narratives that allow all voices to be a part of it because we all know how powerful narratives can be. And I think at the end of the day our big focus is how do we give humanity a seat at that table with technology so technology just isn't shaping us. And so I'd say that that's the big one. It's this really amazing opportunity to kind of look and say, arts and science together, what does that yield and what can we do with story and code?
[00:20:00.565] Kent Bye: Great. And finally, what do you think is kind of the ultimate potential of virtual reality and other immersive technologies and emergent storytelling and what it might be able to enable?
[00:20:12.074] Lance Weiler: I think I'm most fascinated these days, I'd say, around ubiquitous computing. I'm really interested in this cross-section of artificial intelligence, and where it could potentially dovetail into intelligent machines, and what does that look like, and this idea that you could lay narrative across the physical world. I think augmented reality is fascinating. I think there's been some interesting things that have been done in VR. I think VR is a little more challenging just because of where we are in terms of the wearable aspect of it. I think, you know, AR is fascinating in terms of the sheer, you know, with ARKit and some of the movement by Apple and what Google's working on and others. kind of puts it in your pocket. So it'll be interesting to see. But a lot of those things are all in search of use cases. And when I say use cases, I don't mean just a need. They're in search of like, how do I tell stories with these things? What do they look like? And how do people interact with them? And so I think, as I think about those things, I wonder where the great works of the 21st century are going to come from. You know, is it going to be something where AI is helping to augment creativity from a a swarm of people who are working on something globally together? Is it going to have things that kind of tentacle out that make use of all these emergent technology and things that we're not even thinking about yet? So I try to keep reminding myself, I'm into this idea of calm technology. I'm also into this idea of a reduction of screens. And just want to kind of play with that, see what happens. I mean, come talk to me in six months. We'll see where we're at.
[00:21:48.900] Kent Bye: Awesome. Anything else you'd like to say?
[00:21:51.617] Lance Weiler: No, I mean, I think in closing, what we try to do at the lab is share what we learn and make it readily accessible, you know, so people can find it at digital storytelling lab.com. We're always looking for collaborators. So if anybody's interested in collaborating, we're game. And I think a lot of our focus is just being playful and remembering to have fun.
[00:22:13.278] Kent Bye: Awesome. Well, thank you so much.
[00:22:14.479] Lance Weiler: Yeah, my pleasure. Thank you.
[00:22:16.530] Kent Bye: So that was Lance Weiler. He's the co-founder and director of the Columbia University School of the Arts Digital Storytelling Lab. So I have a number of different takeaways about this interview is that, first of all, so I was really interested in the framework that Lance has been developing with his team there at Columbia University, and they had four major things that they were trying to achieve with each of their experiences that they were creating. So number one is a trace so this is kind of like the moment when an improv comedy troupe asks people for input and then you give different input or within the specific Sherlock Holmes case you're able to start to create the whole scene and then add little post-it notes to the little objects of evidence and then you have other people that are trying to take those things that you've generated into the world and then those get woven into the overall story and the experience. And when you'd see some of the things that you had created end up in the story like that, you do become much more emerged and much more engaged and immersed into the overall experience. Secondly, they're trying to grant agency to everybody that's involved. They found a sweet spot of an ideal group size around four to six different people so that if you have too many people, then you tend to have someone who really starts to dominate and take over. And if you don't have enough people, then there's just not enough for them to kind of have ideas be generated. So that seemed to be a good sweet spot for these ideal group sizes when you're doing these types of collaborative storytelling experiences. Then next is that you have some sort of consistent thematic frame such that you're able to step into the role of a detective and we all kind of know the normal tropes of some of these different stories and then you're able to kind of step in as a detective or as a lawyer. So I think it's in part by just being familiar with these wide range of different stories and tropes but also creating a world that is able to really hold that structure, whether that's, you know, you're having AI agents that are calling in a phone and able to kind of give that additional structure and drive that story forward. And then finally, it's about serendipity management of trying to leave enough rooms for the imagination and for the magic of the unknown to really emerge. You know, you're starting out in the beginning, you set a flag out, out in the distance of where you want to go, but there's a completely undefined way of how are you going to get there. And I think that process of co-creation as a group that happens as you go from point A to B through that that journey of creativity and collaboration and that yes and improv energy all of that is sort of like this magic that you're trying to have the possibility for so that it's not so well defined that there's no opportunity for people to really contribute in a really meaningful way. And I think that's what's so fascinating about what Lance and his group is doing is to really come up with these open-ended frameworks that really allow you to really go deep and have these really interesting collaborative storytelling experiences. So Lance actually has a experience out at Sundance that his team has been putting together. It's called Frankenstein AI. So I'm really curious to see how some of these ideas have evolved and developed and start to pull in more artificial intelligent agents within the process. And, you know, the other really interesting thing about this interview was taking something that seems like pretty much focused on entertainment and being able to have these immersive experiences that, you know, really aren't explicitly changing the world. But he's asking these deeper questions about how can you take some of these principles of immersive and collaborative storytelling and start to maybe center around these different questions that have a much more deeper purpose. Or is it able to shape policy? Or people are able to come together and cooperate and co-create something together? And are you able to bring this sense of play and joy when it comes to this shared narratives that are being generated by these groups of people that are trying to really find these different places of common ground? So they're trying to figure out how to make these processes more inclusive, more interactive, and more mutually beneficial for everybody that's coming into it. And that ultimately they're trying to use story to connect us, but also to create these shaped narratives that allow all the voices to be a part of it. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member. This is a listener-supported podcast, and so I do rely upon your gracious donations to continue to bring you this coverage. So you can donate today at patreon.com slash Voices of VR. Thanks for listening.