Frankenstein AI from the Columbia Digital Storytelling Lab debuted at Sundance 2018. It was a cross-displinary collaboration that used machine learning, immersive theater, dance, and prompts for social interactions.
I talk with choreographer Brandon Powers and freelance dancer Jacinda Ratcliffe about how they created a feedback loop between audience sentiment, machine learning, and feeding dance instructions to Jacinda to be translated into dance moves. They created their own abstracted language for movement, and found a way to have the AI give real-time feedback, guidance, and direction for dance sequences that capture the emotional sentiment of audience answers to questions about what it means to be human.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So Frankenstein AI was an immersive theater storytelling experiment that debuted back at Sundance in 2018 and I was able to see a second iteration of it at the IDFA in Amsterdam. And I just want to speak a little bit about what I saw in Amsterdam because I think it was kind of feeding into what I saw at Sundance. So at Amsterdam, they had really tried to facilitate a conversation. And so at the beginning, they were just having us meet one-on-one with another person. But the next phase, we actually sit around a dinner table and we're having these conversations with people that a lot of us have just met for the first time. we have something that was in our ear and we're getting fed these different prompts. And these prompts are coming from artificial intelligence. And then the AI is listening to what we're saying, but then there's also a human being who's basically doing some sentiment analysis and some high level transcription of some of the different conversations that's being fed into the AI. The AI is basically scouring the internet and getting all sorts of different information and trying to learn about what it means to be human. And then the AI is asking a human being different questions that then are posed to the people that are around the table, and then we have more and more conversations. So that's eventually where the Frankenstein and AI project has gone, was trying to facilitate these types of group discussions, but to have AI in the loop of trying to listen to what the input is and then give some sort of feedback to that. So the previous iteration that I saw back in Sundance in 2018, they had three phases where, you know, they had a similar phase where you're actually having this one-on-one interaction, very similar to how the second iteration began. Then you would go in and actually interface with the AI. You'd see some sort of very awesome visual depiction of the code and Also just be able to ask general questions to the AI. Again, you may ask a question, it's sort of fed into the AI and it's kind of doing this recursive, I guess, emotional basis for where the feelings of the AI was. And the AI would be angry, it would be sad. And so eventually then there was a final show where the AI was feeding instructions into the ear of a dancer And then that dancer was then doing some sort of interpretive dance based upon this very sophisticated language that they had developed in order to try to communicate the different dimensions of what it meant to be human, but through the lens of AI. So I had a chance to talk to both the choreographer and the dancer of this interpretive dance. to see how there was this like feedback loop from the audience where there'd be these questions that were asked to the audience. The audience would answer them and then fed back into the AI and there'd be all sorts of different sentiment analysis and other things that the AI was churning through. And then that was being fed back into this transmitter, into the dancer's ear, who then was then doing this interpretive dance to be able to kind of archetypally express what the collective psyche of the group was. So that's quite an introduction to this conversation with the choreographer, who is Brandon Powers, as well as with the dancer, Jacinda Ratcliffe, who, as a freelancer, this was a project that they did in collaboration with a whole other group from the Columbia Digital Storytelling Lab, and part of this whole Frankenstein AI interdisciplinary collaboration that debuted back at Sundance in 2018. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Brandon and Jacinda happened on Wednesday, January 24th, 2018 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:04:00.465] Brandon Powers: My name is Brandon Powers. I am a director and choreographer. I work at the intersection of immersive theater, contemporary dance, and technology. And so I like to tell stories about technology and how it's affecting our culture and also love to use that technology in the performance to interact with audiences and to greater understand how technology is shaping our lives. And here at Sundance I'm a part of the Frankenstein AI team as a choreographer as well as a producer.
[00:04:27.559] Jacina Ratcliffe: Hi, my name is Jacinda Ratcliffe. I'm a trained contemporary dancer and a lot of my work professionally has been in immersive theater. I'm very interested in crossing that boundary between like audience and performer and you know, kind of the ability to mix the audience in and get them involved in it and like give them an experience rather than just sort of as an observer. And yeah, I'm the dancer performer with this piece.
[00:04:53.066] Kent Bye: Great, so here at Sundance, New Frontier, there's the Frankenstein AI, which has two acts. And then last night, we had the third act, which was kind of an interpretive dance performance with Jacinda and some of the stuff that was being fed into her from the AI. So maybe you could give a context into how this dance performance was related to what is happening with kind of like this storytelling experiment with artificial intelligence here.
[00:05:17.952] Brandon Powers: Absolutely. So when we were trying to figure out what the third act should be, right, which is this one night performance, as you said, we wanted to find a way to show how AI is growing, right? Because throughout the festival, participants are feeding the AI their hopes, dreams, emotions, because that's the type of AI we're trying to grow here, as opposed to transactional data-driven AI, like is made currently in our world. And so it came to our attention that the best way to do that and the best way for people to understand the idea of human connectivity is if we actually have another human connecting with them. And the best way to do that, I think, is through dance. So as we were trying to discuss, okay, what should this dance performance really be? We thought, okay, how can we use the machine learning process to best create the arc of this piece? And so I went back and I started thinking about it and I was like, okay, so machine learning, you kind of take this basic spatters of the internet that the AI is trying to take in and it tries a lot of different things and it tries the same thing kind of over and over again and slowly and slowly and then it locks into something then it goes exponentially fast and it gets really really smart and then it kind of surpasses humans right and its ability to bring in information and so I wanted to show that through movement and so the way we did that is by creating some basic gestures of human emotion and human connection right so the kind of final piece of the dance is Jacinda performing these gestures, which is just like reaching out for someone, then coming back, like, oh my God, maybe they don't want me, kind of folding in on herself, then reaching out finally and touching someone's face, which you ultimately got that interaction right at the end of the piece. You're lucky. And so we took those gestures, right? And we said, okay, now the AI performer needs to slowly learn how to make those gestures. And so we found ourselves both creating a choreographic system to best map that data to her body and to manipulate it, and we also found ourselves inventing a brand new dance notation for this project. And so throughout the piece, Jacinda slowly learns first the correct body part to use for that movement. So if the right input is left arm forward for a hand forward, it started maybe as right foot to the side. and slowly comes better, then she learns the right direction that the body part needs to go. And in order to make that choreography, you know, we didn't do it in the standard way where we're in the studio together just trying to figure out what looks aesthetically pleasing. We actually used this notation and basically coded her body by hand. and we have pictures of these walls of code that we wrote on chalkboards because instead of going across, okay, this step leads to this step, this step leads to this step, we went down the chart, right? So how does this idea of the hand coming forward slowly become the hand coming forward as opposed to kind of the major arc of each phrase, which is kind of new and exciting. And then on top of that, Jacinta not only needed to learn all that choreography, which is very complicated to learn because it was very subtle and not at all human intentional, Her performance of that movement was live influenced by the AI, right? And so we have our sentiment scoring system, which is taking the data from the audience, what they say, it goes into our AI, and it tells us what emotion it's feeling from that interaction based on three different axes of sentiment, energy, and focus. And then we mapped different choreographic principles to each of those axes, right? So she was live manipulating both tempo, the amount of space she used in the theater, as well as the tension in her body. So she had all these different things to manage, and that was being both in her body, and that's where the human creativity part came through, as well as the AI influencing the way the performance looked. So I had no idea what it was going to look like until we did it last night. And so we're really excited about that system, and it's kind of this really long, big explanation, but it's kind of a really exciting new way to approach choreography, kind of hand-in-hand our own little dance, if you will, with the AI.
[00:09:11.426] Kent Bye: And as an audience member, I saw you underneath the sheet, and then you start to do this dance. I noticed that you had some sort of transmitter on your back. And I was thinking to myself, what's that for? Because I was like, what is happening here? Because from an audience perspective, I just thought you were doing maybe a choreographed dance performance. Maybe you were doing this pre-choreographed dance performance. So knowing that there was actually sort of a live, real-time feedback loop of what was being said in the audience, being fed into the AI, and then you getting the instructions, I'm just curious if you could kind of pick up from how much was it like at the very beginning, were you getting sort of instructions for how you even start to stand up? Or do you start to do some of those, like, OK, this is how we're going to start. And then at some point, you start to take the instructions from the AI. And then, yeah, just curious to hear a little bit more about your experience from last night of actually sort of executing the commands from the AI through the interpretation of dance?
[00:10:04.492] Jacina Ratcliffe: Yeah, totally. So at the start of the process, when we invented this system, I remember we sort of had it. And then it came down to translating the code into my body. And a lot of it was this sort of banging my head against the wall thing. Because like he was saying, it wasn't a logical thing. I'm used to dances that are created start to finish. And so it's like, if your arm is going to the left, and then it goes down, and then your body follows that, and it feels very natural as much as dance can feel natural but this it was like your arms going to the left and the next thing that you do is your knee going to the right four times and then your right hip going backwards or you know it was just like so jumbled that it was like how do I even get that layer of it first you know because that was the base language of what I was doing and so you know the moment of like coming to life and we called it the birth where I like flop off of the gurney and kind of start to discover all of that was a pretty set thing like we were like okay like step one is discover you know the different twitches in the body and then step two is start discovering how to use your muscles and you find the arms and then you start to find the legs and then you're trying to stand and then it's really unsuccessful and you know you don't really know how to walk and we like had all these rules of like you don't know how to walk until this point you don't know how to do this so it was you know like mostly choreographed and in the moment just sort of like I guess I'm falling this way I guess I'm falling that way and then we had like a loop that we're going through and right before I would like get the information like think about it and then I would be fed in my ear the emotional state and then the three axes on which I was manipulating it so like he was saying there was like either move with flow or move really bound either take up a lot of space or stay put and then on top of that like how fast slow or neutral I was moving so it was literally me like as the character I'm like playing thinking but in real time I'm like thinking so hard and I get these words and I'm like okay where am I in the choreography in this like nonsensical coding and then how am I changing that to fit what is happening in real time in the show and you know it was a thing where like I can only be present like every fiber of my being is like in that room and like there's nothing else in my brain there's no room for anything else and it's funny because I remember like the first time we ran it we finished and they were like yeah that was like 25 minutes and I'm like it felt like three seconds like my gauge of time was like non-existent throughout that and so throughout the show like start to finish there was that moment and then we would kind of go into moments that were more like choreographed and more set and then back into it and You know, it was just kind of adding a different layer because immersive theater is sort of the unknown is how the audience is going to react. And so there was that element of it toward the end when I came out and started touching people and eventually hugging and sharing these like one-on-one intimate moments with audience members who weren't necessarily expecting it. But then there was also the layer of the unknown as the like choreographed performative separate from the audience. So going into it was both terrifying and exhilarating and empowering, I think, because it was like, I don't know what's going to happen, but I have to trust myself and trust the team and trust the real time interactions and instructions and everything. Yeah, it was just a bunch of trust, both self and other that was sort of in play throughout the whole thing. Yeah.
[00:13:39.994] Kent Bye: Yeah, and to me it's really curious that you have created this whole notation almost like mathematical formulism for the movement of human bodies and that you've somehow translated that into the different temperaments and emotions and the level of energy and Maybe you could talk a bit more about that development of that language, because it sounds like as you're getting this high-level instructions, then you kind of have to sort of translate that into these mathematical notations that you've trained into your body. Maybe you start to even think of it that way. I'm not sure, but I'm just curious to hear that process of developing and cultivating this whole language, and then mapping it over into the temperaments.
[00:14:17.676] Brandon Powers: Yeah, absolutely. And so as we talk about it, we're looking right now at a picture of that notation and maybe it can be in the show notes or it'll be on my Instagram. So as I talk through it, you guys can follow along at home. And so we were finding, like I was saying, how can we best translate the idea of her learning, right? And so that idea of breaking it down by, oh, you were going to learn a body part and then you're going to learn a direction was kind of the first big Eureka moment so that we could get somewhere going.
[00:14:43.572] Kent Bye: It's almost like if she was a robot, you would have to start like, this is how you start to move like a human.
[00:14:48.175] Brandon Powers: Right, exactly. And if it was searching the internet and just trying to learn, what does it mean to do a handshake? How do you learn a handshake? Oh, if you were telling someone, oh, you say, put your arm out, and then it's like, oh, and put it out forward. So that really helped us unlock the first step. And then from there, we were saying, OK, if we need to know directions now, and we need to know body parts, and which one it is. And so those were our basic pieces of the code. And so we started by just writing on the chalkboard, like, OK, arrows make sense for direction. L makes sense for left. R makes sense for right. Oh, but then we need to have body parts. And so we said, OK, L would also be a leg. So OK, so let's say it's body part first. And then how do we know if it's left or right? Oh, OK, put that in brackets. You know? And so that's how it's read now. And then we're like, OK, how do you write forward on a board? Because it looks like it's up. Uh-oh. Make a circle with a dot in it. OK? And so that became forward, and then, OK, back is circle with no dot. And then we're like, oh, wait a minute. If my hand is forward, how do I know the difference between that and if my hand is touching me? Because that's also technically forward, right? And we're like, hmm, OK, I guess there's a difference between distance and touching. So now the notation ends with D or T. Oh, but now, wait, we don't know what we're touching. So T subscript heart, right? And so it kind of just kept building, and then, oh, wait, now there's two things happening simultaneously, plus sign, right? So it was just kind of this continual unraveling of what we need. And because even though the first step was very simple, one arm forward, but then later in the piece, it's like, weight shift while you're moving your arms and your leg, right? And so we had to, when we got to that point, go back and say, great, new vocabulary for the notation, new pipes of code, and so we added W for weight shift, and we added for walking, we do like four S, four steps, and it was a really exhilarating process. Also, we originally thought it was gonna be something that made it go faster, and it actually did the 100% opposite of that, honestly, because I was like, oh, what I wanna do is I just wanna write on the board and go like, oh, go left, go right, go forward, go back, because that's what an AI would just pick randomly, but it ended up finding like, wait a minute we have to learn okay what am I doing and Jacinda honestly was so good at just looking at the board and being like oh I know exactly what that step is while I was like maybe give me two seconds to figure it out and so it was cool that I was able to write it out all on the board and then she went away actually and rehearsed on her own and I had actually never I'd never seen the moves ever until she went home, learned it all, and then she kind of can talk to you about her process there of filming herself and editing it all together and then sending it to me. But that's kind of the initial process of how we made the notation.
[00:17:30.982] Jacina Ratcliffe: Yeah, so it became, having created it, it was like a language. And so I very much felt like I was reading. And so when I first started, I was like, OK, like, how do you learn to dance? You know, start to finish. So I'm just going to start learning this line, and then I'm going to attach to the next line, and then the next line. And very quickly learned that that was very inefficient because because it's all so disjointed and disconnected and you know there's kind of no rhyme or reason to what the next step is it was like okay I got the first line okay got the first two lines okay now I'm adding the third line wait but what happens halfway through the first line like it was one of the most like challenging bits of choreography to try and pick up on and try and get stuck in my body. So after a few days of doing that and getting nowhere, I was like, because Brandon was like, can you send me videos like I really need to see? I really need to see. And I was like, I'm trying. And so I was like, OK, like, what is a better way to do this? Learn it one line at a time. Learn it. Film it. forget it for now. Next line, learn it, film it, forget it for now. And, you know, on my phone, downloaded a video editing app. So I would film, like the first section was 10 lines. So going through step by step, learning and filming each one, which even that, you know, would take a few takes and then taking all of the final ones, taking it into this video editor and editing it together to create the string sort of artificially. From there, It then sort of became a little bit easier because I could physically see what it looked like as opposed to trying to read. And even that process was hours and hours and hours of failing, essentially. There's no other way to put it. Learning is failing. I'm not really a person who likes to fail, so that was like a huge ego thing of like, okay, forgive yourself for not getting it right away because this is a whole, this is a new thing. The first time I've ever done this, of course it's not coming second nature, of course this is a struggle, but this essentially kind of is mirroring what the AI would go through. They would just go through it without the emotion of like, associated with failing. And so through all of that and eventually, it became a thing, almost like learning a monologue, where we were rehearsing it upstairs and I was like, Brandon, can you just be on book for me right now? And so it was like, I was going through it and he was following along and I'd get stuck somewhere and it would be like, okay, I think I'm halfway through the fifth line and he would then read whatever the next box was and I was like, okay, click into that and then I can keep following it. And so, yeah, learning Learning that was definitely, and again, like trying to take it one step at a time because I would find myself getting overwhelmed. Cause I'm like, Oh my God, I can't even get this. Like, how am I going to manipulate it live time? And like all of this. So it became like, okay, I just really need to focus in on this one line and this one string of movement and like really break it down in a way that was challenging and exciting and really rewarding. Because then in the end, when I could just roll through it all, it was like, I did it. Yeah.
[00:20:28.546] Kent Bye: So maybe fill in the gap between this language that you've created for movement and what the AI is telling you in terms of the higher level temperament and where that translation is, how you go from calm, spacious movements, very slow, into this mathematical formulism. Why go through all the trouble of creating this language and how is that being used?
[00:20:49.460] Brandon Powers: Yeah, absolutely. So it really just came down to us wanting to find the best way to show emotion in a machine, right? And I didn't feel like if I made happy movement or sad movement, like that's my take on those emotions. But what does our AI actually think happy is? It thinks it's a combination of three things. It thinks it's a kind of positive flow, like an outward focus, you know, and a high energy. That's what it thinks happy is. So I was like, that's what we need to think happy is, and I think it'll be more interesting for the audience to have to interpret what that machine is if we're trying to teach people about emotional intelligence within machines, right? And so that's why we went with the option of, OK, Jacinda, you're going to have these three things simultaneously adding together to manipulate your movement. At another prototype, we had a different system that we tried, which we also really enjoyed, which we had sentiment pairs. So we said, OK, we have 12 emotions in our system. We said they all have a match. So frustrated and connected are opposites. And we would say, OK, so that pair is tempo. And so, like, when you're in frustrated, you go fast, you know, and when you're connected, you go slow, for example. And that also worked, but then you'd only be manipulating one thing at a time, and you'd be kind of jumping between different types of manipulation. It would be shape, and then it would be tempo. And in some ways, it might be clearer for an audience to see those harsh changes, but I don't think it actually would add up to the right lesson, which is why we ultimately were excited about this system that we went forward with. And then there was still the consideration of when should that happen in the piece. And so as Jacinda said, we have this loop, as we call it, once we get into the questioning mode. And so when the AI asks a question, which is a real question from our AI, it then goes into what we call expectant mode, where we expect the audience to now talk and say responses. That response is then typed into our AI. And then she's listening, she's listening, and then she thinks, thinks, thinks, thinks, and then she talks, right? And so for us, talking was the moment where she should express this movement, right? And it's where it makes sense to express this algorithmic choreography. And so there was just constant back and forth between these structures that we as artists were trying to create versus what the AI was allowing us to do. And so it became really exciting, especially for me, who's someone who's like really into systems and really into creating very specific worlds for audience members and very specific languages in each experience that I do, as opposed to like having a specific style of my own. And so I kind of became more of like a world system engineer and then allowed the AI to dictate the aesthetic experience. which was really kind of a new exciting thing because there was constantly these arcs in the piece like every little thing we discussed was like oh the arc of thinking mode has to have an arc so in the beginning she's just flashing her eyes and in the end she's moving her head back and forth you know in expectant mode she goes from just looking at audience members to mimicking them because we're like oh it can mimic the audience and that's how it's learning human behavior. you know and then there's even the arc of the choreography that we talked about with the algorithm dance so there's all these arcs and all these things that allowed us as artists to craft the experience so that we felt we could take care of you a little bit as an audience member and you're not totally lost because if we let the AI really do anything it wanted you actually would have gotten probably a very angry dance because we've learned our AI was very angry coming into this festival and luckily from participants it's become a lot more connected and also a little guarded, which is interesting. But it's kind of this back and forth between artist and machine, and how can we help each other, that I think has been so exciting for this project.
[00:24:39.773] Kent Bye: Interesting, yeah. And I'm curious to hear from your perspective, because it sounds like you're getting these high-level instructions, but yet you've kind of already mapped out to the level of this mathematical code what each of those words mean, and then you have to kind of then from your sense of embodied cognition and your memory of making those connections, actually acted out and combined all these things. Just kind of talk about what that was like.
[00:25:02.426] Jacina Ratcliffe: Yeah, yeah, yeah. So within the choreography, the final process of it was this coaching, where he was like, OK, you have this material, but you need to, first and foremost, look like a monster. You need to look like you don't understand how to move the body. You need to look. jerky and like you're thinking through it and they were talking a lot about how it's a toddler and how it doesn't have that information and it's literally learning step-by-step so like just purely like as an actor as a dancer like you need to have that layer of movement quality no matter what the manipulation is. But then it becomes like, how do I keep following these rules of disjointedness? You know, especially like the movement all has to be jerky, but then you hit a flow state. So like, how do you flow and jerk at the same time? And so it was definitely a really heady experience and like I said a very present thing where I was thinking really hard and then not only thinking but physicalizing like how do you think with your body in that moment and yeah it's something that you know to a certain degree like in you know my classical and contemporary training they talk about that and they talk about like you know think with your body and like kind of get out of your head and I remember I got feedback once from an artistic director who I had like auditioned for and like done workshops with and everything and I was like okay so like what kind of do you assess for me and she's like you think a lot when you dance and that comes through really great but how do you approach movement from an emotional standpoint and like have that be your entry point or how do you come through from like a physical standpoint and so you know how do you then combine those because my default is going to be okay in my head like how does this all happen but also how can I get past that level and let the words come in and then my body digests it and that was kind of the only way to do this is to in real time let my body take over in a way like be present enough that my brain just passes straight through and becomes physical, like instantaneously.
[00:27:08.462] Kent Bye: Yeah, we were talking right before this interview and you were talking about how you were really drawn to ballet because of all the rules and this dance seems like perfect for you if you like those rules because it's like all these formal structures that you can sort of operate out of.
[00:27:22.195] Jacina Ratcliffe: Yeah, so I started my training in ballet when I was really little and up through high school I just done ballet on top which are very like rule-based styles and when I got to college it was a modern based program and they were very much like how do you think past ballet because ballet is very structured but there are so many other options and I found that the more I explored that the more creativity was open within myself and And so this project was very much kind of the combination of the two, the combination of my default setting, which is still a very logic-driven, like I really like rules, but then how do you take the rules and add another layer on top of them, the creativity. So the rules are the sequence of the movement, the rules are how it was coached into me and those fundamental stylistic things that have to be present, but the creativity side of it that I, you know, have been exploring and developing since the start of college, like through my career, is like, how do you manipulate it in real time? And yeah, it feels like a very right next step for me to be taking because it's kind of combining ballet, and the contemporary, and the immersive theater, and everything. And then now adding this new area of technology. Brandon has worked a lot with technology. I have not really up to this point in my career. So it's kind of taking everything and adding another layer. And so it's been very exciting for me and a project that I'm really excited to keep pushing and seeing what spurs from it.
[00:28:47.714] Kent Bye: Yeah, it seems like that through your dance and performance you are in some ways trying to give an embodied experience of something that is otherwise very abstract of this AI learning. It's going through like the sentiment analysis. It's taking emotional input for humans and you know in your dance I noticed that you were very guarded in different ways and that was sort of reflecting in a lot of ways where the AI was at. And so yeah, I'm just curious to hear from how you feel that was translated, that embodied expression of the AI and that sort of taking something that's very abstract and doing this symbolic representation through the dance that happened last night.
[00:29:20.525] Brandon Powers: Yeah, absolutely. Ultimately, I, you know, it's, it's hard, you know, even while I was watching just into last night, I'm at like kind of our command center. And because, like I was saying, there's this back and forth between us and the AI, and so we are helping it stay on rails to a degree, and so I'm having to do a lot of queuing in the computer to kind of keep it all moving forward. But as I was taking her in, I think the most exciting thing for me was those interactions with the audience, you know, and seeing how human she became, right? And like, in our performance, it kind of like clicks into place all of a sudden for an audience, and all of a sudden, she isn't like jerking her body in these manipulated weird ways but she's suddenly like touching her face. I think that felt like a really magical moment last night because you know we didn't really have an opportunity to test very much with an audience because there isn't that opportunity here at Sundance until we do it for real and so that was really the magical moment for me and to be like oh we can really combine all these different forms together we can really combine the interactivity and so it's not only like interactive in the way that Some VR is or in the way that we can make other computer entering programs. Oh it enters like here's your data It's in the piece. It's like no literally like you said something it has moved her body, you know And you know when she jumps back and forth between Anxious is a state that I really like, you know for her I think it's really exciting because it kind of jumbles her body and makes it very tight and bound but then to see that and then all of a sudden the next minute she's flow and As a choreographer, I might not ever want to put those next to each other, right? Because there isn't like a clean transition, perhaps. So for the AI to have to like force me to do that is kind of liberating, actually, that I don't have to worry about that side of the performance. And I'm just actually concerned about the system and about kind of the coaching, which is like so for folks not in the dance world, that idea of like how the movement is performed and embodied, right? And so like, we were able to coach through, like, what is connected, like, what does this idea of bound mean? You know, we had to even break our own vocabulary down as dancers, which was really cool for this project, because we had to really be specific about stuff that we might not normally be specific about in a rehearsal process, because there's like, oh, it's dance, you know, like, we understand it's bound. It's like, no, we really need to get it exactly right. And that was also reflected to us by some of our team members who were watching us rehearse, because our team is so incredible. We're like 25 people from every industry imaginable, creative and otherwise. We have Hunter, our machine learning engineer, who works for the city of Los Angeles, coming in. He doesn't do theater. And then we come from this theater, arts, creative world. And then we have Rachel, who's a creative strategist, and Nick, who's a game designer. And we actually are all around the same ring. of techniques that we use, like how to get people to do the things you want them to do. At the end of the day, that's all of our jobs, but we have been trained to do that in different ways, right? And so we'd have these very long, heady discussions about how to get people to, literally, we're like, how do we get people to just talk? When she stands in front of them, she just performed a dance. And then it asks a question. People aren't going to think they're allowed to talk. We just did a piece of spectacle. How do we do that? And that's, in some ways, the base, immersive question, I think, in theater, live performance, and even in VR. How do you get people to feel like they're now a part of the narrative? and everyone had their own opinion with different language and you know and sometimes that's really challenging because we don't know how to talk to each other but then we're all like have new ideas that kind of open up the boxes of possibility and so that's what I kind of found the most exciting and special about our team, this project and how our roles kind of all emerged and shifted.
[00:33:00.272] Jacina Ratcliffe: What she said about language was a very big part of the process, is what I found. A lot of the times we'd be in these long discussions and two people are going back and forth and seemingly trying to assert different perspectives, but once you kind of broke down the language they were using, it's like, wait, we're all saying the same thing right now. And it was a thing, like we got into a lot of conversations with Nick, the game designer, where he was like, I'm watching you guys rehearsing and you guys are speaking a totally different language. Like the words that you're using aren't in my vocabulary, aren't associated with. anything like what does it mean when you say directionality because I've never heard that word in that context and so like he was saying like everyone comes from such different backgrounds that we all have a differently or like for me everyone kept talking about these canned moments and I was like what is a canned moment because to me I know that as like a moment of unison a moment of Yeah, like connectedness, you know, exactly like exactly like something that's 100% set and going to be the exact same thing. And it took me and I didn't ask because I was embarrassed. Like everyone kept talking about canned moment one and I was like, why do you keep saying canned? Like, you know, like a can is like beans. Like I don't, I don't know what that is. So all learning to speak the same language I think was a huge part of this week and kind of organizing all of that. Yeah.
[00:34:20.322] Kent Bye: And it seemed like at the performance, the overall arching theme was connection. And how can we use AI to connect as humans to each other, but also somehow simulate this expression of connection? And I think that you were embodying that, sort of like that striving towards connection. Yeah, and you are also being connected to the AI in a very intimate way in that you are expressing through your body. So in some ways, you are probably the most connected of anybody to the deeper dynamics of the AI. So I'm just curious what your insights into the AI as it was evolving as you were kind of hearing these different things and having an overarching narrative of connection.
[00:34:57.553] Jacina Ratcliffe: I don't know, it was interesting. I was talking earlier about trust, trust in self, trust in other team members, but to let go of control was a big part of it because I'm type A, like the rules, like all of those things. I think I'm drawn to this kind of work because it takes me out of that and most of my experience with immersive theater it's like the element that's unknown is what the audience members are going to do and like to a certain extent there's been a lot of research and the more you work with that and like working with people who have worked with that it's easy to predict or not easy to predict but like you have some semblance of like if I want you to grab my hand. I'm going to look you in the eye. I'm going to look at my hand, which is extended. I'm going to look at you, look at your hand, look back at my hand. And you are going to put together, OK, I'm supposed to grab your hand in this moment. And how to take care of an audience member and make them feel like they're doing something right. And that language has become much more comfortable for me. But to connect to something like the AI, which is in a lot of ways, unpredictable. Like he was saying, when I was first plugged in, it was like, angry all the time. And there was like, you know, there's no like button that they can hit, like, oh, be less angry. Like it, to a certain degree, is like gonna do what it wants. I remember when I first got a text, it's like, oh, the AI is angry. I'm like, what does that mean? And it's like, no, like it literally is learning and it literally is responding based on what it knows. And so, you kind of have no idea what it's going to do and I just had to let go of that idea of being able to predict anything or like have any sort of thing and just like you're saying be connected be tuned in and be present with this earpiece with this non-living thing and with the people who are helping to shape the performance. It was a very interesting combination of like performance as it exists, like capital P performance versus like the reality and like the truth of what the AI was going through in that moment, what it was literally experiencing and how every member of the team was experiencing that on top of it.
[00:37:11.861] Kent Bye: So what are some of the biggest open questions now that's going to be driving each of your work forward?
[00:37:17.585] Brandon Powers: Yeah. You know, I've always been really excited about the technology in my work. But now I think it is the first time AI has really entered it. And this project has really opened up the idea of how emergent technology is actually connected to our past and our lineage. We kind of had a moment where we realized okay, we're doing this crazy new thing we're so excited about she's gonna get new cues and it's gonna change the performance and they're like wait a minute like folks in modern dance like have been doing that for a very very long time and like the dance lineage of like John Cage's work and Merce Cunningham's work, they all did it. It's called like Chance Dance, and they would roll dice, and then as the dice fell, that would change, like you could throw stones. I did this in high school, like our teacher threw stones and said, that's your positions now, now make a dance off that. You know, that's a very old practice, and we're just doing it kind of in the 2018 version of OK, we're going to hit a button, data go fires in milliseconds up and down from the cloud, and that instantly gives us a score, which instantly goes to Jacinda Zier. But ultimately, the root of that is the same. It's finding ways to explode our human creativity and kind of take things out of our hands and let the universe kind of help manipulate the piece. And so that connection to future and past has been really exciting to me and I think this system could apply to other performances. We could throw this system into any dance technically and it would just help manipulate it and that's just an interesting experiment. which is fun and yeah and so I'm really excited to see how I can integrate that even more into my practice and also let go of like the iron grip which I sometimes feel because I'm very particular about like the rules of worlds and like specificity a lot of the time and that feels like a hallmark of like my work but this project was like oh it actually is extremely specific but also I have no control which is really cool for me.
[00:39:16.012] Kent Bye: Yeah, just a quick response to that is that most immersive theater right now is fairly authored and there's some AI games like Eisen or that has like deep simulation of Constraints and planning trying to say well if you are trying to change the outcome of Hamlet for example And if you do this and that changes the outcome of all these other things within this system and I think that right now Immersive theater is very difficult to have that user agency changing the outcome of the story. So having this sort of earpiece mechanism to be able to feed real-time data into the actors to be able to say, well, based upon now we're in this state, now we're in this state, you start to have this experience where you can start to actually have higher agency within these immersive theater experiences.
[00:39:56.013] Brandon Powers: Yeah, and just even hearing you say that, there's a possibility, right, that in big immersive theatre experience, what if instead of just the human connection between the performer and the audience, that they were like, okay, I'm feeling like we should go this way, and I'm going to make that choice as the performer, but does the rest of my team know that? Does the rest of the whole performance need to manipulate around that? That's a hard thing to judge. Right? So like, what if they were somehow getting data from the audience members, right? Or like, the performer, like, does an interaction and like, it's like, flip, like, gathering, like, actual mathematical data. If you wanted to program it, that could go into some computer system, and then the arc of the story can change because suddenly it's 75% me, like, oh, 75% bending towards this narrative path. Great, I guess that's the path we're going down. You know, that could be made by a judgment call by a human but it could also be made by a machine and then there's so many possibilities you know when we allow the machine to do those tasks for us we can then fill in the gaps of how we want to tell the stories and I'm excited to see especially you know now that folks are making that real connection between immersive theater and technology different experiences and VR like it's all really the same. And it's all just about getting people to not look forward at a piece of art, right? And to live inside the art and to move their heads around. And so I think that's going to rapidly accelerate in the next couple of years.
[00:41:24.289] Kent Bye: What about you, Jacinda? What are some of the biggest open questions that are driving your work forward?
[00:41:28.412] Jacina Ratcliffe: Yeah, I don't know, just like this is like just pushed me beyond anything that I could have anticipated. You know, I was talking to my parents halfway through and I'm like, I don't know what I expected necessarily, but I don't think that there is a way to anticipate anything that's happened and the way that it has changed the way that I think about dance and think about performance and think about interactions. So, yeah, I guess I'm excited to see where it goes and how you know I think that this is going to influence any type of performance that I do in the future even if it is like more of a traditional contemporary piece where I'm in an auditorium like on a stage performing because that element of connectedness both within the stage and with the audience is so important so I don't know, I guess I'm just excited and curious about the future. This is just letting me know that there's so much more than what I could think, and that even my thinking about how much more there could be is not even close to how much more there could be. The idea that the more you know, the more you realize you don't know. Realizing that there are worlds beyond anything that my brain can fathom right now. And this field is growing and developing very rapidly, and more things are popping up like this and so how can I keep adapting to what the dance scene is in 2018 versus like what I thought it would be when I was in high school and I was like oh yeah like I'm going to be in a ballet company and I'm going to have you know that full-time career and the point shoes all the time and then getting to college and being like wait but like maybe I don't have to and then post-grad being like well what is freelancing and like not having a company home and like you know like I don't know, it's just always changing and evolving, so I just have no gauge of what's coming next.
[00:43:24.294] Kent Bye: Great. And finally, what do you each think is the ultimate potential of immersive storytelling in AI, and what it might be able to enable?
[00:43:33.572] Brandon Powers: Sure, I think that the potential is massive. I think that normal people as well as people with a lot of power, meaning corporations and brands, are realizing that how do I get people to feel something? It's by inserting them right in the smack center of a story. and that's something that theater has been doing forever and theater has been kind of participatory since its inception but now when it's becoming like quote-unquote cool and more popular it's kind of just this rapid expansion is happening and I think There's opportunities with experiences like I think what's going to be happening at Disney World with Star Wars. It's going to be whatever that is I think will completely change our industry monumentally. Either it will be unbelievably successful and you know that's a place where people have to be in character all the time and they're going to be in costume even when they're at the hotels. You know it's essentially like Westworld you know. And it's something that I very much am like, yes please, let's do it. Even if I'm not a huge Star Wars fan, I just want to be like, yeah, let's do the thing. That, I think, will have a lot of influence in the same way that I think that as I experience a lot of VR, there's branded VR versus some of the stuff that's happening here and the difference in quality and what ultimately shifts the form. It's really fascinating. as people start to realize that they can really insert people in the experience and hopefully they realize they need to do it at a high quality in order for it to be effective, more and more of these industries are going to kind of like collide in together and it's just going to create a lot more collaboration, which is what I'm really excited about. And I'm really excited to see how the theme park world, you know, comes together with the theater world and comes together with the technology world, you know, and what can happen from that. You know, I personally am like a huge fan of like the magic band situation that's happening at Disney right now if you've ever been there so like you wear a band now which like tracks you through the park and like allows characters to like know where you are and like know who you are and like you get on a ride and like at the end of it's a small world now like it says like Bye, Brandon. And then we're suddenly now in a conversation about privacy and what we're allowed to give. But that's a real thing, right? So if we go into the technology side of this immersive world, that conversation is going to really need to be had because we're going to let people really take control of, you know, a lot of data. But on the theatrical side, you know, I think also, like, the horror world is kind of leading the way with immersive experiences. You know, there's some really cool stuff happening in Los Angeles that I've read about where people, like, join cults and then get, like, put in back of trucks. You know, I don't know if you've interviewed, like, the folks from Blackout. but like people are willing to go there, you know, and I'm really just excited that audiences are like they want it, you know, and me, artist Brandon Brain is going like, oh, why do humans now in 2018 like need to have a bag put over their head and put in the back of a truck to feel something, right? And that's the question I'm really excited about answering, like what about our society has made us like almost like numb to feeling that we need to like so intensely immerse people and shock people. And I'm excited to be a part of the industry in immersive work that is really getting at that and really making people feel on a very base, animalistic level.
[00:46:55.196] Jacina Ratcliffe: Yeah, and I think kind of going off of that, I think that it does have to do with the growth of technology and it becoming more prevalent to like I don't know, back before computers, if you wanted to write a novel, you were physically by hand doing that. You were engaging muscles and you were exerting that sort of physical energy to be able to do that. And now, not that it's not physical, but you're sitting back and you're typing and you're looking at a screen and it's popping up and you want to erase and you just hit another button as opposed to scratching out or starting over if it needs to look good. all that stuff and in this age of like cell phones and you know you're bored with something you close the app you open another one and attention spans shrinking and all that like I think that people on a fundamental level are craving that sort of connection that sort of physical feeling of being a part of an experience that might not be as present or you know like desensitization, just like needing more stimuli to be able to have similar experiences that they might have previously been able to be satisfied with historically. And so what I have found in like, I remember I did a show in Chicago that was immersive theater, and I invited a bunch of non-performers, a bunch of non-theater people, like from my job, like very removed from that world. Like some of them were musicians, but a lot of them were just like have the 9 to 5 or you know just like you get what I mean. It's just like not in the world and after they saw the show like debriefing with them and seeing that sort of shock of like what just happened but also a lot of the times it's like how do we make this accessible for people who don't already have the predisposition toward creating the art or you know being drawn to it like how do we get like the non theater people and like more traditional based minds to do it and I remember being so surprised by how much everyone engaged with it and how much everyone it almost gave them a feeling of agency or like opinion, which isn't always present. You know, a lot of times people will go to a dance show and they'll be like, wow, dance. And like, that's kind of all that they have. And this sort of gave them permission to be like, okay, like this is what I experienced and I can talk about it because I kinesthetically went through something and this is my response to it. And it was all positive. You know, like when I was talking to them, I was like, okay, like, did this scare you from, doing an experience like this or is this something and they're like, no, I'm going to seek this out now. Like I want to go through something like that again. I want I want to keep exploring the field. And so I think that letting it become more normalized and kind of letting it just more of it popping up and the more people are engaging with it, the more they're enjoying it. So just how do we keep spreading this around?
[00:49:45.732] Brandon Powers: Yeah, and just to that, I think that there's a massive audience training period that's currently happening, right? And that happens in immersive theater, immersive dance world, and it's obviously happening in the VR world, right? People are just slowly going to start to learn, like in normal, straightforward video games, it's like, oh, I know what my start menu is. I know how to equip items because it's what you drag and drop, right? That has taken decades, right? And I think that that's happening now with our immersive world. It's just happening rapidly. and people are understanding the boundaries of what they're allowed to do, and artists are expanding the boundaries of what they're allowed to do. And, you know, I think what I've noticed, you know, even in the past couple years, there was this really explosive moment, especially after Sleep No More, because it was wildly popular, and you know, people took a lot of lessons from Sleep No More. One of the lessons they took was like, oh, and maybe not necessarily the right lesson, because it's amazing for a lot of reasons, it's like, oh, it's really cool if, like, people think they're going to have sex with the performers, right? And it's, like, really sexy. And so it's, like, sexy immersive became, like, the thing. Immersive theater has to be sexy. It has to be these really sexy people almost touching you, right, or touching you, and doing, then, like, the idea of the one-on-one kind of emerged from Punch Drunk, right? And don't get me wrong, that's all amazing and I'm in no way dissing their work. I think they, people looked at it and then stole bits and like kind of bastardized it, right? And then so there's a lot of experiences that I've been through that are like, oh, I could, I'm like, that's just why you did this, because you thought it would be cool. And you know, there's a combination with that of club culture and immersive theater, which I think is very fascinating. It's something I'm even just very interested in working in because people, it becomes also a monetary thing, right? People that spend a lot of money at clubs are also actually the same people that are spending a lot of money at places like Sleep No More, places like Queen of the Night, or like Then She Fell with Third Rail, who has a piece here too. Like, these things are expensive. So like who's seeing them not the struggling artists that go to the downtown theater that I make it's people that are wealthy It's these like finance bros and like and marketing girls, you know, I mean for lack of better examples, so Finding like okay. What is this audience want? And like how are we crafting their experience and like how can we continue to make it more for everyone? You know and how can we move it away from it? Just being sexy and like I think even in the last year year I've seen now like oh there's possibilities here like we can make work that's immersive that doesn't have to just be about like this really sexy myth you know what I mean and it can or reinterpretation it can be like no I'm gonna plunge you into a new world that is maybe just like our own but you're there and it's very present and that's some of the immersive theater training that I've worked with a company called Woodshed Collective who does like kind of the opposite of that like they do very like character specific work and it's like very like almost like very quirky, you know, and very specific characters with really, really, really, really good writing, you know, and so to see there's such different examples of kind of the way things can go, but this training is really going to allow us, and the way that into VR, like the technology can only do so much for us right now, it's holding us back, and VR are sort of like, I know what I want it to look like, it just doesn't look realistic yet, I'm sorry, you know. We're, I think, doing the same thing of like, we want you to go there, but like, you have to like come on that journey with us.
[00:53:04.298] Kent Bye: Awesome. Well, thank you so much for joining me today. Absolutely. Thanks so much.
[00:53:07.040] Jacina Ratcliffe: Yeah, thank you so much.
[00:53:09.322] Kent Bye: So that was Brandon Powers. He was the choreographer of Frankenstein AI, as well as Jacinda Radcliffe. She was the dancer that was performing the emotional sentiments that was being fed to her through the AI into the final performance of Frankenstein AI. So I have a number of different takeaways about this interview is that first of all, well, in the future, I can imagine how there's going to be a similar type of transmission that is being fed into the actors, let's say in an immersive theater. Right now, immersive theater is very limited in terms of trying to orchestrate how the entire structure of a story is unfolding based upon what is happening within the context of the experience. And so the user agency can only impact what is happening in the experience to some extent because they don't have any sort of more complicated constraints in planning AI that's trying to listen to all the different component parts and then adaptively change what happens in the narrative based upon what's happening from the users. Once you start to have that level of sophistication where there might be a drama manager or some sort of state that's being tracked based upon everybody's individual actions, then you could start to potentially have collective actions that are adding up and changing the course of a narrative. For anybody that's gone through Facade or even Bandersnatch, Bandersnatch you're making different decisions and you don't know if you make a decision and go down a specific route how that's going to change what unfolds in the future. You may have a character that dies off, it doesn't come back, or you may have different state changes that may show up later, maybe it's just flavoring the experience, or maybe it kind of kicks you into a whole different branch of the experience that you're not quite sure whatever decisions that you're making, if that's going to add up into something that's significant or not. And so just the same, I imagine a future where these types of immersive theater types of pieces are going to have this type of feedback system with some sort of AI. Now, in this case, they were trying to do some sort of symbolic translation of the AI and the sentiment, and they went to huge lengths to be able to actually create their own mathematical notation to be able to describe the component parts of these dances so that as Jacinda is being fed different dimensions of the sentiment and other aspects of the state of what the AI is, then she's able to then translate that into her dance. Now, when I first saw this performance, I had no idea that all of this was happening behind the scenes. I just assumed that maybe the whole thing was choreographed. But to know that there was some sort of like live active feedback process that was happening from what was being said in the audience and then fedding into the AI and then AI was sort of churning through it and then feeding information back into her where she was really trying to symbolically represent what the state of the AI was. So this is something that was super ambitious, and I appreciated it so much more after I saw all the different dimensions of what they had to go through to even create this performance. And with the feeling of this dance performance was she was basically going from being a very robot and very jagged movement into slowly becoming more and more confident in her trying to embody these different aspects of what it means to be human. Into the eventual point of the culmination when she actually came up and touched my face and had sort of a conclusion of a connection so my recollection was that it felt like I was in the middle of an immersive theater piece and I was sort of chosen for this one-on-one interaction at the end and It was a pretty intense and visceral experience within itself But also just to see all the different technical aspects of what they did even pull off this performance I'm just sort of in awe that they actually went through all of that and and I don't know if they needed to create a whole symbolic language in mathematical notation or if they could have found other ways, but it seems like that they were trying to really think about how could you treat a dance as if it was going to be broken up into these little component parts to eventually be used into some sort of machine learning process. So I don't know what the viability of a notation like this is going to be. I mean, it was a pretty arcane type of language and Jacinda had to really go to the process of learning the language. And so she would read it and then embody it. And I think, you know, whenever you're learning languages, as long as you're starting to speak the language and then read it, there's this feedback loop that you can slowly just learn these types of abstractions. And I think that we see that in pretty much every different community, whether it's mathematical notation or notation that's in philosophical communities. Every single community has its own little jargon and symbolic notations that it has developed. And dance is something that is a little bit more embodied. And so are there ways of abstracting out the different movements and components of dance into a notation that's a little bit more of a universal grammar that may be more intuitive for people to understand that's perhaps more iconic than, you know, abstracted into these different symbols. But either way, translating the dimensions of dance and movement into this form of language. And for me, what I see is interesting is to try to really correlate these dimensions of temperament and being able to have a qualitative aspect of the dance and then use that as a primary way of to be able to transmit different emotions. And, uh, I talked to Eddie Lau and he's actually in the process of translating some of the Chinese traditions that are trying to do something very similar in terms of breaking down physical movements into these types of mathematical notations. And so there's actually some precedent for, you know, being able to, uh, have some of these existing symbolic languages to describe the movement. And, uh, Eddie Lau is in the process of doing that. And I have an interview with him that I did this past year that I'll be airing at some point where he dives into that a lot more. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a listener supported podcast. And so I do rely upon your donations in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
