Brandon Powers is a creative director and choreographer who is creating experiences across physical and virtual space at the intersection of performance and technology. He was showing a dance performance at ONX Studios during Tribeca Immersive that was titled Kinetic Diffusion. It was created in collaboration with Aaron Santiago, and featured three screens that were being filled with delayed generative AI footage in near real-time and 30 frames per second, which required eleven 4090 GPUs in the cloud to achieve.
Powers was recording his dance with a mirrorless camera, and then was applying a depth map AI model to extrapolate his embodied movements so that it could be input as a real-time feed into Stable Diffusion with a set of prompts that were precisely timed out. The AI generated images ended up having a 2-8 second delay, which gave the effect of Powers dancing in a duet with himself, but modulated through a series of style transfer prompts. Overall, it was a hypnotically impressive display of generative AI at the intersection of XR and dance. I had a chance to catch up with Powers after his performance to get more context for how it came about, and the long evolution from his previous explorations at the intersections of AI and dance with Frankenstein AI that premiered at Sundance 2019 (see our previous conversation about it in episode #728). You can see a brief explainer video of Kinetic Diffusion within from Powers’ TikTok channel.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So this is episode 14 of 17 looking at the intersection between XR and artificial intelligence. And today's episode is with Brandon Powers, who's a creative director and choreographer who's doing this live dance performance where he's got a camera that is recording his movements and his dance, and that's being fed into stable diffusion, which has got this detailed timing as these different beats and these different prompts, but there's a delay. And so he's basically doing a performance with himself. So it's a duet where he's dancing with this generative AI interpretation of the images are being projected onto these three different screens. And so there's like 11 GPUs in the cloud that's helping to generate 30 frames a second of all these these different generated AI images. And yeah, it's a piece that's called Kinetic Diffusion that showed at the Onyx Studios summer exhibition that they usually have during Tribeca. So it was a really impressive piece where you start to see this cutting edge of real time, 30 frames per second, generative AI with dance performance embodied. And yeah, just a really mind blowing to have a chance to sit down with Brandon to see all the different stuff that he had to do to put all this stuff together. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Brandon happened on Saturday, June 10th, 2023 at the Onyx Studios exhibition in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:40.358] Brandon Powers: Hi, my name is Brandon Powers. I'm a creative director and choreographer creating experiences across physical and virtual space, intersection of performance and technology. I like to describe my work kind of with three main pillars, create, build, and transform, I create experiences, I build interdisciplinary community between artists and technologists, and I run a musical theater XR program called MTF XR at Musical Theater Factory, a non-profit artist service organization in New York City, and transform, thinking about transforming embodiment, our relationship to our body with technology, and I've created a design methodology called embodiment design, all about that, and when we combine all those things together, you have the world of my work.
[00:02:28.547] Kent Bye: Awesome. And maybe you could give a bit more context as to your background and your journey into this space.
[00:02:33.488] Brandon Powers: Sure. So my background emerges from the theater and dance world. I grew up starting dancing when I was in fourth grade, originally in hip hop, which became jazz, which then became classical modern dance, which then became contemporary, which then fused together with musical theater, and then found its way merging into immersive theater, which was a huge part of my background. And I was always really excited about technology, mostly as a kid who subscribed to Nintendo Power, and also just was watching every Apple Keynote all the time. And then found that I wanted to introduce it more into my work, and I was making a lot of performance about our experiences with technology. And slowly it started to meld more and more so that the technology was actually a part of the performances themselves in the way that they're being made. And I have worked with lots of different technologies from AR, interactive dance games, to virtual reality. movement rituals and narrative experiences and other AI experiences, one called Frankenstein AI that was at Sundance in 2018 that Kent and I talked about back then. You could listen to that episode. And that has led me to today where I'm kind of exploring all of these disciplines together.
[00:03:56.423] Kent Bye: Yeah, it's a bit of a callback there Frankenstein AI where that was a big team that was trying to explore the potentials of giving prompts at that point which were maybe I guess sentiment analysis from responses from an audience and then to translate that into Directions that then you would translate into a whole language that would direct some choreography. And so yeah this intersection between Artificial intelligence and embodiment and dance is something that you've been working on and thinking about for a while. I know last year you had a piece called Duet, which was a little bit more of trying to create these emergent social dynamics with people and virtual reality headsets. And now this year you have your latest piece. Maybe you could go through the evolution from Duet into what you're showing here this year at the Onyx Studios exhibition.
[00:04:40.253] Brandon Powers: Absolutely, yeah. I think as you named, there is a lot calling back from Frankenstein AI, especially in this new work called Kinetic Diffusion, which I created in collaboration with Aaron Santiago, who is also our lead collaborator for Duet. So for Duet, as Kent's mentioning, we were really focused on thinking about and reimagining our relationships with isolation and connection in VR. And so in that piece, two participants are wearing headsets and follow instructions through light and sound to dance together in VR while an audience watches on and experiences that ritual. And I think for this piece, this year, Kinetic Diffusion, which is a exploration of artist and AI in collaboration, we actually originally were calling it Duet AI, but we found that was too similar to just duet. But that was because we see it as more duets, right? It's a duet between me, the choreographer, and the AI, between me, the dancer, and the screens, between me and Aaron on the computer, between Aaron and the AI. Right? And so there's all of these triangular duets happening to make the piece possible. And we were ultimately taking a camera, which is pointed at the stage where you see me dancing, feeding those images into stable diffusion in real time, and playing it back at 30 frames a second. So you're essentially seeing like a flipbook quality video. And with that video, we're able to play with all sorts of different choreographic techniques of canon, of theme and variation, dancing in unison with the projections, and create a relationship between me and this new AI character that emerges.
[00:06:25.598] Kent Bye: Yeah, and I know here at Onyx Studios, there's been a number of people that have been looking at stable diffusion. There's another piece at Tribeca called In Search of Time, which John Fitzgerald was involved with helping give some feedback on some of the different stable diffusion aspects of that. But yeah, I'd love to hear how this generative AI come into play for you and how you started to first explore the potentials of what you might be able to do in your practice of choreography.
[00:06:50.711] Brandon Powers: Yeah, honestly, since Frankenstein, I've always been curious about remounting that piece. What's so wonderful about the team behind Frankenstein is that they believe in essentially copy left or like open sourcing the creativity behind the work. So where anyone who's a part of the project or anyone out there can really take it and run with it in different ways. And so when designing this piece, we were even considering remounting what we were learning from Frankenstein. And that was a lot of the bones. The idea of a dancer, a AI, and audience participation. And really thinking about that dramaturgical loop, if you will. And Aaron had been exploring with Stable Diffusion for the last several months. He's really deep in it, deep in all the forums and the discords with it. And so he's really deep in all the conversations about stable diffusion, and when we were trying to strike a new piece together, he was really wanting to see what would be possible, even with just the basic idea of showing people how prompts work, right? And we wanted to try to educate audiences on how advanced it is already, but provide a creative backbone to it. And movement is such a beautiful way of exploring space, especially with the AI, because you're able to really understand the amazing types of images and brings everything to life. And so that's where our exploration began. And we put the camera on, and it started at a very low frame rate. And we were kind of posing with it. And slowly, as things became more advanced, we found like, oh, let's push the speed. Let's push the frame rate. And we discovered a lot of these exciting moments of, what's possible when we start to see a lot more fluidity in the visuals.
[00:08:38.725] Kent Bye: Yeah, so I know that within stable diffusion, you can just give it a prompt or you can give it a reference image through control net and be able to then basically cut that out. And it's using that as a template to draw around that. And so I haven't necessarily seen other people do like as much as 30 frames per second. So that's basically like a lot of images that are being generated. And you're running that on a local machine, or maybe you're doing it on the cloud, but it seemed like it had a pretty low latency. And so can you talk a bit about If you're just running this on like a high-end graphics card that people can buy or if you had to do like Special like extra GPUs or what's the compute that you have to do to be able to drive this?
[00:09:16.302] Brandon Powers: For sure. So we are using about 10 4090s in the cloud to run it. So it's pretty compute heavy. We have one 4090 locally. And sometimes we could also run on like a 3090 cosair just to run the project. But it's really relying on the cloud compute. But it's pretty remarkable because the computers are all over the world. We're just renting them currently. And we're getting to about two second latency, which is pretty remarkable. That also has been brought down significantly from like a seven second latency over the last three months, and that's really Aaron's amazing work to drive that lower, because we needed to be a little bit closer to real time, but it didn't need to be fully. Leaning into the aesthetics of delay was really huge for us, and leaning into the limitations of what the platform could do was important. We wanted to amplify them, and actually say, OK, this is what the AI is giving us. Now, this is my creative reaction to it. I think that's the best way to move forward when collaborating, really, with any technology, but especially AI. But yeah, from the technical side, we have that heavy compute. And so the real determining factor, ultimately, actually, is the internet speed. So here at ONNX, we luckily have extremely fast internet. So the latency is quite low, almost as if the computers were present. If they were all present, we think we could get nearly real time, which would be pretty awesome.
[00:10:39.415] Kent Bye: Yeah, so this is actually a three projection experience. And you're talking about sometimes you're moving and it's being cut out into multiple screens. So like three different iterations of your movements that are also being split out. So you're basically talking about like 10 or 11 GPUs. and that you have 30 frames per second, then is it taking a frame and then taking this frame and then sending it off, and then the second frame and sending it off to the other GPU, and then how long does it take to get the frame and then render it out? Is it two seconds to basically have stable diffusion do a quick cut of whatever the prompt is to help guide the style?
[00:11:13.883] Brandon Powers: Essentially, yes, so what's happening and there's multiple different tricks that we're kind of doing in the background Aaron has built a Completely new custom software to run the piece which is really a remarkable AI plus Q lab essentially if anyone's out there is a theater person who understands how to cue theater and We can not just tell it to run, but we're actually every single beat of the piece, which is about 10 minutes, is precisely cued. It's counting to a metronome. I'm wearing a click track in my ear because we know exactly on certain beats when it's going to change the prompt, when it's going to record me, when it's going to go to delay of eight seconds, when it's going to go to delay of four seconds, when it's going to go back to eight, when it's going to count. There's a lot of precise movement happening under the hood. If anyone happens to have seen this piece by Andrew Schneider, who's an amazing artist called You Are Nowhere, I feel it kind of feels athletic in the way that his work does, and he wears click tracks along with his performances. So we have all those different capture modes, if you will, right? We have quote-unquote real-time, which is technically about a two-second delay. But then we could push the delay longer if we wanted to, which we do in the piece. And so there's this moment that people seem to really be enraptured with where I'm moving across the space on the prompt of like a beach, and it's an eight count delay. So what by the time I get to one end of the space, the AI just starts playing. So it looks like we're passing each other on the beach, right? That's intentionally making it more delayed, right? We also have something we call buffers, which essentially allow us to record a moment and then play it back. We can record an eight count phrase and then play it back immediately, which is really cool. We do that in the piece. And so that allows me to basically do an eight counts of choreography and then do it again. And then the new projection that was just visualized by Stable Diffusion is perfectly in rhythm behind me in real time. And now I'm dancing in unison, which is kind of act three of the piece, right? So with all these different techniques, we can kind of mix and match them for our creative effect. And then a little bit to your point, what the magic Aaron has made possible here is It renders the first count, and then while it's playing the rest of it, it's still rendering. So when it's at count two, it's still rendering count six, if that makes sense, right? So it's catching up to itself, but it knows just play it immediately and work in the background.
[00:13:46.312] Kent Bye: Wow, I am just really blown away with the precise timing of this because, you know, you start off with the metronome and then you count down and you're counting out loud and then you're performing and then, yeah, because I guess you have to time out all these different prompts and as they change as you go through this piece and As you were speaking about it, it reminded me of watching some of these beatbox loop station competitions where people are using their voice to loop. And so in some ways, you're looping some of these movements to then be remixed. One quick clarifying question, because sometimes with stable diffusion, you can render out an entire scene. Are you doing any compositing, like you're only rendering out the dance and then having the rest of the background already pre-rendered? Or is this something that is also rendering out the entirety of all the scenes that we're seeing in real time?
[00:14:32.475] Brandon Powers: Yeah, great question. First, to the point you were making, the easiest way I like to describe the piece to people is that we've essentially built a AI choreography loop pedal, right? That it is literally that. We are able to bank things. And there's even a part in the piece where we're recording while something else is playing on the screen, and you don't realize we're recording. And then later in the piece, during a section that we call Temple Guards, because it's weirdly inspired by Legend of the Hidden Temple in my brain. But it doesn't feel like that at all, just artist terminology. That is recorded early in the piece, and then we play it back at 10 times speed so it looks demented. And you do not realize that I have recorded that, but it was recorded live in the show. We could pre-record it, but we think it's much more exciting to do literally everything's done live. So that's the point to the loop pedal. Sorry, remind me.
[00:15:23.956] Kent Bye: The other question was whether or not there was compositing of whether you're doing the entire image as being prompted.
[00:15:29.952] Brandon Powers: Yes, yes, thank you. So it's a great question because we played with that. Currently we are not doing that, but there was a time where we explored it because we are pointing the camera at the stage and then we are projecting on the wall, right? And so we are seeing in the projection and in all the visual stable diffusion, we're seeing the room, right? And the piece actually begins with us melting from a true live feed into a kind of AI augmented live feed, right? And you see elements of the room, and those elements of the room shapeshift because of the prompt. So a lightning fixture becomes a mountain when we put in that prompt. At first, we actually really didn't like that, and we kept trying to get it to disappear and make it look like I was in a new place. And then, actually, we got to create something where Aaron essentially was compositing my depth map, because we're using a depth model for this. So it cut out my depth map, and then we trained the AI on another AI-generated image. And so it was putting me, essentially, photoshopped on top of another AI image, and it looked like I was in a painting, which is pretty cool. But it just didn't feel as live, actually. It almost looked too good. Like, the flicker effect we're used to seeing with a lot of AI went away. We even played with, in a rehearsal, artificially adding a flicker in TouchDesigner because people expect to see a flicker when they're watching an AI piece. And we're like, OK, we have this in our back pocket. But then we decided actually seeing the physical room and letting it just be exactly as it is makes it so much more grounded. And we never ended up using the compositing ever. We could potentially use that in a future version, but it was an interesting journey with that.
[00:17:17.860] Kent Bye: Your dance performance is broken down into three different acts, and so I'd love to have you just break down the names of those acts and the intent that you were trying to convey with each of those different movements.
[00:17:28.502] Brandon Powers: Sure, so the three-act structure is intended to mirror a creative process, right? A sense of discovery in act one, a sense of exploration in act two, which actually also has a little bit of loss of control to the AI, was a big part of the inspiration for that act. And then act three, this idea almost of mastery with the AI, And that kind of felt like what we are attempting to do in our development of the project. And so in the first act, you see me enter the space, discover the camera, start to play with it a little bit, learn, oh, it's delaying on me. Oh, here's this prompt of a Greek statue in a museum, kind of playing with this idea of like the most classic art, right, if you will, then emerges into some more sophisticated canons and choreographic concepts that are playing across the three screens, versus Act 2, where now that I kind of understand it, I play with this sense of delay on the beach that I was speaking to earlier, and a lot more flow versus the sharpness of Act 1. About three-quarters way through Act 2, we are playing with this idea of like, oh, maybe the AI starts to have more power than me, or playing into people's sense of AI fear with an acceleration that we're experiencing in our society right now. And we show that by the prompts actually melting from one to the next, where we've engineered the prompts to slowly degrade to each other as I move across the space, which creates this really exciting effect. And I mirror that melt in my body, so it almost feels like my body is now connected to the screen in some regard. And then it kind of surprises me with these temple guards, if you will, these crazy kind of glitched out bits of choreography appear on the screen. I stop it. We move into act three, which is the mastery where now I gesture towards the screens to turn them on. I am in perfect unison with them. I'm right on rhythm with them. And then it ends with this really precise freeze-frame section where I know exactly when it's about to take a single snapshot of my body and I freeze and it comes across the space and different screens and I hopefully essentially create the choreography in freeze-frames before this kind of final mega blur of all the prompts together. So that's a bit of the journey of the piece.
[00:19:53.255] Kent Bye: Yeah, it was really, really impressive, and I really enjoyed it. All the different technology coming together, but also just the impact of watching the choreography, the duet, as it were, as you become a dance partner to your own generations. And so, yeah, this kind of looping metaphor. Yeah, I'd love to hear a little bit more about the model curation because there's tons of different models you have for staple diffusion and so if you chose one model to go with the whole project or if you have multiple models that are swapping out because there's some aesthetic differences particularly in the third section where there was a lot of sci-fi more like expansive worlds whereas some of the earlier ones were a bit more of like impressionistic paintings and whatnot, but there's different styles and that could all come from the prompt, but it can also come from different models. So I'm curious to hear if that was just the single model or if it was a variation of different prompts to get that diversity of aesthetics.
[00:20:46.499] Brandon Powers: Yeah, so we are using one main model from Stable Diffusion. We very briefly use image-to-image in the very beginning of the piece, so that we get that melt from live feed, which looks like me, and we kind of always trick the audience into thinking it's just a regular camera, and then it becomes the AI view. And then we switch to depth model. But we are, throughout the rest of the piece, all using the same model and same data set. And that was a lot of error in just kind of exploring what's going to work best for bodies and movement. That's kind of why we landed on depth specifically, because shockingly, pose was so much worse. That's what we assumed. We were like, oh, we're going to use an Azure Connect with a pose, like obviously. And then it'll create these wild worlds with my movement, and it'll change my body. It looks terrible. So depth really brings it home and does all the things we want. And naturally, as stable diffusion evolves, and there's some new models for us to use there, it's going to start to look pretty different. And the piece could look the slightest change in the back end of our system wildly shifts the visuals. So the model certainly will change a lot.
[00:21:58.041] Kent Bye: Well, there's a certain amount of statistical difference with different seeds. Are you picking a single seed, or is it literally different every time you do it?
[00:22:05.805] Brandon Powers: Yeah, so we are using the same seed. It's one, two, three, four right now because there was a moment in our process where we played with that. Aaron was like, hey, you could change this button. And I almost lost my mind because just it's like, oh, that's too many options. Too many options, you know. There's a couple times where it didn't quite give us what we were looking for and we looked through the seeds for like literally only two minutes and we're like, okay, maybe that one's better. But there's just so many little things to change. So we decided to really specify what parameters are worth changing. So seed stayed, but however, obviously the prompts are changing. And we're playing with control weight, which essentially is telling the AI how much it should follow my body versus how it should dream on its own, essentially. And that is probably the main toggle we do play with in the piece, to show this almost lack of control. a little bit or it helps fade the room away and make it look like I'm in a new place. So there are a couple times where we use that. But if we started to change too many of the dials, we almost lost the plot of like what actually we're affecting because even Three days before the show, the lighting grid changed and everything looked different all of a sudden because of the shadows that it cast on my body and that affected the depth map. So then we needed to change the settings and the exposure of the camera to really dial it back in. So tiny, tiny, tiny bits make a big difference.
[00:23:37.981] Kent Bye: Yeah, it's sort of the liveness of the live of all these variations, that it is going to be subtly different. But yeah, I can see constraint to the seeds. For my playing with stable diffusion, you can get some widely different variations if you let it be a random seed. You mentioned a depth map. Are you using Azure Connect or Intel RealSense? Are you using a depth camera, or are you extrapolating that from just a 2D camera?
[00:24:01.956] Brandon Powers: Yeah, we're actually just extrapolating it from a 2D camera. We tried using a Azure Connect and it was much worse, actually, than using the mirrorless Sony camera for the performance, which is then doing kind of a, honestly, it's like an AI model depth map creator. So it's pulling from just a regular old feed and applying its own depth map and it works really well And then for the installation side of the piece which allows audiences to try out the technology for themselves That's just a really cheap webcam using that same depth map model. So That actually just works better than the fancy depth cameras. We'd be excited in the future to Play with multiple cameras in the space and what that's possible and feeding to different screens Or even with some of these parameters we're talking about right like we could imagine a whole piece Which is just similar to like act two where we're like on a beach for the majority of it We could do a whole piece now where it's just like oh We're on a beach and maybe just the seeds change and we're just using one prompt the whole performance, you know And then that's the point But knowing that we were changing prompts, we were like, OK, don't go too wild.
[00:25:13.028] Kent Bye: So you're taking a 2D camera, doing an AI model to extrapolate the depth map, and you're taking that depth map and inputting it into stable diffusion in a similar way that you can do an image into stable diffusion, or you can do control net for video and video frames. But you're just doing isolated images and then putting additional prompts on top of that then?
[00:25:33.874] Brandon Powers: Yeah, so that we are taking that feed and we're essentially playing it back at 30 frames a second, so it looks like a flipbook, right? So in a lot of ways, it's playing with the origins of film, you know? And that's why I love, like, a lot of the history we're playing with here of, like, we're at the precipice of a new medium and, like, let's actually lean into the fact that We are just playing a bunch of images really fast. And it creates this really nice, fluid style. Sometimes it's sharp. Sometimes it's fluid. And that becomes me as the choreographer. It becomes my challenge to then design movement that will do what we want the prompt to look like.
[00:26:13.296] Kent Bye: Great. So what's next with this piece? Are you going to try to take it to different film festivals? Are you going to try to get it into different galleries or performance spaces? So yeah, I'd love to hear if you're planning on exhibiting it more, or developing more, or what happens next.
[00:26:25.697] Brandon Powers: Yeah, all of the above. We're really, this is the very beginning of the project and we've learned a ton in our process and we've learned a ton in the past few days of presenting it with folks here, especially on the installation side where people are starting to play with it themselves. We think it's very modular and so it can work just as it currently is set up with this vertical TV panel with a person just stepping in front of the webcam and that almost, it's inspired very much like a photo booth you might see at a birthday party and you even get to record a video and it emails you directly so we're curious about kind of touring just that around and just letting people want to bring it to their events or it's a great way for people to learn about AI. Even this weekend, we've had some great conversations with people like, yeah, this is great for this conference, or this party, or this event that I'm having. So we're definitely curious about exploring that. Or larger scale versions, you could imagine a huge projection mapped space filled with cameras everywhere that's taking the prompts in real time. We're really excited about trying it outdoors in front of maybe a landmark building. because it's just a camera. So, you know, Onyx is across the street from Rockefeller Center, so if we set in front of the camera in front of the beautiful, iconic fountains there, you'd see the fountains morph and shapeshift with the prompts, which I think would be pretty stunning and help people understand what's happening there. And likewise with the performance. This is the very beginning of, you know, it's currently 10 minutes. We love what we've currently made, but we're excited to also explore a larger scale version with more performers, an evening length, something that really surprises people because we're so used to the idea of like sensors, right, in motion capture. They're like, are we going to max out the number of people? I'm like, no, as many people that fit in a camera frame can be tracked, right? It's just picking up movement. So a big stage with 10 people all doing together, I think would be pretty stunning. Those are just some of our ideas, but we're excited to hear more from other potential partners and put it out there, because it's the type of piece that I think we've found really just gets people to understand the ideas very quickly. And I think we're in a moment in society where this is the type of work that helps general audiences, like, get it, and also make it feel creative and fun. and not like scary, this thing's going to end our world. That was most important to us, that we make a piece that highlights why we need artists to be a part of the conversation and why AI can be a fantastic creative tool, because it does things we cannot, but then we should do the things that it cannot do, right? We should create the dramaturgy of the piece. We should create the story. We should create the movement. Shortly, it can put some input into that. And in future versions, we're excited about the AI maybe creating some movement with me. But at the end of the day, we really want to make sure that we find that perfect synergy together.
[00:29:25.307] Kent Bye: Are you saving all the images that get generated for each performance?
[00:29:28.716] Brandon Powers: We are not, actually, but we are saving from the installation. It records videos, so any of those videos we have sent to you and we'll have documentation of, which is great, but everything else is just cycling through because that's a massive, massive, massive, massive amount of videos, of photos, so we're not sharing those currently, but it would be fun to take those, that's an art project in itself, and like, put them on a wall, you know?
[00:29:54.673] Kent Bye: I was just thinking that if you were to capture and save those photos you could actually like recreate your performance in VR where you could just have those three walls and just playing the video but if you're able to take one or two cameras to do like a motion capture of you then you could be in a virtual space and then play it and rewind and pause it and go back and yeah it's such a rich visual experience that there's a lot of potential there that you could start to you know capture those performances but I think it works really well as a live performance, and I can understand why it's more of a femoral thing. But yeah, I mean, I think overall, it's a really, really impressive piece, and I really enjoyed watching it. And yeah, just one other point, as you were saying all that stuff that I wanted to say, is that probably the limiting factor is going to be the internet speed. And so this might be a use case for 5G, if you're getting the fast internet and the bandwidth of all the data that's being sent over. So there's not only speed and the low latency, but also to get it there in time. I'm just thinking about the film festivals in Venice, where internet's really, really bad. So it'd be very limiting to not have a fast internet connection, or you'll just have to bring all those GPUs on site to do everything locally. So yeah, just some thoughts there.
[00:31:02.002] Brandon Powers: TIMOTHY JORDAN-PECKERSEN Yeah, no, absolutely. The internet allows us to use the cloud PCs, which is fantastic. And it's pretty cool, even for some people who don't even understand how that side of it works. It's like, yeah, the computers are in Japan right now, but they're helping us. And if they were all on site, we'd near real time latency and we wouldn't be reliant on the internet mostly. So that's the other side. We're just aware that that would add about $20,000 to the budget. So versus like an example, like today running it for one day costs about 80 bucks. you know, which is actually pretty reasonable. If you were going to run it for a long period of time, we probably would invest in the PCs to have them on site. So we're not dealing with the marketplace, but for this purpose, it works really well. And it's amazing that we live in a world where internet and cloud computing works that well.
[00:31:53.540] Kent Bye: Awesome. Well, finally, what do you think the ultimate potential of this intersection of immersive media, dance, choreography, embodiment, generative AI, machine learning, all the fusion, all those things might be, and what it might be able to enable?
[00:32:09.421] Brandon Powers: There's so much potential, you know, as I was kind of speaking to you before, I think it's just so important that artists are a part of this conversation, and not just in terms of making the art, but I think artists should be a part of policy conversation, I think artists should be a part of any large commercial conversations that are happening about AI, because we are more than just content creators, we are dramaturgs, we are facilitators, we are people who can unpack things in new ways and present them to people to understand. And I think it's so, so important in our society that we empower artists to really use the breadth of their skill set in this moment, because we are making decisions about technology, both in AI, in XR, that greatly affect our bodies, that greatly affect our minds. And the decisions we make today are going to change the next 100 years. And I fully believe in that. And I think we're genuinely in a five-year window to make some 100-year decisions. And so I think we all need to be a part of that conversation. And artists, especially choreographers, we understand how to move bodies. That's like what we do, right? And we're talking about things that affect our bodies. So we can really help people understand how to make everything feel better. And that's what I love to do in XR. I run the program I do at Musical Theater Factory. And then love playing in this AI space as well because I think dance just activates people's creativity in a way other forms doesn't. The minute you stand in front of that camera and start waving your arms around, people just get it. You know, in a way that if they were to tell someone, like, perform a play, you know, I love theater and I make theater, but like, that's a harder in, actually. Like, if you just start moving, people just start to feel something. So I really just hope that that continues. And there's a lot of amazing other dance artists out there that have been playing with AI for a long time. And I'm hopeful that that all comes back and continues to grow.
[00:34:09.870] Kent Bye: I want to ask you one follow-up question, because you had mentioned after your performance that it's been a while since you've had an opportunity to dance some of your own choreography in front of a live performance, and that you've also been spending a lot of time connecting to communities around the world through TikTok. creating a presence there. So I'd love to hear any reflections on what it was like for you as a choreographer during the pandemic to find these new outlets of choreography and dance over TikTok. And as you move forward and fusing all these other AI and there's these augmented reality filters and whatnot. But yeah, I'd just love to hear any reflections because that's a whole other distribution for dance and choreography. It's been a huge part of TikTok. So I'd love to hear a little slice of your own experiences on that platform.
[00:34:50.047] Brandon Powers: Sure, yeah, we could talk for a whole hour about that. It's my favorite things to talk about in the world. But yeah, so I love to choreograph on other people. And so I do most of my work that way. And so this was my first time stepping into my own work in about 10 years. So that was very scary, but also really exciting. And we did that mostly because this duets concept we've been talking about, and it felt like it needed to be me, the co-creator, dancing with the AI, with Aaron at the computer, right? If it was removed and there was another dancer there, it just feels like it kind of loses a little bit of that spark of the concept we're going for. And yeah, but TikTok, I think, is a amazing place filled with amazing, rich, deep communities, obviously, especially for performance and especially for dance. And I like to describe it as kind of a modern day Homeric experience, like the Odyssey, right? And especially with dance, like one person creates something and the next person interprets it, maybe mashes it up with a different movement, which is a different song. that maybe goes viral. It then gets passed down to the next teenager who tries something, which gets passed down to the next person. And it's this amazing ancient tradition accelerated by technology, right? So I actually find that the communities we're making on TikTok are truly the most human things ever, actually. We just have this new tool that turns the odyssey, which maybe might have taken hundreds of years to get written. We can do it in an evening on TikTok. And I think that's a very vibrant community to be a part of, and I love it also as an educational tool, and that's the majority of what I focus on on TikTok, is helping people understand what choreography is and how it's made, particularly with musical theater. And it's a fun way to share work with a lot of people very quickly, right? And as a performance artist, I think the biggest challenge or kind of mind-blowing moment is like, Oh, I make a video in two minutes in my apartment and, you know, maybe it has seven million views. You know, I've had that experience before and I work on a piece like this for a year and maybe a hundred people have seen it ever, right? So you really start to think about scale and it's made me think as an artist a lot about scale and what I care about and reminded me or told me like, oh, actually, Brandon, like you care about a lot of people seeing your work, which is something that theater artists aren't typically used to. We're used to tiny rooms, right, with people. And not that I don't want to do that too, but like, oh yeah, we need to find a way for, if it's going to be a room with 25 people, why does it need to be 25 people? Because I could also make something that a million people are going to see the next week. Because I have that type of platform. So it makes me think very, very particularly about what stories I'm telling where, and what technology I'm using where, and why I'm making those decisions. And it's also a place to play around. But a lot of technological dramaturgy there.
[00:37:56.019] Kent Bye: So does that mean that we can expect some of the clips from the performances that you've been doing this weekend to show up on your TikTok page?
[00:38:01.659] Brandon Powers: Yeah, so I posted a little video of the process of making the piece on my TikTok. It's funny, because of algorithms, my TikTok is very focused on commercial Broadway musical theater, which is a whole other half of my body and life, because when I start to share this stuff, it doesn't compute with the algorithm, so I don't tend to do it very much. But I'm doing a little bit of teasing there, and I'm actually considering starting a new account that actually focuses specifically on the technology work, And you may also see a little secret account that I'm going to maybe start to play around with that is like an AI dance figure that is anonymous, that is playing into these types of visuals, but you won't know that it's a person behind the account. Because I'm curious about that sort of experience on TikTok.
[00:38:48.003] Kent Bye: All right. Well, maybe someday, someone in the future will listen to this podcast and figure it out. But yeah, is there anything else that's left unsaid you'd like to say to the broader Immersive community?
[00:38:57.462] Brandon Powers: I don't think so. I appreciate everyone listening. And unfortunately, Aaron Santiago, my collaborator, has a beautiful wedding to attend today. So he couldn't be here to impart some of the more technical knowledge. But I hope you'll check us out online and introduce you to more of the work.
[00:39:14.309] Kent Bye: Awesome. Well, Brandon, thanks so much for putting all this performance together. Like I said, it's a real feat technologically to see all this stuff come together real time happening. And just to hear a little bit more of what it took to do all that, it's really quite impressive. And yeah, excited to see where it all goes in the future. So thanks again for joining me on the podcast to help break it all down.
[00:39:31.571] Brandon Powers: Absolutely. Thank you so much.
[00:39:33.408] Kent Bye: So that was Brandon Powers. He's a creative director and choreographer who's creating experiences across physical and virtual spaces at the intersection of performance and technology. And he had a piece that was showing at the Onyx Studios summer exhibition called Kinetic Diffusion. So I have a number of different takeaways about this interview is that first of all, Well, it was super impressive to hear about all the different architecture they have on the back end to be able to run this. And so they have 11 GPUs in the cloud that's taking frames from a mirrorless camera that is extracting Brandon's pose by using a depth map model that is extracting his body. And then from that, they feed that into stable diffusion. And there's a very specific timing that he has with this piece. There's three different acts. And so he's going this whole journey and this arc of mirroring the creative process. And so there's discovery experimentation. And then at the end, he's got this whole like mastery of really getting everything down to very specific timing. So it's really quite a striking piece to be able to see the movements that he's doing the dance within itself is quite interesting, but he's dancing with a looped version. So it's like this beatbox looper, where he's creating and storing in all these different movements, and then feeding them back at different times. And so, yeah, quite a sophisticated method of using this kind of theatrical queuing and the idea of having these delays of two seconds or so for him to be able to Have this click track that's playing at the same time and everything's very much timed out so super impressive to hear all the stuff that they had to do be able to create that there was also a associated thing on the side which is basically like a big giant monitor and you go up and enter in a text prompt and It's like a mirror that's taking these real time shots of this webcam that's using the same type of depth model. So you're seeing the movement of yourself, but then you have all these different generative AI prompts that are on top of that. So it'd be great for like parties or just to introduce people to the idea of generative AI, because it's like an associated thing that's on the side that I think augments this deeper technology stack that they have here. And yeah, just super impressive to see that they're able to do this kind of real time. June of AI stuff and all the different settings and you know, it's very sensitive to different changes but to have the prompts and the certain seeds and all the settings to dial that all in and then each time it's different and very formal in that sense and yeah quite interesting also to hear his perspective on talking about this on these different social media channels and how he's found himself in a very narrow niche of musical theater stuff and so he's going to be expanding out into other accounts to be able to explore these creative technology explorations of this type of work that was really on the bleeding edge of these emergent technologies like generative AI and embodiment and this intersection between performance and technology. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. If you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a less-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
