Future Rights was a prototype at SXSW that aimed to bring bringing Stravinsky’s ballet The Rite of Spring into VR. They were showing a short demo at SXSW, but they have plans for a sort of dance autotune and collaborative performance that uses AI to help augment people as they dance along to live ballet performers. I had a chance to do a quick check in with the collaborators on this project, which is a co-production between choreographer Alexander Whitley, director Sandra Rodriguez to get a bit more context on this unique project and how they plan on continuing to develop it in the future.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to The Voices of VR Podcast. So continuing on my coverage of South by Southwest 2022, today's episode is a prototype project called Future Rites. So there were a number of different prototypes that were at South by Southwest this year. What that means is that it's basically like an early look at something that's not fully baked. There's still a lot of work that this has to come to see the full experience. And they were trying to get some feedback as to how far they're able to bring this as a project. So Future Rights is a piece by Sandra Rodriguez and Alexander under Whitley, and it's the right of spring. It's a famous ballet, and so they have ballet dancers that are within VR being captured within a perception neuron suit. It doesn't have full 6DoF tracking, and that's translated into one person dancing into a whole theater troupe dancing. And so you're able to be in these inversive environments and see this dancing happening around you, but also you're encouraged to also move around to do your own dancing in the context of these experiences. It's really interesting just to see how you're able to take live dance into a co-located environment, to have someone actually dancing ballet, and then to stream all that data into an immersive experience. So I had a chance to do a quick conversation with this team. I was doing interviews on the hour every hour and was able to have some extra time to be able to get a quick interview to cover this project called Future Rites. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Sandra and Alexander happened on Tuesday, March 15, 2022. So with that, let's go ahead and dive right in.
[00:01:45.552] Sandra Rodriguez: My name is Sandra Rodriguez. I'm a creative director for Immersive, XR, and now increasingly AI-led experiences.
[00:01:52.856] Alexander Whitley: I'm Alexander Whitley. I'm a choreographer and artistic director of Alexander Whitley Dance Company. And we make performance work across stage, VR, AR, and kind of all of the digital spaces in between.
[00:02:05.281] Kent Bye: Yeah, I'd love to hear a little bit more context of each of your background and your journey into this space.
[00:02:10.202] Sandra Rodriguez: So I've worked in virtual reality experiences where I really increasingly want people to use the affordance of having a body in virtual spaces and be that, not something just that makes them want to feel keywords that we've heard so many times like presence or immersion, but really feel like their body can be part of a story. Their body becomes a storytelling tool. And I've worked on AI-led experiences where Either we're questioning artificial intelligence or discussing its potential. In this case, it's really hidden way back in the background. The goal is to use the affordances of a new tool, this time artificial intelligence, to see how we can enhance our movements to match them to music, match them to a choreography language, and try to see where that leads us.
[00:02:55.883] Alexander Whitley: So my background is in dance as a performer and I started my own company about 10 years ago with a particular interest in working with digital technology and dance. And that really started out exploring how we could bring motion tracking into a stage context and establish relationships between performers and interactive visuals. And over the years that's really led in many different directions as technology has developed and interaction has become possible across different kinds of digital platforms. And I guess VR in particular really excites me because of how it's capable of tracking movement in three dimensions and then giving visual feedback to action in that three-dimensional space. So another thing that really interests me and excites me about the use of these technologies is how they can widen access to movement-based experiences or to the concepts that underlie dance and choreography, which is so often a kind of fear about or there's an inherent ambiguity about how ideas are communicated through the body as opposed to language. The signifiers are much less clear and for a lot of people that's a problem because they don't know how to engage with it or where to look in terms of understand what's happening in it. And then there's the problem of how you language movement-based experiences or talk about the kind of ideas that are at work in movement. And interactive technologies have the potential to really short-circuit that problem, to get around the issue of how we language an experience. Because you can have concepts baked into the interactive systems. so that you move, you act in a three-dimensional space and you have some form of mostly visual feedback as a result of that and it can suggest a course of action according to the kind of visual feedback you get. So, hopefully, you know, there's a way that by using these technologies we can draw the public and audiences really closer into the heart of dance experiences and make them participatory experiences rather than passive experiences that a more conventional stage setting would offer. So in this case, we're taking a classic work of dance and music over 100 years old and exploring how we can use these very modern tools to, I guess, bring fresh life to it and really open up the door for people to experience it in a really different way.
[00:05:21.855] Sandra Rodriguez: I think it's important to mention that this is a prototype, but I've been really, really floored by the reactions that we've received. And what I mean by these reactions, a couple of years ago when we would say we wanted to do a piece on VR and dance, and it was not this particular project. The feedback was like, well gamers want to do VR and it's all about games, so if you can gamify the dance then yes, they'll want it. But then Beat Saber became really, really popular. Beat Saber doesn't need you to know how to dance, it just needs you to know how to move to a beat and it's gamified. So, what about the other experiences? We have very eloquent experiences that can be shown in location-based venues, and we've seen that these work as well, very well, but they work with a different public, a public that's reinventing theatre, reinventing live performance. They want to see what's the future of seeing a dance performance live. But we strongly believe, when we first met, we strongly believe there's a connection between both. And my greatest takeaway from this is that I've never seen more very tall, large-shoulder men dance ballet. And they were attracted here not because they want to dance to the Rite of Spring or not because they want to dance ballet, but they see this not as a game either. They see this as an experience where their body becomes part of a world that they're creating. And there suddenly, I filmed so many of them, they become more and more elegant in their movement as the experience progresses. So one of the promises that we started first prototyping with the AI. So the AI is a keyword and in this case, as I mentioned before, it's really hidden in the back. It's not what we're testing here. We tested this in a first prototype with what we're calling the dance autotune system. So quite literally, we don't want people to shy off and say, well, there's a performance, I'm not going to dance as well as the performer. Or, well, if it doesn't have a beat, like the Rite of Spring is not a music with beats. How do I dance to a classic piece of music? So the experience is that, well, maybe it's just a flick of an arm. Maybe you're just raising both of your arms. But is there a way that we can make your avatar have hands that seem to extend a little bit more or pace themselves to the music a little bit more than they actually do? And as you're feeling like you're puppeteering this avatar, we're pulling a trick on you because we're puppeteering you into being more comfortable in dancing. So that was the goal of the first prototype. And our promise is we started creating this in the middle of confinement, so we started thinking that was going to be the goal of this experience, to test the limits of that dance autotune system. But we were very happy that we started to deconfine and again we could have a live performer and the value of a live performance really needed now to be tested. Could all this be pre-recorded? Do people dance differently if it's a live performer? So this is the goal of the second prototype. The prototype is split in two parts, the first part and the second part. And the second part really helps us test and evaluate how people are responding to the live performer. And so far, I've been really excited to see how much people move in that space. So I think kudos on the choreography as well, because they are in awe of the choreography, but at the same time, they feel like they are part of it. So I think we build that bridge.
[00:08:28.482] Kent Bye: Yeah, and a lot of these dance pieces that I've seen, there's this question of like, how do you interrogate the liveness of the live? And I feel like in this piece it's difficult to do that because of the lack of 6DoF tracking of the dancer, and also as me as a dancer kind of fixed, and like I have a mirror of my embodiment in the world, but it's also not moving around in a 6DoF way, and so there's ways in which that having things fixed in one location takes away a lot of the plausibility of it being a person. It feels a little bit more robotic in a way, because as I watch the dancer, it looks so elegant, but then when I go into the VR, a lot of that is lost when it's just fixated on one point. So I feel like getting a tracking system that allows the 6DOF movement, but also thinking about this ideas of how to establish a handshake of mirroring or convincing people that what's happening is a person. I mean, eventually maybe that could be AI and it's going to be like the Turing test of how do you really know if it's a person and they're responding to you. And like, there's stuff at least now that you can sort of tell what's more human-like and what's more robotic. And so I feel like that may be a part of it. But yeah, this interrogation of the liveness of alive, I think is going to be a part of what's going to make a piece like this really go from this prototype stage into the full release.
[00:09:38.798] Alexander Whitley: 100% yeah I mean it's it's obviously one of the frustrations at the moment in the relatively primitive or straightforward technical setup we're working with yeah there's real limitations in terms of the kinds of relationships we can establish between performer and participant but obviously the hope is in the longer term that we'll have a bigger more robust an accurate tracking system that will really enable us to firmly establish and work with those relationships. Because it is one of the strongest motivating factors behind making this kind of experience is the combination of feeling the proximity and presence of the performers around you, knowing that their performance is responsive to you. So there's that element of machine intelligence and responsiveness in the use of the AI and in all of the interactive possibilities of VR but also the very human responsiveness of the dancers around you and playing between those various possibilities hopefully will create a really compelling combination of elements and especially with a production like this which you know it's themes of pagan culture and rituals there's dances of the earth it's very kind of earthbound and grounded and a lot of things that both mocap and VR tend to kind of pull you away from you lose the sense of connection to the ground with motion capture and obviously things can look a bit kind of odd and robotic as you say so there's a lot of work to do to really kind of try and really work around those technical limitations and find creative ways of really making it as true to those themes that underlie the production.
[00:11:11.967] Sandra Rodriguez: We had the opportunity yesterday, speaking of creative solutions, to talk about the motion capture suits and how do you use live motion capture and VR experiences. And it's true that trying more accurate like OptiTrack suits, the data is much better, maybe the visuals are more realistic. But it's also very heavy on a system, it's really hard to use for something that's live, and hard to use for something that's live and that could only demand to be played in a Quest, and especially in an untethered headset, for instance. So, because we want all of these to happen, we decide, okay, well at least this system is a lot quicker, a lot faster, maybe not as accurate, but the creative solution may be in the costumes they're wearing. that some of the costumes don't need for the body movements to be very accurate. They look like big dolls or giant puppets. And that makes your brain as a user understand and accept that they move in that way. So we're really trying to push away from realism. And I think it's been asked a lot. The beauty of the movement can and needs to be realistic. And we thought, well, not necessarily. Dance is not about realism. It's about being part of a story through movement. I'm moving as I'm saying this. It's really hard to talk about dance and movement without dancing yourself. So it's about living the music in a different way and living it through your body. And for that, there are so many creative solutions to explore that are not about accuracy, realism. And I think that's the fun we've had in exploring this. We mentioned that it was a prototype, but we keep mentioning because, contrary to other prototypes, I've worked on other prototypes that are presented as prologues, for instance. They're a first chapter to a second chapter, and they're more finished. In this case, because we worked really during confinement too, we really worked in increments. We were getting small funds for a first prototype, we established the back-end. With the back-end we set up to get more funds for another prototype, we're now establishing the relationship with the dancers. But I think it helps create an experience that lives up to our own promise to each other, and that it's all about democratizing dance. So the more that we're sharing with the public, the more we're learning from their reaction to the avatar, their presence, the dancers, the live performers. And we're building as we go, but we're building with the help of how they move in that space and how they're interested in knowing if there's a live person with them or not. So I'm all ears when you're saying some of the feedbacks of what you think worked and didn't work, because that's why we're here. We're testing it out and hoping that it's going to help us build a third iteration. Rites of Spring is a 35-minute experience. We're now up to 11, so we'll see where that takes us.
[00:13:43.667] Kent Bye: Great. Well, I look forward to seeing where this goes and checking in. I'm sure I'll see you all again at some point and get more updates and be able to dive in a little bit deeper into the whole journey and process. But thanks for this kind of quick update. And I'm excited to see where this goes and the potential of the live theater and dance and this AI back end as well. So thank you.
[00:14:01.767] Sandra Rodriguez: Thank you so much. Thank you. Great talk.
[00:14:04.493] Kent Bye: So that was Sandra Rodriguez, a creative director for immersive XR and AI-led experiences, as well as Alexander Whitley, choreographer and artistic director of the Alexander Whitley Dance Company. So I have a number of different takeaways from this interview is that, first of all, so this is a project that has a lot of technological innovations to be able to do live streaming of this motion captured dancing using these sensors that are more 3DALF, meaning that you don't have pure 6DALF representation that's being projected into this experience. And I thought that was probably one of the things that I was wanting to see the most in terms of really feeling like there was another person there rather than what felt like something that was kind of puppeteered as a little bit more robotic. So, I don't know if that's a technological limitation of not having the next degree of motion track technologies to be able to do a full body tracking of some of these dances. But the ballet that I see as I'm watching it in physical reality is beautiful. And then it's not translated nearly to the same degree of what I see from the external. Of course, when you're in the immersive experience, you have this whole other scene design and these costumes that are immersive in their own right. But in terms of the immersive performance, I really want to see this dancer in a way that not only makes it feel like it's clearly a person, but also these aspects of the interrogation of the live moment and the liveness of live. There was an initial first phase that Sandra was talking about in terms of the autotune dance, and I would love to see a little bit more about that as this experience, because what does that mean when you see that virtual representation is how did that change the way that you move? So there's a lot of intriguing aspects there as a project. So I'm looking forward to see how this project continues to develop and like Sandra said it's still a prototype It's still early stages, and this is like a first look. This is a piece that actually as you go around it was pretty striking I saw it actually get written up quite a bit by a number of different journalists just because it was such an intriguing collaboration between Alexander Whitney's dance company in these ballet dancers with other people that are in VR and Stay tuned, looking forward to see how this project continues to unfold. So that's all I have for today, and I just wanted to thank you for listening to the Voices in VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a, this is a supported podcast and I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices in VR. Thanks for listening.
