Inarra Saarinen is the founder, artistic director and choreographer of Ballet Pixelle, does virtual dance performances in Second Life. She talks about the process of blending physical and virtual realities, and pushing the boundaries in creating a new form of dance.
It’s not just about replicating physical reality in a virtual world, but integrating all of the things that are impossible in the real world including hovering, flying, moving your limbs beyond a body-joint movement. becoming an object, animal or dragon, being able to change your skin color, gender, and age.
She talks about some of the limitations of only having 28 bones to work with in Second Life, and her process of scripting out segments of animation sequences, but allowing each dancer to be responsible for the timing of the execution while having the room to improvise.
Inarra doesn’t want to create an automated experience that’s the same every time, but rather capture the vibrancy and vitality that comes with the imperfections and character of live performances. She also talks about how a lot of the participants aren’t physically able to be in a professional dance troupe, and that by participating in Ballet Pixelle the they’re able to have a kinesthetic experience of feeling like they’re performing dance on stage.
It’s interesting to hear all of the insights that Inarra has from doing Ballet Pixelle since 2006, and I imagine that the blending of physical and virtual during live performances will be an area of rich exploration over the next decade. From the VR community perspective, the Riftmax Theater’s Karaoke night starts to explore this blending of realities during live performances, and it’s a bit of an open question as far as what will be considered the most compelling and beautiful experiences within this new spectrum of mixed realities.
TOPICS
- 0:00 – Intro. Founder, artistic director and choreographer of Ballet Pixelle, does virtual dance performances in Second Life.
- 0:40 – How is movement controlled? Creates animations and chunks of movement in scripts and puts them into Second Life where each dancer is in control of their avatar’s performance.
- 1:28 – Create animations in 3 ways. Individual keyframe per 30 fps and import into Second Life. Also uses a motion capture suit, but Second Life only allows 28 out of 206 bones. Also using a Kinect system to put the animations in a coherent sequence.
- 2:31 – How does the dance trouble keep in sync. They keep the beat like another and are in charge of triggering the actions with their keyboard & mouse.
- 3:03 – What’s been the reaction? Lots of powerful emotional reactions.
- 3:35 – What is the audience connecting to? It’s a combination of telling story with set, lighting and movement. Movement is a universal language, and if you put together correctly, then you get an emotional resonance.
- 3:57 – Sleep No More dance performance of MacBeth. Any dialog? There’s a playbill that tells the story of the ballet just like you would in any other live performance. The story should tell itself, but there’s a bit of help provided
- 4:42 – What’s motivating your performers in your dance troupe? Had people in the troupe since 2006. They really get the kinesthetic experience of performing, and a lot of them have physical or other limitations where they’ve never been able to do that be for. They feel like they’re on stage and giving a dance performance.
- 5:45 – The human synchronization and not being driven by a robot. A movie is the same every time, but live theater is not the same every time. Trying to create an experience that’s vibrant. It’s art, not automation. She wants those human imperfections. Choreograph ballets that allows them to deliberately go out of sync and to make order out of chaos.
- 6:54 – Are there auditions? Lots of things are different. Transform, hover, fly and move beyond body limits. But lots of similarities and universals of working with other people. Some of the things they look for.
- 7:44 – Coordinating across many different time zones for live performances. You can teleport in Second Life, but you still have time zones. Have both a European and North American dance troupe. But it can be difficult.
- 8:35 – Other considerations for broadcasting music and clearing rights. Been very copyright sensitive from the very beginning. Made sure that everything is copyright cleared, and have clearance from everyone involved.
- 9:42 – Right for each performance and image release for avatars
- 10:18 – What keeps you engaged? It’s creative and at outer bounds of being creative. It’s a new form of dance. It’s not just adding something. It’s an exploration of physical and virtual movement and blending of realities, which is a different form. What do we find beautiful about virtual dance? Developing a language for virtual dance.
- 11:34 – Things you can do virtual dance: Hover, fly, move beyond a body-joint movement. Become an object, animal or dragon. Change your skin color, gender, Become a child.
- 12:00 – Use all of these components in all of her ballets.
- 12:17 – Pushing limits of what’s physical possible and expanding audience for dance. Gives dancers a chance to experience performances. Teach history of ballet and technique.
- 13:04 – Immersive VR with the Oculus Rift, and future of limb tracking with dancing in VR. Not as interested in translating your movements into the virtual world, because the animations are doing things that you couldn’t be doing. Not interested in replicated the real world, and can’t go out and hire real professional dancers
- 14:36 – Ultimate potential for virtual environments. We’ll eventually live in virtual worlds.
- 15:03 – Next open problem to solve with virtual dance. Limited by the 28 bones that are allowed by Second Life out of the 206 bones. On a world audition tour to do choreographic studies and do motion capture of dancers to study the movement of professional dancers.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.937] Inarra Saarinen: My name is Inara Saarinen, and I'm the founder, artistic director, and choreographer of Ballet Pixel, which is a company that performs all original ballets in the virtual world of Second Life. And we have dancers who dance through avatars all over the world. They're located in Portugal, Spain, United Kingdom, Japan, Australia, all over. And they actually dance with each other and the music.
[00:00:44.066] Kent Bye: And so how is somebody's avatar connected to their physical motions in the real world? How do you make that one-to-one connection?
[00:00:51.658] Inarra Saarinen: Well, actually what happens is I have a couple of different ways to create animations, and I do make them all myself. And then I import them into Second Life, and I put them into little chunks of movement. Each animation, say, is a step, and then a small group would be a dance or a musical phrase. And we have a naming system for each character or each dancer. and a sequence, and then the dancers move their avatars on their own and execute the animations in time with each other and the music.
[00:01:27.656] Kent Bye: I see. And so it's something where the actual movement of the limbs are scripted out, but the sequence of them are controlled by each individual. And how are they actually controlling going from one step to the next?
[00:01:40.593] Inarra Saarinen: Good question. First, you should know that I create animations in three different ways. One is I do keyframe animations, which means 30 frames a second, and I'm actually manipulating a skeleton for every frame to change, change, change. rotation or degrees of turnout or something like that. And then I import them into Second Life. Another way is I use a motion capture suit, but the problem is there's a lot of limitations in Second Life. Out of 206 bones in the body, 28 are allowed into Second Life, for example. And now I'm using a Kinect system and doing motion capture that way, and post-processing it, bringing it in, and yes, putting the animations in a little coherent sequence, but many sequences in the ballet.
[00:02:33.618] Kent Bye: And so each individual, when they're performing, like how are they knowing when to do each step?
[00:02:40.254] Inarra Saarinen: very much like real life. So they're counting musical beats and they're watching their other dancer. So five, six, seven, start. So they're using keys on the computer or a mouse and moving their body with the keys going forward, back, side, wherever on the stage. And then also starting that animation or that sequence.
[00:03:03.486] Kent Bye: I see. And what has been the reaction to these performances then?
[00:03:08.170] Inarra Saarinen: I've actually been very surprised because people have told me they cry. People have come multiple times. I have somebody who's come 10 times to see Windows and says she always cries at the end. I have a Japanese story, Shuzenji, that I wrote about lost love and suicide. Very Japanese. I live in Tokyo. And people are just emotionally part of the performance.
[00:03:35.262] Kent Bye: What are they connecting with? What part of this performance are they connecting to?
[00:03:40.004] Inarra Saarinen: Well, I think it's very much like in real life. It's the combination of telling the story with the set, the costumes, lighting, movement. And movement is a universal language. So if you put it all together right, you get an emotional resonance.
[00:03:55.788] Kent Bye: I say, yeah, I guess I've seen Sleep No More, which was an immersive theater, and it's all done through dance, and they're acting out Macbeth. And so there is a certain power to having, I guess, nonverbal movements to kind of express these feelings. So in your performances, is there any dialogue or anything, or is it all done through the movement?
[00:04:16.088] Inarra Saarinen: Well, when you sit down in a seat, a playbill is automatically delivered to you. So if it's a story ballet rather than just a movement neoclassical ballet, you will get part of the story in there. We also have a master of ceremonies who will tell you a little bit about each act. But really, the story should tell itself. So the ballet should stand on its own. But we give you a little help just as you would in the real world.
[00:04:44.160] Kent Bye: I see. And talking to some of the performers that are collaborating with you, what is it that is motivating them and driving them to participate in this type of performance?
[00:04:54.186] Inarra Saarinen: Yeah, it's actually quite amazing to me because we started in about 2005, 2006. We're the only company that does this and I have people that have been with me since 2006 and I'm just kind of flabbergasted that they would hang out with me that long. But what they say is that they really get the kinesthetic experience of performing. and some of the dancers may be physically disabled, some are retired, some have been hurt, some never could afford to take dance classes, all kinds of different dancers, some are pros currently, but with different backgrounds, and they're able to really get on stage, really perform, they're really dancing, they're not scripted robots, and they get an emotional feeling of what it's like to be on stage and dance.
[00:05:46.124] Kent Bye: Yeah, and I think one thing that I would imagine would be powerful is that people are going through these sequences, but they're not going to be completely in sync as if they were scripted and sort of orchestrated. And so what do you think it is about that component of knowing that there's a human that is behind it that's controlling it versus something that's being driven by a robot?
[00:06:08.398] Inarra Saarinen: Okay, if you see a movie, it's going to be the same every time. If you go see live theater, it's not the same every time. I'm trying to create an experience that is vibrant. I don't want robots. It's art, not automation. So I want those little discrepancies. Some people follow a specific dancer. They think they're more musical than another dancer. And I've also used the fact that they can't always be in sync by developing ballets, choreographing ballets that allows them to kind of float in and out of synchrony and making a statement about order out of chaos.
[00:06:53.495] Kent Bye: Ah, interesting. And do you have auditions and tryouts for these performances then?
[00:06:58.118] Inarra Saarinen: Yes, there's a lot of things that are different dancing in virtual reality. You can transform, hover, fly, work beyond body limits, and so forth. But there's a lot of things that are like real life. I have a lot of drama with my dancers, they're real people. It takes time, rehearsal takes time, and we have auditions, like real life, and we put out a call for dancers all over, and people come and have to follow directions, organize their avatar in space, we see how quick they are, if they know stage terminology, et cetera, et cetera, we interview them, and then we choose them, so like real life.
[00:07:39.780] Kent Bye: Wow, and I guess there's another additional component of they have to actually show up at a specific place and time to give a live performance. And so what is that like in terms of coordinating with people who may be in a variety of number of different time zones?
[00:07:52.786] Inarra Saarinen: Yeah, that's the thing we haven't gotten beyond. You can teleport in Second Life, but you still got time zones for all those humans at their computers. And we often have a Euro company and a North America company so that we try to provide for the audiences in those places a good friendly time for them. And then we also cast Euro dancers and North American dancers so the time is good for them. But it's tough. It's tough, especially for special presentations like today where we can't control the time. So I have people at 3 a.m., people at 8 a.m., people at 5 p.m.
[00:08:33.493] Kent Bye: Just one of those things. And what are some of the considerations that you have to take into account when you're broadcasting music and getting rights to everything? What type of things do you have to do in order to make sure that performance is cleared?
[00:08:48.682] Inarra Saarinen: From the very beginning, I've been very copyright sensitive, partly through work that I've done in computer networking, actually, and domain names and so forth. And I've made very sure that every single thing we use is copyright cleared, and we have some kind of writing okay from each person. Since I create all the animations and do all the choreography, that's clear. Any textures, any pictures, any personal pictures, any historical footage, we use a lot of historical video, historical photos, all of that stuff is cleared. Music is usually composed specifically for the ballet in collaboration with me. That's also cleared way ahead of time, put in an agreement. If not, it's all Creative Commons Attribution 3.0.
[00:09:41.372] Kent Bye: And are the performances by each avatar, is that something that's copy written and they have to give over to you to be able to use in the performance? Yes. Wow. So even though you wrote the scripts, it's someone that's actually pushing the button. That performance then becomes an intellectual property of theirs.
[00:09:58.361] Inarra Saarinen: Well, I'm not totally sure. I'm not a lawyer, so I won't play one on the internet, as my husband says. But I do get permission from them for Machinima to do video recordings in the virtual world. Will I be able to keep this? Will I be able to continue to present it? And they've all agreed to that, of course.
[00:10:18.715] Kent Bye: I see. And what is it about this project that keeps you excited or engaged in continuing it?
[00:10:26.021] Inarra Saarinen: I'm very passionate about it. And it's mostly, well, it's first of all very creative. It's at the outer boundaries of being creative. I think that I'm most passionate about the fact that I think is finally a new form of dance. That it's a kind of quantum leap for dance. It's not just adding something. It's really an exploration of physical and virtual movement and it's a blend of realities. and it's multimedia as well. Multimedia, everybody's done in dance. But the blending of the realities and the blending of the movement, I think, is actually different form. And I'm trying to define the aesthetic of that. Is it a different aesthetic? What do we find beautiful in dance done in a virtual world? Is it a different shape? What if it's past a body joint limit? Is that still pretty or is that contorted? I'm also trying to develop a language for it, like lebonotation, which is a graphical language to record dance, trying to expand that language for virtual dance.
[00:11:32.964] Kent Bye: Are there things that you can do in virtual dance that you can't do in physical 3D reality dance?
[00:11:39.841] Inarra Saarinen: Well, lots. You can hover. You can fly. You can, as I said, move beyond a body joint limit where you wouldn't put your contortionist skills to work. You can become an object, an animal, a dragon. You can change your skin color, your gender, become a child. So many things.
[00:12:01.447] Kent Bye: And have you explored using those elements within your performances?
[00:12:04.947] Inarra Saarinen: yes, they're within all of the ballets. And then even to take it kind of one step further and back, I've also done straight neoclassical ballets as well.
[00:12:16.376] Kent Bye: I see. So you're really taking advantage of the virtual reality environment to be able to push the limits of what's physically possible, it sounds like.
[00:12:23.937] Inarra Saarinen: Exactly. And we're also trying to use that to develop a new audience for dance. We get some young savvy people actually watching a ballet because there's dragons in it. Maybe they'll actually go see a ballet in real life. It gives dancers a chance who couldn't experience it before. It gives audience a chance who couldn't experience it before. Shut-ins, agoraphobics, people in hospitals, and so forth. Lots of stuff about it, and it's also educational. We teach history of ballet, and we teach ballet technique, because you can look at the perfect situation, perfect animation, you know, how this step should be done, how this turn should be done.
[00:13:04.124] Kent Bye: And with the Oculus Rift head-mounted display, being able to bring virtual reality within Second Life, have you thought about ways to, for both the performers, have virtual reality headsets on that are being tracked at their heads, but also limb tracking through Kinect or Leap Motion? I'm just curious about what you see as the next iteration in terms of technology of what may be possible in terms of doing different kinesthetic movements within virtual reality environments like Second Life.
[00:13:34.533] Inarra Saarinen: I haven't tried the Oculus yet. I have an appointment to do it tomorrow. I will probably get a developer's kit and we'll see about that. I'm not as interested in your actual movements coming directly into Second Life. And that is because I'm creating these animations that are doing things that you couldn't be doing. So even if you had a connect or a way to bring your exact body movements into Second Life, that would be you dancing. And you may not be a great dancer. And I don't want to replicate the real world. We're not doing Swan Lake. If I had to go hire professional dancers to all put on oculus and suits and Wii remotes and so forth, then I'd have a real life ballet company.
[00:14:25.195] Kent Bye: I see. And you're more interested in sort of pushing the edge in terms of what's possible. It's creating a new language of dance, it sounds like. Exactly. That's what it is. Great. And what do you see as the ultimate potential for interacting with virtual environments with Second Life?
[00:14:42.938] Inarra Saarinen: Ooh, kind of a tough one. I do think we'll live in virtual environments eventually. So I think that will become our world. I don't think it's Second Life. Sorry, Linden Lab. But I think we will be there. And I don't know. It's a little hard to say. That's about where I would go. OK.
[00:15:03.343] Kent Bye: And finally, what's sort of the next big open problem that you're working on with this dancing within virtual worlds?
[00:15:09.897] Inarra Saarinen: I guess there would be two things. One is that even when I do a motion capture with hundreds of points on the body, all 206 bones that we have, all of that sifts in through a funnel of Second Life that only allows 28 bones and very little other adaptability. So hands don't go in, feet don't go in. So I'd like Linden Lab to listen to me for one thing so we can do better work. And the second is we are now on a world audition tour for real-life dancers to do choreographic studies so I can work on real dancers and also to do motion capture. So we'll be in Taipei, Tokyo, Bangkok. We were just in Africa. We'll be in Hawaii. We'll be in New York City, Toronto next month. We hope to see everybody there.
[00:16:10.223] Kent Bye: Great. Well, thanks so much.
[00:16:11.605] Inarra Saarinen: Thank you.
