I speak with the four creators of Dazzle: A Re-Assembly of Bodies experience, which includes choreographer and dancer Ruth Gibson and visual artist Bruno Martelli of Gibson/Martelli, as well as Bine Roth & Alexa Pollmann of Peut-Porter. This dance performance showed at Venice Immersive 2022, and fused together aspects of dance, fashion, and immersive performance.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that's looking at the structures and forms of immersive storytelling and the future of spatial computing. You can support me on Patreon at patreon.com slash Voices of VR. So continuing on my series of Venice Immersive 2022, today's episode is with Dazzle. So Dazzle is a dance piece that had three different sections. There was a section for people to be immersed within VR with the motion capture suit and wearing all of this mocap gear that had this kind of fashion twist to it. And then there were people who were in VR but not co-located with the motion capture dancers. And then there was people who were watching from the sidelines on a 2D screen and seeing the dancers and the different audience members in VR and seeing it all projected onto a 2D screen. So Dazzle is a collaboration between four different creators of two different companies of Gibson Martelli so Ruth Gibson who's a choreographer and Bruno Martelli who's a visual artist and then Bino Roth and Alex Pullman who is a part of Pouture who does a lot of fashion dance performance and technology so They were looking at this concept of camouflage and the opposite of camouflage, which is this dazzle. You know, as you go in into this piece, everybody wears the same piece of fashion. And so you have this consistency in terms of this social group all wearing the same fashion. And then as you go into VR, then it's a world that has a lot of black and white. And in the actual experience, you have dancers that are dancing around and then they're being replicated in different ways. And you go through three or four different scenes. Yeah, just a really interesting way of using the immersive technology to create this sense of immersion of another time and place, but also just kind of playing with different aesthetics of choreography and dance when you start to replicate people. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Ruth, Bruno, Bina, and Alexa happened on Friday, September 2nd, 2022. So with that, let's go ahead and dive right in.
[00:02:15.888] Bruno Martelli: Hi, my name is Bruno Martelli. I'm a visual artist and I've been working in immersive for a few years and usually with performance and motion capture.
[00:02:23.055] Bine Roth: Hi, my name is Bine Roth and I'm part of the second collective basically called Puprotier and we work within fashion, dance, performance slash experience, also immersive experiences.
[00:02:38.115] Alexa Pollmann: and my name is Alexa Polman. I am also part of the Pupporté team and I think it's quite good to say that Ruth and Bruno are one studio and Pupporté is the other one so they are Gibson Martelli and Pupporté and we found each other around three years ago because we share the interest in working with performance and participation as well as fashion and well obviously technology and especially in this case live motion capture and VR and our piece has come together because we all have different specialisms but at the same time we are quite good in cross collaborating or we enjoy collaboration so I think that's the central element of what brought us together.
[00:03:23.748] Ruth Gibson: Hi, I'm Ruth Gibson, and I'm a choreographer and dancer, and I work with Bruno Martelli, as Alexis just said, and we have an interest also in costume, and always have done, in making interactive works for both stage, gallery, public space, and at home as well, projects that you can actually play or have at home and interact with, and we've been doing that for quite some time.
[00:03:53.740] Kent Bye: Yeah, we've got a broad mix of different backgrounds as your journey into creating this piece of Dazzle that's premiering here at Venice Immersive 2022. So maybe you could give a bit more context to each of your backgrounds and your journey into creating this piece of Dazzle.
[00:04:08.897] Bruno Martelli: How long would you like me to talk for? I don't know, personally I've been interested in camouflage for a long time and the idea of concealment and revealing and hiding and dazzlers type of camouflage which is different to natural camouflages because instead of being a microcosm of a landscape where a landscape is reduced to its essential elements and then applied to in a military context obviously we think about applied to like tanks or something to blend in with the desert or if it's just like a leaf insect where it has the shape of a leaf to avoid predators. This is a kind of camouflage which is used for concealment and Dazzle is the opposite of that and it's a kind of machine age form of camouflage rather than being natural in that it was designed to confuse rather than conceal. It's not about hiding, it's actually about revealing and part of my interest is in the making process, in concealing and revealing the making process as well and so the Dazzle camouflage and the process we have has come together nicely.
[00:05:07.345] Kent Bye: And just to orient on your own personal background and journey, what's your background coming into this project or your expertise that you're contributing to the project?
[00:05:15.127] Bruno Martelli: Oh, well, I'm a visual artist. I've been working with immersive media, but basically beginning to do live effects on stage and then thinking about trying to take the audience on a journey where the stage space dissolves and they get absorbed by it. And so immersive is a kind of natural extension of that, I suppose. Yeah, and really thinking about, without being down on theatre, but trying to escape from the idea of the confines of the theatre and really one of the things I've been interested in is cinema and theatre present a kind of proscenium arch space where the audience is over here and the action that's happening is actually is over there and I kind of want to dissolve that a little bit. And so working with something like computer game engines, the audience becomes, in some ways, a protagonist or an active participant. And they have some sort of agency. Even though the world, like a computer game world, is effectively created by the creators, you, if you're playing a computer game or in an immersive environment, you're actually creating your journey, and you choose what you can look at. You have agency over your points of view, all those kind of things. And this is part of that. Awesome.
[00:06:16.842] Kent Bye: Yeah. And for each of you, just share your background and journey working on this piece.
[00:06:21.342] Bine Roth: Yeah, so, well, originally sort of coming from a maker's background, which is like jewellery and textiles. And after the graduation, Alexa and myself started then Puperty together. And I guess this is sort of the journey of, well, definitely for me, just my personal journey where it sort of started going to be a bit more broad, not only looking at, like, let's say, like jewellery or the actual kind of artefacts, as in something that just sort of like jewels someone or like adorns someone, but more looking at them also from a tech side view as well, and how they can maybe extend the body or distort the body, which then comes also into camouflaging in a way, like you're confusing the body maybe as well. So I guess that's where this sort of started, becoming into this experience, and then using different technology to support it. Yeah and I guess you can see obviously in this experience now that everything comes together and it's not just one person doing one thing and creating something but there is obviously specialisms in the making that comes from my perspective like metal making that comes in or little bits and bobs but then there's always the collective now as the four of us deciding on the things that come into the experience.
[00:07:33.321] Alexa Pollmann: And I think what is quite nice in working with materials in this world is that you manage to move away from this gadget male-dominated space a bit as well. And I think the experience that we hope to achieve for our audiences is really one that they are familiar with in some way from their everyday. So when we work with fashion, and my background is originally fashion and then interaction design, because I think the two are linked anyhow and it's kind of a natural pairing. But what happens when you come into this world of ours, you're really invited in because you also can dress yourself in one of the outfits that we created together with 15 nationalities from all around the world. And I think the main aspect for us or for me personally is that my work hopefully sparks within people that they can become part without having a high level of entry. And I think that's one of the cruxes in a way of technology sometimes. And I think if you allow people to have the opportunity to choose how far they want to go in or give them different points that they can start engaging with, it really helps to open it up and at the same time make it more familiar to everyone. So I think the fashion aspect in this piece was originally really inspired by the dazzle ball that was 1919. And the dazzle pattern and also myself, I'm very interested in patterns and how they were designed for human eye in terms of camouflage and if you look at the normal woodland pattern for example. and all of these artifacts that are within our everyday now that we have no use for in some way anymore because now drones see us differently. The machine has a different way of watching humans, right? And I think that this play between those two worlds is constantly in the piece. as well and there's a lot within the main aspect of the dazzle ball from 1919 where people made their own outfits to come and the pattern came from the First World War where the ships were painted as Bruno explained earlier to conceal but at the same time actually they become more beautiful and I think that kind of play of artistic elements within this very tech heavy world is something that I think really stands out for me in the piece.
[00:10:00.647] Ruth Gibson: As I said before, I'm a choreographer. I'm super interested in audiences understanding the body unwittingly through experiences. So during the project what's been really lovely is to see that and to see the fact that actually people can let go and be less self-conscious but at the same time know that they're in this other world but in the real world. So improvisation is key, a key method that I work with with the dancers. All the time this changes because of the personalities and different people that come to the project. So it's evolving and also that's something interesting for me is that this doesn't stay still, it's not static, it's not rigid. So there's a fluid aspect to this and both the dancers are learning as well. So there's a semi-structure to it obviously because you're wrangling a lot of people and organizing a lot of people in space but there's also a freedom and I think the structure and the freedom and the layers of the project are really showing themselves now and the light and dark and the fact that there are touch elements in it which of course we haven't had in the past two years because of mentioning no names, everyone knows we've been isolated and so there is an aspect of that which is a bit of a giveaway but I think some other projects in this exhibition have come to see it and that's what they're seeing and although it might seem obvious it's something that's really getting under the skin of people and they keep coming back to have a look at that and what magic that is, and this connection somehow that might have been lost. So there's a layering, there's like the dark side, the light side, and it's flipping back and forth continually. And it's how people intelligently, physically engage with all the elements of the project through the costume, music, vibration, and obviously visuals.
[00:12:04.060] Kent Bye: Yeah, and as I'm sitting in this installation, there's a lot of contrast between white and black, and that's the two colors that you have throughout the course of this piece, and I'm really struck by how you started with the concept of camouflage, and the entire world design is that black and white, but then the characters that are in that kind of stand out, given the contrast of the patterns that you have, this dazzle pattern from 1919. And so there's a lot of layers to this project in terms of there's an immersive world, there's an installation, there's fashion, there's the art that you're actually putting on in terms of the motion capture that's crafted metals that's unique and different, taking a fashion sense to the motion capture technology itself. So there's a whole layers of the fashion, but there's also the world design and the choreography. So I'm curious as this project is coming together, how you start to build out one section and then play off these other areas to build up to what we have here at Venice. But did you begin with the world design? Did you begin with the dance? Did you begin with the fashion, the costume? I'm just curious, where do you begin on a project like this across all these different collaborators?
[00:13:07.135] Bine Roth: That's quite a hard one. I can't really remember. But I guess, well, obviously we had the pandemic sort of there to develop different things at the same time and also not really having the time to test it maybe that much. But I guess like all these elements sort of came actually at once. I don't think that one was developed and then we sort of picked another one. It was more like it evolved all together. there isn't really like a specific thing that we said like it was dazzle obviously in the beginning but I wouldn't say like there was like a choreography at first and then the choreography we built on that rather than someone comes in with something and then a reaction sort of happens and that reaction comes to another thing it's like not like a domino effect because that would be too linear I don't think it's that linear it's sort of like a bit broad.
[00:13:54.340] Ruth Gibson: We really get on and You know, there's a respect for the practices of each other's practice, so we know who is boss of each of those aspects, which I think is really clear. And I think probably what brought us together was the history. And the history is just so extraordinary, because of course it's just this library of things we can call on from that period of time. which has similarities 100 odd years later. It's almost like replicating itself, you know. And there's a humanity in the sort of technical progression. And I suppose in some ways there are comments also on the duality of the avatar and the real and the real human aspect of it. So these things keep being picked up just as probably the futurists and some of the incredible choreographers at the time working in the 1990s, sort of 20s, with costume and staging and revealing and hiding and all these things that weave into the concepts as well as the historical aspects. So it's a big source for us and I think we don't stop because we get quite obsessed. We're still doing it now, you know, we're still doing, oh that looks a bit like that or if you add a bit of that. But also colour too, because I mean the girls might talk about aspects of sustainability and I think you will see more and more people wearing black and white because people won't want to use all the energy that it takes to dye clothes, but I'm sure you can say more about that. The worst one, actually. Or to keep, or just to keep, you know. No, but I think it's... Simple things, simple clothes that last a long time. I think we might see that more.
[00:15:29.665] Alexa Pollmann: There's a very strong, I think, in the four of us, as you said, rightly so, I think there's a very strong synergy in the way we also worked before and the interests we have, so the artistic identities seem to just gel really well. So in that sense I think the starting points started much earlier I think with all our works being sort of related to similar ways of working. But what you picked up on I think as well in terms of the idea that our audience is really key. We really want them to be able to bring something to the piece and so Quite early on, and partly through the pandemic, as Bina said, we invited artists from around the world to react. So all the garments are co-created with different designers. And the story of that, again, goes back to the history of the project as well. It goes to Taillat, who was an Italian futurist, and he invented what is now the tuta suit. Tutta is an Italian word for all and he said there's four principles to the suit so it's per tutta la gente, per tutte le occasione, per tutta la corpa e usare tutta la stoffa. So it's for all the people, for all time, covering the whole body and using all the material. those were the principles that we used as well and I think they go throughout the piece. And he released the pattern of that tuta suit in the newspaper, open source basically, so that people could make it themselves. And so Flavia Loscalpo calls the piece the most utopian piece of clothing and I think that's in our shared mind as well. And it was 1919 and I think it's just the joy, I think the joy as well as the darkness of the times are both the elements that influence it quite a bit.
[00:17:16.707] Bruno Martelli: I think about this whole project as a bit like a jazz and basically you've got this kind of underlying rhythm and then you can sort of like go off and have a little bit of a solo kind of moment and so because it's effectively, it's kind of modular, it's a modular piece and there are different elements and so we're trying to make something that's flexible that we can show in different forms. We're all riffing off this dazzle camouflage idea and also because we've got this great big toy box of historical references that we can think about, there's the incoming modernism which is kind of replacing the romantic era before and there's technological shift as well and at the time ragtime music which was basically consumed by people as sheet music and they'd buy sheet music and take it and play at home there's technological shift like records were kind of invented and people could buy records and the new musical form that supported that and sprang up at the same time was jazz and so I think about there's this background and the different names for dazzle it's like razzle dazzle and baffle and jazz patterns It's now, like a hundred years later, that it's come down to the word dazzle to mean this kind of particular, but actually at the time they had lots of different names for it. And so, because it's not quite fixed, and of course the essential idea of dazzle camouflage is that it's about confusion, and so being fixed down to one form, that's not as confusing as ever shifting around.
[00:18:28.612] Bine Roth: It's quite nice that actually with the music, because I just thought around it as a collaboration, sort of the act of listening. So you're just sort of listening to the solo, but you're also listening to the background, because the background, the bone of it, you sort of need that rhythm to keep on going. But then you see like people maybe springing out and doing their little solo, but then you also listen to that solo and react to it again. And I guess that's also what it does when you play music or you're playing the instrument. then you'd also in that jazz piece you have to listen all the time to the whole orchestra to be able to be part of it and I guess that's maybe also for VR yeah it comes quite nicely together then yeah
[00:19:06.745] Alexa Pollmann: Yeah, I think especially when you look at the way that we now have this piece here, it's so great because, you said it yesterday, we start performing ourselves when the audience comes in, right? Because we look after them, we dress them, they go on stage, of course the dancers interact with them, but also we need to watch out for them because they are in headsets, so we go on stage and we all wear costumes when we do so. So it's actually really holistic in that sense and so I think the kind of beauty for me as well is that that all happens naturally and it is different every time and I think that's the beauty of this virtual world because I think even like Bruno is mostly directing the camera for where you can see into the world And so, you know, the camera moves around and it's sort of like freestyling or a bit like maybe a DJ or a VJ reacting, you know, so yeah.
[00:20:02.922] Kent Bye: Because there's many different layers to this piece where everybody that comes into this piece gets to put on a piece of this dazzle fashion and then there's two people who get to be in VR and dance with the other dancers and be fully immersed in the same co-located space and then there's three other people in VR that are not in the physical space but are able to have a window into what's happening in the VR and I'm not sure if they have an embodiment but as a dancer I had a full embodied avatar along with the other surrealistic avatars that you had with the live motion captured dancers who are doing different choreography. And then you have a seat here for people to be able to watch this performance so that they can see on a screen, a 2D portal into this virtual world, but also see the physical representation of what's happening. So I've only had a chance to be immersed into the VR, so I don't know what the experience of what it looks like from the outside, but as audience are coming in, they have different access to full embodiment into the full experience, or just access to the virtual experience, or access to the 2D realm. As you're designing for each of those different people, love to hear how it's gonna be a different experience for each, but what you were trying to optimize for each of them.
[00:21:09.834] Ruth Gibson: I think I'd like to just pull back a bit on what you've just said because it feels sometimes like, you know how you have the chaperone and the guardian in certain different headsets as the kinosphere, the sort of reach of where you can move? Well, in this particular space here where we're sitting with the audience that have an avatar that are immersed in the world, The audience that aren't wearing the headsets can kind of see this sort of ethics of care. So people are the guardian and the chaperone, actually staff, you know, our group. And there's something about that because they belong in that space. And also in the first space with the three headsets that don't actually, they witness the performance in the headset, but they're not in this space. Sorry, difficult to have this on a podcast. but actually they're given a costume as well, so it's a performance from the beginning, from once you enter until you leave, and everybody's part of it. We have a seating arrangement here, which I think really we wouldn't like, and that's becoming something that's necessary for those people who don't want to put a headset on and maybe can't move so well. but at the same time I think eventually we would like to make it more in the round so that it really is similar to the experience that the headset user is. So I think the project will evolve and it will evolve iteratively through every single performance here has made us have an idea about how to do that and other artists coming to see that are also getting ideas which is brilliant because it means that a community of practice are learning new stage directions for working in VR, and so on. But also, coming back to your actual question, it's all about money, right? So, you know, we had to rationalise, so we didn't have all those headsets because we needed the funding to do that, so we need the support to do that. And also, what it's pointed out is actually it's quite commercial, because people are coming back, because they want to have a go at what they didn't have a go at, or they want to be totally immersed, or they want to watch. So it's interesting, so the active want to become passive and sometimes the passive want to become active. So we're learning a lot. That could be a key to the uniqueness of the experience.
[00:23:27.745] Bruno Martelli: It plays off against the idea of the installation and one of the things when Ruth and I started working together we were not frustrated but we realised that there was some people in the UK were afraid to come to performances because they were worried about being trapped in a space with this performance going on that they might not like for some unspecified amount of time. And so we were like, OK, we have to get away from that and we have to start just making installations. And what we want to have the vibe of is you can go in and if you don't like it, you can just leave. That's fine. We don't feel bad about that. You watch it for as long or as little as you want. And so we've made a number of virtual environments and immersive game engine works where basically they don't have a narrative and they don't have a beginning or end and you can start at any point and you can stay for as long as you like and it's about the user having the power or the freedom to be able to choose the length of the experience they want and we're trying to recreate that a little bit here as well but we have been squeezed into this performance format, which is actually squeezing back into a proscenium arch kind of mode. And so I think we'd like to escape back into an in-the-round, just like you go in if you don't like it. If you like it, you stay, and if you don't like it, you don't. Because actually, it turns out that when we showed the single-user version, Dazzle Solo, the scenes are controlled by the user, and they decide when they want to go to the next scene, right? They just have a button, next scene, next scene, next scene, and it's just in a loop, right? And you've just got one, two, three, four, five, six, one, two, three, four, five, six, go around like that. And it turned out that without slots, some people were just in there and they're really quick and they're five minutes and they're like, great, they loved it. And some people are like in there for like quarter of an hour and actually it averages out. It kind of worked out and it was fine. And with this piece, how we're showing it here, we're trying to make it so that the experience is for multiple users. So some people like the thing with having an audience avatar and interact with the performers, like I hate shit like that. I would never do it myself because it just creeps me out. I never want to be, I'm like, that's not for me. I know some people are really kind of into that kind of stuff. So I'd much rather be like a VR person. But we know that not everybody can do VR because they may have astigmatism, or they might have some problem with balance or something like that. In fact, we've got one-eyed friend. He's blind in one eye. So the stereoscopic effect of VR is completely lost on him. And maybe elderly people don't want to stand up. They want to sit down. They don't want to wear a bulky headset on the head. It's too weird. And so we're trying to make it so that we're not trying to be exclusive to one particular audience group. And at the same time, we hope we've got enough exits that it's easy to leave for people who don't want to stay. We're not like, lock the doors. Free to enter, pay to leave.
[00:25:57.014] Ruth Gibson: That's not what we're about. Which is interesting. You know, it does become this space where people think, oh, I can take my headset off. I can actually do that.
[00:26:08.708] Bruno Martelli: the costume is part of the agency because we're all wearing the ridiculous, silly, crazy, monochromatic, everybody's wearing it. It's not like the performers are the gods and the audience are the peasants and the sort of technicians are in the middle. It's actually more like it's kind of blended. So we're not trying to make anybody feel weird, but it's not just like audience, you have to wear this outfit, you have to wear a funny hat. And it's like, we're trying to make it flat, like a level playing field if you like. Trying to break down the separation between the stage space and the kind of audience.
[00:26:35.200] Bine Roth: You choose your level of engagement. That's sort of it, and I guess that's key to all of it. It's like we're not forcing people into a costume, but even though most of the people are actually quite keen to be part of it, because you experience... Because we're not forcing them into it, they want to do it.
[00:26:52.812] Bruno Martelli: It's not compulsory, it's fine. If we said, oh, everybody has to wear a giant bunny suit, or you have to wear a giant bunny suit to do this, it would be a bit like, oh, God, really?
[00:27:02.718] Ruth Gibson: We're also empowering the dancer as well because in another version of this the dancer can choose the scene change. It's not the stagehand or the director who presses the buttons and that's something we want to develop as well.
[00:27:17.764] Bine Roth: We also have the dancers sort of choosing how they interact with the audience. The audience, obviously, every time there's a different person in the headset, so you have to react differently. So, again, that also is quite a lot of attention that the dancers sort of need to have, but still... It's riffing.
[00:27:31.157] Bruno Martelli: It's jazz. We're just riffing off each other. Everybody's riffing off each other. trying to get a shared experience there.
[00:27:36.519] Alexa Pollmann: I think we are really ambitious with this piece but I think the beauty of it as well is that you have not one way to experience it or one journey, you know, and that keeps it playful for us as well and it actually makes it really joyful for us because we find something new every day and I think that's as well something as an artist what you really need. to develop further and I think that's why we were happy to stay in this piece for three years now and actually go on with it because there was always a new element coming in with a new audience member coming in.
[00:28:11.976] Kent Bye: Yeah, there's been a number of different live performance pieces of dance that's premiered during the pandemic and the challenge of understanding the liveness of a live, of a virtual performance is a lot different than having a live performance when you're co-located and you can actually engage and react in an interactive way. And I feel like that was one of the new aspects of as things have opened up, you know, more people are going back to these physical events. that that type of live performative element I think is opening up new things. But some of the other aspects that I really appreciated about this piece was just the world design and the transport of nature being into this whole other surrealistic world that has this monochromatic textures and shapes but contrasted to this dazzle patterns and the different avatar representations that have a variety of different shapes and forms and I guess Bauhaus is like, I don't know if that's a reference or there's inspiration there that reminded me of some of the stuff I've seen from Bauhaus, but also there's a replication that happens with taking the certain aspects that can only be possible within VR. So taking one dancer and then spreading them out into different geometric shapes and forms so that when they dance, it's actually them interacting with themselves in a spatial context that goes above and beyond what they could do as an individual, which I thought was really quite interesting to see the continued evolution of seeing how the dancers, as they're performing, they can see themselves on the screen and see how they're creating these larger patterns and larger contexts that are maybe going beyond what they could do in a physical performance where it's just themselves. So there's a lot of things of the actual experience of this that I think are pushing forward what's possible with where dance is at, what you see with people just in the mundane physical reality versus what you can start to do when you start to add on both the world design of the world that you're putting into this larger context that's transported into this another place in time, but also the avatars and the virtual effects. So I'd love to hear any reflections on that.
[00:30:02.109] Ruth Gibson: It's a really great observation and probably Bruno's better to talk about some of that but for me it's just great. It's like stepping inside a Busby Berkeley extravaganza at times and sometimes I think the dancers actually don't want to look at the screen. Because, you know, it can be super disorientating, but at the same time, sometimes they're just like suddenly, out of nowhere, some movement will happen that's triggered by that. It's like, you know, now I'm a grid, in a grid of a core doing exactly the same as me and all these clones. And again, it relates back to the humanity and the oddness of that, but also the beauty of it and the patterns, you know, being able to see underneath yourself and above and and so on, and yeah, that's amazing. It's amazing as a dancer for that. That's what drew me to motion capture years and years ago is that, you know, it's beyond the body, you know, and these things might move beyond the human as well soon. So there's aspects of that, but that's another story. But Bruno, do you want to talk about any of the patterning?
[00:31:09.094] Bruno Martelli: Well, obviously having a chorus line of dancers would be expensive, and so... There's copy and paste kind of thing in the computer, which makes it cheaper. But yeah, I think Busby Berkeley. We're pulling in multiple references from the time. And one of the interesting things about Busby Berkeley, he was actually brought in to do a drill, to drill the army. And so he'd stand on a big ladder, and he would shout at soldiers to get them to march around in formations before he actually went on to start doing big musical numbers. And so there's this strange. I'm just going off on a big Busby Berkeley riff now. This is tangential, sorry. So one of the designs of one of the stage sets is basically a Busby Berkeley set which we've got a photograph of and then we just extended it into a complete environment. Yeah, we're obviously thinking about all of those things and obviously there's the duplication because we've worked with other performers who aren't here and we've got their motion capture. I guess what we're doing is we're gradually, we can record motion capture from our performers and we can add them as avatars into the world and then each of the performers has their own sized avatar that, because of technical reason, has to have the same limb lengths and the same height and everything. So we've got a really tall performer, we work with Harry, who's not with us now, so his avatar's like extremely tall and so There's a funny personalization and customization behind the actual depersonalization of these ultra-simplified avatar forms, which I'm sort of interested in that because basically all you've got left is when you've got a simple polygon shape is basically the humanness comes out through the movement. And the motion capture we're using, I think it translates extremely well because it's a nice high-end system. Obviously, making everything black and white, that's probably a lot cheaper than using color, maybe. Is it? I don't know. Does it work like that somehow? I don't know, you know what I mean? Yeah, I don't know. I think, well, obviously what we want to do is we want to develop and add more scenes and we've been designing them secretly behind the scenes and we have a bunch of other avatars which we haven't released as part of the project. Now as we develop, we're trying to make the system inside the computer a little bit open-ended. We've got like an array of scenes and then we can add more scenes in and then what we can do is eventually we'll do it real-time where we can decide what scenes we're having. we can do it on the fly, and we'd really like to give that control to the actual performers so they can improvise with that in the moment. Because, you know, the computer world, which is all, in a way it's fixed, we want to make it somehow improvisatory, and so using the live camera is one way of doing that, and another way, of course, is the performers, their movement is not preset. So they have some sequences which they can fall back to, but they also are encouraged to improvise.
[00:33:45.022] Bine Roth: They obviously play with their avatar, or they learn with their avatar, probably as well, I guess.
[00:33:50.384] Bruno Martelli: Yeah, and they've looked at some of the other sequences from the other performers to learn them, and they've looked at it through the computer, because we don't necessarily have any video that they can watch, they have to look at it inside the game engine.
[00:34:01.258] Ruth Gibson: And that's part of it, that's part of the choreography, is that they're actually looking in the computer at the work, so that they learn it from that. So it has this other edge to it that's a bit new to them, that keeps them on their toes, sorry to say that!
[00:34:16.405] Bruno Martelli: It's a weird capability because one of the things we did when we started we were recording the motion capture and then eventually we obviously a good thing to do in a computer is use a loop but then you want to give it some variance so you make it into a palindrome so once you've got a palindrome loop what you do is what's nice is to actually do variance of the speed because one of the scenes at the end there's a big chorus line of high kicking dancers if they're all absolutely the same timing it looks really mechanistic Even though it's come from motion capture, it looks like a computer animation that's gone wrong. And so there's a little bit of coding which actually just variates the speed slightly so that all humans who were doing the proper chorus line would not be able to kick exactly the same time, even with a lot of Busby Berkeley shouting at them. So they're just a little bit out of time, and it's phasing in and out of the timings. The performers, the real performers who are learning the material, they're actually watching these loops back which are actually playing back as a palindrome and also the speeds varying so they have to sort of do quite a lot of thinking because sometimes they're watching the loop is going backwards so they're learning the choreography backwards and forwards at different speeds.
[00:35:21.803] Bine Roth: It's quite nice to see sort of the agency then again it comes probably up to that agency that what you give to people which is not just you but also the performers sort of having their own take and probably also giving in their own you know as Ruth also said that they're sort of bringing up all of a sudden through watching their avatar different movements that you're actually allowing them to also grow with it and grow with the avatar and sort of get into different spheres. It's quite nice to watch. Now we've got also a new performer with us this time and it's really lovely to watch her now sort of starting the learning process. Yeah, she's really getting into it now. Yeah, so it's so nice to, yeah, I think there's always like a special moment when you sort of then all of a sudden you see like different movements sort of happening.
[00:36:04.775] Kent Bye: As an embodied user, my avatar was different from the dancers. It was a lot more bulky and I felt like a little less embodied in mine because it was a little bit less differentiated from what the dancers were. So it was like the contrast between what the dancers are and what I'm doing. But there's also an element, I really appreciated you elaborating on the choreography of being able to have those variances because I've seen a lot of replication without that variance and it does look like it's just one person dancing. There was a more organic feel of having a vastness of many people that were doing the same motions but had that variance. But I think the other thing I just wanted to point out is that during the pandemic, when they were trying to do virtual live performances, it was always going to be the speed of light differences that gave latency. So there's not a same synchronous moment, but here, using the OptiTrack system and having the ability to have a co-located space, you're able to have a lot more liveness of the live. Like, I see a lot of dance performances that were live, but then watching it virtually, there was the question of how do you interrogate the liveness of the live? And there's really no good way other than to be engaged there yourself and to have a direct interaction, almost like a conversation with the dancer, so that they're reacting to your movements in the moment. And that's really, really hard to do virtually because of the latency issues that you have. And plus the haptic dimensions of being able to actually interact with the dancer in real time with a dance, which I had an opportunity to do at the end, which was really quite a nice moment to go from the virtual dance performances into a live performance and actually have that live moment that I felt was able to really achieve that with this piece of Dazzle.
[00:37:38.169] Ruth Gibson: Yeah, that's great what you're saying. We did do some remote production and it was actually last year when we were working with 5G and we were using one of the tools that we made for the project, Motion River, which allows streaming to happen and It's a bit like the holy grail of photorealism, this idea of being totally on the beat. But actually, it doesn't necessarily work. And for liveness to register, it's not about exact. It's not necessarily about that. And that's probably what we've started to learn. Some people will disagree, but actually we've used it as a choreographic stimulus. And Bruno came up with this idea of the manuensis, which is this secretary who takes notes, or the follower or the scribe, and so one of our sequences is just that. So we've been embracing all those things, those tropes that people might think are overused or whatever, but they're not overused, which is this idea of the mirror, the idea of the puppet, the idea of mimesis, and so there are all these things interwoven in the project, and that latency and that lag has been slightly embraced, because what happens is, to avoid that, or to come close to something that's in time, is what you were talking about earlier for virtual production, is that the dancer copes with that and it's like who follows who to maybe bring that closer together. So without giving the game away, that's some of the choreographic things that meets the tech to solve those problems. And it's that energy and that magnetism between that moment of choice And it's almost like a patience. It's not necessarily anticipation, but it's a patience between how the audience receive the dancer and how the dancers relate to one another that can maybe augment that liveness and this notion of proximity that you're talking about that maybe isn't reached because of the distance. But all those things in real space and in virtual space is what we're investigating with the dancers and with the tech.
[00:39:53.882] Kent Bye: Did you have more that you wanted to say on that?
[00:39:56.785] Bruno Martelli: Oh, I could just, I could honestly, I seriously, I could whittle on for hours about all this kind of stuff. I mean, one thing we noticed, there's a failure mode for the audience where, I can't remember how we set it up, but there's a condition where what happens is you have the avatar body, but you're not actually in it. It's in front of you. And so you're like behind yourself or rather, yeah. So you, the real you is behind your avatar. and so you're basically puppeteering yourself kind of thing and it's such an amazing sensation of actually doing that because you move your limb and because the system we have here there's kind of zero latency the limb moves and then you just start like really goofing around but it's kind of stupid because you could just do that Anytime you want, you don't need any technology to do it. But somehow, somehow the foolishness factor comes out. And, you know, motion capture's been called the devil's rotoscope, right, by animators that were kind of disparaging about it. But this thing, this kind of self-puppeteering, is very interesting to me. Because you, you know, you move and you see the avatar move and then you, and when it's flipped around, you know, when it's actually you, because it's not flipped around, it's the same way around that you are, I'm not really describing this very well. But anyway, sometimes it's like a bug, basically. And we're like, oh, because the origin was set wrong or something. Anyway, blah, blah, blah. So we fixed it. But it's like a great error.
[00:41:09.325] Bine Roth: I also sometimes wonder if the puppeteering also happens with the gadgets, maybe, as well, that people sort of get a different way of, like, using their body because they've got all of a sudden more heavy jewellery. The beauty of this piece is in a way also the heaviness of it. Not that I'm going to say you need to have heavy things on you to move differently but it does make you feel your body differently. It makes you very aware of your limbs and things and you see all the people sort of starting to move in really weird kind of handy kind of way. movements and it's quite nice to watch as well because they just get a different perception, which again starts again with this sort of like dressing yourself up, get a different perception and then they start like being in the VR and again having a different, I think they sometimes also lose themselves in a way, not losing like in a... They would definitely lose themselves. Yeah, they just sort of like, they just sort of like wonder. They're on the game, they're not.
[00:42:02.731] Ruth Gibson: They become sort of curious. Yeah, they become curious. Be curious of themselves in a different way, don't they?
[00:42:11.328] Bine Roth: their body. So what is their body? Again it comes like up to this disassembly maybe of the bodies that we sort of have in a way that you're starting to feel a different wrist movement. Again also what the dancers do but it also comes back to the audiences as well because they get a different feel for them wearing these things. Because also on the feet you obviously have like the flamboyant sort of things and they start a bit also goofing around but not everyone. So it's quite also there's this two things again physical, digital or virtual coming together.
[00:42:42.065] Kent Bye: Yeah, I wanted to touch on a couple things on the fashion and then start to wrap up here, because I feel like the other aspect of being a part of this experience was to put on the motion capture technology here, and I've done literally thousands of different demos over the last eight plus years at over a hundred different conferences and So this is the first time that there was actual consideration of what the motion capture technology looked like from the outside because most of the time it's just very utilitarian. It's all about just serving a function and it's probably prioritized the comfort for the user so that it's like tight and not moving around. But this I felt like it was I noticed it a lot more that I was aware of the metalness, but it was also, it looked nice as I was putting it on. It was, you know, there's a lot of consideration of ergonomics, but there's also the fashion. And sometimes it can be in contrast between like something that looks good, but may not feel as good. But maybe you could talk about your own process of trying to take a fashion sense of some of this technology and how to balance some of those trade-offs that you have to go from the utilitarian function, but also the look and feel of how it looks to other people as they're seeing it.
[00:43:42.890] Bine Roth: I think it's like the aesthetics and I think that's also the whole kind of world that we created and that's sort of another entry point to it. So I guess, well, what we sort of felt I guess with looking at the gadgets like they're quite tacky, right? They're really plasticky, so I don't find them really comfortable. So I guess then looking at how can you also, again, the audience, if they come in, how can you make them feel a bit more special and maybe also more, yeah, again, using their body or feeling more comfortable in your body. What you also said just now, I guess that also comes with it, that you're getting... I think metal has also a nice feel to it. there's something around it and again also the weight of it which you don't get with plastic and obviously putting them on every time I put them on like people sort of like getting in with the hands I think that's the moment like the feet not so much but it's more the hands because they watch it more if the feet are too far away but with the hands sort of sliding in then people have this uh-huh, oh, wow, effect and being a bit like, oh, okay, fine, right, and then they sort of start a bit more being aware of who they are.
[00:44:54.634] Alexa Pollmann: I think it's also quite interesting because the idea of comfort or comforting or, you know, there's this utilitarian that is being pushed a lot in the gadget world because it has to just kind of be out of your consciousness. And I quite like that people might feel weirded out to some degree, you know, or might think it's a very... they're not familiar with it. So it's also a world that they can then dive into better actually as being aware of it. And I think the idea of, you know, dress, garment, fashion, jewelry, adornment, often is as well that it's not necessarily about just comfort because then we would all run around in jogging wear which has happened over the pandemic and it's really sad because I think if you go into the street and you see someone really well dressed there's so many senses that we have And I think we react to other people and this kind of interaction between people or looking at someone dressed differently is as much a part of an experience as the actual experience itself. And technology has this tendency to try and hide itself. And I quite like that we also reveal how everything is done in the piece. And I think it makes it more And by making it more prominent, it's something that can cause curiosity in the audience again, and actually a will to understand what's going on.
[00:46:23.016] Kent Bye: Awesome. And finally, what do you each think is the ultimate potential of immersive performances like this and what they might be able to enable?
[00:46:33.158] Alexa Pollmann: I have one. I think that what Ruth also, I think, has been saying in the past is that the performer of the future might be the audience. And I think the idea that really there is this non-hierarchical world that it allows to open, that Bruno talked about earlier as well. You know, it's definitely a thing that gives us the opportunity to experience it individually, but at the same time very collectively.
[00:47:02.267] Ruth Gibson: Yeah, there's a lot of what Alexa just said was something we'd talked about, which is there's been a real push for audiences of the future. So what she was saying is like the idea of performers of the future, which is something I'm super keen on because actually that helps the audiences of the future and the experiences for everyone for the future. So for me, the whole experience, what I like to see is from the beginning to end, from donning a garment and seeing the people themselves in the mirror, they change their way that they stand, their alignment shifts. They're having to wear a headset to see VR, but that change in the alignment carries through by then putting on the jewellery pieces that are functional for the system, and all of it has this elegance. So, instead of this hunched looking at the phone, the deportment really, the alignment of the body, that's what I think is really important for people, you know, as we move forwards is to With all these things we wear and so on is actually to carry ourselves and not injure ourselves.
[00:48:18.359] Bruno Martelli: I'll take a slightly different tack and I'm just reminded of a story and it was about somebody whose dad died and they got their old PlayStation out and it had some, I can't remember, some racing game like Colin McRae or Forza or something and what happens in the game is that you race around the track and then basically you can do the race again and you're racing against the previous winner. and in this case the previous winner was the person's father that died several years before and so they were racing and it's called the ghost car in the parlance of the game and that's the terminology for it and it's kind of translucent car and you're racing your car and you're trying to beat the time of the ghost car which is the previous highest winner and so this person was basically racing against his dead dad and it was a way that the digital was kind of providing this kind of ephemeral moment as a kind of a memory and so I'm thinking about in the future of performance it might be something to do with there's some way of like recording you get maybe if there's maybe these things we have it at home somehow there's some sort of futuristic tracking technology enabled by virtual reality and remote performance in some way it's recorded and then it can be sort of played back and then you've got this maybe you have a memory of you as well as the audience Not only do you have the performance coming in, which is maybe in a particular way, but you're in there too as a digital trace. One of the things that I always think about digital technology, which makes it similar to live performance, is that live performance, one of those things that you have to be there to see it, otherwise it's gone. And we think about I don't know, Yvonne Rainier, performances from the 1960s and then there's just basically all that's left of it is like an old film that maybe we see. And so you're either there or you see this old film which isn't the same thing at all. So the performance is ephemeral, it's gone. But in the same way, the digital technology, it has a lifespan and then basically the operating systems change and you can't necessarily get it back or recreate it or play it back. and we've all got like, well I've got things like floppy disk, I've got floppy disk drive, we've got CD-ROMs that have delaminated, I've got like old hard drives and so in some ways the digital is really like life because it's ephemeral and so I think something, something, something about memory, ephemerality in digital and performance is a kind of, is my future thinking about this.
[00:50:26.323] Bine Roth: Well I guess the experience, the experience that you get and then also I think someone said that yesterday in that talk like also around the emotions and then sort of mixing up the virtual and the physical and what's real, what do you consider as being real, what do you consider as being like virtual and if there is a real virtual, if it is or if there isn't? but what you can definitely say is there's something that you feel and that's the experience and I guess that's that's the thing that you take home and that you've got just for your own and I think that's also what we're sort of creating here there is an emotion sort of coming up and something and an experience that you take and that is just your own and it's different for everyone and what is then the mixing of those two worlds is I think the unique thing that makes this whole experience one thing Also what Bruna just said around the ephemerality. Is that a word? Ephemerality.
[00:51:20.925] Ruth Gibson: That's interesting what you just touched on. The joy of living in a simulation. Are we? Are we not?
[00:51:34.040] Bine Roth: I guess that's what it, what also happens here is that the whole confusion that's sort of happening, and I guess that's also a beauty of it. It's not like, yeah. It's not linear. Yeah, it's not linear. And again, we're coming back to the rounding it up. It's not linear. And there's always movement. And I think the movement is very important for this experience to evolve.
[00:51:59.147] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:52:03.867] Ruth Gibson: Just have fun with it. Don't get bogged down with retelling what we already know that's happening in the world. it's already there. For us to see on the news, if we keep making artworks that are replicating that, I'm not sure where we're heading really. We have to have something that steps out of that. Obviously draw from horrific experiences or identify that or bring it to the public's knowledge, but at the same time there's room for the other, there's room for new knowledge. Not preaching to the converted as well. You can cut that one out.
[00:52:45.814] Kent Bye: No, that's good. Any other final thoughts or anything?
[00:52:50.016] Bruno Martelli: Well, I was going to say, you know, GitHub's really hard to use.
[00:52:58.910] Kent Bye: And with that, thanks for joining me here for the podcast. Yeah, I really, really enjoyed this performance and yeah, a lot of things in terms of the embodiment and the performance and the fashion and the installation and transporting me. The magic circle that you've been able to create here that sets the context for me entering into the virtual world. This is a whole other layer and then coming out back into this as the offboarding. So it's kind of a nice transition of using the physical space to kind of enter into this portal and then coming out and all the different viscerally actions I had. in the context of the piece. And so yeah, I really enjoyed the piece and I'll have to come back and watch it from an observer point of view and get some other perspectives. And yeah, but thanks again for joining me here on the podcast to help break it down. So thank you.
[00:53:35.431] Ruth Gibson: Thank you. Well, thank you. Thank you. Great questions.
[00:53:40.413] Alexa Pollmann: Thanks.
[00:53:41.593] Kent Bye: So that was Ruth Gibson, a choreographer and dancer, Bruno Mattelli, a visual artist, as well as Bina Roth and Alex Pullman. They do both fashion, dance, performance, and technology, and immersive experiences for Porte Preteur. And Ruth and Bruno are part of Gibson Mattelli. So I have more coverage of all these different experiences from Venice and episode 1021 and if you want to recap on that I'd recommend doing that. I'm trying to get 20 different podcasts out today so I'm not going to be doing my full in-depth takeaways for each of these but I'd encourage you taking a look at that as well as episode 1144 will be the immersive panel that I did at Venice that was looking at the art of reviewing immersive art and entertainment, which I think is kind of relevant to having these different types of conversations. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
