From Dust is a 24-minute VR Opera by composer Michel van der Aa that is premiering this week at the Rotterdam Immersive Tech Week, and I had a chance to get a sneak peak at the end of my trip covering IDFA DocLab in Amsterdam. The emotional core of From Dust is the top-notch composition from van der Aa and brilliant musical performance by Sjaella. It was a really powerful and moving immersive journey that seamlessly integrated my GenAI prompts creating a personalized experience. I can highly recommend checking it out to see where immersive music experiences are headed here in the future.
The throughput of this piece is only 3-4 people per hour, and so it may be hard to be able to see it once they start touring it around Europe to different musical festivals. van der Aa told me that this is the lease commercial project that he’s had a chance to ever work on, and acknowledges that this is a piece of subsidized art. There have been a lot of broader discussions happening within the XR industry about the sustainability of these types of location-based experiences, and the need for creators to understand some of the more pragmatic financial constraints for exhibition taking into consideration earlier in the production process. I’m in the process of finishing up the public report from the Think Tank at Venice Immersive this year on this very issue, and the MIT Open Documentary Lab is also actively starting to study these types of distribution questions as well at the R&D Summit at IDFA DocLab this year.
But at the same time, it’s also great to see artists who are able to get this type of work funded who are willing to push the virtual reality medium to the limits of creative expression despite some of the financial impracticalities of exhibition. Especially as it may drive the adaptation of this type of work into a format like the Apple Vision Pro or PCVR where it can be within a form factor that is a lot more scalable, even if there are compromises on fidelity or on the GenAI elements using the open source LLM of Flux that can not be distributed Steam due to Valve’s restrictions on AI integrations within games.
I had a chance to catch up with van der Aa last week in order to get more context on his journey and process on creating the piece, as well as get some elaboration on the degree on some of the branching mechanics. There are some questions that they ask during the onboarding process that categorize the user’s Big Five Personality characteristics, and then they are creating some invisible branching of my experience that was completely imperceptible to me. van der Aa told me that about 75% of the musical experience is the same for everyone, but that there are some explicit and implicit branching that is happening musically, but also a bit more visually as they are integrating my GenAI prompts in 4-5 different places throughout the experience. It’s certainly an innovative and ambitious piece that has some GenAI parallels to Tulpamancer from Venice 2023, but one that shows the potential of combining musical composition with these types of immersive adventures that are able to tell a much richer story when combined.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast, the podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So at the end of my trip at IvaDocLab when I was in Amsterdam, I had a chance to see a sneak peek of a VR opera piece called From Dust. It's premiering this week at the Rotterdam Immersive Tech Week. where you need like a seven by seven meter space. And there's some prompts at the beginning to doing the psychological profiling of your big five personality characteristics and having some invisible branching that's happening in the back end. But there's also some choices that you're making throughout the piece, but overall, all there's the journey that 75 of the people get the same music but there's a lot of other visuals that are also added in that they have some generative ai prompts that are taken in at the beginning where you're asked to think about your past present and future and then those are all seamlessly woven in to this piece that has this kind of like liminal dreamlike feeling to it it's got this point cloud representations and the musical group of sojala has six different members and They're 3D avatars, they're CGI, but their faces are video, and so they're also tracking you as you're walking around. And so it actually does a really great job of making you feel like you're being serenaded. And I think overall, the emotional core of this piece is this really top-notch composition from Michelle van der Aal and this really brilliant musical performance from Sojala. It's a really powerful and moving, immersive journey that seamlessly was able to integrate my prompts into creating this personalized experience. So definitely recommend checking it out if you get a chance. So I think it starts to point to where some of these immersive musical experiences may be headed here in the future. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Michel van der Hall happened on Wednesday, November 27th, 2024. So with that, let's go ahead and dive right in.
[00:01:58.028] Michel van der Aa: Hi, my name is Michel van der A. I'm a composer and director from the Netherlands. I've been working mainly on the theater and opera stage and in the concert halls, but also have been making some work, some digital works, and now my second VR work from Dust.
[00:02:17.889] Kent Bye: Nice. So maybe you could give a bit more context as to your background and your journey into the space.
[00:02:23.032] Michel van der Aa: Yeah, so I started as a recording engineer. I had my own recording company and I was recording a lot of contemporary music. So I really loved the technical aspect of recording, but also the musical aspect. And I did that for a couple of years. I had my own company, but I had a little itch I wanted to create. And I was already composing at that time. And I decided to stop my company and to study composition at the Royal Conservatory in The Hague. Got accepted, did that. And during my composition years, I found that I had a lot of visual ideas as well. And I somehow wanted to extend my vocabulary, my musical vocabulary with staging, with film, with lighting. So I decided to take a year off in 2002 and study film in New York. the Film Academy, and I've studied stage directing as well in New York at Lincoln Center. So yeah, expanding my toolbox in order to be able to tell the stories that I want to tell.
[00:03:26.636] Kent Bye: Okay. So it sounds like you're coming from like a center of gravity of the music and composition. You started to expand out the visual thoughts you had by starting to go into film. How did virtual reality start to come onto your radar in terms of something that this was something additional that you wanted to explore, you know, also with that experience with the stage and having the theatrical spatial context, but maybe you just elaborate a little bit more of how VR came onto your radar and what you wanted to do with some of your first pieces that you started to do with virtual reality.
[00:03:56.977] Michel van der Aa: Yeah. So in my opera pieces and my music theatre pieces, I often use quite a lot of technology on stage. Film projections. In 2013, I did an opera with 3D film projections where the sort of physical space was extended with 3D film and the singers were sort of mixed in the 3D world with 3D singers. And that sort of made me think of... I mean, this is all... I mean, being in a concert house or being in an opera house, it's a wonderful ritual, you know? A few hundred or a thousand people and you experience this event together. But... There's some types of stories that I can't tell on an upper stage, very individual stories. And I think that's why I started to be interested in VR, because I love the fact that the audience member is the protagonist in the piece, the only protagonist in the piece, and you can really make it about the audience member. So I started working on my first VR piece called Eight. It's a 15-minute journey. Basically, you follow the life of a woman from when she was 80 back to 30, back to eight. I worked with a designer named Musk, so we used a physical set as well. So it's a very tactile experience. There's a hallway with sort of stretchy walls that you can touch and it's mirrored in VR and you sort of ignite events in the opera like that, like a membrane almost. But this was a story that was the same for everybody. I mean, still you were the protagonist in the piece, but I wanted more. So a few years ago, I started thinking about ways to really create a personalized opera in VR. And with the development in AI, suddenly that became possible. So that's where I ended with From Dust.
[00:05:46.448] Kent Bye: Nice. I had a chance to watch the trailer for your previous piece of Eight, and I noticed that it was actually commissioned by the Holland Festival, the Beijing Music Festival. Maybe you could give a bit more context for the funding that came about for the first project.
[00:06:01.946] Michel van der Aa: So I think when I did eight, people in the international music world knew me from my operas. I was sort of interested to see what I would do in VR. And we were very lucky to have a great group of financial supporters and commissioners, Holland Festival, Sydney Festival, Google Arts Institute. Beijing Festival. Oh dear, I can't remember them all, but it was quite nice.
[00:06:29.447] Kent Bye: Kunstfest Spiel Harenhausen.
[00:06:32.489] Michel van der Aa: Ah, yes. And Aix-en-Provence, I believe. So a lovely list of presenters and commissioners. And the nice thing about it is that we also got to do the piece, you know, and travel around the world. We were on continents almost. And I found it super expensive to do VR very well. I mean, you need a lot of developers, a lot of hours. And it's at least as much work as doing a piece for Dear Prostage, I would say. It all takes place in this headset, but it's quite a large creative team needs to work on it for many, many months. And from dust, we worked over a year, I think.
[00:07:09.810] Kent Bye: And so with From Dust, did you also have a series of different funders and commissioners? Because, you know, like you said, it costs quite a lot of money to do these types of projects. And so, yeah, I'm just curious if the success from the first piece of eight led to other support for you to spend over a year, like you said, working on From Dust.
[00:07:28.140] Michel van der Aa: Yeah, I mean, the main commission is the Doelen in Rotterdam, which is a wonderful concert house. And they collaborate once a year with the Immersive Tech Week. And they want to present a new immersive work every year. And this year, they commissioned me to do a work. So a lot of the funding came from them, from the Doelen. And we were very lucky that the Dutch Film Fund and Fund for the Creative Arts helped with quite a significant subsidy and a few smaller subsidies. So altogether, it's mainly funded from Dutch money, and we have some international presenters that are interested that will help a little bit as well, but this was mostly funded from here, from the Netherlands.
[00:08:06.680] Kent Bye: Okay, so yeah, I had a chance to try out from Dust at the end of my trip covering IFA DocLab, and I was in this big, giant room, and it was probably a 20 to 25-minute experience, and I sat down and answered some questions, brought some prompts, and then went through this quite epic journey through this acapella group that was singing to me. You had the recreations of all the different singers and really quite elaborate production that we can dive into and pack a little bit more. But I'm wondering like where you began with this piece in terms of like, you know, it sounds like that the audio was pretty much set. Like there wasn't much changing in the composition or maybe if there was i didn't notice and so you know it sounds like the music is the foundation the core of where you're starting and then you know you mentioned that you worked with a previous vr project with eight and that you know you have these visual ideas and so i'm wondering if these ideas of what you could do in these immersive spaces were like feeding back into the composition process or maybe just talk about in your process of where you began with from dust
[00:09:10.290] Michel van der Aa: Yeah, so it began with, I mean, I was thinking about the idea of starting the opera with a particle cloud and everything sort of arriving from that cloud. You touch the cloud, it forms itself into a shape of a woman. From this woman, alter egos are split. And from there, they take you to this sort of magical world. I quite early on started writing a script and this was a very intuitive thing. Five scenes where I thought you would go through as an audience member. During that time, in parallel, I was already composing the music. So the music very much is the heart of the piece and the core of the piece. But this skeleton, of course, will be slightly different for everybody in the audience because people have their own onboarding input. And I wanted to make sure that in a way everybody can find their own way through this opera. And actually some parts of the music you haven't heard because there are options where you could have chosen where you would have heard different type of music. So there are some parallel possibilities there. It starts for everybody the same and it ends the same, but in between it sort of branches off and comes back to the end again. I worked at the dramaturge Madeleine Coiman and we started researching texts by female poets and writers in two directions. One is based on this idea of five personality traits, psychology, technique, you know, you have these five sliders for every personality and you sort of where you roughly can put yourself on these sliders, chaotic, organized, introverted, extroverted, et cetera. So we thought, OK, how can we use this as a starting point for an audience member to be put into this world? So the questions we asked you all had to do with these sliders, these five sliders. And they determine, for example, how close the signals can come to you, how tight the spaces are, how chaotic the spaces are, the way your text prompts are visualized, how chaotic they are visualized, or whether they're very organized. So in a very subtle way, this sort of drips through in the piece. And then we started making it and I'm quite an intuitive maker. So once I've seen things, I get new ideas and we started to deviate from the original script. And I mean, a lot is still there, but also a lot we sort of discovered during the creation of the piece.
[00:11:30.120] Kent Bye: Okay. And it sounds like that there is some branching, but it converges back at the end, but that there are some variations tuned to each person's experience as they go through this piece. There are some choices that were clear, maybe some other choices that were kind of hidden or things that I missed. The music part for me was really the heart of the piece where it was really quite beautiful, the acapella group that you got to sing this composition, but also the fact that It was spatialized in a spatial context and that they were each like singing at me. I thought that it's subtle in the way that there's the spatialization of each of these individual singers. And I'm wondering if you could maybe elaborate on that specific part, because as a composer, you're able to now work in 3D space and locate in space where each of these voices are coming from and how you are. orienting them in different configurations. And if maybe you were experimenting with, you feel like you're in the middle of a circle versus if they're off to one side. And just love to hear your process in terms of the placement of each of those voices as something as unique in the way that people would be experiencing this piece.
[00:12:38.783] Michel van der Aa: Yeah, yeah. If I may jump back a little bit on what you said before that you thought you missed some things, but I don't really consider missing because it's not something that you have to experience. It's just you took your way through the piece and someone else will take another way and there's no missing really because you can't do everything in this piece. But talking about the audio, yes, I really wanted to make use of the 360 space, not only in terms of placements of the singers, because I wanted the audience members to locate them, even if they appear behind you or next to you, you can always find them, and that their voices are very much linked to their characters. But also, I played around a lot with acoustics, so the acoustics of the space. At one point, you're in a huge glass dome that sounds very different from a very tight hallway or an open space. So the second acoustics also played an important part in the final audio mix. And the German group that sings the piece, I mean, I discovered them maybe 10 years ago. I found them on YouTube. There's a wonderful recording of them singing a piece by Purcell in a church. And they have been singing together for 25 years. And they're best friends. So almost like one body, you know, their sounds blend in a way that you can't really achieve with a choir. And this fits very much the concept of the piece, that this one person is sort of split into alter egos and comes back, converges back. And their voices blend themselves so beautifully for that. That's why we started collaborating on this. And they are super nice people, super open-minded. A weird adventure for them, of course, to be captured in a motion capture suit with a head cam and sort of singing to an imaginary audience member. So when we did the motion capturing, I took the position of the audience member so that there was something to focus on also in terms of sound and eye contact. And that was a very strange experience to be so close to someone. And since privately to you, I could already experience how an audience member would feel like in VR when six people are standing around you and address you like that. It's quite powerful.
[00:14:43.322] Kent Bye: Yeah, that was one of the powerful parts of the piece as well was the eye contact that you're able to recreate with the avatars that they're tracking you as you're moving around and they lock eye contact with you as you're moving around the space. And yeah, it just felt like they were addressing me and tracking me and Obviously, I know it's driven algorithmically, but it still had that effect of I was being serenaded or the song was being directed at me.
[00:15:08.776] Michel van der Aa: Yeah, you are tracked. I mean, not all the time, but at certain spots, they track where you are and look at you and address you. Yeah.
[00:15:15.660] Kent Bye: Yeah. And it's just a powerful experience. That thing is distinctly different than if you're just an audience listening to songs. And obviously these are digital recreations and avatars and they're not actually like singing right at me, but it still gave me that same feeling of this is different than if I were just watching this or listening to it in a way that, like I said, I feel like the audio and the music is the emotional core of the piece and all the visuals are supporting it. And I guess another thing is that they're singing in a language. I don't know if it's a foreign language. What language are they singing in?
[00:15:46.517] Michel van der Aa: I mean, they are singing in English, but it's very much layered and sometimes difficult to understand. I mean, it's all rather abstract poetry. So it's not that important for me that you understand word by word. But we are giving next week at the premiere, we are giving audience members the lyrics before they go in so they can read through them. So they have a little bit more of an idea what they'll be hearing.
[00:16:09.209] Kent Bye: Okay, yeah, I guess that's my memory now. Yeah, it is in English. I think when I'm in a piece like that, I'm listening to more of the musicality and the music. Just when I listen to music in general, I often don't look at the lyrics until way later after I've listened to something many, many times. So I'm not the type of person to come out and be able to interpret what their lyrics meant. But what I felt was more of the spatial construction of... both the tonality and what they were singing, but also all the ways that the music is taking me on a journey. And when you asked me the prompts, it kind of goes with feeling at home, feeling something that I want to avoid, and then something about my vision of the future. And so there's these different beats within the story that are having the consonants and dissonance. And so building some contrast between something that's positive, something that's negative. And so maybe you could elaborate on your process of constructing this piece in a way that is trying to both musically go through these different movements or phases, and then visually also try to create some contrast and some consonants and dissonance cycles within the piece.
[00:17:16.021] Michel van der Aa: Yeah. So we asked the illness members on the onboarding to describe, to find a visual description for their present, how they see themselves at present, how they saw themselves in the past and in the future. And we word that slightly different, but these are sort of the main currents. And through the storyline, you see your own prompts back in the current moment, depending on how you, where the sliders are. In your onboarding, you either end up in a very upbeat poppy dance scene, which you ended up in. If you chose a slightly more introverted answer to the question, there's a way more zen type of music and also the imagery is much more less sort of in your face and more pastel. Then there's a moment where you've already taken quite a bit of journey, then you sort of see your present self, the part of you that you sort of said goodbye to or that you don't want to be reminded to or that you lay awake by. And this part burns down, actually. I mean, in a way, your whole world implodes into your hand and the particle cloud is responsible for burning down your past and from the dust itself. your future settles in very broadly. So this narrative arc is there in the piece without being really in your face. I don't want it to be super obvious, but I think you sense it intuitively. Yeah, so that's the way we use the image prompts. And the music very much... follows that line, the larger line. And I knew how I wanted to begin and how I wanted to end. I knew what type of music I wanted when everything burns down around you. So these three markers I composed first, and then I started bridging between them.
[00:18:59.229] Kent Bye: Yeah, and part of the thing that I was mentioning earlier in terms of that saying that I missed things, I think part of the process of creating a piece like this where I go through a certain experience in the piece and then someone else may go through it and may experience something completely different. So there's a part of my experience that's very unique to me and then other people that go through it, they may have seen maybe the beginning and end or something that it starts and ends in the same, but maybe there's different ways, but there's kind of these different points of convergence that are the same. And so when you're architecting the piece, it sounds like that you're taking from the prompts, you're assessing like the big five personality traits. Maybe you are taking me on some of those different branches and then you're kind of ending me in the same and maybe getting different movements or different sections of the music. And so in terms of like the stuff that is the exact same for everybody, no matter what they answer, what percentage of the experience would you say is something that is the baseline bones that everyone sees versus something that would be completely unique to me?
[00:20:01.687] Michel van der Aa: I mean, the music I would say is 75% the same for everybody. The scenes, degradation, these levels of change. I mean, on the core level, everybody sees a unique thing because of the prompts. A level below, there's these branches in the piece. I would say, I think there are nine or 12 possible routes through the piece. different combinations and you see different things and you have literally have a different perspective on many of these places you can actually move to the other perspective the other side of the slider you have this personality slider there's hallways slightly hidden that you can go through so in a way i kind of like that after you've positioned yourself psychologically on this slider there's always a way to move to the other side like in real life i mean we always have the possibility to change and to to a certain extent work on our Sliders.
[00:21:02.145] Kent Bye: Yeah, that's the thing around when I go through some of these experiences and there are these different branches and choices that are being made for me. Some of them are explicit. There's one moment where I can choose to go through one door or the other. But a lot of what you're talking about is kind of the hidden architecture of the piece that is kind of invisible for me. Like I would come out of it and I wouldn't know for sure whether or not there was anything else that was there. Did that present unique challenges in terms of debugging? Because, you know, there is so much variation.
[00:21:31.209] Michel van der Aa: Yeah, I mean, that's really the most difficult part and something I slightly underestimated. With a multitude of possibilities, there's also a multitude of bugs and things to fix. And it's funny because you think you know what the audience would do, but then we did the alpha test and we were completely surprised by what some people did. Not stepping through portals, walking just... straight ahead into an abyss, walking to walls. And so some of the sort of very core understandings we thought we had of this piece, we needed to sort of safeguard more. So that's what we did between the alpha and the beta test. And from the beta test, people got through. We didn't have to stop it for anybody that didn't know what to do anymore. So that's a good thing. But we had some smaller technical stuff, some rotation stuff. We had one gentleman who couldn't sit down on the floor because of his knees. And we had to bring in a chair, you know, stuff like, okay, our collider was too low. So he didn't trigger our collider when he was sitting on the chair. So we thought, okay, we need a little mark. for the usher to mark when someone can't sit on the floor so that we know that he triggers the collider when he sits on the chair, you know, stupid stuff like that. But it's very important because we want everybody to be able to go through if you're in a wheelchair or if you have physical challenges. Yeah.
[00:22:54.249] Kent Bye: Yeah, just a little magic carpet ride in there that you're sitting down for. So yeah, having accessibility options like that. And also, I mean, I didn't know, like when I went through it, I felt it was fairly clear as to like where to go and what to do next to trigger the next scene. But then as I was walking away, I was thinking about, oh, well, I wonder what's the fallback if people don't take that action and how do you progress the story and trigger it if people choose to not do the actions that were intended and, you know, walking through the door, walking through the portal.
[00:23:24.517] Michel van der Aa: So we give people time to do it. And the music sort of loops at some points until someone steps into an elevator or through a portal until a certain extent. And then we have a subtle text in front of you saying, please enter the elevator or please step through the portal. Please turn around and follow the woman. I mean, after a number of seconds for my mom, basically, you know, she wouldn't know what to do. And this is something we experienced in the alpha test. Yeah, we really need that because not everybody understands what a portal is. I mean, you've done a few thousand VR experiences probably, so you know what to do. But most people, I mean, there are a lot of people who have never seen VR or once maybe like five years ago. They've done Beat Saber, but I mean, it's a whole new audience there. An audience that normally comes to my opera house and now wants to experience this, that will witness VR for the first time in their lives. And there's a certain responsibility to make sure that these people get through all right.
[00:24:20.660] Kent Bye: Yeah, and so next week is the Rotterdam Tech Week that this is going to be premiering at. It sounds like this was commissioned in a way that this is going to be premiering there. Yeah. And so maybe you could just give a bit more context for your plans for the premiere and the larger context for the people that were involved in helping to bring this project about.
[00:24:38.871] Michel van der Aa: Yeah. So we worked with a team of sort of core team of 10 people. Rodland Smenk was the lead developer. And we had a separate team for the motion capture sessions, a separate team for the music. Maloum Koyman, the dramaturg, of course, is very important. I mean, if you go to my website, you can find the full list of names there. But it's obviously a team effort. I mean, I would never... I mean, these are the things I can't ever do alone. I need all the developers for that. And... I learned so much from this project, also on a technical level. You know, it's been quite a wonderful, I would say, rollercoaster ride, but maybe carpet ride. Next week, we have the premiere at the Dooling at the Immersive Tech Week. I think these days already sold out, but we have some more time slots left. It runs until the 15th of December in the Dooling in Rotterdam. And after that, we have some plans to tour to Asia,
[00:25:34.466] Kent Bye: a few places in holland hopefully the uk and us germany france so people will have a chance to see it if they want yeah and the location that i saw it there in amsterdam was probably at least five or ten times larger than the actual six by six meter space that you were using and so yeah i'm wondering if you could talk around the footprint the size and how you settle upon that size of a space in order to design this piece
[00:26:02.199] Michel van der Aa: Yeah, so we can run two installations at the same time, and they're stacked in time. And for that, we need a space of 7 by 14 meters, two squares of 7 by 7, and an onboarding area, which can be in a hallway outside of that space. Other than that, we bring a few computers and headsets and routers, and that's it. So the set is rather compact and easy to travel with.
[00:26:26.971] Kent Bye: I think throughput is a discussion that's been happening a lot more in terms of like these different types of installations and what's the minimum amount of people to go through to see it per hour in order to make sure that it makes sense to actually fund and support some of these types of projects. There have been projects in the past at Carnegie Arena that were seeking outside funding in order to make the content feasible to run because it was such a low throughput for how many people could see it per hour. I talked to Miriam Ashard from Phi Gallery that it was like two people per hour up to like 14 people an hour that they're able to make some tweaks and change it in order to increase that throughput. But with your piece, it seems like it's maybe two or three people or maybe up to four per hour. Maybe you could talk around how many people per hour do you think you'd be able to get through that and some of the other larger considerations of things like throughput and the production and the cost and everything in order to make it feasible to exhibit.
[00:27:21.681] Michel van der Aa: Yeah, I mean, it's probably the least commercial project I've done in my career, especially because of this. You need two or maybe three ushers to run the thing. I mean, I also saw Carnet Arena. I mean, they had a whole team, a whole truck of people there. So, unfortunately, we don't have that much technical need. But... Yeah, I mean, normally we would come in with our team with two people, build up and train local ushers and technicians to run it. And then we come back to break it down again. But we designed it in a way that the local people can run it if we're sort of touring it abroad. But I mean, it's just depending on how much you ask for the tickets, it's almost impossible already to get the usher fees recouped from the box office. It's just art, subsidized art, I'm afraid. And so it's a real festival piece. It's a museum piece. It's not a commercial piece. It won't be sustainable. I mean, I can't increase the number of... I mean, I can by bringing in more equipment, but then I need more ushers as well. So it's the usher costs that are really the limiting factor in that sense. But for me, it's not about that. It's about telling a story that I can't tell in another way. And I'm super grateful that I was commissioned to do this and that we got to make this piece. And hopefully, maybe if tech improves and we can run this thing from a laptop, Hopefully, it becomes more cost-effective tutorials. But I'm thinking about that quite a lot, about the scalability of these projects. And this is one of the reasons that I'm preparing another project. It's called Hair, and it's an interactive digital theater that we're going to launch next year for the Apple Vision Pro and for PC VR. It's sort of a platform where we're porting my previous digital pieces to and where audience members at home can experience the works. And, you know, the main reason is that there's a whole new audience out there that can't find their way to a theatre or to a concert house for whatever reason, financially or whether they are neurodivergent or the distance to travel to a theatre is too big.
[00:29:28.052] Kent Bye: and this offers me a way to present my work in the living room and i'm quite excited about that possibility and so you would have both your previous piece of eight as well as from dust as a part of that as well in terms of because there's quite a lot of interactive things that are happening in from dust where you're getting prompts and asking questions so i'm imagining that you would maybe create a journey that is the archetypal version of that journey rather than specific to one individual
[00:29:56.522] Michel van der Aa: Exactly, yeah. I think we can still do a little bit of onboarding in the psychology sliders, but the actual AI, I mean, we're not allowed to use AI on SteamVR, so that's, yeah, so we wouldn't be able to make a PC VR version out of it. And it's also quite visually on the edge of what's possible, so it won't even run on Apple Vision Pro the way it is now. So we start the hair platform with a few other pieces that we sort of, adapted to this platform and then from dust will arrive later next year on this platform
[00:30:28.154] Kent Bye: Okay, well, let's talk a little bit about the AI pipeline that you have in this piece, because there's an onboarding where I'm asked a number of different questions. And in those questions, you're creating this psychological profile, the big five personality score that is directing, you know, which branches or where I'm going to start. But there's also these prompts that you're asking me. And before we get into the AI pipeline, was there any other projects that you heard or saw or thought of that you were taking inspiration to take input from the audience and then use that within the experience?
[00:31:01.590] Michel van der Aa: No. I mean, I know they're there, but I haven't seen any. When I started thinking about it, I didn't know how... elaborate the onboarding process would be i thought it would be way less complicated than we have it now and with less options basically yeah i mean the challenge was how do you i mean we know that we can create images from a text prompt that's quite easy and straightforward these days but we wanted to turn them into a 3d environment so that's what was what we worked on uh actually, this is an R&D project we did before even starting on from dust. How can we do that? Is it even possible? And we wanted to do it locally. We didn't want to depend on internet to do it remotely.
[00:31:41.507] Kent Bye: Oh, okay. So yeah, there was a piece that was at Venice Immersive 2023 called Topomancer, where you sit down in front of a terminal, you're asked a series of different questions, essentially getting at your past, present, and future in a similar construct that you used in this piece in terms of taking that in terms of imagining some image from the past, present and future that is transmitted into the immersive experience. And so when I did the experience, I answered all the questions straight and then talking to another immersive creator named Craig Contero. And he's like, oh yeah, when I answered the questions, I was talking about dragons and really going into this imaginative space. And so I think when I was going through your piece, I was doing that a little bit in terms of doing this creative prompting where you asked me like where I feel at home. And I said, the Scrata Familia, I've only been there once, but it was something that sticks with me and answering in terms of like metaphoric images of Dante's Inferno of Hell, the Seven Rings of Hell. So whenever I'm doing these generative AI pieces, I am thinking about what are the type of images that I would want to see in a piece that are loosely connected to what you're asking rather than literally connected to what you're asking, just because I've had that experience of doing something literal and then it kind of being, you know, generic or something that's not magical or giving me the sense of awe and wonder. So I was really happy with the prompts that I had in terms of- the way that it was translated into both, you know, some of them are just, they feel like images, but they have like a depth map onto them. And so it had like a gargoyle that was in there that had a whole 3D model. You have different levels of which the prompting is generating general 2D type of images, maybe with some depth map information, some that has even more depth map information and more volumetric ways. So maybe you could talk around that process of taking in those inputs and then translating them into the experience and what you were hoping in terms of connecting my imagination with that specific story beat.
[00:33:42.867] Michel van der Aa: Yeah. Actually, since the beta test, we actually altered the prompting a little bit in order to encourage people to also use their fantasy and not think about realistic places. Because one of the points I took from you and from someone else actually is that it sort of helps creating way more personal and exciting spaces that you wouldn't normally see. I mean, if someone in the alpha test, people talked about mountain ranges or forests, and then you get a rather bland, generic forest that you could have drawn from the internet. So the more precise people are and the more they use their imagination, the more exciting these forms become, like you did, basically. So we're going to encourage people a little bit more to go off the beaten path and to think bigger than that. So the way we do it, we have a local large language model. We work with Flux, and it's creating different levels of images. Like you said, dev maps, but also we do some background removal, and we stack the images in layers, like an old see-through box. How do you call that in English? Get shoe boxes, and they open up the top, and they make a little world inside and a hole in the... It's like a diorama or something. Diorama, exactly. That's the word. But then with individual depth layers, so it's more than a diorama. And we used some other techniques to create 3D models. I mean, I already knew I wanted to step away a little bit from realism, because whatever prompt you give, it will never look the way you imagine it will look. So I really wanted to take a step back from that. That's why it's often made from particles that also flow, that are kind of fluid and are morphing in and out of the images. And so it's already a kind of dream state that you see these images in. And then we have a local network and we share the images to the other computers. We actually upgraded our onboarding computer this week because the laptop we're using was too slow, too many crashes. So yeah, learning a lot.
[00:35:38.602] Kent Bye: And so then you're rendering everything on PC and then you're also streaming it onto the Quest wirelessly. Like maybe you could talk around that part.
[00:35:47.031] Michel van der Aa: So we tried a few headsets, but in the end, we found that Quest 3 for now is giving us the best performance. Yes, we have dedicated routers, Wi-Fi 6E routers that will hang above the audience members. And we haven't decided yet either AirLink or virtual desktop to stream to the headsets.
[00:36:06.087] Kent Bye: Okay. And one of the other things that I remember about the piece was that the spaces were sort of like a dreamlike liminal space in a lot of ways, because the architecture wasn't always clear of the context I was in. There would be maybe some columns with particle effects that I would reach out and touch, and it actually had some interactive components, which was really nice to see how interactive the piece was. And there were some like ways that you're wayfinding or guiding people through by having one of the singers encourage you to follow them and you're kind of mimicking their actions as you're going through this piece. But there's also like some times where it's like there's these geodesic domes in the future. There's some elevator stuff that you're going up and like a hallway of mirrors. Maybe you could talk around like the environmental design and some of the spaces that you were designing in a way that Wanted to create that kind of dreamlike place that felt like an in-between liminal space, but also like still have these touch points where you felt like you were connected to a specific location or constrained hallways or kind of abstract. It felt very dreamlike and love to hear about your process of designing these different locations in the piece.
[00:37:17.202] Michel van der Aa: No, good question. I worked with a leader of 3D, developer Stijn van der Ven on this. What we were looking for is create a sort of realism that slightly falls apart, if you know what I mean. So I always wanted to have very concrete surface and on the edges sort of falling apart in dust. So it's always in flux. So we have this Rens van der Wil, our shader developer, made these beautiful shaders where parts of the mesh are sort of floaty and sort of breaking off and leaking into space. So you have the idea that the whole world really is made of particles and of dust and sort of implodes and builds anew. So that concept of everything starting from this particle cloud and ending with the particle cloud is seen throughout the experience. And as far as the buildings themselves, yeah, I just, I had this idea of this huge cone-shaped tower that I wanted the audience to go into. In the end, yeah, and I love sort of the more steel type of look. In the end, I think it's a bit of, it edges a little bit to maybe cyberpunk opera. It has sort of a cyberpunk vibe, some of the spaces, but there's very concrete metal-like brute force spaces are then softened by these particle descriptions that you gave yourself during onboarding. So this is beautiful. I find that there's a really interesting juxtaposition between these two worlds. So they're sort of very floaty, saturated 3D images and they're way more concrete almost physical space that also on the edges sort of loosens up and flows into space again. So yeah, it's this in-between world, like you say, and that's how we started looking at it. And also this is how the characters look. I wanted them to have a certain realism in their faces because they are video, 3D video, but their bodies are constantly, there's these glitches going through, they're eroding, they're leaking particles if they move around. Yeah, so they're also these poetic versions of human beings, I would say.
[00:39:14.650] Kent Bye: Hmm. Yeah, you have some of those science fiction futuristic scenes and there was prompts that when I answered my vision of the future, it was like answering a lot of solar punk type of things with trees. And so there was these moments where I was like, wait a minute, is this creating for my prompt? Or it's like, no, this is way... too complicated to have been generated on the fly. So there was certainly some overlap in terms of each of our respective visions of the future. Although this was a little bit more of a cyberpunk rather than solarpunk vision, but being in a geodesic dome, rotating around and seeing all these giant holographic faces singing at me, you know, some really nice moments in this piece. And, you know, I think the overall feeling was that I felt like I was being personally serenaded too, especially when they were at one-to-one scale.
[00:40:02.429] Michel van der Aa: I mean, this for me is really the core of the piece. I wanted the audience member to feel the connection to these women in the piece. That's really what it's all about. And we worked very hard on the tooling for the women. That's why we expanded with volumetric recording, with pure 3D models that we animated. Didn't work, became these game characters. And in the end, I opted for a combination of a 3D model and a video hat. which isn't a perfect connection, but somehow it's lip sync. You see the eyes, they're there, you know, if you're noticing they're there in their character and they all mocap their own movements. So for me, that was the absolutely necessary thing in this piece to feel seen by the performers.
[00:40:46.127] Kent Bye: Yeah. And there's these subtle layers of uncanniness in some of the ways that they look, especially at a distance or especially when, and I don't know if this is just a matter of time, as I saw early alpha build in terms of some of them weren't always looking at me. And so when they're kind of staring off in the distance and not engaging with me, I felt like actually when they're looking at me and engaging and tracking me that my brain accepts it a lot more than if they're there and they're kind of staring off in space and how having that eye contact made such a huge difference. And I imagine that will be as you continue to go through ensuring that each one of the beats of the story have those eye contact moments. But yeah, it was surprising to see how impactful that was to allow my brain to really have a lot more plausibility and believe and not focus on anything that was that uncanniness of it all.
[00:41:36.127] Michel van der Aa: Yeah, cool. I would say 80%, 85% is they look at you in the piece. So there's moments where they walk in front of you. Of course, we can't do it. Or when they guide you where to go. I mean, it's all automated. I can decide for every moment whether I want eye contact or not. And there was a long process of going through the piece for each singer, determining where would an audience approximately be. You don't want people to turn their head too far. So it's a lot of trial and error. But I think we find a nice middle way.
[00:42:03.976] Kent Bye: So the piece is called From Dust. Maybe you could elaborate a little bit on the title of the piece, what that means for you and how you were elaborating on those themes in the piece.
[00:42:12.844] Michel van der Aa: Yeah, I mean, I've talked about it already a little bit. I mean, the From Dust idea, it's slightly, of course, we know it from the Bible, but also this is the DNA that this woman is made of and her alter egos and the other sides of her character are made of and the whole world is made of. But also these are digital pixels that our monitors are made of and our digital lives, our digital lives. you know, behind these screens our life is made of. And somehow these felt like elemental building blocks that I could use to tell a very humanistic story, a story of things breaking apart and combining into something new again. Hmm.
[00:42:52.173] Kent Bye: One other piece that I just want to mention also is that there's a connect camera that you have at the beginning that you have a little bit of a volumetric capture of the audience members. And I think you had mentioned something like, I don't know, you didn't say this, but it felt like more of like a direction of a sincere reaching out and welcoming and something. And I kind of did a little bit of a cheesy, like with my mouth open, or I don't know what I just sort of in the moment had a little bit more of a shameless take that then when I saw it in the piece, I was like, Oh, maybe I should have like followed the direction and been more sincere in this moment. Cause I feel like I'm distracting myself from how cheesy I was, but maybe you could talk around this moment of having this kind of volumetric encounter with yourself within the context of the piece.
[00:43:35.799] Michel van der Aa: Yeah, I mean, there's a moment in the end of the final scene, scene five, where all the singers are in a sort of mirrored world around you. And so they're all multiplied. Also the voices in the audio track are multiplied. And at one point they all walk to the mirror and hold up their hands. And then we cut to an image of you, the orange member, walking into the same mirror and holding up your hand. And once you touch your own hand, you sort of float up and you move to dust and float up and sort of descend again in your future images, in your future, basically. And I mean, it's very difficult to prompt this one, because people feel very self-conscious. But now you saw where it was in the piece, I imagine you would have thought, OK, I should have done something different. But this is the way it is, of course. But we had two people in the audience that really broke up at that point, where they sort of touched themselves and floated up into their future. They found, when we spoke with them afterwards, found that the most profound moment in the piece. So I have to think about maybe we should change the way we address the audience members for the recording a little bit in order to get them to that stage. That's some work we still have to do, I think. But I don't, yeah, we don't want to spoil what's going to happen, but also it should be more than a technical thing, you know, it should be, yeah.
[00:44:53.322] Kent Bye: Yeah, I feel like it's a challenging thing because it's like I want to reach a sort of emotional authenticity in the moment and have it be sincere. But also like there's a number of pieces that I've seen in the past with the Collider or other pieces where they have me evoke a very specific memory. And then I'm sitting in that memory. And so I think that might be helpful to really think around, like, what is the memory or that emotional connection? And so that way, that memory, when it comes to that point, I can connect to that memory again with that new context.
[00:45:25.363] Michel van der Aa: Yeah, that's a good idea.
[00:45:27.659] Kent Bye: just because of the raw direction, it was like, okay, I don't know the context under which this is going to be used. And so, yeah. So thinking about a past self reaching out and connecting to my future self and that's the context, but you know, what is the memory and the emotion that you're trying to evoke? And so I feel like there's something really powerful with immersive experiences where you In this piece, I think, you know, another part, aside from the music, another emotional core, this idea that I can express something in my imagination and then have that be represented in a way. That's kind of the magic of generative AI is have these different iterations of these thoughts or images, but then just have those specific thoughts and images juxtaposed within a larger narrative context under which it's unfolding. There was a part where my Dante's Inferno of Hell also happened to have like red and there's colors that you're having people choose. And so there's ways that you're able to tune into these subtle emotional cues in someone by their prompt and what they're bringing up.
[00:46:25.843] Michel van der Aa: Yeah, you're absolutely right. I think this is a good point. It's a good idea for people to ask their past self to touch, reach out to their future self. That's a good idea. We're going to try that next week. Thank you.
[00:46:37.451] Kent Bye: Yeah. Well, cool. I guess as we start to wrap up, I'd love to hear what you think the ultimate potential of some of these immersive experiences, immersive music experiences might be and what they might be able to enable.
[00:46:50.549] Michel van der Aa: So the thing that really excites me is that as a maker, I can tell different type of stories in VR that I can't tell on an opera stage. Stories by an audience member is the protagonist in the piece and essential for the storytelling in the piece. And this is what really excites me. You know, I wouldn't only want to make VR operas for the rest of my life, but some ideas I can't do in an opera stage and I can do in VR. So I'm really looking forward to explore that further and to do my third VR piece in a few years, maybe.
[00:47:20.259] Kent Bye: A quick follow-on question because, you know, I've seen a lot of immersive stories and some people that come from a cinematic background may start with a script and a story. People from architecture, I know in theater, start with the spatial context. As a composer, you're able to start with a piece that has a lot of emotional movements and very powerful, especially with the spatial context. And so when you're translating this emotional journey into a spatial context, do you think about it in terms of like emotional beats that you want to hit in terms of the feelings that you want to give or like what's the language you put when you think about this journey that you take people on?
[00:47:56.104] Michel van der Aa: Yeah, I mean, I of course started as a composer, but I've been directing my work for 22 years now. So I have these double two sides of me that when I'm working on a piece, so I have this visual side, this director side that thinks about the visual responsibilities of the core concept of the piece. And the composer in me wants to think about the heart and the emotional journey as well. And I find that sometimes some things I can really tell well with music and some things I really need image for. And I always... play of changing perspectives in that regard. And the fact that I can do it myself allows me to choose for each moment in the piece which of these layers is foreground and which is background.
[00:48:33.598] Kent Bye: And when you're making the music, are you thinking around emotion in terms of the emotion you want to draw? Or is it more that you're listening to it?
[00:48:40.901] Michel van der Aa: Yeah, it's an intuitive thing. Yeah, of course, I know which scene I'm writing for and what's happening in the scene. So I definitely have that in mind. And sometimes the music actually goes somewhere else and I need to change the visuals for the scene or the other way around. So they absolutely influence each other. You know, that's a continuous play between these two worlds.
[00:49:03.976] Kent Bye: Is that something you give language to at all in terms of like naming those different movements and chapters? Or is it, like you said, it's an intuitive process where you listen to it and feel it. And I'm just wondering how you make sense of it in terms of language or naming it.
[00:49:16.687] Michel van der Aa: Yeah, I mean, there's a skeleton for the piece, which is the time I know I needed to take 25 minutes. I know the approximate durations of the scenes and what I want to tell in the scenes. But within this skeleton, I feel I'm completely free to very intuitively find my way through musically and visually. And especially with VR, I mean, you can do anything you dream of. You know, you can be on a flying carpet or you can... That's one of the wonderful things of VR, that there are no limits on what you can do, what you can see as an audience member, which again, I guess another question to your answer, something I can't do in an opera stage. I can't give the audience a carpet and fly to huge glass doors.
[00:49:56.550] Kent Bye: Yeah. That's helpful just because, you know, talking to lots of people that come from a writing background, they may write the script first or a video game. They have the interactions first or building up the space first, but it sounds like it's both the visuals and the music kind of working together as you're doing this piece. So great. Is there anything else that's left and said that you'd like to say to the broader immersive community?
[00:50:16.831] Michel van der Aa: Um, Well, I mean, thanks for having me, first of all. Lovely to talk to you. And thanks for coming to see our test as the first, actually. And if people want to learn more about me, they can find me on my website, ponderaa.net, my last name, dot net. And I'm also on Blue Sky and Instagram, et cetera. Not on X anymore, though.
[00:50:39.599] Kent Bye: So you're going to be showing it at the Rotterdam Immersive Tech Week and then touring around from there as well?
[00:50:45.165] Michel van der Aa: Yeah, next season we're going to start touring and we'll put the dates on the website.
[00:50:50.450] Kent Bye: And touring, most of these kind of types of projects are at film festivals, but coming from the music background, are these more music festivals or what kind of events?
[00:50:57.337] Michel van der Aa: Yeah, music festivals.
[00:50:58.118] Kent Bye: Yeah, yeah. Okay, great. Well, it's great to see not only the piece, like I said, I feel like it has a real strong emotional core with the rooting of that composition and the spatial sound and really beautiful performance from the acapella group and Yeah, I felt like it took me on a whole immersive journey in a way that was tapping into different aspects of my imagination and felt like a full, complete adventure and a journey that you were able to take me on. And you just really thought it was quite moving and powerful and just really well done. I think it speaks to, you know, not everybody can put together these types of experiences, but the way that the spatial medium and immersive quality is feeding back and forth to the music. It really feels like where some of these different types of musical experiences might go here in the future. So Michelle, thanks again for joining me here today on the podcast to help break it all down. Thank you. So that was Michel van der Al. He's a musical composer, as well as filmmaking, and is creating these virtual reality experiences. This is his second piece. It's a VR opera piece called From Dust. So definitely recommend checking it out if you get a chance. Really amazing musical performance by Shajala that is spatialized. And so you're walking around, and it feels like they're singing to you. It's just a really powerful part of the experience overall. Also, it's got this liminal space, like dreamlike quality, where there's all these point cloud representations are coming in and out. And you have the chance to do this generative AI prompting. So ways that you're able to use your imagination, but also have that emotional core that's also tied into the piece. So there's some times where that really worked well and other times where I had this volumetric capture of myself. And I think there can be a little bit more improvement in terms of, you know, the direction that I'm given, because I don't know the context under which this thing that I'm doing with this little gesture works. And so, you know, having a way that I can also recall some of that memory within the context of the piece is something that I also see a lot in the context of these types of immersive experiences where you're asked to kind of recall different memories or different emotions. Especially when you start to think around the branching narrative of this, you know, having these different branches, like when I went through it, I didn't know that there was a lot of these subtle branches or these different choices I could have taken. There was like one explicit portal where I could have walked through one door or the other. And Michelle was saying that there was actually a lot of like hidden tunnels that I could have gone through to discover these other aspects of the big five personality categorization of me on these different sliders. And then based upon those answers or giving these slight variations, like I said, that's kind of imperceptible to me as I'm going through it. But there are these different tunnels where you can see the opposite polarity of some of these different answers for these different big five characteristics. Like I said, the overall, I think the emotional core of this piece is the composition and the performance by Sajala. And even though there is a little bit of this uncanniness of these singers, this group, the six different members, it's got like a CGI motion capture of them, but also like this... mixture of some of their video from the face but also the eyes are tracking you dynamically so it's kind of like this weird hybrid but i think overall it works just because they're looking at you while you're singing and then overall you're kind of overcome by the power of the music and just the fact they're also looking at you there's some times when they were looking away that was the times where it felt a little bit more uncanny but there's something around having this personal serenading experience that i think is also really quite powerful um And it's also something that is somewhat inaccessible in the sense that only like two, three or four people can see this per hour. And so as they tour this around, it's not a type of piece that a lot of people are going to be able to see. So it is the subsidized art piece like Michelle Vanderaal was talking about in terms of this wouldn't be feasible without these commissions. And I think it's still valuable to kind of push the edge of what's possible. It can be just frustrating as an audience member if you're trying to get in and there's just very limited throughput capacity to be able to actually go and see it. However, there are looking at ways that they can translate some of these different types of experiences and distribute it out on the Apple Vision Pro. So they, in the future, they're going to be able to use these artistic commissions to prototype, to see what works and to have this blend between both the music that is being created, but also the visuals and these immersive spaces and these contexts and these different journeys that you're taking on and just the way that you're able to have this full immersive experience. And there were some really powerful moments, I think, where the music just really hit me and I was completely immersed in the space. And There's different images that I was talking around, like Dante's Inferno, these mushroom houses, these solar punk visions of the future, also the Sagrada Familia. So these types of images that really inspire me that have these contrasts between like things that I really feel at home in and things that I want to avoid, you know, Dante's Inferno and Hell. But in the context of the piece, it created this contrast as well. Part of the feedback that I was giving that really encouraging people to tap into their imagination and not to just go back into their mundane answers that they may think of when they're asked the questions. And so really tapping into people's imaginations and being able to seamlessly weave that into these different pieces. Yeah, highly recommend checking this out. And yeah, I think the throughput issue is the thing that kind of is the pragmatic thing that is going to make it a little bit more difficult to see it. But if it does come through in Europe or some of the different places, check out the website, check out where they're going to be taking it. And yeah, definitely keep an eye out for it. And for me, it shows some of the potentials for where some of these different immersive music experiences may be headed here in the future. So that's all I have for today, and I just want to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a supported podcast, and so I do rely upon donations from people like yourself in order to continue bringing this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
