Andrew Schneider’s “NOWISWHENWEARE (the stars)” absolutely blew me away, and it was a story-driven, immersive art exhibit at the University of Texas Performing Arts Center that opened at the end of SXSW. It uses a cube of 4000 LED lights in a dark theater space to create a visual spectacle, but also to hide an incredible magic trick. There was an array of 392 speakers hidden on three different walls creating one of the most complex and nuanced spatial audio storytelling experiences that I’ve ever heard. I was profoundly moved by the themes of grief, loss, and our place in the cosmos as he uses the low-bit visual experience to amplify the richness of the audio storytelling featuring beautiful biomimicry moments, large movements of abstract shapes, as well as allusions to stars and constellations. It’s an experience that transcends my ability to fully communicate how incredible it was, but I did have a chance to have an in-depth chat with Schneider to talk about his journey in creating it. From theatre-maker to NYU ITP, Schneider does a brilliant job of mashing up data inputs with novel outputs, but all serving the larger journey he’s taking us on as a master storyteller.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continuing my coverage of South by Southwest and specifically starting to look at how different artists are using spatial audio technologies, today's episode is with Andrew Schneider of a piece called Now is When We Are, parentheses, The Stars. So this was an absolutely incredible experience. This was at the Texas Performing Arts Center at the University of Texas. And you basically walk into this big black room. There's a whole grid, like a 4,000 LED stars. And you have this whole like immersive experience. I'm going to dive into more of the specifics of what that experience is. But spoiler alert in terms of we're going to be diving into the actual content of of this show within this conversation. And there's certain elements that I think the more you walk into this experience without knowing much about it, the better. And then I guess now at this point, we'll start to dive into some of the spoilers. Spoiler alert is that there's actually like a huge array of like 392 speakers that he set up within the context of the experience that is using wave field synthesis to be able to create these audio objects that are also located with this whole grid of where these stars are at. Now, there's two different versions of this show, one where you actually have a tracker and you're able to have like a little bit more of an interactive experience with audio and lights. But this was a little bit more of like the narrative version where you're just sitting and watching this half hour loop. So I ended up watching it for just one half hour of loop. If I would have had more time, I would have loved to like stay and watch it again and again. But I had to get back to do my interview with Roman Rapic and Wayne Powell and just kind of continue to cover and actually see Roman's band play his set with Pivots. But I managed to sync up with Andrew the next morning and it was just an absolutely amazing experience and so deeply moving. It's talking around like themes of grief and loss and using this matrix of light to create this low fidelity, like eight bit visual field. And I think with that lower fidelity of visuals. you actually have like a heightened sense of the spatial audio that's happening. And so after I saw this show, I actually went back and watched Roman's show again, but without the VR headset and was just listening to the audio. And there was something around just being able to hear the fidelity and richness of the sound that I missed when I was first watching it, just because sometimes the visual field can be so overwhelming. So This is actually a really brilliant way that you're using this kind of low fidelity visual field to really amplify the spatial audio experience. And in this case, Andrew actually is coming from a background as being a theater maker, which I think is also really shaping the way that he's using these technologies and the way that he thinks of it centered in story and theatrically staging it and having these elements of surprise. And Yeah, I think it overall is just, you know, one of the things I am taking away from this year's South by Southwest is just how many insights the XR industry can get from theater makers. And so this is actually one of the first opportunities that Andrew had to more directly relate to what's happening with the broader XR field, just because he comes from the theater realm, but he went to NYU ITP and is all about thinking around data input and data output and ways of kind of mashing that up Through technology. So, yeah, just really amazing experience. And I really, really enjoyed this conversation as well, because he starts to recount some of the beats of the story. And when he did that, my body just got transported to those moments as he was talking about. And it just was overwhelmed with emotion. Whew. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Andrew happened on Wednesday, March 12th, 2025 at South by Southwest in Austin, Texas. So with that, let's go ahead and dive right in.
[00:04:05.821] Andrew Schneider: My name is Andrew Schneider. I make theater, installation, dance, anything that has a live component. I'm really interested in that. I often use neurological, psychological, physiological phenomena as a blueprint for staging or as a research tool, as an inspiration. And yeah, I've never thought of myself as being an immersive artist, actually. I think this conversation is going to be interesting for that reason, because I come from theater. And so... As I've gotten to know more about the immersive world and talk to more immersive-making folks, I'm like, oh, theater is like, that's what we're doing. We don't put the devices on, we put on the room. We know what the theater is already, and so we go into the theater and do that kind of thing. But this thing that they've made now, called Now Is When We Are The Stars, is, you know, it happens on the stage. It doesn't have to be a stage. It's a big room. And it's filled with 4,000 individually soldered, individually controllable points of light that you can interact with at certain points. And a 382 channel wave field synthesis array, plus some other spatial audio going on that sort of creates audio holograms around the experiencer. to tell them the story about not somebody else, but sort of about them in that moment and how maybe thinking about how they got to be there that day, you know, every choice they've ever made in their entire lives and like how that's sort of led them to here and now.
[00:05:37.702] Kent Bye: And I'm wondering if you could give a bit more context as to your background and your journey into this theater space slash immersive space that's kind of expanding out and continuing to evolve. But yeah, just to hear a little bit more about your journey.
[00:05:49.400] Andrew Schneider: Sure. I mean, we can go real far back. When I was a kid, I just like a formative memory is like my parents had a VHS camcorder and me and my friends would just like make funny videos with it. But I remember like turning it on its side and, you know, the classic thing of like pretending to like walk on the wall or whatever it is and realizing like, oh, you can create impossible realities by like using like readily available technology, by just paying close attention to phenomenology and perception and things like that. So because of that, I was super into theater, like making performance stuff. Because I remember going to the theater and being like, when all the lights go off, that's the most exciting part. Like when the lights come up, I was like, okay, now I know where I am. But when lights went out, it was like anything was possible. I didn't know what was coming next. And it was like scene changes were so amazing because it was like... the lights would go out we would be in a living room and then like the lights come back up and you like be out on the street and i was like that can't happen in reality and that's so phenomenal to me because my brain knows that like we're in a real space and like that's not possible so i went to uh it was a big choice between like musical theater school and like fine art photography like black and white photography i was doing a lot of that stuff just really thinking about framing of pictures and stuff like that So I was doing both of those things, basically, like doing musical theater, moving to New York, like auditioning for like Little Shop of Horrors, things like that. And yeah, it was that show specifically after playing the dentist in Little Shop of Horrors for like the third time. I was like, man, I love telling stories on stage, but maybe there's a different way to do it, you know? And all the groups that I was interested in, like the Worcester Group or the Builders Association, They were all using this tech stuff and I was like, this is cool. I don't know why, but it's not about the tech, it's about the story, but it's enhancing the story somehow. And so one of them said, well, you should check out this program. It's called the Interactive Telecommunications Program at NYU. It's a grad school. It teaches artists how to be engineers, teaches engineers how to be artists. And so I applied to just that school, got in, didn't know what I was going to do, found out it was the perfect place for me. Learned how to make LEDs blink, learned how to make a web page, learned how to make servos do their thing. And just had this real realization that you can take data from the outside world. There's a sensor for everything. And if there's not, you can make a sensor sense anything. whether it's like galvanic skin sensing or brightness or you know super like low level easy stuff and these parts are super cheap and you can put them together in clever ways and then you can get that number into a computer and then once you get the number into a computer you can use whatever software and like get anything out of the computer you can just do some simple math and like translate it and like put it into actuators electromechanical servos or like solenoids or displays or lights or sound and that was just like oh you can map anything to anything I can walk around the room and Have some sound follow me for instance as what happens in the show that we're making and that was just like it really opened up the world for me because I didn't I was a very late bloomer in tech like I didn't have an email address until I was like a senior in college, you know and I thought that tech I remember I was living in New York and I had just gotten a cell phone and my partner texted me that they loved me. And I was like, this is the downfall of civilization right here. This is like you can't text someone that you love. You have to, like, write them a love letter. You know, I was like, man, we're going to be talking so much more, but saying so much less somehow in the future. so that was also i i i didn't trust any tech and so i was like i need to explore this from the inside out and then once i went there i like sort of drank the kool-aid a little bit and was like this is this space this tech space mapping one thing to another thing is really what i want to be doing or just like it opens up the palette so much and especially of thinking about theater like You have such a captive audience in theater because you know half the theaters is like you're sitting in a certain seat, you know? So like if you know that that person's in that seat, you know that they're going to be looking where the lights are. So already you have like these two amazing things where you know the location of the person, you know where their attention is focused. Then you can really manipulate their attention. And most people know how to use a theater. They don't have to figure out a VR headset. I don't know that I thought that at the time, but looking back now, I'm like, oh, that's why I was so attracted to it, because people know how to use this building. So it was just natural to me to bring these new media tools, these new experience manipulators, attention manipulators, into a theatrical space, because people have the patience for it in the theatrical space, because they're already expecting to sit in the dark and have their emotions manipulated for an hour, you know, and they're not going to like leave. It's like, that's what a great captive audience, you know? And yeah, so I was making, I was just making little performance pieces and I made a piece called You Are Nowhere, You Are Now Here, all one word, said both ways. and I didn't know what I wanted to say with that piece but I had an idea of like what if the lights were on just on me and then suddenly the lights were on everywhere else and we could do fancy things with blackouts because LED lights had just become commercially available especially in the theater space so you could do these instant blackouts that didn't have this incandescent tail of like where your eyes sort of adjust So you could really do jump cuts in the theater. So you could have a person appear and then disappear if you rehearsed it enough. You could kind of make people disappear, kind of like a magic trick. And that show used all that high-tech stuff, but also a lot of low-tech stagecraft. There's something that happens, there's this huge reveal that happens halfway through that I can't tell you about, but... It's the most low tech thing that I've ever done. But because you're primed for it in a high tech way, you think that it's you think that something incredibly technological or incredibly high budget is happening, but it's not. It's just that we took the time to like do the impossible thing. And that show sort of like got big and toured around the world. And it steadily drove me towards, okay, there's something here that, the reveal that happens halfway through, I realize that the people aren't watching the show anymore. They're watching themselves watch the show. They're questioning their own reality. They're saying, wait, this is, wait, what's happening? This isn't real. Like this can't be, this can't be happening. so they're they're not following me they're they're like seeing themselves see like in the way that erwin and terrell talk about like perceiving your own perceptions and that was like oh that was a huge revelation like the story doesn't have to be about someone else the story can be about you sitting here in the theater and so all of the pieces since then have had some element of like we're not being transported somewhere else we're definitely right here right now then because we accept that reality we can like manipulate the very subtle like the super subtleties the micro realities of the surrounding area so that you do start to question your own reality so it's not the suspension of disbelief of like an outside thing it's sort of like I don't know. It's sort of like what happens in magic where it's like the magician says, I'm going to trick you. And you as an audience say, OK, great. Trick me. And then they trick you. And then you say, oh, my God, that's amazing. You know, your brain believes it again. I don't think I knew any of those things at the time. But looking back, I'm like, oh, I think that's what I was doing.
[00:13:21.201] Kent Bye: Awesome. Well, that's a great recap of all your journey into this space, and I'm wondering if you could briefly recap your experience that you had last year, and then we can start to dive into what you're showing this year. But yeah, Blake told me a little bit about it, and I wasn't here to check it out, but I'd love to just hear you talk about what you were doing last year, and if there was any other big immersive-type project, or projects that are getting away from the proscenium, let's say, that more immersive, and then, yeah, we'll dive into your latest piece.
[00:13:46.693] Andrew Schneider: Yeah. Yeah. So last year we were here, I was working with an artist called Annie Saunders. We make sound walks together. And we had a nice commission from the Simons Foundation. They're an organization, science organization. They make science accessible to non-scientists. And the great northern eclipse was happening in 2024 in April. And so they set out a commission and it's called In the Path of Totality. Any group could propose a project that happened within the path of totality that was sort of about the eclipse that got people excited to come out and watch the eclipse on Eclipse Day. And for us, the eclipse is all about like being in the right place at the right time. You know, three planets, three bodies in space have to align for the shadow to be in the right place. And so, you know, extrapolating that to like, oh yeah, like I turn the corner and I see like the love of my life or I turn the corner and I just miss them or whatever that is. So I was super interested in that and like making that happen, making that real. And with these sound walks, we had in the past, we had done things like We use geolocative audio on an app that we've developed with Josh Kopasek, who's from the UK, makes an app called Echoes, but we worked with him as a developer. So you put on headphones, you know, you scan a QR code, you put on headphones, you put your phone away, and then through the headphones you're taken on a walk. So you hear Annie and I sort of... telling you, hey, welcome to Eclipsing. We're here at the Long Center. We're going to take a walk like this building on your left. Like this used to be a different building completely. And by the way, like JFK was coming to this building the day that he was shot in Texas in like just like 150 miles away from here. And like that sort of starting building those like what ifs. And then you keep walking, you cross the street, you go into a field and then we're talking about like, you know, coincidences, things that could go wrong and things can go right, things go wrong. The asteroid hit the Earth 150 million years ago or sorry, however many years ago. And because of that, all the dinosaurs died, but also, like, we happened to flourish and survive. And you can see that, like, here's the limestone right here. But also, sometimes things go wrong. And just when you're saying things go wrong, like, someone over there, like, drops their bag of groceries on the ground, and someone else goes and helps them pick it up. And you're like, oh, that's a cool coincidence. You know, they were talking about things going wrong, and then those people dropped their groceries. But you don't really think anything of it. And then you go a little further and you look at like, oh, let's stop and look out at this field of people. You know, all these people are here right now. And if you think about where they were yesterday, they were all in different places yesterday. And then you think about where they were last week and maybe they were even further away. And then you can trace that all the way back to when they were born, like what hospital they were born in. Everyone's in probably a different state. Some people are in different countries, halfway around the world. And you can draw a line to here right now. Like every decision they've ever made has led them to this moment right here, right now. And as you're thinking about them, maybe they're thinking about you. And as we say those words, maybe you're watching all these people out there. Maybe they're thinking about you. 50 people out there stop and they look at you. Like, you know, because we hire them to do that. We rehearsed like an ensemble of 20 like core performers. Then we had a bunch of volunteers. who would be listening to a separate track of audio and we just made these little moments where as you're walking around these super serendipitous things happen like you're told like you know there's a rabbit that usually hangs out over here and that's something that we noticed that a rabbit was always hanging out in this field so half the time the rabbit would be there and the other half the time the rabbit wouldn't be there but because we say we don't say there's a rabbit that's over here we say the other day there was a rabbit here and when you see that rabbit because it happens to be there half the time your mind is blown because you're like wait what is a part of this how did you do this you know and because of that everything becomes part of it because anything can be part of it you start to look around and you start to think And like, wait, is that part of it? Is that? And you start your mind starts attributing things that we have nothing to do with to what we made, which is like just one of my favorite things that has ever happened in a project, like especially because it's outdoors and it's such an uncontrollable environment. So opposite of a theater. It's so hard to get people to focus on a thing. There's so much input happening all around all of the time. But with little subtle directions, little subtle cues, you can still make magic happen, which is super cool. And the show sort of continues on like that. As you're sort of led around thinking about being in the right place at the right time because of the eclipse, you're just thinking about your own life and how maybe you're always in the right place at the right time, no matter what, you know? And you can make the most of those moments. So the story, again, isn't about something else. It's about you being right there in that moment. And it was great. You can still do the walk today. I mean, we can't pay the actors to be out there. But yeah, those things live on. And they're accessible. Like we made a show in New York. We made a show in Austin. We made a show in Greece, in Athens, Greece. It was great. Last year was a great experience.
[00:18:38.604] Kent Bye: Nice. As you're talking about all those serendipitous moments, I had a little bit of those series of serendipity coming to your show last night, where I was going to Roman Rapic's show. I had left here at 7.10, and I had to be there by 7.20. I ran into Danielle Giroux. We saw Roman's special audio, and then Roman was like, here, come see this secret installation that we have. And then on the way to there, I ran into Josh Bengston and Dennis from Black Gamers Organization. And then I was like, hey, we got to go check out this other immersive art installation. And so when we got in the car, we drove up, and talking around themes of death and loss, sort of endings, was a theme that was coming up. And so then we get to your installation, and we go in, and we get in right at the start of the loop. And it was like, go ahead and have a seat. And then we watched down and just had this really emotional experience watching it. So it was really quite beautiful. But I'd love to hear how you, it sounds like you've been already working with LEDs in a theatrical stage, and that you were already starting to experiment with sound. I don't know if you were doing anything with the spatial sound with that many 300-plus channel wave field synthesis. But yeah, I'd just love to hear the origins of this project that you're showing here, Now Is When We Are Stars.
[00:19:49.778] Andrew Schneider: Yeah. The first show that I made was, yeah, it's like at the very end after this big reveal happens, it's like, maybe I'm dead. I'm the only person in the show. And like at the end, you're like, wait, is like this a metaphor for death? Is he dying? And I remember like the day before we opened, I was like, this doesn't sit right just ending like this. So I was just like, what if we just end in this field of stars? And I have no idea why I thought that, but I soldered up like 20 little stars, you know, those little like LEDs that I clipped off of like some craft lights or whatever and hooked them up to an Arduino. And like someone at the end of the show, someone's like hiding under the booth just because of some other things that happen. And they hit this little button on a bread box and they, these little stars light up and then twinkle. And that's how the show ends. So it's like, it's not just like it ends with death. It's like, oh, it ends with like maybe rebirth or like something else, just like something atmospheric. So those stars were just like hanging out in my studio, you know, 10 years ago. And then every show since then, it's like, oh, maybe we'll have a moment in this show where like there's a blackout. And during the blackout, like these stars come up like over the audience. And so we did that in the next show. And the next show, it was like, maybe there's another blackout and they're like you know these stars come on and like but maybe this time they're fireflies you know so let's get a different color temperature of leds and so that just kept happening and kept happening and in the meantime we were working with wave field synthesis my sound collaborator bobby mcgover like introduced me to this concept of spatialized audio And we had a residency up at MPAC, and they showed us their wavefield synthesis array, which is a bunch of tiny speakers making audio holograms out in the space. And we asked if we could use it, and they said, absolutely not. We don't let artists use it, which is amazing. And then eventually they saw the show that we were making, which is all about hallucinations and whiteout conditions and your brain making neural noise leading to hallucinations. They saw that show that we were making, and they were like, oh, okay, we can let you use it. It's like the perfect use case. So we made that, and then the Simons Foundation, again, before they helped us out this last year, they said, what is this? Audio technology? We've never heard anything like this. Are you going to tour with this? And we said, no, it's not ours. They said, well, can you... buy it? And we said, no, it doesn't, you can't buy it. And they said, can you build it? And we said, yeah, but it costs a lot of money. And they're like, here's some money to help you build it. And so they funded us building a Wayfield array. And so then we really got to think out loud with the tool. That was a big thing. Like just the time on the tool to think out loud was just such a huge, a huge thing for us. So that was happening concurrently. Like, oh, we can like put audio out in the space. And I'm just obsessed with these little twinkling lights out in the space. And something I was thinking about with Wayfield as we were using it was like, oh, it would be great if you could imagine something in that place that that sound was. It's cool that the sound is out in space, but there's no way to get you to look out there or to get you to walk around there. It's cool that you can bring the audio out and sit it on the shoulder of an audience member, but what if the audience was moving around and they could discover these little audio pockets? And so we were experimenting with projection, like maybe we can project or like moving lights, maybe we can like use haze and moving lights to like lead you to a location so you can discover a sound. And that just didn't do the thing that I wanted it to do, you know? It was like 90% of the thing, but it wasn't 100% of the thing. So I was like, well, maybe let's just put a physical object there. I have these stars hanging out in my studio, so I'll just put a star there. And so that was the first thing. I was like, oh, this is cool. The star could tell you a story. And you could use some tracking data, again, with the idea of if you can get numbers into a computer, you can get them back out and affect stuff. So I just did a simple camera tracking thing, which is like when you walked up to the location of a star, it would get brighter. And like, I would turn the volume up on the Wayfield audio object. And then as you walked away, it would get lower volume and dimmer. And I was like, well, what if you did that like a hundred times, you know, in a giant room, like that would be cool. You could walk around and like these little stars could tell you a story. And that was the first part of it that I made, which there's two versions of the show and you didn't see that version. So there's a version where at the end of the show you actually walk up to every star and it tells you a little story, which is all recorded stories from people who have been through the experience before. And so that was the first part of it, just like one single plane of stars that each one was interactive and could tell you a story. And I was like, well, how do you get people in there? Like, why would they be in there in the first place? Is it just, is it a interactive installation? Is it a story? Like, do you see everything from the beginning or like, can we like sneak people in there? So for me, I really wanted it to be like, you don't know what you're going to enter into, which is like, for me, all my shows are, there's this level of expectation versus reality or like, If you know what's going to happen, or if you show the picture of the installation right before you go into the installation, it's like, you know what's going to happen. So I really try to keep some things at least a secret, so that you know that some things are going to happen, but you don't know that other things are going to happen. So you enter in total darkness, and similar to the Sound Walk, you're invited to think about these things, like every decision you've ever made, leading you to this room right now. you know, the fact that you were all talking about death and grief and loss, like, and endings right before you came is like, everyone can relate to these things, I think. Even if you weren't talking about those things, everyone can relate to them. So it's not the story of someone else, it's the story of you there in that room. And then you walk out there and then you're invited to like, get more comfortable and look at all these, you know, it grew from a single plane of stars into like, well, if people are in there, like, what are they looking at? Maybe make it a 3D matrix instead of a 2D plane. And then maybe the narrator can talk to you. We already have the Wayfield out there, so maybe the narrator can come talk to you as you're sitting out there. It's almost like the dramaturgy of the audience. How do we craft the story, the narrative of the audience going through it? Instead of just like, could it be more than just an interactive installation? There's nothing wrong with that, but could it be like what theater is good at, which is a beginning, a middle, and an end? There's a start time and an end time. Yeah. So it grew very organically from like this idea of like this one thing I wanted to see happen into like, oh, it's a story about you, you know, discovering these things. Yeah. Yeah. I'd love to hear about this.
[00:26:11.274] Kent Bye: kind of homegrown wayfield synthesis array of speakers because the room felt like it was a little bit more of a rectangle than a square. So it felt like there was an array of speakers that were on the longer side and then also on the shorter side. And I ended up sitting on the end just so I could also see all the lights. But if I were to do it again, I would probably be in the middle, but also kind of walk around a little bit. It was dark and, you know, a lot of people were laying down. So the affordances were just kind of sit in one space and listen to it. But It was such a rich sound field that it's a type of experience that if I had more time, do it again and again. But I'd love to hear around how many speakers and what the process was to build this Wayfield Synthesis array of speakers.
[00:26:51.741] Andrew Schneider: Yeah, it's 382 speakers right now, and it's, again, we copied MPAC's design, like, screw by screw, you know? We were like, seriously, what kind of screws did you get? And we're going to do that. What kind of tech flex did you get? And we're going to do that. Each box is about six feet long. Each box has 31 speaker channels and two Dante amps on the back, so it's 32 channels of audio on each box. We only use 31 because, you know, you can have the extra channel for a sub if you want. And, yeah, they gave us the designs. We got some support from the Simons Foundation. We CNC'd the boxes at a CNC little mill. And bought all the components off of Parts Express, basically. And Hold Up in Bobby had a place upstate New York. I was in Brooklyn. And between the two of us, I remember being on tour with other shows, sitting in my hotel room stripping wire. Because we had to strip the correct lengths of wire and label them for these 382 channels of audio going from the amps under the back of each thing to the drivers. And just like the amount of hand soldering and like hand, you know, like literally like just manual labor that went into these boxes so that they can be tourable and plug and play. You know, you could make an array where it's like, sure, you have like 100 speaker stands, but we can't set that up in an afternoon so that we can be performing by the evening. And Bobby was the expert, you know, Bobby like understood fundamentally how wayfield synthesis works and was talking to these great people like Thebo at EarCam, because really there weren't any, there weren't a lot, there were a handful of wayfield synthesis arrays like in the world at that time, you know, whatever, eight years ago. I mean, it was only just a very recently possible audio technology that theoretically it existed, but the processing power wasn't there for it to exist. And obviously like now we're years later and the processing power exists. And so you can actually make these physical objects happen. And the amazing thing about it is that your cam and spat, they made it basically open source. I mean, not completely open source, but at least it's free. So you can experiment with these things and the community was there. So we can, you know, say like, Hey, we're making these filter sets for our three-box array. You tell the computer the physical location of every speaker, and you tell the computer, here's the focus area where I want to be able to have the synthesized wavefront sound like it is. and it creates like a sometimes like a five gigabyte text file of like all of those like every possible basically it says here is every possibility of location volume delays for all the speakers and so it's doing all of this processing in real time it's saying you know if you picture this array of speakers that's 20 feet long and it's 124 speakers long like a desktop speaker just extruded you know to 24 feet long. If you picture trying to get a sound object, an audio object, out into space, out into your head, basically, the outside drivers would fire first, and then the next ones, and then the next ones, so that as the waveform, the physical audio wave is propagating through space, however fast sound moves, all of those waveforms converge at your head, for instance, or anywhere else in the room that we want to put the object at the same time, so it becomes focused there, as if the sound was coming from there. Which is something that I sort of understood when we were making it, but it's taken me years to actually wrap my head around how that all works. Because you can put audio objects out in front of the array, you can put them behind the array. And then most demos I've seen with it, you see the speaker array and then they're like, imagine a cello right here. And you imagine a cello and then you walk around and it's like, yes, the cello, instead of being where the speakers are, the cello does sound like it's coming from the middle of the room. But my brain is always thinking about the fact that there's speakers over there. And so with our thing, it was like, let's hide the speakers, you know, in darkness, basically. And we won't advertise that it's like a Wayfield Synthesis Array show. We'll just say like, it's a show and you can come to it. And then we can do them, you know, again, like expectation versus like... sort of impossible realities. And so right now there's 382 channels. They're on three sides of the room. And we do like some pretty rudimentary sort of like handing off of like, okay, which array is it at right now? The version that you saw is, you know, it's for like 40 to 50 people at a time. There's another version for only eight people at a time. And in that version, everyone wears these tracking devices. We don't tell them that they're tracking devices, but in that version, we're able to know all the coordinates of the people. And so we can, in real time, move the audio object to where you are. So the narrator can always be like sitting on your shoulder, basically telling you something. And then in parallel, the narrator can be talking to your friend over on the other side of the room, telling them a different thing. And you would never know that you're being told different things because you can primarily only hear your own audio object. And then at the end of the show, again, when you visit these little stars, we're able to put these 72 audio objects in stationary positions so that when you walk up to a single star, that audio object is pre-placed there. And as you walk around that object, it does sound like the audio is coming from that single star specifically, which is really a magical thing. And people are like, wait, how? where are the speakers? It's coming from there, but there's no speakers there, which is just a delightful thing to have a conversation about. That's really the magic of it for me, is not just that it's cool, but that it does this impossible thing that hopefully the longer you're in there, the less you Because you walk in that room, and it's a technological room. I think the first five minutes, you're like, wow, this is a lot of stuff. And it's all hidden, but hopefully after five minutes, you stop thinking about all this stuff, and you start just thinking about what I'm asking you to think about.
[00:33:10.534] Kent Bye: Yeah, just a follow-on on the open-source software, because I was looking at the DMV audio, how Roman was driving... With his DAW, he was sending all these channels. And then it was basically just creating an XY plane. There wasn't any Z depth. So it was just able to position audio in this 2D plane. And they measured all the different speakers. There was just 13 speakers that they were using. Can you talk around the pipeline from your DAW into whatever open source software, whether it's a plug-in or what's doing the kind of specialization and then translation of all those channels into all the different delays and everything else that needs to be input into the amp and to the speakers to be able to create this waveform synthesis experience?
[00:33:54.113] Andrew Schneider: Yeah, so we have we have like Ableton Live is driving the whole thing. And that goes out of basically there's like, you know, I said that there's 70 little pockets of audio that you can visit at the end of the show. So those each have a lane of audio. So it's 70 different channels. And then on top of that, there's. There's like 13 speakers above those speakers, which are doing like some basic like VBAP or surround panning above that. And so that's another 13. And then there's just a bunch of automation lanes for them. So there's the hundreds of lanes of Ableton stuff. That audio gets sent out through some Yamaha and RedNet PCIe cards, which, again, we built this eight years ago. And so the compatibility of the hardware was different then. And, you know, we were locked into some legacy stuff now and we're looking into buying some new stuff. But Basically, we're going through four Yamaha and or RedNet PCIe cars, each which can do 128 channels in out. And that gives us enough to do the top surround speakers and the way field speakers. So half of that, the Wavefield stuff, all goes to a different computer. So the Ableton computer talks to two different Wavefield computers. One does the long side, 124 channels, and the other does the two bookends, which is another 248 channels. And each of those computers has one patch per array, a max patch per array, which uses the spat object, which is the ear cam plug-in, basically. And that is doing, again, it's doing XY wave field synthesis. It's using the spat object to do that. in a pretty basic way like we make all the filters for it depending on how big the array boxes are so we tell it where every speaker is we tell how big the room is and from venue to venue it's very important that like that footprint doesn't change so that we don't have to do that math we have to like measure the room every time we go into a different room so we set down the footprint each star is always in the same location so we get to a venue and it's like That first placement of that first star has to be like within the centimeter because we're gonna be off, you know as we set it up but like Once you get off too far, it doesn't do the magic anymore so everything is like locked together that whole footprint of the 4,000 stars and 382 channels of audio is just like we have to get that placement, right and Yeah, so we call them arrays A, B, and C. So one computer receives the audio from Ableton and does the spatialization in real time and sends it back out to the arrays over some switches, network cables. It's really nice because we actually only have to use, we have these tech flex cables. So the Dante amps are on the back of all of the boxes. We have two switches and all of the cables just go directly to the switches the amps are on board And so the deployment of these things is like super quick, which is awesome the locations we only have to measure once I'm gonna measure twice, you know, but like only once like I have to do the hard work of that and then the spatialization is happening in real time from a fourth computer and that is using Aliko ultra-wideband RTLS trackers, which use time of flight basically to just say like, here's where the person is, here's where the person is, here's where the person is. And that updates it 30 times a second. And that XY data, it has Z data as well, which we use for the lighting, but not for the sound. that gets shipped again, that gets shipped over to the Wavefield A, B, and C computers to say, drag this audio from here to here, and that's doing that at 30 times a second, so that instead of using a mouse to drag the audio around, which is the primary interface for SPAT, we're just saying like, let's use OSC to just take over the mouse and drag that audio around. And it's doing that for the lighting as well, so that the lighting and the sound can happen basically at the same time. So as you're walking around in the first version of the show, the narrator's following you around through a real-time wave field object, but also the lights are following you around as well. So it's like, it's cool. It's like once, in theory, where we're like, I think this part can happen, I think this part can happen, I think that part can happen. And when we first actually turned it on and it worked, it was like a real, you know, it worked like 70%. It was like, whoa, this is cool. This is gonna be cool.
[00:38:09.832] Kent Bye: And what was the name of the open source software?
[00:38:12.936] Andrew Schneider: mean we use max msp which i don't think is open source but like the spat object is free and the spat object from your cam comes with so it's short for spatializer and it comes with so many in-built things that and like we were using it so much that we were able to send them feature requests and be like hey can you actually like make a filter that like is like a little more granular than the filter that you already have or like you know can you allow us like to not have that filter and like to make a bigger filter. Cause basically the sound object, it's like a cantaloupe sized sound object. So like you put your ear into one side, like it really does feel like that's how big the thing is. And then the more improvements they get, like the smaller that object becomes actually. So it was like a golf ball sized audio object. And we're like, that's too, that's actually too granular. Like, can we, let's make it the cantaloupe size thing again. And they've been great in like just talking to us and helping us figure out. Cause you know, I mean, we're not, we're good at what we do, but we're not like, we don't have PhDs in like wave, like, you know, fluid dynamics or anything like that.
[00:39:19.720] Kent Bye: Yeah. Awesome. I appreciate all that technical breakdown. Cause you know, it was one of the most incredible spatial audio experiences I've ever had. In fact, it's like right there. I mean, I can't think of anything that was more impressive and immersive and I would just wish I could see the kind of interactive version just to kind of have that full experience. I think part of being in one place, you get a sense of a sound field. But then at that point, it feels like more of like you're in an ambisonic experience rather than being able to really hear the dynamics. So I think because it's in the dark and the affordances of how you're theatrically staging it, it doesn't really afford for me to kind of openly roam around quickly to kind of really hear the specialization of the sound. So I feel like being in a one spot just ends up creating a sound field which could be done with kind of normal ambisonic array or any VR experience. But if we go back to the kind of story that you're telling, because, you know, like I said, we were talking around endings and, you know, there's some moments in your piece that just sort of like build up to this crescendo that is so moving that it just sort of brings emotion back. Just even thinking about it, the context of a conversation beforehand and then leading to that. So just wondering if you could kind of elaborate on some of the story and themes that you wanted to explore in this piece.
[00:40:35.816] Andrew Schneider: Yeah, it was. I mean, I guess the first, you know, you're asked to sit down and, you know, it says we're like, hey, we're going to be here for a little while. So why don't you sit? And then one of the first things that happens is like the narrator just starts listing the time that it takes to do things like. 45 years or like 16 days or 72 years and then he says of being alive like like 15 732 days being alive one day not and then like three months driving the street in front of your house 17 hours pressing the elevator door close button like nine months brushing your teeth And so it's not like he doesn't say like this is what happens in a life, but you put it together. You're like, oh, this is if you took the things out of sequence and put them all together, like maybe this is this is how much time I would be spending doing these things. and it's like there's some funny stuff like you know 92 hours watching someone else's movie on a like in-flight movie which is to me it's like I do that all the time I watch someone else's in-flight movie and it just and then it's split like the audio really splits and like the narrator becomes like 20 narrators and like And the lights are going crazy. So you almost get the sense that like every piece of light is like an event that has happened in your life. Like 25 hours pointing at the wrong place on a map, you know, like 15 minutes misspelling your name on your email, like just stupid stuff and profound stuff like 12 hours giving birth. And it just, so many overlapping, hundreds and hundreds of things that could make up a life. And then it becomes very, very intense sonically. Because it's so dim, it becomes almost overwhelming like visually because all the lights are coming on and off at the same time. And then right when you get to the top, all the lights go out except a single light. And the narrator says like, you know, 17 minutes giving CPR. like 45 hours hugging them goodbye one last kiss on the forehead like one last time sharing an ice cream cone like one last time you held their hand and so you're really asked to like think about all of these things happen and it's overwhelming all these things happening but there's a last time that each one of those things is going to happen which is like devastating to think about but also like for me it's what makes it all worth it is like yeah, there's going to be a last time, but there was also a first time, you know, and there's going to be so many times in there that, um, and we don't, it's like, I, I'm obsessed with, and I do focus on the last time, but just because of the loss of my own life, like, I think about those things all the time. And every time that that happens, every time I sit, like I've watched the show a thousand times and every single time that moment happens, I'm like, I start to cry just because it's still true. Like it doesn't become not true. And, um, Yeah, it's, uh, it's, I don't know. It's, it hits for me. It hits. And I, and like, because, because it's triggering me to think about my own life, I'm not following someone else's story. I'm thinking about my own life. Um, And so the lights and the sound like they keep representing different things like moments in your life or the ends of things. And after that, you're just asked to think about the last time that everything happens, like the last time that the last time a child grew up without a cell phone or the last time that a tree will grow on the planet Earth, the last person to remember your grandfather and the last person to remember you. I mean, all those things are real things that are actually going to happen. which is again, it sounds like well, what a downer of a piece but it's It's saying like these are real things and if we can think about them These are all things that we share, you know, like everyone I've ever known is gonna die and like it's awful and sad but It's also it makes me feel closer to them now in this moment And and so it's not a doom and gloom piece it's like what if we could be present and not think about those things that I just listed actually, like, cause those aren't the important things, you know, like the important things is like being here in this room now and like, and where can we go when we get out of this room? Like, can we reappreciate pointing at the wrong place on a map? You know? Yeah. Whew.
[00:45:24.290] Kent Bye: Wow. As you were recounting the story in that narrative, that moment was such a visceral, cathartic, emotional release. And you just even recounting the story bits, like, took me back to what it felt in my body to be there with the sound and with all the lights. I'm going to put a spoiler warning at the top of the show to just say, you know, for sure, watch this piece. If you have any opportunity to see it, go see it. Cause there's something around the magic of that moment and not knowing what to expect. Cause I didn't know it was like a wayfield synthesis piece. I didn't know like the spatial audio component at all. Blake was just like, Oh, it's going to be awesome. Um, but yeah, just you recounting it because yeah, it's just like this reflection of, um, our own mortality the people that we know and love and um yeah just it felt like a profound moment of grieving and with everybody there also it's sort of a invisible grief ritual because we're all going through our individual experiences i can't see anybody else but i know that when talking afterwards like everybody that i was with was also like crying in that moment so So it feels like this kind of invisible grief ritual where everybody's still able to be in their own space and not really worry about what other people are looking at. So it just feels like this really private, cathartic grief ritual in a way that was just really profoundly beautiful. And I was like... I have to find a time to do this interview. I had basically one slot left in my time here and you happen to be available to be able to like come at 9am, the one, you know, the earliest slot. So I just really appreciate, you know, you coming out to be able to have this conversation. Cause I was like, I have to have a conversation about this. This is so incredible. So yeah, I mean such a beautiful piece and, um, what's been some of the reactions you've received so far for people seeing it?
[00:47:22.047] Andrew Schneider: Yeah, it's, It's been the best reactions. The piece is primarily played in theaters because I'm a theater maker and so I have the connections to get it with theater curators. And then it goes to theaters and people know what they're expecting from the theater and then they go see it. And this is the first time that it's really been able to be shown to more XR people and more tech-minded folks. And it's it's like I don't want to compare the two, but there's been like the appreciation for what we're doing has been like profound to me because I think, again, because of expectation, like we're really not. OK, so it is cool. It looks cool. Like you can Instagram this thing. you know but we ask you to keep your phones outside so that that's not the primary experience like you want to but you can't because you know that you would disturb somebody else and like that also like in builds like a little layer of care that you have like you said before it's like you're in there you're alone because you can't see anybody but you know other people are there you know so you get to be alone together with those people Um, yeah, people come out and there's like, I mean, sometimes it's just like, I just love the descriptions. They're like, I felt like an egg in the bath of the universe or like I wanted to shave my head and like live in there. Like the most profound experiences I've had are, um, people come up to me or someone on my team and they just like have tears in their eyes and they just like kind of look at us and like I look at them and then like we both start crying. I mean like when I was making this piece my dad died and I was so excited for him to see it and he never got to see it. Um, I know, I mean also like he was in the process of dying, you know, so I don't know that he would have like wanted to go there, um, necessarily. But, um, Feel it's like it that influenced it so much that personal loss And like there's some other things in the piece that say like Out of the trillions of possibilities like you found me and I found you which is like You know like all the other people that we've lost in our lives and I just feel like everyone has that that person or like everyone has some version of that experience and so people come out and just like there are people who I can tell that they've lost someone or I mean even without saying something like we just sometimes like people come out they have tears in their eyes and we just hug you know and that's like that makes me know that I'm doing that there's been a lot of tough choices in my life of like should I keep doing this because it's hard it's a hard thing to do or should I like move into something where I have like health care or financial stability and and i'm like no i made the right choice because this is the thing that is the sharing this and because like it's also open enough experience where you can it's a container for your own grief or rage or joy or whatever that you get to put into it whatever you want to and so i think that people also just you know there's some people that come out was just like i just needed that at the end of my day i just needed to like Some people were like, I didn't listen to a word you said. I was just in there, you know? And like, I'm sure they listen to some words, but that's the other thing is like, you don't have to follow a story. There's no story to follow. It's a story of you. So it's a story of your reactions in there, which is again, is, is another profound thing to me. It's just like such a, I feel like I'm trying to be generous with the experience I'm providing. And in return, like, everyone who goes through it is so is like just opens up and it's just so generous upon exiting and you know like people come out and they turn their phones back on and they're like they call the people that they know and they're like come see this which is like that's as a maker of something you're like this is the best gift i could have ever asked for you know like it's exactly what I wanted it to do, which is like where people just want to be in it and, and bring their whole selves to it, you know? Cause it's about them. Yeah.
[00:51:46.805] Kent Bye: Awesome. Well, it's a profoundly beautiful experience and I highly recommend anyone to go see it. If you've listened to this, sorry for the spoilers, but it'll still like the actual embodied experience of it is way beyond anything we could describe in words right here. So, um, Yeah, I guess as we start to wrap up, I'd love to hear what you think the ultimate potential of all these kind of, you know, mixture of theater and technology and what we are calling immersive experiences and immersive art and what you think that might be able to enable.
[00:52:15.580] Andrew Schneider: Yeah. I mean, I was thinking yesterday about like virtual reality or like XR or things like that. And I've always thought it was funny that virtual reality was always like trying to get more lifelike, you know, or like theater tries to get more lifelike as if the ultimate goal is to like just be this, like us here sitting in this like lobby of this hotel, like, you know, like seeing each other and talking to each other. Like, immersive experiences will come around full circle and become so immersive that they're indistinguishable from reality. Or, like, you know, AI will provide experiences that are indistinguishable from reality. VR will provide experiences that are indistinguishable from reality. Except for the fact that we can curate them in ways that, like, real life isn't curated. I mean, that's why I love the audio project so much is, like... If you get yourself into the right mindset, you can provide experiences for people where they're a little bit complicit in their own experience because they've come to you to have an experience and they say, here's my ticket, please provide an experience for me. But I hope that as experienced designers writ large, like VR makers, XR makers, immersive makers, That it really, to me there is no like virtual experience or synthetic experience. Everything is a primary experience. So like when I put on a VR headset, my first thought as a maker isn't like what is the story of the protagonist? It's usually like am I going to do this right? Or like do I look stupid? And to me, like that is primary experience, similar to when I go into the experience that I made is my first experience isn't, oh, now I'm immediately thinking about like the cosmos and the stars. It's no, it's like, am I going to bump into somebody like and will I fall down? But the show is about those things, too. It's about will I bump into somebody and will I fall down? Like both metaphorically and definitely like not metaphorically. And to me, like I think that that is the great potential in like experience. I would just wanted to say experience manipulation, which may have a negative tone, but like I really do think like as experienced designers, if we just think at such a minute level and so deeply about the actual primary experience that's happening, the actual received experience of the audience rather than what we want them to receive, that we can like provide incredibly empathetic scenarios that like that's why I was interested in theater in the first place. It's like there's an actor over there and they're acting out a part of you for you so that you don't have to do it yourself. You don't have to go down to the pits of yourself like they'll do it for you and you can see a bit of yourself in them. And, you know, there's there's like the Broadway version of that and then there's like the homegrown like super like small version of that and I don't think that those things are in opposition I just think that like one allows you to like get really deep in there and think about how to provide bespoke experience. And maybe that has to be for smaller audiences at a time. But I'm excited that like, I think that because we're getting more fluent in these new technologies and in like immersive, so it's not just, does it work? Is it cool? It's really like, what is the nuance of this story being told? And because the public at large is becoming more fluent in these technologies, there's less of a, do I look stupid? Am I going to run into something? And more of like the classic theater model, which is like, I know what to do in a theater. I sit in my seat and I look over there. I think that the more, the more immersive experiences think about audience care, the more that immersive audiences will be receptive to like going along with every bit of nuance that the maker can put out there.
[00:56:02.867] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:56:07.749] Andrew Schneider: I just think that everyone is doing such cool stuff. Not everything is for me, but the fact that the community exists and the fact that everyone's making things that are bespoke for the audience at large is so moving to me. like everyone is saying here i made this for you which is like so beautiful to me like like i didn't make this for myself i didn't make it because i wanted i mean there's a bit of like oh i wanted to be cool but it's like i made this specifically for you you know like one person at a time you know or small audiences at a time and it's people are really thinking about the primary experience of the individual not just entertainment not something that's happening over there but like something that's happening to me with my own neurological apparatus right now. And I just love that people are thinking about that. I'm just super grateful to be in that community. And especially coming from the theater world, I'm super excited that there's been such a nice response from the immersive community. to something that I felt like I didn't have access to before. And now I feel like wholeheartedly welcomed into like as a maker, I just, it's like my dream come true.
[00:57:25.514] Kent Bye: Awesome. Well, Andrew, thanks so much for joining me here on the podcast and you know, just to the sequence events of what happened last night was like, I saw Roman Rapix latest iteration of his wristband and his pivots and spatial audio experience mixed with VR with a 13-speaker waveform synthesis array where I was in VR listening to it. I went to go see your show and we're completely in the dark, not having any visual stimulus other than these 4,000 points of light and this grid and cube and this 382 waveform synthesis channel array. The fidelity of that experience and then to come back and to see the same experience from Roman without the VR and just listening and the way that I could just hear but how much still my visual field was dominating in some sense but the way that you've created a context so that you are dialing down that visual processing power to do very minimalistic and then that raises your awareness of all the spatial audio I think it's sort of a very interesting like contrast of that like I discovered something new around like hey maybe VR is not like a great component to spatial audio experiences just because of how much it's dominating and I can't actually process what's happening so it's just kind of like this really interesting contrast that I had if you have any comments I love that and I don't I feel like thank you for saying that because I don't think about that enough but
[00:58:47.671] Andrew Schneider: Yeah, I feel like we're a little bit away from the audio getting... It's like, I do think that the visual field dominates the audio field. But the visual field, because it requires, like, it goes through your brain. Like, when I was working with Bobby at the Worcester Group, like, he was the sound guy, I was the video guy. He could, like, make jokes in the room by playing, like, audio clips, you know? And, like, you don't have to be looking anywhere. And you don't have to be focusing on things. It just, like, it bypassed your intellect. The audio went straight to your brain. And I think it's true that like your brain is constantly taking in visual data and like process. I don't know if this is actually true, but like it's to me, it's like it doesn't bypass your intellect. It goes into something that like takes more time and so takes more brain energy. And I'd never really thought about how important that is. And like you said, like we're low resing the field of your experience in the stars. It's just like it's you know, it's super low bit visual field. But the audio field is like so rich that I feel like it's finally equalizing that awareness. So you're not being driven by your intellectual brain. You're being driven by like your primal brain reacting to like. you know, the literal vibrations of audio entering into your brain and your body.
[01:00:04.449] Kent Bye: and just the way that the story that you're telling is using that kind of low bit creating these archetypal shapes and forms that are like like you said like an 8-bit video game you know this is sort of like an 8-bit type of visual experience and physical reality that is very minimalistic and still like a spectacle to some extent and as it's moving quickly dominate the processing of the mind but at the end of the day the heart of the story that you're telling comes through loud and clear and they're really working off each other in a way that they're They're amplifying the deeper story that you're telling on loss and grief. I just thought it was incredibly powerful and moving and jumped to the top of my list of different experiences that I saw here this week. So thanks again for joining me today on the podcast to help break it on down. It was a real treat and pleasure just to hear a little bit more about your process and journey in creating it.
[01:00:48.294] Andrew Schneider: Yeah. Thanks so much for having me. I mean, it's, it's a, yeah, it's such a joy to like, to realize things about my own process in talking to you. So thanks for the, thanks for the generosity. Thanks for the questions. Thanks for, I mean, thank you universe for the serendipity of like you having the experience that you had last night. It's so, it's so cool. Awesome.
[01:01:05.816] Kent Bye: Thanks for listening to this episode of the Voices of VR podcast. And there is a lot that's happening in the world today. And the one place that I find solace is in stories, whether that's a great movie, a documentary, or immersive storytelling. And I love going to these different conferences and festivals and seeing all the different work and talking to all the different artists. And sharing that with the community, because I think there's just so much to be learned from listening to someone's process to hear about what they want to tell a story about. And even if you don't have a chance to see it, just to have the opportunity to hear about a project that you might have missed or to learn about it. And so this is a part of my own creative process of capturing these stories and sharing it with a larger community. And if you find that valuable and want to sustain this oral history project that I've been doing for the last decade, then please do consider supporting me on Patreon at patreon.com slash voicesofvr. Every amount does indeed help sustain the work that I'm doing here, even if it's just $5 a month. That goes a long way for allowing me to continue to make these trips and to to ensure that I can see as much of the work as I can and to talk to as many of the artists as I can and to share that with the larger community. So you can support the podcast at patreon.com slash voices of VR. Thanks for listening.
