#1376: Indie Musician Roman Rappak’s Annual Mixed Reality Performance Experiments & Expansion into “Detachment” Immersive Story

I interviewed Detachment director Roman Rappak remotely after the SXSW XR Experience 2024. See more context in the rough transcript below.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So continuing on my series of looking at different immersive experiences at Southwest Southwest 2024, today's episode is with Roman Rapik, who had a number of different pieces that were being featured at Southwest Southwest this year. So one, he had another performance and this is like the third year in a row that Roman and his band has done some sort of live music performance with mixed reality components. Previously it's been with this band called Mirror Shot and now he's got a new band called Pivots, but he's continuing that same type of exploration for doing like a live performance and having like a small group of 15 to 20 people using these virtual reality headsets with mixed reality pass-through. He's also got a startup called BristBand, which is starting to use Unreal Engine to recreate the spatial context to have music events and kind of a platform for independent musicians to explore the affordances of XR. And then finally, this year, he's got a immersive story that's being featured within the context of the competition called Detachment, which is starting to look at the XR industry and deconstruct it in certain ways and exploring the different genres and the affordances within VR as a medium, but also do the satire commentary on the medium, but also at the same time, use the medium to feature his first song from his Pivots band to explore some of the affordances, how virtual reality can start to become a music video. This is a piece that they're in the process of hoping to release sometime here this year on Steam so you can check out this independent musician approach to XR. So definitely recommend checking that out once that becomes available. So lots of different views from an independent musician for how they're starting to use XR technologies to find other ways of making it as independent artists. So, that's what we're covering on today's episode of the VistaVR Podcast. So, this interview with Roman happened on Sunday, March 17th, 2024. So, with that, let's go ahead and dive right in.

[00:02:10.724] Roman Rappak: My name's Roman Rapak. I am co-founder of Wristband, and I also play in a band called Pivots, which we just launched. Wristband is a startup that looks at the way gaming technology and technology more broadly could be used in live events, in music, in anything that brings people together in the physical space.

[00:02:35.534] Kent Bye: Great. Maybe you could give a bit more context as to your background and your journey into VR.

[00:02:39.945] Roman Rappak: Sure, so my journey into VR has been, I suppose like everyone's, a meandering one with twists and turns. My background, as I mentioned, is that I'm a musician and I play in bands and I'm a producer, so I record music. I graduated as a filmmaker at London University of the Arts and focused on sound design and directing and then I started directing music videos for labels in the UK and that all sort of merged together when I was writing some music and one of the music videos I was directing for a major who were all main nameless suddenly as is the case with major labels the artist got dropped and they basically said look we can only pay you half but you can keep this video so the first thing I ever launched with the video that we put out and started an adventure of touring and playing music I did that for I guess about five or six years and got signed a five album record deal and did all the sort of things that when you're 14 and you decide you want to be in a band the other things that you always thought that would be my dream come true and it was right when when the music industry was probably It had been going through its crisis, but it was at the point before people started finding a way out, let's say, for monetization was sort of worked out. And it was a difficult time. It is still a difficult time to be a musician or an artist. But at that time, it was completely impossible. And so I was watching everyone leaving the music industry and noticed that a lot of the really talented people who were really frustrated with how the music industry didn't seem to be doing anything innovative and kind of dying, started moving into gaming, moving into technology and I started to think well is there not a way that we as musicians, labels, A&Rs, radio pluggers, event promoters, instead of complaining about the way that things used to be better Is there not a way that we could use all these new technologies to create something interesting? And so I put together a new kind of live concert, which would be something that originally we thought would be a kind of a secret event that we would do ahead of our show. And that show involved people seated in headsets that would switch between the reality of the show and using the pass-through camera for the more technically-minded listeners. So the pass-through camera would show the physical event. You would have the elements that make a physical event so exciting, which is the person you're sitting next to, the response of the crowd, the uniqueness of a human being's performance on stage. but then also switch into VR at certain points in the track. And crucially, to do this as a multiplayer experience. So it wasn't something where people were on their own in a VR experience. They would switch from being together in a physical space to together in a virtual space. And that began the new part of the journey, which I guess leads us to today. where we've been able to actually tour this, I guess you could call it mixed reality to a sense, this immersive live concert around the world and have been able to sustain our careers as musicians much better actually than the musicians who would be looking at how many plays did we get? Do we get a sync or does our music get licensed in a commercial? And it's been amazing because we've been able to be a band, we've been able to be artists, But we've also been part of this movement, I suppose, which is constantly evolving, where everyone seems to share ideas, where there's a community, which, if you're from the music industry, is a very different atmosphere, because in the music industry it's kind of the opposite. There's the people who are already there, tend to pull up the ladder and it's in their interest not to let too many people up. It's saturated with so many, I think 100,000 tracks uploaded to Spotify every day, so it's incredibly difficult to ever find an audience. Small venues, especially in the UK, are all closing down. So the original kind of rags to riches story of, well, we were just a bunch of friends, and we got together in a studio, and then we played a little show, and then our friends came, and then we played a bigger show. And then one day, someone from a label said, hey, kid, do you want to tour the world? And off we went. That's kind of dead now. And maybe there's a cynicism that's set in the music industry, which is, oh, it's all down to, the fact that everyone's on social media, no one goes to shows, no one buys albums, and artists can't find, not only can they not find their grassroots audience, but they can't do what they have traditionally done, which is just get really good at playing and play these small circuits and perfect their craft. This is a sort of cynical perspective that you just have to have a million followers on TikTok. And you know, there is some truth to it, but I think there is also, there's an argument for not just throwing your hands up and saying, well, that was that, that was music and it's dead.

[00:07:41.149] Kent Bye: I've had a chance to check out the previous two South by Southwest in 2022 and 2023, where you were showing this mixed reality concert. I wasn't able to make it to South by Southwest this year. And so I only saw some videos where you were able to do a live performance again with a new headset, first year with your VR and then HTC Elite. And then now you have what looked like to be like a Pico headset. That's right. Maybe first before we start to dive into some of that, I'd love to get a little bit of a follow-up as to like, originally you were under the name and umbrella of MirrorShot. So what happened to MirrorShot and maybe what is the new bands that you're operating under now?

[00:08:16.453] Roman Rappak: Yeah well Mirror Shot was a band where in a way we had one foot in the old world because we'd signed a record deal, we had an advance. One of the things about operating in a new medium and a new environment is a lot of the rules do need to get thrown out of the window and a lot of the kind of rituals and in music as soon as you do sign to record deal as much as the label that we were on were one of the more innovative it's still down to okay great this vr thing's great this this idea of embracing this new technology is great but we're in the business of selling records and we're in the business of getting streams and so it put us in a in a way in a constant push and pull of well for us this is well perhaps this is how bands are going to evolve you know perhaps music will evolve in this way that this idea of looking at IP and copyright and royalties and that was traditionally how bands would be able to sustain their careers and pay their rent Maybe there's a new way of doing it and obviously that's at odds with a business who their business is okay Well, we actually need to sell records. So so the idea was okay. We've been getting everything ready. We're a very small I mean from the startup that we launched wristband We're a very small team. And it's interesting how many parallels there are with being in an indie, in the sense of independent band, and in an independent small startup. You're living by your wits and innovation. You're trying to do things on a small budget. You're constantly trying to compete with giants. So in the music industry, if you're a small independent band, there's no way you can compete with the kind of stage production that Coldplay or the 1975 or Taylor Swift will have. But you can also use technology in a new way, and you can build a fan base, et cetera. So the idea was to launch, at this South by, was to launch a new band, which is a band called Pivots. We have one single out, and not do any of the traditional release schedules, tour cycles. And really, we've proven, I think, that with Wristband and with Mirror Shot, well, actually, it's possible to exist outside of the traditional music industry business infrastructure. And it felt like, OK, well, all these things have changed and we've perfected the kind of the show and the structure of how we do it. So how about going all in and saying, is it possible for a band to issue all of the way that music has always been done and to embrace this fully and go on tour with using VR, AR? Well, there was an installation in this year, a spatial audio installation. And South By was the launch of that experiment, I suppose.

[00:10:56.890] Kent Bye: Okay. And so there's, I'd say three major strands here. There's the mixed reality live performance element that at South by Southwest is really blending the technology side of the conference and the live music performance side. And you've really been straddling those two, playing a music gig at one of these venues and having a mixed reality component where people can come ahead of the gig and have this whole immersive experience that I've had a chance to see the previous two times. Then there's the wristband, which is your startup that you're doing to have more of a virtual world kind of trying to figure out new ways that bands could connect to their audiences and create these immersive experiences. And then you have Detachment, which is like more of a standalone VR piece that has some of the elements of what I've seen before with this kind of flying through these spaces and having music videos, but there's also a whole other dimensions of narrative components that you're starting to add in. And so I'd love to pass it over to you to update me on each of these three strands of both the live performance, wristband, and what you've been up to, and also this piece that you were able to show in the context of the competition of the South by Southwest XR experience.

[00:12:02.366] Roman Rappak: Sure. So the live performance side is actually, you described it perfectly. It's not about replacing what a concert is. It's about saying there is an incredible technology and also a desire people have to experience new things, which I don't think existed as strongly about four or five years ago. We were often seen as a kind of a interesting, weird art project. And we would get booked for things like, you know, a warehouse in Hamburg that was doing interesting experimental things. Or we would get booked at places like conferences where everyone is looking at what the technology might be in five years that people might start using. And now I think that there's a language of, you know, you only have to look at things like how the Vegas sphere and the ABBA voyage experience. The idea that a live experience or music experience and technology, and let's say like an experiential element, are something that is kind of part of everyday life, I think is much more pronounced than it has ever been. And so the idea is also that, well, All of those things, you know, the things I've just mentioned, Voyager and Vegas Fear, these are like billion dollar things. You know, if you're U2, you can use it. If you're an established act like Abbott, you can use it. And I've got a love for, I think the history of music, you know, what's the Mark Twain thing, that it doesn't repeat, but it rhymes. I think that there's a lot to be said about the fact that in the history of music, there's always these moments where the fact that the cost of something comes down, the fact that the accessibility towards using these tools, the accessibility is easier, then it gives rise to an entirely new scene or an entirely new genre in some cases. I mean, synthesizers are a good example. If you look at when synthesizers first started getting used, they were an instrument that prog musicians used, you know, stadium prog bands, because these things cost more than a house. And they were unwieldy, and you had to have an entire team to tune them, and you had to have a special truck to bring them along. And then look at what happens when they become small enough and cheap enough so that everyone can start using them. You get post-punk, and you get artists using them. You get sort of no-wave New York. You get, uh... bands that can start using them and evolve an art form and then you eventually get dance music you know a lot of the earliest dance music was made on like the roland 808s etc which were originally meant as small versions of complex machinery that were suddenly being put in the hands of everyday people so So yeah, so that's why the VRAR experience, the immersive music experience, is so important. Because it's trying to say, look, we don't have billions of dollars, and we haven't got a Vegas sphere, and we haven't got LED walls, and yet there's an enormous queue of people that can't wait to experience this thing. And we turn up on a flight with everything that we need in the overhead luggage, and we can set up in 35 minutes. and it packed down in about 40 minutes, which, if you're an independent musician, you know how important that is. So that's the first part of what we were doing. The second part, which is wristband. So Epic Games found out about what we were doing, and we started looking at how, as well as this happening in the physical world, you know, we're using game engines. The idea was always take the physical world and a physical concert and augment it with virtual reality. And then we started realizing, well, look, these are things that are happening in a game engine. And the nature of this technology is about transcending time and space and saying, well, what happens if this thing can also occur in a virtual world? This isn't like the Fortnite Travis Scott explosion of people who are in a game and meanwhile in that game an avatar, a pre-recorded avatar plays a song. This is much more about creating a world where you could walk through it and discover new artists and these are artists that are playing in real time. using you know the earliest stages we were using very basic videos of what was happening in that concert but as we're moving forward and things like condensed reality and move ai are getting nearer and nearer to real-time volumetric capture and point cloud capture i suppose for virtual world then the idea that there would be a form of a Fortnite or a Grand Theft Auto, which is a street in Austin, and I could go and I could discover new bands, not even necessarily in a VR headset, just in a game. I think that that's something equally that it's logical that that's where it will go, gradually being able to have a real-time digital representation of a space, well then how many A&Rs and how many music fans and how many labels couldn't get to South by but would love to walk down that street and go oh my god there's this guitarist that is absolutely exploding from Sao Paulo I'm going to go in there and I'm going to not only watch how they perform but I'm going to see the audience reactions I'm going to see what kind of room they're in so that's what we've been building with wristband. and building in Unreal Engine, which the interesting thing about that is we've started to realize that in a way we're the kind of glue or the testing ground for a lot of these technologies. We've worked with Copyright Delta, for example, is a really good example. They've got this incredible blockchain-based solution. It's almost like a Shazam combination where you could listen to something and the artists who've performed on that track will get paid by a smart contract And, you know, this is for me, it's a kind of a killer use case for blockchain, because blockchain has to solve a problem. And that's one of the most difficult things. It doesn't really solve how quick can you pay for something on a credit card, because it's very slow. But what it can solve is something that is really complex. At the moment, if your music gets played on the radio, then it takes about seven months for the revenue collection societies to pick up that money, to send it to the right people. And there's loads of money that goes completely missing, that just gets put in a big box because it's unclaimed. So the reason I bring those guys up is because for them to try and get Ubisoft to use their blockchain solution is completely impossible. So you have all these siloed tools, the same with condensed reality. You have these amazing tools that actually work, but they don't have a kind of laboratory in which to try them in. So that's kind of where Wristband is now. And then I'll finish with what Detachment is. So Detachment is a standalone VR experience in the sort of, I like that we can now say traditional, you know, ye olde VR experience, which is, you know, it's an experience. You download it like a game from Steam and you are on your own in your house and you experience this piece. I originally, I have to say, I was hesitant to do it because the whole point, you know, you will hear from how I've described things so far, was that I think that as much as this is a great way of experiencing VR content, I had much higher hopes for what this would be, that it wouldn't just be Oh, it's like games, you know, you just go on a thing and you download a thing. Have you tried that game? And I actually saw a YouTube video this morning, which made me think of Voices of VR and made me think of You Can, which was, it's been four years since Half-Life Alyx. Four years, and Half-Life Alyx is, without a doubt, a complete masterpiece. It's an incredible game. It is a game that could only exist in VR. You know, and obviously it's built by an enormously powerful, wealthy company. But I kind of still feel like, well, is the whole of this thing meant to be that you go to Steam and you try a thing out and someone says, have you tried that one? And you go home and sit on your own and you put a headset on and you try the thing out and that's it. And that's as far as it's gonna go. Obviously, I don't think it is because I don't think I'd work on it anymore if I thought that was all there was. But it's interesting to see how, from my film background, the way that VR is presented, even at festivals, and obviously I love South By, I'm a huge fan of the way that it's showcasing new and emerging pieces, but I started to wonder, like, well, This is exactly like the things that you do in the independent film industry. You just go to a place, they say, here's the indie films, one of them or two of them win a prize, and it's good for the careers of the people involved, and then everybody goes home. So the idea of me turning this thing that was a show, that it's all about being there, it's all about being together, felt like we were diluting what was exciting about it. And then fortunately, Blake Camadina, as head of XR at South By, highlighted how arrogant and stupid I was. Well, he didn't put it like that, but unconsciously he did. And he said, look, just make it into a piece and submit it because it's interesting and people want to see it. And some people want to see it in that context. So I realized I was being a bit of a pretentious artist saying, no, no, no, I insist it can only be shown in this way. And I thought it might be an interesting challenge actually to say, well, could we do something that is gently subversive in the sense that it's doing what we're trying to do with the show, which is to kind of kick the tires and kind of try and see what the medium can do, but also optimistically maybe make a comment about the state of VR and maybe gaming more broadly.

[00:21:06.867] Kent Bye: Interesting. Yeah. That makes a lot of sense. And as you're talking a lot about how VR has been so much about gaming, I think a lot of it up to this point has been a lot of how Meta has been curating their different apps. I did a whole survey of the 620 or 30 plus VR experiences that have been released on Oculus Quest over the past 4.8 years. And it was around 70% games and the rest were split between apps and entertainment. that's actually flipped on the Apple Vision Pro where most of them have been apps rather than games.

[00:21:39.986] Roman Rappak: Very deliberately, I suppose, on that part.

[00:21:42.768] Kent Bye: Yeah, and I think it speaks to the frameworks of Apple Vision Pro are very much starting with 2D and then eventually moving into more 3D and all the stuff that's on the Quest has been mostly in the game engines of Unreal Engine as well as Unity. And those are game engines. And so it's much easier to create a game out of a game engine than an app that you might see on Apple Vision Pro.

[00:22:00.558] Roman Rappak: Yeah, that's a good point.

[00:22:02.957] Kent Bye: So I think there's other aspects of how the industry has been developing that have been dictated by these big distributors of these platforms. But it is true that these locations at South by Southwest and other festivals like Sundance and Venice Film Festival, IFA DocLab, Tribeca, all these places have their existing ecosystem of film distribution, but there's been an associated XR distribution that has by de facto adopted that model. But yet in the indie film world, there's actually a chance for some of those films to get distributed and out into the world. But yet in the XR industry, that's still trying to really figure out how a lot of these pieces, like the stuff that you're creating is going to have a life after South by Southwest is kind of mirroring how not all the independent films are going to get picked up. There may be 17,000 films that are submitted to like Sundance this year, including short and features and about a hundred to 200 if you include all the shorts are selected. And then of those, it's a small percentage that actually gets distribution. So it's already like. There's ways in which the music industry is worse in terms of it is easy to get it out there, but in terms of getting paid is a whole other issue that seems to be a driving factor for how you're exploring these new emerging technologies to both do the live performance, to do the wristband, and then this seems to be an experiment where it's sort of like I think of it as a music video, but it's also a way of doing a VR essay, which I don't know if you had a chance to see Shadow Time at South by Southwest, but that was one of the first examples of a piece that really felt like it was a VR essay that was deconstructing aspects of the medium of VR by using VR. And I feel like in a similar way, you're exploring the different genres and commenting and critiquing on the medium of VR at the same time as you're exploring the variety of different tropes that we've seen in VR. So yeah, I'd love to hear a little bit more elaboration on this evolution from what maybe started as a live performance, but then adding all these elements that Blake came to you and said, you know, actually what you're doing here may make more sense in this context of showing it in an exhibition.

[00:24:07.323] Roman Rappak: Yeah, and I think he was saying, what would it look like if you did it? You know, because the points are still the same, but the... Actually, there's two Marshall McLuhan quotes here, because in one way, the medium, in our case of the medium of being at a concert, does change what the message is. But another thing, and this is going back to our point about the mostly games point, is that McLuhan says that as soon as there's a new medium, the first thing it does, because obviously it doesn't have its own language evolved, is it simulates or reproduces things from a previous medium. So radio will be, hey, let's just mic up a theater production. And then TV is like, well, we used to do this in radio, so we'll just do that, but we'll just film the people doing it. Video games themselves, you know, as soon as technology was good enough, the holy grail was, it's like a movie. It's almost as good as a movie. You know, how good is the cut sequences? I feel like I'm in a film kind of thing. And then the second part is like, so, you know, this is for any gamers out there. This is not a, like, I love game, you know, I think games have been an evolutionary force in technology. I really love the fact that if you're talking about, OK, games need to be, you know, is it all just becoming games? And you talked about that statistics with Meta about how many of them are games. I think it's also worth mentioning that the entire Silicon Valley is thanks to games. You know, the gamers are obviously going to push this medium forward just by their sheer wallets, you know, and their sheer number. So I think it should be something, you know, shout out to all the gamers. I've got a lot of gratitude for them. being those sort of active things, because otherwise who would be buying headsets, you know? Why would Palmer Luckey have bothered gluing some screens to a ski goggle? It was all about games. It was all because they wanted to play games. And, you know, Atari as being one of the first Silicon Valley companies and evolving in a way what we think of as startups, you know, the kind of t-shirt billionaire, kind of counterculture aspect of it. So yeah, so for that reason, I don't think it's about saying, or I don't think, I personally think that there's too many games and that somehow games distract from what the medium can do. In fact, people making a really powerful VR piece on the Holocaust have a lot to thank for the billions of people that played Beat Saber. And I quite like the idea that something that was ultimately just using a lightsaber to hit some blocks means that the technology reaches that level of ubiquity, that it means that some of those people might go, I wonder what else is out there? And watch an incredible documentary or a VR piece. And that there is that kind of Trojan horse of interesting, powerful opinions and thoughts that can arrive on the doorstep of someone who just used to be someone who played Beat Saber. So yeah, so I think that the context as well of VR, it's interesting because people sitting in an audience and collectively responding to some live music, the way that they react to VR is totally different to when in a booth on the fourth floor of the Fairmont, I put them in a headset and they do an experience on their own and then they take the headset off and you do, you know, there's always that slightly awkward Did you enjoy it? How was it? And then they kind of politely nod and say, that was great. Or they have a, it's a completely different thing to do that with a group of people, you know?

[00:27:31.024] Kent Bye: Yeah. And so maybe you could describe to me a little bit of the mixed reality concert performance this year to help me understand a little bit of how that evolved. And then if there's any parallels between some of the assets that you were showing in detachment were maybe coming up in a live performance, because in the past, I remember that. There's essentially you're up on stage and you're playing the same song over and over again and maybe you have like 10 or 20 or 25 people that are able to come in and put on a VR headset and all these quotes ruminating about the nature of reality, the nature of VR as a medium, kind of a meta commentary usually that would be included. it becomes a little bit of like a ritual of you kind of going through the same song. And then some people get the immersive component, which is that you put on the headset and then you have a whole visual component where you have like mixed reality pass through, or maybe you add some effects. You then would switch into a fully immersive VR and then you'd fly through a space and then you would come out and finish the song and flip back and forth between what you were seeing on stage and what people were experiencing. And then you would actually have some ability to control live, like what people are seeing. So you could mix it up and make it interesting for you to have everybody have a little bit slightly different experience, but allow you to cultivate the editing as you know, your background and film editing, your editing, the experience while you're also playing. So yeah, I'd love to hear a little bit about what you did this year at South by Southwest. Cause there was another mixed reality concert performance that you did as well.

[00:28:52.682] Roman Rappak: Yeah, yeah I'm glad you talked about how there's the things that have been taken from the film industry and then we're talking about things that have been taken from the gaming industry and one of the things that we've evolved with the live performance is we're realizing that it's not like a new language is going to evolve from nothing that's going to be like nothing else we ever saw and you know no mediums ever do that you know cinema takes a little from photography a little from theater a little bit from music and the thing that it comes up with in the end, you know, you can't say that Apocalypse Now is some music with some pictures, or you can't say that, you know, that Dune 2 is some photographs with some talk, you know, that we talk about them as films. And I think the route that we're going to take towards VR experience, and we are already taking, is that a language is evolving, and it's actually a language that uses a little bit from user agency in video game and user experience design. It takes the same things from music it takes things from film and from composition and story arcs and so the piece was meant to be okay well how can we use what are the what's the kind of balance of which of those of the aspects of these things that we use so the audience comes in there's a queue where they find out what session they're going to do we had 20 headsets this year we did the first year we did i think we did a hundred The next year we did a lot less. Again, it's down to hardware and it down to, you know, we have been quite fortunate with headset hardware partners, but it is getting more difficult to just say, hey, we're doing this really interesting experience. Can we have a hundred headsets? So yeah, any headset manufacturers out there definitely reach out. but then they would be seated exactly as if you were sitting in a theatre production or maybe a sit-down concert experience. The music's already starting because we want to make sure that when they sit down they don't feel like oh they're just going to sit down and it will start soon. They should already be getting immersed while we do this and actually it's a really We found it's incredibly effective to have them almost go through a small maze before where there's tones and atmospheres. You'd be amazed by how when we haven't done that, people sit down and they're just talking with their friends and they're sort of people are shouting and someone's standing up and talking. Whereas as soon as you put people through a ritual of you're going through a thing, they go into kind of, oh, it's an installation, I'm an art, you know, that strange way that in art galleries, people's volume goes down because they're experiencing something together, even if they haven't consciously all agreed to do it. So they sit down, The headsets all get started already, the experience already gets launched before we even put them in because we don't want to sit there and go, okay, so you see that button, okay, you press, yeah, so we want it to be like, as soon as they're in these headsets, they are inside the experience. Another thing we've learned to do is a lot of people have never done VR experiences before, find it incredibly intense, awkward, claustrophobic, and we don't want this to be something that is a VR experience for the VR community. We don't use controllers because we wanted it to be something where your curiosity is rewarded by which way you're looking or what way you're facing. We also don't want to have to teach someone how to use a controller press up. I feel like all of that stuff really breaks immersion. Like as soon as you need to be explained something, about okay so you're going to see a box okay so press the box no don't press that because it will reset all that stuff as soon as you have that as soon as someone's in it they almost feel like they're doing a test you know how well do i respond to the instructions so the whole thing is meant to be as easy as possible and what we found was because we have to get 100 people in headsets while the music's going and every you know the first person who's put in their headset technically has to wait until the last person is put in their headset so what we do is we set up a kind of a tutorial so that as soon as they're in they're doing something and that tutorial is the uh in the school of Shigeru Miyamoto, game design, is quietly teaching them how to fly, how to interact in this. And then on stage, I have a button that launches the experience. So it's launched exactly in time with the music and everyone's flying over the same landscape together. They can see each other as small avatars within the space. And the first time we ever tried doing this years back, we just thought, oh, we'll put them in VR experience. It will switch, pass through, and we'll just play a song. It'll be like a music video. And everyone started coming out saying, it didn't really get what the story was. And my first reaction was, well, I mean, the story is the band, the music. No one goes to watch a guy playing a guitar on stage and goes, I love the show, but what was the story? I didn't get the story. So we realized, wow, people have this need for a narrative. You can't put them in these worlds. And people started inventing their own narratives anyway, a little bit like in Fortnite, there is this environment storytelling, where if you speak to my nephew, my nephew could talk to you for six hours about the lore of Fortnite, and it's never explicit. It's like you go in one month, and there was this tunnel, and then you go in the next month, and that tunnel's turning into this, and you talk to your friend, and they say, oh, probably what happened. It's amazing how emergent narratives kind of evolve just from that kind of storytelling. So we thought, okay, well, let's have a voiceover actor and that changed absolutely everything because first of all people are it's all about what they're familiar with in terms of mediums when we've done it in cinemas it works so well because people are so used to the ritual of the cinema you know it's 100 years old you sit down there's red chairs faint smell of popcorn curtains go back and I know exactly what to do as a viewer VR experiences are slightly different because, you know, how am I interacting? What am I supposed to do? As soon as you have a voiceover actor, people just settle back because they are in tens of thousands, millions-year-old idea of storytelling, let's say. And where the music sits is a little bit like, well, we're kind of borrowing from movies. You know, I mentioned Apocalypse Now. It might start with the doors. and jungle blowing up which tells you a little bit of a story of the mood being set by the band of the visuals evoking a certain time and a place but the next thing you see is that rotating fan and a voiceover and so all of those things work together to create this mood and this atmosphere. So the band only really plays, in this 10 minute experience, the band only really plays two sections, a little bit like you would in a movie. You would get the chorus of a track as there was a specific scene. And the idea is that if you have that moment where the band starts playing, people responded so much better because they've been in story mode. So when the song kicks in, it's a bit like in your favorite movie when there's that classic track or, you know, like the slow motion walking in Reservoir Dogs where Little Green Bag comes on. If I listen to Little Green Bag on its own, it's great. But when I watch it in that movie, I feel that song so much more. My aim is basically that, you know, all of these components acting together to create effectively something that is very simple and very primal, which is a show where the audience has a collective series of emotions that they go through and they go through them together.

[00:35:30.999] Kent Bye: Okay. And so as you were describing all that, I was kind of imagining what I've seen before in previous years, but also thinking about detachment and all the different components that you were there. How much crossover is there between that mixed reality performance experience and what the experience of what detachment ended up being that I was able to watch here at home? How much overlap between those two entities are there?

[00:35:53.213] Roman Rappak: That's a good question. I guess after now I've talked so that some people might find it pretentious, but like after I've described that and tried to describe the sort of big lofty things we're trying to do, I started looking at what we would possibly make as a VR experience as standalone. And I realized that, in a way, detachment is almost a spoof of that. It's almost saying, listen, if all of these ideas are so heavy, so weighty, you know, there's Brechtian fourth wall stuff, there is, you know, there's total art, there's the idea of mixed mediums. And I kind of thought, God, there's, in a way, VR does take itself so seriously. Or rather, it only has two gears, it seems. It either has like goofy, shoot the zombie, smash the blocks, or it has a super heavy experience about, you know, dealing with loneliness or something. And I thought, OK, well, now with the show that there's this really big, quite serious point that we're trying to make, I felt more comfortable with saying, OK, well, I should be able to laugh at VR and I should be able to laugh at myself. You know, I'm the one who stands on the stage and listens to the guy talking about, you know, what is the nature of VR? And I thought, God, like there must be a way of saying, I think of balancing it out really so I can describe a little bit about what the detachment standalone piece is so at the moment that was its premiere so it's not going to be out on Steam for a little while but the idea was was to look at how the VR industry or the VR scene or movement as well as the people that work within it have some untold stories if you like and one of the stories by virtue of making this piece started to find out much more about number one so developers are horribly like the crunch time that developers face is absolutely horrible the working conditions in the video game industry in general i mean we you only have to look at the last sort of 20 headlines whether that is blizzard whether it's ubisoft whether it's And it's also, you know, if you watch things like indie game, the movie, you know, and you see the actual day-to-day life of two indie developers, it's a lot like music in a way, in the sense that it's a dream that people follow. As with any dream, there are people there out to profit from people who are optimistically thinking, hey, maybe I could quit my job and I can just put a game on Steam. And so there are these huge success stories, you know, like Braid, like Super Meat Boy, But then there's a tragic and difficult side of it where not as much as the music industry, but there's a lot of games uploaded to Steam every day. There are agencies that will charge you to do the PR for them. It's a bit like, you know, the people selling the picks and shovels are the ones profiting from it. And on the other side, there's the actors, the voiceover actors, the mocap actors. who are the absolute backbone of what this industry is. We started working with Eric Todd Dellums, who is in film, theatre, video games. The guy's done everything. He's in Fallout, he was in Skyrim, he was in Star Wars, he was in The Wire. If you look at his IMDB, you'd be like, wow, this guy must be driving around in a Porsche and having the absolute greatest time of his life. What I started to find was, speaking to developers that we're working with and speaking to Eric, is a lot of them can't even quit their day jobs because it doesn't pay the bills. For doing a voiceover work in a game that goes on to make $2 billion, you're paid about $300 a day. All the big parts go to the same five. You know, this all sounds familiar. And you look at the artist strikes and the actor strikes, et cetera. And then the same time, in my perspective of a musician, I've had music in games, but the amount of money you get for the game compared to how much that game makes, the kind of way that you're treated, because they say, look, kid, if you don't want to be in it and get this very, very small fee with this really terrible licensing agreement, there's about a thousand other hopefuls that we can go for. So the idea was to make a VR experience that made a comment on that. And in the, I don't want to give too much away, but in the experience, the voiceover actor who starts off as this sort of lofty, almost like a sort of Morgan Freeman character, it starts to be obvious that he wasn't paid enough to do this. And it starts to be obvious that he hates the script and he actually is not a big fan of VR. The game developer is being absolutely kind of crunched to oblivion and so hasn't been able to fix some of the bugs in the experience. Then the voiceover actor starts getting annoyed because the game that he's in is buggy, which makes him think, well, am I in this two-bit operation, this tiny project that's not going anywhere? And then the music that comes in also is unfinished and breaks halfway through. So the idea is to show a kind of a VR experience that effectively falls apart before your eyes, even though while it's trying to use all the tropes that we've seen millions of times in VR experience. So in a lighthearted way, It's satire. It's saying, well, look, the world of VR has matured to such a degree that we now know what is a cliche in a VR experience. You can see things that we've seen a thousand times. And I guess what we're trying to do with that is, in some way, poke fun at ourselves, because we're guilty of having used a lot of those. but in another maybe more serious way is saying, well, how is it gonna be any different from the video game industry? How do we avoid all the things that happen in gaming, all the things that happen in music, all the things that happen in the film industry, and is there another way? So the final part of this as an experiment or as a statement is that we are splitting when we put the game out on Steam, we're splitting equally what we're making out of it with the developer, with the director and the voiceover actor and the musician. straight down the middle, which obviously any game development company would tell you is a lunatic move. But I think it's something that, you know, it kind of makes sense as games get easier to make, as the tools become easier, as the distribution becomes easier, then I feel like an actor should be going into a studio to do a session and get paid the same amount as, you know, for his hours to be the same, rewarded the same as the hours of the developer. And I think if there's a royalty, I think they should get it as well.

[00:41:52.391] Kent Bye: And I noticed that there wasn't any writer that was credited. Maybe you could talk about the writing process, if this was a collaborative process of coming up with the script with your voiceover actor, or if this is something that you wrote out, or maybe you could talk about that process of developing it.

[00:42:07.249] Roman Rappak: Sure. It was a collaborative. So I wrote a script originally, but I'm so familiar with Eric as a human being, you know, we'll talk on the phone. I'll say, how are you doing? I'll say, oh, you know, I'm just, I'm taking my mom to the doctor and then I got to take the dog to the vet and then I could do this. I've got this audition, but you wouldn't believe this. So this guy that, and I won't, you know, I won't get into any personal details, but like, I really had a window into what the daily life of a voiceover actor is. And he's someone I love dearly and has been there and helped us for every project we've ever done. And, you know, we've been able to sort of help him out, get him work on certain things. He always just loves doing it because at his heart he's an artist. So he loves the idea of making something creative and subversive. So I wrote it with him in mind using, I'm not very good at doing an impression of a guy from DC, but he's got an amazing turn of phrase when he's just talking. I sometimes ask if I can put him on loudspeaker when I'm with my friends here in Paris because no one, you know, for them it sounds like a guy from a movie, you know, no one here knows someone like that. and so I started writing it and I was wondering is this a bit too near the bone you know that I'm basically writing about his life and how how certain things annoy him and he absolutely loved it but obviously no one can write you know an honest Eric as much as Eric can but it actually worked better when I got him to improvise And I said, look, Eric, imagine you're going to the worst ever audition that you hate, that you can't be bothered. And imagine that you're literally about to just quit. Just riff on it. And so the stuff he sent back was, I mean, there was some of it I couldn't possibly ever release because it was just too offensive. But some of it was hilarious. There was just a moment where he started eating Doritos in the middle of his voiceover. There were bits where he just came up with things that you could only come up with if you were a voiceover actor who's done huge things and worked with amazing people but also worked with some absolute idiots.

[00:44:06.331] Kent Bye: Yeah. So, it's got this dual tone of like the, okay, now he's in the mode of earnestly following the script and then it kind of goes off script, breaking the fourth wall, sharing a lot of his commentary. And you're spanning a lot of different genres of VR and both critiquing them, but also implementing them at the same time. And so, I know in previous conversations we had back in 2022, we had a chance to sit down and you had mentioned that your mom's an art historian and You had sent me a note commenting on kind of the postmodern nature of VR at the point of starting to deconstruct it. And so, yeah, I'd love to hear a little bit of your reflections of that influence of that art history context that you've grown up with and being able to look at the medium of VR through this lens of understanding the dynamics of the different genres and tropes that we've seen and what you wanted to do in terms of both aggregating all of those, but also commenting and deconstructing and critiquing them.

[00:45:01.518] Roman Rappak: Yeah, absolutely. I think the idea of postmodernism is meant to kind of almost be, it's always meant to be as much a full stop and a line in the sand as it is a rebirth. You know, it's never meant to be everything is terrible and it's all been done before, let's give up. It's time for something that's new and exciting, but also it's kind of a celebration you know if you look at like punk's a good example of a post-modern form of music because a lot of it was looking backwards and saying hey there was amazing stuff there that actually is worthwhile and there's a new way of expressing it where you have a knowing glance towards what's come before and it's almost suggesting to the audience You guys are well informed enough to spot when we're doing something that is, as well as making fun of it, it's kind of, it is sort of a love letter to VR, you know? That's why he switches into those really big, heavy spiels about when you're in an art gallery and he's sort of saying this art gallery is here but you are really here but I can't exactly remember the lines but that is genuinely how I feel you know I genuinely in a great VR experience the joy of it is the feeling that that real life has slipped away. And then when you take it off, it's a bit like, well, my eyes and ears have been fed all this information while I was in that headset. And so I felt like that was real. And now I'm back in my apartment and my eyes and ears are being fed completely different information. So it's obviously at its most effective is when a VR experience is at its most effective when you take off the headset and reality seems different rather than just, oh, when you put this headset on you feel like you're somewhere else. And I think in postmodernism, it's about saying that there are new paths forward and that this medium has still got lots of life and lots of exciting things in it. And it's not meant to be, oh, once you've seen an Andy Warhol print, then you can never look at a Caravaggio painting again. But it is saying, wow, this thing is going to be infinite. There's always going to be new ideas. There's always going to be exciting things that haven't happened yet. And so there is a sense of optimism to it. And there is also, there's comments that in the Andy Warhol thing example, you know, there's things that you can say with VR that you couldn't possibly say with painting or with poetry. And you could say there's aspects of painting and poetry within VR. But because there are things that you can only uniquely say to it and feelings that you can evoke with it, then it's almost our responsibility to say, well, what are those things? It's so exciting. All the greatest artists of the last 700 years were unable to make the comment that you're about to be able to make with this medium. And some of my favorite reactions when we showcased it for the four days at the Fairmont were developers who'd gone through it. And there's a bit where Eric starts talking about collision areas, which I gave my mum that experience. She wouldn't know what a collision area is. And there's loads of people that won't and also won't care. But because it was about a collision area and this developer, you can imagine how many conversations and arguments and frustrations he's had. And for a moment, it just said, we see what you're doing and that we understand what you're going through in a way. And the same with the voiceover actor. One of the voiceover actors from another experience came over. and she went through it and she said I think I'm gonna have a chat with the team I was in after this. She said it's really struck a nerve and yeah maybe you know let's be as you have to be as optimistic as possible when you're making something you have to I don't think there's any point in writing a song if you don't think it's going to be a great song or making a VR experience you don't think it's going to be an amazing VR experience and so maybe I guess optimistically maybe If a voiceover actor goes through it, then next time a VR project comes along and offers them a gig, they say, well, look, I'd like to ask this and I'd like to ask that. Or when a developer is asked to do something within a horrible crunch time, then they can say, well, look, to be honest, I can't stand my ground on this because look, this is the deal. Everyone's aware of this and this is what's going on. So, I mean, it is optimistic, but, you know, we'll see.

[00:49:04.365] Kent Bye: Yeah, I really appreciated this shift back into the music video genre that is very much building upon what you've already been exploring with previous iterations with the mirror shot and the live performances. And then again, this year that you're doing that. It really felt like a epic adventure part where you're able to go and navigate through these different post-apocalyptic worlds, which is a really nice introduction that you have. Again, kind of like a callback to the Matrix and a simulation and breaking out of the Matrix sort of dimensions that are also included in here that I really appreciated. But one of the things that I wondered is that you're working with the game engines and starting to play with moving through a spatial context. being able to guide your way through this open world space that's constrained. Sometimes it's very linear, you move forward, and other times you can explore around. But there's also the spatial audio dimension of VR that I'm really curious if you've started to explore the spatial dimensions of audio. And, you know, obviously there's like Dolby Atmos and other spatialization standards that are able to be distributed on something like Apple's platform. And I'm sure other platforms eventually will start to adopt different types of either ambisonic or Dolby Atmos or some sort of spatialization to the audio. But as a musician and a creator working with the spatial medium, it opens up all these new possibilities to start to either work with spatial audio in a way that's maybe built in the game engine or using ambisonics, or maybe you're doing interactive FMOD, you know, where you're really encouraging the player to engage and interact and become a part of the music creation process. And so I'm just very curious as a musician, where you see that going for your own creative explorations for where you want to take the future of spatial audio and the practice of creating your musical art.

[00:50:52.267] Roman Rappak: Well, I'm really glad you mentioned that because we worked with an incredible company called Sphere of Sound, and they are spatial audio experts who also are very much on the sort of artistic and creative aspect of the spatial audio movement, as well as the technical side, which is so crucial. And I've looked at it a lot as a musician of, OK, how can you do this? There's a really interesting contradiction in terms of musicians, live music, and spatial audio, which is, Spatial audio really doesn't work very well if you're in an audience and you're standing there and You have your ears and there is the stage You can make sound come from behind you and from above you etc. But really your focus is on the stage So the effectiveness I think of it is kind of diminished a little bit You can't walk around and that the audio warps because this is the effect for me Then I'm sure there's people especially people like that sphere of sound who are much more technically articulate on this But the effect for me with spatial audio is really powerful when you can move around a stage space which is again why it's so powerful in VR because it sells the fact that there is depth to this space you know you turn your head and there's a dripping tap and the exact mathematical acoustics of that dripping tap if a human being was to turn their left ear this many degrees is incredibly pronounced. So one thing that we did this year at the wristband official showcase was we had a room that had a spatial audio installation. It was about 1,250 square feet, four speakers so it's ambisonic and it was a piece made with Alexander Parsons who's a sort of a composer who's worked on sort of BAFTA winning films and TV. And the idea was to have a loop that people could go in, almost like a kind of a pre-immersion into the VR experience. And what was interesting was that because people could move around it, you had people who would stand in different parts of the experience and feel different things. Obviously, each speaker is doing different things. There is a huge structure in the middle that is projection mapped, which is really, I can send you some videos of it after, which is really specifically timed to what's happening in the piece. And the fact that it's a three-dimensional projection map display lends itself to the fact that there is four sides to this music as well. Because again, like, if you watch a movie and then there is you know, a car crash behind you, what do you do? Do you turn away from the screen and look at, you know, it doesn't actually work in the medium, in 2D mediums. And I think in some ways, a live performance is a 2D medium. I mean, if the performers are doing their job, then they desperately want to hold your attention for a rectangle that's in front of them, so. Yeah, those are my thoughts on it. I'm excited to explore it further. That was the premier version of Detachment standalone. We're now working on the Steam release, which will have a ton of other dialogue that we're even experimenting with using an update every week for different pieces of dialogue, for different sections that you can discover. And kind of, again, playing with it, like there's a part in the experience which has a degree of gamification because I felt like, well, I don't want people to go through this experience and they're just on rails. And naturally the voiceover actor starts complaining, why does everything have to be gamified? Why do we have to, why do we always have to collect things and get points for things? So yeah, we'll see.

[00:54:13.653] Kent Bye: Yeah, the third time I played through it, I was able to successfully get all of the different gems. So that was a lot of fun.

[00:54:19.535] Roman Rappak: That's great. I don't think I've ever done that, but yeah, I think, I think that more of that is great because some people want to go through it and experience the mood of the piece. I like the people that go through trying to look for the Easter eggs. You know, we want to have a leaderboard at the end and maybe have Eric complained about the fact that, you know, why is everyone obsessed with leaderboards?

[00:54:36.836] Kent Bye: Well, you mentioned the sphere earlier and you mentioned that usually in a live performance, it is a very 2d experience. I feel like the sphere is starting to break out of that. I agree with having that huge dome. I haven't had a chance to make it out to Vegas and check it out yet, but I also know that there's other smaller venues, either like planetariums or other domes. Like Cosm is working on like, like 1500 people in a dome rather than like the 17,000 that's in the sphere.

[00:54:59.972] Roman Rappak: So, yeah, we, we met Cosmo. They're amazing. They're doing great, great stuff as well.

[00:55:04.935] Kent Bye: So have you had a chance to see the Sphere experience in Vegas or explore some of these other dome potentialities?

[00:55:11.162] Roman Rappak: I haven't, no, but domes I think are a great example of the thing we were talking about at the beginning, which is this desire for immersive experiential events. It's interesting that in a time where we can You know, you look at the... I hate those little notifications saying how much screen time I've spent on things because it's never good news. It's never like, oh, well done, Roman, you didn't really look at screens. There's always a horrible amount of time I've looked at screens. And I can download a movie or I can watch it on Netflix and I can listen to any album I want. And it's natural that human beings crave that idea of going to a place and doing something, you know, that transports them. And in some ways, maybe we are You used to be able to go to a movie, and if you imagine you didn't even have a TV at home, going to a movie would be like going to a Vegas sphere, you know, just being wrapped in a kind of a Western or something. You would genuinely feel like you're there. And I think as we crave more of that, we're also maybe, it's harder and harder for us to really feel enveloped in something. I mean, obviously, if you'd really love a particular folk artist, you can go to a small bar and sit with 30 people, and that person with a guitar can do more than any headset could ever do to transport you. But on a kind of mass culture level, the idea that you can go into a dome and you can experience something that you can't get from your TV, you can't get from your phone, and you can't actually get from your headset, right? Because you're there with people. I think we're going to see more and more of that in Cosmos. A great example of people who are saying, OK, well, it's not just about U2 and Beyonce, and you have to have enough money to go to Vegas and spend hundreds on a ticket, that this is gonna be something which, in the same way as early cinema, there weren't that many cinemas, and the movie business didn't exist until the cinema system existed. You have to find a way of reaching people en masse and keeping the cost down so it's not an elitist activity. I'm optimistic about that because I feel like, again, referencing the stuff we talked about earlier, It's only when we have these new mediums that we can make these new statements, but it's only when we have the infrastructure for these mediums to exist in that anything can happen.

[00:57:15.982] Kent Bye: Great. And finally, what do you think the ultimate potential of this intersection between virtual reality, XR, immersive media, and music might be, and what it might be able to enable?

[00:57:29.038] Roman Rappak: I think, for me, the ultimate potential, I'll answer it, rather than saying the ultimate potential of the medium, I'll say the ultimate desire that I have for it, if that's okay, which is that it brings about a new movement, a new form of art, a new way of expressing things, the same way that we had impressionism, the same way we got punk, the same way we got rave culture, all of those things, if you borrow down, are down to the fact that you suddenly have new technologies and new ways of transmitting it. And so I think the ultimate thing would be that there's a new era in art performance and society in general.

[00:58:04.953] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?

[00:58:11.106] Roman Rappak: I just think thank you Kent for putting this podcast together. It's this sort of the Bible of VR but it's also the daily news of VR so it's fantastic and then the other thing is that if anyone wants to see these experiences that we're doing if anyone wants to book us to come and bring the experience we're doing one in Berlin next month, then one in Marrakesh, and we're doing one in London in about two weeks. So yeah, get in touch. If you go on wristband.co, you can also join the newsletter and get more info about when we're releasing Detachment publicly.

[00:58:46.055] Kent Bye: Yeah, well, it's really been a busy three years. You've been at South by Southwest each year with a new iteration of your experiences. And I really appreciate how each year is also expanding into both the live performance component that you're doing, that folks can book you to come do this type of experimentation of the fusion of mixed reality, XR and music, and then leading into the actual music performance. And then all the wristband stuff you have and experimenting with game engines. And then yeah, this dive into the narrative components of detachment and commenting on both the culture and the political economy of XR, but also deconstructing the different genres and mediums that we have. And also taking a stab at satire and comedy, which like you said, there's usually just two modes of being super earnest or being gamified and that, or it's like apps that are no narrative components at all. It's just utility of performing a function. But yeah, it's not easy to kind of dive into that satire comedy. It's a genre within XR that's difficult to pull off, but there's certainly some moments here that as a XR user appreciated different callbacks to things like big screen and different kind of matrix references and yeah, just like lots of. inside jokes that are embedded in there for folks who are in the XR industry that whether you're a developer or a voiceover artist or artist, I think there's going to be something in there for folks to appreciate as they reflect upon the nature of the medium itself. And also really appreciate all the deep ways that you think about the medium as well. You're trying to carve out your own artistic practice with integrating the medium and see how it's this iterative process, how you're going back and forth between the mediums and I'm really excited to see where it takes you next year and on into the future. So thanks again for joining me here to help break it all down.

[01:00:25.811] Roman Rappak: Thank you very much. Speak to you very soon, Kent.

[01:00:28.893] Kent Bye: So thanks again for listening to this interview. This is usually where I would share some additional takeaways, but I've started to do a little bit more real-time takeaways at the end of my conversations with folks to give some of my impressions. And I think as time goes on, I'm going to figure out how to use XR technologies within the context of the VoicesofVR.com website itself to do these type of spatial visualizations. So I'm putting a lot of my energy on thinking about that a lot more right now. But if you do want a little bit more in-depth conversations around some of these different ideas around immersive storytelling, I highly recommend a talk that I gave on YouTube. You can search for StoryCon Keynote, Kent Bye. I did a whole primer on presence, immersive storytelling, and experiential design. So, that's all that I have for today, and I just want to thank you all for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.

More from this show