Anti-Gone is a mixed reality play that features two actors performing in front of a wall that has an open, virtual world projected on it. One of the actors spends the majority of the time in VR navigating this world and interacting with the other actress who is a motion-captured suit. They each have simultaneous embodiments in both the virtual world and the co-located, physical reality as the story unfolds through a series of choices they make on their journey through this virtual world.
The narrative structure is based upon the comic book also titled Anti-Gone by Connor Willumsen, and the mixed reality adaptation is described as “set in a post–climate change world raging with late-capitalist rebellion and excess, Anti-Gone is a quest for happiness in the sunken city, where the characters shop, score drugs, and hotbox cinema.” The narrative structure is an inverse tree in that the characters have more options at the beginning of the show, but it narrows as time goes on and always ends the same way.
This gives the characters a lot of latitude to make a lot of choices at the beginning of this experience for how they want to navigate this open world. The interactions are pretty low-key, hanging out, chilling, and largely center around taking psychotropic substances that invoke a variety of different altered states of consciousness.
For me, the implications of the technological infrastructure behind Anti-Gone is one of my biggest take-aways from this experience. They self-describe Anti-Gone as “part theatre, part Twitch.tv, this groundbreaking play is a feature-length livestream game telecast.” It is a seamless blending of the virtual and the real, and the virtual world is being driven by an open session of Unity where the director and lead artist Theo Triantafyllidis is controlling a virtual camera rig with five different perspectives. He can control NPCs in real time and trigger virtual events based upon what the actors do, all while being the actual virtual director of what cinematic, window/portal view that the audience and actors see of this open virtual world.
There’s a lot of other live elements including musician Cameron Stallones and lighting, as well as an off-stage actor to provide additional narration and other characters that appear throughout the narrative experience.
I had a chance to sit down with director Triantafyllidis and musician Stallones to talk about the evolution of this project, their inspirations from Twitch.tv livestreaming and Dungeons & Dragons, how they formed the bulk of the potential narrative arcs in rehearsals, what aspects of the experience are live, dynamic & changing, and their overall experiential design process that is blending the virtual and the real, the improved moments and planned narratives, and the open world exploration and more limited set of possibilities.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So continuing on in my series of looking at some of the XR experiences at Sundance, specifically the immersive storytelling innovations, the technological innovations, as well as the experiential design process. Today's interview is with the creators of Antigon. So Antigon was a part of the biodigital theater. So it was a live performance. There was. The director, Theo Triantafillidis, he was a real-time dungeon master controlling the open world that then was being projected onto this big wall. There was two main actors, one was in virtual reality for over 75% of the time, and so he's able to do these different interactions in the virtual world, but the virtual world was being projected onto this big wall. and then the other actress was in this motion-captured suit and so her embodyment was being projected onto the wall as well. So you kind of have like this third-person perspective of seeing the virtual representations in this open world and it's this whole exploration of them being able to explore around different aspects of this world and then eventually converges into a Sinclair ending. But each performance has slight variations with different changes, and it's kind of like this open world exploration that's trying to blend together aspects of the game engine, open world exploration with this real-time dungeon master, as well as this improv acting, but also having an overall narrative arc as well. So, that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Theo and Cameron happened on Wednesday, January 29th, 2020 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:01:50.500] Theo Triantafyllidis: I'm Theo Triantafyllidis. I'm an artist. I work with technology, I guess, and I've been working with immersive VR, AR type of work for a while now. A lot of game engine work. and usually presented within a gallery or fine arts context. And this is the first larger scale performative work I've worked on and I'm glad to present it here in Sundance.
[00:02:20.357] Cameron Stallones: My name is Cameron Stallins. I'm an experimental musician and composer. I have a project called Son-A-Ra. And I met Theo working on an immersive project that we did together, trying to create generative environments based on improvised music. And then he asked me to come and compose for this piece, which has been really interesting because it's indeterminate. It's an interesting puzzle to solve. So we had to kind of build some systems. So it's a systems project.
[00:02:48.481] Kent Bye: Great, so yeah, maybe you could each give a bit more context as to your background and your journey into this immersive space.
[00:02:54.983] Theo Triantafyllidis: Cool. I have a background in architecture. So I studied architecture in Greece, in Athens, where I'm from. And then I came to Los Angeles about six years ago to study the Design Media Arts MFA program in UCLA. And since then, I've transitioned into being an artist and working with immersive technology.
[00:03:19.456] Cameron Stallones: And Theo's really my main intro into the tech world. I mean, I've experimented a lot with digital tools as far as music systems and software and indeterminacy and cybernetics in that space. But when I met Theo, I mean, I had an interest in trying to make those systems talk to other kinds of systems, visual systems specifically. And when I met Theo, we started experimenting with that. So yeah.
[00:03:42.914] Kent Bye: So maybe you can give a bit more context in the journey of Antigon and the evolution of this project and how that came about that you're showing here at Sundance.
[00:03:52.376] Theo Triantafyllidis: Yeah, so Antigon has been a long, crazy process. We've been working on it for more than a year now. I started off as small experiments with small embodiment experiments, like how to represent a body in a virtual space. A lot of my earlier work is focused around that as well. And then I started bringing a team together and starting to expand upon some of these ideas. And a key moment was when I discovered the original comic book by Conor Williamson called Antagon. Basically this performance is an adaptation of that comic book. And there were a lot of really interesting ideas, characters in the story that really, not only were really interesting on their own, but had some sort of really interesting connection with the kind of questions of virtual reality that I've been interested in. And I found that there was this really interesting dialogue happening.
[00:04:58.348] Kent Bye: Yeah, and so maybe you could talk about your introduction to this project.
[00:05:01.530] Cameron Stallones: Yeah, you know, like I said, Theo and I had worked on a previous project, and so he asked me if I might be interested to come and work on this. They had been using like pre-recorded compositions, you know, like in a traditional theatrical way, but Theo really wanted the next stage of the project, the one that we're presenting here, to have a lot of indeterminate elements and be chance-operated, which meant that the score had to be able to react to the choices that the actors make. That sort of stuff is interesting to me anyway, so I got really excited about trying to make a system that would allow me to play and obviously have some composed elements, but be able to react and rearrange it in whatever way the performance finds itself going.
[00:05:43.262] Kent Bye: Okay, yeah, so I had a chance to see it once, and I haven't seen it multiple times, to see some of that variation. It's hard for me to really know what elements are left up to chance, but there were certain moments that I could feel that there was like a... like a concrescence of something that happened, and then maybe a sound effect, and then maybe a reaction of someone saying something. So, you know, I think there was a collision one time when... Actress said ouch and it was just kind of like, you know There's a virtual boat colliding with each other and just went through them and she's not in the virtual world But you know, so is this sort of this this funny moment? So as I watch this scene there's two actors one has motion capture to be able to have the higher-end mocap and then you have virtual reality headset with the main actor who's got a index with knuckles and or the index controllers now, as they're called. And then there's a third actor who's offstage, who's providing voice for offscreen characters or other entities or animals. You have the musician, you have the lighting, and you have you that's driving different stuff. And so there's this big screen where you see everything. and you're presumably directing it in some ways maybe, like there's a virtual world and maybe you're controlling the camera. Maybe you could just kind of break down each of these roles and how they're all like integrated into each other because you're, it's like this open world exploration, but with improv actors and sound effects and I don't know how much is like prescripted, how much is emergent. So maybe you could just kind of describe each of the roles and what you're doing throughout this performance.
[00:07:09.418] Theo Triantafyllidis: I can start off by describing some of the early experiments because I think they give a good sense of what's happening. The first rehearsal started by making an adaptation of Dungeons and Dragons rules but using the story of the book. So we tried to have the actors think about themselves as characters and as players at the same time. So it was very like in their role-playing mindset. And then on the other side, Matt, the collaborator who is not here right now, and I were playing the game master role and guiding them through the story of the book, but leaving them enough like agency and space to improvise a lot of the things that were happening. and imagine this is now the same system but with all these layers of crazy technology going on top of it but still the core mechanic is that these two actors are the players that are playing in this open world like on our side we are still playing this dungeon master role where we have I would say an arsenal of different like tricks and animals and NPC characters and environmental conditions and all sorts of things that we can throw at them to destabilize them or give them cues for narrative progression or delay them or generally mess with them in ways. And then they have their own character goals and their own character desires that they are trying to follow. And basically, if you think about it in narrative terms, I would say it's an inverse tree format where there is like a lot of different choices and directions they can take in the beginning. But as the piece progresses, those options start limiting themselves and then there is this imminent catastrophe that is unavoidable and will always happen at a specific point.
[00:09:04.293] Kent Bye: So basically the ending of the piece is always the same but the first two chapters are indeterminate basically I see so so there's a bit of randomness that can happen in the version I saw it there was an alligator that was found and then there was like someone getting beat up outside and then when they look back then that whole scene had just like Disappeared and then she was like just oh and then they just kind of moved on like it like that's just a normal thing in this world where you see something that upsets you and then it just kind of disappears but So the actors, how much are they improv-ing then? Are they improv-ing more at the beginning, and then as you go on, then there's more of these story beats that go towards this certain character journey?
[00:09:45.700] Theo Triantafyllidis: Yeah, so they are, basically they are improvising a lot of the dialogue, and they are improvising a lot of the smaller decisions that they do. Like in the beginning, in this archipelago of different islands, and they get to pick an island, and they get to pick what they're going to do on that island. and then they have to move in the city where they know there's some locations and interior spaces that they can visit, but again, they will arrange the sequence. And then some of the characters in these locations are randomized. There is a pool of NPCs that are distributed in the whole world in the beginning every time. And then there's also a set of environmental conditions that we are manipulating, basically.
[00:10:29.120] Kent Bye: What do those environmental conditions determine?
[00:10:31.662] Theo Triantafyllidis: I think you might have, in the one you saw, there were no environmental conditions happening. But generally the idea is that Antigone World is in a near future, in a post-environmental catastrophe, where there is a city that seems like semi-submerged New York City that's overgrown with vegetation. And the weather in this world is very uncontrollable, basically. So there is an extreme heatwave and there is also an extreme rainfall that can happen. So these are used in key moments when we want to... Like when the heatwave happens, the actors know that they have to find an air-conditioned space really quick. So this will determine their actions a little bit. Same with the rain, basically.
[00:11:25.660] Kent Bye: Yeah, and the music design, I really enjoyed the little tunes and the music and the live interaction because I could see you there watching what was happening. Maybe you could talk about what you're taking as an input because there's a big giant screen with the virtual world, you have the two actors, you have some sort of screen in front of you and you have this Ableton control interface with 16 to 64 different buttons or 32, I don't know how many there are, but there's a lot of buttons that you have that are unmarked and that are lit up that you have to either memorize that or have some sort of reference there to know what one of these magic buttons to push to be able to create emergent sounds. So maybe you could talk about how you're able to take in all this information and then what you're outputting by pushing these buttons.
[00:12:08.392] Cameron Stallones: It's a challenge but it's fun and also because we're a small crew I'm also like mixing the mics and doing like just kind of basic theatrical sound as well. So I have an Ableton session set up that has a couple channels that are devoted to being sort of like a soundboard with like sound effects and environmental loops and things that we use just to create the atmosphere. And then the scored elements are all, we wanted it, like I said, to be live and to be played live, and so, as much as possible. And so, and also we wanted to match, you know, the aesthetic of the piece has this sort of down-res comic book aesthetic that has a very specific vibe. Basically everything is being run through a DX7, which is an 80s Roland keyboard synth that was very popular. We've all heard it on hits and classics, but just the fact that all the scored elements are coming from this one sort of 80s synthesizer box gives it a feeling where it feels cohesive. So on top of the soundboard element there's another element which is, you know, there are some kind of sequenced chord progressions that are programmed, they're not recorded but they're MIDI, so they get triggered through the synthesizer and then I'm able to like play live top lines over the top of them. So I basically have like a bank of patches, you know, that I've pre-selected that I can switch between and then improvise with. And like he was saying, we sort of have a general sense of like, you know, all these puzzle pieces go together in kind of any order, but you know, they have specific vibes and moods that we're trying to create, you know, within those pieces. So, you know, I'm never fully having to like make something up on the fly. Like I sort of, you know, we've decided that at this point in the story, this is the sort of mood we want to set. So this is the composition. like the timing and the ability to speak to your earlier question about what am I inputting, you know, it's what makes it fun and challenging is to have to watch the actors and try to kind of play underneath and between their improvised dialogue, you know, because you don't want to overwhelm what they're saying because that's the important thing. So I'm usually watching them and watching the screen and trying to figure out, you know, where I can play and kind of create some ornament in a way that doesn't distract from the actual dialogue or the actual meat of the scene.
[00:14:17.716] Kent Bye: Are there any emergent sound effects or things that would happen that are unscripted or unplanned that then you would need to pull up a jingle sound or something like that? Or are all the sound effects pretty well set out where you're only responding to things that you've already prepared for?
[00:14:34.433] Cameron Stallones: I've tried to prepare. I have a huge bank of stuff based on all of these, like we had these pretty chaotic rehearsals for about a year. Yeah, where I spent a lot of time on freesound.org trying to find sound effects for all these things that they were coming up with. They're like, okay, and now this happens and I have to like, oh shit, I gotta go get like a, you know, like a tugboat sound or whatever. So now I have a pretty big basket of stuff. It's yeah. So I, you know, and we don't want to be like on wifi during the piece for a lot of reasons. So, So I have, you know, I'm stuck with what I have, but we've also had some funny moments where, you know, like a new NPC, like we recently introduced like, or maybe it was always there, but we just didn't use it that much. We recently introduced a snake, you know, that appears at a certain point and I didn't have a snake sound. So I had to like quickly get a maraca sound and like process it in a way that it could be like a eight bit snake sound, you know, so.
[00:15:26.419] Theo Triantafyllidis: Yeah, I can talk a little bit about the earlier rehearsals because I think that's where a lot of these ideas and tools emerged. We basically had a really extended... I mean, we did a lot of rehearsal rounds with a little bit of space in between. But we did these really early experimental jumps where the whole team would come together and we were all completely improvising. We all know the overall arc of the story so we are generally trying to stay within those goals of the story basically. But in these early super crazy improvisation sessions we would all be completely pulling things on the fly and creating everything. Like Cameron would have to improvise all the sounds, I would have to be in the game engine and pushing and pulling things around and dragging 3D models in while they were playing and doing all that kind of stuff. but through this process by the end of that we would look over every playthrough that we did and we would select the most successful elements and now we all have like a palette of things that we know work well and it's all about like keeping the improvisation energy and having everyone on alert and at risk and knowing that anything could fail at any point and this brings a certain like adrenaline to the performance which we enjoy. And then at the same time, the actors know really well what are the capabilities of the game engine and what are the edges of this world. And we know where they might be inclined to go towards. So we have really good communication and understanding of where things can go, basically.
[00:17:12.433] Kent Bye: Yeah, that was the thing that I think was also really striking was the open world nature of it. And not surprising that you have an architecture background, because I think the way that you design the city and that you have this space they're exploring, I think it has that architectural sensibility. And so maybe you could talk a bit about, with your architecture background, designing this large city with a lot of more modern architecture, which is something that has these buildings that are there, maybe represented of the existing New York City. But then you have these more organic virtual architecture that would be almost impossible or very expensive at least to build. So maybe you could talk a bit about like from your architectural background what you were able to play with a bit with this open world that you created.
[00:17:53.699] Theo Triantafyllidis: Yeah, so actually we did a version of the piece earlier that was a pilot performance at Human Resources in Los Angeles and one of the major parts that was missing from that one was that we couldn't make that city because basically in the comic book the city is a pretty important part of the whole world. but to make it would require a proper game studio and it was a lot about using the few resources that we have and finding a form of stylization and making a city that feels complex and alive but with very limited resources basically. And then also, of course, the comic book has such a really lush and interesting visual language that that was one of the early challenges as well, like trying to take Conor Williamson's visual language, which is very much all about the line work and like all the crazy stuff that he can do with the comic book language. and translating it into something that is three-dimensional and is very concrete and specific versus like a fluid line and all the things you can do in drawing. And one of the interesting things I think is that I had a few like really talented 3D artists that worked on this and I tried to find people who would have this sort of sensibility Ryan Decker, an artist from New York, made all the vegetation and all the flora of the Antigone world and these are all hand sculpted in VR and they have this very like gestural quality to them so the vegetation is very soft and smooth but also aggressive in a weird way. And then a lot of the buildings were made by an architect from SyArk and they have this combination of like classical architecture to show that the city has some history and some like it's been there for a while but also these very crazy sci-fi buildings that show up occasionally.
[00:20:07.329] Kent Bye: Yeah, and I've talked to other architects, both building physical architecture, but also virtual architecture, and that there's certain aspects of things you can build and design in VR that just are impossible with not having gravity and whatnot. And so I think that'll have an interesting influence on architecture. But there's still similar constraints where, while you're not buying material, you still have geometries and the complexity of geometries, and there's certain amount of efficiencies that you have to run everything on if you don't have the budget to be able to design that. type of really complex geometries, you would need to have very high-powered computers to be able to run that, but also the time to design it. So it feels like that you're able to take a minimalist approach in some ways, and it's an 80s aesthetic where you've got these early video games and it's like low-res in that way. So I think it really works because it has this retro feel to it that allows for that low fidelity and that The focus is on the actors and the content rather than the visuals and you wouldn't want the visuals to like overpower what was happening in the story anyway.
[00:21:09.139] Theo Triantafyllidis: Yeah, exactly. And also, we had to really invent a visual language that would work. Like, having actors in costumes is such a different visual language than what you can do in a video game. Like, even in a AAA, like, photorealistic video game, if you put actors in front of it, it would look really strange. So we had to invent a visual language that has a certain amount of abstraction and a certain amount of figuration that would work in that context. And also having the motion capture suit in VR and all the glitches and the strangeness that comes from that motion. The characters also had to be of a certain fidelity to be able to accommodate for this strangeness of the motion. We tried more realistic characters and they looked super uncanny and weird and we tried more cartoony and then they were too silly so we had to find a good balance there.
[00:22:07.827] Kent Bye: Yeah, and maybe you could break down a little bit as to what you were doing, because it looked like it was a game engine scene, either Unity or Unreal, in that you have a virtual camera that you're potentially moving around that is like a portaled window into this virtual world that with one of the actors is completely unveiled into virtual reality for most of the entire experience, that he's inside of the virtual reality experience, presumably seeing a whole spatialized version of this world that he's able to navigate around. and then kind of lift up his visor and see the 2D projection of it on the wall. But for a lot of it, he's in VR. But you are sitting there, staring at a screen, presumably directing. Is that right? Or maybe talk a bit about how you're able to, if you needed to design different tools, or how you are able to, as a director, do this real-time directing of what is essentially a theater show, but projected onto it, a little bit more cinematic of a virtual world.
[00:23:01.631] Theo Triantafyllidis: Yeah, so we had to design a lot of tools for that. Basically the piece is running in Unity and it's running in the editor so I still have, like, if something is going horribly wrong I can still pull up a different scene and do a lot of things that a build wouldn't allow.
[00:23:20.175] Kent Bye: Wait, so does that mean it's not a build, you're just pushing play? Oh, so there's different scenes and you're pushing play to kind of cut, okay.
[00:23:26.798] Theo Triantafyllidis: Yeah, I mean, it is one gigantic scene. Like, everything is inside one gigantic scene. And this was also one of the technical challenges. There is blackouts, but the blackouts were more for the actors to have enough time to move between things and to recalibrate the suit. But basically, theoretically, the whole world and the whole experience could be seamless, like you can travel between everything without loading. And what I am doing is basically, I have a bird's eye view of the whole world and I'm organizing the NPC population and the different... I mean, a lot of things are automated at this point, but throughout the whole process I was constantly shaping the world as the piece was progressing. And at this point it's basically a set of cues for each scene, showing prop items, characters, triggering events that happen within the scenes. And then there is this five camera rig that each scene has a five camera rig that works sort of like a live television directing where I'm constantly picking what view to use.
[00:24:42.972] Kent Bye: Okay, so you're seeing all five of those views and then cutting between that to show what everybody else sees.
[00:24:48.035] Theo Triantafyllidis: Yeah, yeah, exactly. And we are trying also to merge like the theatrical language, a more filming language, and also the video game camera work. So it's always shifting a lot between them. Some scenes are more theatrical in the sense that they are nearly 2D and meant to be seen from a specific perspective and some are wild 3D scenes that I use third-person camera orbiting around the action and continuously like adjusting and then we have all the character close-up shots for the action.
[00:25:25.723] Cameron Stallones: Sometimes I wish that we could have like a projection of Theo's screen. I sit next to him and I get to like look and watch his screen which is really interesting because everything he's describing it's like there's you know all these different camera views that he has to switch between and it's very performative for him and for us because you know like the quote-unquote camera work that happens every night is pretty different. I mean there's some like things that are conventions that we do but Like you know watching him perform each scene and seeing like try new things every night like even last night We were documenting the work and I could tell you were trying these camera views We had never tried before just because you're like I could tell I was like But he's just trying to see if this looks good because we have this camera here to film it But yeah, it's it's really cool in that way And it's like we've kind of wrestled this whole time with like how much to let like like your experience of seeing it once a lot of people don't realize when they see it that there is the possibility for other versions of the play you know and And so something going forward that we've been thinking about is how to make that a bit more explicit or how much to let the audience have a peek at the God's eye thing that's happening, you know, behind the scenes, because that's also very interesting.
[00:26:29.558] Kent Bye: Yeah, there's a thing in interactive narrative where you go through an experience and then you're told afterwards that there were decisions that you made that you weren't aware of. And so it's like, then you're like, oh, wait, really? Was it really interactive in that point? So I feel like, yeah, there's a certain, it's a hard thing to convey unless you do go see it multiple times. But I could definitely see how that keeps it alive. And I guess there was one scene where it looked like you were actually physically moving a camera through virtual space as well. So it's not just static shots. You can actually move through and do zoom-ins as well, right?
[00:26:58.935] Theo Triantafyllidis: Yeah, it's like some of the cameras are more static and they're kind of locked onto the characters and other cameras have these fully controllable video game camera controls basically.
[00:27:12.928] Kent Bye: And is the person who's in VR seeing the exact same view as you're seeing? Is it he's going on like a live edited view? Or is he seeing something else?
[00:27:21.104] Theo Triantafyllidis: The headset is actually seen exactly in the same world and he sees like a fully high-definition version of this world, which actually is very impressive if you see it from VR.
[00:27:32.736] Cameron Stallones: That's the next step. It's really a tragedy that the audience can't be in VR too, obviously. We haven't quite found the resources to get, you know, 40 headsets rolling, but that's a future goal for the piece.
[00:27:44.935] Theo Triantafyllidis: Yeah, definitely. And I mean, that was also one of the very interesting things from the beginning of the work, that one of the two actors is blindfolded in the physical space, but he has this like, in some ways he can see more than I can see, like he can see much more detail in the world than I can see from the game engine.
[00:28:06.333] Kent Bye: Well, maybe you'll be in VR directing at some point.
[00:28:08.274] Cameron Stallones: There's characters that appear and disappear from time to time, and it can be a challenge for us with the way the cameras work to find them and to control them and bring them into the visual space. So sometimes I actually rely on him to look around and be like, oh, over here. And we're like, oh, good, it's over there, because we don't know. But he's actually able to look around and see the whole space rendered, and we can't. So it's really interesting in that way.
[00:28:29.799] Theo Triantafyllidis: Yeah, and in the VR view of the world, it's basically spatially continuous. So he will always be in the proper space and see what is happening. And when we are... A lot of the interior spaces are actually also still in the open world scene. So when they are in the coffee shop, he can still see the world around and outside the building. and there is a certain flow of the space. And then he has the boat controls, so basically he has complete agency of deciding how to navigate this world. He's really playing the game, basically.
[00:29:09.033] Cameron Stallones: And the world is huge too. I mean that we've had to play a lot with the speed of the boat because it was interesting to us that you know he would have to start driving and it would literally take him 10 minutes to get from one place to another you know and that's a huge part of it which is like the first time that I saw it I was so impressed at the scale of the whole It's like the city and this archipelago outside the city. It's really vast. Do you know how big it is in cubic feet or whatever?
[00:29:33.091] Theo Triantafyllidis: I don't know. It's pretty large. I think it's around three miles long in real scale. It's definitely compressed because it would take 30 minutes to get from one place to the other if it was in real scale. But it's a very compressed version of a real scale.
[00:29:52.367] Kent Bye: Well, just to kind of start to wrap things up here, what are some of the either biggest open questions you're trying to answer or open problems you're trying to solve with a project like this?
[00:30:03.334] Theo Triantafyllidis: Well, one of the major challenges of this work is there is of course all this technology involved and there is all these elements of improvisation which are important, but at the end of the day we want this to be like a really interesting performance and a really interesting experience for the audience. and we would rather have all this stuff kind of like fade to the background and we want this to be like a really exciting experience and that's also been a struggle like of course it has all these layers of technology but we're trying to push them to the background basically.
[00:30:43.480] Cameron Stallones: Yeah, for me, for the last like five years or so with my work as Sunora, I've been really trying to do similar things musically as to what Theo's doing, which is like, I guess work with cybernetics in the sense of work, like, allow the computer to make choices, to collaborate, truly collaborate with software. And now with like, you know, Maxim SP and a lot of people building some very interesting things, it's very possible to create to create music systems that are generative and emergent, I guess is what you would say, or at least very close to that. Like, so complex that they feel that way. And so, you know, I think we're really excited to, like, continue to refine the whole system so that, you know, I think the dream has always been, in the way he was describing with the Dungeon Master kind of approach, that eventually the system will be, like, able to sort of throw things at us, and we can throw things back at it, and it will be, like, become, like, a real sort of, like, I think about it like musicians, so it's like you're jamming, you know what I mean? Like you're improvising, you know, you're playing and your guitar player does something and then you do something on the drums and that's when we finally get it all the way dialed on a larger scale, I think that would be the best possible version, would be a version where the actors and us the crew can like really Have a participatory relationship with what's going on and it can be like a kind of an even collaboration between the technology and the and the humans involved great and and finally, what do you each think is the ultimate potential of immersive technologies and immersive storytelling and What am I people to enable?
[00:32:15.360] Theo Triantafyllidis: Hmm I always see that from the perspective of an artist and seeing that this is such an interesting territory to be in. And honestly, the parts I'm mostly drawn towards is wild experimentation with immersive technologies and really discovering things about the technology itself, but then reflecting back on us and seeing what it means.
[00:32:47.305] Cameron Stallones: For me the most exciting experiences I've had with VR are all about the creation of space and the scale that's possible. I went to art school too and I had a really good professor who did an amazing thing for us where he took us all to Home Depot. and he showed us these plastic things that you put in your bathtub if you're gonna like redo your spackle or whatever to protect your bathtub and they cost like a quarter but it's like basically a giant piece of molded plastic the size of the bathtub and he was like for a hundred dollars you can get like you know a thousand of these things and build like a mountain as big as like any building you know you can think of and it was a really good lesson Like for me as an artist, you know, who's never had a lot of financial success, you know, like I'm always trying to step to scale You know, I'm always like having big ideas and then having to like winnow them down because of resources So what's exciting about digital space is like, you know I saw a friend's piece recently where it was in a very small Little gallery and you put on this headset and all of a sudden you were in this incredibly vast space it's very similar to like putting on the headset and anti-gun and and that's really exciting to me and so the ability to experiment with creating things at scales that, you know, are not possible physically at the time, you know, is really exciting.
[00:33:59.041] Kent Bye: Great. Well, thank you so much for all the work that you're doing. A lot of really great innovations here. This idea of the D&D, I could see in the future that this is the type of the underlying infrastructure and the foundations it's going to take for these types of emergent narratives and different roles, people playing the game master and people playing the players. And so, yeah, just thanks for all the work you're doing and for joining me and to help unpack it all in the podcast. So thank you.
[00:34:21.411] Theo Triantafyllidis: Thanks a lot. Thanks a bunch.
[00:34:23.603] Kent Bye: So that was Theo Triantafillides, he's an artist who works with technology, one of the lead creators of Antigone. As well as Cameron Stallins, he's an experimental musician and composer who was doing a lot of the live music during the show. So I have a number of different takeaways about this interview is that first of all, well I love the fact that this was kind of like an open-ended D&D type of inspired experience where you create an open world and then it's projected onto the wall. And so then you have the two actors who have different ways of engaging with the virtual world. One of the main actors is actually in virtual reality for the majority of the time, doing different things in virtual reality, interacting with this open world, but also, you know, lifting up the headset and engaging with the other actress but a lot of things he's doing he's got this valve index and the index controllers and he's engaging and interacting with this world and according to Theo a lot of this follows a inverse tree structure where there's a lot of variability and decisions that they can make at the very beginning and then as they move on through these different worlds then they kind of converge and eventually end up ending with the same type of ending each time as you watch this experience. So I think technologically, this is really interesting to see where this is going to go in the future, especially if you think about, you know, what would it be like to have an open world exploration that had that D&D aspect where you're able to explore around and maybe have a narrative type of experience. But then in this case, because it's something that you see over and over again, then it kind of converges on this deterministic singular ending. But we'd want to have some sort of other aspects of the open world exploration, but to see how they're able to use the game engine as this neutral mediator for different interactions with non-player characters, different environmental conditions, different kind of random events that happen throughout the course of navigating this open world. And as a musician, Cameron has to pay attention to all these things and try to do this real-time reaction to providing different sound effects. And so it sounds like they had to go through a lot of different iterations of rehearsal and then through that rehearsal process, coming up with different things. And then Cameron would need to go out and research and get more sound effects. And then as they did enough of these rehearsals, they're exploring the probability of space to the point where they're, they're trying to be as prepared as they can to be able to have a little bit of variation. And Theo, he's controlling the different aspects of the camera. And so he's in some ways taking the cinematic approach of having like this live theater that's unfolding, but yet trying to direct the larger unfolding of the experience through having these different five camera angles within the Unity experience itself. So I thought it was an interesting concept and idea, especially exciting to see if this is a first iteration, where does this continue to go in the future? But also, you know, he's running unity in the editor mode, which for anybody who's done different unity builds, usually you build it out. And from that static build, then you're able to, you know, have a consistent build that you put out there. But in this case, because he's still in the editor, it's a lot faster for him to sort of navigate in between these different scenes. Usually when you. have the full build, it just takes a lot longer to load up. And just for him, it just was faster just to have this approach of kind of running it open ended in the game engine, which isn't necessarily the intended use case of how you use unity to have a live performance and run it in the editor mode. But to me, that was just interesting to see how that was just what worked best for them for their workflow. You know, I think there is some other aspects of just designing this open world city. He has an architectural background. I think you can kind of see that different impact of having a vast world that is pretty large to be able to explore around and having different handcrafted plants throughout the whole world in this near to far future world where there's a lot of climate change and. Unpredictable weather and the version that I saw, the weather is pretty consistent, but I came back and saw the beginning of another iteration, just because I wanted to see what kind of slight differences would happen in the time. Then I saw it the second time. There was ways that they had to like modify their behavior because it's raining. And so they would have these really bad weather events that then they would have to react to. And that would change the way that the story unfolds as well. And so that's just an interesting thing to add in there, which is like in the future, having more unpredictable. weather events and then how to deal with that and how the society is adapting to this unpredictability. So there is a little bit of this live element where you could kind of see that there is a bit of leeway for decisions to be made and everybody's kind of watching what's happening both on the screen and what the actors are doing and having to really pay a lot of close attention and to be very reactive. And so there is this live excitement because you get the sense that anything could happen. And there'd be things that would happen within the simulation where funny moments where there would be in the VR world is projected on the screen, like a random boat that would just like collide and go right through them. And then there's a moment when the actress is just like, ouch, you know, like. like she's feeling this even though she's not really in VR but you know just kind of like little serendipitous moments like that that in different interactions that are happening that they're clearly in this yes and improv way of interacting with each other which created this chill vibe slacker-esque unfolding of people just kind of hanging out but Uh, anyway, I think the types of technical innovations that they had to be able to pull this out, they're integrating lots of different technologies and pulling in like the live music and projecting this onto the screen. Having somebody wear a VR headset for a majority of the narrative that's unfolding, you know, synthesizing all of these different systems doesn't always. operate all that seamlessly, but I thought they were able to kind of bring it together and it felt cohesive in that way. And for me, I'm just excited to see where this goes in the future, not only for these live performances, but what would it be like to have somebody who's playing the dungeon master type of role of having a live Unity open world where you're able to have a lot of different variation, but treat them as a dungeon master who's helping guide and direct the course of a journey or a story that you go on in this type of open world exploration. So finding that type of Dungeon Master DM guide who's really helping to craft not only the world that's unfolding in front of you, but also the deeper story and experiences that you're having. I think that to me is one of the most exciting things to come out of this experience to see where that type of idea will go in the future, as well as trying to synthesize all these things into these different types of live performances. So that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a list of supported podcasts, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
