Bag of Worms is a live XR sketch comedy performance developed by Matt Romein in collaboration with Peter Mills Weiss and Julia Mounsey that showed as a part of the ONX + DocLab MoCap Stage at IDFA DocLab 2022. It’s difficult to fully describe the surreal quality of Bag of Worms, but their synopsis does a pretty good job of summarizing the key points, “Wearing motion capture suits, Bag of Worms uses technology to create a new kind of hybrid performance, with the performers constructing and manipulating a digital playground viewable to the audience in real-time. The lines between the physical and the digital become blurred, challenging the audience’s perception of where the actual performance is taking place.” Because the audience can see the physical bodies of both performers, Bag of Worms plays with “creative violence” in the context of the virtual avatar representations of their motion captured embodiments that are being projected onto the back of the stage. I had a chance to catch up with Romein at the IDFA DocLab to get more of the backstory of how this project came about, how they’re translating some of the language of theatre into rules for this project, and the quality of attention within a theatrical proscenium stage with adding video screens and hybrid performance.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So I'm continuing on my coverage of looking at some of the different experiences that we're showing at if a doc lab Today's episode is covering one of the pieces that we're showing as a part of the onyx doc lab mocap stage and so matt romine has a piece called bag of worms, which honestly was pretty funny and kind of hard to describe exactly what was going on. They had a basically sketch comedy with two people that were acting. They were in motion capture suits with OptiTrack tracking, and they're essentially walking through a number of different interactions between two characters. And you could see the physical interactions, but then projected onto the screen was a whole other virtual realm. And it was kind of playing with how does that virtual realm interface into the physical realm in more of an imaginal, comedic way. And Matt Romine is also on the side as a character, as a facial capture, and you see this floating head that's omnisciently coming in and out of the piece and also commentating. And at the end of the section that I saw, there was a whole unpacking of some of the uses of violence that they have. And so he's doing this deeper commentary and exploration of the use of video game violence and showing a progression of Mortal Kombat 1, and how that was kind of a moral panic at the time, and then all the way up to Mortal Kombat 11, and just the advancement of gore and violence that's happening in that video game. So he was doing a little bit of a commentary in that, and embedding it into the actual narrative that he was showing there as a part of A Bag of Worms. So that's what we're covering on today's episode of the Wastes of VR podcast. So this interview with Matt happened on Monday, November 14th, 2022 at IFA.com in Amsterdam, Netherlands. So with that, let's go ahead and dive right in.
[00:02:05.669] Matt Romein: Hey, I'm Matt Romijn. I am kind of a coder slash performer. I've worked a lot in theater and dance in the past, and then I've been coding with Max MSP and Unreal Engine to make a lot of like, immersive video environments and tools for performers, sometimes my own performance, sometimes for collaborators. And lately I've really been working with motion capture systems and trying to think about them in the context of a longer form live performance that goes beyond just kind of eye candy, but a little bit more of like the physical in front of you is informed by the digital and vice versa, and there's this kind of symbioticness or feeding off of each other and putting it on a stage.
[00:02:51.102] Kent Bye: Yeah, maybe you could give a bit more context as to your background and your journey into making this type of immersive work.
[00:02:56.827] Matt Romein: Yeah, I studied theater as an undergrad. I was an actor, and then I realized I wasn't going to be doing that as a living, but I still enjoyed it. So I also did an art technology minor, and I did a project on circuit bending, which is taking apart old toys and instruments and making weird sounds. And my theater professors saw my presentation on that and made me start doing sound design. and then eventually video design for theater and there wasn't that much in the way of video design systems. This would have been like in the mid-2000s. So I started learning to code. I had these ideas I wanted to get out. That brought me to New York and I worked in a lot of experimental dance and worked as a technician for a lot of companies. Ended up going to grad school for immersive media technologies at a program called ITP. Really got into Unreal Engine and motion capture stuff there through some classes and access to the facilities. And then really dove back into the theater scene and started working more as a designer and really trying to go back to those connections in the world of people I knew there because I thought I could bring something interesting into it and also expose people there to something in a way that didn't feel like it was just a technician telling them to do what they usually do in front of a mocap system. And yeah, now I'm kind of focused on trying to make my own work with the collaborators I really like working with.
[00:04:29.650] Kent Bye: Yeah, maybe you could give a bit more context as how this project that you're showing here at IFA DocLab at the ONIX DocLab stage with this motion capture track system that's here. Yeah, how did this project Bag of Worms come about? Where did it begin?
[00:04:45.761] Matt Romein: Yeah, so when I was studying at ITP, we had a class that paired together dancers with coders, and it was a really fulfilling experience, but it was also very binaristic of like, here are the dancers, here's the coders, and I was like, I feel like I'm kind of equally versed from a theater background more, but I was like, It'd be fun to do this in a way where it didn't feel like there was this like separation of these things. And then I was actually teaching at NYU in the same program until this spring. And after the pandemic, I kind of had that moment of like, I've been thinking about this for a while. I kind of want to shoot my shot. And so I left teaching and I actually did some work for Lisa Jermoy, who is the first person in our showcase with the Maquette Project. We were talking a lot, she had some similar ideas and I did some of the avatar coding for her project and during Tribeca I was a part of a program with Onyx where they had a showcase and I didn't, it wasn't a motion capture work I was showing, just kind of a generated video piece and Casper came to the showcase and on the spot I just kind of made up a, oh this is it, you know, I should get Onyx to partner with DocLab and like bring the stage out, you know, and told them I would come up with my own piece. Lisa's got a piece ready to go, I know Matthew had been thinking about something, my friend Kat, we taught together and So I kind of saw the vision as Lisa was also, I should say, at this showcase performing an earlier version of Maquette. And I was like, yeah, we should put a whole thing together around this. And Bag of Worms... I really just was like, I want to get back to devised theater, playful kind of thing. So I reached out to my collaborators, Peter Mills-Weiss and Julia Mouncey, who have their own theater practice that I actually do most of their video design and technology design for. I said to them it felt like I was the bassist in a band going to the lead singers being like, hey, you know, I'm thinking of putting my own EP together, I'd love to have you sit in on it, you know, but they were really excited and the timing worked out perfectly. We had about a week of writer's room, just kind of spitballing really stupid ideas. I kind of shared some articles with them that I kind of referenced in my talk around violence in video games and my ideas around creative violence and just over the kind of like Looney Tunes kind of like almost sketch comedy kind of nonsense, but then we'll find a way to kind of thread it together and then We just kind of had a three-week dash of me coding and rehearsals, and my biggest rule was I just wouldn't do any coding while we were rehearsing. I'd write down ideas, I'd write down notes, but we can't take a break while I do something. You're going to get bored. Let's just find something with what we have, and I'll bring more tomorrow. And it was fun. The biggest thing was it was just like really playful and really fun. We were just trying to make each other laugh. We were trying to see what we could break and, you know, it was just a really nice, equal, collaborative kind of partnership.
[00:07:59.946] Kent Bye: Yeah, I really appreciated the context setting after seeing the show, because I was watching it, and it was entertaining, but there were some moments I was like, okay, there seemed to be a level of violence that was going up to a line and going over that line, but there was a way in which that it was being projected in a mediated screen, and so part of the humor was that there's a violence in video games that you see through mediated technologies, but yet here we see the physical representation of these actors, and you can see that there's no harm that's being done to them, and physical reality, but in this virtual projection of this world, you see all these things that are happening. And then, you know, there were some moments it was like sort of over-the-top gore and violence and that I was sitting there and being like, okay, not knowing the full context of, you know, how this project came about. And then when you described it at the end, I felt like I really got a lot more out of your artistic intent for what you were doing. Because there's a bit of a trope of film students who go in and do the Quentin Tarantino-esque sort of violence that doesn't have meaning beyond that. But there's a comedic element that you're doing and exploring, but also these provocations from these articles that you came across. So maybe you could set a bit more context for those articles and then how that was driving into the overall aesthetic for what you were going for in Bag of Worms.
[00:09:10.618] Matt Romein: Yeah, when I was at ITP I was teaching a class called Performative Avatars. It was actually based off a project I did for DocLab in 2016 called Meat Puppet Arcade. And I had these naked 3D scans of my body that you would beat up and toss around and do semi-violent things with. But again, with this kind of head tilted, maybe like comedy is the intent, but you can't quite tell. And I, as someone who grew up playing video games and just exploring them a lot and like being interested in breaking them, It just kind of got me asking questions about the context in which we do things in video games, and a lot of what we're doing is controlling bodies, we're playing with bodies, puppeting bodies, and like, especially coming from like a queer dance scene in New York, I mean, all these conversations around like, how we treat bodies, like the autonomy of bodies and how we view and look at bodies. So in a video game space I didn't feel like I was, I felt like maybe more of a self-reflective kind of like trying to think more about what am I doing and the kind of context collapse is a word I keep thinking about of like I'm playing a video game and you know I don't think about what I'm doing but if someone sat down next to me and they see me pull a baseball bat out and beat someone to death on the street with it, they might be like, wait, what are you doing? There's something about when you're in the game and there's the structures it's imposing on you and the tools it gives you and whatnot that you don't really take a step back and think about what you're doing. And it takes other people being in a room and viewing it to kind of really pull yourself out of that for a moment. And I think one of the things I am aiming for with this piece, and especially even just like a theater communal context, is you said something about like, yeah, it almost felt like I was gonna go over the line a little bit at times, or the threshold was like, you're on a little on edge, and everyone in the room has a totally different threshold. And I love a space where a couple could come to the show and one of them is laughing their ass off and the other one's looking at them, realizing for the first time that, like, you think that's funny? Like, so I like when there's tension in a room of, uh, and I feel like a lot of times it's how much mediation you have already or exposure to video games or how rooted you're in that you've already kind of You find it funny because it's a little ridiculous. You've seen things not work. You've seen things break. You know, when a video game tries to be, especially growing up in the 2000s, if it tries to be realistic and earnest, then you end up with something where the animation mechanics, like they're just like sliding around or something. But now I'm just rambling a bit.
[00:11:51.589] Kent Bye: Well, I appreciated both the setting the context of the violence in the video games, but also you were talking about the Sims and the process of killing Sims and your own experiences of that, but also the Mortal Kombat, which I remember playing on the Sega Genesis back in like maybe 1992. Mortal Kombat 1 and you know, some of these fatality kills that they have where you rip out the skull with the spine that's there. And then I hadn't been keeping up with the evolution of the amount of gore and violence that was in the latest iterations of Mortal Kombat 11 in that you showed sort of the equivalent cinematic from that game. And I was really quite taken aback. I was like, wow, that's like, I felt that same sort of moralizing like violence and video games where people were seeing the very pixelated digital representations of that back in the early 90s but then to see how far it's kind of evolved to where it's at now in Mortal Kombat 11 with all this gore and violence that is at a whole nother level of photorealistic realism that is like really like a wow this it felt like your show in a way that it was much more pixelated in a way that was not nearly as bad as some of the stuff that's already out there and so if people don't have that other context though they could be I don't know if you imagine when you're performing this if that you would always at the end contextualize things just to say why you were doing it because there is a bit of an artistic intent that is a little bit unclear as I'm watching it it's just sort of like using violence as a I guess comedic trope But there is a deeper commentary that became more clear to me at the end as you were unpacking all the origins of all these other stuff and going from the sins and the brutal kills into all the stuff that's happening with the latest discourse and violence in video games. And so, yeah, I'd just love to hear any other reflections as you think about what you presented here in the Doc Lab. context and how you think about in the future developing it and if you would sort of have that context at the end or if you just sort of leave it as standing on its own right in terms of a performance.
[00:13:50.760] Matt Romein: I think I'd always prefer to have it stand on its own. I could imagine, it's always context, you know? I think if you walk into a theater, my expectation walking into a theater is that once the show's done, the director's not gonna come out and be like, I need everyone to stay another 30 minutes while I explain everything. Like, I want the experience of what's on stage to kind of stand as its own thing and for me to have a chance to talk with other people about it and think about it. In the context of a festival or maybe more academic, I love to talk about these things and provide more context and engage in a different kind of, not necessarily a different dialogue, but a dialogue that might be more expected. I'm trying to engender a bit more, but I'm a very non-theoretical person in my own practice. I like to absorb things through osmosis. I love to keep tabs of thoughts and ideas and the real world things and articles that kind of like sparking some of the things that are on my mind. I like sharing that. with my collaborators. But we're not A to B, kind of like, we're not saying, we have this article, so we've got to make that piece that reflects it. But instead, like I said, osmosis of like, we download all this stuff. And then when we go into the creation process, which is just a completely different art form than writing a lecture or writing a paper. It's just something that's kind of living in us. And actually, when I was going over, My notes for the lecture, I had completely forgotten about in Brutal Kills, a lot of the characters are modded to be like Iron Man or Teenage Mutant Ninja Turtles. If you've spent any time in VRChat, it's just this... copyright hell of, like, Pikachu with a bong on his head and, like, Nickelodeon cartoons with, like, medieval armor. And so, in our piece, when we're using Coca-Cola, using Shrek and all that stuff, I was like, oh, it's that context collapse I had kind of talked to you guys about. It's the cognitive dissonance of these really intense things happening, but with these pop culture icons that confuse you and like you can still be traumatized but the trauma is more confusing and like we never talked about that is why we brought those things into the piece but that was a connection I made after. I do think if we were looking to make a full evening length version of this work and we're still thinking like what's the structure of the world we're building and the characters in it and I think there's already a little bit of a feeling it's almost like an art lecture, like a very deranged one, like I have this kind of all-scene director talking head character and I'm asking them to play out these different scenes. I don't think it would be so earnestly, like, my talk and ideas translated, but maybe that's something we lean into a little more for my character of like, yeah, presenting things to the audience in a way that feels like a performance lecture is a trope of some sort.
[00:16:53.655] Kent Bye: Yeah, I mean, I don't know what the full-length version would be that stood on its own. The version I saw had the context in the end and I appreciated it because as a documentarian and oral historian in this space, I always enjoy hearing about the process of creation and that was a bit of an insight for how this project even came about with some of these articles that were discussing these things and using those as a bit of a launching point. Because there is a bit of a, I guess in comedy, there is a way of having different contexts mashed together and that There's some aspect of the context collapse or there's ways of things that make sense in one context and putting another context And so in this show you have people in a physical context, but you have a virtual representation of that context And so you have the mediated context juxtaposed with the physical context and between those two you're seeing how those two different worlds interact and I guess if there's a theme that I see is that you're kind of playing with almost imagining if they were in this actual virtual context, what would happen if he was kind of responding as if the virtual world is kind of interfacing with the physical world. And that seemed to be a common theme towards the Bag of Worms interfaces between these two actors and what's happening in the immediate space. And so, yeah, I don't know if that's sort of an accurate recap or how you start to think about the interface between the virtual representation that you have in this piece versus the physical manifestation they have with these live actors on stage.
[00:18:08.101] Matt Romein: I'm not even... I'm not even sure what I was trying to say in doing that, so much as I just, theatre has its own language, and I was like, how does this fit into a theatre language? How does it fit into a language of what an audience is looking at? Where does your eye go? And I really wanted to hold myself to not creating something where it's a one-to-one translation, like there's a body on stage, there's a body on the screen, and it's a little different, the colors are different, or it's this creature or character, and we know it's happening live, but you don't need to pay attention. We actually encourage you not to really pay attention to the body that's doing it, the physical one. Just know the magic trick is there's a person there and there's a thing on the screen. And then that's, I think that's why you end up with a lot of things that feel like tech demos is because it's a demonstration of the technology. So I really was just trying to use this as an opportunity to try and in the context to like kind of talking about of like violence but violence without consequence wait no one actually is getting hurt but this is all up also upsetting you know like I think it's important for me to pull the Peters real body on stage and facial reactions he's giving and I actually didn't realize how much not giving the avatars facial reactions and body language, but letting Peter and Julia really do this kind of physical comedy they do so well, anchors the audience back to the physical stage. So you're getting a little bit of something each on each side. A funny thing that happened that we didn't even realize as we were devising and making all these scenes is Peter kept being the one to get killed. and really like not very long after that and I think Peter just has a really nice physical comedy training that maybe was always just funniest in the room for Peter the one to get killed by something and then we're like oh this is the through line even though we only have 15 minutes we gotta find a thread to connect all of these and that really kind of Peter becoming increasingly traumatized and in the last scene where it looks like he's the one that will kill but he kills Julia's character in almost a mundane way compared to the others and then he's so wrapped up in the trauma of it he can barely go on and Julia comes back with this avenging Shrek angle and it all just kind of is the crescendo of this arc for this character.
[00:20:40.603] Kent Bye: Yeah, I also really appreciated the temperamental qualities of each of the characters of Peter, who's very amicable, quick to respond, eager to participate, and happy to please. And Julia, who's a little bit more soft-spoken, doesn't have much to say, and just kind of more of a melancholic temperament, or just deadpan humor in it. And I'd love for you to expand upon, as you're working with Peter and Julia, how you're developing each of these different characters and their different temperaments interacting with each other as a part of the dynamic that makes it interesting.
[00:21:11.090] Matt Romein: Yeah, it was really fun to discover Julia's character that way too. I think we described it as like, you're a professional, you do your job, but you don't do an ounce of effort more than that. If someone calls a corporate company meeting and we've got to talk about things in an earnest way, you're sitting in the back being like, I mean, you're paying me to be here, I'll be here, but I'm not going to get into this. Just tell me what to do." And she does that so well. I think, for me personally, my biggest questions are still what my character is. I think even the technology I had behind it with the facial sync and the lip busy mouth stuff like it was the most inconsistent in the show and I had a kind of fun broken quality but I'd rather have that be a choice than something that's kind of a limitation of the technology and how sadistic am I how in control am I how caring am I for them you know like I don't think I want to answer the question of why I'm having us do all these scenes but maybe a little bit more of a hint of why the show is developing the way it is and how my character is responding to that. And I think we talked about that eventually Julia will kill me and become the director. That was one arc where I was like, that seems to be where this would end up going. Julia's just got to end up being the one in control of this like deadpan stoic kind of humor. But yeah, just things to work on for the next version.
[00:22:44.787] Kent Bye: Yeah, because you do have like an iPhone camera that's on a stand and that you're at a podium and you're kind of directing things and you have stuff that you're typing on your keyboard to help trigger different stuff, but you're also you've got a voice that is an omniscient voice and sometimes and you don't see a representation of yourself, but other times you actually do have a virtual representation of your face that comes in and that it's connected to how you're moving your head in your mouth movements and kind of that an animated version of the sounds that are coming through the speech that's coming through. And so, yeah, I guess maybe talk a bit about the integration for how you're bringing in all these different aspects of your facial performance, but also the type of stuff that you are directing and coordinating and triggering from your computer as well.
[00:23:29.852] Matt Romein: One thing I really like having technology on stage with me, like I'm always pushing for someone not to be in the booth taking the cues, but for someone who's involved in what's happening on stage to dictate what moves forward and has the readiness to respond to it. And I was kind of just the process itself was me directing with technology and through the ideas Peter and Julia and I was like I just I want to be an actual character doing this. I want this to be a virtual space that I am You know, when we were in the rehearsal room for the show, I walk back, in the show I'm sequentially playing all the cues in real time, like I'm deciding when something happens. But in the rehearsal room I wrote the code that I had a big control panel and I could like, almost in an improv sense, add and subtract things or bring something up if someone asked for it. And so I wanted the integration of the technology to really be a part of the rehearsal process. And I think from there you have the idea of me as a director or me giving like a performance lecture of some sort. We just had to figure out how to stage me in a way that felt a little bit more omniscient, a little bit more not in the same playing field as Peter and Julia. And then also like my interest in just having different forms of motion and capture represented in this. It was fun to not always do a full body kind of thing. It was fun for me to think of like how can I give myself a little bit of something but then I can have my hands free for other things. I can stage myself on stage in a different way and it creates a further juxtaposition of who I am to their characters.
[00:25:17.081] Kent Bye: Yeah. I don't know if this was intentional or not but there was some moments that you were directing and triggering stuff where you were trying to find a certain trigger but you kept on triggering like the wrong thing and so then you had all these things that were kind of falling down and a bunch of worms and you were saying like oh no but it had this other comedic effect where you know the stuff that you were trying to have triggered from your omniscient perspective, then the people who are on the stage and in the virtual representation then have to deal with all this kind of mistakes that you're making. And so I thought that was kind of an interesting, I don't know if it was on purpose or if it was an accidental thing, but I think that's actually kind of an interesting dynamic too when you are somehow as the omniscient person in control, you suddenly are now out of control and that the people who are in this virtual world have to deal with all those consequences of that. So I thought that moment was actually kind of funny and I don't know if that was intentional or not, but. I enjoyed it.
[00:26:07.909] Matt Romein: I think I could be wrong, but I think you were thinking of when I was kind of demonstrating things after the performance. Oh, yeah, yeah, that's right. Yeah. No, but I mean, that was kind of how our rehearsals were like it was just this like total chaos sometimes and something I really was adamant with them about when we first started working on it. I just kind of immediately realized, I want a sense of messiness and clutter. And not just like messiness of what's happening on stage, but every time we do one of these scenes, I want objects, I want candy, I want worms, I want bloodstains, I want things to feel like from the first minute of the show to the last minute of the show. the floor is just like trashed with things and sometimes you guys are having to clean it up or like the context collapse of like Peter's head spraying blood but there's still pieces of candy from the pinata scene before left over there you know like I wanted that feeling of everything kind of climbing over the top of itself. And partly because, like, you know there are times where, like, a 3D object gets stuck in the wrong position or something ends up not behaving the way... There's a part at the beginning of the show where my body always falls onto the ground and I ask Julia to help me up and she accidentally kicks my body off the platform, but... The way I fall is always different and sometimes I fall without her kicking me or other times she does end up having to try and help me so I like finding ways to make sure that the technology is not predetermined. I want mistakes, I want things to, so much of acting is responding to things and technology can be really sterile and performance context because a lot of times the focus is to never have mistakes, to always make sure you know exactly what's going to happen. And I'm trying to find a controlled way of letting some things happen that we have to figure out what to do with in real time.
[00:28:02.492] Kent Bye: Yeah, I appreciated the level of immersion that both Peter and Julia had and in your presentation afterwards when you were triggering all this extra stuff, the frame rate was dropping so low because there's too many objects and they're trying to get the objects off the screen but they're diligently kicking them into the hole or off the screen, you know, but they're kind of immersed into this other world and you sort of have this plausibility that they're actually interfacing with these other realms and that they're not having you hit the button that gets rid of everything automatically, but they're actually doing the type of janitorial work in a virtual environment that you could literally just push a button and have it all go away, but it's creating a sense of continuity of this virtual space that gives this extra layer of believability of this virtual representation of this world that they're coexisting in this hybrid existence between these realms. And I think that interplay between those two is something that's really Interesting to be working with and I guess one thing that you mentioned that I want to have you perhaps elaborate on a little bit is the language of theater and the different insights that you're taking because XR and these immersive technologies are pulling in lots of stuff from cinematic storytelling or video games human-computer interaction, but there is this theatrical component and so I'd love to hear what you think of in terms of the language of theater and how you start to talk about what you're doing on the stage because it does feel like a theatrical performance but also has all these other immersive components and so how do you start to think about or talk about what you're doing more in a theatrical context but how that starts to translate into the immersive context?
[00:29:31.488] Matt Romein: I'm not sure. I guess, you know, one thing, I'll go with this. When I was studying at ITP and teaching at ITP, they really prioritized bringing people from a lot of different backgrounds. So you had graphic designers, you had lighting designers who were theater trained, you had artists that worked from a studio context, you had architects that worked in an architecture firm context. And I was really struck by, you know, everybody's in the same classes trying to make their first VR, AR project, or work with physical computing. I always thought it was a little understated how much you could see the training that people had in a specific field and how that informed the way they talk about things, how they staged it, and probably the biggest way was just how they presented the work. I just loved seeing, someone may have come up, like an architect might have done like a performance, but like what was important to them in the staging of it, of like what the room looks like, what the lighting would be like, or what they think an audience would want, was so different than my own intuition. And I think maybe what I realized as I've gotten older and kind of straddled working on immersive projects and VR projects and coding stuff and also being deeply rooted in theater and still connected to that. I just like, I really want to use both of those. Like, I think I can make something unique and different if I really let both those things I've learned inform one another. And I think a lot of times when we talk about these kind of collaborations between different mediums it is rooted in there being two people or multiple people coming together and trying to figure it out and great things can come from that. I think I'm kind of tooting my own horn a little bit and like what I'm trying to say is that it's nice to have a little bit of both of those things and pretty well and be able to jump in at any moment and have my theater background and my knowledge of how I want something staged to influence how I go back to the code or how I build the control system. for me to be able to stop my actors and be like, we can't build this thing, but here's an alternative of how the physics system works that we could probably build a joke out of and play with that. Because this thing we thought was a good idea is actually really boring to watch because it's really hard for you to use a pitchfork to pick up worms and feed me, which is something we tried and I was like, This is so, it just doesn't work. It's too hard for you to do. Let's totally pivot to something else. And that speed to be able to move between those things is really helpful. And also to say I see many people among different fields bringing all that in and I think that The immersive field is always going to benefit from people bringing something that they're trained in and have a practice in already and really allow to feel like they're versed in both of those or also to just find collaborators where you can go beyond that initial workshop that a university sets up between departments but really start to develop an artistic language together and really interrogate what thought process and what viewpoint each person's coming from.
[00:32:53.968] Kent Bye: Yeah, that type of interdisciplinary fusion that's happening in immersive technologies is what is a really fascinating aspect for me as I do this podcast and talk to people from different backgrounds. And so, you know, I feel like experiential design from the final XR immersive VR AR projects that I see has a certain, you know, qualities of presence of active presence, mental social presence, and emotional presence, and embodied and environmental presence. And I really see that the theatrical tradition is having a consistent spatial context that is distinct from maybe, say, the film language that has a little bit more of the framing that's able to control how close you're seeing things. And there's a lot more cuts and edits. But in the theatrical staging, you have much more of a consistent spatial context that's moving from scene to scene, but yet with the virtual mediated aspect you're able to get another layer of that onto starting to bring in the XR technologies into the theater so that then you have the actors who are looking at the virtual representation of what they're doing so that they have to coordinate their spatial context into what's going to be represented into that mediated context and so that interplay I found really interesting in terms of the Spatial negotiation that they had to do and how well the theatrical performance is moving into the more XR translation And in this case the predominant context is that people are sitting in a theatrical context watching them actors But I can imagine in the future people may be sitting at home and seeing something like this and maybe they'll be watching the virtual representation of a recreation, but they're also see a virtually mediated it'll be like a Incepted layer they'll be in VR watching people who are motion captured but then watching yet another kind of similar mediated context but I guess the point there is that how the language from each of these design disciplines are feeding into existing XR technologies and I think theater is informing the structures and forms of XR and VR and immersive storytelling more than say film although there are lots of cinematic aspects and then on the other side how there's going to be a aspects of the XR technologies that are being infused into the theatrical context. And I think your project is really an interesting intersection between those two.
[00:35:00.110] Matt Romein: Yeah, I was thinking about when we were working on this in the rehearsal room, we came up with a bunch of rules. They're not rules that are established by any means. I just was like, we're going to make that a rule. And one was we wouldn't move the camera. It was going to be a fixed camera the whole time, because I've seen how technology can dominate the theater performance you have to kind of tamp it down or like reduce how much it's going to do otherwise the people on stage get overshadowed like I mean there's like they're in suits with like these little marker ping pong ball kind of things on them with like cameras trained on them like it takes a little bit of work to make the audience stop paying attention to that but then we were also able to find cheats like there's a moment where Julia's head is a camera and there's a projection screen in the virtual world and this is the backdrop of the whole show where each scene is announced and we turn that into a live movie screen. So you're still rooted in this fixed spatial reality that, as you said, theater still really specializes in rather than this framing that film really brings to mind. But then we also found kind of a hack to bring in some of immersives tools, kind of affordances to play outside or just appear kind of like spatial. But I guess for me, I'm so used to bringing video cameras as a designer into theater processes that it felt like a natural extension of that sort of thing.
[00:36:28.684] Kent Bye: What were some of the other rules, if you don't mind sharing some of them?
[00:36:32.090] Matt Romein: I'm going to say there's no... Oh, we couldn't go anywhere else. I was like it's this yellow floor. That's it. We can add and subtract things from it. And that kind of fit into the camera thing but we're not suddenly going to snap or fly over to like a forest. We got to stay right here.
[00:36:47.663] Kent Bye: It's another theatrical conceit, I guess, in some ways. I mean, in theater, you have different scenes, but you're changing the staging, but it's pretty much being in one scene in a play, right?
[00:36:56.687] Matt Romein: Yeah, you know, if you ever see, like, they do this in musicals all the time, like, they have something on wheels, and it's like a bookcase. and then suddenly you're in a factory and they spin it around and on the other side is a bunch of like tools. I like the way that you can use like objects and scenery to suggest a new place that we're kind of doing with our 3D objects appearing and disappearing but like the stage itself is fixed and immutable and that felt like it fit the theatrical language I've kind of been talking about. Oh, another rule was like we have to have things from one scene have to bleed over into another scene. So like I was talking about with 3D objects from a previous scene still being scattered across the floor as we start the next scene. That was something we were really going for. Simple ones I've kind of mentioned already too, like Peter always is the one that's hurt, Julia is always the one that inflicts violence, unless there's like a real kind of fake-out moment where you think it's flipping, which gave us a nice climactic kind of arc to play with expectations. I did try to get them to cheat their body towards the audience. It was tricky, like I just want to applaud them. My actors were thrown into this with no experience in this and like, Upstage, if they had to tilt their backs to the audience, then they could actually see the giant projection of what's actually happening with their digital bodies. But then they would be facing away from the audience. And we only had a tiny 27-inch monitor at the foot of the stage that they could use as a reference. So they were really good about keeping their faces and their bodies aimed at the audience and cheating sometimes to see what's on the screen to get an orientation of where they are. And I think those are all the rules that can kind of come to mind.
[00:38:40.320] Kent Bye: Yeah, I think in terms of world building, you did a great job of building this world out and having a certain logic to that world. And yeah, I guess there's certain elements of the performance that are worth watching to see how they play out. But for me, what I found really interesting was just the temperamental interactions between the two with the real quick, amicable, eager-to-please type of temperament from Peter, and like you said, with Julia, just, you know, not, kind of more laid back and only doing the minimum amount of work to sort of get the job done. And so I thought those, and to play between those two different temperaments was also a big part of what I found like their performances that gave this sense of who they are as characters. There's this Robert McKee quote that says a lot of what characters are is they're putting them into different situations and how they respond to that situation is a revealing of their essential character. So I felt like the characters that were going through all these different scenes, still in an ability through how they're reacting to each of these impossible situations, you get a sense of who this character is as they go through these really off-wall conditions.
[00:39:42.338] Matt Romein: Yeah, I mean, I just want to completely shout out Peter and Julia because they brought so much of that to this and they have their own professional practice and theater company together in their own right that they make really incredible work in New York and that we've toured so I learned a lot from them and how much they brought to the process and they were just so generous and like what they brought to this kind of collaborative kind of nature. And especially for people like, you know, just like a totally different medium, I had to kind of keep reminding them, it's like, I don't think there is a language for this quite yet. So let's just keep trying. There's like no dumb ideas. And I think we, usually our dumbest ideas were the ones that we liked the best or felt the most playful or joyful, even in the context of this kind of over the top violence, which was totally necessary. Like we didn't, We want it to be a little difficult, but we don't want to traumatize anyone. We want to entertain people. We just want it to be a little bit confused while you're feeling entertained. But yeah, they really did so much to create these characters and discover things in the process. And then also even on stage, I think Peter really learned how eager he was to please when we actually had an audience. You know, we talk about in theater all the time, just like the energy of the room. When we did this at Onyx, like our rehearsals, there's times we had one or two people come by just to get, you know, people we trusted for feedback. But Onyx is this amazing gallery space, but the room feels like a XR immersive gallery. It doesn't feel like a theater, like the theater at the DeBrock Grand here at DocLab. There's like wings, you know, with people looking down from the side, and there's this beautiful red, like, paint along the walls and there's kind of like curtains and stuff and it just puts you in this room where there's just a vibration of energy that really helps us as performers just feel kind of like rooted and grounded to what we're doing.
[00:41:41.807] Kent Bye: Yeah, with the chairs also in an incline so everybody has a good view. Yeah, it feels much more like a theatrical performance space than more of the tech demo space of the Onyx, which is great for its own right, but yeah, it's a lot different than having a type of performance like this. So I can see how that proscenium context can help to set a larger performative context for the actors.
[00:42:03.873] Matt Romein: One of the reasons I love a theater context is it's still socially unacceptable to take your phone out or to get up and leave in the middle of it. I mean, people will, and they should feel empowered to do that to a certain extent, but I do think things that want to engage with theater and engage with dance but end up feeling like tech demos, I'd say a lot of the time it's just the place it's presented and the way it's presented. If people are standing and they don't have a good view and it feels like you can kind of come and go when you want or, you know, the things like the acoustics. I'm such a sucker for the lights down that's telling everybody. You have to tell people the piece is beginning and you got to tell them when it's ending. And I really feel like, I love in theater, I feel like you have like 15 seconds to walk out on stage and punch them in the face and then you have their attention and then you can kind of go. So, what I have most fun with is really crafting a way to tell the audience, like, if you give me your attention, I plan to entertain you and I plan to make something that will be worth your focus and attention.
[00:43:13.000] Kent Bye: Yeah, I think it really worked as a performance and a comedic performance. And comedy is not something that I see a lot within the XR industry. And so it's nice to see these other disciplines of established theatrical practices are able to bring the acting chops and the whole staging, the way of creating these comedic moments. And I thought it works quite well. And also, as people try to describe it, there is a certain amount where it does become difficult to really fully describe what the essence of the experience is, because it is so much beyond from anything else that people have normally experience, especially if they haven't experienced a lot of the interface between the XR immersive technologies. And so yeah, I guess for me, there's a lot of potential there for it to be expanded out into a full performance with lots of people. I can see how it could be quite successful. And so I guess in that spirit, what do you see next as you are shooting your shot in this context of this blending of XR technologies and theater? So where do you take Bag of Worms from here?
[00:44:08.917] Matt Romein: All I know at the moment is we'd love to do almost like a 60 to 75 evening length performance, something that could stand on its own. I would love to pursue that and we've already started talking about ways in which we could do that and then at this point it's kind of a question of finding creative partnerships, finding funding, but also context. I think it's been interesting getting feedback at this festival as it's kind of been all over the spectrum of where people see that it could be. And I feel like I've really hammered this in this interview that I really see it rooted in the theatrical tradition, like I want it. I would want it at Off-Broadway more than I would want it at some giant, I don't even know, like, not an XR conference, because I feel like I'm talking bad about a thing I'm at. I love being in these contexts, but I wouldn't want to make, like, the Instagram playground, like, oh, at the end of it, you can put the suits on yourself and take some photos and stuff. You know, like, I want it to feel like you went to a theater show rather than you went to, like, an XR gallery or an immersive event. And in that context I think a question of like trying to get a longer run in one location if we're all based in New York City or the cost-benefit of maybe trying to tour with this or even tour like between theater festivals or tour in these kind of like film festival kind of XR circuits. And then also just Establishing the reality of these systems are still really expensive and really hard to access and we've just been so lucky to get to work with Onyx and Onassis and have their support so far to play with this motion capture system that's still a big lift to get somewhere and install and the amount of people required to do it and I don't take for granted through Onyx and academic connections I've had in the past I've been able to experiment with them, but we'd love to keep experimenting with it and also just engage with other people that have curiosity about how this could be translated into theater and by extension also kind of dance communities.
[00:46:22.103] Kent Bye: And finally, what do you see as the ultimate potential for these types of theatrical performances mixed with motion capture and immersive technologies and what that might be able to enable?
[00:46:35.813] Matt Romein: I guess I'm a very visually driven person. There's just something about What you're seeing on the stage and like it's something that combines like your imagination with kind of a visual reality or like a tangibility almost. I think it's really exciting for me, like I feel like I'm almost like wanting to turn, like I usually do video design for theater and I'm like well what if the whole theater was the video design kind of thing you know there's something that I really feel like. What we found in rehearsal was just how immediately satisfying our ideas were. And I don't mean like based on the quality of them, but like, you know, not relying on this kind of like creating the moment through acting and play and make-believe and kind of like a suspension of belief, but like we could actually try something over-the-top and wild and crazy and actually see it. And not the scene of being like the uber truth of it or anything, but just I just felt giddy a lot. I think we all treated rehearsals as a chance to make each other kind of like feel joy and pleasure and laugh and like it was really easy to do that when you so quickly see your body and their movements translated into these kind of impossible tableaus. So in that sense it just feels like a further extension of creative tools that are available for people to create creative ideas and narrative storytelling in a way that people haven't seen before.
[00:48:05.643] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:48:11.427] Matt Romein: No, I'm just happy to get a chance to talk about it and share my work and always happy to talk more. And yeah, I'm so excited by all the stuff I've seen at the festival. And I'm also just really happy to be back in person for these kind of things. I feel like much more rooted back to why I want to do this by getting to talk to people in person and see them and see their work. And I just kind of have that human connection again.
[00:48:36.173] Kent Bye: Awesome. Well, Bag of Worms, I hope we start to see this performance come to different locations in New York City, or you kind of develop it out. I think it does have a lot of potential. I found it very entertaining. And yeah, I guess for me, a takeaway is if there are elements of the different type of content that you're presenting at the end, if there is a way to sort of bring that into the conversation at some point, or not, I don't know, then maybe that's, like you said, it sort of stands on its own, but as a performative lecturer, yeah, that's just something that I had value out of seeing that at the end, and so there is kind of a didactic turn in that, though, but it does help me kind of reflect upon what I had seen, and almost reevaluate, I don't know, I guess it's like this experience of watching something unfold, and you see the schadenfreude of it, and you know it's a mediated simulation, and it makes you laugh, but then it's sort of like, you know, this deeper interrogation of, like, why that is funny or breaking down. And I felt like there was this interesting social commentary that was unexpected, but for me, made it even more interesting or valuable. So, yeah, that's, I guess, my piece of feedback, if there's a way to kind of add a seed of that into the actual performance, but to allow it to ground into this relationship between the physical and the virtual and how you can start to explore the relationship between those two realms in a way that has these different provocations. And at the end of the day, it still has a level of comedy that just works on its own right. So anyway, really enjoyed the performance and wish you the best of luck as you continue to develop it. So thanks for joining me on the podcast to help unpack it all.
[00:50:08.614] Matt Romein: Thank you so much.
[00:50:10.828] Kent Bye: So that was Matt Romine. He created the piece called Bag of Worms that was showing at the IFA DocLab as a part of the ONIX DocLab mocap stage. So I have a number of takeaways about this interview is that, first of all, well, Matt's coming from a theater background, and I think his intention eventually is to make sure that whatever he's showing doesn't need to have a lot of explication or other lecture-style explanations. I actually found the lecture to add a whole bunch of context. It was almost like the degree of senseless violence and gore that was in the piece. At a certain point, it got to be like, what's the point of why he's doing this? some deeper reasoning for this type of commentary of the way that we interact with video games and video game violence. And I think that this was at the context of Ifadak Lab, and it's a documentary festival, which is usually looking at things that are not in this genre of over-the-top gore violence in a comedic sense. And so, I think part of the set and setting of this piece was a broader context of a documentary festival, and so it was part of this motion-capture stage of trying to explore creative expression. And so, yeah, just the ways that, you know, killing Sims or the degree of violence that's been increased, and so they're kind of playing with that as a trope, and also playing with this juxtaposition of what would it mean to have this type of over-the-top violence where you can clearly see that there's nothing physically happening to these people. They're just imaginatively projecting what it might feel like if they were actually embedded into the virtual context. That way, it is still a bit of a distance in terms of that commentary. But also, for some people, it could be triggering based upon some of the different experiences that they've had with gore or blood or violence. But overall, I think this is a type of experience that I think could translate pretty well for a general audience because just the acting by Peter Mills Weiss and Julia Munsley, they were just absolutely fantastic. Peter with his eager to please, more sanguine, happy temperament and Julie more melancholic and just soft-spoken and only doing the minimum amount of work that's required and just the trope that Peter would always get killed and just playing with that and just their interactions and having a lot of really funny moments but also a number of the different rules that they had created in order to create this continuity and taking these different lessons from theater to try to create a flow and improv nature so that there wasn't hard breaks in anything that they were doing. There were some aspects where they were dropping in new digital objects, but in terms of clearing stuff out, they always had this kind of odd juxtaposition where they're doing these physical interactions to clear out the virtual space or to leave some trace of the previous experience left behind. And so there is this kind of accumulation of different things to give it that extra degree of plausibility in some ways. Yeah, just very interesting to hear some of his different rules that he was having behind this, but also just to hear his own creative process as he's working with both Peter and Julia in a way that, you know, there's no deep coding, but everything has to be prepared ahead of times, and they're just playing with what works and what doesn't work with this motion capture technology. I do think that this has the potential to be expanded out into a show and it could be quite entertaining. I think that there could be something in trying to integrate more of what he was presenting and more of a lecture style into the actual work. I don't know if that, again, starts to explain what they're doing in an unnecessary way. I totally get why he wouldn't do that. But I don't know. There's a part of it that kind of changed the overall tenor of the piece for me after I saw a little bit more of the explanation about what he was trying to do. But if I was at just a normal entertainment experience with people in the audience, I think the audience had a pretty good reaction to it, and I think it has a lot of potential, and I'm looking forward to see where it goes in the future. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a mission-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
