The ONX + DocLab MoCap Stage at IDFA DocLab 2022 featured a number of experimental XR performances using a high-end OptiTrack – Motion Capture System within a theatrical stage with a screen that was projecting the virtual representations. Each of these projects are exploring how to blend aspects of the virtual and physical in the context of a live performance. I had a chance to get the backstory and more details on all of the featured projects from Matthew Niederhauser and John Fitzgerald, who are both from Sensorium & ONX Studio.
Here are all of the projects that were featured at the ONX + DocLab MoCap Stage at IDFA DocLab 2022:
- ONX + DocLab MoCap Stage: Maquette
- ONX + DocLab MoCap Stage: Once a Glacier — Voices of VR #1154
- ONX + DocLab MoCap Stage: Realtime — Voices of VR #1163
- ONX + DocLab MoCap Stage: Bag of Worms — Voices of VR #1164
- ONX + DocLab Mocap Stage: Dance Looper
- ONX + DocLab MoCap Stage: Choreographic Interfaces Past & Future
- Industry Talk: Capturing reality in motion — Voices of VR #1162
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So, continuing on my coverage of looking at different experiences at Ifadoc Lab, today's conversation is with Matthew Niederhauser and John Rodriguez from Sensorium. So, both Matthew and John were part of Sensorium and Onyx Studios, which is from Onassis Foundation, that had basically created this place that's in Midtown Manhattan. It's a gallery space that has kind of been transformed into this hackerspace, coworking space, that has access to all these high-end OptiTrack systems. And when I happened to be at Tribeca 2022, they had an open exhibition that were showing some of the latest projects. And I think that was timed with the Tribeca, because there were a lot of industry folks that were there watching the different pieces at Tribeca. But it was an opportunity to show what the artists from that community were doing with some of these different technologies. And I remember seeing Maquette, which was a really amazing dance performance with these professional contortionist artists who were just moving their body in really amazingly Acrobatic ways and to see how that was being represented on the screen with this kind of time-lapse sculptural Representation where can I've just imagined taking different snapshots as someone is moving but to put it into just one spatial representation So this is all projected onto a 2d screen as you have people dancing in front of you now So there's this larger conceit of having the live performative aspect but having the motion capture technology to give you a window into a mediated experience that you can watch and to have this portal into this virtual world where you're seeing this whole other dimension of reality that's playing out. And so they were working with a number of different artists to see what they can do with this high-end OptiTrack systems. technology, but to think about how does it get translated into a live performance, and how do we start to bring more people into the fold of engaging with these types of high-end, immersive technologies, but in a way that can actually scale up. Some of these, you could fill up a whole theater room with people watching them, and you can just enjoy this as a performance, both from a physical embodied live performative aspect, but there's this whole other dimension of the virtually mediated component of that that is adding a whole other layer. And so each of the different projects were playing with that in different ways. And I already had a chance to air an episode with Once a Glacier, and I'll be talking with Matt Romine as well after this conversation to kind of dig more into his take on this. And then there was actually another project from Ali Santana called E-Cat Sound Pyramid that was an installation that was showing at Ifadoc Lab that was using a lot of the technologies there at Onyx Studios as well. So lots of different projects that were born out of this. And yeah, just an opportunity to talk about how this came about and what they were trying to do with each of these different projects. So, that's what we're covering on today's episode of the Wasteless VR Podcast. So, this interview with Matthew and John happened on Monday, November 14th, 2022 at IFFIT.Lab in Amsterdam, Netherlands. So, with that, let's go ahead and dive right in.
[00:03:18.758] Matthew Niederhauser: Hi, my name is Matthew Niederhauser. I am a co-founder of Sensorium and a studio fellow at Onyx, a production exhibition XR space in New York. And I create new media works, XR works, whatever you want to call it these days. And yeah, just out here at IDFA working on some new projects, utilizing motion capture technology.
[00:03:46.750] John Fitzgerald: Hi, my name is John Fitzgerald. I'm the other co-founder of Sensorium and also a studio fellow at Onyx, where we are kind of half artists and residents, half technical directors of the shared production exhibition space in the Olympic Tower in Midtown Manhattan. Also here at IDFA producing four experiences that are showcased at the Onyx DocLab mocap stage.
[00:04:13.068] Kent Bye: Awesome. And maybe each of you could give a bit more context as to your background and your journey into doing this kind of immersive work.
[00:04:20.472] Matthew Niederhauser: Yeah. I have been taking computers apart since I was a kid, and then worked for a number of years as a photojournalist. And during that time, I started doing computational photography, stitching, panoramic work. And it led me into building 360 cameras when we started our company, Sensorium. and ever since then have been mainly into camera based capture for immersive experiences including volumetric, photogrammetry, all the fun things you can do in the real world to bring inside experiential projects. And since then have produced a host of works for big tech companies like Google, Apple, Samsung, but have also created our own projects out of Sensorium that have been at Sundance, Tribeca, and now returning for a second time to IDFA. So we are very technically oriented studio, but we are still led with new creative ideas and build custom hardware, software solutions to tell stories.
[00:05:33.379] John Fitzgerald: My background is primarily in film production as a director, cinematographer. Early on in the mid-2000s, I got really infatuated with building stereoscopic video images with off-the-shelf parts and rigged together camera systems. I was early on doing a lot of projection mapping in kind of DIY ways. and had some video art installations that fragmented screens by projecting into mirrors. And yeah, I think to fast forward ahead, Matthew and I got really into the capabilities that were newly available by taking apart GoPro cameras and lenses and building early 360 camera rigs in like 2014, 2015, which propelled us into the idea of immersive image making. So yeah.
[00:06:29.341] Kent Bye: Awesome. Well, I know we've crossed paths a number of times in the immersive festival circuit scene, and I'm remembering Metamorphic at Sundance 2020, where you had a piece there that had some Vive-tracked social experience with sophisticated like Quill and Houdini, a whole pipeline that you had to do this interactive piece. But I remember in that interview that you gave a bit of a sneak peek that you're going to be involved with like Onyx studio space that you were going to start to distribute this. And this was right before the pandemic had really taken off. And then I think the next time that I saw you in the flesh and physical reality was at Tribeca 2022, where you had a showing of a number of different pieces that also some of them showing here at IFFA DocLab. So I'd love to hear a little bit more of a context of the ONIX studio and how that started to come about and then the journey from those two points into what we're doing here now at IFFA DocLab with the ONIX DocLab motion capture stage.
[00:07:27.502] Matthew Niederhauser: Yeah, it's been a crazy few years ever since Metamorphic and sitting down in a nice hotel lodge pre-pandemic and doing that interview. But at that time, we were in the midst of building out the new Onyx Studio space. That project initiated out of Karen Wong, who is then currently at the New Museum and is one of the founders of New Ink. And she proposed to Onassis, who had this nice gallery space midtown, to build an XR production co-working exhibition space. And Onassis, which was trying to be a bit forward-looking in terms of their digital programs. And I think like a lot of cultural institutions want to be innovative, paying attention to all the new tech trends. We're into it. And then Karen quickly brought us in as studio fellows, which is a term that sort of combines us being artists in residence and technical directors. So we were there from the start working with Onassis and also New Ink in order to design the space, what technology was going to be in there. And we were all set for a launch in September 2020, which was obviously done with basically me sitting downstairs with a whole bunch of equipment by myself, given the state of the world we were at that time. And to tie it into what we're doing here, one of the first big purchases that we wanted to have available to the artists who are members of Onyx, and that cohort of artists are around 40 right now, was the OptiTrack system. You know, a high-end motion tracking system for people to play with and get access to, which is particularly hard in New York. unless you somehow have an NYU affiliation. So yeah, ONIX was launched as a co-working space, but now become more production oriented. Sometimes the word accelerator is used around for people who have established practices and are looking to get access to production facilities and also funding and sort of other opportunities to exhibit in places like Athens, back in Greece. And, you know, get connected into the festival circuit as well. So, you know, we've been doing a number of showcases, especially during Tribeca. We did Stephanie Dinkins in association with Sundance in 2021. We have a bunch of interesting partnerships that we're developing, public programs. I'm doing air quotes, thought leadership, you know, trying to create a community base for the XR and creative tech world in New York. And as you said, we had a showcase during Tribeca this year, 2022, and Matt Romine, one of the artists who eventually showed here and had been at IDFA before, essentially pitched, I would imagine, after a beer or two, told casters, like, why don't we bring the entire motion capture stage to IDFA? And it seemed like an absurd comment or idea at the time, but we sort of quickly took it up into a larger project. And that's how we ended up here. I mean, I don't know, John, did I miss anything else about the Onassis world?
[00:11:00.846] John Fitzgerald: Yeah, I would say we've been really energized by the way the New York art and immersive space has kind of been growing. And we've been fortunate. We were kind of there early on in 2014 when New Ink first started, year one of New Ink. We were members then. And we were really energized by the way people were just sharing resources and tools. And it was something I had never seen before and something that our friends in LA were jealous about. It was kind of this great, like, hybridity, blending moment. Karen Wong was definitely key to helping, you know, she had seen so many of us create projects built at New Ink and then presented at festivals like Sundance and Tribeca and she really kind of wanted to bridge a bit more of like an art world presentation sense. Oftentimes, presenting something at a festival, it's this great eight-week rush. You get your demo approved, and then you crush for it and build this project, and then there aren't a lot of distribution channels for presenting it afterwards. So you may spend a huge amount of money and time creating something that only a few hundred people see. So I think the vision from Onassis is also how to help bridge this idea of distribution and have a place where workshops and demos can be presented on a bit more of a scale that wouldn't be just like a regular studio visit. And so I think that's one of the really exciting things about Onassis taking this leap and putting an investment in to build a mocap stage there and allow artists to really demo and build things that they wouldn't be able to otherwise.
[00:12:35.179] Kent Bye: Just a quick follow-up on Onassis. You had mentioned Greece and Athens and maybe you could elaborate that a little bit of what the connection is between Onassis and Greece and what this cultural institution is.
[00:12:46.006] Matthew Niederhauser: This is where I'll probably get in trouble for not hitting all of the bullet points that would normally be given by some of our administrators, if you will. But the Onassis Foundation, we sometimes use the analogy that there's sort of like a bit of a Ford Foundation of Greece. They build hospitals, they do public programs, and they do have a big cultural component. And they have had a cultural component in New York, I'm pretty sure, since the 80s. And their promotion of Greek culture, and I think they rightly so, have a big stake in it, they say, oh, you know, we came up with theater, we came up with democracy, all of these major, you know, ideas that ping around Western culture. And they've always just like wanted to plant a flag in New York as a cultural center that Aristotle Onassis, the founder of The Onassis Foundation, you know, he loved the city and has been trying to bridge things back to Athens. So, we do have these opportunities where sort of the digital and innovation team from Onassis Stegi, sort of the cultural side of the Onassis Foundation, to show works out there they want to bring and they have brought artists from Athens to show works at Onyx, do workshops, produce things here. So this year is really the first year things are starting to fire on all cylinders, but we're always excited about building these bridges between Athens and New York and We took Metamorphic, which was that Sundance piece that you saw, and we turned it into a project called Polymorphic, which was taking the VR experience or the core interactive piece of that experience into an LED wall using these markerless motion tracking camera system. And it was amazing going there. Not just the food, but also just going and being part of a big group of artists out there. So they have a lot of big visions and I think they're focusing on building something that would be a mirror of Onyx in Athens. And it would be too soon for me to talk about it in many ways, but we've also seen there where they have these facilities potentially to also exhibit large scale immersive works, which is something that they have a long term vision for. So they've been an amazing partner. I mean, in that sense, like, we work for Onyx and Onassis, but we, like, came in from the New Ink side, and we've been super lucky to help try to guide the space in terms of providing production advice and support for artists, getting technology into the space that we know Onyx members, like, want to use and experiment with. And Onassis has been super supportive about these efforts as well of engaging with festivals like IDFA, and also trying to look beyond the festival as well. Like a few of these pieces, I know that Onassis is excited about potentially like What is the business plan to present this in New York? And trying to move beyond seeing the festival circuit as launching you into something that is potentially sustainable. I know this is something everyone's talking about all the time, but we're excited about what we're doing there.
[00:16:22.456] Kent Bye: Any other comments on that?
[00:16:24.647] John Fitzgerald: The other thing is that for Onyx members, they're really trying to bridge this transatlantic collaboration, and Athens is a hub that Onassis is focused in, because they're based there, but there's a lot of opportunities for other European and North American connections. Yeah, it's an exciting project. We kind of, like Matthew was saying, we got brought in as studio fellows, but really we've been helping them with a lot of the technology opportunities as well because we've been touring works on the VR and immersive festival circuit for a while. We've been able to kind of like make a lot of interesting introductions to hopefully build some new ideas together.
[00:17:02.390] Kent Bye: Well, I have to say that there was a bit of a underground art scene vibe that when I went to the Tribeca exhibition that you had at Onyx Studio, I'd just gone from seeing a lot of the immersive stories that were being shown at Tribeca. They tend to be very story driven and a lot of the different pieces that they're shown there. And then I walk into this Manhattan space, I go downstairs, there's like a room packed full of people and they're kind of crowded around this motion capture stage with OptiTrack and people that are I think I saw the Maquette performance was the first one I saw and it was like these dancers that were doing these incredibly acrobatic things and then there was like this really interesting translation of their body into like time-lapse and combining their bodies and just a really provocative artistic fusion of these different disciplines of the dance and the art and the All the XR technology and so I happen to just watch that tonight again with the exhibition where that was one of the four or five different pieces that were shown here in Amsterdam and so maybe you could talk about the process of working with various different artists to use the affordances of this motion capture technology and to do the different pieces and you had mentioned that you helped produce them or at least work on them and so Talk about that process of working with these artists to produce these works that were shown here
[00:18:18.080] Matthew Niederhauser: Yeah, I mean, for every ONIX member, one of the first things we do is a big onboarding of what technology is in the space and immediately trying to understand where they are in their creative practice. And they apply with projects in mind as well. It's not so much an incubation place like going to New Ink was for us. We do see it as an accelerator in so far we have people with established practices, have big ideas, and we're there to help execute them. I would say insofar as our roles as like the studio fellows there, we don't specifically produce every single piece that comes through, you know, for some of them, we're really there to introduce the technology, get them to the place where they can press record and then let it go. So sometimes I'd say we're more like, thank you, Matt and John, than like actual listed producers. But each one is very different in terms of how they're coming in and ready to use the space. So Lisa had this idea for a project that almost I feel like started with 3D printing bodily forms and was going to produce almost like statuary and ended up after being introduced to it, you know, building an interesting team of people. to create this averaged motion. She turned the statues into a live CGI render of the dancers' movements. And I think being at Onyx allowed her the space, time, and access to equipment to sort of take that idea into a really amazing direction. Then you had people like Matt Romine, who, you know, he taught at NYU. He's familiar with the system. He just came and just plugged in and just created something. that I'm still wrapping my brain around. And then Realtime, which was the project that I created, I've been working a lot with cinematic, like virtual production techniques, and I've been thinking about creating a short film in Unreal. And essentially when we were deciding to take this project to IDFA, I was essentially like, wait, like, let's do this live. You know, like I still want to work on the short film project, but I just got caught up into how we could stage it using virtual production, immersive VR, like an HMD, like headset and the body. And then Jabao, she's pretty amazing. She created this wonderful 360 piece called Once a Glacier that I'd seen. And she was a Nu-Ink member before becoming an Onyx member. I mean, Nu-Ink is a great feeder for like Onyx members. A lot of people come from that family. And Jabao, you know, we've sent out an email being like, hey, like, does anybody else have some works in progress that's utilizing motion capture? And she was like, I do. And I'm like, wait, you just have like a 360 piece. And then she's like, no, we're going to explode it into a dance and theater. And we're going to bring the girl from Alaska to IDFA. And she took the opportunity to turn a 360 piece into a performance. So, you know, there's such a diversity of, I would say, like, practices that are being brought in by this group of Onyx members of all these artists that, you know, they're manifesting in all these different ways, but it's also provided like this showcase that we just had. such a diversity of visuals and use of the technology. So we're there to provide all the production advice that we can give at all times, you know, but a lot of these artists are coming from a place where they're really ready to engage and build something. Oh, and I should say the last one is Kat Sullivan, and she was not part of the showcase, but she had built this tool to allow dancers in motion capture suits to loop their movements to create a larger visual piece in Unreal. But, you know, she actually would run our workshops at first when we first got the motion capture stage just because, I mean, she taught this at NYU. So, you know, as I said, I feel like we're turning into a magnet in New York for people who have pretty established practices in this realm. Some people are still coming more from like a dramaturgy or fine art, but we're there to be a resource for people who are trying to like elevate their creative practice in terms of utilizing technologies like motion capture for sure.
[00:23:00.987] John Fitzgerald: Yeah, I would say one thing that has been interesting for this Onyx DocLab stage in particular is that a lot of artists who work with mocap have been able to set up installations where an audience gets to come in and experience a single piece and kind of like sit with it. And VRI, for example, that Jill Jobim made. We exhibited our Zikr project next to VRI in 2018 at Sundance, and it was amazing to see people kind of really spend time with one project. And almost from the curatorial limitations of time and resources for this project, we kind of smashed four pieces together into a mocap performance, which I hadn't really seen done before. And so it was, I think, from an audience perspective, interesting to just see this kind of spectrum of ways that people are utilizing motion capture, you know, with Lisa's maquette and the depths of dance and movement exploration that she's doing to, like, Matthew's virtual camera in real time and putting a viewer into a VR headset to interact with another mo-cap performer. It's kind of just a wide range of mo-cap functionality in 90 minutes.
[00:24:13.023] Kent Bye: Yeah, one of the things that I was struck with by seeing, I saw all four except for Kat Sullivan's, I missed her performance. But there's a sense of being able to use the motion capture to take this embodied movement and to put it into a virtual representation, but to be able to project that onto a 2D screen while you're still watching the live embodied dance performance. And so what is the relationship between being able to see somebody's embodied movements, but then seeing the virtual representation of that onto a 2D screen? Each of the different pieces, I think, we're starting to play with that in different ways, where in Maquette, there's the live dancers, and they're doing all sorts of amazing acrobatic moves that you could just watch on its own, right? But then to see the abstracted visualizations of those into something that ends up being really surprising and novel, but also beautiful, symmetrical in lots of different, unique ways. Yeah, just to take the motion of the body and to lay it out over into different increments to create an entity with like 30 different legs that are all sticking out so that kind of like Sculpture like entities that are made but also that remains in terms of the sketch comedy that is based upon what you see in the physical reality and then the virtual representation is that something else that you're seeing and And then in your real-time piece, you're showing how there's a virtual camera that's there that's also recording stuff, but the audience has additional information than whoever's in the immersive experience in that context. And so, yeah, just all the different ways, as I was doing a bit of a survey of different pieces that I've seen over the last six years of motion capture, there was actually some new manifestations of how you were blending the physical and the virtual. And these different presentations are being shown here. And yeah, I'd love to hear any thoughts about what you see in terms of capturing the body's emotion and how that is being displayed in real time into this virtual representation.
[00:26:01.772] Matthew Niederhauser: Yeah, I mean, we were really giving a smorgasbord of mocap tech for the showcase. I would say that before trying to speak about each one, that we were really lucky to do this at DocLab, or it just made sense to do it at DocLab, because each one of these projects are actually still in a very early experimental stage. Each of us only showed a piece that was about 15 minutes, but the seeds had already been planting, they've started to grow, and it was the perfect opportunity to show them in these early stages. But each one of them, I think, are, especially after the experience of being able to present it to an audience like DocLab, which just facilitates a gathering of really interested, engaged people in terms of this creative space. They're premieres in a way, but they're also indicating, I think, a much bigger project that each individual piece is trying to continue to build on. Go ahead.
[00:27:04.715] John Fitzgerald: I was going to just add to that note, one of the things that we've been excited about is the way that IDFA and Casper and the curatorial team are really interested in lifting up the hood and exposing the tools and the processes. And so DocLab was the ideal place to do this project. I don't know if this would work anywhere else, not because of any other reason, but that doc lab is just a great place to experiment and show things that are in progress. But yeah, there's an approach that the audience here seems to really be interested in knowing the process and what's happening behind the curtain a bit. So the whole idea of presenting the motion capture stage with an audience and then a screen on the far side where you could really get a sense of how the technology was being used and manipulated and broken was really kind of Ideal for it and also the doc lab team on Friday on day one brought in two creators who also Were trying mocap for the first time creators from Amsterdam that were able to do some pretty interesting experiments and things that I hadn't seen before so it was a good moment to try out some new things and shout out to Avinash from we make VR who helped facilitate that as well and
[00:28:16.763] Kent Bye: Just a quick note on doc lab is that as I look at the work of William Uricchio And he wrote part of the history of the new book that just released with the collective wisdom co-creative processes And he made this comment in that book where he was saying that like 90% of the first films were documentaries because it was the medium that was sort of documenting aspects of the physical reality that was the world around us and so I There's a way in which that new emerging media the documentary aspect of it. I think is a way of forging forward like the experimentation so because of the it the International Documentary Festival of Amsterdam is already in the documentary form and this is really pushing forward the frontiers of what is the next Iterations of those documentary forms and so Casper doc lab for the last 16 years I think has been at the forefront of bringing together a community of practice of people who are For me at least really pushing the edge that's a reason why I loved coming here because I do get to see like where are the trends of where things are going in the future because you get a lot of artists that are thinking about the form in a way of telling stories in a documentary fashion, but using the technology to express things in a new way. So you end up having people who are getting to show projects to their peers of other co-creators who are also thinking about these things deeply. So it tends to attract a lot of people from the industry in that way. So just wanted to make that comment because that may not be obvious for why a documentary festival would be at the forefront of a lot of this immersive technology.
[00:29:42.342] Matthew Niederhauser: Yeah, definitely not a lot of nonfiction going on out there on the mocap stage for us.
[00:29:49.547] Kent Bye: Once a Glacier is, in some sense, right?
[00:29:51.468] Matthew Niederhauser: No, you're right, you're right. I retract that statement. Once a Glacier is trying to address a very real situation. Even though the technology was exposed in the setting, insofar as you do have dancers in mocap suits in front of These CGI renderings of what's occurring on stage I would say one common thread is that we're really each artist was trying to play into How their physical presence and the CGI? representation built on each other like there's a dialogue occurring in terms of what you're seeing on the screen and and the physical representation, which I think leads to a deeper appreciation or just sort of will get an audience mind thinking in new ways about bodily representation, where we're going in terms of having avatars, how we interact online, the whole gamut. I mean, I can speak insofar as how I've personally felt about some of these other projects, but Maquette in particular, in terms of like the temporal elongation of movement, you know, you are watching these super amazing dancers, Nikki and Francois, And they are contortionists, and it's sort of, you're blown away just watching them perform in and of themselves. But by temporally extracting their movement, seeing the CGI representation of striking different poses that build these sculptural moments in engine, and seeing that representation just obviously is a completely different experience. and is more deeply appreciated by seeing them physically build these amazing rendered worlds. Matt Romine and Bag of Worms and his work with Peter Mills Weiss and Julia Mounsey, the two performers, is beyond description in many ways. But once again, sort of the acting of Peter and Julia, the deadpan work verse the CGI world that they're creating this sort of absurd stage that includes violent acts and moments of transgression and It needs to be seen together like Matt Romine is just like created this performance interface that is sort of unbelievable but completely transfixes people in the audience and the humor that comes out of it but also sort of the shock of some of the representations you know if you were just watching these two actors on stage you know they're elevated into this game engine cgi world that just transfixes you and he is Masterful in terms of playing around and he has this crazy show control that he's doing live and he is a character as well While manipulating that world. I mean the whole entire thing is was like wonderful to see play out and I loved it real-time You know, once again, the piece that I made was, I think, especially trying to explode even another layer of representation that can occur and is needed in order to create these like strange juxtapositions, because you have the rendered version of the 3d world that yes, I'm controlling with a live virtual camera on stage. which is fun in and of itself. You sort of understand better how you can like control a cinematic representation of a 3D world, but you also have somebody in a VR headset, an unwitting participant to a certain extent, someone pulled from the crowd who is also seeing this world immersively in a completely different fashion. There's sort of like a third party who you also understand is potentially seeing an entire world in there in a completely different manner. And they're also getting messed with in another way insofar as pre-recorded motion capture moments of the performer, Ali Donnelly. So they're also seeing something that the audience knows is not there, but they think is embodied. And so the VR participant is also getting played with. I mean, they're getting played with the entire time, almost for as a spectacle for the audience. But, you know, there's a juxtaposition of the actor embodied on stage, how they're being represented on screen as me, as like this virtual cinematographer. But then also, why not throw into the mix you know, this person in a VR headset who, you know, adds an element of risk, but also you're trying to understand their viewpoint as a participant in a VR headset in a completely rendered world. I don't know. I still don't have a short, succinct log line for real time. I came up with one that's like, it's a satirical metaverse hero journey using virtual reality motion capture and virtual cinematic technologies, but I don't know if I've quite got it yet, but that's happening. And then almost there, the last one being Once a Glacier. It's a slightly different manner in which the embodied performance of this girl who flew from Alaska to be here. Annika Schmidt, thank you, John, who, even though there's more video as background, it's not like a real-time representation of her movements on stage. There's a juxtaposition of her story being represented in CGI and her being on stage. And there's also a really beautiful sort of opening dance piece by Dorothy Oberle, who becomes the glacier, you know, this object that's so important to the Once a Glacier story. And it just completely gives a different spin on this amazing 360 video that she created and to see it embodied and juxtaposed with the CGI. And I don't know, once again, it's just like a completely remixed layering of how the performers on stage help create a world and the audience appreciating the juxtaposition of those visuals with the performance themselves. And so, I don't know, it definitely feels more like theater in some ways. Yes, like mine actually. bring someone in to participate, but it was a sort of grand experiment in this juxtaposition of performance and how it can help create a digital world that an audience like could appreciate. You could actually also imagine this all being live streamed in some strange way into Metaverse platform even as like flat cinematic video or you know I think that's a potential direction for some of this and in terms of having four months we weren't like we're gonna also do this live online but I think it's been like an amazing first step for the Onyx community and when we come back from this and we show this to everybody else it also gets them excited and from us as technical directors at Onyx as well we're also sort of like okay like this is sort of the strange new paradigm with performance it's not completely new of course but we're playing with it in this very cool way and then How can you maybe even have something like this done so it's amazing for people in a theater, but then it can also manifest itself in a completely digital manner as well.
[00:38:00.897] John Fitzgerald: I think Matthew's descriptions of those projects is an interesting, you know, to hear together all those different pieces that pull from dance and almost like comedic displacement to almost like live TV, reality TV style filming to kind of deconstructed documentary with dance as well. It's interesting. We have a great creative partner in Vallejo Gantner, who's the director of creative partnerships at Onassis and comes from a theater production background. So it's been fun to have him on board and help sail the ship because it's a wild ship with a lot of weird things happening. And also, Podromos Siavos, who's the head of digital innovation at Onassis, who has been excited to take a lot of chances and you know, without them we would just be in a basement in Manhattan trying to pull these projects together. So it's been a really incredible team. They're two names, but it's a huge team of people that have helped make it happen and on a day-to-day basis support the artists that work out of Onyx as well.
[00:39:05.747] Kent Bye: Yeah, and having seen a lot of different iterations of using motion capture for live performance, and sometimes they're virtual, sometimes you're in physical co-located environments, and there's always this, what is the liveness of the live and what makes it a live moment? And I think there is that. element of risk or things that could go wrong or people that are coming into the audience and you know you don't know what's going to happen and it is all this real-time technology that may or may not work and so there's this live performative aspect of it but also just to see the juxtaposition between the physicality of the performers and the comedic performances, but also how the virtual representation is adding another layer of context to what's happening in the physical performances, and how some people may be fully immersed into that other context, and other people are not immersed in that context, but they're pretending like they are. And that's part of the humor, is that there's a context shift there, that if he was in this Matt Romaine's piece of Bag of Worms, if these actors were in these virtual spaces, and they did have haptic suits, maybe they would be actually feeling all these things. expressing of pain, but it's all sort of made up in an imaginal space and physical reality. So even though there's like this contrast between this extreme violence on the screen versus something that you clearly see nothing is actually happening in physical reality. So it adds to that comedic element of that contrast between those two things. So yeah, lots of really interesting frontiers of how these different realms are being blended together. Wanted to go back to the virtual camera because there is a way that you were as you were doing this virtual production live cinematography of Almost giving us the audience a live portal view into what the people who are immersed in the world or what they're seeing and so it's kind of revealing this broader world, but through this Lens of your directing you're able to show the audience, but you're also actually triggering different scenes which I thought was very interesting to see how you're unpacking how you are putting all these different sequences on top of each other and able to kind of have this keyboard on your right hand where you're pushing these buttons and actually triggering different moments in the scene so Going above and beyond just a casual cinematographer would normally be in a film, but actually into this other roles of director, theater staging, triggering, I mean, I don't know, all the different roles that you're combining in that. But as I was watching it, I didn't realize that. I figured maybe someone was behind the keyboard triggering stuff and then making stuff happen. you as the director, cinematographer, staging director. And yeah, maybe you could just talk about fusing all of this into this physical tool that you have this camera with a rig and everything that's on it to be able to be tracked, but also to provide a combination of all these different roles in this piece.
[00:41:40.244] Matthew Niederhauser: Yeah I mean it was at the end like a very deliberate decision to put myself on stage. I still don't know if this piece is like made for a full hour of work but it was a challenge to myself, but also I feel like to an audience to, you know, try to understand what I'm doing, like how I can manipulate like a cinematic view of a live rendered space that I'd created. So in Unreal, I put together all these different sequences of moving environments and also like triggering options for the participant and assign them across essentially like a number pad. So. my virtual camera setup essentially had a built-in way to control the computer that Unreal was working. So I was able to trigger out each of the sequences that the VR participant was involved in, while at the same time creating this cinematic view into their world for the audience. Would it be sustainable for an hour? I don't know. But it also, for me, like, I eventually ran into a time limit for Infit South, but it sort of started getting me excited about, quite frankly, you can start having all these different possibilities in terms of controlling the world. you know, moving beyond into different perspectives of the space and even beyond the one that I'm controlling. But the main point of me being on stage and doing that live cinematic intervention was just to emphasize sort of this layer cake of representation that the piece was unfolding for you. Trying to imagine what the person in the VR headset is seeing and how they're being tricked through having pre-recorded motion capture and live motion capture. Me and my movements as this cinematic representation on the screen. And then just like our physical, corporal embodiment on stage. And I actually even caught myself like making a facial gesture, almost as like a director would for when Allie was, I don't know, she does this one part where she's supposed to encourage the person. And I'm like, sort of like, oh, it was almost as like, oh, wait, what am I doing? Like, I had to like stop. I'm supposed to be like the guy in a black hat and the black clothes, like this cinema background person, not Totally in the background, of course, I'm very, very present on stage, but I sort of broke my character for like a hot second. Yeah, I mean, having the show control on the camera became essential, you know, at one point, especially in terms of a piece like that, like timing is everything. And I do feel it could be turned into something bigger. I mean, that piece itself, is it made for theater? I don't know. I almost feel like it could be more having like more people in headsets, like more actors, more participants in terms of how I think I'd want to develop it. But, you know, there's ways of taking the cinematic pieces that I'm creating and also playing that off of other participants. I mean, it really becomes a playground. And then in Unreal, like I was able to just like quickly build, iterate, I think I spent around like 450 bucks, you know, for all of the, if you want demons, you got all sorts of earth elemental demon options. But, you know, I was just able to create these fun, satirical, metaversal worlds like quickly and put it together in a way that it could work on stage. And people did laugh, so I think there was some success to it. But, you know, as I said, I still want to do something that's like a cinematic short movie, and that's just a completely different production process. And yes, I'm delivering something that can be seen on a screen. But as I said, I saw this challenge of doing a performative base of it. I think that it's sort of, I'm going to be going back to Onyx. We're going to be doing a workshop for motion capture, but I'm also going to be showing how to use a virtual camera as well as like a tool within this ecosystem. So yeah, I mean, it was a big experiment and I hope to like continue to develop it for sure.
[00:46:08.735] Kent Bye: Uh, so yeah, just as we start to wrap up here, I'm curious what happens next with this Onyx studio motion capture, you know, you had this Onyx doc lab stage here and it's a doc lab 2022. So yeah, I'm just curious what happens next after this.
[00:46:24.959] Matthew Niederhauser: What happens next at Onyx? Everything. Well, we've got a new cohort of artists coming in. I'd hate to call everything out that isn't written in stone. We've had to find, you know, working with a big foundation now. Like, they like to announce things at their own pace, and after all the Memorandums of understanding have been agreed upon, but I think we are excited about potentially doing this type of staging at other festivals. Also, potentially developing these pieces and doing something in January in New York, and especially trying to find the successes in an experiment like this at IDFA? And how can we invest in and build something that can be successful for like a wider audience in New York? I think that is a big thing that ONIX is trying to provide is, yes, coming and doing the festivals is an important part of like engaging the community experimenting with new works but we're all looking for new models of trying to take these projects and make them viable for larger audiences and to also like support creative practices monetarily outside of like the initial investment that an entity like Onassis can provide to get a project off the ground and like get us all out here in Amsterdam and on a stage. So other than that, you know, we will continue to be doing public programming in New York. We love the New York creative technology ecosystem, and we're trying to become a hub for that, especially as New York continues to emerge out of the pandemic. So there's still this growing demand for location-based entertainment that includes immersive experiences. I'm not saying like Onyx is going to be the next Van Gogh projection mapping thing or something like that, but I think that there's an audience for something more than that and that people would be willing to see a 45-minute version of Bag of Worms and all of its crazy intensity and especially in a market like New York, you know. So I think the big thing is to try to get bigger with projects coming from Onyx members and continue to take smart calculated risks and I think we're also going to like in Onyx, we're going to be doing things with facial capture, finger articulation, we want to like keep building the technical capabilities of the studio for artists. But yeah, we're excited about what 2024 holds for everyone. 2023?
[00:49:16.954] John Fitzgerald: 2023. I'm also excited for 2024. Yeah.
[00:49:23.022] Matthew Niederhauser: What year is it? I've been awake for seven straight days here operating this thing.
[00:49:30.358] John Fitzgerald: Yeah, I'm really excited for there's a new cohort of Onyx artists that are coming in and some of them have some pretty deep experience with motion capture and technologies that you can do forms of motion capture with, but some are pretty new to it. So I think there's going to be some interesting experiments. Some of them have some projects that they're trying to complete really soon. So I think the buzz of things happening at the Onyx studio will continue to grow. Yeah, also other opportunities to show work outside of New York continue to be on the horizon. Potentially some other things in Athens next summer, following up on the Plasmata Festival that we participate, Sensorian participated in this past summer. So yeah, hopefully onwards and upwards.
[00:50:16.349] Kent Bye: Awesome. And finally, what do you each think is the ultimate potential of these types of immersive technologies and immersive storytelling and these motion capture technologies and what they might be able to enable?
[00:50:30.150] Matthew Niederhauser: I think that they're going to be absolutely essential to any idea of, air quotes again, what the metaverse may be. And sort of embodiment is, I think, is going to be so important for future immersive experiences. And OptiTrack, the current technology that we're using for motion capture in Onyx, is amazing. It is also cumbersome. And expensive as well. And expensive. It's hard to access. I mean, it's great to have an artist space where people can get in and use it. But motion capture is turning into something that I think is going to become more accessible through a webcam, through your cell phone. just being able to use machine learning based motion capture libraries that suddenly can translate bodily movement into usable data for motion capture. And I think it's going to become very important for this flat out like communication, you know, like how people are represented, like moving beyond, we've gone from phones to FaceTime. I think we're going to have more readily available like embodied or volumetric presence. And we're also excited about, especially for virtual reality, we've been working with Elise Ananiri. He was one of the creators on Metamorphic. Everybody came into that experience had to have Vive trackers on hands and feet. And then now being able to place five or six cameras on the outside, and then suddenly having all that information be live for somebody in a headset to be fully embodied with markerless tech, I think is going to be huge as well. So it's going to be really interesting how this plays out. New headsets coming out next year as well. We're excited about what the Apple headset is going to be. And I always have seen and felt that like the iPhone and its LiDAR and all the technology for like the AR pieces, like is going to feed into what that headset could be. And I think a lot of people will take note. And I think that there are already companies are trying to do that, you know, where it's like the phone becomes a motion tracking technology that feeds live into an associated headset. I don't know. Now I'm just riffing a bit, but it's absolutely fundamental for embodiment. I think it's going to become more accessible, and it's going to make for more compelling, immersive experiences. And I think it's going to be exciting to see what happens over the next few years, quite frankly. And the longer term, I think it's just going to get crazier and crazier.
[00:53:23.587] John Fitzgerald: That's a good answer. Some of the things I'm excited about, especially in the near term, are as these technologies hit the market, the ways that artists like some of the folks exhibiting at DocLab are deconstructing the systems and using the components that are on the shelf to build really weird new things and ways of connecting to people. I think that's one of the most exciting things. Obviously we've gone through very difficult past three years for a lot of people and the idea of like human connection has changed a lot and I think I'm still processing what that means and how people will continue to live their life but also as these technologies advance we have all these opportunities for remote and hybrid digital connections so it'll be It's exciting to see how artists continue to break down some of the possibilities for using the technology.
[00:54:18.127] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader Immersive community?
[00:54:25.910] Matthew Niederhauser: Stay strong. Our time will come.
[00:54:28.712] John Fitzgerald: Stay vigilant.
[00:54:32.539] Matthew Niederhauser: You're all welcome to come visit us at Onyx in New York, onx.studio. Sign up for our mailings and, you know, we're always looking for new creators to get involved there. And we do have creators who are based in LA or based in London or based in Lisbon. You don't have to be in New York to be a member. And one of our greatest strengths isn't necessarily being a day-to-day co-working space. It's about having somewhere where you can go and execute on a part of your production for four or five days, or come to New York and have a great place to demo your work for curators, investors, anybody you want to share, potential collaborators. So it's why we are still so involved with Onyx, even though we have our own creative practices. We've just been excited about trying to build a community around the studio. And we also like having a tech budget to buy a bunch of toys that we want to use anyway, but we're also excited to let you use them. I would say that's the takeaway. You should definitely check out what we're doing. And we're always excited about people bringing new ideas into that community.
[00:55:55.075] John Fitzgerald: Yeah, I definitely echo the sentiment that Onyx is really open doors to our whole strategy is, you know, to explore things that we're not really familiar with. So the mocap stage is there. We're always looking for interesting projects to help creators tap into some of these new technologies and build things. So please come visit us or reach out and hopefully we'll see you all soon.
[00:56:19.881] Kent Bye: Awesome. Well, it's great to see the progress from getting the early inklings of what this Onyx Studio might be back in 2020 in January and to actually get a chance to visit there in New York City and to see some of these projects more fully fleshed out and these initial prototypes, I guess you could say, a proof of concept of trying to see what is the affordances of doing this live performative aspect using the motion caption technology. And yeah, I was quite impressed with the range of different types of experiments, but also how compelling it can be to have this live element of a performance with another layer of augmentation that's happening in the background and how that may feed into augmented reality in the future. There could be some early ways of how to blend the physical and these virtual augmentations because you're doing this on a 2D screen, but also having asymmetry of power, people being immersed within VR. And they have access to more information than the audience, but sometimes the audience has more information than they have. And playing with that as a performative element as well is also quite entertaining, and to see all the different manifestations of that through all these different projects. So yeah, thanks for doing all the work to help make all these projects possible, and to sit down today and help break it all down. So thank you.
[00:57:31.454] Matthew Niederhauser: Always a pleasure to be with you, Ken. Always. Thank you, Ken.
[00:57:35.579] John Fitzgerald: Appreciate it.
[00:57:37.341] Kent Bye: So that was Matthew Niederhauser and John Rodriguez of Sensorium, as well as the Onyx Studios that was in collaboration with creating the Onyx Docklab mocap stage there at the IFA Docklab 2022. So I have a number of takeaways about this interview is that first of all, Well, just to recap that there's these five or more different experiences that are being shown there. There was the Maquette, there's Bag of Worms, Once a Glacier, the Realtime, and then Dan Sluber. So the Realtime was the one that Matthew Niederhauser and Sensorium was working on. This was a piece that was just like a virtual production piece that was able to take a virtual camera, but with these different buttons and a little portable screen that gave Matthew the ability to not only control things that were happening, but also to kind of stage this whole Wizard of Oz technique where there's a live actor that's in front of them, it's a motion capture, and there's somebody who's come up from the audience into the stage. And so they have limited information for what they can see and in front of them in the virtual world, and we can see the physical reality. And so there's kind of a swapping that happens at some point where they start to play a pre-recorded embodiment of some of these different characters, but they have multiple characters at the same time, but they're all voiced by the same actor. So there's one actor playing two different characters, and they kind of either have one that's pre-recorded or one that's being live-acted. And there's different ways that they're each kind of touching and interacting with him at different points and different times, just to reiterate that there's the possibility of having multiple people. So there's kind of like information that the audience knows that there's only one actor, but the person who's in VR is having difficulty knowing that there's only one actor. So that's part of the allure of having privileged information as this third person omniscient perspective versus the first person embedded into these virtual environments and playing with that. And so there's all these different ways that Matthew was triggering different aspects of the scene. And the big part of this was also just sharing like how Matthew had constructed this unreal engine scene to be able to have these different layers and to have these different buttons that were triggering different things. And that was a bit of a theme of these projects because they were still the early prototypes of just getting the first five or 10-minute demo, and then from there, fleshing it out into either more of a large experience, but each of them were giving these different talks throughout the course of the IFA doc lab. And so, you would go watch some of the 15-minute demo, and then they would unpack different aspects of their experience. And so, the bag of worms from Matt Romine, I have a conversation with him to kind of really dig into that experience. But that was one that was really unique in the sense of arresting and alluring, and what they're doing is quite funny and entertaining. I could actually see The Bag of Worms as the one that I see as the one that's the most likely to go on and potentially get fleshed out into a full 45-minute or hour-long show, and I could see that having a very successful run in the context of New York City. So it's very theatrical in that sense. And some of the other ones, actually, the Maquette was really technically impressive in terms of showing talented dancers in ways that were doing this modulation of their movements in a way that was taking their dance to a whole nother level. You just watch what they're doing physically from a performative aspect, and that's entertaining, but then there's this whole other aspect of how they're capturing their movements over time and creating this time-lapse sculptural artistic visual representation of that so that was one that i also really enjoyed the maquette from lisa jim houry and i talked to jibo with once a glacier and the dance looper i didn't actually get a chance to see just because you know i was there for a limited amount of time and, you know, doing like 18 different interviews over the course of four days. So I wasn't able to see the dance super by Kat Sullivan. And it sounds like she's actually been a key part of helping to do a lot of the teaching and been a long time artists working on this. So sad to have missed her show and being able to potentially chat with her, but there's so many things that are going on at, uh, if a doc lab it's. a bit intense to try to keep up with all the things that are happening, all the different events, and to see all the things and then talk to as many people as I want to talk to. But anyway, my takeaway of the ONIX DocLab stage was it was really fun to see the different types of experimentations. And so, I feel like the DocLab is this place for more of experimental prototypes and to see what's possible with the fusion of all these different technologies and Yeah, some of these felt very experimental in some way and getting a glimmer of something that there's a seed of something that could be borne out into a larger show. Even the thing with real time has some entertaining qualities of it. The qualities of having people be in an immersive space and maybe may or may not know what the full context of what's happening. And so, yeah, just to see the more improv elements of taking people through that as an audience member. So kind of a blending of different genres. I'd say some of these are a little bit more documentary style versus other genres of comedy and sketch comedy and just trying to experiment with what's possible with blending together different elements of embodiments into the context of a live performance. So embodiment with live performance and these different technologies and and seeing how you can have the fusion of what the physical reality is happening versus what's happening in the virtual space. And the contrast between those two different realities creates some different techniques to play with. And so the artists were all kind of playing with that in some different fashion. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then you can spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
