One of the most powerful explorations of embodiment at Sundance New Frontier 2020 was Metamorphic. The creators describe it as exploring the “ephemeral nature of the self by allowing participants to redefine their relationships to their own bodies, to the bodies of others, and to the environment they share.”
The visual aesthetic is quite amazing in this piece. I’ve already been a fan of Wesley Allsbrook’s Quill art work in Dear Angelica, Sun Ladies VR, and 30-Seconds of Gunfire, but there’s something different about how they created their own custom backend to be able to parse and reanimate the Quill brush strokes with more fidelity and responsive control.
The brush strokes are dynamically animated both within the world around you, but also within your own embodiment. There are actually multiple timelines of animation that are contingent upon your interactions with the world around you including other people. It’s ends up producing a cutting-edge exploration of context-dependent, embodiment that is able to cultivate entirely unique and novel social dynamics.
At Sundance 2020, I had a chance to sit down with the creative team of Sensorium co-founders John Fitzgerald & Matthew Niederhauser, artist & writer Wesley Allsbrook, and programmer Elie Zananiri. We talked about the need to create an infrastructure to facilitate the dynamic animation of art, the collaborative tensions that can come between the creative talent and the engineering implementers, the throughput challenges of location-based entertainment experiences like this, their next milestones and desired architecture for scaling up the viewing capacity by multiple orders of magnitude, new exhibition opportunities at the ONX Studio by Onassis USA and New Museum NEW INC, and the challenges associated with cultivating second-order, emergent social dynamics within an immersive experience.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Here’s the trailer for Metamorphic
This is a listener-supported podcast through the Voices of VR Patreon.
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So continuing on my series of looking at some of the XR experiences at Sundance 2020, specifically the immersive storytelling innovations, the technological innovations, as well as the experiential design process for the different creators. So today's episode is about metamorphic. So metamorphic of all the different experiences that Assault Sundance is probably the one that's most focused on embodiment and playing with this connection between embodiment and identity and different social dynamics and interactions. And this fusing of dynamic art that is from Quill, but they have their whole engine where you have different animations that are unfolding and that are dynamic and changing based upon how you interact with other people. Really a quite sophisticated and interesting experience. So it's put together by Sensorium, that's John Fitzgerald and Matthew Niederhauser. The Quill artist was Sweet Albrook, who's done a lot of really amazing work with 20 Seconds of Gunfire, Sun Ladies, as well as Dear Angelica. And then Elie Zanoniri, he's a coder and writer and programmer trying to create the overall program to be able to do this type of dynamic art embodiment experience, taking all this information from Quill. So talking a lot about the creative process, collaborating with each other, but also the distribution aspect, as well as aspects of embodiment, cover lots of different ground here and packing their own creative process as they were developing this piece called Metamorphic that was premiering at Sundance and New Frontier. So that's what we're covering on today's episode of the Oasis of VR podcast. So this interview with John, Matthew, Wesley, and Ellie happened on Wednesday, January 29th, 2020 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:01:56.859] John Fitzgerald: I'm John Fitzgerald, a co-founder of Sensorium, a production studio in New York, and one of the co-creators of Metamorphic.
[00:02:06.735] Matthew Niederhauser: Hi, I'm Matthew Niederhauser. I'm the other co-founder of Sensorium. We're an experiential studio. We do all sorts of cool projects. And we are here at Sundance as another co-creator of Metamorphic.
[00:02:21.540] Wesley Allsbrook: Hi, I'm Wesley Ellsbrook. I like to draw and write.
[00:02:25.221] Elie Zananiri: And I'm Eli Zanoneri. I write code, and I'm also one of the four creators of Metamorphic.
[00:02:30.503] Kent Bye: Great. I'm wondering if you could each share a little bit more context as to your background and your journey into immersive technologies.
[00:02:37.032] John Fitzgerald: I studied filmmaking, primarily documentary filmmaking. Also was doing a lot of experimental video installations and projection mapping. I do a lot of after effects and that kind of 2D and 2.5D visual effects. And I've always been really interested in fracturing screens and expanding the immersive space that people get to interact with content. and first works were really in like 360 video and from 360 video started working more with interactive software and building more immersive VR experiences.
[00:03:14.475] Matthew Niederhauser: For me, my background started analog, like in darkroom and photography, and I guess also digital in terms of taking apart and putting together computers, and then ended up in visual anthropology, just thinking about how people interact with digital interfaces, and then I was a photojournalist for a very long time in Beijing and still doing, experimenting with installations and eventually came back to New York where we got sucked up into the experiential scene, started building 360 camera systems, post-production pipelines, and then got into anything else that expanded agency within experiences, which led us into game engine work and all these other different forms of making installations. And that's what we do now. And it's a lot of fun.
[00:04:03.925] Wesley Allsbrook: So I went to the Rhode Island School of Design. I was an illustration major there. I worked as an editorial illustrator and a maker of independent comics, like a few of them, for about 10 years. And then I started working in VR when a more famous graphic artist dropped out of a project at Oculus Story Studio. I worked on Dear Angelica, The Sun Ladies, 12 Seconds of Gunfire, and I guess Metamorphic, and a bunch of other odds and ends. What was that piece last year with the dollhouse? But yeah, that's how I got here, I guess.
[00:04:39.213] Elie Zananiri: I wanted to work in animation for a long time, and I was gearing up to that. But then when I went to university, I got a little scared and took a computer science class and fell in love with it. So since then, I've been just writing code that's more graphics-oriented, so working a lot in exhibit design. And I got into VR at Sundance, actually, in 2014. I worked on Clouds with James George and Jonathan Menard. And we were coming here with a version that was all Kinect-based. But then we got a DK1 and tried to make a VR version, too, like over Christmas. So that's how that happened.
[00:05:10.136] Kent Bye: Great. So maybe you could give a bit more context as to Modern Morphic, how this project came about, where it started. So where should we begin?
[00:05:17.060] Matthew Niederhauser: OK. Yeah. Why did you ask me whether I wanted to do something or not at P&A?
[00:05:24.625] Wesley Allsbrook: Well, Matt and John and I used to share the same agent. And I had just seen their work. Objects in the mirror are closer than they appear. And I thought it was really kind of wonderful. And I loved the way that they were doing non contiguous narrative work and I don't know like the install like Matt I'd asked Matt about it and he was like I'll never do that again that was too intense but like the install was so impressive to me and I was like oh these guys are so cool and then I I realized that they had been a part of all these other projects that I had admired over like the past few years at Sundance which I'd never been to before 2015 but like I I was like, hey, Matt, look at this thing I made. And he was like, oh, nice. It looks pretty. And it was just animated quill work. And he asked, what kind of work do you want to make? Do you want personal? Do you want decorative? Do you want sociopolitical? And I was like, oh, a little bit of all three of those things. Maybe we can come up with a mechanic that explores identity in some way. So then he was like, oh, I know this amazing engineer.
[00:06:25.679] Matthew Niederhauser: Yeah. I didn't say they were just pretty. I thought they were amazing. She showed me all of these worlds she'd been building in Quill and was stunned just being in them. But a lot of what we had done, you know, previously, our previous work is like How do you, A, bring people together in such a world? And how would this actually come alive and be able to interact with it? So I sort of had an inkling of the first mechanic that we've been thinking about, which was body swapping. Hypothetical bodies.
[00:07:01.197] Wesley Allsbrook: Yeah.
[00:07:01.737] Matthew Niederhauser: having like a hypothetical body and how do you enable it and then allow them to swap and that was sort of like the first spark in terms of like what became metamorphic and then talked to John about it and then I don't even remember when I first talked to Ellie but Ellie definitely came to mind because I thought he was bored recently and he also likes He also seemed like a guy who liked a big challenge. And then, yeah, that became like the next big step was figuring out how we could, from a technological side, bring Wesley's art into this interactive realm.
[00:07:41.595] John Fitzgerald: Someone said something recently where they were like, you know, we evaluate projects that we do and we figure out, like, there's a threshold of we figure out how many miracles it has to overcome to, like, complete it. And I thought that was funny because this project, in my mind, like hearing Wesley describe what she was doing and Matthew pitch this idea to Ellie was kind of like the whole thing seemed like a miracle. And it seemed like each stage was, like, even more complicated. And yeah, it was a project that kind of, like, the scope kept getting infinitely larger. And I think it continues. In a lot of ways, this iteration is like a milestone to launch it here at Sundance. But we have a lot of ideas for how to expand it in the future. So it's kind of like this big, shining miracle that we're all even here showing it right now.
[00:08:26.962] Kent Bye: So what went through your mind when you first heard about this project?
[00:08:30.220] Elie Zananiri: Well, they had shown me the art and I thought it was really wonderful and I loved Dear Angelica. I thought it was really beautiful. So when they told me Wesley was on board, I was very excited about it. And I've also always wanted to do a sort of project that was like generative drawing where you could like redraw art or like re-see how art was drawn. So this definitely fit. the build for that. But I was just thinking now, while you guys were talking, that I think what really helped me say yes was that it was like, this is going to be a self-produced project, and we're going to take the time we need to make it. So there was a lot of technical challenges. And it wasn't like, OK, we need to have this by Sundance in two months or by whatever, Tribeca in however many months. It was like, let's see what we can do. Let's figure it out. And let's take the time to do it. And whenever we have something ready, we'll show it to the world. And that was like, OK, great. That's how I want to work on this. Because it's half a research project and half an art project.
[00:09:17.031] Kent Bye: Yeah, because, you know, Wesley, you've been using Quill arguably for probably longer than anyone else in the world. You were there from the beginning, working with a lot of the tool as it was being developed. And so it has a number of animation capabilities built within it, but there is obviously some limitations because this piece was able to go beyond any other Quillstration I've ever seen. So you're able, through the engineering, to export and do all sorts of stuff to make it fully interactive. But for you, what was the genesis of seeing what you could do with the capabilities, but then what you wanted to see adding to the interactivity, but then was an expression of identity. Like maybe you could sort of step through that evolution of that process for you.
[00:09:58.217] Wesley Allsbrook: Well I feel like, I mean this is kind of a question for everybody because we all kind of ideated on what we wanted the mechanic to be and what the mechanic possibly could be given our time constraints. The idea initially was to cause strokes to flow from one body to the next and to have this idea of mixed perspectives, people occupying the same literal space but to different environments. And I don't know, I kind of want to kick this question to either Matt or Eli, because Eli is the one who built a system which was modular enough to deal with what was going to be possible with the project. In terms of the subject matter, I just drew what I wanted to draw and what I like to draw. And the only real stipulation was, oh, the environment that one user sees needs to be different from the environment that another user sees. Everybody needs to have a mutable perspective and also a sense of mutable body. Can you speak more to what you designed around those constraints?
[00:10:51.462] Elie Zananiri: It was all kind of new to us. Yeah, I remember the original idea when I kept trying to get people on board, I would email them and say, like, talk about spaghetti monsters. So one person was spaghetti and the other one would be, like, macaroni. And then you had, like, different shapes on your body, but then whenever you would trade, like, you would both be, like, fusilli, which is kind of like a mix between spaghetti and macaroni. I had no idea what fusilli was. Me neither, but I mean, I was just, and then nobody understood what I was talking about. So then we had to rethink of it. But yeah, we were really trying to do something where, like, strokes, brush strokes would come off of one body and go onto the other. And it turned out to be very challenging to do that. And it's not what we have in the current version, but it's still something that maybe we could do in the future. But I had to learn a lot about rigging, like how things are weighed down and you can't really just have stuff fly off from one shape to the other.
[00:11:38.776] Wesley Allsbrook: Can we talk about the rig body size? Because we have one rig for all of the different things. And that meant that I needed to draw things within certain constraints, which sometimes I didn't realize. So you had to fix things afterwards.
[00:11:53.043] Elie Zananiri: Yeah, we definitely had people that were always crouching when we put them into the rig, and we weren't sure why. Or they had really long hands. I think the purple character had extremely long hands, so they just always looked like they were sloths. They were just walking around kind of droofily. And then, yeah, it was just like, why is this happening? What's going on? It's like, oh, yeah, the rigs have to be perfect, and they have to be symmetrical. Some were just a little bit slanted and stuff, and one had a tree growing out of it. So there's all these, like, Things that we would just look at the art and be like, oh, that's pretty. Let's just rig it up. That whenever you try it, it would just make a demon that walks with like its knees inverted and stuff like that.
[00:12:30.007] Matthew Niederhauser: And there were some very beautiful ones too. They weren't all demons. I mean... We were trying to do something, I feel, with the game engine that it wasn't really built for, especially in terms of there's usually a sanctity to your avatar or the rig. It's usually immutable and in static in its form. I mean, yes, it can take on, like, You can play with it, but to like interchange with it and sort of have it interact dynamically with the environment and with other people was a major challenge. And we constantly had to play some whack-a-mole in terms of getting to a place where we actually felt like we were getting to the state where bodies felt like they were leaving traces on each other. and that was a large chunk of the first six months of the project was we did have some beautiful art from Wesley from the start but we just needed to get the mechanics there and then based on what Ellie was building then decide like what would then be sort of like a mini-narrative or a way of bringing people through the experience without dialogue so that they got into the space where they potentially didn't totally understand who they were or realized themselves as something else and then was able to play with that identity. But there was a big battle at first. How were they actually going to show up in Unity? and within the game engine. And then once we got over that was when we were finally able to get a demo together for Sundance and start playing with the actual mechanics of the experience itself.
[00:14:16.250] Kent Bye: As I go through the different experiences and talk about them, you know, I'd say like the metamorphic is the one that's focusing the most on embodiment and identity, specifically these embodied interactions that are really focusing on your body, but also the social dimension to that as well. So maybe you could talk about trying to orchestrate this emergent social dynamic and what type of desired behavior you're trying to have. I know that when I went through it, I happened to go through it with somebody who, I don't know if that was her first VR experience or it felt very new or she wasn't moving around. It was like, I was wanting to really just like explore my embodiment with her embodiment, but it was sort of like... Her elbows were tucked in most of the time. Yeah, so I feel like there's a certain aspect of a social experience where it can really be set by who else is in the social experience. Like at Toma, we were just talking about like how the whole purpose is to be dancing around a tree, but if you get into a social situation where no one is dancing, then you feel a social pressure that could take away a part of that social dynamic. So as creators, can you create a very specific social interaction and how you prime people to be like, okay, here are some guidelines for how to operate in this encounter, which is essentially a social encounter in this virtual space.
[00:15:31.729] Wesley Allsbrook: We haven't figured that out yet. We were worried we had made like a sexual harassment spanking machine. Well no seriously because like you know you put people in a headset and they can't see and they're in a space that they don't know and like it was like Matt was the one who had the idea to be like no we need an actor in there or we need the third party individual who is not mocap who's going to bring them together and they're gonna touch and I'm like they're not gonna touch. I was like, no, no. Like, it's like, I don't feel, I don't feel good about this for so many reasons. And he was like, no, no, no, listen, this is going to work. You know, I have been doing this for a while. It's going to be, it's going to be fine. And then it's actually like the curator here really kind of encouraged us to do the third party thing. And like, it's just like, it's like, oh, shari, shari, like, like Ellie is Egyptian, but we are all white and you're encouraging us to like cause people who can't actually see to touch each other and to be confused about what's real and what's not. So like, okay, sure, fine. But can you talk more about like the reasons why you wanted to include like, cause like you push for this so hard, like from the beginning, you're just sort of like, we need a person who's going to like make them interact.
[00:16:45.434] Matthew Niederhauser: Well, No, no, no, no, no. The way people have interacted has gone quite swimmingly. It's been great.
[00:16:54.555] Wesley Allsbrook: For the lady who was like germs and also I don't want to touch people.
[00:16:57.517] Matthew Niederhauser: Well, yes, you're going to have your outliers, of course. I mean, I think within the context of the experience, I mean, we sometimes call them acts within it. There is this first act where you're introduced to your body in the first place, like in this big mirrored gate so that you understand your potential new self, in a way. And yes, I did push for... But you were right, is the thing.
[00:17:24.147] Wesley Allsbrook: People are really enjoying it. There are some people who are like, stranger danger, for sure. But I don't think that we have, like, super triggered anybody yet.
[00:17:35.012] Matthew Niederhauser: No, not that we're aware of, but we got to go back this afternoon.
[00:17:38.234] Elie Zananiri: I've been respectful about it. Like I'd say, People have been pretty respectful about it. Whenever two people are in and one person tries to touch the other one, and as soon as they make contact, the second person takes their hand away, they just back up. Everybody backs up. There's never been a situation where somebody's just hounding somebody down for eight minutes.
[00:17:54.585] Matthew Niederhauser: They don't always back up. Sometimes they stay attached. We create a space where it's not enforced to interact except for the third person. Then we actually try to do it.
[00:18:07.084] Elie Zananiri: When I have noticed that people are just not into touching anybody, I'll just like wave at them and not try to grab their hand or do anything that's going to freak them out. I think that's a big part of it is we can be respectful of the space. You need to see how people are reacting and react accordingly.
[00:18:23.677] John Fitzgerald: Yeah, I mean, it is like a comfortable space that people are brought into and, you know, the onboarding experience is how we onboard people into the experience becomes like part of the dynamic of the interaction so that people realize they're becoming these virtual bodies. There is like a safe feeling that they have when they're in there. It's not like we're putting them in a headset and they're disappearing into darkness.
[00:18:47.372] Matthew Niederhauser: But I would like to address where the original idea did come from the fact that I think it's a very cool feeling when you potentially don't know whether another entity is real or not in terms of like manifests physically next to you or is digital. And it is like a feeling of trepidation, but also a potential discovery and like astonishment and even joy. And I would say it's been a really fun installation to run because it's been an ongoing little mini social experiment, quite frankly, and all of our docents love to do it because they're in there just watching like, oh my god, are they going to figure out that they can interact with each other or not? that moment when they figured that out and it is usually fingertip to finger this this slow because you're this very abstract character in a very beautiful world and with very wondrous music by Tim Fain who was probably should be sitting here talking as well because his music really brings such a a transcendental feeling to it, you know what I mean? Sometimes I'd be like, you're being too majestical, but he's just such an amazing composer that brought it together. So, you know, how they approach each other, I think it's like with a very good feeling. And when people figure that out, that that connection can be made, and that their bodies can sort of change each other and the environment, people start dancing. They start dancing with each other. There's lots of sensual reactions, people trying to hug. Unfortunately, that's when you lose tracking the most. But people, we've seen Stranger Stance with each other. It is really beautiful to see couples do it, almost rediscovering themselves in a strange way. Naveed and Bessie did it, who are creators from Ink Swordies. I almost cried. They're VR creators, and I watched them go through, and I was like, oh my god, they're like, almost everything we'd hoped and planned for was occurring, and that's really beautiful, and siblings, but yes, there have been two or three people who we've encountered who were like, they did not want to interact with somebody in a virtual environment, and they avoided the other person, and all the digital, like the fake characters, and there was one in particular where this woman was, because we introduced these fake avatars, and was literally running around the room trying to avoid them, and was almost cornered by them, and she yelped, and we went over to them, we're like, are you okay? Pulled up the heads out, she's like, oh, I'm fine. And went back in, and at the end, it was like, because the other person was trying to approach her to make contact, and he'd step forward, she'd step back, he'd step forward, she'd step back. I was like, do you guys know each other? And he goes, oh, that's my brother. And she knew the person well as family, but like that open space, there's a lot of risk going in there in terms of encountering someone, whether they're real or not, and how that's going to make you feel. And she just didn't want to do it. And that's a completely legitimate response. But within the experience, it's not one where the expectations are it's like has to be enforced until we introduce a third person this is like spoiler alert who comes in physically tracked and that person is there to bring them together to the center and they have a realization that Oh, wait, there is a third actual person in here showing up as an avatar. But as Ellie said, you approach them, it's a little bit of an art to it. And Ellie has got quite the moves, quite frankly. Maybe we can share some videos of Ellie working the floor. I think that's something we've been figuring out as well. You know, it's like, you can tell when someone's there and you really have to, like, draw them in. But man, when that moment hits and they realize there's somebody else in there with them or that sometimes they've never interacted with each other until that moment late in the experience. And when that happens, like, there's a lot of, like, delight. And it's something that transcends Their own bodies like a hackles up like where am I and what is happening to me? It's sort of moment that through our mechanics. We were trying to achieve Slightly enforced through the structure of this experience as I said, there's a first milestone It's a bit of an experiment, but we've been having a really good time with it in the installation.
[00:23:26.742] John Fitzgerald: Yeah, I One of the one of the simple installation design gestures is that you know, there's a curtain that divides the two spaces So one of the participants walks in around to the back of the installation and one enters through the front So they can see like through this lights a scrim that you know, someone is over there but when they put the headset on they have backs facing to each other so they even though they are in the same space, they don't necessarily know that the curtain will open. So, I mean, that's a really exciting thing when people, like, go to touch the curtain, and then when they go back to touch it later and it's gone, and they realize they can walk through, becomes this part of the journey of, like, the player discovery.
[00:24:06.391] Matthew Niederhauser: But I will say one more thing. We proposed it really in the sense where they started separate and stuff. And we went back, you know, when Shari came back to talk about us before we were accepted, you know, we were like talking about, oh, actually, like, let's cool it down. Let's take them in through the first same entrance together. Tell them that they're going to be interacting with each other. Let them know. And Shari pushed back on that. She was like, really? I read the deck and I thought they were supposed to come in separately and they weren't supposed to know. And she really pushed us back to an earlier stage. I mean, it comes with greater risk, even though we're there watching it and are making sure that people are behaving in a way that is mature and respectful and appropriate, but with that risk I think comes with the possibility of somebody having like a more incredible moment inside of the headset. And so Shari, we blame everything on Shari. She pushed us back in this realm and I think it's really worked out. But we were ready to like come in and adjust depending like if it really wasn't working or we're getting a lot of bad reactions. Curtain would have been gone, everybody could have come through the front door, but it's super positive.
[00:25:19.457] Kent Bye: I do think that there's like a value for these best practices of consent and informed consent in terms of these different things. But in the context of this experience, if you do that deliberately right before it does have the potential to have this spoiler potential to take a lot of the energy out of that moment. And so you have this tension between the responsibilities and the ethics around having a safe and consensual environment along with having the emergence of dynamic encounters with social interactions and touch that may be more powerful. So I feel like it's like attention that like maybe what needs to happen is that as people sign up and it's just like a uniform like stamps that people can put on like at conferences now they're like don't take my photo you know like do you consent to touch in virtual experiences? You put a little sticker on and then you could start to see, but then you'd have to disclose that in this experience, this includes touch.
[00:26:12.863] Wesley Allsbrook: So I think it's... Lots of your experiences at this point include touch or include some kind of unconventional interaction. And Ellie was saying earlier, it's like, oh, like when you take a yoga class, you just sort of like, you have that thing where it's like, do not correct me physically. You know, and also, like, if we do this again, I mean, the onboarding process does seem to be evolving. Like, we don't have the perfect BDSM contract yet, but also, like, people in that space aren't really amazing with consent anyways, but I think that if we get the opportunity to, we'll continue to iterate on this for sure.
[00:26:45.244] Matthew Niederhauser: Yeah, Elliot, how would you, what do you want, do you want to start talking a little bit about where we might take this with more people and as new milestones going forward?
[00:26:55.373] Elie Zananiri: Sure, yeah with the next step what we're thinking is like it's really fun with two people imagine if it was maybe 10 or like 20 people in a space each with their own perception of the space they're in and discovering it as they go so that's obviously many more chances of of unwanted touches or something like that so we really need to like design it both like technically and like socially like how we would create that experience but it seems like it could be a really interesting challenge because the whole thing is about discovery in a sense like I think the reason why it works is that nobody really knows what's going on in the beginning so everybody approaches it slowly like everybody just puts their hand out to see if somebody else real is there and that's the first touch it's never somebody who knows exactly what's going on and runs to the other person and grabs their waist or anything like that so if we just keep that idea with the 20 person experience where just like you're in the space you don't know what's going on and like kind of feel it out like you're all going to be as confused as each other I think it'll balance out somehow or at least I hope it will but we'll I do like the sticker idea and I think if it was more of a of a festival based sticker and not an experience based stickers like at this festival I consent to being touched in virtual experiences then you don't ruin the surprise and people get to do what they're comfortable doing.
[00:28:05.384] Kent Bye: So one of the most striking things about this experience is the visual aesthetic and the color palettes and the dynamic nature of it. Because I've seen a number of Quill experiences, but it felt like there's a different quality to this experience where it feels more dynamic, more interactive, or seeing things that I haven't seen before. So maybe talk about the process of how to go from Quill to how you ended up with this interactive experience and what you had to do to be able to get there. And so from your creation perspective, been limited some ways for how you have the output in terms of what you can do. But now that you have this new pipeline, were there new things that you could do and explore and ways of expressing yourself through using the medium of VR? Like, were there new affordances that you could do within VR as a medium using Quill that you couldn't just by using Quill alone?
[00:28:55.052] Wesley Allsbrook: Well, I mean, we have rigs. And, like, I would like to find a way some other time to bring Quill's animation capacity, like, to do looping animation so that I could kind of control more of that, like, later on. But what was so great about what Ellie built was that we were able to update assets really easily. And on any other project that I've been on, that has been the major complaint. It's like, OK, like, let's decide what things are going to be, and then you do them, and then you send them to us, and we update them. If we update them, we update them only once. And like what he built, I mean he's drawing a lot on Quill's capacity to record the order and speed at which I drew. And we use that, but he's also kind of tuned a number of other variables so that we can have our own timing. And yeah, like I think that what I like most is the ability to experience regret in any process. And I think that the more any person who's working with an artist can set up a pipeline that allows for both sides of the process, like if they can educate the artist on what's important to them while sort of seeing the way that the artist works and kind of like building that into their pipeline, then that is advantageous. And of course, getting anything out of Quill is useful because like, I mean, like, how's anybody going to see this art anyway? I mean, I can take pictures of it. I can put it online. I don't have a Facebook account anymore, so I can't do that direct-to-Facebook post thing. I mean, the greatest advantage is that I have collaborators, because otherwise nobody would know what I was doing.
[00:30:25.112] Kent Bye: Maybe you could elaborate on this system that you built.
[00:30:27.192] Elie Zananiri: Sure, so Wesley was saying, like the system takes in all the drawing information from Quill and just uses that as a basis for like a mini rendering engine in Unity. So all of the timings of every stroke she made, every point she made, like we have all that in and then everything is redrawn in the game engine. So whenever you start, you don't have anything drawn in and then it follows the same sequence that Wesley drew her drawings in. So for example, we have eight different rigs that you can have. And when you come in and you look at the mirror, you'll notice that some of them are drawn from left to right, some are drawn from inside out, some are drawn from top to bottom. And that's the actual order.
[00:31:00.492] Wesley Allsbrook: You told me to draw the grass from top to bottom. That was the most frustrating thing. He's like, I want you to draw grass, and I want them to be on three separate layers. And I also want you to make sure that you're drawing the grass always from one angle to the other angle. Sometimes you draw the grass from top to bottom. Sometimes you draw a thing from the middle. Why do you do that? How interesting of you. And it's just like, Jesus Christ, man. It's like, do you think I draw like a printer?
[00:31:22.582] Elie Zananiri: So yes, I was putting some constraints in. But yeah, the grass was a special case because the grass does need to grow from the bottom up, but for everything else, she did have the freedom to draw it like any way she wanted to.
[00:31:34.357] Wesley Allsbrook: It was a system that allowed us to reorder the way in which the grass was drawn.
[00:31:38.717] Elie Zananiri: I refuse to build that system. But yeah, so for the non-grassy things, like you could really see Wesley's thought process and the way things are drawn in and that's what I find really interesting. I really wanted to keep that quality to the drawing and that order to the drawing so that it's just like, you know, Wesley's drawing again but this time we're saying we're controlling when and how she draws and how close you have to be to something for it to be drawn in or not. And basically every single brush stroke has its own timeline that it's running on. And then there's also a global timeline for each model. And then depending on what the players are doing in the experience, each stroke either merges with the global timeline or it branches off and does its own timeline thing. And that's how things start looping separately from each other or start drawing in and out from each other. So there's just a bunch of mini animation timelines going on at the same time.
[00:32:24.016] Kent Bye: So it's very cyclical in that way.
[00:32:25.692] Elie Zananiri: Definitely. So to keep it active, things can get drawn in and then stay drawn in, or they can draw in and then draw back out and draw back in. And then we kept that behavior for things we wanted you to pay more attention to. So at the beginning of the experience, there are props which appear, which we encourage you to touch and interact with. And then when you do that, they switch to a different version of themselves. So they kind of switch perspectives in that sense. And to make those more attractive, we have them kind of pulsating and drawing in and out after they draw in the first time. The things that are more static just draw in and stay static, like the environmental elements, like the sky and the more far-field elements. And your bodies yourself, they change all the time. So whenever they draw in, they get drawn and stay static. Whenever you trade traits with somebody else, you'll see things draw out and then draw back in on the other person. And then at the end, when you do a full body morph, then everything just kind of, like, feels very flowy. And it just has this really crazy effect. Like, I really like looking at my arms when it's doing that flowy thing. It does feel very alive, and it's pretty cool. And we can also change the weight of everything to fade things in and out. I didn't want to do any traditional fades, really. I wanted everything to be based on how the art is drawn. So if you get close to an object with the camera, it'll actually undraw itself instead of just fading out so that you don't have a weird clipping issue, but it's also beautiful. So yeah, you should stick your face in a statue next time you do the experience. It's pretty cool.
[00:33:47.824] Kent Bye: So that timeline, the local and the global is interesting to me because it feels like you're deconstructing the monolithic linear timeline where there's only one timeline and everything has to operate in that. But each of these objects, the object-oriented approach, they can have their own timeline. They could be off on their own. They can converge. And so what would be an example of an event that would be triggered by a global event versus a local event? What's the difference there between the local and the global?
[00:34:14.150] Elie Zananiri: So a global event would be when your first character draws in. This is all based off of one drawing. So let's say you're the yellow guy with the big hair, then that's all going to draw in using the same timeline. So this is getting drawn in the order that Wesley drew it in. Then whenever this yellow guy with the big hair meets the red guy and they touch hands, then some parts of the yellow person's arm are going to get drawn out. and some parts of the red person's arms are gonna get drawn in yellow. So whenever that happens, those brush strokes detach from the main timeline, undraw themselves using their own timeline, and then the strokes that are getting replaced are drawing in using their own timeline as well. So it's kind of like, yeah, I guess that's basically one good example of it.
[00:34:52.881] Kent Bye: So there are events with the touching each other that then trigger these other events then?
[00:34:57.203] Elie Zananiri: They're all events either based on interaction or based on timing in the act. So we do like to keep the things progressing. We say, OK, you now have 20 to 30 seconds of free play. Once that's done, we trigger a different event. And at that point, we might completely change the world. We might erase some of the near-field elements and put some new stuff in. And at that point, those detach from the environmental timeline and just draw in as their own.
[00:35:18.607] Kent Bye: And this is all in Unity?
[00:35:20.027] Elie Zananiri: It's all in Unity. It's mostly shader-based, so it's all happening on the GPU. OK. OK, cool.
[00:35:25.917] Matthew Niederhauser: Yeah, and I would say along with these local and global timelines, it also means that everything sort of has to like be spun up and wound down together, especially for the major transitions between acts and music. And so how those land isn't like a linear editor, you know, where it's like, oh, at 26 seconds and four frames, we're going to automatically go into this. We sort of have to think about it as coming to like the coda and like it comes together and then we have the music kick in for a new act. And so it's pretty precise now. It's not like it's imprecise, but you sort of have to like spin it up and then wind it down and then spin it up and wind it down. But I'd also just say in terms of like the back end and working with Wesley's art is that Wesley does have a very intuitive sense for interactivity and what happens when you put somebody in this world, which also made her art as it came in was like pretty ready to go, quite frankly. All the things we're spouting out about the fake avatars and here comes this third person and I think she understood what we were trying to do even if she thought we were a little dotty doing it.
[00:36:38.823] Wesley Allsbrook: I don't think you were dotty, I just think you wanted me to draw the grass a certain way and it offended me.
[00:36:45.065] Matthew Niederhauser: All I'm trying to say is like Wesley would send us the grass, the grass is really almost undid this entire project multiple times obviously. It was a lie. Wesley would send us these like whole coherent worlds. And like, those were some of the best times in our development process when we would talk about all these different things. And then Leslie would be like, okay, I'll talk to you guys in four days. And then she would go off and then send us this new package and be like, boom, like a whole entire world pretty manifest. And she did have to come out to New York like the second week before just to like nail it down. You know,
[00:37:22.682] Wesley Allsbrook: You picked up all the assets and you rescaled them and you repositioned them in a completely new context that was not integrated. This is the offense to which he is referring. These guys are like, oh, it looks fine, it looks good. And I'm like, no, it looks horrible. But I think I would be interested to know actually whether other people would have been fine with the way that they were treating my assets, the things which we call props as modular. And I don't know whether other people would be caring about what I was caring about, because the way that these worlds in particular are painted is so impressionistic that it may not, in fact, matter. It definitely mattered to me.
[00:38:01.408] Matthew Niederhauser: It definitely mattered. And for greater context, there's like a far field, a midfield, a near field, which we didn't mess with. And then there are these props or the things that are actually in your play space. And we were still trying to figure out how to do it together. And we thought, we were like, okay, this looks okay. And after we reordered them a little bit, and Wesley needed to be flown into New York to handle these transgressions. And quite frankly, she came in and this was like where we just had to get everybody in the same room right before Sundance. And she came in, solidly done. And then she took those things and then actually made them look a lot better. We fumbled a few times here and there, but when we got everybody in the same room together, including Tim was there. Yeah, we miraculously had everybody there two weeks before Sundance and yeah, just brought it in for a landing. Yeah.
[00:38:53.240] Kent Bye: Well, it reminds me of a similar dynamic for writers or film creators who are used to working in a medium that's very fixed, but then you start to add these video game components where you have to surrender certain aspects of your agency and authorial control to be able to have these more dynamic interactions that give the player more agency. But you can see how there's a certain part of that iterative process that all the iterative aspects of game design where you do want to fail fast and to iterate and to see how it looks and like having the opportunity to have that regret and to update it and to change it and to create a dynamic system that allows you to do that, but then also to give up some control, but not all control. And so balancing that in terms of like you wanting to maintain the integrity of your artistic vision and that it doesn't necessarily misrepresent what you were intending with what you handed over, but yet allowing some surrendering of that control over that art to be able to create this larger context that allows the interactions to happen. So I don't know if that's accurate summary of that, but it seems like it's something that is happening in other mediums as well from like filmmakers and artists of like part of the aspects of the medium of VR itself is that there's certain surrendering of the ways in which your existing pipeline is happening in order to adjust what has normally been a mostly waterfall process where you have things as fixed objects, but creating something as to this more dynamic flow of this process that's unfolding that you have these different workflows and pipelines.
[00:40:25.498] Wesley Allsbrook: Everybody has personal preconceptions about process, but I don't know. Our main issues were that I live in California and that these guys are always together in New York, and we want to, in the future, find a way to bring other artists into this because, let's face it, we're not the most diverse group, and we can't really make a piece that's about identity if we decide to expand it without bringing more people in and allowing stranger worlds to mix. But it's good I think that if we figure out pipeline wise how to deal with like people in all kinds of locations in the now times. But yeah, I don't know. I agree with what you say about like these ideas of sort of like fixed process versus the way that things need to change with immersive media.
[00:41:09.114] Matthew Niederhauser: Yeah, I mean, this thing would have definitely fallen apart a long time ago if we hadn't had a fun, slightly antagonistic, but creatively respectful dialogue. And I think the best part of working with everybody on this project has been A lot of us have known when we've gone off course, you know what I mean? And people weighing in with really good criticism and being able to take it, and it's really been through our joint efforts, I feel like it's been a bit of a success here. But we contemplated many times even over this past year about, as Wesley said, like how we could bring in other artists, new looks, and we were dealing with some constraints in terms of bringing it here, as we all know, the mad dash to Sundance, especially with a nice holiday right beforehand. and financial constraints and I will say that this was largely just built on our own efforts and without monetary support except for a little stipend from our dev lab with Kaleidoscope and Riot and Oculus. but we want to find a way of getting other artists worlds in there and in the same way that this one is set up so that Wesley ended up creating the two different worlds just because of the nature of the beast we were trying to bring here but there's definitely a future for this where it's not only more people maybe incorporating performing artists and dance or movement, but just the art itself being radically different and the ability for different envisionments of bodies being brought into the system and you know lots to explore.
[00:42:52.638] John Fitzgerald: We saw a wild turkey running down outside the other day and we were like oh my god this would be the most amazing thing if like the third person that entered was actually like a creature and just we put trackers on it.
[00:43:12.460] Kent Bye: Well, it sounds like that you've created a whole platform for other artists, that you've got the infrastructure there now to be able to have this pipeline to be more inclusive of people around the world, to bring more interaction or dynamic experiences that may not be possible just from these artist tools alone.
[00:43:30.043] Matthew Niederhauser: Yeah, I mean, non-humanoid figures. We wanted to have different beasts in there, perhaps. And yeah, Ellie's going to do that next week. He's going to create an entire network platform that you can submit assets from all over the world. They'll automatically rig themselves.
[00:43:45.745] John Fitzgerald: I think by the end of the month, actually.
[00:43:47.086] Matthew Niederhauser: Yeah. But no, it's a great milestone, I think, for the project. And we are excited about doing longer term installations of it. There will be a lot of cool R&D involved and thought around how people interact in virtual spaces together, and also hopefully find a platform where We were able to bring in Wesley's art. We could almost in some ways create, if we were trying to have these like matching worlds but different aesthetics, like create templates or like markers and share them. And then people could submit them along certain lines, could be integrated. Maybe you could have 15 people in there who are seeing radically different worlds and hinting at their worlds to other people. And that's also a fun takeaway when you talk about it afterwards. What did you actually see? What did you actually feel? And who were you while you were in the experience?
[00:44:44.527] Kent Bye: Well, one of the challenging things about VR in this early days is the throughput and getting people to actually see the experiences and each festival has their own approach to how they handle that throughput challenge and Venice they have places where they're scheduled for that specific experience and then you over the course of a week you get to check out different things and so if you're there for a week you get to kind of pop in and out or There's Tribeca where they take a little bit more of all the experiences or you go in and you have the waitlist But there's a mad rush at the very beginning to try to get in there And so then it's good to you know not have that waitlist and then to do more of a ticketed schedule here at Sundance But then you have that still mad rush feeling where you have to get in there And then you end up waiting in line without the waitlist and so it feels like there's this like either you're queuing Electronically or you're waiting but yet people are sitting there waiting in their mind calculating like I spent a this amount of money and went through this trouble to have this 90-minute slot and I get to see of the number of experiences here only a handful of them and you know with your throughput just rough calculations it's an 8-minute experience but then there's the onboarding offboarding if it's like 15 minutes for each person then that's like an average 10 to 20 people for a 90-minute session that could see it so it's sort of like I'm just curious, it feels like there needs to be some larger throughput calculations of like this many people, this long, on-board, off-board, and this is how many people could see it in this slot so that as people go in they could at least know what the throughput estimates are and then maybe schedule and plan a little bit better or have other ways of having cinematic VR experiences while they're waiting or Just figuring out this throughput issue, it feels like it's a high friction, painful, frustrating, and great for people who get to see stuff, but extremely frustrating for pretty much everybody else.
[00:46:30.333] Matthew Niederhauser: I would say, first of all, the Sundance crew has been doing, I feel like, a better job this year than before, and it's always, like, amazing to come here and present, and that's, like, always an ongoing dialogue with them, and, you know, it's not, at the end of the day, the best context for showing these experiences, but in terms of understanding how many people we're getting to right now, like, we want to explode the paradigm that we had to use to show it here at Sundance. That would include Ellie's continued work of potentially tracking bodies without the markers, which would significantly reduce the onboarding process. But even the experience itself, if we're thinking about a 15- to 20-person installation of something that is actually just a continual timeline. It would be something you could enter or exit at any point in order to still be able to experience the world, potentially have those same feelings of discovery. So it could be more open-ended in terms of what you could experience and how you could interact. So you could have 15 to 20 people going in and out as they please over a four-hour period. Would be a more interesting paradigm for us in terms of an installation. And that's why this is a milestone. It's like an experiment. What are people capable of handling in terms of being put into this world? And do they like this mixture of the fake and the real? What happens if there are planned actors within the experience? But to make this viable moving forward, and this is always here the funniest part of these festivals, is it turns into an oral tradition because everyone talks about it because most people can't see it. And I mean, for us as creators, like it's absolutely necessary to think about how these experiences can be brought to larger audiences and hopefully to more institutions that can sort of handle them as well. Because every single festival has a handful of experiences that should be seen by a mass audience. And that still hasn't been figured out for cultural institutions and other places to view art or other creative paradigm spots, things.
[00:48:36.587] Elie Zananiri: My partner had a good idea this morning, actually, which was that, like, what they should do is just have one or two of these at every cinema during Sundance. And then while you wait in line, instead of actually having to wait in line, you can just, like, jump out, see an experience, and then come back in. Instead of having them all in one place and having an hour and a half to go through everything, it would take the stress out of that. And it's like, oh, if I want to go see Metamorphic, I'll see it before my movie at the Egyptian. Or if I want to go see this other piece, I'll see it before my movie at the Park Avenue Hotel. And it just kind of, like, I don't know, relieves the stress of it.
[00:49:05.673] Matthew Niederhauser: I'm not doing metamorphic on the street in Egyptian. I don't know. We obviously haven't talked about this as a group yet.
[00:49:14.236] Wesley Allsbrook: The business of VR seems to be like there's a large section of it which is currently organized anticipating the spaces in which people will be waiting for long periods of time. There are so many people who are like, what about VR in hotels? What about VR in airports? What about VR in malls? And I don't know, like, to Matt's point about, like, the sort of oral tradition of these experiences, like, that's a huge part of the business of VR, too. I mean, you didn't see the VR at Sundance? Great, you know that it has a Sundance thing on it and that people are talking about it. And that's another sort of, like, star on the lapel of, you know, every person who's kind of, like, done the thing. Like, whether you've seen it or not, is it well-reputed? Well, then we can all continue to work. You know what I mean? Seriously, like a lot of these things gain so much more power through reputation than they do through the actual event. Of course I would love for everyone to have seen Metamorphic, but sometimes it's better for them to simply know that Metamorphic exists because then they don't have to, like, they've never been in a Vive Wireless. Think about the way that the festivals work. Not to be too cynical about it, but people should be able to see this stuff. And this stuff is good, and it is fun, it is potentially pushing the bounds of storytelling, and it is doing valuable things, but this is also how we make money.
[00:50:35.252] Matthew Niederhauser: Yeah, and I would say to that, like, we're always hoping year after year that there are more people coming to these festivals with potential opportunities for installation. And, you know, it's having interesting conversations with places like the Phi Center. had a great conversation with the Walker Center, and it is like a hardship to put a lot of work into something, and you know, you did the calculations, and you're pretty close in terms of thinking about, oh, like, at the end of the day, 50, 60 people a day over 10 days, like, you know, we're lucky to maybe hit 500. And where does it go? Does it disappear after that?
[00:51:17.374] Wesley Allsbrook: We need the festival circuit for that.
[00:51:19.830] Matthew Niederhauser: Yeah. And, and the festival circuit has been a great platform for just all of us, you know, weirdos been brought into this, like the limelight of these legacy festivals. But unfortunately it's not like a cinematic paradigm where there's automatic regions and markets and ideas of how it could be played out. Immediately and this is something that we think about a lot individually as artists and also for instance orium our studio And we're hoping that you could come to this and be like okay We're gonna take this to New York Montreal London Shanghai LA and that there's gonna be a burgeoning network of of places where you could find a home for two or three months because people want to see these things. They're not always being run well, but people want to see these things, but how do we bring it to scale? How do we make it be something completely different and at the same time not being like a selfie factory, you know? But there's something in the zeitgeist that is occurring and I hope is going to make it possible because we had a big problem with this with Zykr, which we did two years ago here and always amazing to be here, of course. But we built a super complicated system, four-person networked experience. And, you know, we've taken it around the world, but I don't know, maybe a thousand, maybe 1,500 people have seen it. So we have been thinking about Metamorphic a lot, like Sundance is a milestone, but to really move forward with this project, it's like, yes, how can we get 200, 300 people through in half a day, and for it to also be awesome? And there's a lot to be done on the hardware side, the software side, but I think we're starting to scratch into that realm. But it's been difficult, and it will probably remain difficult for maybe this decade.
[00:53:16.591] Kent Bye: Yeah, the staggered release, like persuasion machines, every five minutes people go in. So yeah, something that has the constant flow of people that has that, so. Yeah, and I think part of the dynamic, I'll just say, as a journalist and covering Sundance since 2016, this is my fifth year, And last year I went to Sundance, South by Southwest, Tribeca, Venice, and DocLab and saw all the experiences at all those festivals, 150 experiences, did 100 interviews, but then only reached about 20% of it because it's like, who's the audience for that? And so then I've had to figure out, what's my philosophy of covering this stuff? like, what's the news hook? If that isn't getting out there in distribution, then it becomes a challenge for other journalists to be able to know how to even cover it. So I feel like, like this year, I want to try to get out all these as fast as I can, and then just sort of like go into other things. But to mark the cultural moment to this, I still have an archive of hundreds of interviews that I've recorded that haven't come out. But to say, okay, this is a model where the cultural influencers at these different museums need to hear about these projects and know about them and be able to distribute them around the world and then be able to have this larger distribution network. So it starts with maybe the museums, but then, you know, eventually percolates out into the location-based entertainment or people that are getting away from the zombie wave shooters and wanting to see a little bit more of the artistic story, immersive stories, experiences. There's a bit of a chicken and egg problem that I've felt that the immersive industry has had, that without the critical discourse and the larger analysis and discussion of them, then it does create this vacuum. So, I'm trying to do what I can, but it's a larger movement that I think has to happen across all the industry.
[00:54:59.050] Matthew Niederhauser: I have one shameless plug. It's not a shameless plug, but we're actually going to be working with the new museum and the Onassis Foundation this coming spring for a new cool studios place called Onyx, which is going to explore how to exhibit these things in the context of especially museums and cultural institutions. because I do feel like this is a proper space for that and there's space of contemplation as opposed to like a IMAX theater lobby and also like people go to museums and cultural institutions for open-ended four to five hour time periods to wander see things and it's a good upsell so You know, I think the demand is there. Hopefully we're gonna find something with Metamorphic where we can take it around the world where there isn't some like golden brick road yet, but we're excited about the possibilities experience, the fact that you doesn't need, it works across all languages, potentially across many cultures, and it's gonna be interesting to see how it develops moving forward.
[00:56:04.910] Kent Bye: Great. And finally, what do you each think is the ultimate potential of immersive technologies and immersive storytelling and immersive art, and what it might be able to enable?
[00:56:18.019] John Fitzgerald: Sure. I'll jump in. So I am a new parent, and so I have all these cliched, funny things happening about the way I'm thinking about learning and discovery. And watching my 3 and 1⁄2-month-old baby discover phrasing has been something that I don't know I'd say this metamorphic project we've been building in the darkness or in the shadows of like we're not like VC backed or it's not a finance project so it's crazy to be out here presenting it and see the spectrum of reactions and emotions and I'm I'm kind of like, I don't know, my brain is watching people in the space interact for the first time and have these moments of discovery and touch. And I don't know, my mind is thinking about my small child looking at me and having these small epiphanies. And I don't know if that's what's happening in the future, but that's something I'm thinking about right now.
[00:57:13.290] Matthew Niederhauser: Yeah, I think what we're most excited about is how to expand agency within these experiences that create better stories and more possibilities for dynamic social interactions. And we like to build this stuff. because we feel that the future of this is people interacting with each other in these virtual spaces and not sort of being, you know, in their own worlds. Although that is still definitely going to be pushing the industry, but this is just a culmination of a lot of thought in terms of how do you drive somebody through an experience. Just constant rethinking themselves and opening up new possibilities for engagement and, you know, seeing the world.
[00:58:03.302] Wesley Allsbrook: Well, I mean, I feel real ambivalence about virtual reality in general. Like, at this point, VR is still, and all of this XR stuff, to deny the fact that these are all products is kind of impossible. And so, you know, as this medium continues, it would be great if it could become space where everybody had access to it and where everybody felt like, I don't know, like they could participate in it in a way that didn't cost them anything. Like every time I use Photoshop it costs me something. Like it's a subscription service and all my friends are, you know, illustrators and They either have to pirate it or they have to pay at least $500 a year for it. So whenever I use this stuff, I'm really kind of conscious of that. And so I can only really kind of hope for that. Like being here is really weird. Like I feel like I'm in some big virtual reality dick measuring thing and everybody's got their dicks in their hands and I'm looking down at my dick and I just don't want it. But, so, you know, all that I can hope is that this community grows in a way that's healthy for everyone and that everybody can participate in in a way that will make them happy. Like, I moved to the desert. I'm buying a gun. Like, goodbye.
[00:59:20.177] Elie Zananiri: I guess what I'll add to that is like I think it's already finally starting to happen but that people see virtual reality or this has a completely different medium and not as an extension of film not as an extension of video games like as you like don't come in with these conceptions that oh I'm gonna watch like a you know an informative documentary I'm gonna come in and watch like a I'm gonna have some agency. I absolutely have to have agency in this piece that you just come in and it's like its own New form of either art or entertainment or whatever you want it to be, but it's not like the weird bastard child of another thing that exists already What's that what am I excited about oh well
[01:00:00.844] Kent Bye: It's funny, the last interview just asked me that and I sort of gave an answer, but what I'd say is that VR has the opportunity to create these deeper philosophical shifts in our culture that I think that we need right now with the consolidation of wealth and power. We want to have a decentralization and decolonization and putting the power back in the hands of the people and I think that VR as a medium can help to shift the market dynamics and the culture and the political aspects but you know we're kind of in a crisis right now in our world and we have the fragmentation of social media addictions and the polarization of society and can VR be a way to allow us to get away from these ways that we categorize people with labels and to see the the common things that unite us all in terms of what it means to be human. I feel like that's the essence of what VR can do is sort of boil things down to this experience that we all share. That's like our sense of our bodies, the way that we interact into the world, the ways that we communicate and we connect and the ways that we have emotions. It's like these are these primary elements of the qualities of experience and all the different contexts of our homes and our families and our identities and our relationships and our careers and all the different aspects of all the multitude of contexts that we have in our world and our own characters and how we develop and change over time and how we can explore the expression of character through stories and our own development of our own character to improve ourselves. And so I feel like in the context of stories, though, it's like with VR, you're able to see the affordances of each of the other mediums in a new way of theater and video games and the worldwide web and the user interaction interfaces and the affordances that the web has had with transmitting information and knowledge. and literature and books, architecture and industrial design, on top of all of the radio and theater and movies and film and cinema. All of this is like being mashed together, but as we look at VR, it's like taking stuff from all these other aspects and trying to fuse it together into a new meta-framework that allows you to have a conceptual understanding for the human experience and story. And that as you do that, it gives you better lenses into these other mediums and their affordances. And everything's not going to be a great VR experience. What is the story you're trying to tell? And what's the best medium to do that? For me, I love to tell the story of the evolution of VR through a podcast. So I feel like it gives me the most latitude to be able to have an experience and to talk about it and to have lots of discussions. And I don't have to have it through my lens of my perspective. I can get the perspectives of everybody that I'm talking to. And I can just ask the questions and let the authority be what other people say. The creators are the most authoritative people in the industry. You're on the front lines. You're in the trenches. And so the more that I can step back and focus on that oral history and decolonizing what people see as authority in terms of the New York Times or whatever, what gets published by the newspapers of record, you know, that's the existing hierarchical way of thinking about authority. But I feel like with technology, it's like you have to actually get to the people who are the artists and the creators and the makers to get to what's actually happening. So I feel like You know, I want to see a memory palace of all space and time, and I want to see a virtual reality experience of the oral history of VR with all these podcasts, but make it an interactive dynamic experience with conversational interfaces. You know, all the dialed in information so that people could walk in and have an education. and how do you do the intellectual property and how do you get the consent and you know it's like there's a new thing that hasn't existed before and so how do we make it in a way that it's not just one person who gets all the spoils but is able to create like a self-generating economic system that is trying to find new models. So the cryptocurrencies, the decentralization, all that stuff that I think is going to come together with VR, it's going to be a part of it, but it's like a larger shift that for me, it's like the most highly leveraged way to bring shift in culture is to work in immersive technologies because it has the capability to change the architecture and the market dynamics and the culture and the public policy. So for me, it's what I've chosen. It's not for everybody, but that's what I chose to do. And I just feel like it's very exciting to see what can change. So.
[01:04:03.367] John Fitzgerald: Yeah.
[01:04:05.869] Matthew Niederhauser: Brave new worlds, greater potentials.
[01:04:07.631] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community? Try harder.
[01:04:16.697] Matthew Niederhauser: Yeah, it's been fun. And bravo to everybody else here at Sundance, and not everybody is here. I'm with Blinders, and we're all facing this together. But there's a reason why we keep coming back, because I don't think it's just because we're sadomasochistic, maybe a little bit, but we are excited about these new potentials. And some of these convergences that you're talking about, not just in terms of head-mounted displays or new mobile devices, but as cryptocurrency AI, new platforms for interaction that are not going to have the same mediated hierarchies from other large conglomerates and companies. We're definitely not in the clear in any way, but what keeps us coming back. It's like why I don't take photographs anymore. I worked as a photojournalist for a long time. It's like the hardware shifting, the software shifting, and there's going to be a lot of possibilities to interject and get in on the ground floor and try to help shape a more fair, equitable future for what is going to be a game-changing technology.
[01:05:22.972] Wesley Allsbrook: Yeah, I mean, congratulations to everybody here. I mean, it's a lot of people here have donated their time and efforts to making something that they believed in because they believed in the project or because they believed in the talents of their teammates. So, you know, good job.
[01:05:37.063] Kent Bye: So yeah, I just wanted to thank each of you for joining me on the podcast. And that, you know, I really love the embodied experience. And you know, the platform, I think I'm really excited about the most in terms of what this can do to bring other art, but also this new model of throughput, because I think that's a huge issue. And I think that this is a milestone mile marker that you've reached. And I'm excited to see where that goes. And I hope that you get the resources and funding to help make that happen. So thank you for joining me on the podcast. So thank you.
[01:06:04.310] Matthew Niederhauser: Thanks a lot. Thanks. Thanks again. Thank you.
[01:06:08.195] Kent Bye: So that was John Fitzgerald and Matthew Niederhauser. They're the co-founders of Sensorium. Wesley Awbrook, the artist on Metamorphic, as well as Elie Zanonieri. He writes code and one of the co-creators of Metamorphic. So I have a number of different takeaways about this interview is that first of all, Well, the most striking thing about this experience is that you have a dynamic body and it's changing and it changes based upon how you interact with other people in the experience, which is something that I've never actually had a chance to experience before. So I was really happy to be able to have that experience of being able to be connected to another person and feel like there was a part of their essence that was coming onto me. Now in talking to Ellie, they actually had to do this very interesting thing of having multiple timelines. And so most of the different types of experiences that you see have like a singular timeline, but this is a type of dynamic and interactive experience where they actually had to have these actions that were kicked off and have their own timeline. And sometimes they merge with the master timeline and sometimes they go off on their own. And so just to have this concept of a more relativistic aspect of having multiple timelines unfold at their own pace, It's kind of like a general relativity of timelines and time and reminds me of Carlo Rovelli's order of time where every point in space and time has its own time that's unfolding relative to the gravity. And so you have different experiences of time. So the fact that they actually integrated different timelines in this experience, I think is pretty mind blowing. a real technical feed, but the experience of it is that it allows you to have these dynamic interactions where you really feel like you're engaging with other people. And that the way that you engage with them is able to kick off these different experiences that change your own sense of your embodiment. So they are playing with embodiment quite a bit and give you lots of opportunity to have mirrors, to look at yourself, to see how you're changing, to be able to look down at your body and to eventually even have a serendipitous collision with other people in the room where you feel like you're separated and then you are now engaging. And, you know, this was a topic that was a bit of a discussion where Wesley was skeptical of this whole idea of trying to encourage people to interact and touch each other when they're occluded within virtual reality, just with all the different aspects of consent and being trauma aware it's not necessarily safe for everybody to do this type of experience but at the same time to preserve some sort of surprise around that but also to maintain different levels of consent and best practices around okay in this experience you may be interacting with somebody and so but just the mere fact of saying that may actually create a little bit of a spoiler that there is going to be interaction if they want to create that as a part of a surprise and then to have somebody who's a bit of the docent and trying to, you know, people aren't inherently already connecting to each other to be able to come in and try to make a connection. When I did the experience, the other person seemed to be fairly new to VR and so wasn't necessarily aware of the different types of social dynamics that can happen. Maybe they thought I was just another recording because there is this combination of having recorded people that are walking around, but there's also opportunities to have like these real time interactions with other people as well. A big part of this experience is to see how your embodiment changes when you actually do have some of those physical interactions. So another big thing that I take away from this conversation was just the creative process and just to see how as an artist and a creator, Wesley really likes the fact that she can experience regret, meaning that she can see her art and to make updates and to, you know, usually there would be one opportunity to be able to deliver on your art. and did not do any type of iteration. And so as this starts to unfold, then to be able to have a pipeline that would be able to ingest the different things in this case, it's from quill to be able to paint everything out in the order that it was painted, and then to like recreate that. And that Wesley likes to do all sorts of things like drawing grass from the top down and bottom up. And then from Ellie's perspective, he didn't want to have to write a lot of logic and code to be able to like reverse things. And so He had asked artists to change the way that she created art, just so that it was able to come out in a very specific way. So that was one point of tension of the creative process for one person, but the needs of not having to do too much engineering to be able to create the effect of grass growing from the ground up rather than from the top down, but also this dynamic of having the near field, the midfield and the things that are far away. And that as they are doing this interactive experience, then the creators have to sometimes shift things around and they had to. bring Wesley in to be able to do some proofing to make sure that it wasn't disrupting the overall gestalt of her artistic expression, the fact that they were kind of moving different aspects around. And so that's just, I think another dynamic of the nature of interactive media and trying to negotiate all those different things and to make sure that maybe there's a, an opportunity for them to come in and be able to make changes, but to have this back and forth dialectic, to be able to have the freedom to move things around, but also to make sure that it has the past from the artists to make sure that still. Maintaining certain levels of integrity of her vision that she's trying to create here. So the other big thing is the distribution and the throughput issues. You know, this is an issue that I talked to a little bit with some of the Sundance creators, just kind of unpacking some of the dynamics here. But the issue is that the work that's there often is limited enough that there's more people that want to see it than have the capacity to be able to show it. And so one of the things that they're trying to create as a future is a vision to be able to have like 15 different people in an experience like this. And maybe they're experiencing different aspects and they're able to stagger them in at different times, but with lots of people who are able to have a very rich experience, but to be able to have a big enough space and to get enough people rolling in and staggering. So that's not just like two people going through for 10 to 15, 20 minutes, which ends up only being on the order of hundreds of people over the course of 10 days can you start to reach that same amount of people over the course of one day if you expand out the Capacity to be able to onboard people quickly and to get them into an experience like this and so there's other technological changes that they'll have to go for but that's kind of the vision that they're shooting for and Matthew mentioned that there is this new Onassis USA and the new museums new Inc announced a new partnership of the onyx studio That's o nx capital o nx studio onyx studio and So sounds like they're going to potentially have an opportunity to be able to show it there and then maybe continue to develop and work on it. And you know, this is super impressive for a team that was essentially bootstrapped. I mean, I think they were part of dev lab put on by Kaleidoscope VR, and I think they were able to get some money out of that. Uh, but other than that, this is essentially just a lot of really technically capable people, super passionate about creating this different type of work and on their own time, being able to actually produce a pretty impressive experience considering, you know, how bootstrapped this was. to really show their technical chops of design. I know that Sensorium has had a number of different experiences that they've worked on, including Zicker and other experiences. There's the Freud interpretation of dreams where they did a lot of work on that as well. Amazing experience, a lot of great production design but a lot of the 360 video world so they're coming out of this like 360 video world and starting to getting into the more interactive they worked on Zicker and so to see their evolution and types of technical competencies that they have I know that when I was in New York City the last time I was able to drop by the our lab and they just had some offices that were there in our lab our lab being this really great incubator site where a lot of the immersive technology Scene in New York City is really happening a lot of the programs and schools and education and like an incubator space. It's a really big space, exciting to see that the government has put forth a certain amount of money as well as different institutions and universities there in town to be able to start to run a number of different things out of our lab. And so I know Matthew had mentioned them as well as being a part of helping to support them in different ways, especially as their offices have been out of there. So I'm excited to see where this goes, especially around embodiment and having dynamic aspects. I mean, I think the underlying technology to have multiple timelines unfolding, that's something that I haven't seen a lot of the other experiences. Cause that's technology that they built themselves, but I'm looking forward to see ways to have more of those relativistic timelines relative to the objects rather than a universal time to see how these objects, as they interact with each other, you're able to have these more dynamic. experiences, but also just the experience of experiencing myself from the art of Wesley Albrook, who she's got a very distinct style. It's the color palettes and the animations that they were able to do within this experience to be able to bring to life the cool illustrations. I mean, Quill has within itself their own animation that's built in, but. The way that they were able to integrate it within this experience felt like a lot more of like taking it to the next level of like almost motion graphic dynamics of really giving this sense of like a living, breathing experience rather than the static objects, but really to give this dynamic experience and really enjoyed what they were able to do with these artists. And I think the good thing is, is that they're able to potentially collaborate with a number of different artists as well, and to bring in different styles and different ideas and concepts and. just kind of exploring what's it mean to explore different aspects of your embodiment and your identity expression and deconstructing it and reconstructing it in different ways and again a theme of trying to see how people are connected to each other through their interactions. And just And just a final note, just because both John in this interview and Felix in the previous interview, they turned the tables on me at the very end to ask what I think. And whenever people ask me that, I like to try to like see what's arising in that moment. And so this was like literally within a half hour of each other where they both asked me at the same time. The thing that came up on this conversation was a little bit more of the philosophical shift that's going to happen potentially. I mean, I think that VR represents this paradigm shift. And so to try to like break down, what is it about the existing paradigm that we're in and how does the virtual reality help shift and change our paradigm? And so that's something that I thought quite a bit about coming home from Sundance, because it was a bit of like. really trying to investigate what does that actually look like. I mean, Thomas Kuhn has the structure of scientific revolutions where he talks about the social dynamics of a paradigm where one existing paradigm will have an anomaly and then a new paradigm will come along and solve a lot of the issues that the previous paradigm is having. And I feel like there's a lot of just issues that you could call it like 2D flat paradigm, which is really kind of reducing down the complexity of things down into a flat plane. That's about the complexity that we can start to think about things. And now as we add another spatial dimension that actually allows us to have these completely new immersive interactive metaphors that allows us to add a whole nother layer of complexity, to think about the fractal dimensions of scale, to see how the whole impacts individual, The qualitative aspects of time just have these immersive experiences where you're able to experience different nuances of the qualities of an experience. And so I can say, okay, it's like job simulator, or it's like Beat Saber. And you, you know what it's like based upon having those experiences, but those experiences are the building blocks of our qualitative experience. And so what's it mean to have. a more robust, unified way to be able to have discussions about our qualitative experiences and having these discontinuities of paradigm shifts. And so there's a book called Dynamics of Transformation by Grant Maxwell that I think is a good follow-on to a book like Thomas Kuhn's Structure of Scientific Revolutions that talks about the paradigm shifts. And Grant's coming more from the philosophical perspective, trying to break down some of the deeper aspects of a next generation paradigm. Like I said in the previous interview, I think there's more aspects of artificial intelligence and the blockchain and decentralized web and category theory and more of a relational metaphysics, moving away from a substance metaphysics and seeing things less as concrete objects and more as dynamic processes that are unfolding. So getting away from more waterfall thinking and more of like a dynamic iterative agile. All of that I think is into more of a relational metaphysics and what I've been inspired by Alfred North Whitehead and his process philosophy seems to be a pretty robust metaphysical framework that I was also really focused on thinking about things in terms of these relationships and these processes that are unfolding. And I think that to me feels like a natural fit for what I see is the inherent affordances of virtual reality is a lot of these experiences that are unfolding and recommend dynamics of transformation if you want to read more about some of the philosophical foundations of this next paradigm shift. And I talk a little bit about very briefly throughout my talk that I did in all space VR, which talk about the future XR trends and trying to pick apart the fundamentals of experiential design. But this is a, I think a larger talk to be able to really unpack this paradigm shift in what it represents and also trying to break out these concepts as they diffuse out. Paradigm shifts usually or trying to address all these different aspects that are the downfall of existing paradigms And I think all those puzzle pieces haven't been fully fleshed out yet So it's not like we can just upgrade to it a new paradigm oftentimes new paradigms and scientific revolutions happen when there's a concrescence of lots of different things all coming together and fitting together at the same time and then really solving a lot of the biggest pain points of the existing paradigm and so I I still think we're still a bit of ways away from that and trying to really connect the dots between a lot of these different aspects of the relational metaphysics and see how it actually gets applied. I think in virtual reality, though, we're going to start to see a lot of the new paradigms that people have and then applied in specific use cases and people having embodied experiences that just give them a direct experience of this new capacity that this medium of virtual reality is going to be able to enable. So I just wanted to unpack that a little bit more just because I kind of went down a rabbit hole of some of the deeper philosophical foundations of what that means. So anyway, for anybody who's really interested in that's some, some pointers and recommendations that I would give you. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast and please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a list of supported podcasts. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.