Forager is a short immersive story that shows the four stages of development of a mushroom through a groundbreaking volumetric timelapse that takes place within a game engine. They had to innovate on automating their photogrammetry timelapse pipeline both in the capturing and rendering side, and they ultimately decided to go with a voxel-based approach for representing and optimizing these volumetric timelapse processes.
I had a chance to chat with Winslow-Porter, co-creator, producer, and creative director of Forager as well as Elie Zananiri, co-creator and technical director on the project about their creative process, the influence of creative coder communities, how mushroom inspire them, their epic mushroom-based installation, where to use audience interaction and where to dial it back, as well as their multi-sensory integrations with smells, wind haptics, and SubPac haptics.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing and the structures and forms of immersive storytelling. You can support the podcast at patreon.com slash voicesofvr. So, continuing on my series of 24 episodes from South by Southwest, today's episode is on Forger, which is a piece by Winslow Porter and Elie Zanoniri. It's a really amazing piece where you get to see the life cycle of a mushroom, and you go through these four different phases. The one that's really highlighted in this piece is the third phase of the fruiting body, where they actually do time-lapse photogrammetry of these mushrooms growing and then translate that into these voxels and volumetrically so they can actually put into these immersive experiences. And so you have this whole sensory haptic experience with both wind and smell and this sub pack as you're laying down and watching this short seven minute virtual reality piece, but really deeply effective of telling the story of mushrooms. And so, yeah, I had a chance to talk to both Wenzel and Ellie about this piece where I first saw them pitch it back at Tribeca in 2023 and had a chance to go to their offices and see some of the different rigs and to see the different mushrooms. It was actually one of the more popular pieces and really difficult to get into. I think there was a nice buzz and word of mouth to have this haptic sensory immersive experience around the life cycle of a mushroom. So that's what we're covering on today's episode of the Wasteless VR Podcast. So this interview with Winslow and Eli happened on Monday, March 13th, 2023. So with that, let's go ahead and dive right in.
[00:01:45.159] Winslow Porter: Hi, my name is Winslow Porter and I'm one of the co-creators and producer and creative director of Forager.
[00:01:53.221] Elie Zananiri: And I'm Eli Zanoniri. I'm the other co-creator of Forager and technical director on the project.
[00:01:58.450] Kent Bye: Awesome. Maybe you could each give a bit more context as to your background and your journey into VR.
[00:02:02.854] Winslow Porter: Sure. Actually, the first VR project I worked on was with this fine gentleman to my right here, Eli. And we both worked on a project called Clouds, which was an early, immersive, virtual reality documentary about creative code, where we could actually step into the artist's work. So it was almost like liberating the stuff that was normally inside of a 2D screen allowed us to see things It was almost like a gift to everyone working on the project because they actually got to see their own projects with an extra amount of depth that it felt like was an inevitability with a lot of it. And it was right when the first DK-1 came out. So it was just an amazing, like the proof of concept was really powerful. So I think we all got bit by the bug then. And we've been collaborating on all sorts of interesting projects since. And it was really just a nice amount of time. A COVID project with photogrammetry, time-lapse mushrooms came together and now here we are today with our premiere. And it's really exciting to be able to share it with the crowd. It's a really good energy to it.
[00:03:01.320] Kent Bye: Yeah, I remember I got my DK1 on January 1, 2014, and the Sundance of 2014 had happened. And I remember seeing Clouds, and I ended up doing an interview with James George later about that piece and had a chance to see it. But yeah, I remember getting really excited about this interactive documentary spatialized in VR. And it was really kind of ahead of its time. But yeah, I'd love to hear a little bit more about your background context and journey into VR as well.
[00:03:24.726] Elie Zananiri: Sure, so like Wenzel said, Clouds was the first one, and then after that I worked again with James George and the Scatter crew to do Zero Days, I was the technical director on that. That was in 2016, premiered in 2017, and that was like an interesting ride, because that was the first time we were using an actual game engine to do VR, and there was a lot of challenges there, but we definitely saw the potential. After that, a few years ago in the year 2000, I worked on Metamorphic with Wesley Alsbrook and the Sensorium guys, John and Matthew. And that was like a really fun project that was really about embodiment and how like you can just, the whole thing was hand-painted in Quill and it was about just how you change your perception of the world through experience. And funny enough, that was the last Sundance right before the pandemic. So after that happened, we all kind of took a break and this is the first time I'm back in VR since, so it's been three years now.
[00:04:16.923] Kent Bye: Awesome. Well, and I know that I spoke with you with Tree, and so I guess there's kind of a theme here of some of the different projects, at least from the last one to this one, in terms of looking at the environment and also this time-lapse element. But maybe you could just give a bit more context for how this project of Forger came about.
[00:04:32.824] Winslow Porter: Sure, I wish you could say it was this amazing story, but really it involves going into a co-op in Damariscotta, Maine and seeing just the most beautiful bouquet of mushrooms you've ever seen. Lion's Mane, Blue Oyster, Royal Trumpet. It almost seemed like they were sculptures, like handed down from Mother Nature. And I bought them. I wanted to buy all of them, because I was like, these are just so beautiful. I bought them, brought them home, and then it was like, yeah, painful, like heartbreaking to have to cut into them and sort of destroy this. But then as soon as I fried them up with garlic, butter, some onions, then I was okay with that. And sort of repeated that for the next few weeks, and then got really interested in actually growing mine. So I got a monotub that I built, got a kit from North Spore, started seeing these beautiful blue oysters just peeking through the soil, and then you blink, and then they're like 3X larger. You come back the next day, and then they're these fully-fruited mushrooms. And then I had a small grant from a program with the Goethe Institute. Because of COVID, they weren't flying people to Berlin, so they put those funds to projects. And then I was like, these things grow so fast, so they're so beautiful and they're voluminous. So they're the perfect subjects for photogrammetry. But what if we did a time-lapse photogrammetry? And that just seems like an inevitability, mixing two different types of technology. So I called up Ellie, I'm like, Ellie, you know, there's gotta be some tech out there that we can use, you know, like a plugin or, you know, like, a unit on Kickstarter or something like that but there was just nothing and that really was a little bit shocking and then we figured out all the things that we needed and in under a thousand dollars we built our first prototype because that was we didn't have much money from that grant too so you know sometimes you do with what you can. And we also worked with another, one of Ellie's former colleagues up in Montreal, Hugues Brière, who was really helpful in us getting the different things together. And once we saw it worked, it was like this big aha, eureka, turn on that light bulb moment. And then it was just like, what else, you know, how can we advance this? You know, how can we partner with the mushroom world? And so we called up Smallhold, And they immediately were like, let's meet. And then we became fast friends with them. They're really close friends and collaborators with us. And we just learned so much. Like, I feel like we're only scratching the surface of people who are micro curious and people who are actually professional, like mycologists or scientists who work in the field. And it's just been this amazing experience where we want to share what we've learned. continue to educate ourselves and explore the immersive possibilities of understanding different organisms on their own timescale. Similar to Tree from back in 2017, but you know, I feel like technology's evolved a lot. It's still hard, doesn't get any easier. VR is exactly the same, like, difficulty level, but as far as the resolution goes, the amount of pixels you can push through the headset, I think there's a chance to make things even more immersive. and also to make people feel really comfortable and feel like they're a part of something that they normally wouldn't be able to experience from a 2D screen. So we're really interested in continuing to push the boundaries of that, not just in VR, but as maybe a culinary experience. We've already started talking to different chefs and organizations who are interested in that, exploring the potential for, you know, what are the medicinal slash uses that can be used for PTSD, you know, the psychoactive and psychedelic applications are really fascinating as well as, you know, how are mushrooms relevant sort of on a larger environmental impact level for myco or bioremediation, getting rid of fossil fuels that have been a challenge that could ultimately, you know, it could save us in so many different ways and also is a fantastic meat alternative. I mean, there's just so many things to discuss. So I feel like we want to continue to develop chapters to dig into each one of those. This is basically like an onboarding just to understand the life of it. And then we can understand how it more directly impacts us through future experiences.
[00:08:21.260] Kent Bye: It feels like there's a lot of different ways you can explore the mycelial branches of this topic alone. But I'd love to go back to the photogrammetry capture. Back in Tribeca, there was a pitch that I saw you both give. And then I went to your offices and saw the whole rig that you have here as well. And so what were some of the things that you had to do, actually, to make this work of figuring out how to do all the lighting and what, on the software side, you had to do and actually do this time lapse to see these mushrooms grow?
[00:08:50.361] Elie Zananiri: Yeah, it was definitely a lot of a learning process there. One thing we learned with photogrammetry is that it's all about precision. You need to know exactly where your cameras are, you need to have the exact right lighting conditions, you need your subjects to not move as you do in full capture. We were lucky enough to partner with RealityCapture, which is one of the leading pieces of software that does photogrammetry from the beginning, and they've been really helpful. We have regular meetings with them where we show them our data, they kind of look at it, try to process it themselves and give us feedback on what we could do both on the photography side and on the software side. But basically, through a lot of trial and error, we realized that it takes a lot of patience, and then sometimes the hardware breaks, sometimes the mushrooms don't grow. All these things can happen, but it's just about reiterating and trying again. One thing about all that that we realized from the beginning is that it's a really long process, and if you do a lot of it manually, it's going to drive you crazy. So from the get-go, a big part of the project was figuring out how to automate this as much as possible. So we just have a rig that runs on its own every 30 minutes. It turns on some lights, regulates the humidity, and then starts capturing images. So it does a full 360. The rigs we have have four or five cameras, and they just capture between 160 and 200 pictures of every... Mushroom reconstructs them as one 3D model, and that makes a single frame of an animation. So that whole process happens first on a Raspberry Pi, which is a microcontroller that gets sent to a computer, which has the software running on it. Eventually, those get cleaned up and turned into mushrooms without all of the noise around them. And by automating this entire pipeline, it really made it easy to experiment, because we don't have to sit there and click around to see if it works. We can just have something running and just kind of get results at the end of the day. And if it doesn't work, we can just fine-tune it, try again, fine-tune it, try again. So that was good. Working out of the smallhold mushroom farm was really useful, too, because we had a lot of help figuring out how to grow the mushrooms, but also we had an almost unlimited supply of, I guess, actors, like, that we could shoot. I think originally we were thinking of shooting in the forest, like, our first idea was, oh, let's bring some cameras out in the woods and shoot them out there, but that would have been practically impossible, because you really need controlled conditions for everything to work, because mushrooms need a lot of humidity to grow in a healthy way, and then cameras do not like humidity at all, so there is a balance there that you need to work with.
[00:10:58.216] Winslow Porter: We learned that the hard way, a few cameras later. And as soon as it goes past 90, it's a little bit tricky. So yeah, we had to really learn from some of these amazing mushroom farmers, too, who became our friends. And they were really interested in what we're doing, because what they're doing is very organic, like physical, and we're doing things that are very like hardware focused, very technical and very virtual in many ways too. But I think that there's a shared fascination with not just the farmers we work with, but just the whole mycology community in general. So there's just this massive knowledge share. When we went to SIGGRAPH, it was a very sort of buttoned up, perfectly on time. You know, here's our technical feedback, you know, white paper kind of crowd. And then when we went to the Telluride Mushroom Festival two weeks later, A lot more loose on the schedules, way more open mind, creative applications for things, and a lot more interested in the cultural implications of things. But again, same sort of curiosity. Mushroom folks are, I'd like to say that mushrooms in general, or mycology, is kind of like an ancient band. where everyone has their own, like, with a massive catalog, and everyone has their own, like, deep cuts that they can go back to and talk about. But it's also, it's for everybody. There's no, like, barrier, or it's not a walled garden. Like, anybody can be a part of it. And it's just a question of sort of, like, what your entry point is. And we wanted to create this experience as something that would get people really, you know, micro-curious, and ultimately speak to other people, join the community. area, you know, a lot of places in the U.S. have their own mycological societies, and we've met different ones, and they're all unique, you know, just like the ecosystems of different forests, so there's a lot of amazing metaphors, you know, that we can say, and a lot of puns, too, but we'll try to, we'll try not to go too heavy on the mushroom puns today.
[00:12:48.963] Kent Bye: Well, so I can imagine that you're starting to get these volumetric time-lapse captures of these mushrooms. Then you have to figure out how to put that into an overall arc and a story, but also have these other layers of the haptic experience and the music experience and the smell experience. And so maybe you could just walk through a little bit of your process of how you start to put together a piece like this.
[00:13:12.302] Elie Zananiri: The interesting thing with Forager is that we didn't really have a set plan from the beginning of our schedule. There was a lot of experimentation. So this version we're showing here at South by is actually the third or even the fourth VR experience that we created for Forager. We had like a full demo start to finish that was working. We had prepared for Tribeca last year and then as soon as it was done, we weren't really happy with it. So we kind of scrapped it, started over and then worked a lot on prototypes. So we had prototypes using We tried growing mushrooms out of your hand. We tried having mushrooms grow that are larger than life. We tried controlling the playback of mushrooms by walking around them. We just had all these prototypes and whatever stuck we kept and whatever didn't stick we kind of threw out. And because we were iterating so rapidly, it was kind of easy to just throw out ideas that weren't working 100% because we didn't spend too much time on them and we didn't have an actual deadline to finish. However, the counterpart to that is that we never really finished because we never had a deadline. So when South by came around, we made a decision like, OK, let's just make this our deadline. Let's spend the next three months coming up with a VR experience that has a narrative arc to it, that has a story that makes sense, that features all of the tech that we've created and all of the learnings we've had from all our prototypes. And that's kind of how it came together. I think Winslow is better suited to speak about all the multisensory stuff, because that's really his jam.
[00:14:27.346] Winslow Porter: Yeah, I mean, there's so many learnings from Tree, obviously, bringing over 100,000 people through the experience. I didn't do every one of those people, but I brought in many, many, probably tens of thousands myself. And you just see sort of what works and what doesn't and the reactions on people's faces when it is timed right. You know, it's not too overbearing in the experience where, you know, you don't want to hit people over the head with like, you know, a smell or with like, you know, haptics or, you know, you don't want to be getting in their ear like David Attenborough telling them how they should be feeling about something. It's really a subtle dance between all the haptics, you know, the sight and the sound. And oftentimes they're just as important because that's also how we navigate our own reality and if we can simulate that in a way that feels comfortable then you have more immersion and with that you'll have more emotion and really yeah it's something that a lot of people in VR talk about or use but like story feeling I think is a very real thing we're trying to express things that are often ineffable like a feeling that can't be described by words but you have to be able to evoke this through you know for us it's a seven minute experience where we start as spores we ultimately go down into the soil as mycelium and then pop up as fruiting bodies and then also with any life there's also the decay as well and that's a really difficult journey to tell with tree just the narrative you know as kids we understand that seeds turn into trees but no one's telling us at least when we were kids, about the journey of a mushroom. It's a lot more nebulous, and so that's something that we wanted to elucidate through this experience, because I think mushrooms are now becoming so much more top of mind in popular culture, but also just for all the reasons I mentioned before, for all the benefits that they have, and they predate plants as well. They've been around, and I think it's just, you know, only a matter of time until they get their time in the sun, although they don't like to be in the sun.
[00:16:19.300] Kent Bye: So time in the dark.
[00:16:20.000] Winslow Porter: Time in the dark, exactly.
[00:16:22.410] Kent Bye: Well, I felt like you have this kind of like beanbag thing that you're sitting on and you're somewhat upright, but still mostly laying down and you have, I don't know if it's a butt kicker or a sub pack or some sort of like haptic device in there. And so maybe you talk about, cause there, there is a tight correlation between the music that's in this piece as well as having the low enough frequencies to actually feel it. And so I felt like this experience of like being so down deep in the roots that I feel like these low frequencies, not only in my body, but also the multimodal audio cues as well. I think those together, I felt like it was tying together both the audio and the haptic experience that felt like a whole journey within its own right. So I'd love to hear you elaborate on how you build like contrast or building and releasing tension when it comes to like the whole haptic dimension.
[00:17:08.403] Elie Zananiri: Yeah, well, I mean, I would say like a big part of that is working with Q Department, which was our music and sound design partner on this project. And they really got it. Like, this is not the first time we work with them. They have a very open mind. They're very comfortable with VR. And we definitely thought that having a sub pack or some type of rumbling would be helpful to ground people, which a big part of the experience, because you're laying down, that's more of a meditative thing. You want to feel grounded. You want to feel like you're just part of the environment. So we knew we wanted to do that, but the rest of it was just sending them builds and sending them screencasts and them kind of understanding right away what we were trying to convey and just working with us to get it done. It was, I'd say, a fairly easy process. Like, they were great collaborators on that, just iterating on their own and just coming up with these solutions themselves and us just putting on the headset and being like, oh, wow, this totally works.
[00:17:53.182] Winslow Porter: Yeah, we thought we were torturing them with just like all the last-minute requests and everything, but they really loved it. And I think what really helped those, we sort of got them into the mushroom mindset early on by bringing them to the farm, sending them blocks that they could grow themselves to sort of have that moment. Because, you know, with any VR, too, it's what the knowledge you bring or the experiences you bring to it that help form what you're experiencing, because it's all contextualized based on, you know, what you've done before in your life and your knowledge of the subject that you're experiencing. And so I think that having that understanding with them and prototyping a lot with them, because VR is not at the speed of thought. If you have an idea, it sometimes can translate to what you see on the screen, you know, in final pixel. But oftentimes it's, as Ellie was saying, it's about iterations. You know, we've done so many different builds. We've done so many different captures. Probably only 5% of the mushrooms that we've tried to put in the jar and have grow through the rig actually came out. You know, we've done probably like 77 captures and we have six that ultimately went from beginning to end. Eight. Eight. But you know, not all of them look as good. And you know, the power goes out in the facility or like there's a pipes froze so that we couldn't get water to the fogger. You know, if anything goes wrong, the whole thing can get messed up. But. Everybody in our team including, you know Q department especially totally understood that this process is experimental and we're trying to do some things that Not to say that no one's ever done, you know lying down experience or people are mixing scent, you know But I think it's really about how we put all these things together and make it so that it's a dance You know a symphony of the senses
[00:19:26.683] Elie Zananiri: One thing I just realized now is that every single member of the team who worked on this project grew their own mushrooms. Because as soon as we onboarded them, if they were in New York or in Austin or in LA, we tried to get them to visit one of the smallhold mushroom farms. But even if they weren't, we shipped them a mushroom block so that they could grow their own mushrooms. That was kind of one of the first steps, the onboarding to working on Forager. And most people, I think everybody did it, and everybody would come back with pictures of the mushrooms growing after three weeks and be like, this is the craziest things I've ever seen. I get it. Everybody kind of got why this was important and why this was a really interesting thing to build. And I think it really helped, not convert, but just bring everybody onto the same page.
[00:20:02.278] Kent Bye: Can you both elaborate on that? Because you've been speaking about the experiential element of the growing the mushrooms. But what were some of your own personal insights of going through that process?
[00:20:11.356] Elie Zananiri: Well, they kind of have a life of their own. They're very easy to grow, and they grow extremely quickly. You can practically see them grow. Once it starts growing, every two hours, you can come back and look at your block, and it gets bigger. And they also change shape over time. So we've grown so many, you realize when they start, you have really no idea how it's going to end up. So it's really kind of a fascinating thing. They don't grow like plants at all. They just have their own way of growing. They take over a block of sawdust, essentially, and then just these gnarly things come out of the top.
[00:20:39.226] Winslow Porter: Yeah, it's like they have order from chaos, you know, like something that would normally be discarded. It's like the most beautiful form comes out of it. And also for us working in the digital media space for so long, posting that first really clean capture, we didn't post much. We weren't putting the word out until we knew that the stuff worked, until we know we got something that looked good. And when we posted it, like a lot of different members of the creative code community were totally convinced that it was procedurally generated. and they're wondering like, you know, what type of software we use, and we're like, ha, that's exactly what the response we wanted. But Mother Nature, you know, we credit Mother Nature as our art director. In many ways, that's true, because we're trying to simulate things as accurately as possible, obviously taking some artistic liberty for the simulations of mycelium underground, but, you know, we want to continue to use photogrammetry as much as possible to be able to bring things that You can model, you can texture, you can add really interesting shaders, but there's so much in nature that is just, again, beyond words, and that's what we want to try to bring in. When we take people to the studio, even if they're not in the headset, just seeing a nice 32-inch monitor and seeing the mushrooms grow, people just start going, they just start making noises. Just like, wow! Oh my, what the? And expletive here, expletive there. And then we're just like, high five. You know, it's just like, that is what we're trying to do. Those moments of joy. Like, how do we evoke that? Through something that's, even if it's a representation of it, is like, that is the feeling that we're trying to instill in the audience. And I think that we're only beginning this process because there's so much more to explore. The more that we meet actual scientists, I think that there's a lot of good findings that we can learn from them and then also hopefully we can inspire some of their work from the stuff that we're doing as well.
[00:22:27.627] Kent Bye: There's something about watching it unfold over time where you get to see this creative process where you kind of imagine what may happen but then it always defies your expectation in a way that it creates this novelty that it's like this perfect sweet spot of that art where you're able to have that level of something that's surprising and creative and novel. So I guess there's this other aspect, though, when you have, like, as you're growing, you're transitioning from one to the next by, like, transforming that into, like, a point cloud representation that then kind of morphs in. And maybe you could talk about those transitions of, like, as you have these volumetric things that get to the point where they're, at least when you stop filming them, fully grown in the sense that they're no longer being captured, but then to transition to the next, you kind of have this, I don't know if it's like a creative coding type of inspiration, but maybe you could talk about that transition.
[00:23:14.495] Elie Zananiri: Sort of. Definitely the point cloud is one of the creative code mediums that people use a lot. In our case, basically we had a challenge. The way the mushroom animations are built is that they are completely individual frames of an object growing. So every single frame is a 3D model, which is completely independent from the previous one. So you can't really do optimization tricks like animating points. It's really just completely different things that are growing one after the other. I think if we had more time and budget, we could figure out a way to optimize that. But for now, we just had to deal with massive amounts of data and reading these massive amounts of data from disk. So one of our lead devs that worked on this, his name is Will Young. He's based out of the UK. He came up with the idea of using point clouds because we could pack the point cloud data into images, and images can load really quickly on GPUs. So we have this custom format that we're using to play back these animations that works really well with points. And by making the points big enough so that they overlap, but not overlap too much, just overlap slightly, it can look like an actual solid mesh and like a realistic mushroom. And one thing about our point clouds that's interesting is that they also still have lighting data, so we know how they can catch light and reflect it, which also helps making them look realistic. So by coming up with this system, we were able to have a workflow that made sense that you could play back in real time. And then we also had point clouds. So we're like, OK, well, cool. Since we have point clouds, let's use point clouds as a transition system. Let's use point clouds as like, let's show the raw data that's hiding under the hood. Whenever we do transitions, or even whenever you do more abstract representations of the mushrooms, we can just kind of play with that idea and bring back the creative coding, I guess.
[00:24:47.180] Winslow Porter: Clouds was point clouds, you know, so it's like it kind of an homage, you know in a way But also in the beginning with the logo, you know, it's a sporulation, you know It's everything is represented as points and it's so it's a metaphor to show how also mushrooms are so responsible for the forest sort of being one entity instead of all these individual things and And, you know, through Michael Reisel connections, there's different types of mushrooms, too. Not all of them are connecting things, and that's sort of an overgeneralization. But, like, the wind is a strong theme in the piece, and how we're breaking things down into spores that recreate other things. Sort of showing that, like, everything is made up of other things, and also... even though they're singular, they're all part of the same organism at the same time. And so, yeah, for the next version, there's a lot of things that we have already done a lot of R&D and work well in their own project file, but crash the project itself. So we're really looking forward to V2 where we can Some really beautiful representations of mushrooms in different types of simulations of like wind and being able to morph into different objects too. No spoilers just yet. But I think the other thing that's interesting is we're also at the mercy of the medium. VR is notoriously bad at doing overdraw or like whenever there's things that are in front of other things, it's very challenging. And also the pixel size. So there are on our screen sometimes the mycelium and the point clouds look incredible, but as soon as you put the headset on, it's just this like alias sort of like muddiness that happens. What's nice about this project is that in two years, three years, eight years, you know, being able to use a higher, maybe 16K headset, we'll be able to look at our piece with a sort of a fresh lens, and we can actually be able to more accurately recreate these natural phenomena as you would see them in real life. Actually, the first day here, one of our mushrooms that was installed by this amazing group called Wild Bunches, our mushroom partner, Smallhold, the light was hitting it, so you could see it sporulating. And it was like right on top of the installation. And so everyone was sort of looking at it. I thought the computers were on fire. But it was just this sort of natural thing that was happening. But you could see how small they were. But actually representing that realistically would not be possible in a game engine. So we made everything a little bit bigger, made everything a little bit more intuitive. Because this is, again, a very unknown narrative that we're just trying to chip away at.
[00:27:14.365] Kent Bye: Yeah, the first day that I saw it the wind and the smell wasn't quite set up or working correctly yet But I just had a chance to see it now and I felt like being able to feel the wind at certain points It's very subtle, but actually like very immersive in a very odd way I mean you think it would like not be as effective as it is But it's actually very grounding in a way that makes you feel a sense of your body and your haptic experience that gets just much more embodied immersion into it, but in the smells I I heard there was like four different smells. I think I could distinguish a couple of them. They were kind of subtle. So I'd love to hear a little bit about that process of both designing the smells and trying to get the dispersal of that smell just right.
[00:27:50.852] Winslow Porter: Sure. So there were five smells. There's the Pine Forest, Resinous Pine 2 is what the actual name was. We work with Scentronics and OW Smell Made Digital for the actual scent delivery device. And this is one of the first times that they were using it as well. So a lot of it is, you know, very much like trial and error, trying to see how things work. And everyone gets a slightly different experience. Ellie, unfortunately, after COVID can't really smell. So I was definitely put in charge of making sure that those smells were approved and sent back, all in a very short period of time. We actually got the smells the night before we went live. But we had already done some back-and-forth AB scent testing, and gave descriptions of what we wanted, and they did a really great job. So yeah, resinous pine was the first one, then it was basically like a forest floor, then it was like an underground, earthy underground smell, and then it was kind of this slightly funky mushroom. And then we also go back to then the forest floor, we go back to the resinous pine, and then we have this sort of sweet exhaust smell to sort of, along with the visuals where you see like trash accumulate and you see, it's sort of subtle, but you can see the shadows of buildings that we wanted to literally foreshadow the fact that there was a city, sort of see time passing, and also the sounds of the city too. So all these elements together allow you to advance the narrative in that way. But then what's interesting, you mentioned about the wind, and wind is something that's really fascinating because there's no HD wind, there's no 4K wind, like wind is wind. And it's also something that our skin, we trust that sensation just a little. We're just using one speed, but we're able to click it on and off at different intervals, so it can simulate just like a slight gust. but you trust your body, your nerves to tell you if that's real and it knows that that's real. And so I think in many ways it can be more immersive than the visuals because your brain is telling you, okay, this is, I'm willing to go along with this, but I see pixels, you know, I see some tearing because of the frame rate, but then wind is, wind is wind. And it also has a dual purpose of cleaning out the scent too. So people aren't just stuck with that one smell.
[00:29:55.370] Elie Zananiri: I think one thing about all the extra sensory things that aren't visuals, it's kind of like with audio, like whenever the audio works right, you don't really realize it. You're just like, wow, this piece feels like cohesive and it makes sense. And whenever it doesn't, you think that something's missing. And I think everything we did regarding wind, regarding rumble, I guess is the sense you would say, and smells, we tried to go with the same idea of like making it just feel natural like you know because it's there everything feels right and then if it wasn't there you'd know that something was missing so it's it's a weird balance there and I think it works because those are all the senses that are actually 360 degrees whereas vision is more like a little less like 170 I think is what we see but all these other senses are senses that you can feel all around you.
[00:30:36.156] Kent Bye: Yeah, one of the things I also appreciated is in the second chapter of Mycelium that you have your ability to use your hands to kind of draw these mycelia trails and at first when I did it my hands were rumbling and then you know that's usually the universal cue when you're doing a VR experience to do something with your hands and so it took me a It took me a moment to figure that out, so I did it at the end and then I just did it again here, but just to be able to weave my hands around and be in the creative process of creating the mycelia on top of the rumble and the music I felt like was also, and I think at that point it was like the underground smell, so I felt like it was The confluence of all these things together just felt like this really cool journey into the underground, but I'd love to hear about that interactive component, because most of the other piece is non-interactive, but that is one aspect that is more deliberately interactive, and I'd love to hear how you think about that in terms of your embodiment and interactivity in this piece.
[00:31:27.007] Winslow Porter: One of our amazing devs, Ming-Chi, who's actually a student of both of ours at NYU ITP, super creative, and also loves to work really late, too, with us. It's just me in the chair, and we were trying to figure out how to debug it so that we knew that those were our hands, and we're like, we know it's difficult to get people to use the controllers if they haven't been, so we're like, did he turn on the debug mode where it shows two translucent hands? Because we're like, no way, we're not using translucent hands. Get that out of here. People will figure it out. And then it was like, no one's going to figure it out. Because we already have this amazing mycelial simulation from our technical artist, Georgios. But we wanted to make sure that people knew what was theirs and what was something that was already pre-rendered. And that, along with the vibration, I think just a lot of people, this is maybe their first VR experience, maybe their first multi-sensory VR experience. So we don't want people to feel like they missed something or that they left out or they did something wrong. So for us, that was you know, we also felt like we needed to have some type of interactivity in the piece. There's other things that we want to do, but also with creating a story that is trying to simulate nature, nature doesn't make choices the way that humans do. So we don't want to do something that also subverts the narrative. And so for us, we found that that was a nice piece where you could be painting this world and sort of feel like mycelium. And there's definitely some camera shifts or some like, you know, perspective changes. And that was conscious because, you know, we didn't want to make something where you're mocapping a mushroom, where it becomes comical. So there's a lot of things that we want to continue to do with interactivity, but also understand that nature doesn't function the same way that humans do.
[00:33:03.872] Elie Zananiri: And yeah, we made a lot of conscious choices regarding interactivity. Like I said previously, we made a lot of prototypes leading up to this. Some prototypes were fully interactive, where you actually had to walk around the mushroom at a room-scale setting. And that was just confusing for a lot of people, so we had to scale that back. We had another version where we had a really lush forest that you were free to roam around. And we realized with that that nobody was looking at the mushrooms because there was so much other stuff to look at. So that's how we ended up settling on this idea that where you're mostly on rails, So we can kind of guide your focus and make sure you're looking at the right things and appreciating the things that we spent a lot of time showing you. But we still ended up with a moment where we felt like interactivity made sense, and that was the mycelium portion of the piece.
[00:33:42.781] Kent Bye: And you have a pretty epic installation here, which is all these mushroom block things. Maybe you could describe what your installation's made up out of.
[00:33:50.508] Winslow Porter: Yeah, so we were really lucky to be able to work with some people that we've done other VR experiences with before, like Justin Durazzo. He was at Droga when we were doing Tree, but when he was at Droga, he also worked on Procession with Ellie last year. So it felt like kind of getting a band back together, you know, six years later. So we had a lot of learnings and we knew what works in the space, but we also knew that a lot of people, unfortunately, at these conferences can't witness the VR. Just it's a numbers game. We only can do two people every 15 minutes. So we wanted to be able to give a broken down version of the experience in pieces as an installation. So that's why we have the volumetric capture rig, which is sort of the beginning of the experience. Sort of the mushrooms at the beginning where you see them and then the volumetric capture and a big mushroom. that was 3D printed by Pink Sparrow using a Massive at 5000 printer, and it was like seeing something larger than life. And then being able to see the final mushrooms through the looking glass display, and then also the microbrick structure. We worked with Accenture Song Sustainability Studio. Daniel Perlin was able to contact the architect David Benjamin and his studio living in New York. And we were really inspired by the idea, how can we make an installation that is fully compostable to at least the brick structure? And how can we inspire? It was way more work than just having like wood blocks or plastic things from Ikea or like, it was so much more work that went into it. But I think that inspired people. I think that people felt the amount of like love that went into that. and the consideration for the design that we spent, you know, we reworked so many different things, and then when we were actually on site, continued to rework it to make sure it worked. And then also it's a metaphor too, so the bricks, you know, represent the mycelial layer underground, and then on top we have the mushrooms, so it's like the forest floor is represented there as well. always trying to tell the story before the headset experience. And then the off-boarding as well, they get the pouches that have a small mushroom, sort of remember that time when they were witnessing the life cycle, but also a map of old growth forests that are still here in Texas. You know, I'm not sure how many people will actually drive an hour east and go foraging, but I think it's to make this feel specific to Austin. And we'd love to do that for every other place that we show. And I think that that makes it personal, like that feeling of being back in the headset, like that memory that they had, even though it was virtual, it still can feel real.
[00:36:14.227] Elie Zananiri: Just to clarify, I'm not sure you said it, but the bricks are made out of mycelium. So they're grown in molds and they are mushroom bricks. So I think that, yeah, like Wenzel said, it was important to do that because we're thinking if we're making a project about climate or about the environment, it doesn't really make sense to just like have a bunch of plastic everywhere that, you know, it's going to end up in a landfill in like four days. So we really felt like it made a lot more sense to go the extra mile and find something that I guess more conscious.
[00:36:37.722] Winslow Porter: And that was made by a company called Ecovative. And we thought that these bricks were something that people were building structures out of all over the place. And they were amazing. But they're like, you want how many bricks? The final number that the structure was was 666, which is very good luck. And that was the largest brick order that they've ever done. They had to make new molds, and they did it in such a fast pace. And we also got the last order of bricks the night before we went live. So everything just happened to fit in place, just like the bricks, perfectly. But yeah, I mean, it was very much experimental from beginning to end. But I think it really paid off.
[00:37:13.881] Kent Bye: Well, what's been some of the reactions from people who are the mushroom people, the mycologists? How is this inspiring them to reimagine what's possible?
[00:37:23.505] Elie Zananiri: I think we saw that mostly at the Telluride Mushroom Festival. We just went there as we weren't planning on showing anything, but we did have a laptop and we had some visuals going on. And the few people we showed it to, some were just foragers, some were scientists, some were genealogists. Everybody was really, really fascinated with how true to the shape our captures were. You could really see all the grooves and all the detail in the gills of the mushrooms and all that, and they were really fascinated by that. A lot of people were really curious and trying to look deeper. They were very interested in playing with scale. blowing it up and seeing everything you can't see with the naked eye, everything that you would usually need a microscope to see and some special equipment you can just see on a computer screen. So that was the main takeaway. Everybody was like, wow, zoom in on that thing. Enhance it. I want to see inside of it.
[00:38:08.762] Winslow Porter: Yeah, and I also think, you know, when you go to the tech world or academia, it's like white papers and you sort of have to go through the chain of command or, you know, sort of this caste system of knowledge. But with Mushrooms, though, these leading experts, scientists, biologists, you know, authors, poets, like everyone was interested. I didn't feel like we were super accepted because we were trying to see things from a new perspective that they were also interested in. So there's just this intense knowledge share going on. And I think that now that we have the project complete and we're premiering, Like, we want to bring it back to New York and show all the mushroom farmers there, bring it to Telluride this year and show everybody. You know, we were just showing them a proof of concept and they were like, hey, like yelling, you got to come check out this shit. This is crazy. And, you know, we're like, OK, I think we're on to something here. So actually being able to sit them down and do it and like, I think we're going to get a lot of hugs. And that's really what it's all about. Colorado is beautiful and that I think that we were sort of not fully prepared for what we were going to like immerse ourselves in but I think now we understand the community better and I think that It's the perfect time for us to be able to show this because mushrooms so hot right now
[00:39:18.423] Kent Bye: Yeah, I know that this was one of the more popular experiences and surprisingly difficult to get into. I know usually during the press preview you were able to see stuff and it was a relatively short experience but pretty much booked out for each of the days and so it sounds like it's really resonating with the crowds here at South By. But yeah, just to wrap up here, I'm curious what each of you think the ultimate potential of virtual reality might be and what it might be able to enable.
[00:39:42.154] Elie Zananiri: potential of virtual reality? I think for me like the projects I'm really interested in making and the projects that I find successful are projects about embodiment like really you know seeing things from a different perspective, understanding things from a different perspective. I think that's really where virtual reality really shines and I think the more we can improve that and make it be more I guess work better and better like make the encompassing How can I say this? Like, make the encompassing portion of virtual reality better is what's going to make it more successful. Whether that's by increasing the resolution of the screen or making everything lighter so it feels like it's not on your head, by adding different sensory components like we're trying to do here, whether it's like tactile or like through olfactory taste even. I think just bringing that all together to build something that's more immersive and just, you know, you make sure your relief transports you somewhere else as something else is what excites me about it.
[00:40:33.284] Winslow Porter: Yeah, I mean, ultimately, I think, I think that the problems were so caught up in these hype cycles, you know, VR, then AR, and then like crypto, and then NFTs, and now AI and metaverse, like, it's all buzzwords that people who don't really understand the industry or the potential, keep sort of over inflating, and then it never matches expectations. I think it's all about connecting to a narrative. Some of the best VR experiences that I've been able to, you know, lucky enough to be able to witness are from like six years ago. like, draw me close, like, even thinking about that, like, I could burst out into tears. In fact, I do, and I describe that to my class. I, like, needed a moment. I, like, started getting goosebumps, and I was just starting to cry, and everyone's like, what the hell is wrong with our professor? Is he, like, bipolar? Like, what is going on here? But no, it's like that stuck with me in such an amazing way because there was a certain humanity to it. There was a certain, like, it wasn't about the frame rate. It wasn't about the clarity. It was about the connection. And I think that also people, yeah, are putting way too much importance on the technology. Granted, the technology could be lighter, could be higher resolution, could be cheaper, you know, the access to it. And I think that there's a lot of buzz around the metaverse, which is, I think the good part of that buzz is that people are, understanding that it has a potential. But I think that connecting brands to people is maybe not the best way we should be thinking about it. Or how do we recreate an office setting in a headset? Like, how do we get people to work more in VR? Like, that's not what it should be for. It should be about enabling narratives, embodiment, things that you can't do in real life. How do we have hyper-real experiences? How do we have surreal? How do we witness things on another time scale? And, you know, and empathy. That word gets thrown around a lot. But ultimately, I think that there is a huge potential for a new generation of people who now know Unreal like the back of their hand and can do things really inexpensively with a group of friends. They could turn out something in a few months that will really impact people's perception of things that previously were not thought possible.
[00:42:30.672] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:42:37.842] Elie Zananiri: I'm excited that it's happening again. I think since, like I said, last big VR festival I've been to was Sundance in 2020 and everything kind of died down after that. They went online for a bit. But now that we're at South By, there's a lot of amazing projects here. It's really, there's a lot of variety, which is really fun. I think you can see the medium maturing and it's really exciting that it's still going strong.
[00:42:57.056] Winslow Porter: And I'd also like to thank you for believing in this project, for believing in VR, for believing in immersive media, for actually fostering relationships with the storytellers, for the amount of work that you've put in. I think that we all owe you a huge round of applause.
[00:43:11.875] Kent Bye: Awesome. Well, thank you so much. And thanks for your gracious hosting of a party last night. That was really nice. And this experience is a lot of fun that I feel like you're able to tie together a full multi-century experience that is really fascinating to dive into the world of mushrooms and to understand the different stages and to get to see them unfold and flower over time. And yeah, it's really one of my favorite experiences this year at the South by Southwest. So thanks again for joining me here on the podcast.
[00:43:36.809] Elie Zananiri: Thanks a lot, Kent. It means a lot. Yeah. Thanks again, Kent.
[00:43:39.972] Kent Bye: So that was Winslow Porter, a co-creator, producer, and creative director of Forger, as well as Elie Zanonieri, who's a co-creator and technical director on Forger. So, I have a number of takeaways about this interview, is that first of all, Well, the installation of this piece was absolutely on the next level in terms of you going into this brick-like entity with 666 of these mushroom bricks. You lay down onto what is kind of like this beanbag, but it's slightly upright. And then inside of that, they have a sub-pack. And so, in front of you, they have this wind machine that is dispersing five different smells. And so, you go through this immersive experience. It goes through these four different phases of a mushroom. I forget what the first one is, but the second one was the mycelium and then the fruiting body and then the decay. You kind of get transported into the forest, you go underground with the mycelia, you have this interactive component as you move your hands around, you're able to kind of draw this procedurally generated mycelia, which was a really cool effect and your hands are vibrating as you're going through that. But I think the third phase is really the star of this piece, which is the fruiting body, where they have these time-lapse animations of these mushrooms that then do this pixelated point cloud representation morphing from one to the other. And it turns out that in order to actually show these animations, in order to be optimized for a real-time experience, they had to turn them into these voxels, with these voxels that are overlaid. And through this overlaying of the voxels, you're able to get the shape of these mushrooms as they're unfolding. I didn't realize as I was watching it that they were voxel based because they were so overlapped in that they created this volumetric sense of it, but going from one mushroom to the next, you could really see those point clouds morph into one to the next. And so that was just a really effective way of using this creative coder type of aesthetic to be able to go from one to the next. Yeah, just really fascinating to hear about their editor design process and to get to the point where they were able to really lock down what the story was after all these other different dimensions of interactivity and whatnot and then they kind of dialed it back to be more of a passive immersive experience where there's some interactive components in the second phase where you're moving the mycelium around but there's nothing drastic where you're controlling the unfolding of these Mushrooms, and they have a lot of plans for where they can take this in future I mean mushrooms are endlessly fascinating and they're such a vital part of our ecosystem And so yeah endless amounts of possibilities for where they could take this in the future So really excited to see where they continue to take this series and this as an immersive experience It's just really solid highly recommended folks to go check it out Just because it was super satisfying and I think the topic within itself is also really compelling And all the other different installation components and how they think about the onboarding and offboarding and everything else. They're just these seasoned experiential designers who know how to take what is really quite compelling to see these mushrooms grow over time and just all the different technical innovations they had to do to even just capture these. photogrammetry and that just all the different times that it failed with all the different conditions and whatever else was happening, the mushrooms just not growing or other technical difficulties. And so, yeah, just a lot of patience for trying to capture these time lapse captures and then to put them into an immersive experience like this. So, yeah, excited to see if they're continuing to capture these different images and to see where they take it in the future. And like I said, it's one of those things that could make a great location-based experience because it's just really short and quick and really satisfying. For me, I didn't necessarily smell all of the different smells, so I don't know if it was just the way that it was being dispersed upon my experience. You know, like, my smell may not be as sensitive. Who knows? It's one of those things that's perhaps a bit difficult to debug, because, you know, as Ellie, suffering from COVID, has no longer any... distinct sense of smell that is really relying upon Winslow. But I know that they were putting all these things together in this fast iterative cycle with the deadline. I think each of these different creators, as there's a deadline, they're working on it up until the last moment to get everything working. So the first day, actually, the smell wasn't quite working, and I came back and did it again, and I could smell some of the smells, but not quite all of them. But the wind haptics were actually really Immersive as well and so like I said just a really deeply immersive experience And I think there's a future for where this go into a number of different contexts I think it would be a bit of a smash hit and a museum context or other things like that where people are able to get this real deep immersive and sensory experience So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
