Marshmallow Laser Feast’s latest piece Evolver premiered at the Tribeca Immersive festival, and it takes you inside the human body in a transcendent journey into how the blood flows through our heart and throughout our body. They collaborating with the science & research organizations like Allen Institute and translating 2D visualizations of blood flow by Fraunhofer MEVIS into fully immersive VR visualizations. Being immersed inside of a fluid dynamics simulation of air flow and blood flow within the human body in Evolver allowed me to experience an interconnected aspect of reality that calls into question the boundaries between where the outside world ends and where the inside world begins. I had a chance to unpack some of the deeper philosophical inspirations & motivations behind of Evolver as well as the journey in creating it with Marshmallow Laser Feast co-founder Barnaby Steel at the Tribeca Immersive festival on Friday, June 10, 2022.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
From EP Terrence Malick and Edward R. Pressman comes EVOLVER: a free-roaming, immersive journey through the breathing body directed by @marshmallowlf. Narrated by Academy Award-winner #CateBlanchett.
Tickets available https://t.co/urxTIKTAi6 Capacity is limited! pic.twitter.com/Sa9xFrVoyq
— Tribeca (@Tribeca) May 25, 2022
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, this is Kent Bye, and welcome to the Voices of VR Podcast, a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support this project at patreon.com slash voicesofvr. So today's episode I'm continuing on my series of looking at the different immersive storytelling pieces that were featured at Tribeca Immersive. And so today I'm going to be diving into what was my personal favorite piece, Evolver. Imagine if you were to go inside of the human body and see the fluid dynamics of how the blood flows through the body. and through the heart, and instead of being normal scale, the person's 100 feet tall, and you're walking around looking at all these different things. And it's a real spiritual transcendent piece that is looking at different layers of interconnection and interbeing as you look at the breath that's coming into the body, and as that oxygen through the blood is flowing through the circulation, and just to be able to visualize what does the blood flow look like through the heart, and do it in a 3D spatialized way. Really, really super compelling and one of my favorite pieces. And we talk a lot about the process of producing this as well as the onboarding and offboarding and yeah, just the deeper philosophical principles of trying to get away from seeing us as separate beings and looking at more of the interconnectivity and how can VR as a medium start to express those aspects of this embodied interconnectivity and interbeing. So that's what we're coming on today's episode of the voices of VR podcast So this interview with Barnaby happened on Friday, June 10th, 2022 at the Tribeca immersive in New York City, New York So with that, let's go ahead and dive right in Hey, I'm Barnaby steel.
[00:01:49.736] Barnaby Steel: I'm one of the co-founders directors marshmallow laser feast and Yeah, I guess we're well the first thing completely exhausted all my juice. I put all my juice into this project. I'm completely spent and it's been a kind of four year journey. And I suppose, in a way, this piece embodies a lot of the philosophy and the reason we're working with VR in the first place, in that it creates possibilities of experiences that you can't have contained within your skin. Obviously, it's a simulation within your body, but you can give somebody an out-of-body experience to explore the inner architecture of their inner branching being, the ecosystem of the human body. So, it's endlessly fascinating. I mean, we can go into it, yeah.
[00:02:34.610] Kent Bye: Yeah, well maybe before we dive into the experience, you can give me a bit more context as to your background and your journey into immersive storytelling.
[00:02:41.739] Barnaby Steel: So, you know what, there's been sort of a shift in our approach to work. I think there's obviously like the situation that we're in. in terms of our relationship to nature and the sort of dominant narratives that shape people's relationships like the kind of myth of separation, the sense of you as an individual in competition to other individuals and that when life is framed that way then the actions that you take don't necessarily or acting in self-interest is not necessarily acting in the interest of nature so we're really interested in what is the experience of interbeing like experience of connection scientific narratives for us at the heart of that because they reveal these relationships that are unavailable to our senses. For example, following breath, which is one of the key themes that runs through our work. If you follow that thread, the atmosphere is a co-creation of all breathing beings. We make it together and we rely on it for life and it's also a narrative that takes us back to this ancient relationship of photosynthesis to respiration and So it's a great way to smear the boundary of where you end and begin and just reveal the beautiful interconnected richness of existence that we live only in relationship to the pollinators, the plants, the trees, the rivers, the whole caboodle, you know, the big bang expanding. We're part of this process. And so what is the experience of that? You know, that's really an interesting area to discover. yeah and that's kind of what we're doing working with scientists exploring these different ideas and then making them available as virtual reality experiences so you can step into dimensions of reality that you might understand conceptually or you know you might hold ideas in your mind about what's inside your body is maybe it's a diagram but that's very different to be able to step inside it and explore it in the same way you would explore a forest ecosystem and allow your curiosity to guide you and that's kind of what we're up to.
[00:04:39.022] Kent Bye: Yeah, as you're speaking, I'm reminded of Alfred North Whitehead's process philosophy, which is a lot about moving away from substance metaphysics of treating things as these concrete objects that are seen as individualistic or individual properties on top of that, versus the more process-relational approach, which is seeing that how we're all interconnected at a deep level, but that you can use the building blocks of relationship and processes to describe all the nature of reality through these fractally nested, or these muriologically nested, so holes within parts. And so you have one organism, but it's part of a larger organism and a larger organism. Those biological organisms, as a metaphor, can go all the way down into ecosystems of energy, patterns of energy. So I feel like that's a thread that your work that I've seen over the years comes back to again and again of just trying to see how we're interconnected and interrelated, but through that breath, how previous pieces of We Live in an Ocean of Air is about the trees breathing out the oxygen, and then this experience in some ways is a human being breathing in that. So in some ways, your body of work altogether is creating the full cycle of the breath.
[00:05:47.910] Barnaby Steel: Exactly, yeah, and I think when we're sort of diving into the science, a lot of the collaborations, you realize that there's huge institutions like the Allen Institute for Cell Biology, their whole focus is on the cell, so they're using a range of different techniques to 3D model and understand and break it into its components, and whereas another institution is looking at the Fraunhofer Institution in Germany, they're looking at blood flow through the heart, so they're using fMRI scanners to create time-lapse sequences to then from that black and white data, they can work out the speed and direction of blood flow through the heart. So different institutions have got their lens focused on different things. And I guess that's the nature of observation. Whatever lens you're looking through, you have to choose what you're focusing on, right? So it sort of excludes everything else and focuses in. I think it's interesting to be able to collaborate with all of these different partners and bring it back together. We're really interested in how these pieces combine and the underlying rivers, basically, that flow through life. And that's kind of what we've been doing on this project. So in some ways, you know, ocean of air, whether you're looking at a forest or the human body, those systems exist in relationship to each other. They're really similar and they're constantly in exchange. So seeing a certain way, a tree is as much part of your body as your own lungs. And I think when these works start to combine as we build this sort of larger narrative, I really believe that it's We sort of talk about when we're David Attenborough's age, you know, then can we imagine these sort of virtual worlds, these ecosystems where you can follow all of these different threads that flow through different organisms and cycle around. And that, I think the sort of holistic vision of that, being able to like maybe experience photosynthesis in an electron micro scanned leaf, where the leaf's the size of a cathedral, but that's just like one fragment of this larger collage. And in itself, it might be an exquisite kind of meet the creator moment, you know, the sort of, the light that flows through the plant kingdom and weaves the entire food web by being woven into hydrocarbons. That's sort of like, yeah, we're made out of sunlight, this beautiful flow of energy. But then that nested within a bigger picture where you can flow through a forest, through a human, experience how death becomes life. All of these boundaries just smear into this kind of existence tissue. David Hinton talks about In his wonderful book, Existence, A Story, he talks about this idea of existence tissue and actually it's in relationship to Chinese landscape painting. So as a spiritual practice, a blank canvas represents nothing and from nothing emerges something in the form of ink and as that ink is applied, The artist paints himself in the third person as part of the landscape, as part of this tissue. And the way the clouds roll into mountains, roll into trees and roll into humans, the whole thing is sort of seen as this tissue of existence. It's just sort of rolling in and out of being. that was like, yeah, that resonates, like, yeah, that's what we're exploring. We're exploring the tissue of existence and how that is hard to define anything. And actually the process of definition and labeling is just a thing that we do in order to point at something and explain it, very much attached to language. And obviously science, as I was talking about earlier, is doing that as well through the nature of observation. And so piecing that back together is just a joy and super exciting. And I think it's the shift that we believe is the source of behavioral change, you know, to think of a tree as as much part of you as your own body, that's not an experience I've ever really had. I don't actually feel like a tree is part of my body, but I think that maybe these perspective shifts that you can explore within virtual reality can be a sort of poetry that opens that conversation at least, maybe it can flavor certain ways that you might think about your relationships to nature outside. So that would be our biggest hope, if you can just like We were thinking about, you know, when you wear sunglasses, say you've got yellow glasses on, that tint can be powerful experiences, can offer that kind of perspective shift that travels with you, you know, after you take the headset off. And that would be the hope of the work, that it can be a kind of perspective shift to see that beauty and interconnection, because it's pretty crazy, isn't it? Existence, the whole thing, you know? So, yeah. So I waffled on a bit there, mate.
[00:10:06.705] Kent Bye: Well, as I first came across some of your work in Sundance 2016 with Through the Eyes of the Animal, and then eventually saw Tribeca, there was Tree Hugger, and then Sweet Dreams, and then We Live in an Ocean of Air, and then Dreams, and then now with Evolver, I don't know if there may have been some other stuff that I haven't seen or stuff that even came before that, but You said you've been working on this for four years, so that takes us back to around 2018. How do you think of your body of work, of each of the different themes that you're exploring in each of those pieces, and what led to this piece of Evolver?
[00:10:39.684] Barnaby Steel: Yes, I think the first thing is that MLF is three creative directors, Ersan Han Ersan, Robin McNicholas, and me, and we're old friends, and we direct our own projects, but there's loads of cross-pollination, so we're just better together, and we love it, and we've got a lovely little family, It's a real joy to work in such a creative, inspiring gang. And so within this sort of bubble there's lots of cross-pollination. I guess the projects I'm talking about are not really my projects, is what I'm saying. They're really the process of this large team and there's lots of different influences and projects can take a complete U-turn at a certain point because someone has a great idea about a different direction. So it's very much an organic, collaborative process. But what I'm bringing to the table I suppose is just being born in a city and recognizing within myself that my sort of detachment from nature and then through having a child and having the opportunity to travel and have some really deep experiences that have changed my perspective. I feel like playing with virtual reality and sort of the nature of this medium is just a wonderful way to bring deep experiences of nature into like an urban setting. And I'm not saying it's like the same thing as experiencing nature. It's very different. It's an artistic expression of something, but also it's fascinating to think about if you were stood in front of a tree or, you know, looking at somebody else's body, there's a limit to what you can see there. So then being able to peer inside the body or look at working with scientists to understand different flows and movements and all of the myriad of different scanning technologies. They offer a glimpse into these dimensions of reality that can be so beautiful. It's like the beauty of observation. Often that data is, the beauty is not so interesting to the scientists, you know, the data is maybe more important. And it's just been my sort of privilege and experience that diving into these narratives just creates a sense of awe and wonder that I never had before. The more you understand, the more you dive in, the more cosmic and unbelievable the whole thing is. And I think that process for me is what drives my artwork. And in a way, the final piece is an echo of this sort of baptism that me and the gang have been through. We come out very changed by the process. And we're really lucky to be able to spend our time doing this. And in a way, the artwork is a kind of echo of that. So it's quite selfish in a way.
[00:13:08.284] Kent Bye: I think it was in 2017 where I saw Treehugger here and then went on and then created We Live in an Ocean and Air as a location-based experience at the Saatchi Gallery in London that I had a chance to see. But what was the moment where you had the idea for this project and what was that turning point?
[00:13:26.447] Barnaby Steel: Yeah, so actually when we were doing We Live in an Ocean of Air, the conversation was always about the relationship to your inner branching lungs and cardiovascular system, that relationship of breath. We wanted to reveal the lung structures and the blood flow within the avatars of the bodies, you know, the people in the space. So the idea that you can be with your friend in a virtual reality experience, see each other, but you're seeing their heart beating, you're seeing that blood flow pump round, and when they're breathing you're seeing the flow in and the out-breath. When skin's transparent, race, gender, everything evaporates, like it's a kind of universal underlying I guess it's just there's something about it that was really powerful. The more we started to explore it, the more sense it made. And we also started to look at the medical data. I think the real turning point was browsing YouTube, like hunting, spend a lot of time trying to find, like, what is the kind of Hubble Space Telescope for the human body? Like, who are the people right at the forefront? And I came across the Fraunhofer Institute, Mavis, and they had loads of stuff on YouTube. And one of them is blood flow through a heart. So as I was looking at my phone was on my leg and there was some sunlight coming in through a window bouncing off the phone onto the wall and so the lights on the wall was moving to my heartbeat and it was just such a familiar rhythm and you almost block it out but if you listen to it you can feel it you can hear it this heartbeat rhythm but what does it look like? It's not like the highest fidelity video, but the more I sort of meditated on it, I was like, no one's ever seen this stuff. And it had like 600 views. So this is blood flow through a heart. Something so familiar, and yet what shape does it make? Because the beating of it, you know, the shape of that is the fluid dynamics. And so it's not about the muscle contractions, it's about the movement of the fluid. And that's what this video showed. And so that was, I guess I was looking for something, and that was the turning point. I emailed them, and they were like, oh, we know you, Marshmallows, let's collaborate. And they had an art fund, so they supported the project, and we've been working together for years now. It's a beautiful relationship, and yeah, I think that was the turning point of the project, and where the real deeper conversation started from.
[00:15:44.688] Kent Bye: So these doctors at the Fraunhofer Institute studying blood flow of the heart had already been familiar with your work?
[00:15:50.332] Barnaby Steel: Yeah. Bonkers, right? I think maybe Eyes of the Animal or something, one of them had seen it. Yeah, it was great because I email quite a lot of people and don't get many replies, especially scientists. They're busy and they're often doing amazing, really valuable work in the medical, just saving people's lives. They're really focused on that, so maybe artists are a bit of a distraction, but she responded really quickly and I was like, yes! Sometimes as time goes on there is a feeling that when a certain vision is flowing through you, like I don't really believe that I'm coming up with an idea, I think ideas sort of flow through you and then people tend to congregate around these things and the synchronicity is It just seems to me quite unlikely, although sometimes Google has a hand in it. It knows what to show you at the right time, all the algorithms and stuff. So who knows if it's digital synchronicity. But there was a lot of that on this project, just things aligning in a way that I could never have predicted. I couldn't have sat down and mapped it. And I think also the underlying vision of interbeing, the experience of interbeing. I mean, this phrase comes from, I can't pronounce his name, Thich Nhat Hanh, I think. Thich Nhat Hanh. Thich Nhat Hanh, that's it, yeah. So I've been watching him on YouTube and he's full of beautiful wisdom and it just, I think a lot of the Buddhist philosophy resonates with me and at the essence of this I was really just thinking about there's an experience of interbeing that is very different to a sort of intellectual understanding of it or the conceptual understanding that I think can be made available through this medium and through the sort of drawing out of scientific data to offer these experiences of connections that just kind of all came together as an underlying sort of intention for the project.
[00:17:38.775] Kent Bye: Yeah, well I guess to expand on that moment that you had from the 2D version of seeing the blood flows, as I see the fully formed version of Evolver here at the Tribeca Film Festival, and to be able to see the three-dimensional version of that, it was pretty mind-blowing to see that. It was like, oh my god, that's how the flow of the heart? I mean, I could sort of sense that that's what was happening, but it was almost like something from science fiction that was like so unlike anything else I had ever seen before. And I really like how you phrased that the Hubble telescope of looking inward at the human body. It seems like this would have already been common knowledge of everybody knowing this already. But also the thing you said earlier of how there's a lot of the different institutions that are working at different parts of the body and that in some ways this project is a big interdisciplinary collaboration amongst many different organizations to be able to come up with that full picture of the innards of the human body from a very fluid dynamics perspective because I was just thinking like mathematically the Navier-Stokes equation of like how liquids flow and how you're able to recreate that but also create an interactive fluid dynamics in a way that is also very difficult equations and just ways in which that it's not insignificant to be able to recreate a lot of it's probably a good reason why we haven't seen a lot of those types of immersive experiences that really have a lot of that fluid dynamics that are expressed in that way and I'm sure there's in a real-time environment a lot of simplification or artistic license to be able to actually make it work in a real-time environment But for the overall experience of recreating the human body, as I read through the credits, that's I guess another part of, as I look in the wall here and look at all the different people you collaborated with, or watch the credit sequence within the piece, it does seem like you were needing to pull in lots of different folks, not only from the scientist's perspective, but also other co-producers, and it's a big project to be able to pull something like this off, so When you think about the story of how it came together, you found the blood flow analysts at the Fraunhofer. How did it congeal after that in terms of all these other people, like these puzzle pieces that were putting each segment together to be able to create the whole picture of the innards of the human body?
[00:19:44.764] Barnaby Steel: Yeah, so Evolver is basically the latest in a series of projects that explore the human body. So the inception of it started with The Tides Within Us, which is another piece that we did, and explored that as a series of limited edition prints as well as video installations. And they're kind of stepping stones because working as an artist, you have to take opportunities to develop concepts forward. So, you know, you're looking for funding and sometimes the funding might be a little bit less and you might do a series of studies where you're looking at some aspect of the data, but like over that four year period, there was a point where Evolver landed on our plate and it came from Rene at Kaleidoscope. Rene was like, hey guys, you want to direct or be part of this amazing vision? And he had the most beautiful deck, been trying to get funding for quite a while, but it just hadn't quite clicked. And so Evolver as a project already existed and we offered the idea of saying, look, we're doing this human body stuff. We want to take one chapter of this project that was going to be five installations over multiple years, a sort of musical journey through someone's life. So we'll take the death sequence and we'll do death into life. And we talked about the human body stuff. And like, as the project progressed, it shifted. There was a lot of people involved. We were super honored to have Terrence Malick in from the start. You know, Rene had built an amazing team around it. And just, yeah, as the artwork started to just sort of find itself, it shifted and morphed and ended up being what it is now, which is a really kind of stripped back version from the starting point. Essentially, the concept is if I was to 3D scan you, so I get your entire inner branching being, scale you up to a hundred foot tall lie you on your back and I go right I want you completely invisible and I just want to see breath like what is the journey of breath you're a hundred foot tall so it's going to take like 15 minutes for a breath what would that look like and What would it be like to walk around with your mates as though you're walking around a forest and just explore, just see that journey of breath that's, you know, it's spiraling. Even when you breathe in, I think it's probably making a little vortex, like when you pull the plug out in the bathtub. So it's like a tornado forming as it just, you know, as it just kisses the ground. It's kind of like that spiraling in, rolls down the throat as it goes through your mouth. It's defining your teeth and your tongue and the inner shape. It's like a sort of mold of your mouth and it rolls down your throat. Actually, in the experience, you can see teeth and tongue if you're really careful, that the airflow, which is the only thing you can see, as it vortexes down, it's rolling through the mouth. So if you look, you can see a tongue shape. but then it's branching through the lungs and so you have this, you know, it's like an oak tree on its side, you can walk down through the flow and then as it diffuses into the blood it's a little bit like a kind of leaf landing on a fast-moving stream, it just gets whipped round and the blood out of your heart's moving at three feet a second, so it's a really high pressure, takes about a minute to do a lap of your whole body and about three seconds to get to your fingertip, so it's a We're under a surface, we're like a wild river. It's seriously intense. It's kind of like, as I was studying it, I was like, ah, that's why in Game of Thrones, when you get your throat cut and it splurts up so high, you realize that that main artery there is as thick as a wine bottle. It's incredible what's going on under the surface. So yeah, from the heart center, it's then going on a journey that eventually reveals the whole body and diffuses through tissue into every cell. And that's another angle on it. I was on the plane and I fell asleep in an awkward angle. My arm was completely numb. And as it was coming back to life, sort of feeling the flow back in and thinking, oh yeah, my cells are almost dead. They're almost dead. I can't feel them at all. There's nothing going on there. And without oxygen, without that delivery of oxygen, it's game over. And you need that constant flow. And so because it's infusing you to such a degree, the more I thought about that, this sort of portrait of breath through your body, it infuses every aspect of you. And so really that's what this piece is about, just exploring that. And when you get down to the cell at the end, the climax of the piece, it was almost like, who's breathing who? You know, the way the moon pulls the tides up a river estuary, you know, it's that... You know when I'm breathing in half the time, I'm not even thinking about it, but I'm breathing in to deliver that I'm a complex multicellular Organism and it's just I mean, it's I don't really have the answer It's just a really big complicated mystery and it's bonkers and yeah I think somewhere within that you recognize the full spectrum of everything you can sense is only a fragment of the complexity of your own body there's a myriad of different organisms and different sensory perspectives within you and And so that obviously doesn't end at you. You're like a cell within the body of the planet. And so that's the kind of fascinating thing to recognize that everything you could ever experience is just a fragment. It's a tiny little lens on who you are. And I think that's the kind of, yeah, it's a bit bonkers, isn't it?
[00:24:50.286] Kent Bye: Well, yeah, I think as you were sort of walking through those phases, you're describing with words what I'm seeing visually, but when I'm in the experience, it's so difficult to always put language on what I'm seeing, and so it does become this sense of awe and wonder because it's beyond anything else I've seen before, but it's also the most intimate aspect of being human is like this part of all these flows that are happening within our bodies. And I had a similar experience when I was watching both We Live in an Ocean of Air and Treehugger of watching how the water was going up the tree and the oxygen was coming out of the leaves. It's making things that are invisible visible and being able to see the interrelatedness of how the cycles or the boundaries between what we see as objects are not so firm. They're actually ways that we really get down into it there's very little of those boundaries and I feel like that was a part of the main message that I got out of this as well is that as you start to really analyze this and think about the boundary condition it becomes difficult to define because it is much more like these ecosystems that are these holes and these parts that are within each other. So I feel like that was, again, going back into these metaphors of biological organisms, what Arthur Koestler described as holons, that Ken Wilber has picked up, and something that is holes and parts. And philosophically, that's mereology, which is things that are individual entities within themselves, but also a part of something larger than itself. And so I feel like that's a theme that goes out throughout the course of your work, and also as I hear you talk about this piece. But the experience of that, I can describe it philosophically, but there's something that when i'm in the experience of it it turns into this poetry that feels like the spiritual transcendence that i watch it because it's putting a mathematical structure to the dimensions of human or the dimensions of reality that i have up to this point never seen expressed in that way and it provides this metaphor. It's like a philosophical provocation that's trying to challenge your ideas of what the self is or what the individual is. The more yang expression is the concretized self or like metaphorically the yang is the sun and the sun is putting a shadow on individuals during the day and you can really identify what it's an individual but at night with the moon it's more of a reflection of the sun but it's more non-differentiated so it's more about the interrelatedness of all those things and through more of a yin archetypal journey it's seeing how you as an individual are connected to something of a cosmic larger whole. That's the takeaway that I get from a piece like this is that it is amplifying a lot of those deeper philosophical things but the poetry is it's hard to describe in words it's almost like you have to go see it and experience it for yourself to leave it up to your own sensory perception to kind of make sense of it but that's at least my process and so that's Part of the reason why it's my favorite experience is because not only was it an amazing poetic experience, but it also is a lot of deep philosophical provocations that's really challenging a lot of the substance metaphysics ways of thinking about the separateness of ourselves and really emphasizing the interconnectedness. So anyway, that's... I wanted to elaborate on that a bit.
[00:27:51.625] Barnaby Steel: You're so articulate, dude. That was, that was grand. Oh man. I mean, you express it in a ways that I could never, I haven't got that jibber jabber. That was some quality jibber jabber. Bravo. Yeah. I mean, I think you captured it there. I mean, What can I say? I suppose there's this thing going on, you know, if I ask you, what does an apple taste like? You know, the richness of the sensation doesn't translate to language at all. And this is the beauty of virtual reality as a medium that What you're describing is really hard to translate into words and in some ways the beauty found in the data, in the observation, like the source data is so critical to the experience that you're describing because that's where the magic is. You know, if I try and imagine the blood flow through a heart, if I was to animate that with no reference point or maybe a bit of research, it just wouldn't have the same richness. It's so detailed, that fluid simulation that we've got and just the subtleties that come from attention to detail in the observation carry that beauty and resonance, something that's familiar and alien at the same time. So, yeah, I think it was so well articulated. You got it, but I'm chuffed that you said that because that's what I'm hoping people take from it.
[00:29:11.284] Kent Bye: Yeah, it's one of those things where it's like go see the experience and then like listen to us try to make sense of it because it's like, it's like, it's one of those. So I guess on that point, there's five people at a time, four to six people that are going through this at a time per hour. I don't know if you've increased that, but like, what's the plan to be able to get this out into the world with something that is such a powerful experience, but also from a throughput perspective can be very challenging in terms of giving people the full experience of that.
[00:29:38.723] Barnaby Steel: Yeah, this is the big one. This is the one. So for us, we realize that there's a strange in-between moment in terms of hardware. Backpack PCs are obsolete. Wi-Fi 6 is doing something, but you can't get mass audience. But the promise of 5G, and I mean, I don't want to get too much in the tech, but I think mass audience VR is very close now and not that virtual reality is the only thing that we're focused on but I think this project we plan to tour it for the next four maybe five years and that the next phase for us is to basically expand it into a really robust touring show that's like the whole flight cases thing the business model we've got some really good backers and it's got momentum so our plan is to tour it as a location based experience And then there's further plans beyond that, you know, discussions about how to bring it maybe to the home. Yeah, I think just more generally, we talk about David Attenborough as a teenager with a kind of black and white wind-up camera filming monkeys in a jungle going, you know, I can't wait to show mum, this is great, like, I can't wait to show people this thing. You know, we're like, it's early days, it's early days, we're going to be making this when we're old men. And so, you know, the mass audience thing will be solved. Eventually, the kind of concepts that we're playing with here will hopefully reach millions of people and hopefully like if it all works out for the sort of narratives we're exploring then you know it might be that kids you could take over the school gymnasium and the kids are able to see science not as separated into like physics and biology but as an interconnected ecosystem experience where all of these boxes and books and pages and chapters that the way that science is broken up and taught and all of the language is is something that is taken from the source which is the experience of reality and the observation and I think that there's a beauty and a wisdom and a resonance in the observation of it that gets lost in the language and the words and so that's kind of the dream for where we're headed and we're going to be doing it for a long, a long time I hope and so this is just a fragment of where we're headed I think, yeah.
[00:31:45.613] Kent Bye: So I saw it with four total people, so there was three other people, well there's one person that was sitting down and three other people, so I saw the other people as these white balls of light as the head and then two smaller hands and I found actually that the hand tracking was frustrating because my hands kept losing tracking and so I had to like keep my hands up, otherwise you would have the little blip of the white ball occlude my vision and so then I would have to raise my hand to restore tracking and then put my hands down so I found it difficult to watch it with my hands down by my waist because then my hands weren't being tracked and then it got confused and so I almost would have preferred to have hand track controllers so that I could have more of a resting position without having to worry about the hand tracking part losing tracking and so I can understand why hand tracking would be alluring for a mass audience to not have to worry about holding anything at all But one thing I did notice is that there was another person as I was going through it was moving and playing a little bit more erratically in a way. And so I felt like if I were to not be paying attention, get smacked in the face or something like that. So I'm wondering the decision to have lots of other people and what that meant for you to be able to have it as a shared experience rather than something that you're doing by yourself on top of the fact that it's one of the biggest shared VR spaces that I've been in in the context of an art piece. I've been in other location-based experiences from Zero Point VR in Las Vegas, where it's a big massive space, but first-person shooter. But this in terms of an artistic story that we're all in this massive space that you have here. So I'd love to hear a little bit more about the scale, but also what you were thinking about in terms of what the other people are adding to the experience of this shared experience of Evolver.
[00:33:26.445] Barnaby Steel: Yeah, so I think although the experience is almost finished, like there's going to be an extra phase of development and the avatar is basically, if you've got the Vicon tracking system or like an expensive piece of equipment like that, I mean there's lots of different tracking systems you can use but then you know exactly where the hands are, the shoulders, the feet. And so you can have a skeleton that's matched perfectly and makes people really visible. Obviously using the Quest with the inside out head tracking, when your hands are not visible they disappear. And certain restrictions for this, bringing it to Tribeca, it's quite a stretch to make these things happen. So I think the plan for the full touring show will be to have an external tracking system to solve that problem, basically. And then that gives people rock-solid avatars. And then, clearly, like, what you pointed at, which is clearly an issue at the moment, is that when you're not certain where another person is, you become nervous because you don't want to get hit in the face. And so it needs to be really clear that there's a person over there you can reach out and you can touch their hand. And so once you've got the confidence that the tracking's rock-solid, the experience is amazing and like on Ocean of Air we had the hand tracking working really well it was a different system but people held hands a lot they went on the journey together and you don't need that much in terms of visual cues to make that connection so I think for the next iteration which you'll get invited to for sure All of that stuff is going to be resolved, and I think that the shared experience is really beautiful. We've got the Viacom system back in London, and when that tracking's rock solid, then you can go and get someone to bring them over to the view you've got, and you grab their hand, and you can see it really clearly. You can look at the heart from down here, so it's a different thing. So technological limitations, budget restrictions, and it's very tricky to bring these things to a festival. It's such a stretch for us financially to make this thing happen, so that's my excuse.
[00:35:21.235] Kent Bye: I just saw The Infinite by Felix and Paul in collaboration with the Phi Gallery where one of the things that they have is you go through cohorts with 14 people at a time but up to like 168 people per hour so it's a massive throughput in large space but the International Space Station is shaped like an H and so they can kind of like rotate the H to be able to direct people into different segments of areas to not have as many collisions. But one of the things they did was have me and my partner have the same color of an orb so that I knew where she was at all times. I could always see her and then other people would only show up as the blue orbs when I was close to them. And so as a way to negotiate a shared experience within that so I feel like that was a really helpful to have a shared experience and if there's multiple people that are in it because this feels like an experience where in order to get large enough throughput you could add 10 people if you get all the tracking and all the different technology I mean I'm sure there's a hard limit at some point technologically a hard limit but also experientially a hard limit At some point, it's not going to be as enjoyable if there's too many people in the experience where it starts to feel crowded. So there's the trade-offs between needing to get enough throughput to be profitable versus not corrupting the experience in a way that takes away from the experience because it just feels like there's too many people in there.
[00:36:38.608] Barnaby Steel: Yeah, good old Felix and Paul. That's a great way of knowing who your friends are. Yeah, I think there's always these things to wrestle with to try and find the sort of optimal setup. If you think of this giant virtual human ecosystem, it can be adapted to the space, but the best expression of it is to have a hundred foot long, vast sort of warehouse space where you can walk from head to toe. In that way, it's kind of like this one-to-one relationship like exploring a forest and It's probably once a larger audience number puzzle is solved, maybe it's a 5G thing that's coming, then you can have a lot of people in the space and I think that's where this project is headed. It's also really interesting to explore other lenses on the essence of this experience. The beginning is a sound meditation, a sound bath. We had the great honor of having some music, Chaos Cave by John Hopkins, which is a beautiful piece from his Music for Psychedelic Therapy album. And so we got a new mix of that with voiceover by Cate Blanchett and poetry written by Daisy Lafarge. So we've been working with Daisy for a while as a way of appalling all of these interviews with scientists, all of this data, all of this research, through this beautiful human that can weave words in ways that's so beyond me. She came up with this line, where do you end and begin when sunlight is under your skin? I was like, dude, that's just magic. She's got the magic. So I think what she can do with language is capture. She really understands the science, but she's able to capture a texture with words that evokes the imagination in ways that is really interesting juxtaposed against the experience. So starting in a pitch black room, going through this kind of spatialized sound experience with poetry is a great way to vibrate you and unwind you and set an intention, a sort of openness, set a rhythm of breathing and that preparation is really important. You come in from the busy city and that element to it I think creates the opportunity for spaciousness or it clears your mind and creates an openness for the next part which is obviously quite a deep journey into the body. So I think as this piece expands we're going to be exploring a number of different light installations, video installations, to build it out into a much bigger experience.
[00:39:00.840] Kent Bye: Yeah, it was interesting to hear that it was a binaural recording. Was it like hemispheric synchronization to try to create brain resonance of any type of frequency? I know there's Holosync and then the Monroe Institute's Hemisync to create different levels of brain entrainment to facilitate different wavelengths and modes of being, I guess you could say. But was it a binaural recording that was trying to do that type of, or was it just a normal audio recording?
[00:39:26.816] Barnaby Steel: I don't know the full richness of it but for this Tribeca experience it's just a stereo mix and we work with this amazing composer James Bulley so he's basically my right-hand man on these projects not only in terms of all of the sound design and like spatialization of the music, but also conceptually we develop a lot of the ideas together. We've got a great ping-pong. With this piece, there's been a bit of spatialization that's been done by James and a balancing for the space on headphones, but headphones are absolutely not the best expression. You know, you want 32-channel speakers set up, you know, big subwoofers and to really explore the full power of the music. And in that kind of context, we haven't really gone into a deep dive with John yet, but hopefully there'll be an opportunity to really break that music up into its fragments and think about it as a spatial experience, you know, movement through sound. And there's another amazing collaborator, Catherine Tempah-Lewis, from Kinda Studios, and so she's got a wild brain. She knows a lot about everything, it seems. So we're talking to her about, you know, some of the healing qualities of sounds and different frequencies and tones, and so she's been doing a lot of research into that space. Yeah, so it's more than just music at that point. There's certain frequencies that have a certain sort of vibration to them that can actually put you into a flow state. And so it's kind of interesting to explore these things and see how that can be part of this transformative journey. We're kind of thinking about these shows as almost like rituals or sort of like a baptism, that you come in with a certain concept about who you are in relationship to the natural world around you. You come out slightly changed, kind of like, vibrate you into a softer edges and that involves a lot more than just VR and it's collaborations with all sorts of amazing people. I think that's the way we're headed to really expand these narratives and collaborate with lots of different artists.
[00:41:23.397] Kent Bye: Yeah, the whole onboarding and offboarding I think is a challenge to be able to set the context for people to come in and set the tone, but then as they come out as well, and I know that when I just saw The Infinite by Felix and Paul in collaboration with the FI Gallery where there was a whole streamlining of the onboarding, but the offboarding was interesting in the sense that you're walking through a number of different immersive art pieces. to help decompress and so I know that here you also have a kind of retrospective of motion prints of scenes from different experiences you have in these giant LCD screens in a vertical orientation set up like an art gallery so that after you go through the experience and coming out walking through all this different pieces of art so Yeah, just thinking about the whole journey. You're creating these immersive journeys for people and transformative technologies in some ways of trying to give these provocations that are inspiring the sense of awe and wonder, but hopefully, as people are leaving, coming up with a larger sense of themselves. I guess one quick follow-up, and you mentioned the Cate Blanchett, maybe just talk about that work with her in terms of process.
[00:42:25.773] Barnaby Steel: So well because we were already working with Ed Pressman and Terence Malick Which was just great because you know two wise gentlemen who? We would share work in progress and have the occasional call, and I think there was just some really helpful steers that I You know, like, you've got an idea and you're moving in a certain direction, but sometimes that outside perspective from someone that's thought so much about cinema and the relationship of music and creating these epic, awe-inspiring scenes. I think Terrence Malick and awe, you know, he's like an awe master. So, you know, some of that juice is definitely in this piece, and I feel, like, really grateful for being able to have that relationship. And then, And then, yeah, we were talking about the poetry, and he suggested Cate Blanchett. We were like, yeah, that would be amazing. And so we recorded her in London, but she was filming. So we didn't have a huge amount of time, but there was the themes. And actually, the poetry in itself really resonated with her. So she's come on board as an exec producer, and will be supporting the project into the future, and hopefully into future iterations of it as the ideas expand. But yeah, she's just very down-to-earth and it was nice. Yeah, the whole thing's been quite mind-blowing because we don't work with Hollywood people at all. It was really Rene that started this thing and then suddenly the knock-on effect of having these big names associated with the project has elevated it in ways that we might not have planned that. So it was just a really happy accident and yeah, I feel quite lucky just to have landed on our plate like that, yeah.
[00:44:01.745] Kent Bye: Yeah, well I know that Rene Pannell started KaleidoscopeVR and then eventually did artist funding and then started a spin-off of Artisan. Now KaleidoscopeVR is actually kind of shut down in some ways. So I know there was an event that I went to that may have been a KaleidoscopeVR or Artisan. It was like at the Vulcan Center at Paul Allen Institute in Seattle that you presented at in 2020. Had Rene already come on to the project by that point?
[00:44:25.467] Barnaby Steel: Yeah, so actually that was the point where, so René already had the project, we came on, we developed the treatment that is basically very close to what we're showing here, and so at Vulkan that was our first sharing of the vision, and we actually got some funding there that helped us create a 360 video, so actually we did it all in V4, which is the software we used to do all this sort of fluid particle winker-tinkering, but we had a bit of seed funding there to do a first iteration, and we took that to Cannes as a 360 video. So from the real-time software you can kind of export 360. And that was kind of like a proof of concept and then from Cannes we managed to get funding by Nicole Shanahan and so she came on board and has been like super generous and a great supporter of the project and she gave us the funds to make this piece which is It's pretty much finished now. The VR piece is finished. The next stage for us is planning the world tour. So yeah, kind of like stars aligning. Because after such a long time of holding a vision in your head, because it has been quite a long time, I haven't quite digested it because it's been such a long journey. and I'm pretty knackered. Like last night was we reached the peak and it was finished literally just before the first audience came in. It's super dodgy to do that. It should have been done a week before, but we're like just tweaking all the settings and I think everyone was striving for perfection. You never get there. And it was like the clock, it's like the guests are coming in, you've got to stop. But it literally was like... that's it and yeah so it's kind of just I haven't really experienced it properly yet you know I need to go in and just be like what's going on it's a crazy process actually because you're creating a virtual world with all these different parameters it actually ends up in a unity timeline so all of these parameters from v4 actually vl is the version that we use but all these parameters get exposed in a unity timeline so once you've got everything there it's like you can control the whole world like particle size, colors, what if the velocity is affecting the color just a little bit that suddenly enhances a rhythm within the movement that you wouldn't have been sensitive to so there's all these subtle details they can only happen right at the end so that's why in these last few weeks you're kind of tweaking the look in the entire lead-up to that you're basically building a piece of software And so I don't have that much to do. I'm just like sitting on my hands waiting. But it's always the end where you really get to kind of sculpt it. Yeah, it's pretty fun.
[00:46:58.950] Kent Bye: Well definitely the film festivals are a forcing function that sets a deadline for artists to be able to actually deliver things and so I've definitely noticed that you're not alone in working on it to the 11th hour and then sometimes even changing it even throughout the course of the festival. So yeah, that's part of the reason why I enjoy so much covering the festivals because you really see the bleeding edge of bleeding edge of creation of whatever's being created in that moment. Yeah, well, it's an amazing experience. Congratulations on finishing it. I know when I saw you in 2020 in Seattle give the pitch for this as a project that you had already been working on it for a couple of years. And I think you even mentioned to me it takes so long just to make these things happen. So you're at the metaphoric finish line now. So congratulations on the piece. And one other sort of logistical question on the production, because I know Atlas V, Atlas 5 has been working on it as well. And I don't know if you want to mention any of the other collaborators on the project as well.
[00:47:52.116] Barnaby Steel: Yeah, dude. Yeah, so I haven't mentioned the gang so Antoine was our right-hand man, and he was sort of instrumental working alongside Sam Pressman and so those guys were doing a lot of work in terms of fundraising and the kind of exec producer roles alongside like Nell and the whole gang, the Marshmallow Laserfeast gang. So, I mean, outside of that, there's Jennifer Garrison from the Buck Institute, and she was like a sort of lead science advisor. I mean, when you start going into the human cell, it's like a minefield of terminology that you never heard before. It's quite an alien landscape. Also like a lot of the work was built in Houdini so like our workflow went through Houdini to kind of tidy things up and this wonderful animator Louis Saunders has just been crafting away he's got all these anatomy books out and I'm like wait how does that bit connect to that bit? I'm just like trying to work it all out looking at the shape of the heart because you have to get the flow right. I'm like wait a second It's going the wrong way. The blood flow's got to connect that pipe down there. It's really like the plumbing. So even though the source data's there, it all has to be sort of like tickled into shape. And yeah, like I said, Sam Pressman, he's such a character. He's just been epic on the project. And especially now it's done, he's come into his own in terms of relationships and I think hopefully the project's going to resonate and there'll be these opportunities to get the tour together and we've got a great gang pushing it so yeah, fingers crossed on all of that. I think also just like big thanks to Nicole Shanahan who's funded it and The Allen Institute, we've got this amazing cell data, and Fraunhofer Institute for the blood flow data, and just like everyone at the Marshmello gang, we're pruned. All of our juice has gone into it. We've been working so many weekends, especially in the last month or two, so just big love to everyone involved in it. I probably missed some really important people. My brain's frazzled. I'm really sorry if I did, but yeah. It's been a big passion project for everyone and we're really proud, I think. Seeing Louis Schwartzberg from Fantastic Fungi shed a tear today, that was the moment when I was like, yes! He came out and was speechless and had a little tear rolling down his cheek and that was like, yeah, I made Louis cry! That was a real moment for me, like, okay, it's working. Did you cry?
[00:50:22.568] Kent Bye: No, well, one of the things that happened in my experience was that the first five minutes didn't show the visuals, and so I missed, and I had to watch it again at the end. So I missed something at the beginning, and then I had to re-watch it, and I was like, oh, okay, so I was somewhat distracted, feeling like I missed a part. But I was able to see it, but I had to, like Pulp Fiction, take one part and put it at the beginning and, like, project out. But aside from that, I really was moved by the piece overall and was really in a state of awe, really. One of the producers told me afterwards was that, she said, oh yeah, that's my body in there. I was in a fMRI for 16 hours actually measuring. So the degree to which that you're actually measuring and capturing new information to be able to put into this. So yeah, just the whole scope and scale of all is just really amazing.
[00:51:10.180] Barnaby Steel: Yeah, Emma Hamilton. I can't believe I didn't mention her. So she's been, we've done this as a total tag team. And yeah, so she went to the Fraunhofer Institute and spent 16 hours in an fMRI scanner. And you've got to be really still, you have to do it again. If you move a little bit, the data would get messy. So it's kind of like a long exposure photo. But yeah, fuller. She did the scan, and now it's quite weird that everyone's walking around her body. In fact, when you come out of the head, even though the shape of the head is defined by blood vessels, you can kind of recognize her face. You can just say, it's Emma. Yeah, it's just very bizarre. Yeah.
[00:51:48.659] Kent Bye: Awesome. Well, finally, what do you think is the ultimate potential of virtual reality and immersive storytelling? And what am I be able to enable?
[00:51:59.616] Barnaby Steel: I suppose in a time where we need to, I guess, we need a new story. I've been talking earlier about this story of interbeing and I think that there's a potential for transformative experiences that can take you beyond the limits of your perception and reveal connections and relationships. And so I honestly think that there's kind of like scientific hallucinations, that there's something in this that can create experiences that then will shape the way you think about yourself in relationship to nature. And that's, I think it's really, the possibilities for transformative perspective shifts, I think, are really powerful and that we're just at the beginning of that. And yeah, it's just kind of lucky to be able to explore these themes. And yeah, I think that's where it's headed for our little niche, our little niche in VR.
[00:52:48.294] Kent Bye: Yeah. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:52:57.269] Barnaby Steel: Yeah, I guess. I'm not even sure. I'm pooped. I'm totally pooped. I'm out of juice, Kent. I'm completely spent. I'm actually going to a remote island in Italy on Monday. I need to massage my wife's feet and say sorry for all the long hours I've been doing, spend some time with the family. Yeah, I'm going to have a bit of time off now, so yeah. I don't know, take a holiday. That's the thing, when you're working really hard, you have to book the holiday in advance so that you know that it's coming. And then pamper the wife, so that's my plan now.
[00:53:33.148] Kent Bye: Awesome. Well, congratulations again on finishing this piece, and I really, really enjoyed it. And yeah, I look forward to getting out and having as many people as possible to come see it, because I do think it'll have the potential to open up their minds as to what's possible with not only the medium of VR, but also to, like I said, become into more of a relational orientation and a spatial metaphor to kind of understand some of those different ideas and concepts. You know, who knows how it'll continue to unwind and get unpacked by each person that sees it. But I can only speak from my own experience of just having this deep sense of awe and wonder of experiencing it. And yeah, this transcendent meditative poetry that you're able to achieve with the piece. So, so awesome. Well, thanks again for joining me on the podcast and helping unpack it all.
[00:54:17.064] Barnaby Steel: Thanks, Kent. Always a pleasure. You articulate rascal.
[00:54:23.062] Kent Bye: So that was Barnaby Steele, one of the co-founders and directors of Marshmallow Laser Feast. So I remember different takeaways about this interview is that first of all, well, I really, really, really loved this piece. It was my favorite piece of the festival, not only because it gave me this sense of wonder and awe and really taking me into another place, but I also think that it's using the structures and forms of immersive storytelling and the medium of VR to look at these deeper aspects of the interrelated nature of reality, but also the process relational view of seeing how these blood flows are flowing through our body. And yeah, just to be able to step inside of what Barnaby said is that there's the Hubble's telescope that's looking out into the universe, but there isn't really one entity that's looking at the Hubble telescope or the Hubble microscope looking inward into what's happening inside of our body. There's lots of different organizations and scientists that are looking at different bits and pieces, but there's something about the immersive art that's able to tie it all together. It's a lot about just starting with the data visualization of what is the actual data and then just presenting it in a way that you're able to just experience it in an embodied fashion. I mean, I can tell you that you could imagine what do you think the blood flow through the human heart looks like if you were to just take away all the different structures of the heart and just look at the fluid dynamics of the whole thing. You can look at some of the videos that Barnaby said from the Fraunhofer Institute, you know, you can see those videos, but even when you see the 2D, it's like taking these slices. They're doing fRMRI, and when they do that, you're able to piece together all these 2D slices and then start to come up with these different vectors. But when you're actually immersed into a space and you get to see a spatialized version, it's pretty hypnotic and pretty amazing that this is just not more well-known, the structures and forms of what blood flow looks like as you walk through the human body. Yeah, I really love a lot of the different pieces. Marshmallow Lizard Feast has done over the years, Through the Eyes of the Animal, Sweet Dreams, you have Tree Hugger, then We Live in an Ocean and Air. Each of these are looking at things that are invisible and making them visible. And then as you make those things that are invisible visible, you start to see the permeability of how the boundaries of what is yourself and what is the outside world. And for me, the philosophical concept is myriology, where you're both a whole and a part. And the boundary conditions between the whole and a part ends up being a lot more fluid. Sometimes it's a whole entity within itself, but sometimes it's a part of a larger whole. So that's these fractally nested holes and parts that go all the way down in terms of these organisms within organisms within organisms. So I think the medium of VR is actually able to just give you an embodied experience of what that's like. I mean, I could speak about it philosophically, but to actually step into it and experience it as a whole other experience. If you get a chance to see Evolver, definitely go check it out. It's going to continue to evolve and hopefully do other tracking so you don't have the dropping out of the hands and stuff like that where it gets a little bit frustrating for having to hold up your hands at your head level so that it's not blipping in and out. Yeah, just have a little bit more consistency there and also assuredness of where other people are in the space as well. But for all the different constraints that they had, really amazing that they're able to do that. I guess one other thought about this piece is that they were starting to run into some limitations of Wi-Fi 6. We didn't get into it a little bit. I had a little bit of discussions with some of the other co-creators in terms of like just being in New York City and doing a project like this and the different interference that you have. And potentially this could be a use case where you start to go into cloud rendered content that's being transmitted through 5G and you're able to have a little bit higher scalability. Imagine a hundred foot long And so maybe that's something where the 5G is going to be able to come in and start to be a use case that starts to show the power and potential of having these shared social experiences that are delivering to you a high bandwidth, low latency types of experiences where you're able to walk around a space, but also be able to interact with other people. And so I think that's something that's going to be a big part of the future of 5G. I think that's something that's going to be a big part of the future of 5G. start to interact a little bit. I mean, I mentioned the infinite and Felix and Paul, which by the way, they're still in Seattle, they've been extended into September. So if you're in the Seattle Tacoma area, definitely go check out the infinite by Felix and Paul and my previous conversation. But in that piece, they don't have any way in which that you're actually interacting with the world around you. And this piece to have a little bit more interactivity where you can, you know, start to play with a little bit of fluid dynamics and the fluid dynamics are reacting to different ways in which you're playing with it, which is also really quite compelling as well. So Anyway, definitely go check out Evolver if you get a chance as it starts to get out into the world and just to support their project and just to see it for yourself. So that's all I have for today and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a list to support a podcast and I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
