One of the most visceral experiences I had at Sundance New Frontier this year was Marshmallow Laser Feast’s Sweet Dreams, which incorporated taste as part of the narrative in a surrealist virtual reality story world called Luscious Delicious Land. Marshmallow Laser Feast is known for exploring the multimodal nature of the sensory experience with exploring the perceptual perception of insects with In the Eyes of an Animal and exploring the use of smells in Tree Hugger.
I had a chance to talk with director & Marshmallow Laser Feast co-founder Robin McNicholas and producer Nell Whitley right after experiencing Sweet Dreams. It was absolutely fascinating to see what aspects of the experience stuck with me, and what I remembered or didn’t remember. It turns out that my lack of memories for aspects of the experience mirrored what they were finding in others, and just how mysteriously powerful taste can be within immersive experiences. In debriefing my experience afterwards, I was able to recall specific memories associated with taste in the process of conversation about it, but during the actually experience it was almost as if the creators were starting to paint with an invisible palette of a matrix of unconscious memories and associations that I had with these tastes. What that looks like and how to best do that in the service of a narrative goal turns out to be a VERY hard problem as we explore in this conversation.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So one of the most visceral experiences that I had at Sundance this year was Sweet Dreams, which was incorporating taste within their story world that they've created called Luscious Delicious Land, where you're introduced into this world with a couple of immersive theater actors, where they're establishing the rules and setting the context because they have to really build trust with you because you're basically going to be unveiling yourself and putting on this virtual blindfold and then they're asking you to go into these virtual experiences and to experience these different tastes that are somehow woven into this deeper narrative that they're trying to get at. So it was a pretty wild and crazy experience and actually probably one of the hardest experiences to get in at Sundance New Frontier, just because it took probably around like 20 to 30 minutes per person and they could do two at a time. So there wasn't a lot of people who were actually able to get through and experience this. So I just feel lucky to have actually gotten in and to experience it. I had a chance to talk to a couple of the creators, Nell Whitley, who's the producer of Sweet Dreams, as well as Robin McNicholas, who was director, as well as the co-founder of Marshmallow Laser Feast. So Marshmallow Laser Feast, they're one of my favorite VR production houses, just because some of the experiences that they've done through the eyes of an animal, you're able to do this visual depiction of what the world will look like through a number of different insects, which was this amazing sort of point cloud visual artistic depiction of trying to give you this different sensor experience of what it would be like to be an insect. And then Treehugger, they were able to do this whole LiDAR scan of a giant redwood tree and to show the visual depiction of the water cycle and the production of air and really visually depict this deeper aspects of nature to really give you this embodied experience of that. And then their latest experience that they have is in a gallery called We Live in an Ocean of Air. And I haven't had a chance to see that yet, but hopefully, It'll still be running when I go over to the United Kingdom at the beginning of March to give a lecture at the Architecture Association, and hopefully I'll be able to catch it there. But they've been really exploring the sensory experiences. They had smells that were included within their tree hugger experience. And in this, they're really doing this pioneering exploration of how to integrate taste within a virtual experience. And so lots of very interesting insights and unpacking of this experience, and For me, this is really trailblazing some experiences that I've never had before in any sort of immersive experience. And so, yeah, I'm excited to kind of unpack it a little bit more. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Nell and Robin happened on Saturday, January 26th, 2019 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:02:53.525] Nell Whitley: Hi, I'm Nell Whitley. I'm the producer of Sweet Dreams with Marshmallow Laser Feast.
[00:02:58.973] Robin McNicholas: I'm Robin McNicholas, I'm the co-founder and director of Marshmallow Laser Feast and here we are at Sundance with Sweet Dreams.
[00:03:08.057] Kent Bye: Great, so maybe you could just describe for the audience like what Sweet Dreams is. I had a chance to see it but I just want to hear from you in terms of how you talk about it.
[00:03:16.803] Robin McNicholas: Well, it's a taster of location-based experience. We've been focused on the multi-sensory and very keen to embrace and explore the potential of untethered VR. And in this case, in continuity with other projects where we've involved different sensory elements, both on a technical and creative standpoint, smell and sight and sound and so on, we've included food and drink in this. And yeah, it's been an extraordinary journey.
[00:03:53.112] Nell Whitley: Yeah, so what we've done is we've created a place called Luscious Delicious Land. This is the first time that Marshmallow Laser Feast has worked with narrative in any kind of meaningful way. So we've spent a long time writing about a world, it's about the land of plenty, and we invite you, the audience, to come on a journey through that world. It's very much our ambition to create this as a feature-length piece, much more around looking at more like an immersive theatre model, where we bring in kind of multi-sensory immersive technologies into theatre. and the plan is to have this as a shared experience with multiple people, kind of the future of dining perhaps.
[00:04:31.135] Kent Bye: Yeah, and the thing that comes to mind is the immersive theater piece, Then She Fell, because in that piece, they are giving you shot glasses of different flavors of alcohol throughout the entirety of that experience. And so that was the first real immersive experience that I had that was really trying to blend in your taste and your sensory experience into some sort of immersive entertainment experience. I mean, there's obviously lots of other people that have been doing this for a long, long time in terms of like food tours and other stuff like that. But in terms of trying to blend narrative and story mixed with taste, then something about the body that's so visceral that you could start to tap into different aspects of either people's memories or aspects of their embodiment. So I'm just curious how, as you start to think about integrating taste into the sensory experience of a VR experience, like how do you start to think about how you start to add things in to either amplify or create contrast within the context of a narrative?
[00:05:24.759] Robin McNicholas: Yeah, it's a tricky one because obviously when you have a visitor and you're essentially blindfolding them with a HMD, there's serious intimacy considerations. We want people to feel safe, confident and also we need to keep a level of exploration and curiosity. where we're literally putting things in people's mouths, blindfolded, and people are walking around. There's a whole contract of trust that we've very carefully designed to essentially front-load the experience with lots of hand-holding and onboarding that is sensitive, that is kind of all wrapped up in the narrative and framing of the story world. But yeah, it's fascinating. The first few attempts, we completely failed. And so to see it operate here in a live environment with a whole range of people, some completely fresh to VR, some who are complete experts that we look up to and admire. And as a result, there's a fine balance that requires a level of flexibility in the work, a plasticity. and it's curious because you learn on the job and you learn through the process and I think that that is one of the real-time expressions of what separates this scene of new forms to things that are locked like film, for example, in an interactive way, you know.
[00:07:01.505] Nell Whitley: Yeah, I don't know if I have much to add apart from I think it's just a really fascinating journey. We often ask the question, why are we doing this? Why are we using virtual reality or why are we using this other technology before we choose to go on that endeavour with it? And I think for us, It's a really interesting iterative process to work out, is bringing taste into this valid? Does it make sense in terms of the story? Are we doing it for a gimmick or are we doing it because it's central? And the whole point of this story was to make it about food, about gluttony, about desire. That's very much the central themes of the work and therefore the food and drink needed to be completely embedded within telling that story. And I think we're getting somewhere. I think it's been an interesting journey to work it out.
[00:07:44.056] Robin McNicholas: I feel it's quite important to differentiate where we started, which was more of a novel diner. We were thinking, essentially, we've been dabbling with VR for a long time and wafting our hands around. I've always been frustrated with that. I've always been frustrated without any kind of tactility. It's getting better. It's by no means sorted, but we've embraced it and we want to bring a tactility and blur that line between the virtual and the real in a way that helps immersion, but also there was a junction where it's like, wow, this isn't just going to be a VR feast. Actually, frankly, what we realized was we love eating in the real world. So what do we do? Do we just make glowing novel objects that light up and animate and provide unfamiliar, otherworldly elements, or do we, we zoomed out and realized, no, the first thing we need is to have a solid story in place, and that's where The Junction took us away from just novel dining with glowing food to one where there were more sensory pairings that served as narrative conventions that we were able to dial in with the careful collaboration with The writer and super analogue, pen and ink, Simon Rowe, the author, he's a food author and completely oblivious of all things tech, which is a welcome and wonderful thing to have in the creative process. And as well as that, the BFI, the British Film Institute, who are so story-centric and very familiar with pulling out their machete and just chopping things down and being constructive with their criticism in ways where they're learning on the job, in terms of new forms, we're learning. in parallel with their pedigree in terms of knowledge of storytelling and it's fascinating. It's a fascinating dance between, it's like a 30-legged race where everyone's feet are tied together and we all have to walk in line.
[00:09:57.835] Kent Bye: Yeah, one of the things that comes to mind is how when you see a movie, you go in and you go through a ritual in order to start to watch the movie. You go into a movie theater, you start to see the lights go down, and then all the things that they're doing is slowly preparing you to be ready for story. And I feel like there's an equivalent ritual that is trying to be developed in the context of virtual reality immersive theater type of experiences where you're coming from the mundane world you enter into like a learning of what the rules are into this magic circle that you're about to step into and then you step into it and I feel like that transitional liminal space of how you go from the mundane into the magical It feels like there's a process by which we can look to immersive theater, we can look to experiential advertising, we can look to how people are starting to do this, but this is something that it seems like you've done a lot of thinking about.
[00:10:46.351] Robin McNicholas: Well, absolutely. And you've hit the nail on the head. In fact, the pageantry that surrounds cinema going, you know, going to the movies and that contract. I've always regarded the tearing of a ticket with a human being as a kind of symbolic, like, OK, this tearing of the ticket represents the line between the imaginary world that you're about to enter and the world behind you. And so it's great to hear that your thoughts in line with ours to that extent.
[00:11:16.942] Nell Whitley: For me this is part of these technologies being evolutions of art forms, not new by themselves. It's not like VR is, oh VR is not like that's our paintbrush or our kind of new art form, it is a tool in which we are embedding within existing art forms which have got long histories and they've been perfected over years and people have tried them out in different ways and are evolving them slowly and so therefore the pageantries have really important to honor and to not think that just because you're using a new technology that the way you treat your audience should be new in some way or disregarded because it's all about the technology and so I think often you know as long as asking why are we using this technology we have to say in what context are we using this and what is that audience journey from the kind of moment that you're in knocking at the door to go in right through to coming out the other side and we do a disrespect to our colleagues in theatre, our colleagues in film and other art forms if we don't really respect what we're trying to evolve, if that makes sense.
[00:12:15.990] Kent Bye: Yeah, yeah, for sure. One of the memories that I have from covering the virtual reality space is I went to TechCrunch Disrupt and there was a Westworld travel agency booth that was there and I hadn't heard of that at all. And so I walk up to them and said, do you have a VR experience in here? And they said, yes, we do. I was like, I would like to see it. And then I just thought that they were a travel agency. And so they put me on a list, a wait list, and then I actually got in. And then they had me sign this waiver, like, oh, you're about to see some nudity. And I was like, that's weird. Why would a travel agency be showing me nudity? And then, like, I went through the experience, and it was like this whole, like, CGI process of selecting a hat and selecting a gun, a lot of things you see in the Westworld show. But I was like, what is going on here? And then I went through the experience, and I was like, what is this? What is going on? What did I just see? And then they told me what the show was about, and then I was like, ah, I just had a direct embodied experience of what the Westworld experience is like. It was an experiential marketing campaign where there was a failed intention behind what they were on the outside saying they were, and then you find out something that's different. And so it feels like they were embedded in the context of a tech conference that I was covering as a tech journalist, covering VR. And it just so happened they had a VR experience that I had this whole thing, but it felt like either an experience like this is going to be unveiled in some sort of like surprise pop-up type of delightful, completely blow people out of the water like they didn't even expect to even have. anything close to that or it's some sort of experiential dimension that they know they're going to go to some sort of location-based virtual reality entertainment to have a Marshmallow Laser Feast experience. And so it seems to be like as these types of experiences go outward it's like how do you at the same time hold different things in in terms of what the experience actually is but give enough of an incentive to go be exploratory to be able to even see something like this. And so that seems like a tricky balance.
[00:14:14.374] Robin McNicholas: It is a tricky balance and it's fascinating seeing the balance between, especially in the advertising sector, where there are budgets to explore these in perhaps different ways to freer situations such as where we are with Sweet Dreams. But I think all of it exposes a picture emerging where audiences are certainly keen to have new experiences and there's a lot of influencing factors, right? You know, people can see in terms of media saturation at home, they can see and hear music and film in a land of abundance that we live in, in terms of that kind of level of saturation. And so the LBE, the location-based experience, from our standpoint, provides something new, a new experience, a new memory for people to walk away with. And in parallel, there's all these extraordinary developments in the technological realm and through education, where our team is made up of There's actors, there's designers, there's architects, there's technologists, unity developers, traditional CGI artists and it's a huge collaborative venture that requires lots of parallel development and trust on all parts including curators and visitors as well to the experience so it exposes this almost crystallization of a new form which we hope at least the LBE will be something compelling and familiar to more people and will draw audiences in to engage in story worlds in a full-fat circus way but through 5G and various emerging other at home and mobile devices that allow for windows into the world and weirdly serve as convenient advertising models to draw people into the larger LBE, the full fat experience. So it's curious. What I love is there's no experts and the scene, especially at places like New Frontier, is incredibly thrilling and part of the reason why we're involved in this scene is that The multitude of creative exploration and tackling all these technical problems and, you know, there's easier ways to make a living, right? Definitely. But there's something about the curiosity and that drum that keeps banging out that just, I don't know, it's really infectious and we've got the bug big time.
[00:17:01.681] Nell Whitley: I think for me, in terms of audience, at least from my perspective within Marshmallow Laser Feast, we're always working out how being an audience member is as good as being a creator. And I think this is something I know our guys have talked about a lot in the past, which is, especially in the early days of making this immersive content, it's definitely more fun to make than it is to watch in terms of distribution, how you get it out into the world, how the technology works, the hardware problems. It was so much more fun as a creator to kind of delve into these worlds and build them out. Getting it out to your audience has been a much bigger challenge and as an audience member having a quality experience has been harder in all contexts. But I think that's shifting a bit. I think over the last couple of years, things have evolved. This notion of a location-based experience or immersive experience is increasingly being refined. Obviously, void, dreamscape, there's people doing it more and more often. And I think from our perspective as a studio, we're trying to say, you know, for example, our show in London at the moment, We Live in an Ocean of Air, is very much our first kind of out there into the public realm. Like, this is going to be a very high quality worth my money experience for that mass public and I think increasingly we're trying to do that that's what the ambition is with Sweet Dreams as well it's not just to think about this from a how fun is it to create but it's how fun is it to be part of this and being an audience member from that and ultimately that's what I think us as creators have to do in order for this distribution models for them to get nailed for us to really understand the value in what we're creating.
[00:18:28.675] Kent Bye: Yeah, I remember talking to Barnaby Steele at the Tribeca Film Festival a couple years ago and at the time he personally was way more interested in just pushing for the most emerging technology at that tree hugger experience you had like smells that were being integrated and so there was like a You know, in order to be on the bleeding edge, sometimes you have to be a little bit more willing to use these duct tape prototypes that aren't ready for primetime and to be able to have like a throughput of hundreds and hundreds of people see it day after day. You know, it's okay to hold the hand of something like that in the context of a film festival experience where you really can be on the bleeding edge of what the technology is, but in order to actually sustain that in the context of a location-based entertainment experience, you have to either still be on the bleeding edge, but find ways to find more consumer off-the-shelf products that could be swapped out, not so that you're handcrafting and building everything yourself, but be able to have an ecosystem of hardware and other resources to allow you to do that. And it seems like the HP backpacks are a part of that as well, enabling people to not have to worry about how you actually get a computer onto your back as you're wanting to be mobile, but More and more that there's going to be a consumerization of some of these products that maybe if the location-based entertainment market takes off enough that there'll be more of an ecosystem for that outlet for a product that may be very expensive for one LBE to buy but way out of the price range of an individual but still allows the evolution and the growth to happen within the industry.
[00:19:54.437] Robin McNicholas: I find that really interesting and one thought that cropped up was where in terms of locations these take place. Is it always going to be in derelict buildings? Do we integrate into theatres? Do we go into cinemas? or galleries as well. And what's interesting is there's places popping up that can serve for LBEs, such as the Phi Centre in Montreal, for example. There's the Shed in New York and the Factory in Manchester in the UK, where we're starting to realise, oh right, there's these kind of media arts spaces where we can actually rely on the local tech team that effectively works in our favor in terms of the economics. If we don't have to fly and accommodate people and we can actually just send the packaged LBE as a turnkey system, then that's going to be great for the industry. It's going to really give buoyancy, but it requires a global network. It can't just be these one-offs. It needs these centres, either traditional theatres that are adapting to the times, or these venues that are just incredibly well equipped, that have the flexibility to accommodate all the different types of work. Because of course, you know, we bring a processional VR experience with food and drink. It's not necessarily going to be the same as an LBE that is focused on bringing animals into the space which we should do.
[00:21:32.322] Nell Whitley: The infrastructure is the same. I think it's quite important to differentiate between what we're talking about in terms the arts and culture sector and advertising and experiences happening in pop-up spaces in other real estate areas whether it be malls or you know new spaces opening in you know towns where there's kind of maybe what would have been retail is now pop-up experience space. I think MLF hasn't really done what, for example, Meow Wolf are doing in Las Vegas or 2-Bit Circus have done in LA, which we really respect and use that as a reference quite a lot. We come from arts and culture in terms of our backgrounds and we're very much interested in leveraging that industry more, so hence our piece being in the Saatchi Gallery. hence us looking at theatre partners. We just won a big research and development grant with the Royal Shakespeare Company and Manchester International Festival in the UK and Punchdrunk Theatre. We're looking at together as organisations on how we push those forms in that sector and bring that audience into what we're doing and I think that there's absolutely similarities, there's absolutely resources that will get used across all of those places but it's perhaps a nuance in terms of that different venues approach and evolve their resources to match what we're trying to do.
[00:22:41.057] Kent Bye: Yeah, and just in terms of integrating a taste, some of the thoughts that I have in terms of where it's at now and where it's going to go is that there's going to be perhaps some approach of trying to translate taste and flavors into temperament, into qualities of an experience, and then to match up that quality into different dimensions of overall experience to try to use that either for contrast or for narrative tension. So thinking about how either sound and visuals and how that works within the visual storytelling language of film, how you're able to create the consonants and dissonance cycles to build and release tension. Narratively, I think there is going to be an equivalent building and releasing of embodied tension within the context of an immersive VR piece in that one of the big open questions is what is the underlying philosophical framework that you're going to use in order to categorize the different qualities of these different tastes and how they fit into the context of everything else that's happening within that experience. And so that's some of the thoughts that come up as I start to see this is that I see like traditions like herbalism or Ayurveda or these traditions that are trying to take tastes and flavors and match them to different temperamental qualities and then from there see how they're going to fit into the larger ecosystem of a story that's being told. So I see that's like in the long run where things are going to go and that just to start to see how there's the integration of the taste and the level of trust that you need to do that There's a lot of issues that I think have to be worked out to even see what you can start to do with it. But I feel like this is a great first exploration for how taste and flavor fits into the overall context of a narrative experience.
[00:24:16.508] Robin McNicholas: Yeah, it's been fascinating and of course this kind of cultural differences in terms of the nostalgic Proustian cues, you know, those that grew up in the 80s in the UK might have a very different, they do have a different palette to those growing up in millennial New York. There's different flavours, there's all of those considerations of How do we embed this and detach ourselves from flavor that is personalized? Because all flavor is the makeup of our palate and associative way in which we engage with taste is so personal and so subjective to the individual. How do we white room it? You know, there's these white room flavors.
[00:25:04.487] Nell Whitley: Should we even do that? Because I think, you know, often we're kind of thinking that's not the right thing to do at all.
[00:25:10.972] Robin McNicholas: Oh, certainly. Just dealing with taste alone, it has been fascinating because it's not only the flavours, the scent of course is one of the major parts of it, but also the texture, the way in which it interacts within the mouth. as well as the temperature of it as well and all that wonderful creative opportunity in terms of cognitive dissonance you know if you see something that looks searingly hot and you hear it sizzle and then experience it as freezing cold do we trick the brain into perceiving it as searing hot you know and there's all that playful terrain, fresh snow to run out in and dial into these experiences and I've got to say it's just so much fun sitting down from a creative standpoint and everyone eats and everyone can enjoy coming up with these peculiar dishes and it's something that is a leveller, you know, it's so much fun to sit back and go It's the same when we used to work with drones a lot, still do occasionally, but it's like when they first came on the scene, it's like, my God, you know, the things we could do with this. Drone hairdressing and elaborate usage and reappropriation of this kind of tech. And that's exactly what we're doing here. on a human level and is really educational for us and observing how audiences react to the flavours that we've been lucky to develop here has been just completely fascinating.
[00:26:54.064] Kent Bye: Well, one of the things that I've noticed about Marshmallow Lizard Feast is that a lot of the pieces like Through the Eyes of the Animal and We Live in an Ocean of Air is that all of these are using the spatial metaphors of the Earth to give us an embodied immersive experience of different aspects of the world around us so that we maybe see the world in a different way and it changes our relationship to that world. So there always seem to be like this deeper thread of connecting to the world around us and connecting to the physical Earth around us and to build a reverence and a relationship to that Earth that is so out of balance in our relationship right now. And so this piece, I'm just curious if you could sort of contextualize how that sort of fits in the larger trajectory of where Marshmallow Laser Feast has been and where it's going.
[00:27:39.922] Robin McNicholas: Frankly, we're obsessed with putting people in different sensory perspectives. And in this case, with a narrative-led piece, what we've done is adapt understanding and the work that we've done with In the Eyes of the Animal and with Trier and We Live in an Ocean of Air have all kind of amalgamated and we've realised Yeah, we can adapt these sensory perspectives for stories too. And really, from a creative standpoint, it pushes us outside of our comfort zone. We've got a very solid understanding, based on the touring work that we've put out there, of how to deliver experiential environments that are kind of Earth-centric. But actually, at the core of this, despite its sugar-coated, cartoonish veneer, the sensibilities remain in that what we're doing is, in the case of Sweet Dreams, is developing a story world that's focus is the debt to desire. So things like shark fin soup, Things like cling-wrapped bananas. All this kind of land of abundance that we live in currently, we've just taken that and amplified it and given it, from our position, a fusion of a kind of Beetlejuice cross between Who Framed Roger Rabbit, shmush. that delivers an important message, the use of the Earth's resources, but in a comic veneer that in some ways exposes and is a reflection of the age of distraction that we live in anyway. And that was the idea. And it's complicated. I hear myself speaking and it sounds quite complex. At its core they're different sensory perspectives and new challenges that we put on ourselves keeps us on our toes and keeps us thinking and keeps us thinking just discussing collectively where do we go next, where can this location-based experience scene develop.
[00:29:50.920] Nell Whitley: I think there's, from my point of view as a producer, there's multiple things going on within the company which somehow come together under one marshmallow banner, which is that we are all interested in multi-sensory worlds, we're all interested in making work which uses technology to tell a story in a different way or in a way in which that technology will allow you to do that. So that might not always be VR, it might be augmented realities, it might be room-scale projections we don't know and we don't like to kind of define that they will be we will use whatever technology makes most sense as we go forward and there is something else like thematically in terms of what we're doing as a studio which is very much asking ourselves the question of why should this story be told but why should this story be told right now and for this audience that we're hitting for. And climate, resources, our kind of human behaviour, our perception of the world are all really key themes in a lot of the work that we do. So Ocean of Air may take itself more seriously as a piece in terms of the wonder of nature and Sweet Dreams may be a very comical kind of take on that, or trying to be comical anyway. But they're both kind of at the root of those things, exactly as you described. They're very much both about understanding what we have, protect it, look at our own behaviour within the world, and ultimately offering up pieces of work that help people take in different perspectives about the planet.
[00:31:15.439] Robin McNicholas: In a non-apocalyptic way, research has exposed that climate issues, for example, the oversaturation of images of burning rainforests and so on, go over people's heads, tragically. And so the themes that we're exploring to expose wonderment, expose reflections that aren't finger-wagging is a very important aspect to our practice in that, not that we're adverse to apocalyptic messaging, it's just through our observations that's the state of play. And the other thing is, it's a slight aside, but VR in terms of this age of distraction that we live in. is an incredibly powerful tool to just allow people to engage, to ignore their phones, to ignore the outside world and be able to immerse themselves. And what's wonderful is that if you've got people's undivided attention in these spaces, we can dial in this messaging in ways that, you know, that is playful, that is full of wonderment and celebrates us as human beings. It's ongoing, and as a collective, we, over time, have a just really enjoyable community of developers and shared sensibilities.
[00:32:49.686] Kent Bye: So after I saw the experience, you said you had different things you wanted to ask me or to get my feedback on. I'm just curious if you had any specific things you wanted me to sort of address or talk about my experience in any specific way.
[00:33:00.380] Robin McNicholas: Well, I'm firstly curious about the tastes, whether you can recall any tastes and whether there was any level of nostalgia associated with them. That's one question.
[00:33:11.180] Kent Bye: So I certainly did remember some nostalgia from the second taste that I did have so you know there's a certain and also just the haptic experience of that and how taste with haptics on top of that can call back to different aspects of my childhood and then to sort of be thrown into a wonder world at the same time it was just kind of like it felt I guess very surreal because I think that's the thing about you were mentioning earlier about taste and flavor is that it does have such a strong connection to memory and our own direct experiences and so you know maybe it was at an unconscious level because the visual stimuli is so overwhelming while I was having it that it wasn't like I was just like shutting my eyes thinking back to my childhood and you know driving to the store where I buy this type of taste and eat it and as I'm talking about it now I'm able to reflect on it more and so it's a little bit of like maybe in conversation with another human being it allows me to tap into a story mode and share a story about that but in the absence of that being abstracted from any other human beings and being sort of thrown into a wonderland of visual stimuli it didn't give me a lot of time to really deeply reflect upon the nostalgia components of those flavors. But now that you ask, I can sort of recall and sort of bring those back. But it's difficult to do that in the context of a VR experience in the absence of someone who's really prompting me or receiving that story. Because I think a part of telling stories is creating a container in which there's somebody there to receive those stories and to listen to that. And so if that's the goal, then I think it's a different design aesthetic that results in someone who's in the experience with you who's there to either question or to really have a live interactive experience because a lot of what I'm hearing is sort of a pre-recorded canned voice and so I'm following the prompts but I'm not thinking in my mind I'm gonna have a meaningful interaction or conversation with this person who's talking to me right now.
[00:35:02.345] Nell Whitley: I've got a question about whether what you remember from the story, did you take anything from the story that you heard?
[00:35:10.054] Kent Bye: Well there's a bit of expectations that are built up and then whether or not those live into those expectations because there's a bit of like priming me to sort of think about the most amazing flavor I've ever tasted in the world and then like that's a high bar to then sort of walk into an experience and then have something that's not that. Because whatever it is, if you ask anybody, think about the most amazing thing you've ever experienced in your world, and we're going to match that. And then it's like, OK, are you sure you're going to be able to match this thing that I think is absolutely amazing? And so there's a bit of contrast that you play with as a narrative designer to create an expectation and then to be able to then either deconstruct it or pervert it or to do something else. And I think people want that. In a lot of ways, when people are looking for things online, they want to have their curiosity peaked or they want to be filled with awe and wonder in some ways and do something that defies their expectations. So even if you would have delivered me the exact flavor that I mentioned at the beginning, it would have probably been a disappointment because I would have expected it. So I'm thinking like, oh, there's no way they're going to be able to live up to that. And then it creates another arc within the contrast of an experience.
[00:36:16.835] Robin McNicholas: What was fascinating during the user tests that we were doing in London was that at a certain point within the piece people lost all memory of what went on in terms of the piece features a song and we asked the question yeah could you hum us a song or do you remember the context of the Well, sorry, I'm just thinking back to what actually went on and six people could not recall there being any music at all. Was there a song in there? Oh, well that's... You were dancing!
[00:36:57.420] Kent Bye: Oh yeah, yeah, so... I was dancing to the song, but I wasn't, yeah.
[00:37:03.321] Robin McNicholas: But the thing is, and what we were thinking about is that there's moments within it, haptics, and of course we track people's feet, as well as their hands and their position in space. And when we bombard the sensors, There are moments where I think the limbic system kicks in and fight or flight time where it's just like, okay, priorities, where on earth am I or am I still functioning as a human being? And it's just extraordinary. I'd love to sit down with some neuroscientists and just explore the work from that kind of cross-modal, multi-modal, flavour perception standpoint, to gain a better understanding of us as living beings, an understanding of how the brain works.
[00:37:58.642] Kent Bye: Yeah, I feel like that there's a certain amount of my own direct embodied experience as I reflect upon it, that the haptic and the taste almost dominate. Like they cover everything else that I have that's sort of like that's what I'm left with. There's a bit of not remembering the details of the song. I remember I was dancing and I was like feeling what the floor was like and I was like moving around and having a great time but I couldn't like hum a tune or like it was just sort of like I wasn't trying to remember it, I was just like I was in the moment and I think as you're in the moment and you're fully in tune with what your body is experiencing and the haptic experience and that's actually what people are left with rather than any of the details.
[00:38:35.776] Nell Whitley: Totally and that's what we're learning and through seeing people here as well in a different context seeing the work than when we were testing it in London. in an exhibition context as well, and like what are people left with and therefore how do you tell the story? Are we telling it in the right way? Is it the best way to deliver the narrative? And how do those haptic tastes, you know, how do those bits all amplify the thing you're saying, not necessarily take away and make people forget the fact that there's actually a narrative thing going on as well? And I think that's a really wonderful thing for us to be doing as creators to kind of keep iterating, keep working it out and keep working out how you do that not for 10 minutes but for 60 minutes and how do you give people rabbit holes to go down and you know to explore new things so kind of dig deeper and deeper into those sensory elements.
[00:39:20.405] Robin McNicholas: How do we add to the story world? In parallel to the actual LBE, we use social media to roll out and give context. We shared a menu just the other day that helps provide more context and more narrative that people who've been in or are going to enter can explore and so there's all kinds of different ways in which we can bolster and provide a deeper understanding of what we've been up to. So the fragmentation of story and drip feeding and the distribution of narrative elements through different devices help essentially rework the traditional sense of you go to a movie, it's bookended, you enter, you leave and it's contained. Whereas of course now, much like the gaming world, there's iteration and there's fragmentation as well that allows people to have as little or as much of the story as they desire.
[00:40:29.051] Kent Bye: This is a tricky dilemma for anybody who's into immersive theater has to navigate which is how much do you research about a project and be prepared to have your best experience versus you know you just go and experience it. So I think that there's some people who don't want to have things spoiled or to be surprised or to have some sort of expectation. But there's other aspects where you could feed these things out and it could actually amplify the experience rather than take away from it. So I think it's a tricky thing for each person. They have to decide where they fall in that spectrum as to whether or not they're going to know anything about anything before they go experience it versus they're going to get a little bit more of a context and then be fully prepared to go have that full experience.
[00:41:07.032] Robin McNicholas: Yeah, I agree. I think this is only going to get more interesting as the production values improve as well as access to equipment. You know, we've already seen in the few times that we've been engaging in these film festivals and sharing VR work. The leaps and bounds, of course, have been dramatic. There's so much potential and so much in terms of shared experiences and essentially borrowing and celebrating some of the wonderful things that are thriving in modern culture, such as the gaming world. but to embrace theatre and film and storytelling in completely new ways. There are genuinely new ways to tackle these things and we're very curious about it and I think everyone has this collective vision of where it could go. And I feel as the tech develops, tracking for example, and AI enabled systems that will, for example, negate the need for ping pong ball suits, for mocap, you know, we're already seeing incredible developments there in terms of markerless capture. and the other aspects when using AI personalization and really honing in to gain an understanding of a visitor and tailoring content for those visitors in ways that essentially provide people with new and unique experiences. You know we can all watch a movie and take something away from it and I'm so big into films and it's one of my favourite things to just sit in a cinema. I think it's a perfect example of a brilliant system that provides a wonderful escapism and of course cinemas have got the flow of visitors nailed, you know, but the truth is that as human beings we've got the capacity for so much more and technology is enabling us to provide more. and to create more engaging personal level experiences and it certainly is present all around us in the way that each of us have a timeline and each of us have a thirst for unique stories to share and to just call it their own.
[00:43:41.426] Kent Bye: And finally, what do you think is the ultimate potential of virtual reality and immersive experiences and what they might be able to enable?
[00:43:51.913] Robin McNicholas: I think the ultimate potential from my personal standpoint is that they will help enable us to be more human, to celebrate as sensory organisms and allow us to enhance the capacities of human beings. Sounds very sci-fi but I think that to an extent we've made that shift and the important thing now that we've made that shift into a globalised network there's tragic consequences of that such as the fact that we're living in the Anthropocene that my hope is that we'll be able to tackle those very present issues in ways that on a human level we all desire. There's just a problem at the moment in getting everybody aligned to realize the priorities.
[00:44:49.508] Nell Whitley: Yeah, I think Robin said it all. I think it's about seeing things in new ways, just shifting perspective or allowing new insights, whether that's a technology that layers on reality, whether it takes you to a new reality, whether it's offered in health care or in arts and culture or in whatever it is. I think the potential is all the same, which is to allow us to help tell stories in ways that maybe shift perspective or open up new ways of thinking.
[00:45:17.846] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:45:22.010] Robin McNicholas: Well, keep on going. VR ain't dead. It's just getting started.
[00:45:29.057] Kent Bye: Awesome. Great. Well, thank you so much for joining me. Thank you.
[00:45:31.240] Nell Whitley: Yeah, thanks.
[00:45:31.860] Robin McNicholas: Thanks again. Cheers. Thanks so much. And I really appreciate the fact that you danced in there. Yes!
[00:45:39.832] Kent Bye: So that was Nell Whitley. She's the producer of Sweet Dreams as well as Robin McNicholas. He's the director as well as the co-founder of Marshmallow Laser Feast. So I have a number of different takeaways about this interview is that first of all, well, the moment when they were asking me like what I remembered was a pretty fascinating moment because in that moment, I didn't remember that I was dancing or that there was music. There's something about the sensory experiences of those tastes that almost completely dominates my entire memory of the entire experience. which is funny because that's what they were actually finding is that the tastes were either bringing up these deep associations or memories. For me, I actually don't remember tasting the first and the third taste. I don't know if they made it up or they created it. Certainly the middle one was a very popular taste that I've had as a kid. It actually did invoke these different memories, but it's funny because during the actual experience, I wasn't thinking about those memories because I was just, Robin said you're being bombarded by this overwhelm of sensory experience. You're like walking around on this cushion. You're like the visual depictions of what they were showing you is just like absolutely crazy. So your visual systems can really dominate the way that you process and make sense of the world. And so Robin was saying that in the future they'd like to really play with like what's it mean to, for example, show something as searing hot, but when you eat it, it's like absolutely ice cold. This spectrum of polarity points of hot cold wet dry, you know There's these different things that you can give a visual depiction of but what's it mean to actually start to? pervert that a little bit within virtual reality and start to Play with what your expectations are based upon your sensory experience versus what you're actually tasting or experiencing but there seem to be a lot of just trying to correlate these kind of crazy visuals that were like really out of this world at the same time as you were like ingesting these different tastes and flavors. But the way I think about it after talking to them is that there's something about those, those tastes that really dive right down into these deep memories that you may have. And the funny thing is that you're kind of tapping into the matrix over those emotions and those memories without actually explicitly bringing up any visual depictions of them. Like as I was in the experience, I wasn't, Thinking about like my own memories, you know, my relationship to these tastes as Robin was like talking to me afterwards. I had an opportunity to kind of think about it. Oh, yeah. Like I remember as a kid going and getting this at a store and tasting it. And so it actually was tapping into these deep memories, but it almost felt like at a completely unconscious level, because like, I wasn't actually thinking about those actual memories during the actual experience. So I think that's something that they're discovering and finding, but also like if they're trying to use these tastes to be able to tell a story, then how much are people getting the deeper aspects of these stories? And I talked about this a little bit in the dial, just in terms of like, there's a certain amount of story that's being transmitted through the spoken word. And when I'm in a virtual reality experience, it's a lot harder for me to parse and understand the deeper complexities of the human word. And it's actually the thing that like I remember the least. So if you have some sort of like visual depiction of something, I'm going to like remember that a lot more than I'm going to remember dialogue or even like the specifics of a tune or a song that might be playing. So I think as narrative designers, especially with immersive media, looking to something like sleep no more than she fell and seeing how they're using this symbolic translation of embodied movement and bodies moving through space and trying to do this like symbolic translation of these deeper principles and almost like turning it more into like dreamlike imagery, like that translation that may be a little bit less clear, but it may be more viscerally memorable. And so there's an interesting challenge for me as I'm like going through these different experiences and I'm put on the spot, like, Hey, could you like describe the narrative that you just saw? And I'd be like, uh, I remember dancing and I remember what the taste was, but it's an interesting dilemma for these experiential designers is that if you are trying to use this medium to tell these stories, then how are you going to best use the affordances of these mediums? And for me at my experiences, after seeing a lot of these things is that the more that you turn it into like this weird, crazy abstract dream and use those. spatial metaphoric images to be able to communicate what you're trying to say, that is gonna be easier to remember. The challenge is that it becomes so abstract. It's like, what are you actually trying to say? So for me, it was interesting to actually talk to both Robin and Nell about this because they're talking about this age of destruction and being in this anthropomorphic scene, which is a geological term that every 10,000 years it has a new name and we're moving from the Holocene to the Anthropocene. which is saying that humans have destroyed the earth so much that it requires us to name an entire geological epoch of 10,000 years after humans because we've screwed up the earth so much, but that we're living in this age of destruction and that they're really trying to explore what they call this debt of desire. And I think there's this connection between our desires and the way that our behaviors are acting in the world to have these really exotic types of experiences. And this felt like this over the top experience, the point where you're just being completely satiated by these exotic different tastes. And what is the deeper cost? And that was what I got from. Speaking to them and I think that in the way that they actually communicate that within the experience is still like they're trying to use this really wild and vivid dreamlike imagery to communicate that so One of the things that Robin said is that you know people will be shown images of forests completely burned and you know These different visual depictions of the earth that's in this very desperate state and it just completely goes over their head and they don't remember it or actually, you know, it doesn't stick in any meaningful way and so They're trying to use these mechanisms of wonderment and these deep contemplative reflections in order to still maintain this aspect of apocalyptic messaging, but to create these participatory worlds and story worlds that are trying to explore these deeper issues. And that to me, originally, it felt like Sweet Dreams was a bit of a deviation from some of these other experiences that were much more centered in the earth. And they were more about creating a context and an environment and having a direct embodied experience of some of these deeper metaphors of these ecosystems. But now they're trying to incorporate query more of these explicit narrative components within these experiences to see how they can tell these deeper stories. And so it's a difficult blend to figure out how to actually fuse these things together, because what is the thing that you're actually leaving people with? That's a challenge because as a new art form and how you actually use all these different tools of the spatial communication medium in order to like say these deeper messages that you're trying to get across. But, you know, once I said that all of their experiences are about the multisensory experience and how they're trying to put people into different sensory perspectives, I can see that there is this consistent through line through all their work and that they're using the latest and greatest of all these emerging technologies to be able to play with the human perception and to talk about these deeper ecosystem issues, but also these different patterns of human behavior and how this is all related to each other. Noah saying that she does see this as this evolution of existing art forms and that while there are a lot of new and novel components of virtual and augmented reality there's a lot of things that they're drawing from from both film and theater and the pageantry of the rituals and the best practices for how do you do this transition between Your mundane world and going into this magic circle to understand the rules like in this experience they said if there's anything that's pink you can consume it and so I when you're in the experience and you see something that's pink, you're like, okay, well, I guess I can go over there and eat this and have a whole embodied experience with that. Yeah. It's just like establishing like the magic circle, the rules of what you're about to experience. And then I think they also wanted to really have this very careful handholding because they had the, a couple of immersive theater actors who are really onboarding you and giving you a lot of context as you're about to enter into this world. And so you have this whole. interactive improv moment with these actors who are priming you into going into this world. And then once you go into the world, then you're kind of on your own exploring all these different aspects within the experience. And finally, just the point that creating these experiences is amazing because you get to play with all this technology and you get to have the best experience you possibly can. And then when you start to think about, well, how do you actually distribute these out to people and ensure that you have a quality experience for everybody that gets to go through it? That's a lot harder of a problem that is still in the process of being solved. But it does seem like that some of these either location-based VR entertainment locations or specifically these galleries like they're showing the We Live in an Ocean of Air, the Sashi Gallery in London, which I think it's actually still playing. So if you live in London, you should definitely go check it out. But the distribution aspect of these experiences are extremely difficult, especially when you think about getting to the point where you have like these commercial off-the-shelf hardware to be able to be able to distribute some of these different experiences. And I think that there's a growing ecosystem, especially with the HP backpacks and more and more haptic technologies. And, you know, at Tribeca a couple of years ago, they had this whole device where you could actually have these different smells that were incorporated within this experience. But there's challenges there of taking some of these prototypes and concepts that you can see in the context of a film festival, and then being able to actually have the maturity of the technology to the point where you can actually rely upon it and have like hundreds of people or even, you know, potentially thousands of people a month. go through these experiences, then it has to have like a certain amount of stability and rigidity to be able to handle being deployed into these different location-based virtual reality experiences. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
