Play episode

#1344: “Turbulence: Jamais Vu” Wins IDFA DocLab Immersive Non-Fiction Award with Embodied Mixed Reality Doc on Depersonalization & Derealization

January 23, 2024by kentbye

I interviewed Turbulence: Jamais Vu creators Ben Joseph Andrews & Emma Roberts at IDFA DocLab 2023. See more context in the rough transcript below.

Here are my 19 episodes from IDFA DocLab 2023:

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Podcast: Play in new window | Download

Share this
FacebookXRedditEmail

Rough Transcript

[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR podcast. It's a podcast about the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So this is episode number 19 of 19 of my series on IFADocLab 2023, where we've really been diving into a lot of the digital storytelling and immersive stories from the International Documentary Film Festival of Amsterdam's DocLab selection. So this final episode is with what ended up winning the Ivadoc Lab Award for immersive nonfiction. It's a piece called Turbulence, Jamais Vu. So it's by Ben Joseph Andrews and Emma Roberts, who I previously had on the podcast on their piece called Godwanna that was showing back at Sundance in 2022. But Turbulence is a mixed reality piece that is exploring this experience of vestibular migraine that Ben Joseph Andrews experiences. And so I'm just going to read this synopsis paragraph. When extended reality artist Ben Joseph Andrews suffers an attack of vestibular migraine, he loses his sense of orientation, balance, and spatial awareness. It also affects his experience of reality, where everything familiar suddenly seems new and different, a neurological phenomenon called jamais vu. So there's déjà vu, where everything feels like it's very familiar and like you've experienced it before, but jamais vu is like the opposite. It's like everything that should be familiar is suddenly like totally alien. So when Ben experiences this vestibular migraine, it's basically like this experience of depersonalization and derealization. which is a condition that people can also get from doing virtual reality experiences where they feel like nothing is real and they're not really who they are. It's like this fundamental disconnect, this disconnection from who you are, your identity, and this condition of vestibular migraine is evoked as it's inflamed in different ways. So this was a mixed reality experience that was trying to recreate different dimensions of this disconnect from your body So there's a way that the world is inverted and flipped and so you kind of have this Disconnection from your body in a way that can actually cause a little bit of nausea and nauseousness but I was also trying to mess with your normal proprioceptive reactions by Expecting when you move your right hand you see your right hand move but in the visual feedback is that you're seeing your left hand move so it's got this use of the mixed reality technologies to be able to invoke this experience that Ben actually goes through. So, yeah, kind of a provocative use of things that can be somewhat nauseating, but in a way that's really serving the story of what he's trying to describe and give someone the experience of what he's going through and his depersonalization and derealization experiences whenever he's experiencing Xiamen Vu. So, that's what we're covering on today's episode of the Wizards of Yara podcast. So, this interview with Ben and Emma happened on Monday, November 13th, 2023 at IFA DocLab in Amsterdam, Netherlands. So, with that, let's go ahead and dive right in.

[00:02:58.742] SPEAKER_00: My name's Emma Roberts. I'm an artist and creator based in Wathaurong country in Australia, which is in regional Victoria, near Melbourne. I've been working with Ben, who introduced himself in a second, for about eight years now, making different kinds of XR, but with a particular focus on LBE.

[00:03:15.096] Ben Joseph Andrews: And yeah, my name's Ben Joseph-Andrews, also a new media artist based on Wathaurong country. And yeah, outside of working as an artist or practicing as an artist, I'm also a researcher into kind of immersive embodiment.

[00:03:27.528] Kent Bye: Great. Maybe you could each give a bit more context as to your background and your journey into working with immersive media.

[00:03:32.667] SPEAKER_00: Yeah, absolutely. So I came from film. I was a producer but was asked by Ben to help out with a project he was making in 2016 and became immediately really fascinated by how not cinema VR was. The work that we were making was a live scored LBE with a lot of performers sharing space with the people in headsets and we were immediately really fascinated by Just the beauty of being able to work really intimately with an audience like that. I think when you come from film, you're very separate from your audience. You create your piece. It might be six months, a year before it gets to an audience. You might not be able to be there with the audience. You certainly can't shape around that audience. And I really loved how much you could co-create together, I guess, with your audience. And that was what really drew me into the field today.

[00:04:21.353] Ben Joseph Andrews: And yeah, I also came from a filmmaking background as a kind of director and was very interested in more immersive and experimental forms of storytelling rather than traditional kind of narrative-based approaches to the medium. And through a kind of forced hiatus through my PhD, I started to experiment with VR in 2016 and kind of, yeah, as Emma alluded to, became quite entranced by the differences in the two media and particularly around the kind of the live nature of an experience and even if it's playing back in a linear way, the experience, the atmosphere is alive and can be shaped around and immersive through various multi-sensory kind of interventions. And basically since the very early experiments, we've been really fascinated by the kind of liminal space that you occupy in VR. that kind of in-between space. You're in a headset, you're in a virtual three-dimensional space, visually and sometimes aurally, but your physical body is also still in another space. And those overlapping realities, shall we say, has always been a really strong fascination for us in our work.

[00:05:34.450] Kent Bye: Yeah, and before we dive deep into your project here, Turbulence, maybe you could give a bit more broader context for the location-based entertainment or other VR projects that you've been working on for the past eight years, since 2016.

[00:05:46.812] SPEAKER_00: Absolutely, so we created two of the largest, I mean, I think probably still kind of the largest location-based experiences in Australia, both of which were live-scored performances, almost like immersive theatre using VR. They were called All the Stars They Bleed Together and Starless. We became really fascinated with the way that you could, in a relatively short period of time, bring people into these really alive, performative spaces. All the Stars They Bleed Together was kind of this sci-fi journey through a wormhole with live musicians where we brought people into a space that seemed like a typical XR, I guess, showcase where you just had a headset and stools but then once you put on the headset we kind of used the fact that it also acts as a blindfold to completely alter the space around people and fill the space with live musicians so that when you'd gone on this virtual journey you emerged in literally another world from the one that you entered into. Starless was an immersive near-death experience based on, I guess, those archetypes that we hear about all the time. That was just this huge, incredible, live-scored, live-performed piece for four users at a time that played a lot with sensory mismatch, drawing you out of, like, your physical embodiment and expectations by really playing with, like Ben said, that liminal space where you're both present somewhere that you're not and absent somewhere that you are. And I think most people maybe, if they've seen our work before, might be familiar with Gondwana, which is a durational artwork ecosystem that evolves over a day's exhibition that premiered at Sundance last year. It's been touring ever since, and particularly as an immersive installation where it's not just the headset. but also using projection and real-time lighting and sound to really bring you into this co-present multiplayer virtual ecosystem that evolves over a day's exhibition in line with climate data.

[00:07:42.072] Kent Bye: Any other comments on that?

[00:07:43.413] Ben Joseph Andrews: No, I think that covers the kind of mischief that we've been up to over the last few years.

[00:07:47.765] Kent Bye: Yeah, I really enjoyed the Godwana experience that I saw at Sundance. It was virtual, so we didn't have a chance to meet, but yeah, we were able to unpack it a little bit more in a conversation. But I really appreciated the ways that the sound design in particular of that piece of Godwana was evolving over time such that it was really reflecting the changes that you can really see. But there are some visual changes, but for me, the way that you were able to have the sound design of that ecosystem change reflecting the impacts of different species dying off that would result from climate change. Yeah, it was really quite a visceral experience of that. So yeah, really appreciated that project. But yeah, maybe let's move to Turbulence that's showing here at IFA DocLab 2023. You had a chance to give a little bit of a preview of this project at the IFA DocLab Live that happened the other night where you were talking about this condition that you have, vestibular migraines, and so maybe you could talk a little bit more about the impetus of this project and where it started for you.

[00:08:45.967] Ben Joseph Andrews: Yeah, so I have a chronic condition called vestibular migraine, which is very much unlike the dominant conception of migraine, which is the kind of pain-dominant episodic variety that might contain a visual aura. Vestibular migraine is kind of wholly different to that, and it is 24-7 and constant, and expresses itself sometimes through pain, but not always. The kind of primary ways that it manifests are through essentially misperception of motion. So that is vertigo, that is dizziness, that is basically everything that your vestibular system is responsible for. So balance, equilibrium, stasis, that's what becomes unsettled. So I've had kind of a chronic migraine condition since I was 20 and this variety has been with me for the last kind of six years. And migraine is a condition that evolves over time, so it doesn't remain the same, and your preventative measures aren't the same, nor are your symptoms. So it's this kind of evolving, changing, but very mysterious force. And it kind of gives rise to all kinds of different phenomena that sometimes seem very, very disconnected. So, Turbulence Jamevu is about a phenomenon called derealization or depersonalization, which is essentially a very unsettling experience where you kind of lose your continuity that underlies your stable sense of self and your stable sense of reality. And all of these things quite suddenly become very strange. people are probably quite aware of. Déjà vu was a kind of an experience of feeling like you've seen something before. Jamais vu is kind of the opposite of that. It means never seen. And essentially it's when things that are very, very familiar to you become suddenly unfamiliar. And for me, I know that a really big migraine flare up is about to occur when I look into the mirror and misrecognize my own reflection, and then I look down at my own body and I feel strangely disconnected to it, as if I'm seeing it for the first time and working it for the first time. So, yeah, so this is very much a part of exploring and trying to express, trying to find language, trying to give an experiential context to phenomenon that are very, very difficult to describe through words.

[00:11:11.585] Kent Bye: Yeah, and it sounds like a really intense experience in that there's certain ways that you're using VR as a medium to start to give people some vectors of access to this as an experience. And so, Emma, I'd love to hear from your perspective, as you have heard about this condition, what it was like for you to start to potentially use a medium of VR to help other people experience, but also for you to experience it for yourself to understand what Ben might be going through.

[00:11:37.917] SPEAKER_00: Yeah, absolutely. Because Ben and I have been working together for so long, and we've known each other for a very long time as well, this has always been something that we've been, I guess, working through together in our work. Like, as you can imagine, VR is often quite a hostile medium when you have a condition like vestibular migraine. It's feeding all this motion data to you that sometimes doesn't match your own and that can cause motion sickness for most people, let alone someone who's very sensitive to that. So Ben and I have always been like, I guess, quite open about trying to communicate and understand this between the two of us, particularly when it does get really bad and we have systems in place to work together to try and be as accommodating as possible when those flare-ups are really bad. But it was only when we started to really look at it in the eye, for a long time it's been this thing that we kind of have to work around and push past. But we became really interested after Gondwana in kind of exploring vestibular migraine with the same curiosity that we've explored subjects in a lot of our work. So it's been this really incredibly interesting process because I think we've previously talked about it through language that doesn't quite express exactly what's happening from that felt and embodied perspective. But the closest parallels that Ben had to offer me were when VR goes wrong, when something's awry with your embodiment, you lose tracking and your arm is suddenly drifting through the wall. And so that became something that we decided to lean into. And with this work, it's been this really incredible process of learning, I guess, for the first time, even though we've been talking around it for a very long time, learning what these things feel like, what those analogies are. I think VR What we've become really fascinated by is how much VR can be used as a vehicle to unveil our own perception, to show us different ways or expanded ways of perceiving the world around us because when we build something like Chamois Vu, Ben has had to... we built it ourselves as well. There's been so many times that Ben's built something in the game engine and gone, oh, this is actually, I think, how my brain is working. And so it just gives us these metaphors and these frameworks to understand exactly what's happening inside your head that we just didn't have before. It's been such an interesting creative process, I think.

[00:14:14.473] Kent Bye: And can you elaborate a little bit more about your journey into VR? Because it seems somewhat improbable, given this condition of vestibular migraine, that you would even spend much time at all within VR. But what is it about the aspects of VR, but also if there is a, like, for me, I have a lot of motion sickness triggers that I've had since the beginning. I've sort of accommodated over time in terms of like locomotion through VR was something that was very triggering of motion sickness at the beginning, but I feel like I have been able to deal with certain inputs and have like the quote-unquote VR legs, which I was very Skeptical of because I still get motion sickness triggers. There's still things that will make me motion sick But I feel like I can do the more intermediary type of locomotion So I don't know if you've had a similar experience of like knowing your triggers or knowing what you're capable of and but yeah I'd love to hear a little bit more about your journey into this because it feels like it's somewhat improbable that you would even be involved with VR Yeah, it is a very improbable kind of thing.

[00:15:13.798] Ben Joseph Andrews: So I Yeah, I guess I've always been really interested from the very beginning into, as Emma alluded to earlier, this kind of notion of sensory mismatch. And what I will say is that even though I've had this condition for a long time, I've only actually been like formally diagnosed with it reasonably recently in the last kind of 12, 18 months. So for a long time, I knew I had a migraine condition, but I had no idea of this whole other dimension of it called vestibular migraine. So I've always been incredibly susceptible to motion sickness, like extreme motion sickness as a child, and there is a strong correlation between extreme motion sickness when you're a child and developing chronic migraine conditions. And it's really interesting, and we've kind of been reflecting on this quite a lot, is that now when we look back at some of those really experimental live performance works that we were doing in VR in kind of 2016, 2017, there seems to reveal this very latent and unconscious interest in the mismatch of vision with other senses and this kind of like need to kind of almost like attempt to reconcile these dissonances into an experience that I think at the time I didn't have a conscious understanding but now when I look back I can kind of see that perhaps that fascination with this kind of experimental approach towards essentially using sensory incongruity as the main aesthetic driving force or affective driving force of the work as some kind of unconscious attempt to understand, I think, what was happening underneath all of this that I wasn't really aware of until I kind of received this diagnosis. I understood why I've found VR so challenging for so long as a participant that has to, that goes through other work, but also develops work and, you know, goes through a lot of non-optimized builds and finds that really challenging. but also I think that it kind of put into context a lot of the deep fascination with the medium as well and its potential.

[00:17:28.738] Kent Bye: Yeah, and for me, as I've gone through different VR experiences, you know, I have this sense of being able to have these different qualities of presence, whether it's like active presence and the extent that I can express my agency and interact with a piece and how responsive to that feedback it is, or the mental and social presence, which is like the plausibility illusion from Mel Slater's terms, which is like this suspension of disbelief and really believing that I'm in this world and then There's like emotional presence, which different films and music and lighting that starts to really cultivate this sense of being emotionally connected. And then there's this sense of embodied and environmental presence where I feel like from Mel Slater, there's a place illusion and also the virtual body ownership illusion where I feel like I'm actually transported into these locations and actually have some sort of embodiment within those. And so I feel like that I've over time cultivated this ability to more and more suspend that disbelief and have deeper qualities of presence. And so I know what it feels like to be embodied in a virtual space and to have this sense of virtual presence. And hand tracking is something that is, something that I think is particularly strong at cultivating that sense of embodied presence because it is so responsive to what I'm seeing and that my mind attaches my virtual hands with what my actual hands is just because it is so responsive. And so in your experience of the turbulence, Jamais Vu, It was that experience of depersonalization or this kind of disembodiment or feeling disconnected from my sense of embodiment in the experience in a way that was quite striking. And you're using a lot of tricks to invert as I turn my head left, you're actually turning it right. and there's actually an inversion as well so usually when I am moving my left hand I see my hand on the left moving but it's actually the hand on the right so there's also another switch so I feel like some of these things were achieving that sense of that depersonalization and this disembodiment but at the same time I know my motion sickness triggers and I know that But if I continue to move my head in a way that the head rotation isn't actually reflecting, then it's going to make me extremely sick. So I ended up spending quite a lot of time either very slowly moving my head or minimizing my head movements or shutting my eyes in some cases because I knew that that was going to make me sick. But at the same time, Kaspar Sonnen said that, you know, this is the first time that he's seen that someone that has really deliberately used that type of nauseating effect for a narrative purpose, because that's mimicking what your embodied experience is with this condition of vestibular migraines. I feel like there's this thing where I could see the narrative purpose but also there's a deeper questions around if people are aware or consenting or just recognizing their own motion sickness triggers and especially here in a festival context could wipe them out for the day. But for me, I felt like it's the strongest use that I've seen of using this intentional use of nauseating type of effects within VR to connect to the story because it's something that's very personal to you.

[00:20:26.850] Ben Joseph Andrews: Yeah totally and you know I feel like with Jamevu we weren't trying to go for disembodiment as much as we were for misembodiment because I think that that's very true of the experience that I'm referencing through the work is there is this kind of like rupturing in kind of almost like this emotional bodily somatic connected continuity that I think that we often take for granted that we can remember a lot of things to do with our environment and to do with our body and we have this kind of memory that we draw upon that often feels very, very, very stable and robust. but it's very fragile and tenuous and something like migraine of all things can kind of upset and unsettle this kind of cache of understanding. So for example, even the voice of my parents, my loved ones appear really really strange to me during these kind of flare-ups because it's almost like I'm hearing their voice for the first time and I know that on one level that these things are from should be familiar yet they're not familiar at that moment so it requires this kind of negotiation but it's not a complete disembodiment or a disassociation it's kind of almost like a misassociation it's like there is this navigation that needs to take place between the disruption that's happening on a kind of like a sensory and a mnemonic level but Also on the moment-to-moment overriding these things that actually are familiar and still feel familiar but now have to be kind of approached anew. So within the work we're really interested in not shying away from the intensity of the experience of it because The condition is something that is really incredibly difficult to navigate and its intensities of misembodiment are on such a formative level. It's not an intellectual disorientation, it's very much a discoordination of the entire sensory and bodily apparatus. And that's something that we were interested in from the very beginning, to place audiences within something that is challenging, that it is a non-normative way of being embodied in your own body. you still are your own body like it's using a camera on the front of the headset so it's not showing you kind of a 3d modeled avatar that's kind of like following your hand tracking like you are seeing your actual hands but kind of made strange but Yeah, and the intensity of it does require you to go very, very slowly, but I think it's also still something that we're trying to work out with a piece of how to also communicate that to audiences, because there is this kind of need to reconfigure how you are within this kind of environment and this kind of experience. It's very, very different to others.

[00:23:22.948] SPEAKER_00: If I could just add to that as well, I think one of the challenges of creating the work and something that we've really tried to hold on to as we do so is really taking into account that spectrum of, you know, some people have no motion sickness at all and want to get in there and wiggle their head around heaps and get really exploratory and then there's also a spectrum of people who are super sensitive to movement and so we spent a lot of time really trying to tune and craft Even though we wanted it to be a challenging experience, we also wanted the experience to give you permission to find it challenging. It begins by asking you to ground yourself and literally telling you that you can close your eyes at any point if you need. There's a couple of moments where we ask you to do that just to reset, but also keeping it short so that it is something that we're not trapping you in for a long time. We always try and tell the volunteers that are running things, or if we're running things as well, you can leave at any point or like here are these tools again like encouraging people they can just close their eyes so it's been a that's been something that I think we've really tried to take care of as much as possible because it is I don't know I've been in so many VR experiences right where it's like deeply sickening but that you're trapped in it you don't want to be rude you don't want to take off the headset and disrupts and things or you really want to get to the end of the story or whatever that might be. So I'm trying to build in those moments where people feel comfortable enough that they can just close their eyes and listen or just stay still and move slowly. So that was a design consideration for us as well.

[00:25:08.902] Kent Bye: Yeah, yeah. It makes me wonder, because you're doing a couple of things at the same time, is like when you're looking left, you're turning right. And also you've inverted the view of the handstand. So I don't, I don't know if like one or the other could achieve the same effect without having additional potential, or maybe those you can't decouple because it's the inversion of that is sort of like, yeah.

[00:25:29.425] SPEAKER_00: It's the same mechanic that's doing both of those things. So all that we've done is just invert the view. So what that does is just anything that you do is in the opposite way around. So if you move your head left, it moves right. If you move your right hand, it looks like it's your left hand. So it's just the one mechanic. There's not two separate ones.

[00:25:50.344] Kent Bye: Yeah, but I'm wondering if you could headlock it or, I don't know, I mean, because it's the head turning that is like the real nauseating thing. The hand switching is like I can handle, it's just like when I'm turning my head, so you could potentially like model the body and actually project it out and it'd be less of a pass-through experience and more of like a simulated experience at that point, but I don't even know if it's technically possible with mixed reality to composite out hands and be able to do that, where you're able to swap one thing without having to swap the other, but yeah, I guess that was a limitation for that decision, so.

[00:26:20.223] Ben Joseph Andrews: Yeah, and we considered using the kind of, because it's using a depth camera on the front, we considered using the depth mode to capture perhaps the hands and then, but yeah, like you say, the recompositing kind of like is the difficulty because what that compromises on is the kind of strength of the mixed reality aspect, which is the physicality of the environment around you and how that matches and mismatches your visual experience of it. And then the compensations that need to take place on a bodily and proprioceptive level in order to navigate and make sense of that. But you're totally right in the sense that the whole work is built on this conflict, right? The mechanic is a conflict, it's reversing the very habitual ways of moving and that's something that we really did want to explore consciously with this and deliberately with this. almost resisting and challenging the very normative modes of embodiment that are often given within VR that are often as challenging for me as something like this. So it is something that we really wanted to kind of explore and to ensure that that was very much the aesthetic ground in which the experience takes place.

[00:27:36.268] Kent Bye: Yeah and there is a table with objects and it's using a mixed reality pass-through mode that has this shader effect that has these edge detections that makes it feel like a black and white outline where it does have this experience of you have this different type of inverted environment that does have this kind of depersonalization or derealization effect where it's uncanny in a way that does make me feel somewhat disembodied but yet I'm feeling the haptic experiences so there's something that the haptic experiences that I'm feeling is different than the visual feedback that I'm getting with what my body is doing and so you have us do a number of tasks like pick up the cup and put it on a coaster and then get out some aspirin and then put it in and I had enough proprioceptive awareness where I was almost like tuning into this other awareness or I don't think I shut my eyes but at least I reached out where I thought it would be rather than what the visual feedback was because There's this mismatch between my proprioceptive experience of my body versus the visual feedback of my body, which is this disjunctive experience that you're creating. But yeah, these different tasks that you have people do to have like a deepening of that experience I thought was also quite effective of illustrating this experience that you've had.

[00:28:47.377] Ben Joseph Andrews: Yeah and it's designed around like it's almost like a rising difficulty curve as well like it begins in a very kind of reassuring and grounding way and then essentially the opening third or half of the experience is more an open experience where you're just kind of encouraged to explore the environment on your own terms. So, you know, some people will find the little rocks on the desk and kind of just play with the textures of those, where others will kind of be a little bit more active and rummaging around. And then others again will be very, very, very tentative. And I guess part of the reason behind that is to not be super prescribed in what an audience can do there, but also to allow for people who might be feeling a little bit uncomfortable, not feel like, oh, I have to immediately do all of these really, really challenging things. And essentially, you then start to get given, through the voiceover, these very easy tasks to do. Reach out, grab the mug, put it here. But obviously, through the underlying mechanics of the work, that is really difficult. More challenging for others than some, but certainly very challenging. And as you're doing that, and then you're being told, oh, now get these aspirin from this container that's also on the desk next to the books and put it in the mug. We need three pills. as that's happening the stability of the image is starting to also break down and you're starting to get this kind of temporal delay takeover so it's not just that you're in this kind of black and white edge detected environment where like textures and objects are looking dissimilar and it's not just that your hands are flipped and you're trying to navigate that but then you're also navigating this kind of like temporal layer which is then again almost disrupting and interfering with your sense of continuity in the space. But it's really interesting, you know, through a lot of conversations with people is that the way that the experience actually kind of like asks you to move away from the visual input as much as anything, even though the visual input is actually quite strong in terms of what it's saying, but you really need to almost rely on the kind of haptic, tactile and proprioceptive cues of the body And that's kind of interesting on another level for me because a lot of vestibular migraine can come through and get worsened by, it's not the thing that causes it, but it's a compensation mechanic that worsens over time, is visual dependency. And it's basically trusting, this kind of unconscious trusting of a visual input more than the bodily input of motion. So when I see a car moving, my visual dependency is so skewed that I will feel that the car moving is my movement. And it's essentially the same thing that underlies why people can get motion sick in a film, in a 2D film, if there's a lot of handicam work, a lot of shaking camera, you can feel this sense of movement that isn't yours, yet your body is taking it on as its own. And there's this kind of interesting thing that I think takes place within the experience where you have to kind of actually move away from vision and you have to kind of rely on these different modes, these latent modes of where your body is in space to kind of reconcile all of the elements of it.

[00:32:07.639] SPEAKER_00: Yeah, I guess it's really interesting watching people go through Jamevu because it reveals this very different way that we are all in the world and that we understand it. I think there's some people who go through Jamevu and immediately, I think because that's how they go through the world generally, are navigating through tactility rather than navigating visually. So immediately, as soon as they're asked to do something, even before they've realised that perhaps that things are inverted, reach out and explore some of the objects on the desk. Maybe you're reaching for a pen. immediately put their hand on the desk and slide their hand towards the object that they're looking for and that's a much easier way to navigate in this inverted space because you're relying on this like concrete physicality. Some people completely can't deal with, I guess like it reveals that there are also more visually dominant people who don't immediately default to that and can get really lost in that space and I find that really deeply interesting. that the way the experience works reveals how we all have this multiplicity of ways of understanding our world around us.

[00:33:23.708] Kent Bye: Yeah, there was a paper on a code of ethics that was looking at different potential risks of the virtual reality medium that Thomas Metzinger and his co-author was talking about. One of the risks is the depersonalization and derealization that there are some people that may be susceptible to when they have VR experiences, they will have this implied experiences of derealization or depersonalization and you had mentioned that those are sometimes connected to vestibular migraines so I'd love to hear you elaborate a little bit on if you've had these experiences of depersonalization or derealization and how that fits into the broader context of VR as well.

[00:34:00.107] Ben Joseph Andrews: Yeah, so Jamais Vu is based on my own experiences of depersonalization and derealization. And yeah, I think that that's something that I think gets lost in the loaded charge term of migraine is pain and that everything else is somewhat obscured by that dominant conception that migraine is pain. But migraine is actually this whole disruption of the processing of information, I guess, you know, the whole neurology of the brain is disrupted through this kind of... malicious activity, I guess, or just kind of other. It's just like this kind of alterity, I guess, that gives rise to, yeah, all these various phenomena. So depersonalization, derealization, but also aphasia, which Emperor is also based on. I also get aphasia, not to the degree that that piece explores, but yeah, the kind of like jumbling of words, the phenomena of ataxia, which is loss of kind of motor coordination, I also get Alice in Wonderland syndrome, which is the sensation of being larger or smaller than your environment and your body. And also this phenomenon that sea legs equivalent, but through my own sense of movement through the world. So if I'm moving or catching an elevator, Memory of that movement will stick with me sometimes for hours sometimes for days just kind of on this like weird strange feedback loop not really that dissimilar to like a delay pedal in music that was capturing this kind of infinite field of signals that are kind of just like replaying but becoming saturated with other signals that might be entering into the chain, so It's like a reverb Yeah, kind of a reverb, kind of a delay, kind of a loop, some kind of combination of those effects. I think there is like an analogy there, I think, that fits the kind of sense of movement that resides within me and doesn't get cleared. So the experience of Jamais Vu or depersonalization for me, it's almost like I find myself performing myself. You know, I almost find that I'm puppeting this thing called my body and that's quite an unsettling feeling. And what I liken that to is something has happened that kind of reveals the degree in which my perception of self, of world, of all of these things is mediated. and it's almost like you're confronted with this layer of mediation and you're kind of behind that and within that and beneath that but this kind of mediation of perception that's constructing this very thing called this moment becomes incredibly clear yet incredibly ineffable in terms of how to go on, how to go through that and how to continue with just doing day-to-day tasks when you kind of All of these things have been kind of unsettled and it's interesting, even with a migraine specialist, you have these kind of conversations and they'll be talking about symptoms, but it's never like they'll sit down and say, well, how's your grasp on reality going? So for a very long time, I didn't realize that these symptoms were part of migraine. And that was a huge, tremendous change because I could locate and integrate these really strange existential phenomena into this kind of other experience. And this wider understanding of migraine, which gave me a lot more respect and almost reverence for what this condition is and can be and the levels it can disrupt. It also reminded me so strongly of VR and especially Pass Through because what Pass Through does reveal is the mediated layer of how your vision works with your bodily connection to the world and it asks you to essentially navigate through this additional layer that makes strange your connection with your physical surroundings. And while these cameras are quite good, they never match the precise stereoscopic vision that you might have and the way that your eyes work. So there is always this kind of virtuality to pass through into mixed reality. And immediately that felt like it had to be the medium through which this experience took place.

[00:38:16.625] Kent Bye: Certainly with the Quest 3 and MetaQuest Pro there's a reprojection warping happens that your hands are kind of blurring and it does have this uncanny depersonalization type of effect where it feels like it's slightly not completely my body or at least I can see the layer of mediation. I haven't had a chance to try out the Apple Vision Pro. Apparently the pass-through is a lot better and a lot more closer to having this level of fidelity that you actually need apart from some of the existing reprojection warping effects that you get as most of the mixed reality pass-through that we have today. But you had mentioned earlier that this is a condition that is happening 24-7 but you've also mentioned that there's flare-ups and so I love if you can differentiate between what's the baseline experiences and what the frequency of those flare-ups and if like VR actually is a catalyst for some of these flare-ups. So yeah.

[00:39:06.341] Ben Joseph Andrews: Yeah VR is certainly probably my biggest trigger and actually for a long time during the end of the production of Gondwana I was quite convinced that I wouldn't do anything more in VR because it just was too hostile for me because like Emma mentioned before VR sends you so many signals of where your body is in space virtual space that is conflicting with where you actually are in physical space and because I'm quite visually dependent all of the cues of 3D vision are very inconsistent with that. So for a long time during the final stages of Gondwana, I couldn't even stand up from the bed. So intense were the symptoms of vertigo and dizziness and movement. I've got much better control over it now and it really catalyzed a huge shift in more deeply understanding some of the lifestyle changes that you can make to help control. Because essentially there's this threshold in migraine and vestibular migraine. So say the intensities will reach a certain threshold where they become disruptive and then they'll reach certain thresholds in which they will basically, like you mentioned before with some VR triggers for yourself of just like, you'll be wiped out for a day. The thresholds in vestibular migraine, the upper thresholds are you could be wiped out for a week. You could trigger this threshold that then just, it starts and it just doesn't stop again for a very, very, very long time. But these thresholds are ever-changing and are completely invisible, so essentially the preventative game is to lower what you can control as much as possible so you're staying away from the edge of this cliff that exists in the darkness. And, you know, there are things you can control, so diet's a big one, sleep, stress, how much I use a headset. how much motion input I'm putting into my body so there are days where I have like kind of almost like low sensory days but they're like low movement sensory days so that could be how much I'm moving around in transport or watching films or just what I'm exposed to that can feed that feedback loop. Then there are also a lot of things that you just can't control so like shifts in atmospheric pressure are really really big triggers for example that yeah you can't really change of the weather or sometimes even know when that's kind of happening so it's a condition that's kind of always there and these symptoms are always with me and I think that the depersonalization and derealization is kind of there on one level always but it will flare up with the shifts in the activity and then as it shifts it makes me more and more sensitive to greater and greater climbs towards the threshold so it really is kind of like a feedback loop that is Yeah it's quite difficult to control but I must say that actually in this work, this kind of creative work in which we've been exploring the aesthetics of motion misperception and I've kind of integrated that from something that I was really kind of ashamed by in some ways because it would often be something that I'd have to hide and persevere through because it was often derailing my productivity and you know the very capitalist world we live in and it's very hard to kind of remove the hooks that it has in you in terms of missing meetings or just getting behind on things. And this is the first time of really integrating that into my life, I think, and into my identity, but also creatively to look at it as a subject of curiosity and fascination. And I must say that that's been one of the single most transformative and cathartic parts of this entire process.

[00:42:51.278] Kent Bye: Yeah, that's really helpful because I have my own experience of motion sickness and motion sickness triggers and I have my own awareness that I've cultivated where I think there is often a delay between when something happens in VR when people actually experience sickness and so they won't be able to identify those triggers early but I can identify that discomfort early enough and I've actually pushed way past that threshold to the first time where I actually got motion sick and then continued on and ended up vomiting for the first time that then I knew, okay, this is a new level of understanding where that cliff is and going over it in terms of motion sickness and what I can handle and what I can't handle. And if I am sick, then when I should stop and when I should take a break. So I can definitely identify with my own experiences of motion sickness. And also just in terms of this experience that you've been able to kind of recreate some of those different aspects as well for people as they go through. So. Yeah, I don't know if you have any other things to add in terms of as you're working on this project and creating an experience that is helping to empathize with this type of depersonalization, de-realization for Jamais Vu.

[00:43:57.738] SPEAKER_00: Yeah, I mean, I guess just from the creator's side of things, I think it has been this really, like this whole project of, because this is, of the back of this exploration, we're creating three other, like the sibling works, I guess, that are looking at the ways that the headset works and Ben's brain works and trying to draw these parallels in a way that is, I guess, like looking at the seams of the technology rather than leaving it as this unquestioned, like, seamless immersion tech. It's been a really like I think rewarding process of like self-understanding to create something like this and through that self-understanding I feel like both of us have a lot more language and like processes and ways of thinking about how these thresholds and processes, different ways of perceiving the world, are working. And so I guess, yeah, I hope with this project, but then also with future iterations, that we're able to express that in a way that resonates with people with a variety of different ways of perceiving the world, both normative and non-normative. I mean, I feel like there's so many different ways of being in and seeing the world that normative doesn't really seem like a useful term. But I guess, yeah, it's quite an amazing process to think so deeply about the way that we perceive.

[00:45:29.490] Kent Bye: You mentioned that you had worked on your PhD or got your PhD and that you were interested in researching different aspects of embodiment. So maybe you could speak a little bit about the research agenda, both above and beyond what you're already doing with these experiences, but if there's other things that your research questions that are driving you as you move forward in terms of embodiment within VR.

[00:45:49.012] Ben Joseph Andrews: Yeah, so my PhD is ongoing, almost finished as soon as I get a chance to finish writing it up. Essentially, yeah, that research is very much based on exploring the immersive condition of VR as being something that's not just kind of primarily audio-visual, I guess, or that that's where the experience is being kind of inputted from, but is is also exploring the non-visual aspects of VR. I think what gets lost in a lot of the conversations around presence, telepresence, is the inverse of that, which is absence. I think that a lot of the immersive condition in VR is navigating between these poles of being present somewhere and being absent somewhere. Yeah, so that research is very much interested in kind of like exploring what happens when the non-visual gets curated for because it is an intrinsic part of what it means to be in VR where we're in a hermetically sealed space and that means that we are somewhat separated from our environment and obviously mixed reality is kind of another part of that continuum but so that's kind of what has driven a lot of the earlier work that we've kind of done and definitely informs this and then with Turbulence and the kind of the ongoing project in its future chapters and not just Xiamen who was the first chapter is really kind of exploring motion misperception and kind of almost like deconstructing how the headset works, how it's constructing its sense of space, its sense of continuity, its sense of orientation, how it knows where you are in space in order to refresh its images in a particular way, and the kind of like eerie parallels that has with the human inner ear and the kind of like anatomy of the vestibular labyrinth. And in a lot of ways that's really what's drawn me back to VR and has really renewed my fascination and interest within it as a medium is that it actually helps uncover the processes in which we actually are understanding and perceiving and experiencing movement in this very, very latent level which is very difficult to see and to touch. And in a lot of ways, by deconstructing the way that the headset is working, you can really get at this glimpse into this mysterious world that's happening within us. And so I think that that's kind of like a lot of the research questions that are stemming from this project is kind of around how we perceive motion, how the headset perceives motion, and this kind of this interesting middle ground between the technological and biological anatomies of movement.

[00:48:28.217] Kent Bye: Awesome. And finally, what do you each think is the ultimate potential of virtual reality and immersive storytelling and what it might be able to enable?

[00:48:38.650] SPEAKER_00: I should have prepared for this. I mean, I think what virtual reality is really fantastic at is this expansion of ourselves beyond our habitual selves, right? The projects that excite me the most are those that allow us to accommodate for a different way of being in the world and with the world than we would in our habitual reality. So whether that's purely visual or it's through like multi-sensory performance or whatever that might be, I think this is a technology that is exceptional at breaking down what we think the world has to be and how we have to relate to it.

[00:49:23.682] Ben Joseph Andrews: Nice answer. I think I'll keep mine very short because I think that by revealing the way that we perceive the world, I think that VR has the opportunity to shift the way we perceive the world.

[00:49:36.960] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader Immersive community?

[00:49:42.085] Ben Joseph Andrews: I guess just, yeah, thank you to you, Kent, for all the work that you do and promoting all the voices of creators but also of, you know, raising a lot of really important topics within the industry and particularly your series around access, XR Access, is something that we're really interested in and particularly around vestibular access because you know the headset is a fundamentally vestibular device and that's something that we are really passionate about and particularly with the projects that we move forward with of really working out how to address that and kind of do it meaningfully.

[00:50:14.605] Kent Bye: Yeah, that's probably the one thing that people have in terms of accessibility is their tolerance for locomotion within XR and having different accessibility options of different locomotion options. So there's a lot of universal design principles that people can understand the broader implications of accessibility for how when there's more options, it's just better for everybody. who want to have different modes of experiencing different experiences. So yeah, that's a good point of vestibular access of including that into that because that's actually a lot of people may have different accessibility needs with an XR where they may not be self-identifying as that, but it's something that they definitely experience motion sickness and need those different features to really truly enjoy the medium.

[00:50:55.858] SPEAKER_00: Totally. I guess I also just want to say thank you to the industry and community generally. I think like one of the things that I didn't say before about making that shift from film to XR, like it's just so exciting to be in a community that's so open and generous and excited to experiment and do weird and interesting things and share knowledge and I think that's just such an important part of the creativity that we all have as an XR creating community. So yeah, just wanted to say thank you for that to everyone. And yeah, that I hope that that's something that we can hold on to as the industry matures.

[00:51:35.200] Kent Bye: Awesome. Well, Ben and Emma, thanks again for creating this experience and for helping to break it down and talk about your own condition of the vestibular migraines and how you're able to use the medium to help to both explore the experience of that, but also to help understand yourself and how your brain is working and all the other work that you're doing to kind of explore these different other aspects of embodiment and the mismatches of our embodiment in XR. So yeah, thanks again for joining me and help breaking it all down. So thank you.

[00:52:02.659] SPEAKER_00: Thanks, Ken. Thank you so much, Ken.

[00:52:05.136] Kent Bye: So that was Ben Joseph Andrews and Emma Roberts, and they created a piece called Turbulence Jamais Vu, which ended up winning the Iffy Dock Lab Award for immersive nonfiction. So I've a number of different takeaways about this interview is that first of all, well, I was completely fascinated with this conversation and the ways that you could build different dimensions of empathy for these different types of conditions. One of the pieces that run the third place award at Venice this year was called Emperor and that was exploring this condition of aphasia where your language is jumbled you're not able to communicate and you kind of mix up words in a way and so aphasia can be one of the symptoms of the vestibular migraine experience that Ben experiences. So yeah, just my connection to motion sickness is that it can be like a buildup over time, a cumulative, and then at some point you kind of fall off a cliff. And so very similarly, there's different triggers that Ben has that can cause him to go into different symptoms that flare up even more with his vestibular migraine condition. So yeah, just the fact that you can use the mixed reality to start to explore some of these different embodied experiences and really diving deep into embodiment and virtual reality's modulation of that. There are some dimensions of like nausea that have like, when you move your head to the right, you actually have the visual feedback moving to the left. And so even just that alone of y'all rotation, that is not corresponding to what you're expecting to see that can create a sense of nausea. And it's a huge trigger for me. So when I was going through this experience, I was really trying to like be very still and move very slowly. I mentioned this during the interview and he's actually said afterwards that he was thinking about, Oh, there actually could be like, if there's a camera that was above you, you might be able to preserve some of your head positionality, but because basically they're, they're flipping the orientation. So you're trying to swap your left and right hands, but when you're doing that, then you're also swapping everything that's happening as you move your head around and can be very nauseating for folks. And so, This is one of the pieces that if you weren't careful, you could walk out of it wrecking your day with the rest of what you're trying to watch at the doc lab because it can be inducing of motion sickness. But this is one of the first cases that I've seen that's really using that affordance of the motion sickness inducing to serve the narrative of kind of reinforcing what he's actually experiencing in his day-to-day life through this condition of vestibular migraine. And I absolutely love just hearing more about the experiences of depersonalization and derealization. and just the ways that you can start to use virtual reality to invoke some of these different types of experiences. It's kind of a miracle at all that Ben is even involved in becoming a immersive creator and extended reality artist, given the fact that the medium itself can be so triggering to this condition. But yeah, highly recommend folks going and checking out the interview that I did with him about Godwanna, which is a a really brilliant 24-hour durational take of a simulation of a forest over 100 years that starts to metaphorically and symbolically show how a forest changes with climate change, both visually with CGI, but also just like this really rich sound design of how the wildlife will change as more and more of these different species die off with the impacts of climate change. So you get this real visual experience of how the soundscapes change with the impacts of global warming. So it's another project that they worked on that I had a chance to see and cover earlier from Sundance in 2022. But yeah, this piece, Turbulence Jamevu, really impressed with this conversation and took a lot of weight out of it. And I could see why the jury awarded it to the IFADOCLAB Award for immersive nonfiction. In fact, their jury statement is, our first award goes to a mixed reality project that couldn't be done in any other way. It uses simple techniques to drastically adjust our perception, enabling us to experience the world through someone else's eyes. This confronts societal notions surrounding illness and questions the dependence we have on our own unique visual realities. So, yeah, just a really provocative piece in that way. And exploring different dimensions of embodiment and how those different types of experiences of embodiment can start to support stories that are being told within Turbulence, Jean-Méveux. So that is my coverage from IFA DocLab 2023. Hopefully you enjoyed these 19 different episodes. I'm going to have one episode from the Sundance New Frontier selection this year. Actually, there's really only one XR project from Sundance New Frontier that I'll be covering here on Tuesday. And yeah, if you do enjoy this coverage, please do consider supporting me on Patreon. I also just dropped a lot of money on the Apple Vision Pro. And yeah, if you're looking forward to that coverage, then I can certainly use a lot of support and help as well. It's not a piece of consumer hardware that I expect very many people to be able to just afford outright. But yeah, I'm looking forward to exploring what Apple is doing within the context of spatial computing and the Apple Vision Pro, their first-gen entry into the XR industry. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.

More from this show

Menu

#1344: “Turbulence: Jamais Vu” Wins IDFA DocLab Immersive Non-Fiction Award with Embodied Mixed Reality Doc on Depersonalization & Derealization