Chloé Lee’s immersive art piece Temporal World: A Haptisonic Virtual Reality Memory World uses a custom-made haptic vest that is intimately integrated into the ambient sound design of this point-cloud, spatial exploration around memory. Lee wanted to incorporate open-world wandering, but also explore the fragmented nature of her own immigrant identity and disjunctive blending of memories across multiple cultures. She uses the sparse point clouds to mirror aspects of this immigrant experience, while also innovating on hand-gesture driven locomotion schemes and an overall “haptisonic” fusion that’s incredibly immersive and compelling in a way that amplifies the experience beyond just the visuals or sound design alone. There are more coherent audio spheres connected to the memories that you can portal into via discovering objects, and my biggest complaint was feeling a bit too lost in the second portion where I felt like I was not succeeding in discovering everything. But Lee seems content in preferring this mode of open world exploration without being connected to the desire to be a completionist. Perhaps this feeling of being lost or aimless wandering also serves to replicate dimensions of a sort of exile that immigrants may face, and so thematically it may be resonant with her deeper design intentions and intuitions for how this all came together.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing and the structures and forms of immersive storytelling. You can support the podcast at patreon.com slash voicesofvr. So continuing on my 24-episode series of the different immersive experiences at South by Southwest, today's episode is about a piece called Temporal World, a Haptisonic Virtual Reality Memory World. So this is a piece by Chloe Li, who actually collaborated with the Research Institute to create this haptic vest that you're wearing. And so it's kind of a custom bespoke piece of fashion, really, that you're putting on your body, and it's vibrating across these different dimensions of your shoulder, your back, your arms, and really seamlessly integrated into the sound design of this immersive experience. So it's a piece that's exploring dimensions of memory and aspects of the migration process of moving from place to place and the sense of home and exile, but also the fragmented nature of memories of these different places. So you're exploring around a point cloud representation of these different worlds. You're interacting with these different objects, which then launches you into these different memories. And there's a sound design of those spaces as well. And there's also like a gesture where you put like a triangle in front of your face with the leap motion that's detecting it and kind of like locomote you through the space. And as you locomote and move around the space, then it gives you like this haptic feedback of you moving through the space. So yeah, it's a piece that's really exploring different dimensions of memory, migration, and this fusion of haptic experiences with sonic integrations and virtual reality storytelling as well. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Chloe happened on Wednesday, March 15th, 2023 at South by Southwest in Austin, Texas. So with that, let's go ahead and dive right in.
[00:01:56.968] Chloe Lee: Yeah, hi, my name is Chloe Lee. And well, this is technically my first VR project. And my background is in the arts. And I have also background in film production. And so here, I ended up working with a research institute to make a haptic jacket. I did all the photogrammetry in this world and all the sound work and all the haptic composition, and I worked with a developer who's amazing, Lucas Martinich. And yeah, I think in this project I conceptualized this based on my own personal experiences, having moved around to different places, and then also my family's history of migration. and originally the project, Temporal Worlds, was a research project on memory. Specifically, I made it personal because I think I had been working in these themes before in other versions and other projects, and this is the first time I actually, I think, just went into a new medium on this scale. For this, I had dabbled too in some 3D modeling, a little bit of animation, and I have also worked with a lot of cameras, so I I think living in New York and freelancing, I had always been one to quickly pick up the programs and jump from editing to color correction to kind of juggling it all. And in this project, because the team is so small, I ended up playing a lot of the roles. And I think working with a developer really helped expand the project, too. And then working with the Institute and having the knowledge in different fields. I also hired a fashion designer to help me design a jacket that functioned well for what I needed. MARK MANDELEYER.
[00:03:42.913] Kent Bye: Yeah. Maybe you could give a bit more context as to your background and your journey into making this VR piece.
[00:03:48.430] Chloe Lee: Sure. So I am American and Canadian. I was born in California. And so I made this project in Berlin. So I was living in New York. And I think this kind of sets the context of moving around to different places and the themes of migration that I'm exploring. Basically, I wanted to explore memory. And the project idea first came about when I was in Singapore. My mom is Singaporean. And there, when I had visited, mainly to get closer to my grandmother, I start to hear stories about how the country used to be, what the landscape was, that it started off mainly as villages, a fishing village. And especially in the last few decades, they had, since the 80s, the industrialization had changed the cultural landscape and the way at which people move. And so a lot of times people, when I talked to them, were talking about how they couldn't relate to the area that they lived in anymore. They couldn't root themselves and identify with their landscape. And I learned that some of the buildings around them if they're 30 years old and iconic, were coming down to build up higher and bigger. I was thinking about how development affected these people. Even though I think it's a bit nostalgic to look back in time and want something in the past, I think that realistically, people had better living conditions. Most people were happy about the change, but there were certain things they wanted to have a say. you know they wanted certain places preserved but they still wanted to move forward and some of the interesting people that I met were actually younger and so they were turning old dilapidated spaces instead of building them into something entirely different they kind of took it into their own hands to maintain them as community spaces or they would mash up like Chinese opera with electronic music and I thought this was a really interesting approach. And for myself growing up, I was always interested in my family history. I was always one asking too many questions and annoying my parents and my grandparents. And I think I'd always been interested in preserving memory. What does it mean? What should be remembered? How should we remember it? And during the pandemic, I got my hands on a headset and I had started working in 3D worlds because of course I wasn't out with my camera getting new footage. And so I thought about using VR because when you're talking about development and progress, I thought the idea of having VR, the history of that piece of technology tied to like this need to advance and progress was an interesting contradiction when you're talking about preservation. And I wanted to explore that contradiction and I don't necessarily think I have the answer with, you know, the conflict in that using this technology that represents the very thing that causes or the mentality that causes places to change like this. But I thought it was just an interesting way to explore that question, to bring it up.
[00:06:43.765] Kent Bye: So yeah, that's a really great background and context in the journey up to getting into VR. And I'm wondering, how did the haptic and the haptasonic aspect of this project come to be? Because I see a lot of VR projects, and I've seen point cloud explorations where you navigate, but I haven't seen as much seamlessly integrated haptic plus the sonic experience that is tied into an immersive experience. And I thought it has actually worked. really quite well for me in terms of just like the type of immersion that I got even though it was like very sparse visually with these point clouds and as I'm navigating them around I've been in a number of different point cloud worlds and I first of all appreciated the interactivity of this point cloud world that I did feel like it had this dynamic interactive nature and then these kind of portals that I'm going into through these objects but you know for me what really stood out was this haptic integration that was I don't know quite how to explain it because it was basically like stimulating different parts of my upper body because I have this jacket on with this haptic suit, this haptisonic haptic suit, meaning that it's also tied and correlated to different aspects of the sound design that you have. And so there was a multimodal experience of hearing the sound and feeling things in my body in a way that was really quite provocative. So I'd love to hear how that dimension of the experience came about.
[00:08:00.256] Chloe Lee: Sure. Thanks. Um, so the haptics came in later. So I had already proposed this project for a Fulbright scholarship during the pandemic. So the idea also, I was trying to externalize this in internal space. And I think part of it was because I was in lockdown in this space. And so when I went over to Berlin, the Fulbright scholarship had me collaborating with a research Institute matters of activity, and they're really great. And A lot of it is professors from Humboldt and other parts of the city who come from different disciplines and they work together to address similar themes and issues. And there I found a haptic prototype while I was in a residency that was hosted by them. And this prototype, they call it a haptic handbag that was developed by Maxime Lecave, Yoonha Kim, and Fardin Golomi. And They had developed a flowing open garment kind of similar to a vest. It's a traditional Korean hanbok. And I had tried it on and I had already developed soundscapes for my project. And I thought that this was actually very interesting because it translates sound. to frequency and it's natural frequency. Every sound has its natural frequency and with the frequency and the magnitude it has a natural vibrational pattern in the system. And so I developed the system more for my needs and I also, since it was a personal experience that I was trying to emulate in the visuals and in the sound, everything comes from my field recordings. It's all 3D scanned places using photogrammetry of places that I've been in Berlin, also New York and China. So New York and China were all from my archives, so I had no intention of making those places. into 3D interactive spaces. They're just something that I tried to assemble and they turned out as they are. And so I wanted something similar with the jacket where it emulated how I experienced vibration. So I rewired the system entirely and I even went to this great ensemble with orchestral instruments and I placed myself near our speaker and it was an over an hour concert and I just was very conscious of how I experienced vibration depending on how the sounds changed. And so that went to my research. And of course, I had read a lot on haptics in terms of this technology and how people experience sensation, hands being the most sensitive. And I even developed haptic gloves for that first version. But when I was looking at it from the science point of view, the actual sensations, the visceral experience wasn't what I was going for. So that's why I went to the show to see actually how do I feel vibration on my body? higher frequencies were at the shoulders, the neck, and lower frequencies tend to be at the fingertips or in the gut. A lot of people experience it that way, at least from the people that I've talked to. So just to clarify that you you went to a concert with the haptic vest on and or you're just feeling it with your body feeling it and taking mental notes and then trying to Recreate this and since the sound is directly connected to the haptic jacket I needed to have a very specific motor placement and then how did I create you know vibration to the core for instance and and I had a limitation on the amount of motors. So the first version had 42, this version has 36, only because I took away the gloves. And so how do you place those to create sensation through the body? So I used the idea of points of relation. So sending vibration through the bones, not necessarily needing to put a motor where I wanted someone to feel vibration. It was just the area and how vibration was sent through different parts of the body. I didn't follow entirely the high-frequency, low-frequency, top-down that I just described because the end product wasn't working the way I intended when I did that. So I switched. Actually, I put the low frequencies also at the neck area because then it would match the gut and then it would send vibration through the core of the body. And so there were things that I tweaked. But as a model, I tried to create this end experience or this simulation with that end goal in mind, and it didn't necessarily mean that the motors were placed in that. exact way on the body. High frequency motors were not placed necessarily on the highest parts of the body. But I also composed the soundscapes while wearing the jacket. And so I was also using spectrum analysis, too. Since it was all based on frequency, I could also visually see what I was composing. Most of the sounds were made that way, the soundscapes, the music. Not all of it. And the generative part, where the hands, I'm not sure if you noticed, but there's a few things. It's the movement that changes one instrument sound. There's all these instruments I made for the jacket, while I was wearing the jacket. So when someone moves, one of the points that changes the lower sounds is your head, your head movement. Another is your hands. So when you move your hands around, this is another instrument. And depending on where you walk, because in the second half of the experience, you generate your own path after having followed my path. Where the ground builds, that is another sound also. So all the sounds in the world were composed with the jacket on.
[00:13:27.128] Kent Bye: Okay, wow, that's really quite interesting the way that you were developing this because the end result for me was that it was a really unique and visceral embodied experience that felt like it was almost like a slipstream training my body into a certain resonance that it deepened my experience of the piece in a way that like it's very subtle in a way that it's Preconscious it's like there's things that are happening into my body that I can't quite put a finger on because it's a new sensation in a way that I was being Stimulated in a way, but there was a correspondence to the sounds that I was hearing and so it felt like also another multimodal resonance at that point and
[00:14:06.895] Chloe Lee: Also, I'll add too that I think it ended up being a sound installation for the body a bit. There are certain parts where the haptics were specifically giving you information, like there's a teleporting or jumping feature that sends this full body vibration that I was trying to emulate moving through the air quickly, so air turbulence. But because this is an analog system, it was interesting. There's a naturalness to it that you're talking about. The sound really corresponds to the vibration. It makes sense. In the next version, I'm planning to have it a Bluetooth integration, which means there's total control. If you were to grab something, for instance, you would feel vibration on the parts that you're interacting with an object, for instance. But because it's analog like this and it's only corresponding to the sound, I didn't have that control, so it's a bit of a trade-off. Certain parts would give you some navigational information about your environment, but mostly it was more expressionistic.
[00:15:05.955] Kent Bye: Yeah, I don't think at any point I really necessarily realized that my body movement was having any direct impact onto the sound or the haptic experience. I think it's mostly because when I see a correlation between my body and some feedback of that, it's either have to be like explicitly being able to kind of like wave my hand and hear it immediately, but I don't know if it was like the lower frequency or it could also be that I just wasn't expecting it. Like in most experiences of physical reality, my movements aren't corresponding to the music in any way. And my memory of it, it's a piece about memory, and so it's like I saw all the experiences that I hadn't seen before, and there was a sense of like the ambience, it's kind of like ambient music. How would you describe the genre of music? Was it sort of ambience or what?
[00:15:54.862] Chloe Lee: Yeah, it's ambient music, I would say. I would say soundscapes, ambient music. It wasn't meant to feel like it was specifically focused on it. It was more to create atmosphere around the spaces and trying to differentiate spaces. I really, when I was making it, I had to carve aside time and I tried to go into the area or that memory recount that event as I was making it or sometimes it would just be the mood that I would be in while I was thinking about this and I think intuitively that maybe just came out and when I got the soundscape complete it would feel right to me or else you know I didn't include it there were some that didn't feel right it was intuitive the way that I created them.
[00:16:38.720] Kent Bye: Yeah, yeah, and also just generally my experience of ambient music is that I don't associate ambient music with my agency And so it was just sort of like something in the background But I guess the other really cool thing about this piece was that you had a little bit of a unique locomotion mechanism where you're making a gesture with like a triangle in front of your face with the leap motion type of tracking that would able a to allow me to teleport and whenever I would teleport I would feel like this jolt in my body and I felt like that was really quite effective of giving the sense of you know the impacts of moving through a space in a way that mostly when I locomote through a space virtually I don't have any sort of haptic feedback but to create a discrete teleport mechanism that each time I teleport I feel a haptic experience gave me the sense of that I actually was moving through a space in some way that it gave me like a kind of close the loop for what I might expect when I walk through. In physical reality, I feel like wind resistance or some sort of bodily experience. This is a little bit more of like, you know, it's a little bit like a shock type of haptics because it's like vibrating in a way, but it's not like electrical shock, but more of like a vibrating and some sort of resonance. And so, yeah, I felt like that locomotion mechanic as well as the haptic feedback also deepened my sense of being embodied into the place.
[00:17:52.050] Chloe Lee: Oh, that's great. Yeah, I mean, that's exactly what we were going for. It was also the experience when I moved to Berlin, coming from New York, I came from this frenzied state needing to constantly move. And I think in Berlin, I was learning how to slow down and notice details in my environments. So this idea of slowing down. directly correlates to this air turbulence that when you're moving at quick speeds which the fastest that you can move in the world is through this gesture to move through space quickly and you actually see the environments disintegrate and then rebuild and you feel that disruption that when you're moving very quickly everything is disrupted. And this was to try to show, to try to get you to slow down a little bit. It's actually not possible to move even faster. There's actually a limitation put on the distance that you can teleport. So you have to make many of these gestures with your hands. in order to get from point A to be wherever you want or decide to go. And I think that maybe this could have been frustrating for some people, but I think in talking to some people, I think they initially wanted to go faster, but then they realized that this was how it was meant to be and that they actually became accustomed to it and eventually come to like it. Or at least I hope that's true.
[00:19:15.587] Kent Bye: Yeah, well I had an experience of the piece and I wanted to ask a clarifying question to see if it was architected in the way that I eventually came to conclude that it may have been put together, but you have this pathway at the beginning where you're going down a path and you have these objects that you can touch and then when you touch them it sort of activates something, but There was a little bit of a delay between when I touched it and when it dissolves everything and takes me into another reality. And so there's a moment where I was going through and touching things, and then I saw someone I wanted to go see, like a waypoint far away. And I was like, oh, I'm going to go there. And then 10 or 20, 30 seconds, maybe even a minute after I had touched a thing, it teleports me into another reality. And then there's a little vignette of a memory. And then once that memory is done, then I get back into that path where I was. And so I'm concluded at that point, whenever I touch these objects, it puts me in a portal, but there was a little bit of a time delay. I don't know how long it was, but it wasn't like an instantaneous, like I touch this thing and it transports me. So is that a deliberate architecture where you're trying to give people the feeling of touching something, but then slowly going to another place?
[00:20:19.341] Chloe Lee: Yes, I wanted there to be a slow transition that they had to wait and anticipate versus having things instantly appear in front of them that they had to notice. And that's also why the objects are a little more discreet. They're on the ground and they're kind of everyday objects that maybe you might move past. You know, if you're very busy, you won't notice it. But I also think it's in the everyday that there are these meanings that these objects are laced with. There are also objects that are depending on, well, like there's a flower in there. I mean, people have different attachments and associations. This is not some rare object that someone has never seen before. And so I really wanted to keep it to be simple objects that people could place their own meanings into. And this also goes with the atmospheres, with the sound, that they're left a little open. Yes, you hear voices, but you can't quite generate a narrative from it. It's not like I'm talking about being in New York in 2012 and this is what happened to me. It's more the feeling that I have You know, maybe it was meditative at that moment and there someone relates that feeling of that meditativeness to an entirely different experience that they have. But essentially there are many of these objects that someone can discover and people have to kind of wait. It's things that slowly unravel in front of you that you have to be patient for. It's only if you notice it. that maybe you can gain something from that. And there are endless things like that all around us. We could choose to talk to this person on the street or not, or there's another person on the street, and this could lead somewhere else. I think that eventually we're still developing it. There'll be some more objects, but you can go from place to place or touch different objects and reopen some of the spaces. The idea there, too, is When you go from one place to another and then you come back to that same place, it's not the same, but we want to develop it where now something that was already there before has another layer to it. And you now see that place differently than you did the first time. So it is about noticing nuance. In order to do that, you have to slow down.
[00:22:27.712] Kent Bye: Interesting yeah, I didn't necessarily pick up that I needed like my speed was impacting anything I did notice that when I was moving fast sometimes the whole world would Actually, I noticed sometimes the world be bouncing up and down even more aggressively but I didn't tie it to how far or how fast I was moving and I did have the experience in that first initial path where it was the first object that I had pressed, and then I was like, oh, I'm going to go over there. And then right as I was starting to go over there, then I was teleported into this other reality. And I did feel that sense of yearning to be in a certain place, but then it was ripped away from me. And I don't know if that was part of the intention of the way that you designed it, is to talk about this migrant experience of being in exile in some way, of leaving one place and having it erode away and going into another place. Yeah, I'd love to hear a little bit more about how you were trying to translate that exile-migrate experience into different spatial design elements in this piece.
[00:23:23.115] Chloe Lee: Yeah. I'll add, too, about wanting to go see this place and then not being able to, or also not being able to fully get the resolution you want up close. You can't make things out. But then when you move around to the other place far away, you could see from afar clearly. And then when you get closer again, it disappears. more about memory and trying to capture something in its original form, but you're never ever really able to do that. And there's a bit of frustration in that, but it's deliberate. Sorry. But the migrant side, so for instance, I use fragmentation. I tried with different places. Like there's my family's ancestral home that someone may or may not find. And there's also my apartment that I lived in for six years. Out of the 10 years that I was living in New York, These two places, I assembled the photogrammetry through my archive. So this wasn't using 360 video or anything, it was just video walkthrough I happened to have. And New York, especially surprisingly, was extremely, extremely fragmented. I think it's through the fragmentation visuals, really. It's the glitchiness and the holes, what you can't see, that represents the disconnection. Like, New York, I called it home for 10 years and I never imagined actually leaving. I thought, you know, this when I moved there, that it was indefinite. I was also young though, early 20s, kind of wide-eyed. But then I left about a year and a half ago to do this project in Berlin, and somewhere along the way I woke up and I thought, you know, I don't feel really so much attachment to New York anymore. There is attachment and fondness to certain memories, but as far as the idea, people ask me, are you going to go back? I don't feel that, and it's strange that I feel disconnected to this place, because at one point I never imagined that I that wouldn't be home. And so the place that I lived in the longest in the project is actually the most fragmented. I don't know if you managed to see it. It's fragmented staircases that the scan wouldn't even assemble a floor. And so this one is actually probably the most abstract place in the whole piece. And it's just spiraling staircases from one landing to another. And the soundscape is equally as disruptive. There's hand drums at one point and a disconnected weird glitchy phone call. Yeah, so there's varying levels of connection. I thought that that was actually interesting that somewhere that I had connected to the most. was actually represented as the most disconnected place. And so I was thinking also about migration in general, like what's happening around the world, people being forced to leave their home. They don't have a choice. I mean, for me, it's entirely different. But I was just thinking about that feeling and that actually that could be the largest disconnection and how people then create community. They're forced to, though, you know, but they find it in that constraint.
[00:26:15.120] Kent Bye: Yeah, well, regarding whether or not I'd seen one of those things, I'm not sure. But I wanted to ask a little bit of the overall structure, because my sense was that there was a first part with the path where I was going through and getting these objects, and then a second part, which was much more like a flat plateau that I could see where I was walking, but that there was less objects that were being triggered into these different scenes. Is that accurate, that there was two major parts?
[00:26:38.806] Chloe Lee: There were two major parts. Yeah, that's right. So the first part is, it's more or less the introduction to kind of let someone have a little more space to learn through this path that I've pre-created before bringing someone into this bigger space where it's open-ended, where the path then is created by where they walk. So it's essentially they create their own path, but it's a vaster world that they come into and there's more objects to be activated and discover. The first part is kind of a more elaborate, just one memory. And that gives them the option to, you know, engage with touch. Because some of the things are on the ground. I wanted people to move a little bit differently. So I think some people aren't naturally inclined to do that. So there are objects on the ground, so you have to actually bend down and you have to touch it. Because sometimes people would try to gesture to touch it, like do the teleportation. But people have different intuitions in how to move, and so that was interesting to design for, because some people had opposite inclinations. And the second part was about getting people to walk. It was a question for me on design, like, how much do I guide somebody? Like, how much do I tell them what to do or how they should navigate the world? I think some people naturally wanted to walk around and meander and were okay in the second scene not being told. There's multiple directions that they can go and the path generates underneath them. So they're essentially mapping the ground in the physical space. They can actually look back after walking a few steps and see the ground and everywhere else it's black. they get to choose where they go in the second scene. So one object will open up one scene and then within that scene there'll be another object that they can touch and then go to another scene. And they're actually in some cases sent down and then they can look back up and see the last place that they've been afar and then they see the path that they just traveled down.
[00:28:38.166] Kent Bye: Oh, OK. That's interesting. So if you open one memory, then there's another memory within that memory? OK. I didn't discover those. So my experience of the open world was that I had first learned the mechanic that if I touch these objects that go into these other portal spaces, so then once that portal space was open, like previously, I learned that then I go back out. So then I was assuming that it was going to like take me back out. But my experience of that open world space was that I don't know if there was any memories that I got into in the second world because then I would go to the edge as far as I could go and I would like do something and then the big structure that was in the distance would slowly dissolve. And I was like, okay, that was at least something that I was able to do. And then I walked all the way over to the other one and then It dissolved and then there was nothing else that I could see that I could do except for find a thing that I had already touched before which was like a computer monitor thing on the ground and I touched that and when I touched that then it completed. But there was a bit of a confusion as like I was wandering around an open world without finding any of the next portals to go into. So maybe it was a portal that I'd opened up in the It was intending to go into the next portal, but it was a little bit of like the affordance was that you have this big sparse point cloud space, and then there was one kind of more higher resolution object on the ground that was differentiated. And when the memory came up, it was all uniformly created in a way that I didn't see any differentiation of an object that would indicate for me that there was another portal, at least from what I perceived.
[00:30:04.903] Chloe Lee: Interesting. Yeah. So there's two options. You can go back. There is the object that you discovered that sends you back, but there's at least one or two other objects per scene that can take you somewhere else. So, yeah, I think I also thought some people don't even make it into the second scene because they're standing still in the first scene. And some people are completely content with that, which is totally fine, too. I guess my intention I don't have an expectation. as to like, they must go here or you must see this. It's for open-ended discovery and exploration for someone to roam through the worlds. And I think the goal too, it wasn't, even though there's two parts, like you're saying, it's like the first part, you could see it as like level one, level two, you know, like I really didn't see it as that. It was, I just wanted to guide someone enough to be able to navigate around the world. And I think it's okay if they don't get to the next part. I think unlike an experience, I was trying to stay away from the idea of having a direct objective. Like I don't have an expectation in that sense. I think I would like personally more experiences like that. And so that's, I wanted to make something. like that. I think it leaves more space, less pressure for someone. But at the same time, I think that many experiences have this objective and we're conditioned to think in that way where, OK, what's the objective? What am I supposed to do? How do I get to the next part? I want to move the fastest. Actually, I won't say who this is, but, you know, I'm asked sometimes it's like, well, how do I get through it faster or how do I get to the end? You know, and I just say it's OK. You know, there's no it's not a definitive end, like wherever you get to is fine. I wanted to make an experience that felt open-ended like this.
[00:31:45.867] Kent Bye: Yeah, and I think part of your intention is to create these little disjointed types of experiences with the delay and whatnot. And that's part of the intent. And I certainly experienced that. And for me, I don't know if I'm doing it right, or if it's broken, or if it's like there's a confusion that gets introduced. And if the intention is to create confusion, then you succeed. But then at that point, I don't know if I'm missing something. Like the second part with the open world, that felt like a little bit more of like wandering through a desert, metaphorically, because it's pretty wide open relative to the previous ones. That's like trying to find where's the thing to trigger the next happening otherwise if I'm sort of wandering around a virtual space without anything happening then a certain point then my brain is kind of like desiring a sense of like stimulation or novelty or it feels like I also don't want to be missing something. I want to find out where the arc was going. All these experiences, I'm trying to figure out what the story is and how it's being told through the spatial medium. And then if there's a confusing aimless wandering, then feeling like I came out, was it broken or did I miss something? Because sometimes that does actually happen when it's broken. And so it's like, oh, was there supposed to be something that I was supposed to find? It's one of those challenges of creating open world exploration, but then at the same time, if there is a desire to have, for me at least, a sense of knowing that I'm experiencing all of the, like, basically I'm a completionist. I want to see everything. And when I go through something and I don't feel like I saw everything, then I feel like, oh, what did I miss or where was it at? So it kind of feels like, did I see everything in order to kind of really understand what was there. So I guess that's maybe where I'm coming from as a as an XR journalist trying to like see these different pieces and understand them to their full capacity.
[00:33:30.765] Chloe Lee: Of course, yeah. And I think you're not the only one who has that desire. I think it's not possible to have all the same experiences, especially the direction in which the memories open up to. It changes for every single person. And so they reveal a little bit differently. So I don't think it would be possible to see everything and experience it in the way that someone else might either. Yeah, and I think that eventually, too, when we put in this part of the development where you come back and there's this other layer, it adds a certain complexity that, especially if it has this time limit, as it does for exhibitions, it's not going to be possible to see everything.
[00:34:13.911] Kent Bye: Yeah, but when I did open up the portals, I did appreciate being transported into this memory-like fragment that had a spatial audio design that also situated me more into that location. And it was fragmented enough to not have a full context as to what the place was or how it was connected to an overall whole, but just in terms of the different affordances of the medium to be able to go into those memories I feel like I'm at least situated in the sound field of a place, even if it's not the full visual fidelity of a place, if that makes sense.
[00:34:45.125] Chloe Lee: Yeah. And I think that this will be developed more. But for instance, sometimes you don't have a visual in a certain area. There's a hole, per se, that maybe there's sound there. And with that, there's vibration there. that's something that you don't see, you feel, or you hear. And I thought this was also something interesting to have in VR, something that's a visually dominant medium, and it's the same with society, and that maybe it's very sparse, a sparse world, it seems like it, but actually there's some layers there that can be felt or heard instead, or perceived in some other way than what we would expect. So we'll continue to push that in the next version.
[00:35:33.613] Kent Bye: I think it's because of that visual sparseness combined with such a high fidelity, interesting haptic and sonic experience that, for me, made it even more immersive in a way that felt like I was really transported into another place. And you had mentioned that you were collaborating with a fashion designer to make the jacket. Maybe you could elaborate on that, because that's another component where it was a haptic device, but it also looked pretty cool.
[00:35:59.548] Chloe Lee: Thanks. Yeah, so the designer is Bumezi, which is by Drew Blumenscheid. She's also based in Berlin and she's designing for fashion in Paris. And working with her, I knew I wanted to create a garment that didn't look like a haptic jacket. And the inner lining even shows that she advised me actually to have the inner lining that was something that tied back to the virtual world. And so what you see is actually a texture photo from my family's ancestral home, which is one of the scanned places in the world. And so when people are wearing it, you don't see it from the outside. So it's a bit of a trace memory. And that was kind of what I was going for, too, with the person who was in the headset, where they're just a trace. you see traces of your movements, you're traced in this memory. There are little traces left around that could go unnoticed. And so the inner lining of the jacket was also just a trace. If I was wearing it, no one would see it, but I knew it was there. And it was kind of the initial intention, this, you know, these ideas coming back to my own family's migration. And so that was just a little reminder as to like, How it started and for her the style of it is in a kimono fashion growing up She was highly influenced by this and so that's where that initial idea came from But I worked with her it told her basically what I needed what I was going for and so she was basically inspired by the story like my story and she tried to make a jacket that was Helping it function, yes, because the initial prototype, I think I wanted first, of course, a functioning piece that could hold all the motors, but also I worked to place the motors. Then I told her, you know, this is where we need this. I developed kind of the flow of the wires, and then I worked with a hardware engineer at the Research Institute. So it was their system, and I said this is how basically we should design it for this jacket so working with all three people together that's how we created the jacket together and then that was the prototype that's the first one I'm not sure you saw it that was like a neutral color but the one that you wore was I actually made that one myself So that's the second version. So taking what worked and what didn't work, and I tried to make that a new one. So the second one doesn't have the haptic, we have haptic glovelets also, just because the fingers, you know, the most sensitive. And so we had that version, but then it was affecting the hand tracking so much. And in an exhibition like this, South by Southwest, having so many people come through, I just removed that factor. So hopefully in the next version, I'll have some sort of haptic glove again.
[00:38:40.131] Kent Bye: OK, yeah, that makes sense. And yeah, I guess as we start to wrap up, I'm curious what you think the ultimate potential of virtual reality and these types of haptic sonic experiences and immersive storytelling experiences might be and what it might be able to enable.
[00:38:56.819] Chloe Lee: Sure. Well, I'll speak then what interested me about the technology was, well, the presence of embodiment, of course, using the technology to kind of reveal dynamics that are already in our environments that we can't see or perceive easily. I think that having these kind of visceral experiences is very powerful to elicit an emotional response that can I guess change people's minds or influence people in a way that other experiences may not be able to. And I think it's also just exciting to have this possibility, this medium, I think, kind of blurs boundaries. The fact that even the way that the work is presented, there's no clear defined way. And people are doing it in a variety of ways. I think that that is, in itself, disruptive. And because this is my first VR project, I'm just excited to see more of these different projects and see new ways in which they could be presented to a larger audience.
[00:39:56.356] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:40:02.015] Chloe Lee: I think there's one point coming back to your first question that I think just about the project and what is represented in the world, why it's that way. I had moved to Berlin initially. I chose Berlin over going to China, which I had initially thought that maybe I would do this Fulbright research memory in China because, you know, my family comes from China. And that's what I was initially thinking during the pandemic. But I chose Berlin because actually it's a place that I have no personal history to. And so I wanted to follow that idea of what does that mean to create that community and connection in a place with no personal history in terms of themes of migration. But then also just I think that this project, it was like just a reflection of how I relate to place, how people can relate to place. And China in there was more a framing of how do I relate to this place that seemed to dictate so much of how I came into this world, my family's history, and I've only visited this place for three hours, and why is it represented in probably the most archaeological way of all the places, and New York being this place that's extremely fragmented. I just, I think that this for me was an important part to explain what the framing of this project is, like the places that people see and the visuals that they see and the sounds that they experience. This is what they're seeing. These are the questions that I was trying to answer for myself.
[00:41:31.176] Kent Bye: Yeah, well, you had mentioned earlier that you have a very intuitive process for how to synthesize this whole multimodal haptic and sensorial and visual experience. And I think the end result for me, it was just a really deeply immersive and embodied experience that I found really quite compelling to move around in and explore around in. So yeah, I think there's certainly something there when it comes to tying all these things together in a way that thematically is exploring these concepts of fragmented memories. Yeah, I think there's actually historically been a lot of projects that have been exploring dimensions of memories in VR. So I think it's kind of a natural fit of having these representations of these memories in this way. So, yeah.
[00:42:11.689] Chloe Lee: I think I will add, and on that too, it's a bit, I wouldn't say disparate, but these places don't really connect with each other except for the fact that I have been to those or they connect to me. And I think in the past there's been some sort of, or at least I felt it, expectation that I, Talking about migration that I should speak about the Asian American diaspora being an American citizen having been born and raised there especially with the tension now in the media for representation of Asian people and Yeah, I'm not sure I like that expectation and so in a way showing all these places Berlin and just a dinner scene with friends and just my place in New York in that time is and including the family and social homes, kind of addressing kind of these different sides of myself. And it's not just about migration and being an Asian person and talking about migration and diaspora in that sense. But hey, here's just an everyday memory of friends. And it's not about me being Asian. You know, it's the multiplicity in who I am. And yeah, there are different sides that I identify with. So I was able to do that in a way without actually talking about it directly.
[00:43:22.247] Kent Bye: Yeah, awesome. Well, thanks again for joining me to help break it all down. So yeah, thank you.
[00:43:26.228] Chloe Lee: Yeah, thanks so much, Kent. It's great.
[00:43:29.269] Kent Bye: So that was Chloe Lee. She's the director of Temporal World, a haptic sonic virtual reality memory world. So I have a number of different takeaways about this experience is that, first of all, well, I really, really appreciated the integration of the haptic vest. It's a very customized bespoke piece of art that's really quite beautiful within its own right. It's a little bit of a process to get on just for onboarding offboarding But you know once it's on and you're able to explore around this world the way that it was integrated with the music It felt like you're hearing the music but you're actually like feeling it in your body in a way that felt deeply deeply immersive and the images were quite sparse with these point cloud representations that were dynamic and moving around as you're moving through the space and so there's kind of this interactivity as you're moving through the space the space is reacting to you in a certain dimension and But this deeply immersive quality of how, even though it is visually very sparse, it left a lot to the imagination for you to start to fill in the gaps, but also this tight integration between the haptic experiences and the sound design and the sonic integrations of this hapti-sonic-vis, is what she calls it, that I felt like just really deeply connected to the world around me. So Chloe really wanted to create this sense of exploration and open world nature. And I think there is this tension that I found at least where I kind of want to have the story beats and not be too lost. Like I felt like the first part was pretty much guided, but when the second part, it was a little bit too much of an open world because I felt like I was wandering around without triggering anything that I felt like I should be triggering and She said there shouldn't be any desire to see that, but also I had this experience of like metaphorically kind of wandering into a wide open desert without having anything to kind of like latch on to. So I feel like there were certain aspects of as you go into a memory, there may be other things that are nested within those things that you can nest in and go even further deeper. And I miss those opportunities to go into those. And also I'm not sure if I've found all the different objects in the more open world exploration. Like if that was her intention to create that type of aimless wandering and you know frustration the creator then yeah I was able to feel that but I feel like there's also a part of that I really want to understand the ways that she's using the medium to be able to explore these different dimensions of the memories and the integration for her fragmented nature of these memories but also all the different places that she's either connected to or that she's lived. So yeah, coming out of it felt like I'm not sure if I saw all the different dimensions of those experiences, but I got the gist of using virtual reality as this portal into exploring different dimensions of memory. It's a theme that has come up again and again and again over the years of memory and virtual reality. And so yeah, this is another exploration of doing that. And also point cloud representations actually have been the go-to for representing different dimensions of memory. So yeah. looking forward to seeing where this continues to go. I'd love to see a little bit more of this custom integration of the haptic devices into these immersive experiences and Chloe said it was a very intuitive process of really paying attention to these different frequencies across her body and then doing this translation of this ambient sound design that was happening in this piece and how that was also tightly integrated into the haptic experience as well. So the piece was called Temporal World, a HaptiSonic Virtuality Memory World by Chloe Lee and Yeah, keep an eye out to see it. I feel like in terms of haptic experiences, it's one of those things that really dive deep into that. At South by Southwest, there was also Symbiosis, which I've had a chance to see at the Portland Art Museum Center for Untold Tomorrows. I had a chance to do three of the six different characters. So there was a symbiosis experience that was showing at South by Southwest. And that was one of the pieces that people absolutely wanted to see because it is an integration of both this very unique haptic experience, but also like a smell. So it was a multi-sensory experience and also different haptic integrations that were happening within In Pursuit of the Repetitive Beats and Forager were also really interesting haptic innovations that were happening in the selection this year, which In Pursuit of the Repetitive Beats also covered previously in my coverage of the doc lab last year. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voices of vr. Thanks for listening.
