I interviewed Empereur (Emperor) co-directors Marion Burger and Ilan Cohen after Venice Immersive 2023. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my series of looking at Venice Immersive 2023. So this is episode six of 35, and the first of three of looking at the context of communication. So today's piece is called Emperor by Miriam Bourgeois and Elon Cohen, and It's a piece that actually picked up the third place prize at Venice Immersive this year. And it's a piece inspired by experience that Marion had with her father who suffered a stroke and a condition called aphasia, which means that you can receive and decode language, but you can't encode language properly. So you have a hard time finding the right words to use. And so it's a piece, a context of family and health and communication, trying to communicate with someone who has a hard time finding the different words. And they're trying to give both the perspective of the daughter at the beginning, and then trying to imagine what the experience of the father might be like in this sort of dream logic context and just does a really beautiful job of telling the story of these gaps in communication and jumbled up language and trying to imagine what it would be like to suffer from this aphasia type of experience from a stroke. So it's in the center of gravity of emotional presence is a really deeply moving narrative, very well written and very lyrical script, both the beginning and the end. There's a lot of interactivity in the context of this piece. And so a lot of ways that they're really exploring how the interactions of this piece are really tightly coupled to the narrative, probably the best use of interactivity that's connected to the narrative that I've seen this year at Venice Immersive. And then finally, there's different elements of environmental design that they're taking you through these different spaces and taking a whole spatial journey through like this dream logic context, and a lot of different aspects of embodiment as well, exploring how different degrees of hand tracking and then trying to simulate some of the different frustrations that you would have if you were suffering from this by trying to have these different embodied interactions that are somehow not engaging exactly the way that you'd like them to be. So that's what we're coming on today's episode of the voices of your podcast. So this interview with Marianne and Ilan happened on Friday, September 15th, 2023. So with that, let's go ahead and dive right in.
[00:02:34.127] Marion Burger: So I'm Marion Burgère and I used to be a production designer for cinema. And Emperor is my first VR experience as director and my first experience as director at all. So yeah, it's about my father's story and my relationship to him. So it has been a story that I really wanted to talk about, but just in my way. And when I met Ilan, it made so much sense to make it in VR because he talked to me about that. I had no idea about what was VR. So I was telling him, yeah, if we are doing this way with this medium, we should do it together. And so it was really the beginning of the project. So in my head, so he has been there from the beginning. So it's really our project of us two.
[00:03:26.265] Ilan Cohen: Yeah. And I'm Elon Cohen. I also come from cinema, classical cinema. I am a first assistant director. I also write and direct made a bunch of music videos and in the process of getting my own fiction projects off the ground. And in the meantime, when I met Marion, like she said, we started working on this project and eventually developed into what was recently shown in Venice. And it's been quite a ride.
[00:03:50.796] Kent Bye: Awesome. And maybe each of you could give a bit more context as to your background and your journey into working with VR.
[00:03:56.730] Ilan Cohen: Sure. So I think I've had an experience as a appreciator and user of video games since I was a child. And I have some friends who do a lot of 3D. So I'm sort of familiar with behind the scenes of like how things are made within that space. And I've done projects with a lot of special effects. And so I'm kind of have an understanding of general understanding of how what's required for things to exist in a 3D space. I was introduced to VR years ago when a friend of mine wanted to scan the sets of a film that was being shot in Japan and in France. And he wanted to use an architecture scanner to have the point cloud information from these different sets and turn that into polygons. And back then, that wasn't yet something that could easily be done. He wasn't able to unlock the tech, but I did have the experience of going around to Japan with him and in France and scanning these different places and thinking about writing an immersive narrative piece for VR around the life of this Japanese artist. And so it became just this fascinating thing to just kind of explore just mentally the possibilities of writing and directing within that space. So I kind of had that in the back of my brain. And then a couple of years later, I met Marion, she told me about her project. And I just kind of suddenly realized, oh, this is like a really nice canvas for this, and her idea could really suit the medium quite well, and we just started exploring it together. It's been a whole adventure, really figuring out how that works, and it's been really fun just delving into this sort of wild west of VR filmmaking, where There are rules, and then you can break those rules. And the week later, that rule doesn't exist anymore. So there's a lot of boundary pushing and a lot of exploration, which doesn't feel it's not something you can quite do the same way in filmmaking, where everyone has done a version of whatever your idea is. Whereas here, there's a few things that feel like you're really just going out into the nether region of the medium and trying things. And it's exciting to work with such a youthful set of tools.
[00:05:58.081] Marion Burger: I didn't have any experience in VR and I was not a gamer. It's not my world. I'm much more from concrete materials. I'm making sets, so it's quite far from me. I've always been very interested in mixing techniques and mixing departments. and what we are doing in like cinema is just working together and making things all together with which everybody's doing something different, but just we making a piece together. So I'm always been very interested in that mixing. Yeah. Just not to be just in my thing, you know, just trying something else. So I was very interesting in this kind of innovative way of work and just like this new cinema, that has been very experimental, that I really, really liked the idea that you can try things. And then everybody is beginner a bit, not everybody, but just like it's very beginning tool. So I was feeling much more at ease trying to do things as a beginner, much more than a director on a normal film. Because I think everybody is very interesting in other points of view. You know, there are so many people coming from art, contemporary art, from dance, from so many places. Everybody's interesting in this mixing arts medium. So I think that's a great way to meet other people, other way to think things. And just like mixing our point of view is great. It's a very great tool.
[00:07:28.130] Kent Bye: Yeah. And so I had a chance to see Emperor at Venice and actually had a chance to watch it again this morning, just to kind of refresh my memory of it, because there is a lot of dream logic type of sequences that felt like I had an emotion at the end, but then I had to see it the second time to really understand all the different beats of where all the different associative links go. But in the story, it's basically like told from the beginning and the first person perspective of a daughter. talking about her father going through a stroke and having this condition called aphasia. And then it kind of switches perspective into the father's perspective. And so Marianne, I understand that there's a bit of this story that you're very personally connected to. So maybe you could give a bit more context to how the story came about.
[00:08:09.504] Marion Burger: Well, when I just tried at the beginning, it was like, eight or ten years ago, you know, it's kind of, really for my father, he's not dead, just like I was like, he's still here. But when I realized that I needed to communicate with him, just because he was still there, and even if it was hard to communicate, I really wanted to try to talk with him in a way or just try to communicate with him as an adult, because it happened when I was 21. And I didn't have any relation as an adult with him. So I missed that as a way to, you know, you can ask your father, just question of adults, just like, how did you leave at my age of 20s? How did you feel about life? And I couldn't do that. It took conceptual ideas to talk about with him. With all these broken worlds, it was complicated. I had this frustration. At the same time, I was thinking, he's still there, so I must do something. And in this way, I was thinking, OK, I need to understand how the language, how it's done in his head. How does it work? So I read many things about neuroscience. I just stand beside the speech therapist and him, sometimes just listening to the classes with him. And then observing, I took notes. And then I let this note for years. I just let it in a book. And when I began to ask myself if I wanted to do something about it, I just took back this note and just I read it. And I couldn't really understand the meaning because it was only notes. It was a bit of words. And sometimes what I was observing of his gesture and things. And then trying to interpret these notes was a way to try to understand myself about understanding him. And then I understood quite a link between some words and some images. And then I remembered these really moments of, okay, sometimes there is a word that he wants to say, but then I feel it's really absurd, but at the end, it's not. And when I was there, just beside him one time, he had this weird behavior. Well, he was trying to, I don't know if it's a good moment to explain this. I don't know. Yeah, because it's a good example. It's working in French, but it's hard to explain in English. It's all the weird thing about the project is because all the play with words is working much more better in French than in English. But for one of these exercises with this speech therapist was He has to try to make her guess what word on the card. There was a picture and the specialist cannot see the picture. So he was trying to tell some words to explain the picture.
[00:10:51.831] Ilan Cohen: So he's looking at the picture of a duck and trying to have her guess that word. Yeah, so he's holding a card with a picture of a duck and he's having to have her guess the word duck, but he can't say the word. And he's, instead of trying to mime like the animal or to explain, talk about an animal, he starts talking about a beach and coffee. And at one point he points to Malian who's sitting in the room and the speech therapist is completely confused at this point. But Marion made the connection because when she was younger, they lived by the seaside and he would often go and have like a cup of coffee and she would come in and she would grab his sugar and dunk the sugar inside his coffee, which in French is known as a duck. like that action is a duck, but it's almost like what was prevalent was the emotional connective memory rather than the logical attempt to find a different way to explain the animal. He went into this very personal kind of meandering other meaning that had this emotional tie to her. And we were really fascinated by that example and realized that there are a few others where we could like kind of like pinpoint that there is this emotional core that seemed more important than the logical core, connecting him to the words and to other people through the meaning of those words.
[00:12:14.637] Kent Bye: Yeah. So maybe it's worth expanding on a little bit about this condition of aphasia, because at least from what I understand is that there's a certain amount where if someone has a stroke and has a side effect of something like aphasia, then they may be able to understand what's coming in and be able to process it, but their ability to communicate and maybe write is also impaired. And maybe you could elaborate on that because this seems to be a core theme of what this piece is exploring is this condition of aphasia.
[00:12:44.593] Ilan Cohen: Yeah, well, it's a very complicated disease. There's three main branches, he suffers from the main one, the Brocaphasia, which essentially it's like your filter for language is all garbled. You can decode, but you can't encode properly. So it's like you're looking for words in those little drawers of your brain, and you think you're taking out the right word, but you're taking out another word. And despite using the word that you're sure is the right one, you're actually saying the wrong word. And it takes a while for you to understand that you're not saying the right thing. And that becomes a very frustrating process because you can't trust yourself to express yourself properly. Whenever you're attempting one time out of two, it's going to be wrong. So it creates this very... There's a high potentiality that people lock themselves up into non-communication as a result. And certainly with Manuel's father's case, it's been complicated. He's a shy human being, and so there's a good amount of him that is just locked up in his own thoughts. But he's still there, and his whole past life is still active in his mind, and that's something we wanted to explore and pay tribute to.
[00:13:50.268] Kent Bye: So yeah, it seems like a common theme in this piece is communication and establishing a communication of being able to both connect and create a bridge of meaning. And it feels like part of the experience that I had in this piece was the experience of that disconnect of being trapped and lost in this dreamlike desert experience. That's very impressionistic and associative as well. And there's a lot of telephones and tape recordings. And so memories of moments of communication in the past, but yet you're trying to strive for this type of communication. So Marianne, I'd love to hear you expand on the seeds of what you were trying to explore when you think about trying to create this interactive virtual reality experience of trying to cultivate this experience that you've had with your father after suffering a stroke and having aphasia and trying to recreate moments and aspects of that to try to transmit some of these experiences that you've had within this piece that is exploring this striving and desire for connection and communication.
[00:14:56.798] Marion Burger: if I have to be sincere with what actually my father hears, is very close in his bubble. Like he's really not really trying anymore to communicate just because it has been too complicated and too tiring to try to find always a good word. And years after years, it took much more energy for him to try to do the the work to go to the other people because when everybody's together it's very easy to forget him just because he cannot really be in the conversation and you have to make an effort to try to take him into conversation. And so that's the main problem, I think, just like, it's hard. It's complicated to go to him. It's complicated to make links just because it's frustrating for him. It's tiring, and it's not easy. So this project is about also me letting go something, you know, like just Link accepting the situation that it's, I maybe will never be able to understand him properly. And maybe it's in something that because I'm doing this project, I have been diving, trying to find information about him and everything. So I did a walk to him to try to understand him. Is that the way to communicate with him was just me showing that I wanted to do something with him. But at the end, the real communication is not really existing just because He is shy, he doesn't want to show himself like in this kind of state. And so there is like, it's a lot about my point of view of his condition. And I cannot tell that it is really what he thinks and what he lives. But I feel like at the end, I think it feels lonely just because and it's hard in this kind of this desert and is lost in just this sometimes he just in his world, just forgetting of everybody talking around. So that's quite a sad, thing to see and to admit. But at the same time, if you accept something like that, and then you be careful of all the aspects of the communication, just look, the gesture, sometimes just take my hands, and it's very strong. And, and I really enjoy this moments, just because I know that it's taking much more emotion for him than just trying to tell me badly something. So yeah, I took like treasures, these very small things that we have together right now. But I'm not like trying him to communicate with him the way I would like him to do. I think just trying to accept what is right now.
[00:17:37.628] Kent Bye: So as you're starting to put together this virtual reality piece, there's like a whole component that feels like a very well written script that almost feels like I could listen to it as an audio podcast, especially at the beginning, because I'm seeing the images, but the story is like really being told by the audio narration, then becomes a much more of an immersive interactive VR piece. And then it comes back to certain aspects of the narration near the end. And so as you're putting together this as a piece, there's the script, there's the design of the piece of what the virtual reality experience has, there's the interaction parts. And so how did you start to piece this whole experience together? Where did you begin?
[00:18:18.226] Ilan Cohen: Well, so there is this thing that I always refer to when I'm working on a project which is like the lighthouse. Like you have to have this one core thing that's never going to move that you really believe in and whatever's going to happen throughout the writing and rewriting process and everything gets thrown away and you have to start over again, you still have to have this one lighthouse in the dark that you can go back to and say, this is why I'm doing it. If that doesn't exist, then the project probably shouldn't be made, in my opinion. In this case, the lighthouse was not only Malian's first-hand experience and the fact that this was her own story, but that she had written early on this text, which is, I would say, 95% of it is still basically that introductive and concluding text. And it was so powerful to me. I found it so beautifully written and such a good condensation of her experience. Just like a beautiful literary piece that I connected to. And that was the lighthouse. We always knew that was holding up the whole structure. And that's the thing that opened all the doors. People were very reactive to that piece of writing. And so we knew it had to be in there because without the context of the personal story, all the interactive first-person empathy-centric journey wasn't really going to land. And so we always knew we had to have that text to start off with. And at one point in the writing, we decided to split it and have the end of it at the very end. What happened in the middle? There have been so many drafts. We tried so many different things. It's really hard, I think, for either of us to be able to pinpoint the journey to like, this is finally what we ended up with. But it was a combination of wishing to explore the medium, for sure. We did not want to get trapped into something that was just like, you're sitting there and it could have been a movie. it had to feel like we were making use of the medium, because as occasional VR users, we would get very frustrated with pieces that didn't, to us, mind the potential of a medium that's also a very physically aggressive medium. You have a screen right in front of your eyes. As soon as I get bored in VR, I get very impatient. And so we were like, all right, how do we keep this from being an impatience-making machine? How do we keep it from being boring? And I think one of the ideas was very conceptual. We just wanted to have the sensation that you're constantly learning to do more things and getting to do more things. So we just kind of devised everything from that standpoint where you start off with almost zero interactivity. and then it grows progressively, and it gets bigger and bigger, and you get to see more, and then you get to teleport. We would have liked to push this even more, but there was this idea of this gradual unveiling of the interactive features, and the gradual unveiling of what the father could do. The more he goes into his own memories, the more he's able to use his body again. He can bend down and pick things up, and he suddenly has the use of both of his arms, whereas the experience starts with him seated down and the user early on has selected something with his arm and so the VR experience knows which dominant hand is the user's hand and then we remove that hand from the experiences at the table where he suddenly he's having if you're right-handed you have to do the exercises with your left hand and so which it's a trick and obviously people think is a bug at first and you kind of realize like oh no this is part of the it's a feature And that was fun, trying to really create something where you feel very frustrated and you're playing with the limitations of the hand tracking and the imprecision therein, but then you actually get to do more and just have this gradual unveiling of potentialities. So I think that guided, structurally, a lot of the ideas. I don't know, we just kept throwing ideas into the pit and Mayum always knew if they felt right or not. And we did a lot of digging into his life and the things that he used to do that he used to love and tried to like work them in. So he was a parachutist and he did. a great number of parachute jumps, so we knew we wanted that in there, and we found a way to work it in. Every time we had an idea for an interactive feature, it had to have some sort of metaphorical resonance with a word that had been misused before, and so we had a lot of interplay between those. It was just fun trying to weave this relationship to mistakes being said and then connect them back to emotional memories and have all that be the canvas for an interactive feature. So yeah, it was a strange and convoluted writing process and somehow we ended up with this version.
[00:22:44.519] Marion Burger: As we were making always bridge between one information at the beginning and then an object and then something else. And that everything had to be linked to something and just to do a ping pong of, yeah, oh, I saw that. And then I understand well now, like my process of understanding him, like he tells that. But at the end, he showed me, he pointed me, so maybe there is a link between these two things. And the link is making sense. So that's what we were trying to do, just put things at one point and then we'll do a bridge with this something. So it was hard at the end if we want to reduce it. We were just very aware of reducing it if we didn't have so much money, because everything was so much linked to each other from the beginning to the end that just taking something out of the script was very complicated just like everything was just like so in between that we were like very worried to have to And I want to say that also we had this idea that we didn't want this to be like a perfect puzzle.
[00:23:47.546] Ilan Cohen: It's not like you finally get it. It's more like you maybe you're chipping away at something and you start to see a shape at the end and you have this vague impressionistic idea of how things might connect, but we wanted that feeling to be just like a very abstract poetic resonance and something very pointillistic and not just like a clear understanding because I don't think there is such a thing as a clear answer to what is going on in his brain, but there might be a relatively accurate poetic interpretation of what's happening and that's what we were trying to aim for, just something where We're just triggering feelings within the meaning rather than like just, you know, very absolute meaning, if that makes any sense.
[00:24:29.148] Kent Bye: Yeah. Yeah, for sure. I know that when I went through it the first time, it felt like I was in a dream. And the second time I still got that feeling of a dream, but I had knowing where the entire experience was going. It was a little bit easier for me to understand the dream logic structure that you have really certainly in the middle part of the piece. But I think, you know, one of the other things that both times I watched it, that was super striking to me was just the aesthetic of the very minimalistic using what feels like watercolor paintings that are kind of like unfolding and just a real beautiful kind of hand drawn feel of an animation style that is very sparse. I mean, you have mostly these like snippets or fragments of images that are kind of building out. But for the most part, you don't get a full experience of a scene. And so it very much leans into this aesthetic of how I see memories usually represented within VR. And so it has this kind of, I'm walking through all these different memories of these experiences, So I'd love to hear a little bit more elaboration of how you landed on this very unique aesthetic of the style, this kind of black and white animated watercolor like animation that is throughout this piece.
[00:25:42.328] Marion Burger: Yeah, from the beginning, the first idea I had really for the project was maybe because I just wrote things at the beginning. And the first idea was just an animation movie, just because it could be the best way to do transformation, like a morphing between worlds, you know, like making images of morphing worlds, going to something, to something else. And so I didn't want something real and didn't want something like too much documentary. Yes, the idea of not showing very clearly things were much more about the idea of a perfect aspect as broken words, as the missing words that my father had, like not to showing everything was very important, just to have this kind of discrete feeling of not showing the faces for example was just like didn't want to have something too much uncanny look of because it's in between the documentary and something more dreamy so i didn't want like to look at someone that doesn't have really a real look because it's too much weird so Yeah, it was the idea that if you're not showing everything, then you can imagine more. And it's really what I like in sketches is that you really feel something much more. Sometimes it's very sensible, but you don't have to see everything just to be touched. It's exactly what we meant in this kind of aesthetic. And Ilan has really had the same things in his head when we're talking about it.
[00:27:19.366] Ilan Cohen: Yeah, we lined up immediately. I was also at first, the very first thing felt like this should be animation. But then there was also this idea of if we want to go inside his head, VR is the best medium. And then we quickly realized we can probably combine the two aesthetically. And so we very quickly kind of developed references that were very aligned. And we're pretty close to, I think, what we initially wanted. We were able to work with someone fantastic. And he really was able to translate our vision. But I think one other big element of that is we're both very attuned to the use of off-screen in filmmaking, just off-screen presences of feeling things and guessing at things and suggesting things that are not within the frame. And so the whole question was, well, how do you do that in VR if the person can actually turn their head and then make that part of the frame? And one of the answers was, well, we don't show it. It's just something that you hear, something that you feel. And so we decided that it would also be very helpful just to guide the eyes of the user to the place that we want. We didn't want to have something going on that they might miss. And so always the best solution was like, well, we just erase the rest. And we could always do that. And we had a good thematic excuse for it. And it was just a very good set of parameters that went well with the subject matter, went well with our constraints and filmmaking problematics, and that we were able to solve through this aesthetic. So it was a good combination of needs and wants. And yeah, it was a lot of fun to explore. It leaves a lot of room for playing with sound. And you can give a lot more importance to the sound if you're not representing everything. And that was a whole fun other chapter to explore. Yeah, no, it was a very common desire to explore this aesthetic. We also wanted something that had a very textural approach. And we wanted to have imperfections and a certain granularity that it didn't feel too flat. And so there's this big idea of all the textures are often sand or rock. And there's this evanescent fog throughout the experience. But the fog is covering up these rugged imperfect textures and there's this whole thematic of the crack and the collapse of this giant statue of the emperor kind of permeates throughout the whole experience as something that is beautiful but also very barren and dry and granular and textural. And that was the best aesthetic translation of that feeling.
[00:29:48.653] Kent Bye: Yeah, and there's certain moments where you kind of are slowly constructing out the scene and other times you're in a scene and it's slightly morphing or changing. And overall it reinforced, for me at least, watching it where it felt like I was in this dreamscape and sometimes it would do a hard cut into another scene or sometimes the existing scene would slightly morph and change in a way that is very surreal and unlike any physical experience I've ever had. And so I feel like there's a certain way in which the medium of virtual reality is able to tap into this really surreal dream-like logic that you have, especially in the middle of the section where you are much more embodied into the perspective of the father who is kind of lost in his own mind in some way. And so I'd love to hear any elaboration of, it's almost like a tonal shift as you go into the perspective of your father, kind of lost in the desert as it were, or in this dreamscape where I had this experience, especially the first time that I was watching it, where there's an instruction to point to progress. And then sometimes there's a sound design that's really helping to focus you on where to point and also like a little visual cue. So you're in this pattern of progressing through. It almost is like you have to solve the puzzle of how to progress to the next scene. But sometimes then you'll have a scene. where there is no puzzle to be solved. You're kind of stuck. There's nothing you can do other than just to wait. When I first went through it, I was trying to continue to progress. And I felt like this frustration of this expectation of like, oh, I need to continue with this rhythm of the piece. But sometimes there's scenes that are deliberately put in there to disrupt that flow. Which I don't know if that was a design to recreate this frustration of the lack of ability of having this expectation of being able to connect or communicate or progress. So yeah, I'd love to hear how you were playing with the spatial design in this game mechanic that then almost intentionally gets deliberately obstructed at certain points to take the user on a certain journey emotionally.
[00:31:55.708] Ilan Cohen: Yeah, well, so one of the guiding things was really getting the user out from his comfortable observer position into standing up. There's a whole element of storytelling that does kind of happen like right after you've stood up the first time, you're basically invited just to turn around on yourself. And it's not a very interactive part yet. There's a few hidden interactive features, which some people may or may not discover. There's a few little almost Easter eggy things, like the film projection sequence. You can actually play with your shadows, the shadows of your hands. But you would have to be wanting to try to do something at that point. And the experience really isn't inviting you to, but it happens to be there if you do. But then, yeah, from that point on, there is this whole sequence where basically to get from one place to another there's this new mechanic of teleportation that's introduced and there's a few interactive items to pick up from the ground that are half hidden and then the last one we wanted to introduce a mechanic like picking up things that are half hidden in the ground but then the third time you're expecting to take out this suitcase and it's the handle of the suitcase is actually a telephone. So just trying to set something up and then trick you by thwarting the expectation that we've just set up was something that we wanted to play with so things didn't feel too repetitive. We're always like one step ahead of you in a way. But there is also this element where when all of that is finished and you get to the statue at the end, the last teleportation that you do just takes you back to a place where the father actually leaves your body. And this was very important to us is this whole story is the interpretation of the daughter. And so, we needed to make sure that it was very clear that we were seeing it through her eyes. And so, he leaves us again, and we go back to her perspective. Because deep down, that was the most important thing, was just to recognize that this is her journey, and it's her path to acceptance, and her attempt to connect to what all this could feel like. But it's always going to be hers, and his will always be a mysterious thing that he's keeping to himself. And so there was this desire to also suddenly break that kind of like rising interactivity with a return to, but we don't really know. This is just our interpretation. And to me, that's always where the emotion comes in. I'm just like, that was a fun foray into something that is incredibly lonely, but we tried to keep it light and playful and surreal and everything. But then in the end, like he's just going to go back to his table and stare outside.
[00:34:21.029] Kent Bye: Yeah, Marianne, I don't know if you have any other things you want to add to that.
[00:34:23.714] Marion Burger: No, no, no. Just like getting stuck. Well, frustration was one of our sometimes goals. Sometimes we want people to do some things, but just for a little bit of time to try to figure out what. But yeah, it's playing with that, just frustration a bit. And sometimes it's not very used like that in VR, but I'm very happy to be out of things when you get out.
[00:34:52.525] Ilan Cohen: Yes, because like the whole structure of like especially the first half where you have these kind of frustrating exercises to do at the table with a speech therapist, the whole idea is they're rewarded with a memory and the memory is giving you insight into a different moment of his life. And so we really wanted them to feel like rewards. There is one we had to simplify a lot, unfortunately. But I think the first and the third are quite rewarding. There's the sled. That is a big surprise from the completely locked-in syndrome thing that you're experiencing. And then suddenly, you're on a sled. You can actually control it a little bit with the tilt of the head. And we wanted to go even faster and to be longer. But we had to take into account people who are a little more susceptible to motion sickness. But then also, the third one where you're suddenly parachuted into his mental state, that was something that we were very excited about and really wanted to do to just get this feeling of like, okay, we built up a lot of frustration purposefully, but now you get this, as you're falling into his mind, which is a very sad place to be in, it's also that there's this sense of excitement and reward. So we were trying to play with the elements of frustration, but then to release you from them a little bit.
[00:36:08.019] Marion Burger: Yeah, it contrasts with really great people at the moment.
[00:36:11.080] Kent Bye: I think the first time that I had seen it, there's a certain rhythm that you get with the pointing and then you're supposed to point to go to the next scene. So it becomes kind of like this puzzle mechanic in some way in order to continue to watch the piece, you have to do something. But then I remember the first time I was in some of the scenes where there's actually literally nothing you can do to do anything. I was continually like trying to like, okay, do I do this? Do I do this? And so the second time I watched it, I was like, okay, I'm not going to do that this time, because I feel like there's certain deliberate design decisions that you're making to create this sense of frustration. So I was a little bit more patient the second time that I watched it, which was interesting to see the pacing of that. from the first time doing that versus the second time of knowing what to expect. But going back to like the overall, I think of all the different pieces I saw at Venice this year, I feel like Emperor is the one that is trying to use the type of interactivity that is tightly coupled into the progression of the story or trying to have these embodied actions that are actually reflecting and taking me deeper into the story. Whereas a lot of times they'll see, I mean, you do this at some points in Emperor where it's essentially like pushing a light switch and you progress or something that's very procedural where you're just clicking a button or turning a dial to be able to progress and There's certainly some of that in this piece, but I think some of the different types of interactions that you have in this piece are actually like trying to get into the root of the story that you're trying to tell. What's it mean to go through a stroke and to not be able to write in the same way. And, you know, like you said, very elegantly choosing what is the dominant hand when you're selecting at the very beginning and then taking the opposite hand of the non-dominant hand to force people to write with that hand and then to deliberately mess up the writing. So it creates this frustration of. what someone who has gone through a stroke would experience to not be able to actually write properly. So I felt like the use of interactivity and embodiment in this piece was really trying to tightly couple into these very specific experiences, whether it's some of these things of having the loss of your agency and being able to communicate with your body or these different moments of frustration that you have throughout.
[00:38:18.142] Ilan Cohen: That's really great to hear because for sure our guiding principle was always to find, to only include interactive things that were connecting back to the story and the theme. And whenever we had like cool ideas that just didn't fit, be like, ah, it'd be so great to have this, but it would also not mean anything. So we immediately discard it. It always had to make sense. Even I think those very simple, like, flicking a light switch to keep things going is at the very beginning but we did try to create that as it's almost like those moments are supposed to be like these like memory capsules where you're like you know when you're starting to uncover a memory and you just see like a little bit of it and then suddenly something clicks and it's almost you get this like one second of like this medley of boosts where you can suddenly hear and smell and remember what that felt like for a few seconds. And we were trying to emulate that. So when you do hit that switch, then you get the sound that comes with it and you get the context. And it's like, there's this memory bubble that pops in your brain and you're suddenly transported there just for a few seconds and then it's gone. And so we were trying to emulate that feeling in the introduction. So it was always trying to be tied into the theme of like, all the different themes that we're going in, but obviously memory being a chief one. But yeah, we're glad that you were sensitive to that. There was this other thing, just to finish from the previous point, where we did try to be deliberately very subtle with the UI to a point where, for some people, I think sometimes it's a little confusing or you just want to get places faster. There's a lot of audio cues that people don't necessarily pay attention to because we didn't add any visual cues. And sometimes that's a bit of a radical choice, but we felt we never wanted to have something that took you out of the experience. even just to show you it needed to be woven into the fabric of the world.
[00:40:03.170] Kent Bye: Yeah, this is a very audio driven piece, just from the narration and the sound design, but also the music is also really quite amazing as well. And I'd love to hear you elaborate on the process of creating the music for this piece.
[00:40:15.846] Ilan Cohen: Yeah, we were very, very lucky to be able to work with two musicians who are also good friends of ours. And so we were able to dig in and it was quite conceptual. There was quite a lot of ideas that precluded any writing of music because the project is fairly conceptual and we felt that You know, we wanted to emulate that in the process of making the music. And one of the musicians, Gespar Kraus, is a gifted cellist. He's a great improviser as well, and not a super structured person. And we wanted to make sure that the cello, which was the right instrument for most of the music, we wanted to not fall into the dangers of it being too melodramatic or too intense. It can get a little dark. as an instrument if it's doing like atonal stuff and it needed to be neither of those things but it needed to have that like very touching emotional core of those like quivering strings and so the approach we found was to have the other musician Jamie write songs that would be his emotional connection to the material and to write like actual songs like he found lyrics he built like an entire track with like multiple instruments leaving very little room for the cello but then we use those pieces as the core kind of just the structure and the architecture, the musical architecture of the piece was basically that song that we were never going to hear in the experience but then we knew we could use this part of it and redo it with the cello or in the case of the daughter theme there's these embellishments there's like 140 tracks of cello around the song that he made and then we just remove the song so all you have is like kind of this empty husk like just these like really beautiful ornamental string parts, but nobody would ever write without the middle central harmony. And just not having this sort of ghost theme, not having the central part, but having all the rest creates this weird sensation of something lacking. And there's this very like melancholic feeling to this theme that we call the ghost theme. So it was very conceptual, but also it landed quite beautifully. We're very happy with the process and the results. We made the soundtrack available. In the soundtrack you can hear the original song, so you can kind of see what the pathway was from one song to its different permutations as they were being used in the final experience. And then we also wanted to have as much of the sound design be created from the cello, so there's a lot of incorrect fully where we're using the wrong sound for something. But because it's animation, because the way we react to sound very often in context, it suddenly feels like, oh, yeah, that could be the right sound. But most of the sound design is actually being done with the cello, which is very odd. But you can get a lot out of that instrument as just a sound making device. So it has a cohesiveness, but it's also like it takes you into this kind of dreaminess. Yeah, it's a weird logic.
[00:43:14.626] Kent Bye: I wanted to ask about the dream logic sequence that you have with the father going through these different scenes. There's these different moments of picking up a wine bottle that's part fish and flies off in the air and picking up an airplane that then is a bird that flies off. You have these weird juxtapositions of spatial dream logic that you have as you're going through this journey. And it's almost like when you wake up and you go through a dream, it's sometimes difficult to remember all of what happened in the dream. And so it kind of creates this weird, blurring memory of you're transcending expectations, but you're also in this kind of logic of a world that you don't actually know what the rules of this world actually are. And so I'd love to hear about this process of dream logic that you're able to cultivate within this piece.
[00:43:57.769] Marion Burger: Yeah, the first idea came from, I think, the first mistake we had in mind from my father trying to write in French 28. It's 28 or 20. So we translated in 21 because we wanted to make a Hi, I will just explain this part because it's not good.
[00:44:19.389] Ilan Cohen: The English version is 21 because 21 can sound like 20 wine and that enabled us a good translation to having that word become like a bottle. But in French, it's a little better because it's like wine trout.
[00:44:36.906] Marion Burger: 28 is Weintraut, yeah.
[00:44:39.327] Ilan Cohen: 28 with just one added R can become Weintraut. This is a real mistake he made. And it just suddenly turns a number into a very amusing combination of two words. And so there's a lot of that going on. And so sometimes we went from reality. Sometimes we just tried to find alternative things that would work as a dual object that would be fun to play with. And we retrofitted a meaning to it. So we knew we wanted to go into a dream logic. And some of it came from reality, and some of it was engineered. You want to finish, Mayum?
[00:45:10.521] Marion Burger: Yeah. No, no. It's just like we use objects like words. We make objects like my father would make words, like just there's one of this and one of that, and it doesn't mix together. But it seems, but it's not. At the end, it's not exactly what he wants. It's just like this always deceiveness of just like, I have the good words, but not. You know, just like this kind of weird feeling of, but it's fun, you know, it just, it's also light, weird and funny. It's like, hey, okay. Yeah.
[00:45:44.024] Ilan Cohen: And there's this idea of uncovering them too, of like, you're digging for your words that you think you're grabbing one thing that you grab something else. So that was a pretty direct metaphor. And then we were very tempted, we wanted to like bring things into a much weirder place. And of course, there's like, time and budget limitations. But we were able to like, you know, at least we're happy, we're able to play with scale, and to have just all sorts of different changes within the mental world where it's not one universe, you're not just like walking through a desert, you're kind of being transported. The way you kind of remember a dream, the locations of a dream, you don't really remember how you got from here to here, but it felt pretty fluid. It's just like, oh, maybe I forgot that part, or maybe that part never really happened. But suddenly you're teleported into another room, and it still feels like it's part of the same room, but like, suddenly it's just this giant memory palace and you have this big abstract hall with these flying columns. But then you're back to the desert and it doesn't feel completely incongruous, even though nothing in the environment around you was leading you to that big room. It just became a transition. We like things to just float in like that and then float out again. You just have this, like you said, this oddly remembered feeling of having been through someone else's dream. Hopefully that was at least the intention.
[00:47:08.374] Kent Bye: Yeah, that makes sense. Just a quick comment on that is that it does seem like there's slight variations between the original French version and the English version and different ways that, you know, you're taking these juxtaposed words that are kind of misspoken, but then create these new objects that are very surreal. Throughout the course of the piece, there's a lot of listening to tape recordings. You're listening to like phone messages. At the very beginning, you show this experience of your father going through a stroke and then talk about the phone message being changed. And then throughout the course of this piece, you have like these recordings of your father, or at least the character that's playing the father. And I don't know if in the French version, it's actual like recordings that you're taking and including. But I'd love to hear a little bit of this use of these recordings of either phone messages or tape recordings that you have of the father that's throughout this piece because it seems like at this point it's a lot of broken communication or lack of ability to make this bridge to connect and that some of these recordings are maybe moments in the past where there's been that opportunity to connect, but sometimes it's misconnection because it's a phone message or it's a tape recording of something that's happened in the past. So love to hear any elaboration of using these recordings as a motif throughout this piece.
[00:48:23.290] Ilan Cohen: You want to talk about your father's habit?
[00:48:27.300] Marion Burger: Yeah, when I realized that one of the favorite habits of my father was just to record voices from our family. And so I was feeling that it really was ironic that he was able to do that and I won't be able to do the same for him. So when I was thinking that, I was just like, okay, I have the same desire just to try to keep a trace of his history, like he was doing. So it was quite a strong feeling. And I remember that I had a lot of tapes with all my voices when I was a child, and I was able to find his voice on it. Because the beginning of Framper is about this realization that she called her parents and then she realized it was not the voice of her father on the voice message. And so then, when someone dies, you can hear again the voice, it's very powerful. And you won't have any more of this voice because now he's talking like you can hear in the generic of the prayer, you hear the real voice in the generic, in the credits.
[00:49:27.741] Ilan Cohen: It's very whispery and feeble.
[00:49:32.073] Marion Burger: you know, it's kind of like it's blocked there. So it's not in more his real voice. Well, not the one I used to know. So all the pieces of his voice on tapes were just like gold. And so when I was coming back to these tapes, I was just listening to his voice and was feeling very emotional. But also, I realized also of this power of messages and these kind of really small things that you can tell on a voice message that doesn't mean anything, it's just like nothing so much, but just it's something when you just cannot hear anymore. And it's also about time passing and all this moment of voice message. I really like going to like time passing and trying to connect, but not, you know, you are not, sometimes you cannot, or when you hang the telephone and you cannot answer, but you're trying to talk to you, but you cannot answer. You are VR, but just like, that's the ironic part of using voice message like that. In French, it's not my real father voice, only in the credits.
[00:50:36.017] Kent Bye: Okay. Okay. And finally, what do you each think is the ultimate potential of virtual reality and immersive storytelling and what it might be able to enable?
[00:50:49.274] Ilan Cohen: Well, our hope was that, first of all, I think I've always had hope for video games to be art pieces in and of themselves. And I find the video game name to be a little bit, it's just, it doesn't play in favor of the potential for the medium, like automatically calling it a game. And I think with VR, even more so, there is a potential for emotion that can be very strong because of the first-person immersiveness. And so our hope was that we would be able to utilize the medium to tell a very personal but also universal story that ultimately some people could find moving. So we sort of realized as we were making this, different people were telling us, oh, this isn't a project for people who have aphasia, it's for the people who are around them. It's for all the helpers. And we realized, yeah, it's true. A lot of the project is born from observing Marion's mother, and obviously her father, but just realizing that there's a lot that gets taken for granted when you have someone who's essentially almost like the family dog or a piece of furniture. You just kind of start talking about them in the third person. And as much as you care and look out for them, there is an element of just kind of forgetting what's going on inside their brain. And I think even more so, a lot of people from the medical field were telling us, oh, this project's going to be great for, nurses and people who have to interact with, you know, diminished human beings like this every day. And they just kind of completely forget that there is an ongoing mental activity in there. And there's a personality trapped within there. So ideally, some people might connect to that in ways that will be healthy. But in terms of VR in general, yeah, I think there's just so much more that we can do to turn these different mediums into just vectors of emotion and empathy and art. So hopefully we'll have contributed in some small way to not let that dream of having the medium evolve past its entertaining possibilities and grow into something a little more artistic and emotional. I certainly feel there's a great potential for that and I'd like to see more.
[00:53:08.076] Marion Burger: Yeah, well, I agree with Ilan and just don't know what I could tell more. It's impressive how it can be strong for some people to have emotion in VR. It's weird. Sometimes it just gets to a point that it just beats you already. Just because, I don't know why, it's just the feelings. I don't know, it's more linked to the body. I don't know, there's something much more immersive. It's weird. Ilan told me about some reactions. I was not able to be on them as a church for trying, so I didn't have the opportunity to see so many people trying it. But what I hear is that sometimes it can be very, very strong. Yeah, it's weird because you feel that you can write something honest and you want to put some emotion in it, but you cannot really see the power of things like in VR. Cinema is powerful sometimes, but VR is somewhere else. Sometimes it can just connect to people in a weird way.
[00:54:17.170] Ilan Cohen: Maybe it can be therapeutic, not necessarily this piece, but when it connects it can be so powerful that this immersive empathy can possibly have, you know, be a force of healing in certain places. I know, I remember seeing these videos of people with some level of dementia who have been unable to speak for several years, and then they're just being shown their neighborhood on Google Street View, but in VR, and they suddenly start going, oh, this is the house that I grew up in, and they just start talking. it immediately unlocks, just being transported back there is enough of an emotional and mnemonic connection to something meaningful to them, that language just starts coming back right away. For sure, I'm sure there are ways for this to be a very powerful tool for some forms of healing. But yeah, we were very touched to see some very intense emotional reactions of some people. Personal stories get mixed up with the story that we're telling and they have these like intense reactions which I think are definitely the reason why they're so strong is because of the medium and there's something to that. Of course we're not really able to see that anymore. I keep saying that it feels like we've been like setting up like a thousand dominoes in this really elaborate way and then like we launch them but we don't get to see the result and someone else tells us about it or like oh yeah did that work like okay cool. And it's a very bizarre feeling where we're just seeing problems, but apparently some people connect to it. So there it is.
[00:55:55.212] Kent Bye: Just a quick follow on, because this is a piece that's trying to dive into the experience of someone with aphasia. Did you work with doctors, neuroscientists, or other folks in the medical profession to try to get, you know, some grounding for what this experience is like, or did you take it more and just a purely poetic documentary direction of just your own interpretation?
[00:56:17.387] Marion Burger: Yeah, there is, I think, two persons that just inspired this. But at the end, it's an interpretation. But first person was the book of Jill Boyd Taylor, who just talked about her stroke. That's very powerful.
[00:56:34.958] Ilan Cohen: She's a neuroscientist who had a stroke and suffered aphasia and almost fully recovered. She could explain what was going on as it was happening because she had an understanding.
[00:56:51.805] Marion Burger: Yeah, when I read that for the first time, I was like, oh, that's crazy, because she's describing what my father maybe had experienced, just because she is able to put in words because he's not able anymore. He cannot tell me what happens, but he's there, he could maybe, maybe he remember, I asked him, but he's not able to tell. But she was able to put words on it and just to analyze because she's a doctor, she's amazing. And it was exactly the same part of the brains than my father and it was touching also the language for her. So, but she's okay now, she can talk and well, some difficulties, but she can. And so, okay, that's the first way to understand just how it works in stroke, like what's mixing, what's the feeling and everything. And then it was just talking with the speech therapist. I read many books, but just it's very theoretical. And just speaking with my speech therapist, my father, she's just great. She is just really empathetic. She is kind of an amazing job. I just discovered an amazing job. They just like to try to gets inside other minds. She's actually doing every day with everybody, like just like this child with autism, with so many people that just don't express normally. She's doing this job every day and just an amazing job.
[00:58:11.692] Ilan Cohen: It feels like she's able to hear and understand so much more than we are when we listen in. It's almost like when you listen to someone's toddler kind of say a few words and you have no idea what they're saying, but the parents can make out everything they're saying and you're like, oh my god, you've decoded this garbled nonsense. And she has that same ability. One thing that we took from her and Mayon's father's relationship is he keeps pointing to places, everything's not far, it's just over there. It certainly became very clear that he's exploring this almost physical landscape where words and their meanings are. Everything is always just around the corner, just a little further, it's really close. And so it kind of became a guiding interpretation for us was he's lost in this country called Aphasia, this faraway land. And he's trying to find his way back to meaning. And that was from listening to the two of them. But no, for sure, we did a bunch of research. And in the end, it needed to be a pure expression of Marion's point of view. And so I think that's why that scene at the end is really important, just to acknowledge and remember and be truthful about the fact that all of this is still just her interpretations.
[00:59:27.115] Kent Bye: Awesome. Well, it's certainly a really beautiful experience and I was really quite moved by it and, uh, love to give you any last opportunity. If there's anything that's left and said that you'd like to say to the broader immersive community.
[00:59:40.876] Ilan Cohen: Well, I don't know. I was just touched by the fact that you've done it twice, because we've always thought of this experience as something that should be done a second time. There's a lot going on that we set up in the very beginning that probably doesn't have much meaning at all. But hopefully, when you do it again, it takes on sort of a broader, more specific, and more emotional meaning. So if you try Emperor, try it again, if you liked it. It's worth it. But yeah, I would say to the broader VR immersive community, use sound more. Focus on sound. There's so much to be done. It's half of the experience. And it's the best possible setting for people to hear what you're doing. Most of them have headphones on. And it's just like, there's so much going on with the spatial mixing. It's an amazing feature. And I think it's heavily underused. So yeah, just go all out. It's amazing. Sound is great. Sound in VR in particular.
[01:00:35.155] Marion Burger: And don't forget to get up.
[01:00:37.560] Ilan Cohen: Stand up.
[01:00:43.070] Kent Bye: Awesome. Well, Marianne and Elon, thanks so much for joining me here to help break down Emperor as one of my favorite experiences that I saw at Venice and congratulations for taking away an award from the jury as well. I think it was like the third place prize or I don't know what the name of it was, but certainly deserving of all the awards and all the praise that it's getting and looking forward for more folks to be able to get a chance to see it. Cause I think it's. It's exploring this impressionistic dream logic embodied interactive first person experience that really transports me into what feels like these deeper archetypal relationships between children and their parents and the different aspects of health that comes up, whether it's this specific experience or some sort of modulation that people have gone through. I think this is an experience that can resonate on many different layers, even if the specifics are different. I think the way that you've done this poetic interpretation is able to get translated into many other contexts beyond just aphasia, I'm sure, for folks to connect for all the different ways that they may or may not be able to connect and communicate with their parents. So yeah, it works on many different layers and I think it's just a really beautifully poetic piece that has a lot of heart and soul and yeah, just really technically is, is, is really pushing the edge for embodiment and interactivity and immersive narrative. So thanks again for taking the time to help break it on down.
[01:02:00.906] Ilan Cohen: Thank you. Thank you very much. Appreciate that.
[01:02:05.441] Kent Bye: Thanks for listening to this interview from Fitness Immersive 2023. You can go check out the Critics Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot of digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
