On July 17th, 2014, Malaysia Airlines Flight 17 was a scheduled passenger flight from Amsterdam to Kuala Lumpur that was shot down while flying over eastern Ukraine killing all 298 on board. This has been huge news in the Netherlands, and when someone asked writer Lisa Weeda with family from Ukrane if she felt guilty, then she got really angry and inspired to write a story about the crash from the Ukranian point of view. Part of the plane crashed in a sunflower field in Rozsypne, and so she collaborated with co-director Nienke Huitenga-Broeren in transforming a series of scenes and broader context of the ongoing conflict in the region into an immersive virtual reality story.
I had a chance to talk with Weeda and Huitenga-Broeren about their experiential design process of creating Rozsypne, the larger cultural context of Ukrane as a border country caught in the middle of many international power dynamics, their iterative process of translating specific contexts into a series of different scenes, and how they wanted to use VR to put you into the shoes of the Ukranian villagers from Rozsypne who discovered the downed airplane and the loss of life.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So continuing on in my series of looking at some of the narrative innovations out of the International Documentary Festival of Amsterdam's DocLab, So in today's episode I cover a piece called Rozipna. So this is a narrative piece that was trying to explore a new way to tell the story of this flight that was shot down back on the 17th of July 2014 MH17. It was a Malaysia Airlines flight flying from Amsterdam to Kuala Lumpur in Malaysia and as it was flying over eastern Ukraine it was shot down killing all 298 people. This was flying out of Amsterdam and so that was a story where it's pretty big there in the Netherlands and lots of investigations and one of the co-directors here, Lisa Veda, she actually has family from Ukraine and someone had asked her, you know, if she felt guilty and then it made her so angry that she wanted to try to give a broader context as to what was happening and try to tell the story from the perspective of a person from Ukraine that was discovering this crashed plane. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Nienke and Lisa happened on Friday, November 22nd, 2018 at the IDFA DocLab in Amsterdam, Netherlands. So with that, let's go ahead and dive right in.
[00:01:34.267] Nienke Huitenga Broeren: I'm Nienke Huitinga, director of co-director with Lisa Vera of Rossypne. And I would call myself an interactive producer, director and play around in all kinds of immersive designs. Going from transmedia strategies to storytelling and playing around with VR and AR.
[00:01:54.791] Lisa Weeda: Hi, I'm Lisa Veda. I'm a writer, but I try to do writing for other media, so kind of theater. I made a podcast, too. And now I'm the co-director for SIPNE together with Nienke Huizinga.
[00:02:07.802] Kent Bye: So maybe you could each give me a bit more context as to your background and your journey into virtual reality.
[00:02:14.638] Nienke Huitenga Broeren: Ah, good one. I think my background is a mix of going to university, learning about film. So it's kind of the original art where I started. But what really caught my eye was everything that happened on the fringes. So things that question what is film, question how we experience film. So very quickly I was very interested, I think way back in 2007, What is going on with film on the internet, but mainly when it's designed for browsers and what can you do when the viewing experience becomes a little bit more layered and interactive? For me, it was a very easy and intuitive crossover where I wanted to play more. When I walked into a great artist who wanted to do a mythical story about a man building his own wings, told from our current times, mixed with technology that we use today, but mixed in with an age-old dream of humankind flying, this project is Human Bird Wings. It was a transmedia story told on the internet with all the social media channels, but for us it was a mix of a very strong visual story told through YouTube clips. But for me, that effect when we feel approached by good stories from the media, It was a very immersive job for me to do, to create that kind of experience. But of course, this is fragmented. So I think transgressed from making something a little bit more coherent, creating a space where you can bring in that immersive effect, where a story becomes a world where you really can stand in. So I think that was how I got interested in VR.
[00:03:57.347] Lisa Weeda: For me it's a bit different. I think I already knew Nienke. We worked at a film festival, the Dutch Film Festival, way back when. And I did Art Academy as a writer, so I did creative writing for four years. And in my second year I thought, fuck this shit, I really don't want to do it. I don't want to write all the time. So I started to experiment with images, photographs, with audio and this was evolving and evolving more and more and I really like to experiment with form and just think of stories and which way they fit into a certain medium and not just I'm gonna make a book and I'm gonna make the story fit into the book which is sometimes impossible. So I was experimenting with that and then I met Nienke again in 2016 and we did a hackathon and it went very well, I had a lot of fun. I made a virtual, like a desktop game of a printer, a kidnapped printer in the war zone in Ukraine that was actually kidnapped by separatists. And the people that used to own the printer were artists and they could trace the printer all the way from the Donetsk city all the way down to Crimea. And it was an amazing story. But at the same time my nephew got kidnapped and killed and he was untraceable. So we tried to make that into a desktop printer interactive thing. within 48 hours and when that was finished we had a lot of beers and Nienke and I said to each other let's do VR and that's how it started I guess.
[00:05:27.042] Nienke Huitenga Broeren: I think that's the most honest version of the story. I think the romantic version of the story is that sometimes when a good story comes alive, even when it's captured in a browser, there's always this one bit where you feel, could you maybe even come closer? Like, how would this be if it's not just a flat screen? What can you maybe experience a little bit more up and close from this event or from this moment? That's how we started talking about VR. We never did any VR projects before this piece that's here at IDVA DocLab. So we learned everything on this project. And we were amazingly supported and collaborated with three amazing creative technologists. And one of them, we really want to thank, Frank Bosma, who is an amazing spirit that just keeps on going, which is very important in VR, because not everything will be as easy to create. You need to create a prototype and then discover it's not working the way you want, and then redo it again and again.
[00:06:30.089] Kent Bye: So yeah, we're here at the IDFA doc lab, and you have a piece that is essentially about this plane that took off from Amsterdam, was going to Malaysia, and it was presumably shot down with a rocket. So maybe you could talk a bit more about that context and why you wanted to try to transform this particular story into VR.
[00:06:53.253] Nienke Huitenga Broeren: I'll give a short intro, because I think Lisa has a lot to say about this from her personal angle. This moment, that year, 2014, 17th of July, when this incredible tragedy happened, I was, by chance, at the film festival in Armenia, in Yerevan. And I realized when I saw that on the news that it involved a lot of my fellow countrymen and women. And that of course is a very banal sensation that you suddenly feel so close to people you don't know but they are your countrymen and women. But also that the same trajectory was one that I flew over. And then the second thought is, if this is a route that we fly over for our holidays, for the more romantic destinations, suddenly there's this whole conflict and this complex politics that is laid bare, is open and is coming to you. And you realize, maybe as a conclusion, that you have never really thought about that country in the middle, called Ukraine, that is in between of a lot of holiday destinations and Western Europe. An interesting fact is that Ukraine, as a word, as a title of this country, is like a border country. That's the translation of the word Ukraine. I think a row of these thoughts just really grabbed me that I couldn't really grasp the sadness and the loss, but also the hurt that comes from war. And I just really wanted to, in a way, go closer and investigate how you can make that more human and understand it a little bit better.
[00:08:31.522] Lisa Weeda: Yeah, that's I think a little bit more the Dutch side, but also the difficult part of not knowing a war zone and not being in this plane, luckily. And the moment the plane came down, my family from Ukraine that also live in this war zone came to the Netherlands because my grandmother, their sister turned 90. there was a big party and they all came over and we had a lot of fun and I told a lot of people about it that my family was visiting and I was at a party and someone asked me so your family's from this area in Ukraine do they feel guilty about the downing of MH17 and that's the point where as a writer and a storyteller I thought oh now we need to we need to get Things in perspective because it was 97 days of war already in this area Crimea was taken there was a revolution on the Maidan Square in Kiev earlier from 2013 to 2014 for 99 days hundred people died and So actually I got very angry, then I wrote a piece asking more questions about the perspective and what kind of story do we see. And this is something I think we tried to combine in the Rossypna piece, make it more personal in a way by telling a story of one person, an elderly lady that lives in this war zone and experiences this war every day. And at the same time, this plane comes down and 298 people are suddenly gone. So these two crises meet each other but we tell it from the Ukrainian perspective because in the Netherlands this was a very big story. We still talk about it. There's a joint investigation team that still tries to find the people that downed the plane and I think it's really good that people try to do that but it's not the main point. It's about lives of people. You don't get these people back by doing that and also not by pointing at four guys, or men, or I don't know, who just pushed a button and downed a plane. So we try to tell the story about grief, I guess, and about war, and yeah, change the perspective, hopefully.
[00:10:45.668] Kent Bye: Well, it's a very striking setting, because you're essentially in a field of sunflowers. Did it actually come down in a field of sunflowers, or what's the significance of the sunflower field?
[00:10:56.752] Lisa Weeda: Well, the plane came down above a bigger area. It's like a five kilometer area. So the Rosibne village, that's where the nose of the plane came down. And yes, parts and pieces of the plane came down in very big, beautiful, Summer sunflower fields, so that's also the not ironic, but it's a weird image. It's like a juxtaposition you have to super wonderful image of beautiful flowers in the midst of summer and then all of a sudden there are dead people and luggage and stuff so for us this was I think the best metaphor also
[00:11:35.922] Nienke Huitenga Broeren: Maybe. I guess the role of sunflowers, as it is now, came in the very last bit of our development process. It was already there, because we felt we wanted to show you the most authentic, or most, let's say, convincing rendering of the landscape. Because Ukraine is known for the big cornfields, like the corn and flowers. It was always there. We built a very realistic version of a house and a household. It was a garden with a vegetable section. It was very realistic, also produced with photogrammetry techniques. But just before this summer, we had a sit-down with Frank, the three of us, and we thought we need to do one intervention, like pull something out to get you to the point where you feel the landscape is really growing towards you, like it's coming closer to you. So we pulled out all the walls of the house. So it's like this artistic or artificial trick where you understand you're in a house but there are no walls. So there's furniture but there are a lot of sunflowers around you that kind of mark the limits of the space. And then it kind of became right. So flowers for Ukrainian people are an important symbol. They enjoy flowers a lot, as we saw on our travels. Sunflowers have some kind of happiness and innocence and we juxtapose that with the life of our character is falling apart. So there are bullets raining from the sky and tanks going through her garden. So we play with those harsh elements that invade your life because some other people have an argument.
[00:13:17.077] Kent Bye: Yeah, there was some really striking moments in the piece where, you know, it starts off, you're in the sunflower field, and then you have, like, this tabletop scene that plays out where you're able to get introduced to this character, and then that blows up to be in the real scale, but there's no wall, so you're still in the sunflower field, but you're with her, and so you're kind of juxtaposing these two contexts together, and then bullets start raining down, and then you are then transported into, like, a grocery store, and you see these milk cartons, and essentially that's the only thing that she can get to milk. You have this really interesting, like, as you move your body through that space, you're, like, blipping out some of those milk cartons, and you see these other scenes come up. So maybe you could talk about that scene in particular, because it's one that's very striking in terms of very unique to VR, because you're moving your body around, but you're changing the space around you, and you're having the milk go away, and these other objects come in. And so I'm just curious to hear about that process of that.
[00:14:14.841] Nienke Huitenga Broeren: Why we chose to do that? That particular scene has, I think there are five or six versions of it. Because it was the hardest one to decide what should it actually do for you. So you see like a partial supermarket in a way, like it's one aisle with milk cartons on both sides. That's an actual iconic picture from the war. So a lot of products do not come in anymore. So some of the main elements that they produce in the Ukraine themselves are still there, but they become this in abundance product that you can only get. So that's kind of the truth element. And then we hid some grenades and religious icons on one side and on the other side you see some family pictures of Lisa's family in Ukraine. And we chose to do this because we, in the very last bit of our development, realized we do not really want to design for interaction coming from the user, so you can poke constantly at the system all the time. This is actually an element, of course, that will happen when you move your head, but it's hidden in the landscape. It's revealed when you Allow the landscape to do revealing for you. So it's yeah, it's a little bit of a reward for being curious enough to Move around in this landscape. So it's like a it's not really an easter egg, but it's like a small detail that we liked Kind of the sadness behind the harsh situation. Yeah
[00:15:43.860] Lisa Weeda: It's hard to add to that. We had so many discussions about these milk cartons. Yeah, I don't know man. I think I loved our decision because in earlier prototypes we actually had physical objects. For example, in the end scene there's this stick. Nina, the character, she holds a stick with like a white piece of cloth on it. These were the sticks that miners took into the fields where the plane came down to mark all the spots where people came down or stuff came down or luggage. And we actually had this stick so a person could hold it in virtual reality. And it was amazing, but it was like this interaction where you could poke the system. And it was also kind of a metaphor, but it didn't work because some people started to do ninja tricks and old people started to use it as a sword. So it was completely distracting from the actual story you wanted to tell. So the word scattered, which Rosibne is freely translated, it's scattered. It's a really nice find, I guess. So that was kind of the code we wanted to use for all the scenes and I think the pictures, it's like a scattered life. You have all these mail cartons but you just want your normal life back or all the family members clothes and grenades with the three holy icons. It's like this typical Ukrainian thing but it doesn't fit anymore because it doesn't provide for you or there's no safety anymore. I think, is that a good explanation, Inge?
[00:17:16.007] Nienke Huitenga Broeren: I think so.
[00:17:17.685] Kent Bye: Yeah, I think what I've found is that there's a certain amount of, like, dream logic sometimes, both in the language of film, but also in an immersive experience like this. And the milk cartons, it's clear that, like, you get this real experience that there's not a lot of other options and that it's just repeated. And then when you transmute those into grenades, that's obviously the metaphor for war, but there's also the religious iconography around it, and then some random family photos. And at the time, for me, it wasn't necessarily clear. It was like, OK, what's this mean? But there was also an interesting interaction as I moved forward. It's actually changing the world around me. So it's like taking one context and blurring it into another context. And so maybe that's another metaphor for the blurring of the context of things getting transgressed and stuff. But I find it a little bit of like interpreting a dream where there's some things are clear and other things aren't necessarily as clear and that was one of the instances where I Didn't necessarily know why there was like religious iconography in the scene in that way and juxtapose with the with the grenades and what that meant
[00:18:18.094] Nienke Huitenga Broeren: I'm just looking at Lisa if I'm going to explain this the right way, but first your interpretation is very interesting and I think one of the correct ones that... Yeah, well, of course, a work when it's out there for an audience, they can have their own conclusions and interpretations. Yeah, it's put there as symbols, as details that give you not a clear clue, but it will give you a vibe of... the things that come into play when you're in this situation. And I think the grenades and the icons, religion is still very prominent in Ukraine and the grenades. And I'm looking at you, Lisa. I think war has marked this country for a very long time. In some ways, as we know in Europe, war and religion often have some kind of a relationship. And what I learned from you, Lisa, is that a lot of the reasons that it's for the Russians, they're interested to keep Ukraine instable. It's because they have this very long history where Ukraine was once a part of Russia. And there is this, of course, intertwinement of weird history that they try to explain through some kind of religious history or some kind of weird claim on Kiev used to be our main capital. So I'm not even that much of a Russia expert or Ukraine expert, but the intertwinement of these very old war and religion still play a part also in this very new war. Did I explain it the right way?
[00:19:47.978] Lisa Weeda: Yeah, it's such a difficult topic. And that's also why we kept very far away from this in the virtual reality piece. We tried to not be political. I think that's a really important thing to mark. But for example, we have three language options. So you can go Dutch, English, I think you did that one, and Russian. And that type of Russian dialect is the specific dialect of that area. But that's, I think, the only political choice we made. There's so much to say about this, but the borders of this discussion are so difficult. Because, for example, my grandmother, she's not just Ukrainian, she's also Russian. She's also a bit Cossack. So there we are already lost, but it's about nationality and not about nationality. It's about not being Russian, still talking only Russian or choosing to talk Ukrainian right now. So this part of the country has always been, it's like a block that you, like a Lego block. You put it first on one side of the line and then you put it on the other one and they're just, they're bounced around all the time. So it's a very hard thing to talk about.
[00:20:58.113] Kent Bye: Well, as we're talking about in the past of this specific plane crash, as I'm flying in from the United States, you know, the big thing that's happening there is the impeachment hearings of Donald Trump. And Ukraine, of course, is in the middle of that as well. And, you know, just however that's going to play out, it's going to play out. But as I'm talking to you, I'm just struck by how, again, Ukraine's kind of thrown in the middle of this almost as a pawn in a larger chess match between these larger political forces so I don't know if that if you have any comments on that in terms of a continuation of this story of how Ukraine has been in the middle of these types of conflicts.
[00:21:34.287] Lisa Weeda: Yeah I just there was a tweet and I just retweeted it from Christopher Nolan because I just saw that Trump was on Fox News and he said some stuff about Ukrainian ambassadors and Zelensky and about his picture not hanging in the embassy in Kiev And then this journalist, a very good journalist, Christopher Nolan, he typed in Twitter, I come there every week and Trump is still hanging on the wall, he's lying. So I know Trump is not the best example for fact-checking nowadays, but yes, this country keeps going back and forth and this Ukraina, this border country word, It has been, like Nienke already said, it's always part of their history and they can't seem to get rid of it because it's our European safe space when you look at Russia and I think Trump wanted to give money, I don't know, or Canada for example. There are a lot of Ukrainians living in Canada. They support Ukraine, but it's also a statement to put in. There's not much army power in Ukraine, so they cannot fight. Europe doesn't want to fight. What do you do with NATO, the EU? And I think that's also the difficult part with the MH17. do some workshops in the summer and there I did one working day with people who want to explore writing or storytelling and we tried to rebuild also like the MH17 site, crash site, within the woods. It was really nice and we talked about it a lot before we started working. It's a one day course. And one man said, and I still think about it sometimes, he said it was not really, in the Netherlands we called it a crisis. And he said, I think maybe it's an act of war, but everybody's afraid to act on it. And I think that's where the problem already starts, because we just don't want to burn our hands on this situation.
[00:23:32.388] Kent Bye: Yeah and going back to my experience of the piece at the very end I had a really profound moment when the main protagonist character is walking by and I was kind of like looking off and other things and I kind of noticed that she had kind of walked around a corner a bit and so I had to walk actually physically walk through space and as I walked through space I see on the ground like this backpack and this stuffed doll this monkey and But there was something about that moment of tying back the loss of human life into this plane that's crashed in her field. And I just felt that that was a really elegant way to kind of symbolically really get at the loss without, you know, having to actually show any of the bodies or anything. So maybe you could just talk about the construction of that moment there as you're ending.
[00:24:17.957] Nienke Huitenga Broeren: Yeah, I really enjoy your description because it's actually what I wish for for the audience. I think it's the one scene that we kind of, a lot of the parts are feel poetic. You already mentioned dream logic, so there are some elements that feel like not reality. This is the only scene that we try to direct as a scene where you are together with her in that sunflower field. You're walking through the flowers and there's that moment, and this is based on truth, where the locals were the first ones to check, are there any survivors? What do we need to do? They put blankets over the fallen bodies. Of course this scene we left all the gruesome stuff out because we felt that that goes beyond the point. But we kind of wanted to reconstruct the thing that press doesn't really allow you to do in a more human way. Like often it's sensation. Because the pictures are the thing that make people go to your website or watch your show. So these gruesome details we wanted to kind of boil it down to you are with them in that moment where she has lost people in the war but she's also the first one to care for your loved ones who were innocent and shot down. I think that's the most straight-up scene we wanted to make. It belonged to a version of Rosypna that we had finished in March this year, which was completely different with all kinds of physical interactions. And it's the only scene that survived from the whole different version that we have. Because originally you would walk down with a stick, so you also had a stick like her, and then you could kind of poke the ground. But it was just kind of choosing to not talk and just look, just be there. not supposed to be meditation or being zen, but it's just kind of, you know, just realize that you can, this is it. This is where we are all talking about, and it's basically human loss, the lives that are lost in war and tragedy. So I think we directed this one with the amazing actress Anastasia Lubchenko. She's an Amsterdam-based mime actress. She's way younger than the character you see in the VR piece. But she's an amazing actress because she kind of, yeah, it's a little bit method acting. So she made the movements of our Nina, who is about 70. Yeah, we had a, I think we all fell silent when we were recording that scene. So she had this motion capture suit on with these, you know, radio points on her body. She was so convincing, like we were dead silent in the studio. It was just a walk from A to B that we mapped for her on a line. We could already see, even for us, she transported us already to that place, because Eastern Ukraine is not a place that we actually visited. It's too complicated and too unsafe for us to go there. But we did go to Ukraine to get an idea of the landscape, so we traveled around Kiev. Because we had that image in our head and then her performance, that's what the scene is all about. Kind of transporting people to the most honest moment where nobody can have weird opinions about using a word of crisis or not.
[00:27:45.382] Kent Bye: Yeah. Well, when you think about storytelling and writing for virtual reality, do you think of the traditional three-act structures? Or is there something new about how you actually architect and write a piece like this?
[00:28:00.403] Lisa Weeda: I think we had some... No, there was no three extra, there was nothing. We just started with a lot of research and things we knew and things we wanted to tell and I think we started shaping them in a way we liked. Trying to put stuff together, trying to 3D build a lot of stuff, telling Frank to make shit and then he said it doesn't work and then we said it has to work and then we had to try again and we had him mold a complete... What's a curl? What's the word for it? hole in the ground and it took him I think three months to make it look very convincing and we took it out so I think it's more like the visual research in my work as a writer that's very important I really don't like the like the original or the classic way of writing, it's not for me. So it's more like looking for scenes and see how they fit together and how they talk to each other and how this as a complete world talks to each other again and then if we have to start restructuring everything again and then deconstruct it again, put it back together, go back to paper, test it. So it's more a test trial and I don't really It could maybe be like a 3X structure right now, but we never thought of it like that.
[00:29:17.752] Nienke Huitenga Broeren: I think what's maybe interesting to know for your listeners is that we started developing the story simultaneously on three levels. So Lisa wrote a couple of scenes. A kitchen scene, a garden scene, a cellar scene, which is the first scene you see. It's like a miniature cellar under her house.
[00:29:38.480] Lisa Weeda: The milk scene.
[00:29:39.261] Nienke Huitenga Broeren: The milk scene, the supermarket scene that we talked about earlier. The second thing that we did was kind of make a mapping. It's like a visual mapping of reference pictures of Ukrainian life in the countryside on a really big white canvas where we... Also to help our designers to understand, so where should I place a tree or where... I don't know, we have like... What's the weirdest element? Oh, there's like a big fish on the wall somewhere. So we kind of made reference maps, but when you're making these reference maps, a story also kind of is created. So the space dictates a little bit how you're gonna pace your experience. And I think the third one is the interaction for the user. And I think, in a way, we were amateurs, so we thought, let's do this all simultaneously. Well, we need all these elements to make it work, so why not just work on them all at the same time? So at some point that felt like trying to put a triangle in a square or a square in a triangle slot. I don't know. So the thing with storytelling is when you have written scenes on paper, they try to be dominant in a space, but the space also tries to be dominant. So when you try to craft the experience, you realize, oh, we need less text. Yeah, let's get out all the text. Let's just keep a few lines. And then you're like, OK, this is better, but now we don't understand what's going on. So it was a lot of going back and forth between how much can you actually listen to in VR and how much can you actually let the space tell you. So Lisa has gone through some darlings killing. Frank unfortunately also had to kind of pull out a beautiful hole in the ground. But also the hole, we had a very elaborate garden, you only see one little bit of it. Yeah, so storytelling and spatial experience, they are both very dominant for our senses, so it was like a box fight between them. That was a process, basically. Yeah.
[00:31:38.478] Kent Bye: Yeah, I often think about how architects will think about designing it, actually design it, build it, and then there's, like, in a film, there's, like, the post-production, so there's, like, the three phases, and I feel like that approach within game design is, like, the opposite, where it's just iterate and try to find out what the interactions are, and having these two worlds come together of that, the waterfall, pre-planning, linear approach, versus trying to mash that up with the iterative approach, it feels like that everybody's still trying to figure out how to exactly do that. So for me, it's just interesting to talk to different creators to see how they're trying to take these two things that are fundamentally opposite of each other, but yet see how they can work with each other. Because you can use both pre-planning, but also not get so locked into the script as to not change it based upon after you've actually created something. So it sounds like just in the process of you creating the piece that you actually had to do that and find ways to kind of mash those two things together.
[00:32:33.994] Nienke Huitenga Broeren: Yeah, it's exactly that. I remember those moments where we felt, well, almost crying. What's the solution then? Because a voiceover narration gives a lot of freedom to not having to create meaning through interaction. So it gives you a very strong base to let someone go free. So just, you know, walk around, look around. but listening, it takes all your attention. If I would do it again, I don't know if I would have a better way, but I'm pretty convinced that when you're trying to understand your scene, you do need to work both in words and visuals at the same time, or at least make a combination like a mapping, or if you're a good drawing artist, use anything you have to make sure you can show it and tell it. because the voice actor really needs words to understand what he or she is going to say. But that was where we made the big decisions that finally when we were kind of making little test videos, we made a screen recording and then just play around. So where would you want to hear something? Oh, maybe we'll just keep it silent here. Okay, so we moved the narration a little bit further along. Yeah, you need these weird hacks on tools to be able to make everything work in the right position. But yeah, there's not a really smooth way to make VR, I think.
[00:34:04.176] Lisa Weeda: No, there's not. And I think one of the wonderful elements is the soundtrack by Mark Ijzerman, the sound artist. It was great because we already worked with him I think one and a half year ago and he completely freaked out. We made a lot of good stuff and wonderful music and then we started to do stuff in FMOD. It's like this tool where you can make 3D spatial sound and then our whole crew was crying. in the studio, getting very upset. And then we did a prototype, another prototype, and just some weeks ago, before we had to finish the final version of Rossypne, Mark came back and he made this wonderful soundtrack. And I think as a podcast maker, audio is also, and for film and for a lot of media, it's such a wonderful tool and I think Mark did the final layering of the emotional part also. He did it so well and he filled also a lot of, not a lot, but he filled the blanks where we still needed stuff to get you into this feel, but also feel safe, but at the same time, not be afraid to have a little bit more freaky sounds or a bit more dark sounds. And I think sound is a wonderful element in VR and I would really like to explore that in the future.
[00:35:27.672] Kent Bye: And finally, what do you each think is the ultimate potential of immersive technologies and immersive storytelling and what they might be able to enable?
[00:35:38.864] Nienke Huitenga Broeren: Very good question. I should have prepared something for this one. For me personally, because it's maybe too easy to talk in big trends, but where I see myself with the development of this technology, I just love being able to travel to very special spaces. So I love traveling in real life, but I think the thing that really gives me goosebumps is that VR can It can transport you to the importance, well, maybe dreamlike spaces, but also real spaces that you maybe need to see with your own eyes, and maybe in this case, then reconstruct it in virtual reality, that can really give you different perspectives. So I can imagine that important parts of history can be seen from a new perspective. Me, I'm more working with the poetic way of storytelling, but even when it goes for more journalistic or objective revisiting of facts, for example, I truly believe that the immersive quality is getting better and better in how we have more high resolution renderings of our world. I think the places we have lost, maybe when we're talking about climate crisis, I think VR is going to have a very special role in convincing people to have a different look, have a look again. the things that we take for granted or the things that we have forgotten. I think that's maybe a very storytelling kind of answer, but yeah, that's what changed it for me, definitely.
[00:37:14.899] Lisa Weeda: I think perspective is the biggest word. I know a lot of people right now are not that virtual reality savvy yet, so it has to be a little bit more accessible maybe in the future. That will be a good start. But as a writer, I think more writers should actually do this. Because I really like books and I like reading about other perspectives or worlds in that medium. But it opens up a totally different way of thinking and thinking about how can I use the context of this story and how can I actually shape stuff that I cannot shape on paper. Because some things are actually unshapable, like a kidnapped printer. A kidnapped printer that is talking on paper like, I'm kidnapped. They have taken me. I'm in a car right now. It sucks. I tried it. It's horrible. So in that way, there are other ways to explore it. And maybe that's not really the answer you're looking for. But I think the openness to the medium, I think it can help and do a lot. And the rest is all Nink said.
[00:38:22.028] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:38:27.835] Nienke Huitenga Broeren: Yeah, well I think our sound artist has a request. So Lisa mentioned that we work with FMOD, which is this kind of interface plugin for making dynamic sound effects in a Unity space. It was amazing up to this point where we tried to integrate a piece of recorded voice that belonged to the motion capture elements. So our character was supposed to say a lot of things in Russian. We all pulled it out because what happened, and this is a question to the community, it muffled her voice so much and we just couldn't find the setting. Like the one, I don't know, switch that we needed to flip or this parameter that we needed to change, but it just fucked up all the voice recording we did with our voice actress where she enacted a lot of the speaking the character was doing in her household. So you see a silent character at the moment. Because Avmod just really fucked up her voice. And we have been looking for a solution for two and a half years. And our sound artist has asked, I think, 12, 15 people. We can't find a solution. So if anybody knows... There actually was a moment where a sound artist cried a little bit. Yeah, he did. So anyone, please. Yeah.
[00:39:46.098] Lisa Weeda: Yeah, I think that's the main thing. And please, all editors, writers in the world, Google VR. Look it up. Try to explore it, because it really is amazing. And it helps. And get out of your book shelfy life, please.
[00:40:03.253] Kent Bye: Awesome. Great. Well, thank you so much for joining me on the podcast. Thank you.
[00:40:06.896] Lisa Weeda: Thank you. Thanks.
[00:40:09.302] Kent Bye: So that was Nienken Hausenhut. She's a director, producer, and director and working in immersive storytelling, as well as Lisa Veda. She's a writer and director. So I have a number of different takeaways about this interview is that first of all, Well, as I cover these different narrative experiments, for me, it's just interesting to talk about their process and to hear a little bit more of trying to pick a story to break down the elements of that story. And it seems like this story was obviously a very complicated story, but they were trying to boil it down and to do something that you couldn't do on any other medium. And I have to say it's very striking some of the scenes where you're in this field of sunflowers and then the culmination of the piece is you're walking with this woman who is from Rosipna and she's walking through the sunflower path and you kind of go around the corner and you discover this plane that has crashed and then the artifacts of somebody that was clearly a child. So you get this direct experience of the loss of human life. And so as they looked at this story, there's a lot of political dimensions and other aspects and trying to figure out who actually shot down the plane and whatnot. But they really wanted to just boil it down to just this was a simple story about the loss of life, but also a complicated story of all the other deeper context of what was happening in the Ukraine at the time and to really tell it from the perspective of a woman who's kind of caught in the middle of all this war. And so the challenge that they set forth is to try to, you know, set this broader context and to use VR to be able to put you into specific scenes and to allow you to glean what it's like to live there, so you're seeing her in her home and this tabletop scale she's small and you're able to get a sense of like you know she's just living the simple life in the middle of the country and then you cut to going into this store with just everything's just milk cartons that's the only thing that they have and so get this sense i think they even have some narration at that point saying you know there aren't very many other options for food because of the war that's happening. And then as you move into these more cartons, the more cartons kind of like blip out, like go away. And then you're replaced there with like these grenades and religious iconography and these family photos. And so I think this is a theme that I see a lot of how do you transform a context into a deeper story where you have these different objects that may mean something very specific to a culture, but to use some level of symbols to be able to like talk about a broader picture. So to use the religious iconography juxtaposed with the grenades in this supermarket that are moving out, you're kind of blurring the context and seeing just how the war is starting to overtake their lives, but also there's a religious context to all these conflicts for many, many years. But I think that's generally a challenge I see with immersive storytelling is to pick these symbols that are going to somehow be universally translatable. I mean, later in the piece, you see in the sunflower fields, you see these tanks going through these fields. So you also get this sense of being there in this place in the midst of being in the middle of a war. And this woman who's there is on the front lines discovering these dead bodies. And I don't think they actually have this within the experience, because they're not showing any of the dead bodies. But part of the point they wanted to say, connect people in Amsterdam to this story in a way for these people who are the locals, who are the ones who are on the front lines, and trying to cover up the dead bodies, and to discover them, and try to recover them so that the remains could be sent back into Amsterdam. So just in talking about the process and unpacking a little bit, there was this exploration, maybe in different iterations, designing stuff and seeing what worked, what didn't work. And so they kind of broke it down into these three different phases, which was kind of the script and the writing of the different content, actually building up the spatial context and the worlds and to really populate them, and then to figure out what people are going to be doing in it. and it sounds like in early iterations as you're walking through this beautiful like photogrammetry sunflower field they had these sticks and that you know people were using the stick to try to interact with the environment and so it was supposed to be just a walking stick but yet if you give somebody an interactable object as they're walking in through VR then they're going to want to try to interact with that interactable object with the world around you to see how plausible it is and so if you give people too many options to do with some of these objects you're giving them, then I think there's this tradeoff between that agency and the story that you're trying to tell. So at the end, they ended up dialing back a lot of that type of interactions. And the interaction you end up having is just more of a you moving your body through space and to be able to walk around the corner and to discover a new part of this field to kind of reveal the down plane and the different fragments of the plane and these artifacts that are representing the human life that was lost. So I feel like it's still very early days of trying to explore these different types of storytelling modes, using spatial context to be able to start to try to tell different aspects of the story. So they did this interesting thing where they were blurring the context. You're in the middle of this sunflower field, and then you're looking at this house with like a small scale, and then it blows up. And then once it blows up, they actually remove all the walls. So you're still being in the context of being inside of the home, but kind of blurring the context of still being in the sunflower field and then they start to rain down these different bullets. So what they said is that, you know, they would write these different words and then, you know, have different script and then they would get into these spaces and then they would keep on dialing back different ways of talking. And then they moved more towards trying to show things rather than tell things. And so I think that's one of the fundamental precepts of a film is to try to show and not tell whenever you can. And so within a virtual reality experience of how to actually show different aspects of the story, and then do things like raining bullets down from the sky, just to give you the sense of orient you into this specific location, and then start to add more symbols and things that happen within that world to be able to give you even more context as to what's happening. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a list of support of podcast. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. Just five to $10 a month is a great amount to give and just allows me to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
