I interviewed Aufwind (Upwind) director Florian Siebert at Venice Immersive 2023. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast about immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com. So continuing my series of looking at different immersive experiences from Venice Immersive 2023, This is episode number 32 of 35 and the second of two of looking at the context of career and workplace. So this piece is called Offwind. It's translated to Upwind. It's by Florian Siebert. It's a story that's looking at the composite characters from the early 20th century in Germany, looking at women who are trying to make it in aviation. And so they're being sabotaged by the men who don't want the women to be in aviation. And so then the context of career and workplace and looking at the domains of sexism and historical aspects that are being represented within the context of this piece. So this is a volume cap captured piece that the different actors that are working with each other being projected into these different scenes. And so the primary center of gravity of this piece is both the embodied and environmental presence, just really exquisite world building that's happening here, a lot of historical reconstructions of these different locations. And it's got this real cinematic feel as you go through these different places, as you traverse through them. And it's also got different aspects of embodiment because there's different haptics feedback. It's got a motion platform that you're on. And so there's, you know, the secondary mode of presence is the different aspects of interactivity. So you end up driving around a car through some of the scenes. There's some moments where you're flying around an airplane and through that you're experiencing haptic feedback as well, but in the context of this motion platform that's moving around. And then the third aspect is just the emotional presence of the story that they're telling. So, that's what we're covering on today's episode of the Voices of VR podcast. So, this interview with Florian happened on Sunday, September 3rd, 2023 at Venice Immersive in Venice, Italy. So, with that, let's go ahead and dive right in.
[00:02:12.918] Florian Siebert: Hi, my name is Florian Siebert. I'm from Dusseldorf, Germany, and I'm a director, writer, and sometimes producer, depending on the project. Great.
[00:02:21.740] Kent Bye: And maybe you could give a bit more context as to your background and your journey into working with VR. Sure.
[00:02:27.913] Florian Siebert: Well, so my background is actually coming from the classic film, so to say, making documentaries and also developing scripts for fictional feature films, so to say. And my first contact with VR was about seven, eight years ago. with the producers and friends from Neon Real, which I made Aufwind with, and we made something called Time Ride VR, which is actually, there was an actual tram, a streetcar from the year 1907, which was actually built, physically built, and you sit in it, and then you get your VR goggles on and then you can drive through the old town of the city of Cologne and experience the atmosphere, the people, the sounds, whatever. So you're actually there 100 years ago. And that was the first project we made together. Then it developed. Another project we made was called Essen 1887, which takes place in the city of Essen. It's actually an AR project. You have AR glasses on and you walk through the city physically. At certain points there are scenes, like movie scenes, with real actors. that tell you the story of Alfred Krupp who was an industrial mogul at the time in the late 19th century and this day we are telling about is actually the day of his funeral and since he was like kind of a determining character for that time and for that city it was very important for the city of Essen when he passed away so it was kind of a changing of times so what comes next how is the city going to develop and so on We were telling the story, but as a, so to say, movie you can walk through.
[00:04:15.971] Kent Bye: What was it that originally drew you to virtual reality as a technology?
[00:04:20.540] Florian Siebert: Well, I think it was like the idea of telling stories in a new way, to be experienced in a new way, that really interested me, because I like the classic storytelling of film. So I never thought this is not working or not good, but I immediately thought in virtual reality or augmented reality or whatever the new medium is, there are new tools, new way of telling stories, new experiences, so to say.
[00:04:50.966] Kent Bye: Right, so maybe give me a bit more context for how Offwind came about.
[00:04:55.970] Florian Siebert: Well, Offwind, the basic idea came from Jan, from one of the producers. And Jan and Peter, they approached me about two years ago and told me about the basic idea, about the two female pilots in Germany in the early 20th century. and what they had to struggle with and what their dreams were. But the men at that time, they didn't really like women in aviation. So they really had to struggle. And I really thought this is an important story to be told. And I thought this is the perfect story to be told in virtual reality. But at the beginning, we didn't have much more than basic places, basic times or events that we would want to tell the story about. And then we started developing in terms of story, but also in terms of how can we actually do this? How can we tell the story in virtual reality and use the tools that are there? And then I wrote the script and we always discussed how this is doable. So I wrote a scene, for example, And then I went to Jan and Peter and we discussed it and tried to figure out, is this actually a thing we can make in virtual reality? Or do we have to alter the scene or the storytelling? And that was an ongoing process for the whole script. And in the end, we had a script that we agreed on and said, well, this is strong, this is emotional, it's a great story. And then we approached the Volucap studio in Berlin. and talk to them about how we can actually record the actors and actresses. We knew about volumetric capturing because we had used it before, but it was a little bit different system of volumetric capturing. and now we had more, like, the bar was higher now. We needed more resolution, we needed more room for using it, and so that's why we partnered with Volumecap Studio in Berlin, because they were, or still are, I think, one of the best volumetric studios in the world, actually.
[00:06:59.216] Kent Bye: Yeah, I think I had a chance to visit their studio when I was at VRNOW in 2018. It's not actually in Berlin, but it's in another city, what's it called?
[00:07:07.724] Florian Siebert: Babelsberg. It's actually where the old Babelsberg studios are, where Metropolis was shot and all the old classic movies.
[00:07:15.566] Kent Bye: Yeah, so I was able to check it out and have an unpublished interview with one of the folks from Volucap. So maybe you could roughly describe what's the difference between what Volucap is doing versus, say, Microsoft Mixed Reality Capture. I imagine it's a similar type of capture system, but what are some of the differences between the two?
[00:07:33.796] Florian Siebert: Well, I'm not a technician, so I can't really go into detail what specifically the technical implementations are or what the differences are. But what intrigued me about volumetric capturing and what they are able to do is that you can actually record the art of acting, like the real performances of actors. Because I knew motion capture, I had worked with it, and performance capture as well. I mean, it's a great technology and it's often used for a lot of projects, even in feature film. But I still thought that there is something missing when you transfer the data of movement to a virtual character or to a 3D character. Something's lost on the way. And volumetric capturing actually was the possibility of really conserving all of that and keeping the real emotion, the real acting, the real humanness. My producer Jan yesterday said volumetric capturing captures also the soul and that's actually pretty good. That's actually what it really is. So that's the point, that was the point where we said okay let's do this project with volumetric capturing. All the actors will be shot in the volumetric capture stage and then we will have to build the whole environment and everything in the depth of our scenes around it.
[00:08:49.587] Kent Bye: Are you able to do a volumetric capture of more than one actor at once or do they have to do it like one at a time?
[00:08:55.973] Florian Siebert: well It's like this, it's a pretty small round stage, about maybe three meters in diameter. And you have all these cameras, over 30 I guess, and they are recording the volume of this stage. So if you put one person in, you have more information than if you put two persons in, because then two persons have to share the volume. So that's a pretty good image to realize what actually the limitations are. So the more people you put in, the lesser quality you get. And that's what they told us. They said you can put two people in, but then we have to see how much quality can we get with two people in it. And I decided very early on that I wanted my two main actresses, the two main characters of the story, I wanted them to be in there together. Because they have to have the energy, they have to act together. We took the risk of lesser quality in order to get the realness of human interaction. But most of the other scenes and other characters we shot separately.
[00:10:03.514] Kent Bye: Yeah, because there's always the challenge of eye lines when you have people at different times. But also, you know, if the actors aren't being able to act off of each other, then that is a whole other dimension where they have to similar to like a blue screen where you don't have the context environment. But in this case, they would oftentimes with other systems, you would have to capture folks one at a time. So I could see why you would say that this is a better system for actually capturing acting performances with multiple people.
[00:10:30.170] Florian Siebert: Yes, absolutely. I mean, of course, we also had the issue of eye lines. So I had to prepare very detailed where the actors are in relation to other characters, in relation to props, in relation to, I don't know, an airplane where they have to look at or whatever. So that was quite a challenge because normally for a classic movie you have the one camera position and you know your frame and then you can say, well, there in the background there's something happening and you have to look there but when it's all around you 360 then it was a real challenge for me to have all that in mind and get all that like worked out like beforehand but also on the day of shooting when there is like the stress and the energy of a shoot and then you'd like to focus everyone on this is your mark, this is your eye line. Then he goes from there to there and you have to follow him with your eyes and whatever it is. So it's really challenging but I also was very interested in this challenge like could it be done? Not necessarily like am I great enough to do it but Could this be done, this storytelling, this story with this complexity in volumetric capturing and even with other technologies combined? I mean we also shot extras for the background with a classic green screen shoot. So we had to put them in and this was also a challenge in terms of performance, in terms of that they can't be too close but they can't be too far and all that stuff.
[00:11:55.655] Kent Bye: And so when I was talking to Liz and Michelle, they were talking about how there's a cinematic quality to this piece where you have a lot of really exquisitely designed environments and buildings, felt like very high resolution, and it's also a bit of a spatial journey because you have a motion platform and you're driving through cars, you're flying in airplanes, and so you're really moving us through space in this experience. So I'd love to hear a little bit about your design process because you talked about the script and I don't know if the script was really nailed down first and then from there starting to build out the environments or this relationship between the worlds and the world design and blending in the story on top of that and also the other motion platform capabilities like how did you start to blend these things together in what order?
[00:12:39.867] Florian Siebert: The process of developing and also producing the environments was kind of a ping-pong game. Because we are quite a big team working on this project and while I was working on the script, others were already starting designing, for example, the big construction facility or the airfield or whatever. We used old photographies and other data we could get to really build the buildings and the airfields and whatever exactly as they have been. As far as we could see or tell or find out. So this is all actual. For example, when you sit in the car behind the two women and you drive through the city of Essen, this is completely built, new built in 3D, but it's all real. It's all like the actual buildings from that time. We had a lot of references. and we actually recreated this. It's a famous street actually in the city. And yeah, this was quite some effort. And as I told earlier, when I created a scene, when I wrote a scene, then I was always going to producers or to the artists, digital artists, like to tell me, is this doable? Can we do the scene in that way? Or do we have to change something? Or the other way around. They had built something in 3D, came to me, or I came to them, but they asked me, Can we tell you a story in this environment? Can we use this environment the way we built it? Or do we have to change something? And it was quite a team effort, actually.
[00:14:06.724] Kent Bye: Yeah, and maybe talk about the motion platform. At what point did you start to play around with integrating a motion platform that's moving up and down and with different wind haptics as you start to experience this experience?
[00:14:18.651] Florian Siebert: Yeah. Well, we actually, very early on, we had the idea of using the multi-sensory technology, like emotion, seed, smell, wind, vibration, and so on. But we also decided that we would not want to have some kind of technical gimmick experience. So if we use this technology, it should make sense. And there should be a reason why we use it. But we knew we could do it, like we could use all this technology. We didn't really know how specifically we would put all that together and how it would work out, but we knew we had experiences with motion seeds, with smell, with wind and so on. So when I wrote the script, I always had in mind, okay, this is, like I know how to write a script for a movie. But now we have these other elements, like now we have emotion seed, now we have smell, now we have wind. So how do I put that in the script so that it makes sense, so that it's exciting, so that it immerses you in the story in a different way or even in a deeper way? And yeah, that was actually kind of the approach I was using while writing the script. And again, I was going to the team, to the producers, or to the coders, or to the artists, and always asking, can we do this? Does it work? Does it make sense? Or do we have to adjust something? And actually, that's the way how we did it with the motion seed and the smile and so on.
[00:15:41.407] Kent Bye: Yeah and I'd love to take a step back and have you tell the broader story of these two women and the time that they lived and what this arc of their journey is over the course of the story.
[00:15:53.053] Florian Siebert: We tell the story of the two first female pilots in Germany and our story takes place in the year 1912. They were both women in their mid-twenties with high dreams, high goals, but the world of aviation was dominated by men. And they didn't want to see women around there. And they actually manipulated their planes. If women got a chance to be in a plane and to fly a plane, there was always the risk of a man manipulating it. break something or take something out or whatever so that they would really crash. It actually happened. So this was something that drew me into the story that I thought well this is unbelievable but at the same time it has to be told and it has to be told in virtual reality because if you experience this like it can maybe even change your perspective on how we see things today. So it's about these two women, and we had all the actual events that were taking place around that time, and we know the places that were important, but we also decided that our story is supposed to be fictional. So we're not saying we tell the exact story how it happened, but we take the actual people, the actual events, the actual sites, and put them together in a fictional story. and actually the way how Aufwind will be presented when we do the location-based experience, when we open up the store and after that we will also do a roadshow, then there will be two rooms or two stages before the actual film, where you can learn more about the time period, about what it really was like to be a pilot in that time, like the actual life stories of these two female pilots were and so on, because we know that we are telling a fictional story, but we want to give the audience the actual background of what was really happening.
[00:17:50.350] Kent Bye: And so even though these two main protagonists in your film are fictionalized, maybe you could just elaborate on where they're situated in each of their careers and their aspirations and, you know, just a larger relational context under which they're operating that helps to give a broader sense of that place in time.
[00:18:08.222] Florian Siebert: Okay, so they both were pretty much well situated, so they were not from poor families. They had a background of at least a little bit of money and like chances for their lives and they had the support of their families because otherwise I think they couldn't have done it. The times were totally different to today and women in a field where they're not supposed to be through the eyes of the men, they didn't have a chance. And like Melly and Charlotte both, they had like an entrepreneurial spirit in themselves. Because at that time, if you wanted to fly, you couldn't just buy a plane. I mean, you couldn't go to a store and buy a plane. You had to construct and build it by yourself. You had to pay for it by yourself. So they really got into how should they be constructed to be safer, to be better in the wind, or whatever it was. And also, how can we find a way for more girls and women to come into aviation? So they, I think both of them, opened up a flight school so that other people, and especially women, could get into flying, into aviation, make their pilot license.
[00:19:21.325] Kent Bye: Yeah and in this experience you are sitting on a motion platform that is moving around maybe in three degrees but it feels like it's reactive to a number of moments in the piece where you're actually riding in the back of a car then you have a chance to drive a car because in this motion platform there's actually a steering wheel in front of you and there's some wind machines that are in front of you that are blowing wind whenever you're in this motion. And then near the end, you get a chance to actually fly one of the airplanes. And so I'd love to hear you elaborate on this process of designing these more embodied and interactive motion platform sequences within this piece.
[00:19:57.202] Florian Siebert: All right. So you're experiencing this story through the eyes of a little girl, through the eyes of Tia, which is a 12-year-old girl. And I think you can experience the situation of that little girl best in a way if you really feel it, if you really get the feedback to your body. That's when all these technical sensory aspects come into play. You are actually a protagonist of the story. The other characters talk to you. They interact with you. I mean, you can't answer, obviously, but still, you're immersed into this moment, into the situation with other characters. And that's the same reason why we use the motion seed or why we use wind, because you're actually there. You're in this world. You're interacting with your environment, with wind, if there's wind. So there it all comes together.
[00:20:50.847] Kent Bye: Yeah, and I'm curious if you've had any feedback on the potential triggers of motion sickness. When I happened to see this experience, I knew that some of these scenes could potentially make me motion sick, and I proactively squinted my eyes to at least reduce my field of view, especially in the sequence when I was driving, because I know for sure that when I'm driving and I'm turning left and right and there's this Yaw rotation moving the camera left or right when I'm steering that for me is for sure a big motion sickness trigger that I knew that Okay, I have to get to the scene, but I'm gonna close my eyes because if I keep them open I know by the end of it I'm gonna feel a little motion sick I think the airplane for me was better because there isn't as much infection that's going underneath and you're kind of flying in the air and and also there's a fixed plane that less of a drastic shifts in the horizon line when you're up in the air and It's a little bit more in the far field rather than in the medium field as you're kind of driving around. So, but yeah, I'd love to hear any of the feedback you may have been receiving in terms of this potential for motion sickness in this experience.
[00:21:49.285] Florian Siebert: Well, the issue of motion sickness was something we talked about from the beginning, actually. When we decided to use a motion seat, we also knew we would have to have an eye on that. But the challenge with motion sickness is that everyone experiences it in a different way. Some people do car racing games every day. They don't have motion sickness. They don't experience it because they're kind of used to it. Other people have never had VR goggles on and they have it immediately. So how do you deal with that? How do you do it? And actually I would say we're not quite at the end yet about that issue. We heard different feedbacks about it. and I think that we will address especially the car driving scene after the festival and see if we can like adjust it a little bit more so that it's safe for everyone. Because we have gotten the feedback that for some people, as you just mentioned, there's some tricky moments where you can actually are on the edge of getting motion sickness.
[00:22:47.031] Kent Bye: Yeah, the turning for me I know historically has been a big trigger of anything. Anytime that there's like a camera or shifting the horizon line and I think the big thing is like there's a disconnect between that shifting and the lack of the liquids in my ears that there's a disconnect between what I would expect to experience in my body versus what the visual cues are and it's that disconnect that really does me in.
[00:23:06.453] Florian Siebert: Yeah, that's true and there's also one factor that you have to take into account and that's the turning of the seat because Depending on what seat you're using the seat can like move to the side or to the front or to the back But some of the seats can turn So if you're in the scene driving a car and your seat doesn't actually turn when you turn in the scene, then you have the problems and the seat we have here can't turn because it isn't the exact seat we would want to use here because it didn't arrive. That's something we will have to look at, as I said, after the festival. Is the problem of motion sickness gone when we use the new seat with the turning? Or is it still there and how can we change the scene, make the road straight or whatever to avoid motion sickness?
[00:23:55.651] Kent Bye: Yeah, yeah, I mean it's it makes it a little bit more interactive to have to turn but yeah with the lack of the physical haptic feedback of that turning then yeah, it does me in motion sickness wise but yeah, there's another aspect of the piece that after I came out I asked a question of why it was sort of dubbed in English and they said that it was a request from Venice Film Festival to have an English version. But I would almost would have rather have seen the original recording in German and to get the performances of the actors as it was captured and then potentially even have subtitles. So I'd love to hear a little bit of that elaboration on that decision.
[00:24:33.727] Florian Siebert: Yeah, that's interesting of you to say that because this is the first project that we have done in this specific way of like actually doing so to say a movie in virtual reality. We didn't really know how the audience would want to experience it in terms of language. For the classical storytelling in the theaters or on streaming platforms, it's kind of learned by this day to see it in original language with subtitles. So we had a version with subtitles. And then there was, at least from our view, it's kind of weird to have the subtitles floating around in the room, in the space. Then we experimented with fixating them on one point and if you look around then you just lose them out of your field of view. But it wasn't the solution in that matter as well. So then we thought, okay, in order to make this accessible for as many people as possible, we dub this. But interestingly enough, there were more people than you to ask about why didn't we see the original version. So this would be something maybe for the distribution phase to make it optional, for example, that you can choose between the dubbed version and the version with subtitles. Maybe you have experience with the use of subtitles in an experience, virtual reality experience, that uses this kind of storytelling and fast-paced storytelling, like how you work with it, because we weren't quite sure how the audience would react to it.
[00:26:07.385] Kent Bye: I think a lot of the experiences that I've seen where it has worked well is then like having like a triangle of having at least in two or three or maybe sometimes four locations so no matter where you're looking you can feel free to look around but it's locked to the world it's not locked to your head like whenever it's locked to my head it just drives me batty because I feel like it's Too distracting. It's like having a HUD element that is too much but in terms of accessibility I think it's a broader question that the larger XR industry needs to also be thinking about in terms of like encouraging the use of Pushing forward the technologies for captions because it is an accessibility feature that as I watch more and more different shows on TV I often like to read the subtitles it gives me like a making sure that I'm not missing any bits of dialogue and I feel like I'm able to get everything and then still appreciate because I can read quickly and see things. But yeah, there's two elements in this particular piece. Why I say that is because for one, it's just very glaringly obvious that sound is not synced to the actors. So that's one thing. And the second thing is that I don't think that the English dub would have necessarily matched the emotional tonality of the original acting performance. So I would have rather seen the original performance as presented by the actors that were recorded because I don't think that the dub version was on the same level. Or maybe it was, I don't know. I just, maybe falling back to the breaking the sense of embodied presence in this environment when I can clearly see that there's a disconnect between the lip movements and the sounds that are coming out. It sort of breaks the presence for me. So yeah, because of those two reasons, I mean, the subtitles can also potentially break that presence in a different way. And maybe as we move forward, we'll have AI technology is to be able to perfectly sync that so that there's not a disconnect. But as it stands right now, that disconnect was a little bit jarring. And it was more of a curiosity of like, I wonder what the acting performances would have been like had I've seen the original performances rather than the dubbed version.
[00:28:03.403] Florian Siebert: I agree totally and I have to admit that this was actually a matter of a tight schedule before the festival. So I think we can do the dubbed version even better, because the recorded voices are actually pretty good. In most cases they are very close to the original German acting. But we didn't have the time to really match it perfectly and to really like finalize it in a way that it's smoothly integrated and it gives the same emotional experience. So I agree to you totally about that, but I still think that the dubbed version can help people get immersed who are not the fast readers or who feel maybe disturbed by the subtitles being in the frame. I think the best way is probably to have the option for users to watch it with subtitles or with the dubbed version.
[00:28:55.771] Kent Bye: Great, and as we start to wrap up, I'm curious what you think the ultimate potential of virtual reality and immersive storytelling might be and what it might be able to enable.
[00:29:06.340] Florian Siebert: Wow, the ultimate potential. That's a really big question. I do think that virtual reality is absolutely the next step of storytelling, of visual storytelling. Also in the sense of capturing the essence of our being, so to say, our history, what makes us human. Because storytelling has always been a way of survival, because we grow through stories, we survive through stories, we feel better through stories. And if you can make these stories even more immersive and more life-changing and more that they really stick with you for a longer time or through something deeper, deeper inside you, then I don't see a limit, like what can be done with virtual reality.
[00:30:01.303] Kent Bye: Great. And is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:30:07.350] Florian Siebert: Well, maybe about the big topic of AI that's like floating around. I don't want to even say pro or contrast about AI, but I think that there's always something human in all of us that we live from, that we need to live. And I think even in a world where technology can do more and more, and to bring it close to what we have done here with our project, we use so much technology, But we always said the essence of all of it has to be the human story, the human performance, and the soul of what drives us forward, what makes us feel being one human kind. So that may sound a little bit pathetic, but I do think that the more you can immerse into stories, the more you can really find what's important for us as humans and what is maybe the barrier towards technology, to AI, to robots, whatever there may come. Yeah, I do think AI will, like the technology of AI, will get even bigger and you can do more without it. You can make even complete films through AI. But I do think we as humans, we will always want to see the humanness in a project.
[00:31:30.068] Kent Bye: Yeah, yeah, I totally agree. And certainly there's some provocative experiences here this year with Topalmancer and it's going to continue to move forward. And as we're in the context of a writer's strike for the WGA in America and the Screen Actors Guild, AI is certainly a big part of those discussions and the role of these technologies when it comes to the human labor of creating these stories as we move forward. So it's certainly a hot topic that has still yet to be fully resolved. So it's certainly worth mentioning that as well. Yeah, well, Florian, thanks again for joining me today to help break down your experience of Offwind and all these different dimensions of trying to push forward this scenes that have actors that are acting together, as well as all the different cinematic reconstructions of really beautifully designed environmental spaces. And that was, for me, a lot of what I'm taking away is just the vastness of these different worlds that it feels like a bit of a time travel back to another place in time, transporting me into this historical context under which the story is taking place. Yeah, so thanks again for joining me to help break it all down.
[00:32:29.720] Florian Siebert: Thank you very much.
[00:32:31.400] Kent Bye: Thanks for listening to this interview from Venice Immersive 2023. You can go check out the Critics' Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot to digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
