I interviewed Flow director Adriaan Lokman at Venice Immersive 2023. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my series of looking at Venice Immersive 2023, this is episode number 8 out of 35 of the series, as well as episode 3 of 3 of looking at the context of communication. So today's piece is called flow and it was actually the second place prize winner at Venice immersive this year It's by Adrienne Lokman and it's a piece that's trying to push forward some of the more abstract aspects of the language of visual storytelling within the context of virtual reality and so Adrienne is looking at this deeper concept of how to visualize the invisible and he's looking at the air element and wind and then a within Cinema 4D creating these objects and so you get to see the outline of these painterly particle effects as you see these lines of air that are moving through the world. And so he's essentially taking away everything that is visible and making it invisible and everything that is invisible and making it visible. That's the central conceit of this piece. And he's taking you from these associative links and taking on this spatial journey of error throughout the context of all these different situations. And so it's the communication context where he's trying to push forward the different grammar and language of visual storytelling within the context of virtual reality and it's got a center of gravity of presence of mental presence because you're really trying to piece together and understand what is missing because you're trying to imagine and fill in the blanks but also trying to follow the different associative links and you're kind of puzzling together what's happening from moment to moment in this more associative fashion. And it's also exploring these aspects of environmental design and presence of looking at how can you start to visually tell some of these stories at the bare minimum of using these different spatial contexts to communicate a deeper relational dynamic and deeper story that might be unfolding here. So this interview with Adriaan happened on Sunday, September 3rd, 2023 at the Venice Immersive in Venice, Italy. So with that, let's go ahead and dive right in.
[00:02:19.552] Adriaan Lokman: I'm Adriaan Lokman. I am director of animation films, short animation films, independent films. And a couple of years ago I dived into VR. So this is actually my first VR festival in Venice, which is like the temple of VR. So it's very nice to be here.
[00:02:37.732] Kent Bye: Maybe you could give a bit more context as to your background and your journey into VR.
[00:02:42.795] Adriaan Lokman: My background is really like I said cinema. I studied arts in Rotterdam and for a long time I had my own production studio. I did a lot of animations and I've always been interested in VR but I thought the technology was kind of Too complicated for me and when I started my last project flow I already thought that it would be interesting in VR But then that was in 2013 and still the technology felt complicated still so I thought like well I'm gonna first gonna make the film but what I immediately did was make it in stereoscopy so in 3d because I thought it needed to depth to really make it work and And then when I continued with the film and VR became more accessible and less complicated and computers more powerful because that's needed as well. I started experimenting with it and we looked for financing and so we started the project.
[00:03:34.176] Kent Bye: Maybe you could give a bit more context as to getting introduced to virtual reality. What were some of the first experiences and what catalyzed you to make the leap into making some VR?
[00:03:44.101] Adriaan Lokman: It's funny that I didn't really see a lot of VR before I started. I came across some at festivals. For instance, I suppose everybody knows Gloomy Eyes. And there was another one. What's it called? What was it about? It was about somebody that lost her husband in a flight accident on a plane. And it was all scanned. So it was like you had the feeling that there were real people walking around. It was a 6DoF installation. And I was amazed by the fact that the intimacy VR adds to a moving story. You don't have, of course in cinema you can cry if there's a really, a very sad story, but in VR it's so much stronger. You really have the feeling that you can have someone in front of you. And it even works with animation. Even if you know that what you're looking at is an artificial image, it's like curated by someone, it's not real, then still it provokes emotions much more easy than a projection on a flat screen does because your brain really thinks that you're there in front of someone. For me that was an interesting experience which inspired me to continue in 2VR and to use it as well in Flow where you have the main thing in the film is the wind but then it turns around a woman and I tried to develop a more intimate relation with the woman than I would have been able to in the film and just using close-ups which I think it works.
[00:05:03.059] Kent Bye: Yeah, and maybe you could give a bit more context as to your previous animation pieces that you've done and just a little bit more so I understand the evolution up to Flow.
[00:05:11.345] Adriaan Lokman: Okay. My first film was about light. And to my surprise, it's very simple, it's kind of a minimalist film with just cylinders and light sources moving and camera moving and that's all. And it became a big success which kind of surprised me a lot. It won the Grand Prix at Annecy. and that suddenly opened doors to me and made me decide to stop with my commercial work and go further into that. What I try actually is in each film, or each project, because it's not only films that I make, is to find new languages to communicate, new visual languages. So I experiment a lot, which means that it's not always easy to understand, but I found that it's like stimulating if your brain has to do some work as well, so that you have to fill in the gaps so that you can kind of construct your own film in your own mind and fill in the gaps that I leave to the audience. And of course everybody can do that in a different way. So I think if you look at the spectrum of my films then you see every film is another experiment and goes in kind of another direction. Although people of course say that there's always kind of my handwriting in there, but they still look quite different.
[00:06:22.833] Kent Bye: And is your previous work in the same aesthetic or style? Because you have these lines and they're moving and it feels very much like liquid or fluid dynamics that you're visualizing. And so in your previous work, have you cultivated that style in 2D?
[00:06:34.995] Adriaan Lokman: No. Well, my third film, maybe, because it's a film about clouds. But then the software and the technology was still so difficult because that was in like 2007. And I'm still not totally satisfied with the result. But apart from that, the other films are very graphical, really. The first one is totally black and white. and they don't look in anything. I like flow, so flow was for me totally new just to go into a much more organic style. It was a new experiment and I'm sure my next project will be totally different again because I have to stimulate myself as well to find new ways. I don't feel comfortable in repeating myself and doing things I already did before. It bores me to repeat myself in a way.
[00:07:15.463] Kent Bye: Yeah, so I really appreciated that aspect of flow where you're really cultivating this new spatial visual grammar of dynamics and movements and trying to tell a story not only through like this outlining that's like I'd like to think of it as like fluid dynamics but with air so you have this air dynamics that are painting the outlines of things so you get a sense of the outlines of it, but also through the sound, the sound design was very rich with the spatial sound. And so, yeah, maybe you could give a bit of context for how you think about what you were trying to explore with Flow and what the story you were trying to tell there.
[00:07:49.972] Adriaan Lokman: I had the idea of making a film about the wind or air for a long time, but I didn't really know how, so it stayed in a drawer for quite a while. And then a day I came across an illustration, I don't even remember if it was a billboard or a magazine, the aerodynamics of a car, which is one way to sell a car. So you see the car and then around it you see these lines that illustrate the wind or the air that goes around it. And I thought like well if you would take away the car from the image then you would still recognize that there was a car there before because it's tracing the car. And I thought okay that could be an interesting visual concept to actually show air and show the wind. So that was the start like taking away everything that's visible in normal life and making visible everything that's invisible in normal life, which is the air around us that we breathe, that we cycle against, that we use for energy, etc. So it's one of the most important elements in our lives, but we don't see it. So I thought it was interesting to turn that around. That's the start of a concept, but it's easy to say, but then you have to develop it, and that was like something else. So what I did was I started to experiment with solid objects in 3D, which is a simple 3D animation, like Cinema 4D I use for that. and build scenes with that, with cars, or balls, or walking people, and then use those objects to bounce particle streams against. So, there's these particles of fountains that, like, spray little dots, and they bounce against the object, or they trace the object, they get stuck to it, and then the particles, well, you add some lines, so the particles, the dots become lines. and then you think like well then the object will become visible but it was not as easy like that because of course you get all clutters of lines and particles and then it becomes kind of a ball of wool in which you can't recognize the object anymore so then the complex thing starts that you have to To start playing around with all these parameters of gravity of wind force turbulence speed line thickness, etc And also kill lines so that some parts of the object become more visible than others Which means like you have a block and wherever line that gets into the block is immediately deleted so that that's all things you can play around with so there's a lot of things going on until you finally end up with the image in which you can say, yes, this really feels like you're painting with the wind and with air. And that took me about almost three years, I guess, to get to the final look with which I felt comfortable and which I knew that I was enough experienced to start playing around with it. And then the other big challenge became, it's like, okay, but what I'm going to do with this knowledge and with this experience? Because having this technique and having reached this visual goal doesn't mean that you have an experience yet or a film. It needs a storyline. And that's always something in my work because I'm kind of experimental. I was always in the beginning considered as an abstract artist. But I think it's very important to give the audience a hand. You have to guide people from A to B. Otherwise they lose interest, they lose concentration, and it would become moving wallpaper. Maybe really nice wallpaper, but that's not a movie. That's more an installation, something you hang on the wall, but not really you follow from start to finish. So it needed a story. So that's where I started writing and thinking about, OK, what could you do with it? And I thought I'd try to... develop something that makes sense with the wind. The wind tells a story that shows the turbulence that we have in our own lives. We can flow in an easy way on a sunny day and walk on the street but life can have its strange turns and turnarounds and can put us totally in a different direction without us wanting it and getting us into a literally a storm. So that is something that I used and I created a little story with that.
[00:11:32.237] Kent Bye: So yeah, just to clarify, so you're using Cinema 4D to put objects in and use all the parameters to get the look. And then from there, are you then using that as pre-visualization just to get a sense of it? Or is that something that you're also eventually exporting into 360 video that you're able to create everything in the context of Cinema 4D? Or that you were just using that for previs and doing more of a hand-drawn approach for getting things in the look and the feel that is in the final piece?
[00:12:01.308] Adriaan Lokman: No, there's nothing hand-drawn in it. Everything is digital. What was very important was the plug-in is X-Particles with Cycles as the render engine. So what you see is really all computer-generated. And that's also why it took so long to have this feeling, because it's funny that you ask, was it hand-drawn? And it looks like hand-drawn very often. And of course that was a goal as well. to have that feel that is like, which makes it more organic. Although it's not like throughout the whole film, it's not always that organic, like in the storm it becomes very graphical, which is actually, that was not on purpose. In the storm you see very thick graphical lines going around and those were low resolution renderings that I thought I was just using for testing the movement. And then I played it with some sound and music and I actually liked it a lot. And I thought I keep it like that because it gives a big contrast with all the little subtle hairy lines from the beginning. So that was kind of a surprise. But you need this type of surprises available of course. And that's what I like about experimenting. When you experiment then you come across things that you didn't think of and that you can eventually use. also the other way around. You think like, okay, now I know how I'm going to do it, and I do it like this, and then you make it, and then it doesn't work at all. Of course, you come across that as well. You have good surprises and bad surprises. But no, so like Cinema 4D is like everything, all the solid objects are there, but they're not rendered. So you just switch them off. So only the particles and the lines remain, and so that's what you see. And then that provokes this enormous Transparency and that surprised me as well works so well in VR. I never thought that it would work that well So like I really think that the VR version of flow is You can't say better, but it's different, but it works much better The concept works much better in VR than the film who I thought was also quite minor. It is not bad at all I like it a lot as well as a film and but in VR you're really in there and because of the transparency, the depth works so well. You can look very far, you can look through the lines and you see objects behind, so it exposes a whole world around you.
[00:14:03.130] Kent Bye: And it's a 360 video, right?
[00:14:03.871] Adriaan Lokman: Yeah, it's a 360 video. We tried 6DOF at the start, we did tests, because I thought it was very interesting and very nice. It's a wonderful technique if you are able to get your visual concept across, and that was exactly where it went wrong, because I'm talking about three years ago already, so maybe now, maybe in five years it would be, or maybe even in two years it would be better and it would be possible. But we couldn't get the same kind of feeling of painting with the wind in 6DOF just with a couple of objects, one or two or three. But then there was just too much information and detail which the computer couldn't handle. So it became too slow or you could just have like not enough lines, too few lines to really trace an object and to make it work. It would have meant a totally different kind of style which I thought didn't fit with the concept.
[00:14:52.660] Kent Bye: So the limits of processing means that it wouldn't be as robust, that you can get a lot higher resolution with the 360 video, kind of get the full scene that you can render out. How long did it take to render out the full piece?
[00:15:05.828] Adriaan Lokman: Good question. Actually I don't know because I didn't count it, but it was long. because, of course, you have to render it at a much higher resolution than I am used to for cinema, which already is quite high, because I'm finishing in 4K stereoscopic, so that's also quite heavy. I think there were scenes where one image took nine minutes.
[00:15:24.939] Kent Bye: That's 90 images per second? No, 25. Oh, okay, so it's a 25.
[00:15:29.401] Adriaan Lokman: Yeah, that would surprise me as well, because I see that a lot of experiences have a higher frame rate, And I thought, oh my god, I have to erase that as well to get a good result, which meant really re-rendering also a lot, re-caching also a lot of particles I had left over from the film, so it meant double work. Then I experimented with it and 25 worked quite nicely. So far I haven't heard anybody saying like, hmm, this frame rate is off, it doesn't work.
[00:15:54.356] Kent Bye: Yeah, I was talking to Aardman and they were saying that Wallace & Gromit is animated at like 12 frames per second or something so because it's a slower pace of movement for every movement it's like every 12 frames or so. So you're using Cinema 4D and then you mentioned that there's some plugins that you're using or like what are the plugins that you're using in Cinema 4D?
[00:16:13.364] Adriaan Lokman: The most important, very important, is X-Particles from Insidium. Without that, it wouldn't have been possible. And what's funny is, when I started, they were really starting as well, so it was really the earliest version. And while the project was evolving, the plugin evolved as well. So there were updates and updates and updates, and it became... more capable and more interesting, but also more complex, I must say, and it's also, well, more bugs as well, but I guess that goes with the development as well. So that one was very important. And then Cycles for the Cycles 4D for the rendering, but that could have been another one as well, but X-Particles I think is difficult to replace.
[00:16:47.883] Kent Bye: So the Cycles plugin is what allows you to export in like Equi, Rectangular, 360 video?
[00:16:52.814] Adriaan Lokman: What, stereoscopic? Yes, exactly, that's what it does, it's the render engine. And of course, because it's a render engine that works on GPU graphic cards, you render a lot faster, so I could add graphic cards to my computer, because with the film I didn't do it like that, so I didn't use a GPU render engine, and here I did, which helped a lot in shortening the render times.
[00:17:15.157] Kent Bye: And so are there any camera movements in this piece or is the camera mostly static because you're the head and everything's just moving around you?
[00:17:23.182] Adriaan Lokman: No, there's a lot of camera movements and that was also kind of the big challenge because in the film it floats. It's funny because I didn't look at the film for a long time and it's just a couple of days ago I looked at it I thought wow it really moves enormously and at the start that was a challenge because these camera movements that are in the film they are there on purpose to give you the feeling that you're floating with the wind. almost like you're in a boat, you go up and down. So I wanted to have this feeling in VR as well, but of course I did watch other VR experiences as well, and I get motion sick quite quickly. So I'm quite sensible to that myself, so I immediately started to test that. And it was funny, I tested it first with the solid objects, without the lines, and then it was not working. I really got this motion sickness very quickly. And I thought, okay, I'm going to try it with the particles and the lines anyway, see what works. And then it turned out that the same camera movement with these lines really didn't provoke as much motion sickness as it did with the solid objects. I thought it was very interesting, although I still had to remove a lot to calm down the camera compared to the movie, that's for sure. But throughout the film it's almost all the time in movement. But that really was something I had to test. I had scenes that were too violent that I had to skip. But there were actually scenes that I thought were not fit for the VR that were in the movie and I wrote them out of the scenario and then while making the experience I found out that the motion sickness was even less of a problem than I thought. And I thought, I'm going to try those scenes anyway, which was the one where you go through the windmills. And it wasn't a problem. It didn't provoke motion sickness, so it was fine. But there were scenes as well where it did provoke motion sickness, and then I tried something out. It was like when you are in the metro, in the corridor, and then you're going forward. And, of course, going forward in one direction often provokes discomfort. And it's kind of a traveling shot and there's lines going with you. And I thought like, well, if I have some lines that go in different direction that are coming towards you, at the same time that there are lines that are traveling with you. And it's funny that they compensate the motion, they compensate the movement. So you lose the motion sickness because there's another movement going in the other direction. But I guess it only works because it's transparent, so you can see everything.
[00:19:31.987] Kent Bye: Yeah, there's different motion sickness triggers for different people. Different people have different triggers, but one of the things that gets a lot of people is vection. If you're walking, you see ground moving in your periphery, that can be a trigger. So if you're in space, then that can often be a little bit more comfortable because you don't have that same type of vection that happens. But yeah, I mean, over time, my tolerance to motion sickness has increased greatly. So I'm not always the best judge to whether or not something's comfortable or not. Although there are stumps that does make me sick. But like I said, people have different motion sickness triggers. So it's hard to know. But one other follow-up question, technically, that I want to ask is, Would you be able to render it out at a lower resolution? I'm just trying to get a sense of, like, you seeing the shot and making sure that it works, maybe at a lower resolution, maybe it's faster, or would you have to, like, render out everything at full resolution and see it and just to be able to know whether it's working or not?
[00:20:22.854] Adriaan Lokman: Yeah, yes, you do. I did try a lower resolution rendering, but then again, if you do that, okay, you win some time, but that means that if you like it, you still have to redo it, so then you still end up with a higher render time than you would have if you would have started immediately with a higher resolution. Then again, I think that's also throughout the project, because I've been working with the project and of course with the film as well. The first test I started in 2013, which means that I built quite an experience myself to know eventually what is working, what is not working. So for the VR I didn't have to do a lot of tests. Once I was comfortable with how I could animate the camera, It was more a challenge to tell the story in a 360 environment. And how to let people focus on the right action and not be diverted by something that's not important in the story. And how to fill up space that was not there before, that was empty before. And also play with the dimensions in space and the scale of things. That was also important. But after a while you get the feel of it, you get the hang of it. So you don't have to test that much. But on rendering, rendering is really like when I started with the first scenes, I guess like two or three years ago. I immediately started rendering, so since then, I think if you ask the question, how much time did it take? Well, I think a year and a half, maybe, of rendering total, because it was rendering all the time. But not every frame is nine minutes, because that's kind of the longest one. You also have scenes that go in 20, 30 seconds.
[00:21:48.073] Kent Bye: Yeah, but it's still a painstakingly long process to set it all up and render it out. On average, how many iterations did you have to get per scene before you got it right? Was it a couple, or 10, or 20?
[00:22:00.275] Adriaan Lokman: That depends a lot, because there are scenes that were quite easy, but there are also scenes that are very complex, and then I did it in layers. So I had rendered first the background, on which I could test, like, is the camera movement okay? Because often you can see just at the background, like, okay, well, this camera movement is not going to give any problems, or it's nice. and then add the main characters or the main action to it and then render that one and that one and then later on composite it in After Effects. So that works quite well. Many scenes I had like the most important elements ready and then later on I added like subtle strokes of air or more like for decoration or adding more ambience and environment.
[00:22:35.488] Kent Bye: and you're spot-checking it in VR all throughout the whole time?
[00:22:40.370] Adriaan Lokman: Yes, but also there you get an experience, so it's not necessary all the time, because that is also something like, it means you have to finish the rendering, then you have to transfer it. It takes time to put it in your cask and to be able to watch it. So it depended on the scene. Of course, quite often I did that, but not always, because if you know that the basics are right, then you can kind of read it from a flat screen as well, in a way.
[00:23:04.348] Kent Bye: Yeah, that makes sense. And, uh, yeah, I thought it was just a really quite magical experience because visualizing the invisible, I think of it as like the platonic realm of ideal forms. You see like the math structures from Aristotle, he's got this idea of formal causation. So like the math structures that are subtly informing and shaping the world. And I feel like a piece like this is really visualizing that air element of the wind and just the idea that VR could also start to visualize the invisible. So yeah, I'd love to hear any elaboration on that.
[00:23:35.015] Adriaan Lokman: Yeah, yeah, that's true. Well, that's why I found like being on this festival really really very inspiring because I see a lot of things I think okay, you can do it like that as well. You can play with this you can play with that. It's really VR is really like it still is in its infancy. That's for sure in technical ways, but not only also in In creation, we just start to scratch on the surface, I guess. And by scratching on the surface, the technology is also evolving. It's also getting better and giving us more intuitive tools to be able to use our creativity and to develop that creativity. So I think it's a fascinating medium. in a lot of range, not only in art, but also in education and even in medical. There's so many things where it's going to be a very important part of our lives. But there's, of course, still at the moment, that's one thing that's the big disadvantage about VR for now is like this. You need these glasses. You need to put something on your head. And that still is a disadvantage. But I guess like in the future, we might come up with some solutions there as well. I'm curious.
[00:24:35.031] Kent Bye: Yeah, my experience of watching Flow was that it felt like a deeply awe-inspiring piece because it was such a unique aesthetic that I've never seen before. It's part of the reason why I wanted to unpack all the technical bits of how you did it because it was so magical in that way. but also it felt like a piece where you were really crafting this visual storytelling language where you're able to create these relational dynamics in both how things were connected to each other visually, but also moving from moment to moment, but also the sound design is also there augmenting everything and adding additional context. I guess the challenge of this more abstract visual storytelling language is that there's a lot to be left open to interpretation in terms of what is being said and communicated and that there can be some gaps in terms of what you were intending to communicate versus what people take away. And for me, I guess I'm noticing different objects and different things in the moment, but I'm not sure if you asked me to explain what the story was about. I'd be able to comprehensively recount each of those moments. It felt like moving from moment to moment, but my memory of seeing all the different projects here at Venice and like trying to remember the specifics, I guess, that happens to me on a lot of different projects. But in this project in particular, it felt a lot more abstract in that way. And like there was this cultivation of the visual storytelling language, but the specifics of that story being told are more ambiguous. And that's fine for me because I feel like I was walking away with a sense of awe and a sense of emotion that was just really powerful, but I wouldn't, it's sort of like this pre-conscious or unconscious, subconscious feelings that were being evoked without being able to communicate it. And neuroscience sometimes is referred to as like sub-symbolic. So sub-symbolic associations without having the ability to articulate in words exactly what it's about. So I don't know if that was the intent or if there is also a degree of which that you wanted to communicate a larger story with this abstract visual storytelling language.
[00:26:26.133] Adriaan Lokman: Well, I think that's very interesting what you're saying, because that is actually what I'm doing. It's like normally in, well very often in VR, but in cinema, in storytelling, is you start with the story and then you're going to think about how you're going to tell the story and how you're going to visualize it. And I turn that around. I think about first what element do I want to show. and how do I want to visualise it and then I need the story to visualise the element. So in that sense it's right that the story is important but at the end of the ride if you don't recall everything or if you don't understand exactly what was going on or what was happening or what the story was about it doesn't really matter as long as you felt that element that I wanted to play around in your head did what it had to do. And so the story serves the concept and not the other way around. So there was a very good analyzation that you made there. So that's the way I've been working with all my films, actually.
[00:27:18.584] Kent Bye: So what's the core element then of this piece?
[00:27:22.245] Adriaan Lokman: Actually, it's what you said already. It's like everything that we don't see but is around us, is surrounding us, and which is much bigger than everything we do see.
[00:27:32.827] Kent Bye: Yeah, maybe you could talk a bit about the sound design in this piece, because I felt like that was another key part of helping to add context to things, because you can see the outline, but then, you know, like hearing a hairdryer as an example, it really helps to visualize what's going on, whereas I don't know if I would have been able to identify, oh yeah, that's definitely a hairdryer, but having the sounds helps to give a broader spatial context to what's happening. So yeah, I'd love to hear your process of adding the layers of spatial sound to this piece.
[00:27:58.556] Adriaan Lokman: Yeah, it's true. The sound is very, very important. Actually, I immediately start with sound. Because the images are kind of abstract and experimental, you indeed often don't understand immediately what you're looking at, depending on the scene, but it often happens. So sound immediately explains it. And in fact, myself, I have to go through that process as well. So what I do, From the start I look for sounds on YouTube, on the internet, so I copy-paste them into the tests to immediately be able to see whether what I'm making is comprehensible enough to continue and what type of sound is needed to explain what you're looking at. And there I want to kind of work on the limit of what is understandable and what is not understandable. I don't want to explain everything too explicitly as well. So it's like kind of looking for things that are just enough to be able to understand what's going on. Especially image-wise, that is. Because the sound, of course, like you say, like a sample of the hairdryer, you hear immediately that it's a hairdryer. and if there's a car passing by you hear that it is a car and in the image I just have to add just enough to give you the feeling that there is a car so you understand there is a car. So yes, that's from the start I do that and then at the end when the whole experience is more or less finished I work with sound designers that kind of redo my work but then professionally so with real sounds they often they have libraries of sounds or they Even better, like in the case of Flo, I had Samy Bardet, which was a Paris sound designer. And he made very, very beautiful wind and air sounds that we couldn't find in any library. So he was really very good at that. Because before that, like, I looked for them myself in libraries on the internet, never came across, well, some of them, yes, that worked. In general, having the subtle flow of air around you is hard to capture. And he had, I don't know what it's called, he had kind of a software with a pen on a tablet he could like move around in circles and that kind of provoked sounds that went extremely well with the experience. So, thanks a lot to him for that element because that really is important. So yes, that's how it works. And with the composer as well, the music also, like at the start, I immediately want to know what music can do and what it can add to the emotion of the scenes. I send him early tests. He sent me first tryouts of instrumentation, music composition, and I paste them under the test and see what it does. And then I send it back. And so it's kind of a game of tennis a little bit from the start, actually. So it's not that like most films or most experiences, like you finish and at the end they add the music. And that's not how I work.
[00:30:28.329] Kent Bye: So did he mix down like a stereo mix or is this like a spatialized ambisonic mix or how did he create the sound design?
[00:30:36.181] Adriaan Lokman: It's ambisonic, it's like the music is ad-locked stereo. Apart from the musicians in the Metro, of course, because that's like real-life music, so there it's like ambisonic as well. This was my first experience with ambisonic. I played around with it myself in a kind of amateuristic way because I was curious, because I thought it was magical that you have like, you put your little headphones in and you listen to something and you have the feeling that there's something behind or above or wherever it is. It's a wonderful technology. So I played around with it a little bit, but I'm not a professional sound mixer or sound designer, so it's much better if I let that eventually to professionals. And that was Kom Jaliber who did the ambisonic sound mix. He already had a lot of experience there. That was actually the last stage of the production. And that was also a wonderful experience to see how he did that and how he played with a presentation of the experience on a big cinema screen. And then you can just paste the sounds to the objects and then follow the objects that later on go behind you or go in front of you. It's a magical way of working.
[00:31:34.540] Kent Bye: Yeah, I'd love if you could expand on some of your producers of this piece, because I know Claire from Lucid Realities is someone who's been in the location-based entertainment space and VR for a long time. And yeah, just a little bit more context as the folks that were helping produce this piece.
[00:31:47.698] Adriaan Lokman: Yeah, Lucid actually, it was funny, we were looking for financing in France as well, because I live in France, so it's like, it's kind of logical to work in an international co-production. And I worked with Valk Productions, my film producer, for a long time already. That started in like 2007 or 2008 I guess. And for this production we were looking for a French producer and it was really by accident. Yeah, yeah, now I remember. There was a Dutch curator and she was interested in my video installations. I make video installations as well. And then she came and watched my installations and then we started to talk about it and said like, well, you know, I'm looking for a producer. And then she just came in contact with, by accident, with Lucid Realities. And she thought, well, maybe you should talk to them. So I gave them a call and she connected us. I talked about the project and presented it to them and they were immediately interested to have a go and find some financing and support in the project. So that was really my first contact with them and I now much more understand how broad their experience is in the field and how many contacts they have in distribution as well, which is very important of course because you want whatever you make, you want it to show to the world. and they're quite professional in that so now I start to understand how professional they actually are and very much looking forward to work on a new project I'm working on with them as well so that's for the well it's not even for the future already started so
[00:33:13.328] Kent Bye: So what's the plans for distribution? I imagine you might have a festival run with Flow, but also, do you have any, like, museums or location-based contexts that you would hope to show a piece like this? Or any specific contexts you think this would work particularly well for audiences of the general public or a specific location or context?
[00:33:32.434] Adriaan Lokman: That's an interesting question because that is something I found that VR is still, in general, trying to find the right answer to that. Where does it belong? Where should you watch it and where should it be presented? If you think about it, VR is really like a personal experience that you could very well, but not for the installations, they can be too complex, but a simple 6DoF or 360 experience you could do on your own couch, in your own living room of course, which sounds kind of logical because it is an individual experience after all. But at the same time installations can really be a piece of art so then it would belong in a museum and that could be a good spot. At expositions, festivals as well. But I see festivals also struggle with how to present it and how to get an audience. Here it's very well done but I've seen festivals that you think that it's just like somewhere behind the cinemas in a spot where they made like not even cabins where you got some aluminium frames and it doesn't look like anything. Maybe because they think like if you put on your glasses then you won't see what's happening beside anymore But here it's all like very very nicely styled and presented which puts the VR as a whole kind of a pedestal But going back to your question, I don't know exactly but I know the distributor they do have contact with museum that they talked about here I know I have my next exposition with my video installations in October in Holland, so I would like to present at least Flow VR there as well, but there's not going to be many glasses, maybe only one set. And then there's an initiative in Amsterdam where they might want to start a VR center, but there's going to be 20 there, so that's interesting. So it's popping up, it's getting more and more, it's spreading, you can feel that.
[00:35:10.086] Kent Bye: Your piece flow is very poetic and lyrical and there's an ineffable quality to it where it can be sometimes difficult to translate into words what you just experienced in this very abstract type of experience. As you've been here at Venice Immersive 2023 to show it to folks here, what have been some of the reactions that you've had from people who had a chance to see it?
[00:35:31.597] Adriaan Lokman: Well, they're quite general. It seems that everybody likes it, which is lovely. I never had a reaction so far that people said, well, it's not for me. So apparently, it seems to be a crowd pleaser in a certain way. That's very nice. Then again, I don't get to talk to many people. And that's one of the disadvantages about VR, because you don't meet everybody. Unless I would be there all the time to wait until they finished and then ask, or they would know that I am the director. But if you walk around here, they don't know that it's me who made it. They're not addressing me immediately to say like, ah, yes, you're the director of Flow, because how could they know? And in a film festival, you're presented on a podium in a cinema with 1,000 people, and immediately 1,000 people know who you are. So then that is much easier to react afterwards. And you get immediately personal reactions. So in that sense, you don't get that many. But the ones I got, they're all positive, yes. And of course, there's professional reactions you get from distributors or festivals or curators that are positive as well.
[00:36:31.063] Kent Bye: Awesome. And finally, what do you think the ultimate potential of virtual reality and this type of immersive art and immersive storytelling might be? And what am I able to enable?
[00:36:43.227] Adriaan Lokman: Yeah, well, I think for an artist or a filmmaker or whatever you want to call it, it could actually be anything. It could be a painter or a sculptor or, I mean, VR, you could use it in any way. The most interesting aspect about VR, and that is fascinating, is that you can really put someone totally in the middle of your world, of what you want to show, you want to transfer, because there's no distraction. You can literally, and that's why, of course, it's like kind of... immerse people into your creation, and that is fascinating. At the same time, it's also complicated, because you can do that in so many ways, so you have to think about the best way. And the only way you can do that is to watch a lot yourself and to experience, and then you let your brain work. In my case, that's already happening.
[00:37:28.804] Kent Bye: Great, and is there anything else that's left unsaid that you'd like to say to the broader immersive community? Hmm, I don't know.
[00:37:37.677] Adriaan Lokman: I'm really looking forward to the future, to see how it will evolve and where we will be in five years, because it's going so fast. And also, like artificial intelligence, we didn't talk about that. That's very fascinating as well, to see what that's going to bring us. Of course, I can see the dangers as well, there certainly are. But at the same time, it might make art more democratic, maybe, because even without skills, you could tap from a source of creations and combine them into something new that maybe without that you could never do because you lack the skills. But having said that, like I said before, there's also danger because then you could, of course, use the work of others that have been created before. So where are the copyrights and who owns it, etc. So that is a very interesting discussion. So I'm very curious to see what's that going to do and what the role of artificial intelligence will be in VR. Actually, I didn't have the chance to see the experiences here that are using... Yeah, Topelmancer and Shadowtime have a lot of AI stuff in it. Yeah, well, Shadowtime I saw. I love it. Shadowtime, I thought it was great. Really wonderful experience. Yeah, one of my favorites.
[00:38:49.193] Kent Bye: They have some generative AI stuff, more style transfer on top of existing scenes, and then you have Topalmancer much more into the large language model and generative AI type of stuff, exploring all these things. So, yeah, it's certainly a topic I've been covering, and it's certainly a lot of open questions as to where it's going to end up. I think artists are pushing forward what's even possible. Now, once they've created that, now we can reflect upon, OK, what are the ethical implications of it all? So I think that's kind of what the process is happening right now.
[00:39:16.076] Adriaan Lokman: We'll see where it ends up. But anyway, I'm not afraid of it. I'm very curious. I haven't tried yet what I would do if I would make something in the style of Adriaan Lokman, if something would pop up. Because if it doesn't, it's kind of disappointing, isn't it? Because it means that it can't find you. And if it does, it's kind of frightening.
[00:39:34.228] Kent Bye: Awesome. Well, I really, really enjoyed Flow. Like you said, it's a crowd pleaser and it's one of those pieces that is abstract enough where you don't actually need to understand any specific language. It's a visual language and a grammar that you're starting to develop and cultivate. And yeah, going from one scene to the next and exploring the affordances of the spatial medium of storytelling within VR and yeah, quite a provocative piece. And yeah, just loved seeing the invisible become visible and I look forward to see where that type of technique goes in the future. Thanks again for taking the time to help break it all down.
[00:40:05.974] Adriaan Lokman: Thanks to you as well, because it's always nice to talk about, to get these kind of questions, because it forces me to think about why I'm doing things, and it also inspires me to go further in research and continue in creating. Awesome. Perfect.
[00:40:22.475] Kent Bye: Thanks for listening to this interview from Fitness Immersive 2023. You can go check out the Critics Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot of digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
