I interviewed co-directors Nienke Huitenga and Lieven Heeremans about Drift that showed at IDFA DocLab 2024. See the transcript down below for more context on our conversation.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continuing my series of looking at different experiences from IFA Doc Lab 2024, today's episode is with a piece called Drift, which is part of the digital storytelling competition there. And it's an open world podcast that is using all these generative AI techniques. So the story is set 500 years in the future in the Netherlands and Amsterdam. And so the open world aspect of this podcast is that they generate probably five or six different podcasts per day using these different generative techniques. They're very short, like one to two minutes long, but they're just kind of giving you this vignette of these different characters that they've constructed. So they're trying to project out 500 years in the future to show how the impacts of climate change had taken place. And they're looking back into this present time and And they're also creating their own custom AI voices as well. So everything is actually a generative in this podcast. So it's all generated by AI, but they're able to prompt it. So they're creating these different characters. They're using these generative AI techniques to prompt all these different stories, but also train it with all these data sets from climate change and all these predictions and extrapolations. And so they're setting in the future and the speculative fiction in order to look back into the present, reflecting upon these climate change issues. So you can actually take a listen to it. It's at Merrell, M-E-R-E-L dot club slash drift. And you can subscribe to the podcast feeds there. One thing to note is that they actually wipe everything clean every month because it's based upon the lunar cycle. So it starts on the new moon and then they start afresh and then basically have a new world that's completely constructed. So you can go check it out at Merrell dot club slash drift. And this is about when my voice started to give out throughout the course of the week. So you can hear my voice progressively getting more and more scratchy from this point on. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Nika and Levan happened on Tuesday, November 19th, 2024 at InfoDocLab in Amsterdam, Netherlands. So with that, let's go ahead and...
[00:02:23.308] Nienke Huitenga: dive right in hey i'm ninka immersive director and producer i have a beautiful audio piece but i also do vr directing and let's say everything interactive and play i'm very fascinated by the interplay of let's say the meaning making through narrative but also using your body and really connecting so for me creative technology is like a tool that supports a way profounder process that is kind of connecting and finding yourself in an immersive dynamic
[00:02:53.895] Kent Bye: Maybe you could give a bit more context as to your background and your journey into this space.
[00:02:59.618] Nienke Huitenga: So my background, let me think about that question. Sometimes I find it hard to explain because I'm such an autodidact. I think maybe as all of us, we kind of grow up with this new media, new technology. I was originally trained as a film schooler, but I got really fascinated by, I think, the early 2010s when interactive documentary web-based interactive experiences came around. With all of my curiosity and also I think through IDVA DocLab I actually got a second education. So I kind of dove into this world where all kinds of makers just like me were wondering like, so what happens at the fringes if we not make it linear? So I started making transmedia experiences, mostly online, and quite quickly, I think after 2015, I started experimenting with my first VR piece, Ross Sibne, which premiered here at DocLab, which was a very short, beautiful non-fiction experience about the loss of the MH17 airplane above eastern Ukraine. where we merged two perspectives. The loss of many Dutch people who were in that plane, but also the start of the war in Ukraine. And it kind of opened my eyes. And I built up a studio, Studio Zap, where I collaborate with all kinds of other makers, other disciplines. Yeah, I think that's a bit.
[00:04:19.127] Kent Bye: What year did you first start going to DocLab? And maybe you could elaborate on how it was like your second education.
[00:04:25.839] Nienke Huitenga: Well, I think it was around 2013, I think. I kind of finished my masters in 2010 and my partner was a catalogue writer here. He was doing the daily journal they made at IDFA. So through him, I kind of roamed, and I noticed this event. They were then on the side, kind of a side program. I think the group of people I met there and the type of artists on stage, it was so much joy and curiosity and adventurous thinking about how would a story work if we do it together, have an interactive experience. That just showed me a whole new world to how storytelling can be explored. And I bravely applied to the IDVA DocLab Academy, I think for 2014. I'm not sure about the year, but I think it was one of the first or second academies that DocLab organized. And I actually also met one of my collaborators, Haai Kranen, who's also in the Drift production. And although that was not a full education, it was like 10 days of fun during the festival, but that just really proved to me that I'm not alone in this fascination. Yeah, and I just took it from there. I think it's actively going to all DocLab conferences every year, meeting this group of people. We're still seeing each other here occasionally. I think it's quite remarkable because the way also projects arrive at DocLab, the mix of curation here, stems from not a medium, but everyone has a question about reality. So it can be slightly fictional, it can be very documentary, but I think we all wonder something about our innate human nature or how we relate to the world. And I think that flexibility in the question you can ask and the format you can find for it, it still feels every time I come here, I say my thinking is expanded a little bit further. I just really like that feeling that I'm still learning.
[00:06:23.286] Kent Bye: Yeah, I've had a similar experience here at DocLab. And I first came here in 2018. I actually was coming back home from Germany via Now. They had me give a keynote talk there. And I happened to have a 23-hour layover in Amsterdam. And DocLab happened to be happening. I don't know how it got onto my radar, but I essentially came and Michelle had hooked me up to see the collider, the horse, basically all these amazing experiences. I did interviews and then I had an idea of what documentary was. It's like, okay, you have film and you're documenting physical reality. And then when you have VR and immersive media, then you're creating this CGI type of stuff. And so I remember interviewing Casper being like, how do you define documentary? And he's He's like, well, John Gerson says it's the creative treatment of actuality. And I've come back to that again and again, because I feel like the creative treatment of actuality, I mean, that could be as far as fiction, but it kind of gives them leeway of this blending and blurring of the modes of fiction and the documentary forms so that don't know it feels like it's kind of like this unique blend that looking through the lens of VR has you look at creating these simulations and these reconstructions and these virtual worlds and it's all this world building and then you know to then interpret that into like how does this expand the ideas of documentary so yeah I don't know if you have any thoughts especially you know considering your piece that you have here called drift
[00:07:45.928] Nienke Huitenga: Oh, totally. Yes. Well, Drift kind of came about in roughly two steps, which is way too simple to explain. But first we actually had a technical fascination because Drift runs on an RSS feed. We were like, why is podcasts still linear? Like, well, it runs on a protocol that's easily interactive. I think the ads come in interactively, dynamically. So why do we treat that reality of making podcasts so straight, so straightforward? But I think as soon as we found out that you can play around with RSS feeds, like we made a little plugin, we built our own, let's say, story engine that kind of can mix up. If you imagine a podcast episode in, for example, three, four or five parts, as soon as you press play, we can kind of mix it on the spot and we give you a unique iteration or variation. So that's what we found out. And then we thought, like, wait, the type of story you can tell with that resembles a bit more like, for example, the series Westworld, where they, within the fictional world where you have these AI hosts kind of roaming on a loop, we can kind of make characters that maybe day by day have a very particular routine. So I think that's the first thing that we wondered, how the narration changes. And I said, well, maybe this was the third step. I was wondering... The hardest relation we have to the future is that it's either very far away or very abstract. But with a flexibility in even short story clips, I think you can make playful combinations between the now and the then. In Drift, the then is somewhat abstract. It's kind of like 500 years into the future and Netherlands where... the water has reclaimed the land it's actually a real scenario we're here in amsterdam but i think large parts of amsterdam will in 200 300 years not be here anymore so depending on how we deal with that like we can maybe build floating houses or we can move away i think the most practical scenario is that most of people will find dry land But it also came about, I think like many people, I'm mentally preparing for, let's say, a nearby climate that's less comfortable. It's maybe going to be a bit of a hell in the next 50 years. But I'm also a little bit fed up with the feeling of inaction, kind of passiveness, because all this negative reporting... I believe it, I embrace it, but I don't know how to move forward. So this creative way of dealing with reality, maybe in Drift, is that we embrace that and make a jump where we assume we will adapt, we will find a way. So Drift kind of depicts a community that slightly in a happy-go-lucky way, but also in a nostalgic way, describes how the changes have happened in some rough big steps, but also in these very intimate personal moments where someone describes how my garden is now like a floating vegetable patch. My kitchen works like this. We also have an octopus that actually holds up beautifully recorded monologues with the voice of Shana Bosman. She reflects our human nature, but also how humans are kind of stuck in a linear timeline, while most of the animals are in a circular timeline. So we try to mirror that our thinking about our climate crisis, our thinking about the future is kind of stuck on an A to B point, while actually it goes in circles. And if you embrace the circularity, there's a lot more change easily you can do because you don't have to promise to only be better, but you can also kind of try if you're successful, like, I don't know, letting your car in the streets and not, you know, going on a plane flight, for example. So that's a win. But even if you fail, there's a next chance, there's a next cycle. So it's speculative, it's non-fictional, but we do use actual data sources. So we did some research on what will happen to the landscape of the Netherlands. Rivers will go beyond their riverbeds. Of course, the sea level will reclaim quite a bit of the West Coast. We took in weather data. We made a huge archive of relevant news articles. So we made all these little databases that infuses with the right facts and correct framing this speculative future world. So, yeah, we used optimism to manipulate a relation to reality because I feel I need it. Otherwise, I don't know how to actively change.
[00:12:11.906] Kent Bye: Yeah, just being here in Amsterdam, people talking around the dam that's here, protecting Amsterdam. Someone asked me, do you know that Amsterdam is actually below sea level? I was like, no, I actually didn't know that. And how much of the Netherlands is actually below sea level because of all these dikes and dams that have reclaimed land? And so maybe just give a little bit more context as to that. Because given that context, Amsterdam ends up being on the front lines of some of these Climate conversations just because they're really you know at peril in a number of ways So maybe you could just kind of elaborate on the the context of what's happening now in Amsterdam. I
[00:12:48.212] Nienke Huitenga: Oh, so much. So I would call the Netherlands some sort of a water machine. Let me start with this. So the reason we chose a future that is roughly 500 years into the future also has to do with the fact that the way the Netherlands looks, especially the area around Amsterdam and kind of the West, whole area from, let's say, Alkmaar, if anyone knows Alkmaar, and it goes all the way down to Rotterdam. It's artificially managed. 500 years ago, around 1533, the invention of making polders, making wetlands dry, is quite a crucial... The way you know Amsterdam, the way it can exist, has to do with the fact we invented these mills that could mill out water and then you have a polder and then suddenly you have also land that you can build on, kind of grow vegetables. In 1533, there was quite a famine. There were many troubles. It was like somewhat of a modern time, but people were kind of clinging on to some hard, let's say, environmental situations. With making these polders, you see all these flatlands in straight squares kind of appearing. And that's also a way where how Amsterdam kind of expanded. So the way we know Amsterdam right now is actually an artificial, really man-made island. To the now. So everything works with dams and dikes to keep the water at a level that we keep our feet dry, that the drinking water is not mixing with the salty water and etc. But our modern problem is that, let's say, the bureaucracy and the policy is both doing the upkeep. But with a neoliberalistic way of working, the upkeep of our gates and our dams and our dikes has been delegated to commercial companies. So not all of that infrastructure is kept well. So last year, Amsterdam nearly flooded because some of the gates just were in bad state. So in our very modern, modern time, we not only have a climate problem, we also have like a structure problem. So it's a bit of a technical explanation with all the artificial interventions we have in our landscape. Most of the nature you see around the Netherlands is managed. We don't have that much wild landscapes. I think if you go more to Limburg, the south, it's a bit more, let's say, wild and rough. But thinking about the problems with water, water is maybe, it looks like a soft element, but it's the most powerful element we have. Like as soon as a body of water kind of decides to become a wave and thunder to your city, you're pretty fucked, like it's quite forceful. So we were fascinated by the fact that us Dutchies kind of tempered it, kind of found a way to manage it. But it's an element, especially with climate change, that is so destructive. Amsterdam is maybe a nice icon to think about. So going into the future, I figured if that happens in 1533, so we could find a way to create land and manage water, we will also find a way to live on it, or kind of a new way to be in sync, be in balance with quite a powerful element as water.
[00:15:54.671] Kent Bye: So welcome. I'm wondering if you could, no, no worries. Maybe you could introduce yourself and tell me a bit about what you do in the realm of immersive media.
[00:16:02.277] Lieven Heeremans: Sure. My name is Lieven Heeremans. I'm a podcast maker and I'm one of the creators of Drift, a generative open world podcast, but you probably already mentioned that. Not even. Not even. Okay.
[00:16:13.186] Kent Bye: Well, great. We were kind of holding off until you got here to really, really dive in. We covered a little bit of it, but yeah, maybe you could give a bit more context as to your background and your journey into this space.
[00:16:21.809] Lieven Heeremans: Sure. I actually worked at DocLab for a couple of years. This was like five years ago. I did the summit and stuff they're doing for the industry. And I was working at DocLab in 2020 also when it was mostly online due to the pandemic. And then I was making my own podcast. It was a podcast about cheese. And I was using a system called Art19. It's a hosting platform where you can program adverts. So people use it all the time to program ads. And it's actually bought by Amazon now. But I was adding trailers to my podcast episodes of podcasts made by friends of mine. And then I found that through the system where you plan the campaigns of the ads, you can actually program a lot of stuff so you can see where ads should appear and which should appear on which day. And then I thought, hmm, maybe we can turn that ad system into like a generative system where everyone who listens to the podcast has a different experience based on either where they listen or which day they listen. And then I went to Haya Nienke and we came up four years later. We have Drifts.
[00:17:30.147] Kent Bye: Great. We talked a little bit about that generative nature. And so I subscribed to the RSS feed. I tried to download stuff ahead of time. I had some various issues. It might have been firewall issues I wasn't able to download. But I was able to listen when I was connected to Wi-Fi. Does every person get a different feed? And is it somehow tracking me as an individual? Or does everybody get the same feed and it just is kind of dynamic and generative each day?
[00:17:56.994] Lieven Heeremans: Yeah, this is one of the things why it took so long to develop because we had to find out either if we wanted to make like an app or something like that so you can actually track people or at some point we decided to use the OpenRSS platform because it's ancient but still running and we kind of hacked it. And you can still make private feeds for everyone. So people who pay for podcasts sometimes get a private feed. But that's also a lot of hassle to onboard people. So that's how we came down to decide that everyone basically listens to the same. But the way it's set up now is that you don't have to listen to everything because I think at the moment they're like over 150 episodes and we're about two-thirds of the month now and next month everything will be reset and we generate a new version. So depending on when you listen you still get a different experience.
[00:18:49.255] Kent Bye: OK, yeah, so I immersed myself and dug in. And you said it was tied to the moon cycles, but I don't know if I was able to discern enough, or maybe I was listening to it in a fragmented way. It feels like it's a very fragmented experience just in general, just the way that I was consuming it and just the way that it's set up. I mean, I've published nearly 1,500 episodes on the Voices of VR podcast, so I'm very well aware of overwhelming people with things in the RSS feed. RSS feeds were really designed in a certain way have people publish one episode a week. And that's kind of like how most of the podcast applications are built. And so I feel like even the way that I produce my podcast, I sometimes have to fight against the way that the default, like most people, use RSS feeds. And so with this experience, it feels like you're doing a big process of building this world and giving these characters these moments and these segments of like, here's the weather. And now here's this bar and a jokes. And here's kind of like different scenes. We have the essential workers, and you have the octopus. So you have all these little fragments that are building up a larger world. And so maybe you can take me back to the process of mapping out this world, and then how you're going to incrementally in these one to two minute little clips start to build out this world through the characters and the world itself.
[00:20:11.816] Nienke Huitenga: It's a good description. So the concept behind this, what we thought about is what if you can roam with your ears. So you have room scale VR, you have games where you explore a level and then for example with Zelda or Animal Crossing you have this village where several characters walk around and you can choose who you want to talk with. Of course, with an RSS feed, you cannot actively offer a map, so it's just a list. But we figured, what if we have multiple characters that come back daily? So it's not only generative on a moon cycle, but it's also kind of every night unique episodes are generated. But you meet these characters and they have some sort of a loop. And you can either follow one character, or you can kind of hop and snack and binge it, or... So the big volume of short episodes, maybe it's not meant to listen to it as a completionist. Like maybe it's not really a podcast, but you can roam it.
[00:21:11.625] Kent Bye: It's like a river that's flowing that you're dipping your toe in.
[00:21:14.039] Nienke Huitenga: You're dipping your toe in and we do notice, like, I listen to it differently than how Lieve or Hai or anyone listen to it. But there is a plot and you can follow it through the octopus and the character Sigrid, who's somewhat of a woman in a political position in the future Netherlands that has to do... make some big decisions about the water management. But all the other surrounding cooler locale type of episodes are residents, citizens of Waterden, we call it, that kind of describe more the context. So you can also stick with the cafe and there's a voice. She describes other citizens. So even if you just want like a really laid back listening and you don't really bother about plot, you just hang out in a corner. It's something that I remember from playing Skyrim. I'm maybe like a casual gamer. I really like to hang out in these villages where you have these really random conversations. And yeah, so part of that, I think we try to hack into the concept. So if you go into it and you're looking for a podcast type of experience, it's quite confusing. But if you go into it and you, yeah, it's more like an Easter egg. Like we have like a huge list of Easter eggs. Yes. Yeah. Roaming with your ears. That's why we like to look at it.
[00:22:27.731] Lieven Heeremans: Yeah, it really, comparing to your 1,500 episodes, we're trying to break people out of how they usually listen to podcasts. So actually the feed turns into like an interface where you can browse on your own. And it's also kind of hard to onboard people, I guess, because you need to like... get them into the story, but also explain how to roam, because people are not used to it. And also I'm hearing from a lot of people, my mom actually said, yeah, I listened to 80 episodes now, I still have 50 to go. Very optimistic, because people are just overwhelmed, even though most of the episodes are between one and three minutes. So it's like 10 minutes per day, which you can easily manage, I think, if you're an average podcast listener. People have to mentally switch, like thinking, ah, this is a world I need to explore with my ears. And actually yesterday we were talking to a student they were visiting with their class and he was like, yeah, I listened to one of the first episode I saw on a feed, which was a one minute thing and didn't get me into the story at all. And we were like, ah, right. The very first episode is called How to Drift and it appears every week. So that's more like an instructive episode of like also one minute, I guess. We should put that as a trailer so it sticks at the top of the feed so people actually know what to do. So we're going to do that soon. But you find out things as you go along, I guess, because this is the first big test we're doing.
[00:23:46.997] Kent Bye: You mentioned it as a generative podcast. And it's a huge amount of content. It sounds like you've got these voice actors. So did you train a large language model or use the AI to start to build out this world? And maybe you could elaborate on that process of you mentioned some of the training data sets that you're using. But yeah, just what pipeline you're using and how do you use the affordances of a large language model to construct a world and prompt it and then have that consistency in the characters and then be able to guide and direct from a narrative perspective something that is pretty chaotic as a latent space that hallucinates and is really difficult to wrangle. So I'm just wondering how you manage that.
[00:24:26.676] Nienke Huitenga: Oh yeah, that's the fun part. Let me start with the voices. So when we started our technical research, there was no chat GPT yet. So we totally modeled the initial format on like a very classic database style digital storytelling. So we had like a mixer. We made our own story engine. We did a couple of things. We could trigger episodes on time. We could mix like four or five fragments into one unique variation. We could add music, we could layer different sound effects. So we felt like, oh, that's cool. Like we have a very old school media machine. At the end of that year, suddenly there was JetGPT. And I think two years later, we noticed that plugin, the API of OpenAI we use. If you combine that with a software like 11 Labs, so you can have not only text to speech, but you can also kind of prompt it. And that's where we saw that if you have a very specific recipe on the basis of the research and our speculative ideas, we created a world building that is kind of the, how do you say, the core source. We always, with any prompt, this core source will come along. And then we put a little prompting recipe on top of it that will have quite a few restraints. So we tell it particularly who's talking, from which perspective, how many words. And even sometimes we give it a little bit of a script structure so we know how it begins and how it ends. So when Eleven Labs became so solid, I think it's a year ago that we discovered it can actually work. We did notice that most of the AI voices that were offered were so polished and a little bit empty. Like they sounded so perfect. You do not really have an emotion with it. Like it's like a superhuman talking to you. So we figured that playing around with voices that were maybe less well-trained actually might give a bit of humanity back to the audience. So your question, we did not really do a whole training of voices, but we asked our friends and colleagues and we put our own voices in Eleven Labs just on the basis of like a 30 second, one minute clip. And we don't try to perfect it. So it has flaws. So we have, I think most of them, all of them are Dutchies and they put in like a Dutch fragment and then Eleven Labs kind of translate that into English. So we all speak very neat English. Thank you, Eleven Labs. But sometimes you do hear an accent and sometimes you do hear, let's say, that certain tone of voice that comes with the Dutch language. But I think that somewhat imperfect way of speaking that comes from, let's say, a not so well-trained voice. makes it a more human world to listen to. So we chose to stick with that because it has a particular signature sound. I do play around a little bit with... I noticed that when you tell in the prompt how slow or how fast someone can talk, you can actually influence what Eleven Labs puts out. So we use this connection between OpenAI and Eleven Labs. So they kind of, I don't know, these two black boxes talk to each other and they make something that has become part of Drift. And what I really enjoy is something that I actually discovered two weeks before the premiere at DocLab. If you give someone a particular bio, so in my prong, for example, we have a barman who provides therapy with specific cocktails, specific waters. So I described in my prongs, this is a guy who grew up in Paris in the streets, was a poor boy, but he migrated to London and he kind of climbed up in the restaurant sector. And now he's a very successful cocktail barista. I don't know where it happens. Maybe this is somewhere in the magic between OpenAI and Eleven Labs. But Eleven Labs made him speak both French and English in one clip. So he has freedom to say what he wants, but there's a specific order in which he says things, so he always starts with a greeting. And then instead of the typical, hello, mister, what would you like to drink? He suddenly said, well, hello, monsieur. And I was quite surprised that somewhere, I think also in the programming of Eleven Labs, apparently you can mix languages. So part of what you hear in Drift is a bit of a surprise as well for us. We did give it, of course, limitations. So no racism, no misogyny, you know, like all those healthy limits we also kind of reiterated in our back end. So I think the imperfectness we also embrace. So whatever comes out, we do try to, the way I approach the prompting is that I want to have an interesting mix where imperfect results are in balance with the meaning making that needs to come out of it. Most of the voices you hear don't have these weird sounds. So sometimes when you put a voice through a laughing-lapse, it goes... It makes these guttural noises. So we try to find a balance that it doesn't do that. But when it does a weird break, like it pauses longer, or sometimes it ends a sentence in a question tonation, which it was not there, not in the script. I was like, yeah, okay, well, that's maybe how future people talk. Yeah.
[00:29:34.267] Lieven Heeremans: And then Hai, our programmer who couldn't be here at some point during his holiday, he sent a picture of him sitting beside the pool reading a paper on the MFAIR model. Do you remember? And this is a model where basically the AI generates like a writer's room. You give them a world prompt so they know what's happening. And then they create a script, and then judge it, and then make a better version, which led to some different results. But the interesting thing, I think, is that it actually can remember what happened the day before. So the essential workers, basically, they leave logs for each other, but they know what happened the day before, so they react to one another. And that's different, I think, from if you just use 11 Labs or GPT. So that's actually really cool, because...
[00:30:21.738] Kent Bye: even though we don't know what's coming out it's still responding to the world as it generated itself and is there a curation process where you're like editing or deciding like generating like five takes and then you know taking one of them or are you just like one shot this is what it is
[00:30:39.594] Nienke Huitenga: It is what it is. So we embrace whatever AI wants to co-create. But it goes through some testing. So I try to fine-tune the recipes. I find it has a good range that I find agreeable. Okay, you can have this output. So when a recipe, a kind of a prompting does not work, when it goes, often AI is very chatty. Like it really likes to talk. So most of the recipes that I try to find a way to tell it to really stick to a certain tension arc, sometimes it's like, do not generate more than 100 words or always end with this. But no, what it generates is the output. Yeah.
[00:31:21.870] Kent Bye: OK. And then it basically has an arc for a month that's kind of following a lunar cycle, and then you're wiping it clean and starting over again. I wanted to just comment on some of my experience of listening to it in terms of the speculative futures that you're building because, well, first of all, there's a lot of jargony language that you have in this world that is explained in some episodes, and then some other episodes are just used. You know, you get this underlying confusion sometimes when you watch sci-fi because you're just trying to figure out, okay, we're in the future. There's things I don't know what they are. And then you're like creating these new cultures and new technologies and new practices that these people are doing. So there's a bit of that that is through listening, but also there's this retrospective looking from the future perspective, like looking back into today. And I feel like that's, Kind of like the heart of the piece is like looking at how all this data from climate change and these reports and then people in the future looking back and like, oh, these silly people in this day and age, they thought blah, blah, blah. You know, like this kind of critique from the future that gives this speculative world building of, actually grounding us into the reality of what's happening right now through this mechanism of these speculative techniques. So I'd love to hear a little bit about that overall architecture, you know, like we were discussing earlier, the creative treatment of actuality, using these speculative worlds to be able to reflect on the reality of today.
[00:32:44.787] Lieven Heeremans: Well, to your first point, did they ever explain to you what a subspace defibrillator was in Star Trek?
[00:32:50.811] Kent Bye: No, probably not, no.
[00:32:51.863] Lieven Heeremans: So I think, especially comparing it to games, you just get thrown into a world and you just have to figure out what's happening. And actually, yeah, some stuff is explained, but even to me, stuff still is confusing. But it doesn't really matter. You can leave that be. That's the interesting experience here in the podcast, I think, that not everything is explained like it is generally in nonfiction or fiction, like podcasts are set up now in six-episode series. To the second point, I think... We had some brainstorm sessions and then in the Netherlands there were these huge protests of highway blockades and these were organized by Extinction Rebellion. So climate was a really big topic. Actually last year when we had to pitch our project here at the DocLab Forum there was an insane big climate protest here like right around the corner. So that was really... actual at the time and I think we said before we're not an activist podcast but it's something that reflects our time and then as Nienke referred to earlier we wanted to actually switch future present and past in the piece in that we earlier the Netherlands is made from water and we're probably going to return to that whether we want to or not. So we have to deal with that and that made the future world. But little did the Dutch know that the future would hold this if you extrapolate the climate predictions to any random place in the Netherlands as it is set up in some of the episodes now. There's also this piece Ancestors which works with the same theme where you have to think about what you're doing later on. I hope that actually works in a similar way if you listen to this piece.
[00:34:28.226] Nienke Huitenga: Yeah, I think it's a term from psychology called surrogation, to look at a surrogate who is in a different development state than you. Like that actually works in the here and now. So for example, if Lieve is in a certain point of his career and I am striving for a certain development, like I could learn from Lieve, how did he get there? What steps did he take? maybe the role of the speculation slightly fictional world you hear is kind of to have a mirror to kind of surrogate like is this the trajectory i want to end up with is that the end point or do i want to kind of make changes in the now so i don't have to live in a floating house or i don't have to filter all my drinking water every day. So I think the role for that for me personally also as a maker is that I want to be able to feel that because feeling time is hard but to feel that jump like okay what could be a possible connection between now and then because most of the images that come to us they are mostly images are The protests, the floods, like last week in Valencia and Spain, these images are horrible, but that is going to be more frequent. And these storms and rains, one of the most solid predictions that will happen between now and 10 years is that the rains will become kind of these stormy rains and tougher wind. So even if sea levels is like a really big threat, but on a daily, like your, let's say your micro atmosphere is continuous rain and extreme droughts. I think we can picture that, but the consequence of having that every week, every month, every year. Yeah. I'm trying to offer something you can mirror yourself with. Like, is that the bigger environment I want to be in? Yeah. Yeah.
[00:36:18.795] Lieven Heeremans: hopeful and awake so it's we didn't want to make it like a doom scenario but more like solar punk because that's a really little explored genre in the netherlands i think generally also so yeah in that sense it's also hopeful that there's still people living there but they're living together with the water in the more and ai in more collaborative way
[00:36:40.529] Kent Bye: Yeah, I sent you an article that was getting into some of my deeper thoughts in terms of this relationship between these speculative futures, these possibilities, and the actuality. So there's a large set of possible futures. But when things come down to it, those sets of possibilities get collapsed down into an actuality. And so there's a way in which that you can play with those speculative futures in a way that helps people to imagine these different courses. And I think the difficult thing is always to then Okay, then now how does that translate into action they're going to take now or the next step? And so when you think about Drift as a podcast, what do you hope as the most exalted result for people kind of immersing themselves into this type of speculative future and then coming out and then what? Is it just in the realm of culture of going into this world and thinking about it, or is there any specific political action to be taken? So I guess that's the question that everybody who is doing these types of impact projects has to think about. And it feels like this is a piece that's very experimental and playing with the technology, but I'm just wondering what the most exalted potential of that would be.
[00:37:47.000] Nienke Huitenga: I think to have maybe more imaginative conversations with your family and your peers and your friends. Because I think we can all have that doom talk all the time. But I think we don't really take anything out of it besides agree on it. But I think imagination really helps to be able to imagine yourself in another setting or kind of in a new lifestyle. Even if it's the smallest thinking about... I think it's quite precious that we have drinking water flowing from the tap. But that's the first thing that will stop. I think our daily rhythms, we don't reflect on that. We really like to reflect on our big achievements in a year. But I think the small things are what you can actually embrace better. And if you have an imaginative stuff, like, oh, what if... Even if you think about a simple thing like remodeling your house, we need to have more imagination about being future-proof. and there are all these tech companies and they also have good intentions but you know with the green washing around like you actually don't try to be led by what a company decides for you but also think about what's my way of living like if i stick in the city and i want to be able to not have it so hot in the summer because the summers are getting hotter Could also be that you move out of the city. I think the imagination on going into the future is not like standalone solutions, but actually think about your long-term daily rhythms. So I don't assume this is easy for everyone who listened to it, but I noticed with myself that the more positive imagination comes into my life I can actually see more structures and patterns that I feel I have control over because it's very easy with a doom future to also just let go of like oh but I can change nothing about it so that feeling of oh I can actually maybe make a small change I think that's what I'm trying to aim for but also at least imagination is the first key
[00:39:38.334] Lieven Heeremans: Yeah, I went to Drinking Brecht yesterday, and it's really about Brechtian theatre and how it's different from Aristotelian, where the conflict stays within the piece. And with Brecht, he actually wants to move the audience to do something in the real world. I don't think we're that strong. But I think, personally, I went to those climate protests the day after we had our brainstorm. And that was hopeful in a sense. And I think what we want to give with the project is that real world actions do have real world consequences. So we can also decide to do better. And then theoretically, if the podcast would go on forever, the scenario would change. So there's no real interactivity in the project itself. unless it's over a super long period of time so yeah i think that makes it more interesting um yeah and have people reflect on what they're doing now that actually has consequences for the future even though it's very uncertain still awesome and and finally what do you each think is the ultimate potential for this type of immersive media and what am i people to enable
[00:40:47.478] Nienke Huitenga: Oh, the big question. Well, thinking specifically about what's behind Drift, technically, the way I think video games became kind of moving from an abstract, simple experience like Pong, moving into these very large scale RPG games. I think they also developed into like a poetic medium. So I think what we are doing right now is quite poetic and abstract. But I can see this listening, let's say, audio dynamic that we created. I think it's very interesting to use in fiction. We've made a roaming experience, but I think you can also make like a very journalistic approach. So if you want to include sonification of data sets, kind of offer people a sonified perspective of a bigger structure. If you would do something about the big planets and the cosmos. I think the medium of sound is so unexplored that really attracts me to it. At the moment I'm really in this format world and see all the potential in these tiny different possible playful formats. We're doing it speculatively, but I think journalism could use this, documentary could use this. So also there's this film about Brian Eno that's also generative. So I think we have a parallel there. They express it in film. I think we did it in audio. But you also, as an audio maker, you have tons and tons and tons of content. and you don't maybe all want to edit it with you sitting at the computer. So if you use AI as a collaborator to find some parallel or interesting cross-editing possibilities, I think we have a new engine to do so. Yeah.
[00:42:25.382] Lieven Heeremans: it's funny because i listened to so many of your episodes and i didn't formulate an answer to this question yet but i think speaking of creative achievement of actuality what i think creators are trying to do is create as you said poetic moment or aesthetic experiences that have meaning and just now i talked about how we can technically change the trajectory of the podcast story if we do something better in this world now I think for me, what was really meaningful is that at some point one of the episodes reflected on the fog that was in Amsterdam yesterday. And I was like, ah, OK, now I have a real world connection, but it was in a really short experience. But it was as meaningful as maybe while not changing the future of the world. That created a poetic or aesthetic experience. Even though it was very short, it was meaningful. And I hope that's, well, it's not the ultimate potential, but there's way more potential there.
[00:43:21.291] Kent Bye: Awesome. And is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:43:26.508] Nienke Huitenga: Oh, please do more with sound. Yeah, I think especially in, let's say, our wave of synthetic media, of course, it's also quite scary kind of how voices are easily replicated and synthesized. But I think like the imagination you can open up in someone's mind through audio, through storytelling, voices, soundscapes. I think we do it way too little. I love VR. I'm also directing a VR, but I think catering for vision is, yeah, it's done. I think there's this virtual world that is only created by our neurons by entering it through our ears. Yeah, that's where I want to focus on. Yeah.
[00:44:08.052] Lieven Heeremans: Yeah, you don't need that much to tickle the imagination, I guess. I just wanted to say that I didn't listen to all the 1500 episodes, but I listened to a good chunk of them. So I'm very honored to be on the podcast. So thank you for having us. And yeah, people, close your eyes and see. I guess that would be my giveaway.
[00:44:26.614] Kent Bye: Awesome. Well, that's a great closing thought. And yeah, I really enjoyed listening through Drift. And I think I have to, like I'm more of a binge listener than a casual listener, especially because I actually don't have data on my phone. And so the RSS feeds will only work if it downloads online. And there was issues with something for whatever reason. I was able to listen when I was connected to Wi-Fi. So this seems like it's something that would be nice if you're kind of walking around and listening to it. My listening style is more of a completionist, listen to everything. And so I feel like I'll have to pay attention to it more and listen to it, maybe even download it and use WhisperX to get transcripts and analyze it, to really look at it. Because it feels like it's so rich in so many different aspects. But yeah, I just really appreciated that spirit of the speculative world building to project ourselves out in the future and look at what's happening now. And yeah, it's just a really interesting technique, especially in the context John Gerson idea of the creative treatment of actuality in the form of documentary. So yeah, thank you, Neekan Levin for joining me here today on the podcast to help break it all down.
[00:45:31.195] Lieven Heeremans: Thank you, Kent. Thank you so much.
[00:45:33.876] Kent Bye: Thanks again for listening to the Voices of VR podcast. And I really would encourage you to consider supporting the work that I'm doing here at the Voices of VR. It's been over a decade now and I've published over 1500 interviews and all of them are freely available on the voicesofvr.com website with transcripts available. This is just a huge repository of oral history, and I'd love to continue to expand out and to continue to cover what's happening in the industry, but I've also got over a thousand interviews in my backlog as well. So lots of stuff to dig into in terms of the historical development of the medium of virtual augmented reality and these different structures and forms of immersive storytelling. So please do consider becoming a member at patreon.com slash voices of VR. Thanks for listening.
