Another AI-based project at SXSW this year was SWEET! by Dutch designer and artist René van Engelenburg of DROPSTUFF MEDIA. This is a projection-mapped physical installation designed to facilitate a very brief conversation with an OpenAI LLM-driven virtual being about the parallels between how sugar is added to all of our food and how AI is currently being added to all of our technologies. The virtual being uses a sort of Socratic method of questioning to solicit your opinion in four leading questions that are making an argument about the commonalities between sugar and AI. It’s a short five-minute experience, and so I did the piece three different times to understand the underlying mechanics of how it’s built. It never felt like a true conversation as there’s a superficial acknowledgement of my responses, but always immediately progresses onto the pre-set monologue that is pre-written within the series of prompts without having what I see seemingly impact the substance of the conversation beyond some prompted images that are projected onto the white set of a candy store. So instead of a truly, open-ended Socratic dialogue, this experience feels more like a Socratic monologue where questions are only driving a superficial sentiment analysis before moving onto the delivery of the next random fact or story beat. We still have a long way to go before these LLM-based experiences can balance a convincing conversation while also telling a compelling story, but this feels like the type of XR immersive installation providing a holographic embodiment of the virtual being is a compelling form factor that I can see being further developed in the future.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of special computing. You can support the podcast at patreon.com slash voicesofvr. So continuing my coverage of different immersive stories from South by Southwest 2025, and also diving into the theme of artificial intelligence of different AI-related projects that were at South by Southwest this year, Today's episode is with a piece called Sweet by Dropstuff Media and Rene van Engelenburg. So this is like a short five minute interaction that you have with a AI virtual being. The AI ends up asking you like four questions and you answer them. And then from those answers, it kind of goes to this script where it's essentially comparing artificial intelligence with the sugar industry and all of the way that sugar is integrated into everything. Now we're starting to integrate AI into all of our different products. And so there's these different parallels that are trying to be made through this system that ends up having like a number of different components. You walk into this installation, you see all of this white blank candy. And then once you start interacting with this AI virtual being, there's this like red button that you're pressing and you press to talk to AI is asking you questions and you're responding, and then it's kind of progressing through a set script with the hierarchy of different prompts that are going through an order. And as you're answering, it's also taking your answers and then doing image generation and projecting on top of the candies. And so there's another layer of visual complexity as you're having this conversation. So my experience with AI, large language model related pieces can be mixed because sometimes it ends up being either the limitations of not being fully understood, or in this case, it felt a little stilted in the way that is able to really feel like I was having a responsive conversation. I ended up doing this piece three different times. And so I went in and was able to try out different strategies. And more or less, it's like the same type of arc. You are able to say something, but then it's only briefly acknowledging the sentiment of what you're saying of like, oh, that sounds good or bad or controversial. And then it immediately switches into the next part of the question. And so it feels like, is this really a conversation or is this kind of a monologue where you're not really able to really shift it or feel like you're able to steer it in any way so it feels very on rails and when you have an open-ended conversation that feels like it's on rails then it doesn't really feel like it's much of a conversation at that point it feels like you're receiving the story so those are the different types of tensions that I felt when I went through this experience and so I start to discuss some of those different aspects with renee but also just trying to get a broader sense of drop stuff media and their entry into this intersection between ar virtual reality xr ai and how all those things are coming together so we're coming all that and more on today's episode of the voices of vr podcast so this interview with renee happened on monday march 10th 2025 so with that let's go ahead and dive right in
[00:03:17.779] Rene van Engelenburg: Hi, I'm René van Engelenburg. I'm from Drop Stuff Media. We are a media arts collective. We make immersive arts and interactive experiences. We are based in Amsterdam, Netherlands. And we work mostly for museums. We work for festivals and we make our own autonomous projects. We started with physical structures and adding virtual and digital layers on top of that. We did a lot with virtual reality already since 2016 and after that a lot of augmented reality. Now we're more in a mixed reality because I believe it's very interesting where the physical world and the virtual worlds mix. In my belief that is where very exciting things still can happen and will happen in the next years with all the technology that continues to develop and So right now, of course, AI is a very big topic and we're here in Sardbein to present a work about artificial intelligence through a metaphor of a candy store. Artificial intelligence is, in my opinion, a big box of candy and we all like it and it's very sweet. But just like sugar, it has also a negative side and we should see it and try to find any balance in how we use artificial intelligence. how we can implement that for the better.
[00:04:46.598] Kent Bye: Great, and if you could give a bit more context as to your background and your journey into this immersive space.
[00:04:52.978] Rene van Engelenburg: My background, I am educated as an architectural designer and I did a master in media design. So we started Dropstuff Media in 2014. And also before that, we did a lot of media arts, media artworks, a lot of video, interactive video. So that kind of stuff. And yeah, what shall I say more?
[00:05:15.727] Kent Bye: Well, yeah. And just in terms of like, what was it that really sparked your imagination to get more into the immersive side of things?
[00:05:23.467] Rene van Engelenburg: For us, the technology is always a way to tell a story and to immerse people really into the story with all kinds of senses. Not only using your eyes, but your ears, your haptics, your smell, taste. And the best immersive experience is where you use all of them. And so I like to create, or my team likes to create, exhibitions, experiences where you use all of them and where you really feel immersed as a total person, where you also feel physically immersed. So right here, here in Sao Bai, we present something where you really have to go into also physically. So it's not just a headset putting on and you're in a different space. No, we use the physical space part of the design.
[00:06:15.856] Kent Bye: Yeah, and I'm curious to hear this evolution from starting into VR and AR and then coming back into this mixed reality with the physical installation. And so where did you begin with your explorations into, like, let's start with virtual reality?
[00:06:29.696] Rene van Engelenburg: The first project we did in virtual reality, it was a project that we showed on the Biennale in Venice in 2013, I guess. And it was with one of the first glasses. And it was two minutes, 360 degrees video, like a roller coaster ride through Venice. So you were sitting physically on this small children's attraction where you put in a coin and it will start to move. then you wear a headset and in the headset you make a tour in about one minute like a roller coaster ride through Venice so you're done as a tourist and we did the same for Amsterdam a tour through the boats through the canals of Amsterdam and so you could choose which video you wanted to see you paid two dollars and then you did the touristic stuff so it was a statement in the center of Venice about overtourism. And it's a place where a lot of tourists come, of course. There are still people living there and there are massive people coming for the arts exhibitions. And the funny thing that we recognized all three groups were there in line waiting to do this VR experience. And that was really funny because, okay, we use a form of folk culture, you know, these small attractions, children's attractions, and people like to see it. People are wondering what's happening inside the classes, but people are also interested in being part of the total artwork because they are also physically engaged because it's funny to see people sitting, adults, seeing adults on these small children attractions with their glasses on. So that's where we noticed, okay, this really works also within a physical context.
[00:08:15.823] Kent Bye: Okay. And were there any augmented reality projects that you're working on or did you skip straight to like mixed reality stuff?
[00:08:23.044] Rene van Engelenburg: We did also a carousel merry-go-round. We used cultural heritage materials from the Eye Film Museum in Amsterdam. They have this collection of the first moving images from 100 years ago. It's like a tiny tin box with holes in it and a strip of paper with a walking horse. And if you turn it, you'll see a very small video of a horse walking or running. And these paper strips of 100 years old, we digitized them, put them in augmented reality. We used the HoloLens of Microsoft. And when the carousel starts turning, you will see a little theater play consisting out of these very old images and animations in augmented reality. So you will see the real thing. You see the carousel, you see it turning and in the same time the carousel is a stage for images for more than 100 years old. So this is the way we like to use the technology. We connect the newest technology to the oldest technology.
[00:09:33.726] Kent Bye: OK. And at this installation that you have here at South by Southwest, you have quite a robust physical installation. And then you also have the virtual being with the AI screen-based. You have these interactions where you're able to have this conversational interface. But you said you're getting more and more into mixed reality. Is that what you mean in terms of having both these physical installations but also have these interactive virtual components? Or were there other previous mixed reality projects that you worked on before what you're showing here at South by?
[00:10:04.095] Rene van Engelenburg: We did a lot of mixed reality projects. Last year, here in South Bay, we presented in the Creative Industries Expo a project that was called Fiera del Suono, which means something like the fairground of sound. And it was a 300 square meters experience with big inflatables and two spatial audio systems that made you travel through the ages in the history of the fairgrounds. And using only sound and objects. So you were physically walking in this space. And every sound is like somehow hidden on a specific spot. So you will hear the story told by a woman who leads you through the story. And depending on your location, you hear a certain part of the project. So also in this context, we use physical components and virtual components, digital components to create a story.
[00:11:00.990] Kent Bye: Okay. And so the topic of the piece of Suite is all about artificial intelligence. And so when did you start to integrate AI within your own projects, or is this the first time that you've really started to dive into it?
[00:11:14.815] Rene van Engelenburg: We had a project before where we had bumper cars, bumper car track. You could step into the bumper cars and then they start moving like you're used to. You can steer the wheel and you can go wherever you want. But suddenly all the cars were taken over by an AI. And the cars started to dance, to make a dance together. And as the driver, you couldn't use your hands anymore because the computers start to control the cars. So that's also a way of expressing what happens if computers take over control. So that was like four years ago and here of course technology improved very fast and still in a few months it will be different from now. So yeah, we developed this the last three months with the technology that we have right now and it's made with five different AIs. So both for the hologram, the host who is in the candy store and who will talk to you, the translation from what you tell her back to text and the images that she will create and to project on all the candies so if you say okay i want bright colors she will make bright colors if you say i want body colors she will project body colors and even the music is lively generated through an ai so it's all happening in the same time and this makes that you can have a conversation with her and it's only a statement of three minutes and in this time she makes sure that the history of sugar is sweet and it's addictive but it has a bad side of it has a complicated history and this is the same with AI it's a candy box we can use it but it has a negative side as well
[00:13:00.934] Kent Bye: And so in terms of looking at AI as a topic, thematically, you're really interrogating some of the larger implications of how AI is being integrated into everything, just like sugar is integrated into everything. And so in this piece, it's fairly short, three to five minutes. And so I had a chance to go through it three different times because at the very beginning, you have the ability to choose like a different color, like a political color, national color. And maybe we start with the different types of choices that they have at the beginning, because it seems like the piece is always converging to a similar point in the story. And it gets there in many different ways, but that it's still, after doing it a number of times, it's still converging with these non-sequitur facts that were being shared, that I was told that they were not supposed to be sharing facts. So there's kind of like this uncontrollable nature of larger language models in the sense of trying to bound it from going too far off the rails. So maybe you could just start with talking around some of the choices that people are making at the beginning for how you're going to enter into this topic through these different themes.
[00:14:07.526] Rene van Engelenburg: Yeah, let's say the first 40 seconds we only need to explain to people what they can do and how they can handle this and how they can answer. So very technically we have to make her explain that herself. So this already takes some time and then she will ask you questions because a lot of people will not even answer. They're just used to watch and not interact. But the fun things exist when you really interact with her. She will ask you a question, what kind of color would you like? But you can say, okay, I want to see this color. You can be brave or a bit bold and say, okay, I want Mondrian or I want Van Gogh colors, whatever. And then immediately you will see that she will respond and it will give a much nicer effect than when you say only yes or no. So we have also to somehow to educate people that this is not a real person, but it's still an AI. And we're also still surprised because every time it's different. Sometimes the things come out that we never even, they're not in the prompt. So, and we're still tweaking and how can we make, because yeah, if you give it a chance, it takes 10 minutes, it's way too long. So we have to somehow cut her off and, okay, this is the end. This is the story. This is the morale of the story. So, yeah, it's a puzzle to make the people have a nice experience in only three minutes.
[00:15:31.517] Kent Bye: Yeah, it's a little like you're trying to tame a wild beast in some ways of trying to put guardrails on it. In terms of the technology stack, is this just using ChatGBT or what's the large language model that you're using here?
[00:15:42.921] Rene van Engelenburg: It's all based on OpenAI, but many different apps. So there's a special one for the music, Suno, and we use Google for the text translation from speaking words to text. So there are five different programs we use simultaneously.
[00:16:00.347] Kent Bye: Okay. And in terms of the intent of the story, what was the catalyzing moment when you decided that you wanted to tell a story around AI?
[00:16:12.390] Rene van Engelenburg: I don't know when this moment was exactly because we were doing a project around candy. And then suddenly we found out, ah, that's a nice metaphor for AI. And I guess it was somewhere October last year that we found out and we had to subscribe for Surbuy. We thought, OK, this is an interesting angle. Let's write an application for that. And then we got selected and thought, oh, my God, now we really have to make it. So, yeah, this existed only since the last three months.
[00:16:45.225] Kent Bye: Okay. And can you elaborate a little bit on the structure of this piece in terms of like, there seemed to be an initial question and then draw these parallels between the harms of corporations with sugar and then the harms of AI. I found it was asking a lot of these deeper moral questions. But also, when I was doing it multiple times, there seemed to also be a lot of variability in terms of the specifics of how it was unfolding. So there would be maybe a similar theme, but it would be set in a different way, or it would have different pathways. So it seems like you're exploring a space of potentiality, like a lot of possibilities, but still at the same time across a consistent theme. So just talk around the process of trying to sculpt and shape and constrict that type of possibility space.
[00:17:33.124] Rene van Engelenburg: It's like molding and shaping with clay. We choose five different forms of color, national colors, corporate colors, body colors, military colors. What we found out is that the AI has a morale itself, very much. Because, of course, we tested it as much as we can and tried to make her angry. And let's see what's happening if people are really rough to her. And then you find out that the AI is already so much programmed that it has an opinion itself, which we cannot implement. This is in the program already. This is the opinion of the AI. So it can be sometimes very moralistic. And that's not what we put into. But that's already the nature of this technology and how it's programmed. So, yeah, we try to say, okay, you ask two questions and then you give a fact. And then she somehow tries to tell about the comparison between sugar and AI. So these are the three ingredients that should be in every conversation. But sometimes it goes all the way different. And the next interview will be following the prompt. So, yeah, it's really, really not very stable. It goes everywhere still. And you can change few words and then it goes all the way different. So, yeah, it's still a lot of finding out what's happening if we do this, what's happening if we do that. And, yeah, this is how far we come until now.
[00:19:05.444] Kent Bye: And so is this all created with a single prompt or a series of prompts that you're interfacing with ChatGBT? And yeah, just elaborate on this prompt engineering aspect of creating experiences like this.
[00:19:17.771] Rene van Engelenburg: No, it's actually a huge series of prompts. So if you are in this stage of the interview, you have to do that. If we are in this stage of the interview, we do, and we use Unity for that. Yeah. So it's basically a sequence, a sequence of prompts. Okay.
[00:19:36.243] Kent Bye: And then in terms of the actual visual representation of this virtual being, can you talk around the design of the avatar and the actual physical installation? Because it's got this kind of hologram-y light box type of feel. So maybe just talk around the design of the aesthetics of the virtual being.
[00:19:55.117] Rene van Engelenburg: As I told before, we would like that people really feel really immersed also in a spatial way. So you go through a curtain and then you're in the candy store, which there is a physically counter full of white candy, lollipops and candy canes. And also all the walls, they have a print with the same lollipops and candy canes printed on them. And then behind the counter there's a holobox and there's a hologram in there, an avatar, and we call her Candice. And Candice starts to talk immediately to you, so welcome in our candy store and we have all different colors and you can choose from the blah blah blah. and after she gives you the introduction you can press a big red button and as long you press the button she will or the system will hear what you say and translate it and then she really reacts on that that's really funny and she reacts not only in speech but also by the projection that is mapped on the candy so depending on what you answer her immediate quite immediately the images will change And that's what I, in an artistic way, like the best of this installation.
[00:21:06.959] Kent Bye: Yeah. And I had a chance to do it three different times. And each of them were different in different ways. And so the first time that I went, I went in with somebody else, which is a whole other experience of doing it with someone else rather than just myself. There was some glitch where it didn't give proper instruction. The second or third time I did it, it said, okay, if you speak, then you should be able to see the microphone bar move left and right, and then there'll be a series of green dots that are giving you some visual indication that there's a delay from the time that you ask the question to be processed and then to have an answer. Eventually, once we got it, what was interesting around doing it with another person was that we went back and forth for who was answering. And the person I was with was answering the questions in a completely different way than I would have answered. It was asking about these deeper questions around exploitation. And this person I was with was an investor. And he's like, well, it's just the free market. And it was very pro-economy. And I was like, that is totally not how I would answer that question. And then whenever I was answering Avatar would basically say, oh, well, that's a very strongly worded way of saying that. And then it just kind of continues on without really unpacking. It feels like I could say just about anything. And so I think as time went on, I would be more hyperbolic just to see what kind of reaction I would get. So anyway, it feels like there's a variety of different... reactions, but I found it that doing it with another person was frustrating in a way that wasn't really expressing how I would say it, but it was still getting me a sense of the experience. So I don't know if that's the type of thing where you prefer to have people bump up against each other with these kind of frictions and have these other social dimension, or if this is something that you want to have people just see by themselves.
[00:22:54.878] Rene van Engelenburg: Yeah, most people go in alone. We found out now that when there are more than three people, it doesn't work really well. So because people start to talk with each other and then especially when there are two voices, then she doesn't understand it very well. So yeah, preferably you go in alone because then you are really concentrated. You also hear what she says and then it makes somehow sense. So yeah, we heard this before, what your experience is. Yeah.
[00:23:26.399] Kent Bye: And as you've done this project now, where do you want to take this in the future? Is this something that you want to continue to develop or continue to explore? Or what's next for a project like this?
[00:23:36.627] Rene van Engelenburg: Yeah, we're meeting here a lot of interesting people and people of course they react in a way for I have this festival and could we show it and it will be in autumn. Okay, yeah, but already in autumn it will be different because the technology goes so fast and maybe the way of speaking of Candice will improve a lot and also the visuals will improve a lot. So at least that's what we expect. So yeah, it's an ongoing iterative project and I guess we will work on it for the next two years. For two years, is that what you said? Yes, next two years because this is the first time we show it and we can iterate a lot on it still and there are quite a lot of festivals that are interested to show this piece and so every time we will show it, it will be different. And we will be developed further. So yeah, we also like to do that.
[00:24:35.306] Kent Bye: Are there other projects that you're working on where you see some of the core technology exploration with using AI and large language models that you're seeing other use cases for other projects that you're also working on?
[00:24:47.850] Rene van Engelenburg: Yeah, a big project we're working on. It will have a premiere in Amsterdam in July. It's called Rise of the Rainbow. It's telling the story of the LGBT community and the whole emancipation in the last 100 years. it will start in amsterdam during pride week in the great theater in amsterdam and it's a augmented reality experience so it will be a physical like a physical structure of inflatable like an inflatable city where you walk through with 90 people in the same time And in augmented reality, in different parts of this inflatable city, there will be six different topics. And you can walk through them. For instance, freedom, fighting, or loss, or desire. We will use cultural heritage materials like footage, images, and sounds to create this atmosphere of this theme. So it will be live theater. It will be augmented reality and it will be a social project in one. And we are aiming to make this travel after Amsterdam, both in the Netherlands but also in Europe. And we are making connections right now here in the States to have this, for example, in New York or in Los Angeles. Nice.
[00:26:10.227] Kent Bye: And in terms of this project, it reminds me of this intersection between like XR, meaning like mixed reality. There's installation and all the virtual reality technologies. I see this XR is like the front end and the AI is the back end where you're able to have this conversational interface where people have a lot of agency, a lot of freedom to explore different concepts, but then there are like nworld.ai that is able to like really put like a bound on things. I think when you have a raw interface with ChatGPT, sometimes it can kind of feel like it can be totally open-ended and difficult to really constrain, but at least there's this ability for, on the front end, people to have this interactive participatory expression of their agency that allows them to have a lot of agency, but creates this spatial metaphor for people to actually have some sort of concept for what they're interacting with. And so you've created an embodiment of the character and this whole installation context. So then in the back end, you have all the AI that's kind of tied all together. So I'd love to hear any of your reflections on this intersection between XR and AI.
[00:27:22.231] Rene van Engelenburg: Oh, that's a complicated question. Yeah, of course you have the front end and the back end. Some people really want to see behind the curtains all the computers that are standing there. But it's of course not meant for a show. But yeah, somehow I can understand it has an aesthetic of itself, which is totally different than the front end. But that could be another project to do something with that.
[00:27:52.526] Kent Bye: Nice. And so when you think about the ultimate potentials of virtual reality and artificial intelligence, what do you see as the ultimate potential for how all these immersive and emergent technologies are coming together?
[00:28:08.246] Rene van Engelenburg: Of course, when we're working now on this rainbow project, we have to get some characters alive who lived like 50 years ago, where we don't have any footage from. So we have to do some interpretations and AI can help us with that. so we can create new voices from people from who we don't have voices and we can create their images maybe from only one photo so in this way it is a tool to recreate figures and to recreate but it can also be a tool for a real interaction within, for example, augmented reality. So things are generated at the spot. Things can be generated. So it will be totally different for everyone what experience you will get. I think that will be the most interesting part. But we're not there yet. We're not there yet.
[00:29:09.021] Kent Bye: Awesome. And is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:29:15.703] Rene van Engelenburg: Yeah, we're talking about immersive and most of people mean with that VR or AR and I would like to stretch this up and have the full body experience where you can really get immersed like when you jump in a pool or when you make a stroll to a forest after the rain and when you can recreate this feeling that's really immersion.
[00:29:45.345] Kent Bye: Awesome. Well, thanks so much for joining me here on the podcast to break down a little bit more about what you're doing in this intersection of XR and AI, these kind of like spatial pop-up interfaces. And yeah, I feel like it's, like you said, a hot topic around the world where a lot of people are interested in seeing ways that people can engage with this type of technology to get a little bit of understanding of it. And for me, I had to do it a number of different times just to kind of understand the deeper structure. But at the same time, it's still kind of like a user-facing type of project where people can start to experiment a little bit. Yeah, I still feel like the large language models have a way of being uncontrollable or just kind of hallucinating, and it can be difficult to really rein them in, and so there's friction that I still see in some of these different projects where I have sometimes a spotty experience based upon the uncontrollable nature of that. But overall, it's a really beautiful installation, and I appreciate the overall themes of at least trying to take a critical lens of what's happening with all with everything while at the same time using the technology to make it so it's kind of like a paradox in that way so but anyway yeah thanks again for joining me here on the podcast to help break it all down definitely definitely paradox because we're using it all the way right yeah thank you thank you very much Thanks for listening to this episode of the Voices of VR podcast. And there is a lot that's happening in the world today. And the one place that I find solace is in stories, whether that's a great movie, a documentary, or immersive storytelling. And I love going to these different conferences and festivals and seeing all the different work and talking to all the different artists And sharing that with the community, because I think there's just so much to be learned from listening to someone's process to hear about what they want to tell a story about. And even if you don't have a chance to see it, just to have the opportunity to hear about a project that you might have missed or to learn about it. And so this is a part of my own creative process of capturing these stories and sharing it with a larger community. And if you find that valuable and want to sustain this oral history project that I've been doing for the last decade, then please do consider supporting me on Patreon at patreon.com slash voicesofvr. Every amount does indeed help sustain the work that I'm doing here, even if it's just $5 a month. That goes a long way for allowing me to continue to make these trips and to to ensure that I can see as much of the work as I can and to talk to as many of the artists as I can and to share that with the larger community. So you can support the podcast at patreon.com slash voices of VR. Thanks for listening.
