I interviewed director Seán Hannan about You Can Sing Me on My Way that showed at IDFA DocLab 2024. See the transcript down below for more context on our conversation.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continuing my series of looking at different experiences from IFA Doc Lab 2024, today's episode is with a piece called You Can Sing Me On My Way, which is a part of the Digital Storytelling Competition. So this is an audio installation piece that uses a lot of generative AI techniques. And so the idea is that it's trying to recreate the Shanoi way of singing. So this is a traditional way of Irish songs that are singing about the news of the day, or at least singing about culturally significant moments. And so it's integrating a number of different generative AI pipelines where it's looking at the news of the day from a number of different sources and then taking highlights from that and then translating that into a song or a poem. And then from there, it's translating from English into Irish. And then they are applying it to a whole model that they trained on the Chinoy way of singing. And it's in an installation of what looks like this sphere with a line in between of it. It's this like floating sphere. And it is kind of like singing this Shanoi singing. So that's what we're covering on today's episode of the Voices of VR podcast. I should note that I was also progressively losing my voice throughout the course of my interviews at InfoDocLab. And so my voice is a little bit crackly through the course of this conversation. So this interview with Sean happened on Wednesday, November 20th, 2024 at IFFA DocLab in Amsterdam, Netherlands. So with that, let's go ahead and dive right in.
[00:01:38.554] Seán Hannan: Hello, everyone. My name is Sean Hannan. I'm an artist. I'm from Amsterdam here. I'm Irish and Dutch. So those two backgrounds come into play into my work a lot, which I guess we'll talk about a little bit later as well concerning the work I'm showing here at DocLab. My role in immersive art and digital storytelling is... To be honest, so far quite limited. I'm more of what I suppose you could call a traditional conceptual artist. I got invited to show this work specifically at IDFA because I'm working with AI and DocLab is very interested in storytelling through new processes, through new digital processes. So that's kind of my experience with that. I've never made an immersive experience with that in the back of my mind, let's say. It's always just been an installation and the viewer gets to see it and make up their own mind about it, really.
[00:02:29.731] Kent Bye: Nice. And maybe you could give a bit more context as to your background and your journey into the space.
[00:02:35.566] Seán Hannan: So my background as an artist, you mean? Well, I graduated here in Amsterdam in 2009, and I've been working as an artist ever since. My interest, especially after I graduated, I became very interested in what we can believe, what is real, this overdose of information that we're dealing with online, conspiracy theories becoming more and more of a thing. Back in 2009, 2010, I was basically just, I graduated, I had a couple of shows after I graduated, and then all of a sudden you're just an artist sitting in your studio, and you have to decide, well, what's my next step? And I just kind of caught myself watching a lot of conspiracy theories and all of a sudden just had a realization, well, if this is what I'm doing now, I suppose I should start making work about it. So that's kind of how, I guess, what you could call the backbone of my practice, where it emerged from. So it's kind of just out of interest in what people believe. And then from there, when it comes to working with AI, let's say back in 2019, so just before the pandemic, I did a collaborative project with NYU, University of New York, Tandon School of Engineering. And I was doing a project working on deepfake. So coming from that background of wanting to know like, You know, what can these new technologies really do? Are they dangerous? Are they going to, you know, create even more fake news and fake imagery that we don't know to differentiate, you know, that we can't differentiate reality from fake things anymore. So, yeah, I went to New York, worked on a project there. Obviously, a couple of months later, the pandemic hit, so I had to come back here. Took a little bit longer than I was hoping, but at the end of 2021, or was it 2022? Yeah, let's say a year and a half later I managed to finish the project, working at a distance as well and finding new engineers to work with and made a project then that was showed in Rijksmuseum Twente here in Holland, which is in Enschede. Yeah, part of that project was that you basically just walk around the exhibition space. And we had these IP cameras set up. And we're making live data sets of the visitors. So obviously, there was a disclaimer there letting them know, hey, if you go close to these cameras, you might end up being deep faked. But really the entire process was quite straightforward. I have only like four or five VHS tapes from my own childhood that due to circumstances, that's all the documentation I really have. And what I wanted to do was deep fake those VHS tapes with the visitors of the exhibition. So really what happened in a nutshell is if you visit the exhibition and the cameras are made a dataset of you and it reached the benchmark that we needed to make the deepfake. About half an hour later you'd see your own image back in my childhood images. So let's say, you know, we'd make a deepfake of you Kent, you'd see yourself, let's say, sitting down as my father at the Christmas table in 1993. That's it really. And that kind of piqued my interest in AI as a, you know, I don't want to call it a medium because it's not, it's much more than that, of course, but as a field and interest that I could work with, you know, and experiment more with, really. So, yeah.
[00:05:43.223] Kent Bye: Yeah, you mentioned you're interested in looking at conspiracy theories, what's real, what's not real in this project around deep fakes. Can you talk around the inciting incident or that turning point when you saw... AI is a possibility for exploring some of these different things in an artistic way. Was there an experience or technology or capability? What was it that really drew you towards starting to dig into some of these AI bits?
[00:06:07.282] Seán Hannan: Yeah, really. I think I said it all when I started working on that Deepfake project, and that was my first interest. I wasn't really interested in AI that much before then. But then working together, I mean, I don't make my work on my own. I have to say I'm not a great programmer. I don't know how to write code. I mean, ChatGPT has helped me a lot in doing basic stuff these days, but I always work together with it. you know, more knowledgeable minds than myself. So in this case, for the project we're showing at DocLab, I collaborated with Bram van Es. He's an AI engineer. He works for the UMC, which is, I think it stands for Utrecht Medical Center, Medisch Centrum. But it's a collaboration with the Amsterdam Medical Center as well. So mostly his line of work is in using AI to to understand illnesses better and make implementations that hospitals can use to combat whatever it is that he's working on. And the first project I did with a good friend of mine now, Dan Hameskerk, and he's also a programmer, AI engineer. And, you know, seeing what they can do with AI or, you know, being so close to these people and seeing what the possibilities are of what they can actually do with computers was really fascinating to me. Actually, at a certain point, I started doubting my own practice as an artist, thinking, well, you know, in future programmers are going to be artists, really, because... really all you need is a computer and programming skills to make anything. So that was very fascinating to me and it still is and that's kind of how it leaks into my practice and I'm also kind of exploring this space still with every new project I do. I have to say, and maybe that complicates matters a little bit, but I'm also, as much as it's fascinating to me, I also kind of hate AI. I'm a big lover of what it's capable of, but I'm also quite, to be honest, fearful of what it could become in the future. And that duality is something I'd like to explore a little bit more in my work as well. Yeah. I hope that answers your question.
[00:08:06.043] Kent Bye: Yeah, just a quick follow-on, because I've typically covered a lot of VR industry for the last 10 years, and sometimes I'll ask people how they got into VR, if there's an initial experience. And part of that is to see if there's a specific VR piece or a piece of art that inspired them to get into VR. And so part of the intent of that question was also to get at Because AI as a technology is a little bit less immersive in the way that there's videos and concepts and ideas and other technology people that are doing it. So I was just trying to get a sense if there was like a piece that you saw a deep fake and you're like, aha, now I want to do that because now it's capable and I can do that. So I'm just trying to get a sense of one of those turning points or catalysts that really turns you on to really dive into this.
[00:08:47.987] Seán Hannan: Yeah, I get where you're coming from. All right. Okay, so not really an artwork necessarily, but while doing my research into deepfake, I came across a couple of things that I found very fascinating. God, you'll have to forgive me for forgetting their names, but there was a couple of YouTube channels that I also reached out to at the time. that were doing amazing deep fakes. I mean, they were re-editing complete scenes of The Shining, turned Jack Nicholson into Jim Carrey, and it looked so convincing. And not that I necessarily saw that as an interesting artwork in and of itself, but it was something so interesting to me that you could change someone in an existing movie and make it look so completely convincing. You know, coming again from my interest as an artist in what can we really believe? What is all this information doing to our thinking to... to how we treat reality, that was very inspiring. And maybe that was like the moment that sparked it for me. Then there was another interesting thing. The Dutch government here, they, you know, the late Navalny, the opponent of Putin in Russia, he was reaching out to, at the time, like a couple of years ago, to the Dutch government, but European governments in general, but also the Dutch government to try and, you know, form alliances to kind of combat Putin or, you know, come up with strategies to combat what he's trying to achieve. And one of his head guys in his organization, his name was Volkov. They had a meeting with him. This was during a pandemic as well. And they did this Skype call or whatever they did, like Zoom. And this guy wasn't Volkov at all. This was just a guy pretending to be him. But they passed it off as it being a deepfake, because deepfake technology was kind of coming up. So I found it very fascinating that the Dutch government got duped into talking to this actor guy. Literally, if you saw him on screen, he was wearing like a robe and had a big mug. I think it even said, I don't even work here or something like that. It was completely obviously fake. But they were so embarrassed by that, and they knew this deepfake technology was upcoming and it was a big thing. that they said, well, you know, that was a deepfake. You know, we got fooled by a deepfake, which is better than saying, you know, we got fooled by, we didn't do enough background checking. This guy wasn't actually the guy that he said he was. Let's just call it a deepfake. Obviously, when you're working with deepfake technology, as we were doing, just started getting into at the time, you know, doing my research with a university in New York as well. Especially back in the time, you couldn't make a live deepfake and you still can't really. So there was no way it was a deepfake. So maybe that was fascinating to me as well, like how from the other side then, You know, even from the upper echelons, the top levels of politicians and government, this was being misused, you know, on the other end of the spectrum, let's say. So I didn't really see an artwork, but just several videos and certain events that were taking place that inspired me to do something with that and see how I could... in that specific piece put some kind of skin in the game in a way as well like say right what happens if I deep fake the only existing video documentation that I have left of my own childhood like how does that make me feel how does it also make you feel as a visitor you know when you kind of become complicit in erasing physical tapes and altering them forever so yes there was that extra layer that I found interesting to kind of explore
[00:12:03.160] Kent Bye: OK, so that puts us up to around the end of 2021 or 2022. And so you have the project here. Was there anything else that you did before this project? Or did you go straight into this as being your next project of AI?
[00:12:14.485] Seán Hannan: Well, this is the next big project with AI I did, I think. I did a couple of different things. I worked on pub mirrors that had AI design new pub mirrors based on a bit of a painful chapter of Irish history. The Irish used to be called monkeys by the English. I mean, this is back in the day, guys, don't worry. But I found that kind of interesting, so I wanted to explore what AI would think of that and what it would come up with if we'd make pub mirrors based on monkey drinks. So those pop mirrors I actually made physically then afterwards. So basically I designed most of it. Then concerning AI, I made some videos as well, which are just simple generative AI videos. I'm going to deal with certain concepts. In between that, I also cursed an egg, which has nothing to do with AI, so I kind of move in and out of this space, really. Whereas now I'm really back into exploring it again with this specific project.
[00:13:06.306] Kent Bye: And how did this project begin? What was the inciting incident for you to start to dig into this as a topic?
[00:13:11.753] Seán Hannan: Again, this is like, I think for me, what really sparked my interest in voice AI or voice deepfakes was a thing when I was even working back in 2021, working on a deepfake project. Voice deepfakes were becoming a thing. So this was, you know, in the news, oh, you should be careful if you get a phone call and it sounds like your daughter dies. than it might not be because it might be a voice deepfake and they're trying to get your bank information or whatever. People trying to scam you into doing things. And I think this wasn't even that long ago, let's say last year or a year and a half ago, there was a song released called Heart on My Sleeve, which was supposedly sung by Drake and The Weeknd. and it was completely AI-generated. I mean, obviously a lot of post-processing went into that, it's not just an AI that came up with the song, but it wasn't Drake's voice, it wasn't The Weeknd's voice, and it was released on Spotify, and I think it got like 20 or 30 million views before anybody even realized that it wasn't them, and it was a really good song. You know, as far as Drake songs go and The Weeknd songs, it was actually really good. So I guess that really got me more interested and more enthused to work with voice and do something with song and AI as well.
[00:14:18.921] Kent Bye: Great. So maybe you could just give a bit more context for this project and where you began and starting to kind of piece together and where you ended up.
[00:14:27.301] Seán Hannan: Yeah, sure. So really here with this project, I started researching Shannós singing. Shannós in Irish, not like an Irish accent like you might hear me having now, but the actual Irish language, which I speak like five or six words of. But it means in the old way or the old style, really. It doesn't really translate to a style of singing. It encapsulates more than just singing. It's also dancing. It can be loads of different things as long as it's in the old way. And Shannós singing, for lack of a better, more appropriate term for it, was a form of singing that originally was intended to record history. There wasn't an awful lot of written history in Ireland like eight or nine hundred years ago, so before the English colonised Ireland. Everything was kind of transmitted orally or almost everything. So this was a way of keeping local stories alive. You know, storytelling is a very big part of Irish culture and telling the stories, retelling them and passing them on from generation to generation was the way of, you know, yeah, saying, well, this is important to us. We need to remember this. And I found that fascinating as a subject in and of itself, but also to do something with with A.I., So what we did in this project, me and Bram van Ness, we took whatever we could find on Shannos singing and made our own AI model. So we fine-tuned an existing Irish speaking model to start singing Shannos.
[00:15:54.954] Kent Bye: Just a quick clarification, because there's Shannos, there's Irish, there's Gaelic. Is Gaelic the same as Irish language, and was the language that you were training in, was it in the original Irish or Gaelic?
[00:16:05.721] Seán Hannan: Yeah, so it's always a bit confusing. Irish is actually a language, it's called Irish. Gaelic is actually the name for a family of languages. So you've got Scots Gaelic, you've got, jeez, I don't know, is it Welsh Gaelic as well? I don't even know, to be honest. Gaelic is more of a family name of languages, whereas let's say, I don't know, like Dutch would be a Germanic language and so would German, but so would English be as well actually, or Saxon at least. They're all from the same kind of language family. So is Irish a Gaelic language then or not? Exactly, Irish is a Gaelic language, but Irish, is Irish. It's not called Gaelic. But it gets more confusing because in Irish, Irish is Gaelge, which translates to Gaelic. But let's just, for the sake of the argument, just call it Irish. And yes, it's trained in Irish. So all of the songs that we put in our model as well to train the voices on, to train the singing on, are all from the Irish language, yeah.
[00:16:58.412] Kent Bye: Okay, okay. And so in your training of the model, because part of the idea, I guess, is that it's singing about the current events of the day, and so you have the shanos, which is the songs that were sung that were reflecting to the time of that day. So how do you update it to be able to, what do you train the model on in order to actually have it kind of sing about what's happening now?
[00:17:24.972] Seán Hannan: Right. Well, really, Shannos singing hasn't changed much. I mean, again, there really aren't that many new, as far as I'm concerned, at least as far as I know, there aren't any new songs really being written. There might be a handful of people that do it, but it's, you know, it's not as important as it used to be culturally. There's, of course, a lot of Shannos singers still that sing the old songs and keep it alive in that way. So they keep the tradition alive. So we base the singing on what is known on Shannos. So, you know, everything we can find really, hey, this is Shannos, this is an Irish, this is how to sing it. It's a specific way of singing. I don't mean to deviate too much from my line of Tata now, but there's also different ways of singing Shannos depending on which region in Ireland, you know, the songs come from. I mean, for instance, Munster Shannos or Connaught Shannos is known for more ornamentation in the singing, whereas like Other regions might have less ornamentation in how the voice is used. All that aside, yeah. What was your question, Ian?
[00:18:21.210] Kent Bye: Oh, because, so the way it was described to me by the curators when I had the conversation was that there's this way of transmitting news and stories about what's happening by the Shinnos, and so I guess I'm wondering if what we're hearing, if it's just based upon a remix of those existing versions models of what's trained on or if it's trying to somehow scan the news of what's happening in the world today and trying to update and do a mashup between the news of the day with the data set or if it's just purely trained on chinos and kind of remixing from that
[00:18:57.111] Seán Hannan: Right. No, actually, as we speak right now, it's making new songs based on things that have recently happened. So it does scan the news, as in news sites or newsworthy things, which could be about our neighbor as well. If somebody put it online and it decides, you know, we build that into the pipeline to sing about that person or sing about that event. So really how the pipeline works is that first, you know, the query is we want you to sing a song about us, something that you think is important to remember about us as humans. So then it goes through all the news, all the things that go on in the world. It says, right, this is the thing I'm going to write lyrics on. So then it writes lyrics. Then it translates those lyrics into Irish. Then it goes through our model, so it starts singing the lyrics. Then you get your output. Output goes through a little bit of post-processing, and then really it's played, and it does that constantly. So it's constantly looking for new things to sing about. Really, you know, I was thinking about this this morning before we did our podcast. Like, what is important about this work for me? Like, why do I want to do this? You know, looking at my own background as well, you know, obviously... Working with my own background, being Irish is a thing, but also why do I want to re-instigate this almost dead language and almost dead way of singing to sing about us now? And maybe that has to do with what I said earlier, that I have this fear that I might, you know, like, let's just say my P-Doom is quite high.
[00:20:23.084] Kent Bye: It's the probability of everything going wrong and everyone being destroyed. P-Doom, probability of doom.
[00:20:28.708] Seán Hannan: Extra risk P-Doom. There's all these terms. Yeah, so the probability that AI in some way is going to be malicious towards us in the not-too-distant future, I personally feel like it's quite high. I'm not trying to scare you, but it's just something that I've been struggling with in a way, especially as an artist working with these technologies as well then. So maybe in a weird way when I started this project, I already had these kind of fears and these realizations of, well, this could really become something that... might be a problem for us in the future. Maybe I just wanted to kind of take a head start and say, well, if we're not going to be there anymore, then maybe we should have a computer start singing about us, what we're doing now, because it's going to have to be remembered somehow. Maybe in some kind of weird romantic notion that I had about that, that's what the project is to me. It's taking a head start at what possibly could go wrong.
[00:21:19.997] Kent Bye: Okay. That's helpful to hear different parts of that pipeline. I have a few follow-up questions just to kind of clarify a few things. So when you said you were training the model, were you inputting text or were you also having the singing that was coming along? And I'm, I'm trying to, cause you know, we're all familiar with like, I guess, large language models that are very much a corpus of text, but you know, we also have multimodal large language models that are also doing. voice as well as video and so yeah just curious with the model that you trained if it was feeding in just a lot of Irish text of the snow songs or if there's also audio that was being a part of that model yeah sure so there's two components to that mostly the model um this is um and I have to be careful what I say here because again I'm not the AI engineer that's all Bram so maybe you should be interviewing him instead concerning this question but
[00:22:07.296] Seán Hannan: And we work with a Vitz model. And basically, it consists out of almost 10,000 samples of that we could find. So not full songs, but samples, so you know, cut-up songs. So audio or text? Audio and text. So you need both to train the model to understand what is it supposed to do with the text. So on the one hand, you have this is you know, the sound bite, let's say, or the sample, which is generally speaking about seven, eight, or nine seconds long. And then next to that is the sentence that's actually being sung. So you train it on understanding what it is that it's phonetically, how to phonetically pronounce the text. We did this based on an existing model that already spoke Irish, because if we didn't have that, we wouldn't be here now. I mean, it would take so long and so much research and energy. So we took the existing language model, which was already there, or the text model, the speech model that was already there, and over time, slowly but surely, got it to understand, hey, I can sing. I can change the way I'm supposed to pronounce this word into, like, you know, that this A should be longer because that's how you sing. And that's kind of, you know, started training itself to do that. And with each iteration of our model, it became better and better at doing that. Having said that though, I mean, we had earlier models which were just synthesizing voice and using it to, you know, basically we could take your voice and have it sing an existing Shano song because that's easier to understand for a model, right? So if you want to just sing the same song over and over, that's quite simple. Well, it's not simple, it's still, you know, it's amazing that it's possible, but... What our model really does is create its own songs. So, you know, the one's going to be better than the other because we have no control over how it decides to sing it. It just understands, hey, this is kind of what Shannos does. So a lot of factors come into play, like how long are the sentences? What are the sentences about? Do I understand these words even? So based on all those things, it really differs on the quality of your output. But that's part of the project as well. I mean, accepting that it can't be perfect, at least not for us.
[00:24:10.416] Kent Bye: But that's kind of part of the charm at the same time. Okay. And so that's the model that's translating into the chinoise at the end. But then at the beginning, you said it was kind of scanning different parts of what's happening in the world today in terms of the news. And so maybe you could elaborate on those news sources and if it's updated every day and it's trying to reflect on what's happening in the moment or if it's very specific to Ireland and the Irish news. And so maybe just expand on the corpus of data that it's ingesting in terms of trying to comment on the news of the day.
[00:24:41.468] Seán Hannan: Right, so yeah. So that was actually the other part that I kind of skipped over. Yeah, and then there's this written language component to it as well. So yeah, how it comes up with its lyrics is it does scan for news. It scans for things that have been posted online. Where is it looking? We gave it several categories. Again, this is a question that's better answered by Bram because he built this pipeline. So I'm just kind of answering what I know about it. And I know how it works. I just can't reproduce it. But it's looking on different websites, really. And it's looking for categories. So we gave it a couple of categories. One would be Ireland. Another one would be culture. Another one would be sports. Another one would be politics, biology. So, you know... basically asking it, look at these specific categories and then make a selection for yourself and then come up with a decision of what you think is the most interesting thing. That sounds like really heavy, like as if it can actually think, but really what it does is say, well, I find this more valuable than that and this is what I'm going to sing about. Which is kind of funny because then you end up with text, like one of the early texts that generated was about the Delta IV heavy rocket that was decommissioned, which was a beautiful piece of poetry, especially coming from a machine. It was calling it its brave friend going to the stars and how it'll miss it. It's kind of touching. But most of it is to an extent also. AI can only go so far. It doesn't actually really think in that sense. At least ours doesn't. I'm not sure about ChatGPT. So it kind of makes a random selection to an extent, but we do ask it to look for certain subjects and choose on what we think and then it thinks is relevant.
[00:26:19.837] Kent Bye: Gotcha. Okay. So that's a little bit of the AI pipeline that's ingesting everything and then creating at the end of it this Shinnoh song with this synthesized voice that We've actually been able to hear in the background here during this conversation and probably almost all my conversations here at InfoDoc Lab this year. But there's also an immersive art installation component to this piece. And so maybe you could talk around what you were thinking in terms of how you're actually going to display this piece of art.
[00:26:48.072] Seán Hannan: Right, yeah. I really wanted it to be something that felt like a presence in the space, first of all. So, I mean, obviously this is all digital, you can put this on a speaker and it'll still basically do the same thing, but I wanted to give an artwork experience, something that would be more than just hearing something, but also maybe create a presence for this voice, say this is the voice almost. so yeah i mean and that was an interesting process because that's like sidelining the entire thing right so on the one hand you're working with this difficult complicated ai pipeline that needs to work and then at the same time we're thinking about well how do we display this so all the way back to now um what you see is quite a large golden sphere that's cut in half hanging from the ceiling so it gives this kind of sense of being suspended or floating in a dark space with a light on it and then there's a light coming from underneath it and then also the split in the middle is lit up with a neon led strip so it kind of creates this holy presence almost something that you might weirdly see in a church but you've never seen in a church but it kind of feels like it's something holy And that really came from, you know, back when we were starting the development of the pipeline, I thought we're working on an AI project, so we might as well incorporate AI all the way through. So I started asking the DALI model, which was back then still separate from ChatGPT, but asking questions and asking it to come up with generations of what. something or a sculpture i think i asked it what is a sculpture that sings forever looks like so what is a sculpture that sings forever what does that look like and it kept giving me images of giant spheres in a space and like i think i mean sometimes it came up with something else but a lot of them were spheres so you know going through everything that it came up with let's say i generated 100 images i'd say like at least 80 of them must have been some kind of a sphere in a space or a sphere outside or spheres spheres So I just decided to go with that, and then delving a little bit into my own art history as an artist, I realized, hey, hang on, there's this painting that I really love, and especially when I was an art student still, by Magritte called The Voice of Space, and it's basically just a landscape, and there's three white spheres with a split in the middle, almost like bells, floating, and they're giant, they're massive, and they're floating above this landscape, and I remember seeing it in Guggenheim in Venice for the first time in 2005 when I was still studying in Rietveld. And it really struck me that this painting almost made a sound to me. I could understand the title so well. It was like the voice of space. I could hear this hum almost coming, resonating from this painting. You know, I just added one and one together and said, well, it's supposed to be a sphere in space, apparently. And then there's this painting. So let's just go for one of those spheres from a great painting. That's what I chose to make. It made sense.
[00:29:39.426] Kent Bye: Yeah. Nice. So throughout the course of my time here at Vodok Lab, I've been checking in periodically to there's that little QR code that's a part of your installation that you can take a snapshot of and then pulls up a website that gives the live lyrics of what's being sung at that moment. And then, you know, so I've done that a number of times, and I'm wondering if we could either look at one of those, or maybe we could walk over there and see what it's singing right now, and just have you read through it and comment on it, because I'd love to do that. Okay, let's walk over. So we're walking over. Maybe I'll record it just for a second here, just to give people a little sample.
[00:30:18.790] AI Installation: Okay.
[00:30:34.243] Kent Bye: OK. So you got the QR code, and you're reading it. So maybe you could just read that for us.
[00:30:38.994] Seán Hannan: From the sun the journey stretches in the hearts of the people there's the burden a new leader with the heart of youth Chris Evans fighting for freedom hope of the people in the captivity of night force of the wind coming together life of the young with the sorrow and breaking and the form of the dead moving gently Look at the actions coming together in a fight against the beautiful, relief for the people who are in captivity, who recognize the authority with the heart of youth. Sorry, I lost my track here. In the heart of the sea. Oh, we just kept to a new song. Sorry, so we're on a new song. That's as far as I could go. Now we're at, oh, in the heart of the sea, that is the tale, a little isopod swimming with a brightness and pale pentaceration. You're an engineer. What does that say? Pentaceration?
[00:31:37.017] Kent Bye: Yeah, pentaceration here.
[00:31:39.491] Seán Hannan: Fork and Brewer. There you go. A name clear from the harbor in Kilmenau. That's in Ireland. It is coming near. It is. With the scientists in New Zealand, it can be seen. And the water is flowing with a beautiful scene. Treasure of the ocean, the creatures in the world. And the river is cleansing. with every wave unfurled. Oh, as the material is being dissolved, it is the dream that keeps life moving forward in the heart of the sea. The connection is strong with the little creatures. The joy is where they belong. As the waves come with the sound of the wind, they signal that life is ever growing. It is the dream. It is the way. Pentaceration, Fork and Brewer. Interesting, I have no idea what that says. In memory, it will stay.
[00:32:28.462] Kent Bye: there we go nice okay let's let's go back over here um so that chris evans ones actually got a screenshot of that because that came up earlier so you said it's a new song every half hour but it's been singing for a while so does that is it just like continuing to sing the corpus of songs that it's generated throughout the course of the time here Yeah, really, that's what it does.
[00:32:47.421] Seán Hannan: So every new song that's generated ends up in the same file. And I think, like I said earlier, I'm not sure if I said a journey interview or not, but it creates a JSON file, which has a translation of the text. And then it has the generation of the song itself as well. So those kind of end up in the same file, and the JSON files are read out by the web server, which we use for the translation so you can see the English text. But yes, every new song that's generated gets thrown in the same file, and that file is used by the player, used by the object, really. So the Chris Evans one can come back every now and again. What do you want me to do?
[00:33:22.338] Kent Bye: Keep writing? I just wanted to complete the thought just to kind of get the end of that first one that mentioned Chris Evans because that was one that I had taken a screenshot of it before. So just pick up from that and then we'll talk more about it.
[00:33:34.323] Seán Hannan: I don't know. How familiar are you with what Chris Evans does?
[00:33:36.764] Kent Bye: Because I'm not very. It's just that he's an actor who's like in the Marvel series and stuff. So I don't know if he's Irish or why he's coming up there or, you know.
[00:33:45.828] Seán Hannan: So then, honestly, I think what our pipeline did here is saying, well, there's this, probably there was this article about Chris Evans being in a new Marvel movie fighting for freedom. So maybe it took it very literally and taught that Chris Evans was going to save the world. I don't know. Maybe he is. We're going to have to Google that one. I don't know.
[00:34:02.902] Kent Bye: so maybe we'll just go from there into like what are some of the other themes that you've seen in terms of like you know because it's essentially we're we're throwing a bunch of stuff within these ai models and kind of alchemically mashing them all up and then kind of seeing what it spits out so what are the types of patterns or things that you're seeing in terms of like taking these old ways of singing with this contemporary information playing it all together and then what are the things that you've seen in terms of the patterns Right. Yeah.
[00:34:28.513] Seán Hannan: I mean, there's a comical component to this as well in a way, right? I mean, saying that Chris Evans is going to save the world and then writing a text about that. I mean, it's kind of endearing in a way. Like, you know, you train this AI model to take it super seriously and then you read what it's about and you're like, oh, all right, that's what you chose this thing about. I mean, if it were real, it's probably very serious. There's another one, if I can just name another example, which I found very funny. A couple of days ago, it started writing a song about Claudio and how he was going to make Rome great again. And that kind of confused me, because I know that our pipeline is supposed to be singing about contemporary times. So I thought, well, why is it singing about Emperor Claudius in Rome? But then, after googling, I figured out that that's the name of the new coach of the Rome football team. Soccer, I should say. Sorry. Soccer team. So, yeah, there's funny things like that. Really, the patterns that you see is, you know, we take these, you know, these newsworthy things and the model then translates them into poetry. Really, the segment that we use to make the poetry with loves certain words, I think, or it has a certain way of, you know, writing poetry, as far as you could call it that. And then I think something happens in the translation then as well, because Irish is quite a difficult language. It's hard to translate from English into Irish or vice versa, really. So certain words become almost like abstract notions of things. Like it loves saying the word lovely. It loves saying the word beautiful. It loves saying the word crown, which I think it kind of uses for the word head. or mind. So, you know, there's patterns in how it treats the text and the way it sings. As far as I've heard so far, every song is quite different. There's the same way of singing it, but that's, you know, because we trained it how to sing Chanos and Chanos kind of the songs don't differ that much from each other. I mean, obviously one song is different from the other, but there's a way of wailing almost, like singing. There's another part of Shanno's singing called keening, which is almost like crying in singing. So it does that quite well, but it comes at an expense as well sometimes of it losing its voice a little bit. Like you were telling me, you lost your voice a little bit. So really our model can do that too sometimes, depending on how complex the text becomes. It gets this robotic undertone. If it has difficulties understanding certain words or can't translate them, then what is it going to do with that? So it becomes this robot almost, which is an artistic choice that we made as well. Because again, we could have just synthesized the voice and have it sing the same song over and over. We chose not to do that and have it be a unique voice and a unique way of singing every song. And in that sense, I mean, everything's really a pattern in it. But it does create quite a bit of variety at the same time.
[00:37:11.647] Kent Bye: Yeah, one of the things that my experience of actually experiencing this piece that I found was that I walk up, there's a big globe. It's singing in this cryptic language that I don't quite understand. And the piece is beautiful just to look at. But then to kind of like understand what it's saying, then I scan a QR code and then I end up like standing next to it and kind of reading my phone. So then it becomes more of like an experience where I'm listening to something that is kind of in a different language. And I'm just kind of staring at my phone at that point. And so I'm wondering if, you know, like I experienced this a lot in like mobile AR pieces where it ends up being a screen-based experience. And I think, you know, part of the affordances of immersive art is that you kind of get away from that in a way that I don't know if you've thought about other iterations or things that you might continue to develop this as a project where you don't have to have people look at your phone that you can still kind of be absorbing in and you know, there's going to be some cryptic stuff you don't always understand. And you're like, you know, there's the meaning of what's being said. And but if there is a way to kind of more closely tie to the sounds that we're hearing to then the words or I don't know, it's always it's kind of a larger challenge of language translation. And like from one language to another is always going to be a little bit difficult that It's kind of a problem that's not fully solved in any context. So given that, I'm wondering if you've thought of any other ways of making it so that people don't feel like they have this split experience where it becomes more of a screen-based experience than something that's a piece of immersive art that they can completely surrender into.
[00:38:40.441] Seán Hannan: Yeah, I completely understand your question, and the honest answer to it is I don't have the answer really at this point. I mean, I wanted to make this work, and really, I hate putting it that way, but the QR code that you can scan with the translation, it's more of an afterthought for me, because for me it was the most important thing for you to hear this thing sing, and just experience it in a way you want to experience it, and it didn't really matter on a conceptual level, you know, being somewhat of a conceptual artist, that you couldn't understand it. Again, going back to I find it important that it's actually doing it and keeps on doing it as an act of a machine singing about us. To me that's a very romantic notion and to me that's what the piece really is about. And I agree with you, what ends up happening if you scan the QR code is you get the translation and you end up looking at your screen and listening while you're reading and you don't really pay attention to the object itself anymore. That's why I think this is the best option I could come up with, rather than having a screen there, because then automatically you're going to start reading the screen, whereas here it's optional, you don't have to do it if you don't want to, if you just want to listen and say, fine, I don't care that I don't understand what it's about, but I get the concept and that's good enough for me, you can also choose to do that reading. But it's a difficult one and something I'll definitely think about and I'm happy to hear your experience with it as well because I had that feeling too, like it becomes a screen-based experience, even though it's not meant in that way, really. I mean, I like it as a component to the piece, but I'm not sure how to solve it.
[00:40:05.993] Kent Bye: One other solution, which is probably not a great solution, because I think it goes against the thrust of what you're doing, would be to have it singing English so that people could hear what was being said and sung in the same style, but rather than in the Irish language, it would be the English language. But I think there's a certain part where I don't like that as a solution in the sense that there's a certain way that languages are being lost because of the driving towards the kind of colonial nature of English as a language that is kind of having these different languages go extinct or go dormant. And so there's a part of the cultural history that's preserved within the context of there. But that would be one solution for it to be in English that people could understand what was being said without having to mediate it through screens.
[00:40:48.888] Seán Hannan: I think it's a very good idea in a way, but yeah, you're absolutely right. I mean, the fact that it's sung in Irish, to me, is important in the context of the work, because, you know, as Shannós is a dying form of singing, you know, to an extent, I mean, it's kind of reviving a little bit now, but the Irish language almost died out, you know, over 800 years of... being part of the UK, we weren't allowed to speak Irish. So you were only allowed to speak English and not allowed to speak Irish. So that obviously did a lot to the language and to, you know, this language is part of Irish culture, of who the Irish people are, of how we express ourselves. And to me, that's why it was important. You know, it's kind of interesting. If I do another iteration of this work, I might consider making another sphere that does a translation in English, maybe in a different or adjacent space, so that on one side you have the Irish, and then maybe on the other hand, we'll make it a silver ball, why not? And then, I don't know, maybe a little bit smaller or something. That might be a good solution, something to think about, for sure, yeah.
[00:41:49.653] Kent Bye: I would be curious to just to hear like, cause it's singing Irish right now, but to hear it singing the songs, but with the English language and if it, if the model would be able to even do that functionally because it's trained on that, but if it can take the input of the English and if it would pronounce it in an Irish way and it would just be unintelligible, it's just a more of a theoretical question in terms of an AI model trained on another language. If you try to have it speak in a different language, if it would still work at all.
[00:42:17.153] Seán Hannan: Right. On this specific model, it has some English words, and because you can't translate everything into Irish, especially names or certain things, it won't actually have a translation in Irish. So we'll come up with something closest to it, or we'll just leave it as an English word, which I think happened in the text we just read. It said something like... or something. I don't remember exactly. But I think that's not a proper translation because it just didn't have a word for it. But it does, because it's trained on an Irish model, it will pronounce it as if it's Irish. So I think there's a name in there somewhere, Marcia or Marci or Mark maybe even, just Mark. And it pronounces as ,, which is just because you say it like that in Irish. Irish is a difficult language. It has 18 letters in the alphabet. I mean, it uses combinations of different letters to make certain phonetic sounds, which are really very consistent throughout the language if you look at it. It's more consistent than English is, really, because you can have an English word, you write it the same way, but it can have three or four different kinds of ways of pronouncing it depending on the context. You won't find that much in Irish. it then does become difficult because certain letters are pronounced differently or certain groups of letters are pronounced differently depending on how they're structured you know grammatically in the sentence so yeah english words will kind of sound weird still we'll have to train a new model but we can train an english model to sing shadows i mean sure why not i think i'm not sure maybe yeah well i think i think it's
[00:43:45.154] Kent Bye: Because you said you're a conceptual artist and that the QR code was a little bit of an afterthought, my experience of it was mediated through the QR code, so it's difficult to know what my reaction would have been in the absence of that. I might have had a completely different take on it in terms of just kind of appreciating what was there. I think I would have wanted to know what it was saying and the meaning just because conceptually it's interesting to kind of be scouring for the news and I would want to know like, okay, what's the AI? What is it seeing as it's important through this model? And so Yeah, I don't know. I think it's just something that's it's hard to explain what my reaction would have been had you done it a different way. But as you move forward and go forward with this project, where do you want to take it next? And if it was kind of living up to your ideals for what you started as a conceptual artist, if it was kind of like fully expressing, exploring this concept that you wanted to start with and then how it ended up.
[00:44:35.723] Seán Hannan: Just to reply to what you were saying, just before your question, when I was installing the work, the piece, I sent a picture to a friend of mine, and I mean, I think it looks great, but I just sent it just to show him this is what it looks like, and the first response he gave was, the first thing he said was, oh dear, nobody's going to see the amount of work that went into this, because you don't see what's going on, you know, behind the screens, it just looks like a ball that's singing, and it could be, but you know, you're going to have to take my word for it, it's coming up with the new songs now, but you don't know that as a viewer. Where do I want to take this from here? I mean, this is the first time I'm showing it. This is the premiere of the work. Again, I think my last show on the level of IDFA, let's say, was in the Stedelijk Museum here in Amsterdam a year and a half ago, two years ago almost. And then I did some things in between. But I'm more familiar with that kind of a context, so I'm hoping to show it after IDFA in a museum somewhere as well, maybe refine it a little bit, you know, having the experience of having it run for over a week now at IDFA, you know, you learn some things about it, but generally speaking, I'm quite happy with the work. Where it's going to go from here, I really don't know. Hopefully, I'd love to go back to the States, so if you have any connections for me there, let me know. Other than that, we'll see.
[00:45:42.468] Kent Bye: Really?
[00:45:42.869] Seán Hannan: Yeah.
[00:45:43.931] Kent Bye: Great. And finally, what do you think the ultimate potential of this type of immersive art and immersive storytelling might be and what it might be able to enable?
[00:45:53.942] Seán Hannan: That's a really, really big final question. And I'm going to have to tie it back into what I said earlier. I think we're, I'm really trying, I'm sorry, the Irish in me is trying to pepper my words with curse words a bit. I'm trying to avoid curse words, but it's a really difficult question. Depending on what you want to believe, I guess. It might just take us over completely. We might not be needed to create immersive storytelling in the future. I think that might be a while off, but I think that's where we're heading. Everything can be AI-generated, really. I like generative AI. Let me just make that clear. I really like it. I like it as something to work with. I like what it does, what it represents in a way. So I'm not afraid of generative AI. I'm more afraid of the thinking capabilities that it might gain in the future from large language models or whatever. Jesus, I'm losing my train of thought here. I think anything is possible in immersive storytelling with AI, and it's going to get better and better. And obviously, we're at very early days, right? So let's see where we are five years from now.
[00:47:07.000] Kent Bye: Yeah, it reminds me of that famous quote that someone made on social media that said that they don't want to surrender their artistic practice and creation of art to AI so that they can do more laundry, that they want to have the AI do the laundry so they have more time to create this art. So I feel like there's this... colonial impulse for the way that there is so much existing data and art that is available to be trained on that that happens to be the first thing that's taken away. I think it's coming to festivals like this where I can see how artists are starting to embrace the generative techniques to not only start to explore the potentials, but still have at the core a reflection of their own human experience in a way that feels like they're able to preserve their own artistic practice. And I hope that we preserve that. I'm less of a AI is going to take over everything, but I think that's the default in terms of like the structures of capital and these companies that have an interest of doing that because there's so much money to be made. But I think at the end of the day, it could be taking away our creative expression and process of storytelling and As of right now, the storytelling's not really all that great coming from AI, and there's a lot of critiques that are also coming from critiquing the quality of the art that's coming from AI. Even a NYU professor critiquing the banality of some of the poetry that's coming out of AI. So I feel like there's still a place for the human connection and the human experience. And that I'm not, based upon the work that I've seen so far coming from pure AI, I'm much less interested in that and what other humans are creating with the tools. And so, but that may change in the future. But I have more optimism and hope that things are not going to go down that really dark path. So anyway, I don't know if you have any thoughts.
[00:48:52.776] Seán Hannan: I'm really with you on that line of talk. I mean, I hope that, you know, when you look at my piece here, that's something that jumps to the back of your mind as well. I'm not trying to replace humans. I'm a human making something with AI and really using it as a tool in that sense to say something about the world we live in today, right? So whatever you take from that, I mean, that's still a very human experience. It's about you. It's not about the AI. The AI is not trying to sing for you. Well, it is singing for you, but it's not singing in your place. So, you know, you can take from that experience whatever you feel like is important, is worthwhile. But yeah, I mean, I agree. I hope we're all going to need humans, you know, backpedaling on my last answer a little bit. I do think, I feel, this is an interesting evolution of a technology because I think it's, stop me if I'm wrong, you might know more about that now, but I think it's the first one we've seen in human history that is kind of cutting straight through the middle. It's not, you know, replacing workers in a factory or, you know, taking manual labor, like I think that quote that you were I think the quote is, I want my AI to do my laundry and dishes so I have more time for my art. Something like that. Yeah, exactly. Which is very true. At the same time, I think what it's going to do first and foremost is make sure that you have more time to do your laundry and dishes, period, because it's going to do other things. So that's where we're kind of heading into now. I don't know where we're going to be five, ten years from now, and it's a different story. But yeah, it's kind of cutting through the non-physical labor first, right? So management and HR things, I mean, everything's just going to become automated by AI to a certain extent. And yeah, I mean, maybe it's a good thing. Maybe we're going to have a new appreciation for people that do physical labor. We're going to look at them differently, like, wow, you're really important. You do stuff that a computer can't. I'm hoping for that kind of a revolution. But we're always going to need humans. And I hope AI will see that as well and will agree with that. But who knows?
[00:50:51.658] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:50:57.923] Seán Hannan: No, not really. Pleasure to meet you, and hopefully see you in the future. And if you want to look up my work, I have a website. I don't know if you placed that somewhere or not. So it's seanhanan.nl, N-L for Netherlands. So seanhanan, S-E-A-N-H-A-N-N-A-N.nl. And follow me on Instagram, I guess, if you want to. I'm really horrible at this kind of stuff. Sorry. Nothing more to add. That's it.
[00:51:25.905] Kent Bye: Great. Well, Sean, thanks again for joining me today to talk about You Can Sing Me On My Way that was premiering here at IFA Doc Lab. And yeah, I really enjoyed having a chance to break it down a little bit more to hear not only a bit more about your process, but all the things that are going on behind the scenes that we can't see on exhibition here, just to see where this type of AI and voice generation is at right now and some of the things that you're interested in exploring within the context of this piece. So yeah, thanks again for joining me here on the podcast to help break it all down. So thank you.
[00:51:54.216] Seán Hannan: You're so welcome.
[00:51:54.736] Kent Bye: Thank you so much. Thanks again for listening to the Voices of VR podcast, and I really would encourage you to consider supporting the work that I'm doing here at the Voices of VR. It's been over a decade now, and I've published over 1,500 interviews, and all of them are freely available on the VoicesofVR.com website with transcripts available. This is just a huge repository of oral history, and I'd love to continue to expand out and to continue to cover what's happening in the industry. But I've also got over a thousand interviews in my backlog as well. So lots of stuff to dig into in terms of the historical development of the medium of virtual augmented reality and these different structures and forms of immersive storytelling. So please do consider becoming a member at patreon.com slash voices of VR. Thanks for listening.
