Elele is an Venice Immersive experience that was selected as a part of the Biennale College Cinema selection, and it’s a quite compelling embodied experience exploring that transforms your hands via hand tracking into a spatialized living sculptures that dynamically transform as you move your hands. It’s really quite a hypnotic and compelling experience, and I actually experienced phantom touch during one of the surprise twists at the end. I had a chance to talk with the director Sjoerd van Acker, who may be one of the first XR native creatives that I’ve spoken with (i.e. that they’re not coming from film, video games, or some other domain). He talks about his inspiration for creating this as a non-material gift for his cousin, and stumbled upon something that people didn’t want to put down. He then got into the Biennale College Cinema program, and won one of the grants. He highly recommends applying as he was able to get a lot of great feedback on the piece. It’s a short and simple piece, but extraordinarily powerful and compelling, especially as I was able to have such a sense of embodied presence as to invoke my first experience of phantom touch. To that end, hand tracking has the potential to help people get deeply immersed within VR.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that's looking at the structures and forms of immersive storytelling and the future of spatial computing. You can support me on Patreon at patreon.com slash Voices of VR. So in today's episode, continuing on my coverage of looking at different pieces from Venice Immersive 2022, today we'll be taking a look at a piece out of competition called Alele. So this is a part of the Biennale College that was a special program that they have to be able to bring in different projects and have artists and creators help mentor, but also they have different grants that provided. And I think this is one of the winners of one of the grants and They also show a number of different pieces each year in a special Biennale College selection. So, Lele is a embodied experience where you're looking at hand tracking and taking hand tracking and using your hand movements, but translating it into this radial sculpture-like object. And so, you can imagine like a pot, but it's made out of your hands. And so, as you move your hands, then you're able to have this dynamic moving structure. And there's other twists in this experience. that I think actually led to me having my very first experience ever of phantom touch. And so having such a deep embodied experience that having some sort of interactions that make me believe in my mind that I'm physically there and that I should be expecting something from haptic feedback. But yeah, so I think it's this whole dimension of haptic touch that has been, you know, people who do full body tracking within the context of VR chat to invoke this sense of virtual body ownership illusion. And once you have that, you have this sense of phantom touch. The first time I ever experienced it, so I had a chance to talk a bit about it with the creator, Sjoerd van Acker. So that's what we're covering on today's episode of the Visited VR Podcast. So this interview with Sjoerd happened on Sunday, September 4th, 2022. So with that, let's go ahead and dive right in.
[00:02:06.761] Sjoerd van Acker: I'm Sjoerd van Acker from the Netherlands, and I have my piece here called Ellele, which won the grant from the Venice Biennale College Program. Yeah.
[00:02:20.894] Kent Bye: And maybe you could give a bit more context as to your background and your journey into making VR and immersive art.
[00:02:28.150] Sjoerd van Acker: Actually, VR is my primary medium. I graduated and immediately got into virtual reality. I studied very broad studies, image and media technology, which really focused on not thinking about mediums, but thinking about stories and combining mediums to create new experiences and other things.
[00:02:54.627] Kent Bye: So that's part of the design curriculum and where was that at to study these things together?
[00:02:59.102] Sjoerd van Acker: That was in the Netherlands, in Utrecht, at the University of Arts there. And my way of working is working out of fascination for stuff. I'm pretty technical, so this experience started with just trying out stuff. The experience is all about your hands, and in the experience your hands are mirrored in a circle. And on paper this sounds kind of boring, but when you try it for real, it's really fascinating. You can all make this super interesting geometric shapes with your hands, build moving sculptures, like I like to call them. And I like to believe that this kind of things, you can't think of them on paper, you just discover them by creating stuff and just playing around.
[00:03:50.695] Kent Bye: Yeah, I can definitely attest to it's a lot more compelling than what it may sound because you have your hands and so you have this embodiment but as you have this mirroring effect you can see your hands and you're identifying with your hands but then when you start to replicate it in like a circle or a mandala effect you have this almost like distributed embodiment where you're actually feeling like you're controlling and having agency over this spatial object that starts to transcend the shape and form of a normal hand and combine and mix together to create this larger, what I, remind me of like when you're making a pot in pottery and it's like going around in circles and it's changing shape as you put your hands on it, but this is your actual hands and you move it, you're creating these different shapes and having your hands combine together. I really, really enjoyed this piece and think that the degree of embodiment that it creates, but also the agency to be able to change the ways that it's morphing and shaping, super, super compelling. And so where did this project begin? What was the initial provocation for you to start to experiment with this kind of hand tracking and embodiment in that way?
[00:04:52.377] Sjoerd van Acker: I was doing a lot of research into embodiments. We had access to a full-body motion capture studio where we had two people in like the black suits with the ping-pong balls and the VR headset on and we just did weird stuff like sharing bodies and also having multiple bodies and that's kind of where the seed got planted. There it was also just fascinating. I stayed there nights just on my own, jumping around in this VR mocap suit. And then it became Christmas. So my cousin, we had Christmas and we did like a secret Santa, and my cousin wanted something immaterial. That was on his wish list. So I was thinking, what can I give that's immaterial? And then I thought of an experience, a VR experience, because I can make those. And that's actually where the first seeds began of this project, where I was also just experimenting and then came up with, oh, hand tracking and the mirroring effect. That's like really fascinating. And then I gave the presents and I realized people wouldn't get out of it. Like you literally had to pull the headset off the person to get them out so somebody else could try it. So then you know, yeah, this is something really interesting. So that's the source of the beginning.
[00:06:15.009] Kent Bye: Interesting. So it started as an experiential gift to your cousin to be able to create an embodied experience. And so what was his reaction to it?
[00:06:22.223] Sjoerd van Acker: I think he really liked it. We added music there, like music is really essential for this piece. It makes people move easier. So that's really added a lot to the, yeah.
[00:06:34.553] Kent Bye: Well, this project is showing here at Venice Immersive 2022. It's a part of the College Biennale where maybe you could talk about that process of applying and being a part of that and how that's related to what's showing here at Venice Immersive.
[00:06:46.303] Sjoerd van Acker: Yeah, so I showed this project to my producer and then heard nothing back from him for a couple of months. And then he came back and said I couldn't get this thing out of my mind. So that was the first moment we thought of working together on this project. So we applied to the college program and every year the Biennale has an open call for project ideas and 12 projects get selected in the first round and there are multiple rounds. They got workshops, it was amazing, like the best people of the world in VR. get to help you with your own projects and loads of really interesting subjects, like social VR. And then, over the course of multiple workshops, both online and in VR and physical in Venice, you develop your idea and your experience. And in the end, one project gets selected to win a grant. I'd really recommend it to anybody who starts in VR to try this amazing opportunity.
[00:07:52.160] Kent Bye: What type of feedback did you get as you're showing people this prototype at the College Biennale?
[00:07:57.455] Sjoerd van Acker: So we had a prototype, which was already amazing to just share. And as I said, and as she also mentioned, you can't really describe it in words what the experience is like. So having the prototype was really amazing. And that was also very insightful to see people's reactions. That was the main thing. And the other thing, I can't spoil the experience. So I can't go into it a lot, sadly. Yeah, there's a...
[00:08:25.878] Kent Bye: There's a definite experience of sort of like phantom touch that I experienced where it was sort of like you feel like you're touching something but there was a like a bolt of energy that was sent through my body at this end of this experience that was quite unique in a way that took this experience to a whole level this twist at the end that I thought was really quite amazing and really appreciated it and yeah we'll have to let people see it for themselves so they kind of experience that dimension but I did actually like experience a physiological unique experience of that phantom touch. I don't know if you've ever experienced that in this piece or heard other people talking about that but like kind of feeling like a certain level of I think that's part of having so much of a degree of embodiment in this piece where I really felt like I was identified and so when I see the visual feedback of something then It was almost like my body was sort of reacting for what it was expecting to react. And it wasn't actually feeling the haptic feedback of what I was experiencing. But it gave me this jolt through my entire body, which was, I've never experienced that before. And I just want to see if you've had other people report on having this kind of phantom touch experience.
[00:09:30.232] Sjoerd van Acker: Yeah, that's really amazing to hear. I think you're the first one who told me. But I have heard of the phenomena. Sadly, I haven't experienced it myself.
[00:09:41.127] Kent Bye: Yeah, I think that there's a part of like you being really embodied and with the predictive coding theory of neuroscience theory Which is like your mind is a prediction machine and it's making predictions and so you have what you would expect and so I think I was in a situation where I was expecting through an interaction to experience something in that giving the visual feedback of that and you know this deep deep sense of embodied presence and then having sort of a reaction that the jolt was almost like a ghost like feeling where It would have been like if I was touching something that I would have expected to have that touch I would have had that feedback but like the absence of that my brain was almost like whoa I'm expecting something but it's not there and so I'm getting the signals of what I would get but the lack of the stimulus is actually creating this kind of diffusion where it's a subtraction of what I was expecting so it's sort of like this Ghost-like presence of this feeling that I'd normally feel but it wasn't there So it was that's the best I could sort of describe the phantom touch But yeah, I think your your experience was able to invoke that and yeah Hopefully other people will have a chance to see it and I'll perhaps elaborate more as people get a chance to what so speaking of that What's what's like the next step for where this project goes from here? Oh
[00:10:53.393] Sjoerd van Acker: So we're really hoping to tour festivals and the great thing is the experience is about movement and about music. So not only VR festivals but also music festivals and performance festivals, dance festivals. That's our short-term plan and hopefully after that we'll grow it to something that people can experience at home.
[00:11:16.492] Kent Bye: Awesome. And finally, what do you think the ultimate potential of virtual reality and this type of embodiment and immersive storytelling, what the ultimate potential of that is and what it might be able to enable?
[00:11:29.027] Sjoerd van Acker: Yeah, I love VR. And my dream is to use embodiments to blur the line between yourself and your environment in a way. Because I think the definition of yourself is fluid, or hopefully is fluid. And VR is the way to experiment with that and to discover if it is or not. Hopefully, we can find the limit of that.
[00:11:58.157] Kent Bye: Awesome. Well, definitely, I think you're making a lot of strides with this Alele here at Venice Immersive and looking forward for more people to see it. And do you have any final thoughts or anything that's left unsaid that you'd like to say to the broader Immersive community?
[00:12:10.627] Sjoerd van Acker: For new makers, definitely recommend applying to the college program. It's an amazing opportunity. Even if you don't get the grant, you'll get amazing feedback on your ideas and develop them further.
[00:12:24.478] Kent Bye: Awesome. Well, thank you so much for showing here at Venice and for helping kind of unpack a little bit of your experience. So thank you.
[00:12:31.630] Sjoerd van Acker: Yeah. Thank you very much.
[00:12:33.292] Kent Bye: So that was Sjoerd van Acker. He's the creator of Alele, which we're showing there at Venice Immersive 2022. Yeah, just a short and simple experience. I think you really have to see it to really understand what's happening, but it's really quite trippy to be immersed within an experience where you can see your hands and you start to use that as a three-dimensional sculptural dimension, but in a way that you understand your agency and how you're moving your body and how that's translating into the spatial context. really quite trippy type of, you know, sense of identification of embodiment with these sculptural objects, which are essentially your hands, but are being transformed in lots of really compelling ways. Really amazing experience, highly recommend you get a chance to check it out at some point and Yeah, and I had a chance to experience the phantom touch for the very first time and so just very interesting to be able to Experience it and try to describe it. It's kind of trippy Yeah, I think you have to actually experience it yourself to really know exactly what it's gonna feel like for you But yeah, just a really cool to be able to experience that and to talk to short about LLA really fascinating to hear about how it came about as a non material gift to his cousin and as well as the whole college biennale, which I think is a whole other dimension of different things that are happening in Venice. I'm happy to be able to talk to him and unpack that a little bit more, as well. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
