I interviewed Soul Paint co-directors Sarah Ticho & Niki Smit remotely ahead of the SXSW XR Experience 2024. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my series of looking at different immersive stories and experiences at South by Southwest, today's episode is with a piece called Soulpaint, which ended up winning a special jury prize at South by Southwest 2024. So sort of like the second place prize within the context of the XR competition. So this piece is by Sara Tico and Nikki Smit, and they previously did a piece called Deep, which I've covered previously back in 2021 when it was shown in one of the Unity showcases, one of the virtual online festivals. And that interview is unpublished. They're hoping to release Deep at some point on the store and make a lot of different updates. And so hopefully I'll have a chance to air both the previous episode, but also do a new episode whenever Deep is going to be made available. And so in their previous experience of deep, they're using a lot of your hand gestures to correlate to your breath. And by using your breath, you're able to actually navigate through the spatial context. And it was just a really amazing mind body connection that they're able to achieve with deep. And so continuing on that theme with SoulPaint they're using virtual reality technologies to do this kind of body mapping technique to step-by-step walk you through this process to draw your emotions with this tilt brush inspired mechanic but is built from the ground up to be super streamlined to fit their specific use case of being in the first person perspective and noticing your body stepping out of your body and then painting emotions on your body and then stepping back into your body and then stepping out again and listening to the stories that you're telling about it. So just a really masterful experience that is exploring how to use these immersive technologies to become much more connected to your own inner state of your emotions, and be able to create a context to be able to share that with other people and to be able to tell your story of what's happening in your life. So really quite powerful experience and actually one of my favorite ones that I saw this year at South by Southwest just because it's Really digging into some of the real affordances of embodiment and how you can use virtual reality Technologies to look inward of what's happening inside of your body So super excited to see this get out and have this mechanism to do this type of body mapping technique Which has been kind of proved out in these larger research contexts and to have it available within this virtual reality context. So I had really quite a powerful experience with this piece called SoulPaint. So that's what we're covering on today's episode of the WysisVR Podcast. So this interview with Sarah and Nikki happened on Wednesday, March 6th, 2023. So with that, let's go ahead and dive right in.
[00:02:45.662] Sarah Ticho: Hi, my name is Sarah Tico. I'm an XR artist, but I also work broadly in the field of policy as well. So alongside making projects at Hatsumi around mental health and well-being, also work closely with the NHS through my role at the XR Health Alliance.
[00:03:04.403] Niki Smit: And I'm Nikki Smith, co-founder of Mona Banda. And through my company, I've been making interactive embodied, playful art for the last 16 years. And when VR came along, which, you know, it's a body medium, we quite naturally evolved into VR artists because it's such an embodied medium.
[00:03:27.779] Kent Bye: Great. Maybe you could give a bit more context as to each of your backgrounds and your journey into VR.
[00:03:33.634] Sarah Ticho: Sure. Yeah, it was very accidental for me. I studied anthropology at university, so I just found people broadly fascinating. And then it was actually just as I was graduating that my dad passed away and I started working in an arts organization that was like a former Regency church that commissioned site-specific installations. So I think that that was actually my introduction into immersive art. and seeing people talk about what those artworks meant to them and just kind of seeing them immersed in it and the conversations that it brought out was I think part of my interest in how VR could do that in a bit more of an accessible way. I then kind of went through my own sort of journey of grief and going through the healthcare system and became really interested particularly around how VR could be a tool to provide insight into lived experience and the different realities that people experience. And so it was through learning about other projects that were talking about mental health. I also found out about how VR was being used to improve mental health. And I just thought that was so, so fascinating. So I ended up becoming a curator at a place called the Big Anxiety Festival, where I was putting on an exhibition about that kind of interesting overlap of talking about and improving mental well-being. And so it was through that that I actually met Nicky when he was first making Deep. And that was the first project that I showed. And learning about projects like that just made me want to do more.
[00:05:12.813] Niki Smit: Yeah, for me it was, I come from a background that is half game design and half theater. And the beautiful thing about theater is that feeling of being in the moment and being there with your real body, like that real in the flesh, ephemeral, expressive moment of being there. And then VR came along and that fits so well with all the kind of playful, spatial installation work I was doing before. that I started to tinker around with that. And around that time, I met my co-designer on Deep, Owen Harris, and we started making Deep. And it was such a logical fit to start using my game design interests of making people aware of their own body and own creativity. And then suddenly having a medium where the body of the audience is there in the medium itself. So you as a game designer can suddenly not only address their mind and their empathy and their inspiration, but you can address their body too. And that just opened this whole new playful door to start making those kind of things.
[00:06:23.731] Kent Bye: Awesome. Yeah. And I feel like the piece that you're showing here at South by Southwest 2024 with soul paint is a bit of a extension into some of the spirit of what I experienced within deep, which we have a unpublished interview. Hopefully I'll be able to get it out because there's a lot of really amazing stuff that you were doing with not only with the breathing and I saw it just with my hands. But I know that there were other devices that you're putting on your body to really make this connection between the body and your phenomenological experiences, your mind body connections that you're able to really explore in such a unique way. And so I'd love to maybe give you an opportunity to give a little bit more of a update as to what's happening with deep where it's at, what's happened since the last time we talked back in July of 2021, and where it's going to now.
[00:07:07.802] Niki Smit: Yeah, so we've been trying to get deep onto the App Store, but at the same time, most game companies, they have a certain runtime for their development, right? You have to make your game and at a certain point you run out of money, so you need to bring it to market. But with Deep and the same with SoulPaint, is that at the same time, we're working together with scientists and research institutes, which allows us a totally different timeframe to work on. So at a certain point, when we ran into problems of getting onto the App Store, we sort of got sidetracked by working together with a great research partner to dive deeper into this very juicy research. and yeah that sort of postponed the launch for a while until we got some stuff figured out and now through that research that we've been doing with scientists we also got to look at our own game with new eyes and what we thought was ready for launch two years ago now we're like yeah you know what We're going to rework these things that we made with the scientists. We're going to rework them a little bit and put them into the at home version too. And yeah, get that ready for launch later this year.
[00:08:25.942] Sarah Ticho: Yeah. And to add to that as well, I think it's been a really interesting journey for both Project SoulPaint and Deep around where do we best want to put our energies and which channel is best for distribution. Because I think for us, it's really important that we make these things as widely accessible to people as possible. And so bringing it to the App Store is one avenue for that. But we also know the limitations of there's only so many people that use VR in the world. But there's also this whole sphere of bringing it into healthcare, how it can be used in healthcare context and prescribed, and that that is potentially a slightly different experience and a slightly different strategy. And the working and research is this really interesting space in between where you get to collaborate with really brilliant people to think about it and expand that. And yeah, working out all those different places in an industry that is quite new to VR has been a really fascinating journey. And it was sort of through the experience of creating these two projects that led me to do some of that policy work through the XR Health Alliance as well.
[00:09:35.965] Niki Smit: And also to add to that, what we see that's happening the last, I don't know, 10 years, not only in game design, but in broadly in the world of design to make it last 15 years is that you have seen this shift from what is the role of design. So the role of design used to be, you know, luxury projects. But more and more design has shifted to design for societal impact. Right. How can we use design to make a better world? And you've seen the same now happen with game design, where traditionally you had like the fields of applied gaming and serious gaming where it weren't traditionally game designers, but more like policymaker or researcher together with a few coders. make this bare bones game. But the last couple of years, you see all these artists stand up and from sort of a grassroots place that is outside of academics and outside of healthcare innovation labs, starting to make their own things and then pull in scientists to inform it and get that to a place where it's, I like to call our process art led science backed. So, it all starts still with this very pure and emotional and synesthetic artist view. But then talk with your scientists and go like, hey, the sort of behavior that you're facilitating that has this and this and this impact. And that's interesting. And then we go, oh, I didn't know that. That's great. So let's lean into that a little bit more, et cetera, et cetera. So but that then leads to a new sort of segment of entertainment where it's not a therapy app, it's not a fitness app, it's not a traditional game, but it's art, but it's also good for you. And that sort of new segment, that is what makes it, I think, hard for us to find the correct positioning on something like an app store that is still working in sort of old pillars, old genres. And what we like to do, my goal with the kind of stuff I make is I want to prove that our work can be at home just as well in a museum as in a therapy center or hospital waiting room.
[00:11:51.417] Kent Bye: Yeah, I really feel like your work is able to be used across all these different contexts and look forward to getting that conversation out about deep because it was one of those that I had a really visceral experience with it. And just like soul paint, I also had a really powerful experience with it. I had a chance to play through it. I showed it to my wife who also played through it and I thought, There's also really amazing conversations that can start to get unlocked when you start to do an experience like SoulPaint, which is all about using what feels like a little bit like an open brush integration to start to explore painting your emotions. And so maybe you could just set the broader context to SoulPaint and how this project came about.
[00:12:29.875] Sarah Ticho: Yeah. So back when I was working at this festival, the Big Anxiety Festival, I first came across this fascinating arts and health research method called body mapping. It's been around since the 80s. And it is a tool that was designed to think about new ways that we could talk about the embodiment of emotion and feeling and sensation in the body. So traditionally, you would trace around your body on a large piece of paper. go through a mindfulness onboarding experience and then maybe be given a prompt like, what does anxiety feel like in your body? What sort of locations do you experience them? Do they have a color or a texture? What does it feel like falling in love or being stung by a bee? And I just love that as a process of thinking about how you can translate these kind of intangible, impossible to describe feelings into something visible that you can use as a way of talking to people about. I'm just not very good at drawing. And I remember doing my first body map and just feeling quite frustrated by that process. But I'd tried things like Tilt Brush and love that you can float through space and be in these kind of amazing environments that I think can create this level of focus in a way that it's not about being immersed somewhere else, but like you're actually reflecting more on your experience. that you can paint with fire and electricity, and the idea that we could create almost infinite palettes that could be representative of different sensations. And that can be feeling, that could be a shiver down your spine, it could be grief. And what we wanted to do is create a brush set that felt like it was quite broad, but that we're never telling people, this is the joy brush, but really creating something that gives people that ability to subjectively translate their experience and by bringing it into VR being able to draw on the skin or inside the body or even outside the body and I really love that process of that it seems that people draw all around them as well that feelings and sensations can be you know, totally all-consuming and relational. And so we're really interested through this process of what can we also just discover about the diversity of human experience? Like, does everyone draw sadness the same or other different sensations, does chronic pain look the same in people or how does things like language or culture inform the way that we think about our bodies because we talk about how we relate to our bodies in so many different ways. So it's been a really interesting process of working with Nikki and all the different researchers to think about how making it interactive is such a complex journey in some ways from taking quite a simple concept to how you onboard people into creating something quite interactive.
[00:15:42.529] Niki Smit: Yeah, and that has been the challenge, I think, to see that the prototype that Sarah had where you could, in VR, make a body map, there it was just reworked tilt brush functionality, you know? That hasn't changed that much in that three years' time. But what we've done in those three years' time is create an experience around it that makes you land in the right place that leads you in because the technical functionality of just saying here's a brush in your hand here's a transparent body go to town on it that is not that hard but bringing people to a place where they can understand this fairly abstract concept of, hey, not how are you feeling, but where are you feeling? That takes a little time. So, you know, you need a little bit of centering, a breathing exercise. You need some onboarding, but we don't want to make it this medical tool. So this has to be not like this stiff tutorial, like, sometimes emotions, it needs to be an experience that you're drawn into and that you feel that you have agency and that it almost feels like you're diving inwards into yourself to start exploring yourself. in an artistic context instead of like very cerebral therapy context. And designing that flow, getting people from, I know nothing about this and I'm very externally motivated for impulses and then sort of slowly drawing them inwards into themselves. That is what took the most time to try and do both in scripts, but also in getting the controls as accessible as possible. Yeah.
[00:17:34.969] Sarah Ticho: Yeah, and a big part of that process was back in October time, then when we did a pilot as part of this social prescribing program that we were doing with Story Futures Academy in the UK. So we had a prototype of SoulPaint in a public library, and we were looking at how it could improve well-being, as a public engagement tool, but being able to bring in people that had never used VR before and learn how best to present that in a public setting was really interesting. So we did that across two weeks. and had loads of really valuable feedback just around anything from how you represent the body to what kind of brushes people want, how comfortable people are about sharing, how you can create really informed consent and really create that journey through an installation-based experience where you feel like you're being held by someone that can also speak to you if challenges come up. So we also went through mental health training as well. Because I think a big part of the experience is not just the experience itself, but like you were saying beforehand, it's the conversations that it brings up with people. And I think that is the thing I'm most excited about this, that this is a way of getting people to talk about things that are hard to describe, and that you could share that with your partner or your best friend or your doctor or your therapist, and how it could change the way that we understand one another, as well as like what we can learn or sort of more research level too. But I think right now for us, the most important thing that we wanted to do with this was release it to people and think about ways that it could bring value in the way that it feels most important for that person to use it.
[00:19:25.847] Kent Bye: Yeah, you're really creating a context where people can draw their inner life and their emotions on the context of their body. And then you have the mechanism to be able to record audio and then have you share potentially at the end, you have the option to upload your drawing and to listen to other people's drawings. And I thought that that was also a really powerful way of just being able to bear witness to other people's stories, what's happening in their body and Sometimes it's very clear when you see like someone's shoulders hurt and you see all this kind of redness and radiation of lines coming out of that indicating a sort of pain. And so, yeah, as we were talking about that, I was just like, oh yeah, and this could be like an augmented reality filter where you're like, you know, walking around with your inner representation of what's happening in your body and just. another layer of understanding what's happening in other people's lives by doing these types of body mapping paintings that really helps to take what's happening on the inside and give some sort of spatial representation to allow people to connect to each other.
[00:20:22.918] Niki Smit: And also, it shows that thing that on the inside, it transforms it into something that is not awkward or vulnerable to talk about anymore, because you, within your own playful agency, drew it yourself. So there's this form of control over it, where you can go like, you know what, my anxiety is really bad, and I feel it like this black, sticky, undulating knot in my stomach. But drawing that yourself makes you go, oh yeah, This is how I visualize it. Oh, that's quite smart of me. It really looks like what I feel. And then suddenly there's this sort of pride and this agency of your own creation and association attached to something that used to be very elusive and awkward to talk about.
[00:21:13.206] Sarah Ticho: I showed one of the first versions of the project right before I met Nikki to a friend that suffers quite severely from body dysmorphia and we were hanging out in the summer and I showed her that artwork that she did about three and a half years ago. And it was also really fascinating that she was like, I remember that. I remember that feeling so well. And it's made me realize how much I've changed as well. And I think that's something that really sparked a lot of interest in us is this installation version is a one-off experience typically, but the idea of it as an immersive journaling tool that you could continue to create artworks over time and thinking about how that is a way of documenting your journey and thinking about how you change and being able to look back on that the way that you could with old diaries. I'm really interested in how that may change people's perspective on the stories that we all tell ourselves as well.
[00:22:19.100] Kent Bye: Yeah. Well, a quick technical question. Are you using the open source version of open brush that you've integrated into this project?
[00:22:27.023] Niki Smit: We started with that, but there's so much stuff attached to that. And we only want to use a few things that it was actually easier for us to make something like that from the ground up. Because if we would use that, then we would, when we only want to sink, we would get the whole kitchen into the project, if that makes sense. So yeah, we took notes from it, but then started all over.
[00:22:53.905] Kent Bye: Okay, well, so it fills very much in the vein of something like open brush, same kind of mechanics, but something that's a lot simplified, I would say. And I wanted to clarify that because, you know, when you go into something like open brush, it's like a blank canvas, you know, you you have the tension of being able to do anything you want. But yet, as a immersive experience, or a narrative context, or in the context of therapy, you could really be lost as to where to even start. So I feel like the way that you have the onboarding of step by step of this experience, even the little tutorial where you're drawing and you're taking whatever you draw and have it go away, like it's not permanent. So you don't get attached to anything that you're making. You're just getting the mechanics of how the tools work. you know, at first I was like, Whoa, this is like frustrating. I can't get anything to stick around. But then I was like, Oh, this is just kind of getting the feel of it. But it helps to create more of a flow state. So you don't get too caught up and making something look perfect, but you're just getting the first steps of how the mechanics work. So I felt like the onboarding from starting from no experience of people have never even tried something like tilt brush or open brush that you're able to step-by-step guide them along the way. And also put a larger narrative context around it that allows you to have a tutorial and you're slowly learning each of the mechanics. And so you have this game design element where you're slowly building up the skills where you can eventually then do the more open-ended body mapping process that you have us go through. So yeah, I'd love to have you maybe elaborate on that type of onboarding process and really striking that right tone to avoid that experience of creating this open ended sandbox that's overwhelming, but to create something that's much more bounded and accessible for people to really engage with this project.
[00:24:44.176] Niki Smit: Oh, I'm so happy that you caught that. Yeah, that was one of the things because we started out with the wish of giving people as much choice as possible. But there was this tension field that it was also saying, like, if you give them too much choice, they'll clamp up because they don't know where to start. So, yeah, you really have to sort of guide them in and that it's not like Here's the tool set, you can do anything with it, go to town, but literally that it almost becomes a little narrative of discovery to learn every little thing. Hey, if you pull the trigger and you wave your hand in the air, you will then 3D in the air, draw a line and sure it will fall away again. And maybe that's frustrating at the start, but at a certain point you'll get a body and there. we tried to make is the tension field between making stuff accessible but still making them feel like they have an inner logic in the world by which I mean we struggled for a long time around the tension field of sketching or drawing for real right away and also trying to avoid that micromanaging of people. And then when we realized, oh, but we could just, if people want to try out the brush, but if they try it in the body, it's already on the body and then they have to delete it again if they don't like the brush, you know, that is tedious. So what do we do? And then we realized that it's quite organic to let people just draw in the air and then it floats there for a second before it falls away. So literally all the air around you becomes a sketchboard. for you to try out things before you do it for real in the body and then at the same time it has this internal fantasy sort of this internal poetic language where it says like hey all emotions need to originate from within so as long as you start from within the body with drawing you can draw all the way out and have like these big clouds of chaos coming out of your head but remember they have to start start in your body. And that then also solves the technical problem of if you then later move with your avatar and all the brushes are attached to you, where do you attach them? Like, are they attached to your elbow? So it was like, we have a bunch of those quite elegant little solutions that touch both on usability on a poetic story layer and just on technical pragmatics. But that was the huge puzzle to make those things.
[00:27:21.645] Sarah Ticho: Yeah. And I think what we found as well as particularly for first time users but for everyone is that just the tools are so seductive and that people are so excited at the ability of being able to draw in the air. That is what the magic of Tilt Brush and all these different drawing experiences has been is just that it's so freeing, but that we're asking such an important question and essentially trying to, we're asking people to become synesthetes and to think about something quite complex and abstract and locate it in their body and then think does it have a color or a texture and that is a lot of onboarding alongside just understanding like Nikki was saying the controls and the interactions so I think there was a real need for people to get it out of their systems in a way so that there is that ability to focus and just again touching on that whole idea of like What we wanted to explore in this is that all these feelings that we have are ephemeral and temporary and that they only exist for a certain amount of time. So I think that was part of the joy of having everything fade away. And yeah, obviously that narrative continues throughout the rest of the piece as well.
[00:28:37.002] Kent Bye: Yeah, and the canvas that you're drawing on is your body. And so because you do have a first person perspective from a virtual reality headset, then you have to do some tricks to do some inverse kinematics and actually map out the body. So there was a real magical moment I wanted to ask you about because I felt like you were either doing a magic trick or something that was really actually kind of like detecting the breath because you have us kind of breathe our body in. And I had this experience where it was like, okay, take a breath in. And then when I stopped breathing, then the body stopped coming in. And then when I breathe in again, the body, it almost felt like you were somehow detecting my breath. I was like, how are they detecting my breath? Is it through motion based or are they, are they listening? And like, or is this just some sort of magic trick? But I was like, wow, that was really, a nice implementation of that. Obviously, your previous experience of deep was all about the breath and breathing in and out. And you had us move the controllers to dictate when we're breathing in and out. But I'd love to have you maybe expand upon or do you have some sort of like magic way of detecting the accelerometer? Or is this just something that was kind of paced out to give you the sense that this was actually following your breath?
[00:29:48.555] Niki Smit: yeah so oh that makes again that makes me so happy yeah so there are tricks to measure the breathing both using the camera both using the microphone or minute movements of the headset but honestly you know we're using none of that this is just the power of story because you're there and you're standing in front of that mirror and the narrator tells you like hey look at this space and take a deep breath in and we know that most people will do that after somebody tells them and you're already sort of standing and focusing and If somebody tells you, take a deep breath in, we just timed it. So we go, she says, take a deep breath in now. People then start breathing now. A human breath takes just something about this long. And it's just an animated, we just animate that shader coming upwards. So your body slowly appears, but it's a linear animation. We're tracking nothing, but we're hoping that most people will sort of instinctively start breathing at the moment that they're asked to. So yeah. it's smoke and mirrors.
[00:30:52.560] Kent Bye: Yeah, I think it was probably smoke and mirrors, but it was just so well done that it was like one of those magic tricks. I was like, wow, that was quite incredible. But once the body's there, then obviously you have a certain degree of inverse kinematics to give you a full body representation. I tried to move my feet a little bit. There wasn't like actually tracking my feet, but it's mostly from the waist up anyway as an experience. But you're now looking into a mirror and you're asking us to step through a portal and start to actually kind of draw out our emotions. And so love to have you elaborate on this process of instructing people to use these tools that you've created and ported over, built from scratch, like you were saying, to then kind of use the affordances of this spatial painting to describe their emotions and how to really set a good context for people to really get into that.
[00:31:40.320] Sarah Ticho: I think at the beginning of the process I was so inspired by projects like Freud Me by Mel Slater and just this idea of being embodied in yourself and disembodying from yourself and that ability to create a completely different perspective on your body using that process like always really fascinated me as well as being able to just draw on the original concept of body mapping that in the original process you're traced in so exploring you know how could you trace in a body Is it that we have the sound of a sharpie like spatially around your body? Does that actually make sense in a virtual environment or not? And so I think we explored different ways that we could really make people feel embodied in that process. It was really interesting thinking about sound as well, how you can use sound spatially to create that sense that you're being created. And we could talk for hours about the journey of exploring avatar representation as well, because I think it is such a complex thing to do. Right in the early days of, you know, prototyping, I was trying to find you know some standard obj files of a body online and they're all just you know incredibly muscly men or hyper sexualized women and of course that's just like not appropriate when we're asking people to think about exploring their body not as something that is Yeah, you can't have something like that as a representation of everyone. And then also lots of people, even when we tried to make something vaguely gender size neutral, would say, I can't relate to that. And so we went through such a process of, you know, do we have a series of avatars and different levels of chunkiness at the beginning so that you have something that you feel like represents you. But of course, that was incredibly off-putting for people because we're asking people to explore how they feel within and then trying to create that evaluation of your size and shape from the outset was just so unpleasant.
[00:33:54.227] Niki Smit: Really insulting, yeah.
[00:33:55.328] Sarah Ticho: Yeah, incredibly insulting.
[00:33:57.449] Niki Smit: Yeah, we just wanted to give a bunch of different body sizes so you can choose at the beginning the sort of size that you identify with. But of course, what happens then is you thrust people into an experience. So you basically ask them, choose how fat you think you are. And that doesn't work at all. So then we need to go back to a universal body size because making people choose doesn't work. but then universal body size also doesn't exist. So it was this Catch-22 that we ping-ponged on from side to side for a long time until we found a shader for the avatar that was sufficiently smoky enough to sort of give the feeling, oh yeah, now it looks like the icon of what a human shape is. It's a sort of smoky thing, and then it's a little bit easier to To internalize, oh yeah, this is not literally this body, it is the icon of roughly my canvas.
[00:34:57.620] Kent Bye: Yeah. And so you have an opportunity to draw on your body and then teaching them how to speak into this little ball to record things. And then they're able to really tell the story of what they were able to create, which I also found that those super powerful to not only draw it, but also record something. And then having the opportunity there at the end to listen to my own story, but then to listen to other people's story. I think there's a real nice moment there at the end where you can really hear some of the stories from other people. And then sometimes those stories will just land and kind of really get me into a much more of a deeper place of emotion of feeling with whatever they might be going through, whether it's grief or pain. And then you get to listen to your own account again near the end. And so Maybe you could talk about that process of capturing the audio story of it and giving the option to share it, given the fact that, again, you're kind of dancing in this area where this is really intimate information that people might be sharing. At the very front, you say, hey, look, we're not going to be recording anything. You have the option to share it if you want, but just so you know, the audio that you're doing is completely private. And so you have this process of making this map, but then having the choice to share it at the end or not, but maybe walk through that process of allowing people to record their own story and then listen to it.
[00:36:16.025] Sarah Ticho: Hmm. I think the artworks themselves, it's so hard to untangle what they all mean without having someone describing it to you. And so I think that was always my favorite part of the process when you would just ask someone to explain it. And it's just you don't even need to ask a lot sometimes. You just see people just unleash all these stories. And, you know, why did they choose that color? Why did they choose that texture? or what is that story and I think there is something about the meaning that we create for ourselves when we're able to say it out loud that it's not just you now searching within but you have something to look on to start to tell that story of that I think also creates such value for a person.
[00:37:02.123] Niki Smit: It's an anchor point.
[00:37:04.164] Sarah Ticho: Yeah, definitely. It's been a real interesting journey for us, even in the testing of how and where we present that. Because if you are in a public space, we know that people are much more apprehensive about sharing that story. And so that's why we've tried to build an installation that can create more of that sense of seclusion. But I think I really love this idea that you're almost kind of collaging throughout the experience, that you're creating this visual expression, and then there's this audio expression, and that we try and capture your movement through that, and that that moment at the end where you see it all coming together, that is that kind of, you know, seeing yourself from that outside, that Freud-me style sort of perspective and sense of self-compassion that we hope that people can have for themselves. Yeah. But of course, I think there's such importance in the value of seeing yourself in the context of others. And I think that we can all get so wrapped up in our lives and stories. But I think this is why I've always loved about the arts and kind of, I guess, takes me back to the beginning when we were talking about how we all first started, that I love seeing other people's stories. You see yourself in that. And I think that is the hope with soul paint of knowing that we're not alone in feeling things that are awkward and complex and I think that's something that we really want to continue of how can we find new ways of sharing stories and that is important for everyone but how it can particularly help people that do struggle to find the words anyway like our first prototype we made particularly for children and young people and thinking about how it can be valuable for them to talk about their emotions, thinking about how this can support people that are neurodivergent as well in the autistic community, and that there's all these avenues that we're so interested in about particular areas that it can create benefit. But I love that moment so much of that reveal. And I think a lot of inspiration also comes from things like where thoughts go as well as an experience where it is really about sharing your story and the kind of delight in seeing yourself in the context of others, I think is such a beautiful part of the narrative. And I think a huge part of what it means to engage in mental health, that we need to have time to think about our own experiences and see it from another perspective, but knowing you're not alone in that is extremely important.
[00:39:28.767] Niki Smit: And then the way we structured... So, again, it's very simple elements, but it's about finding the right structure so it flows very logically. And we knew that, as Sarah says, the painting itself is almost collaging. You are in a play state. and you're highly associative and you go like uh this goes here and this goes here and you're most people don't do that very cognitively they're just in the moment and they're they're slapping colors onto that body but then we realize if you immediately after give them the chance to reflect on what they just did that becomes really joyful because then that gives them a chance themselves to take a breath, take a step back and go like, Hey, it's quite cool that I did that. Yeah. And I did that because of this. And that's quite a cool creative choice of myself. So it gives them this moment where you allow them to elevate their own associative power. which makes them feel you know good and in control and it's that moment where you start to own your own drawing but then we knew okay now you have something but now we're going to show you something from other people and then you hear other people and you have a dancing scene in between where you really land in your body and you're not thinking at all And then we know, because you get exposed to so many other elements after you talk about it, first dancing, then seeing stories of other people, hearing stories of other people, that then it allows us to create the chance encounter with yourself. where you've kind of forgotten what you've said already because you've done it all in the moment and it's sort of gone to the back of your mind. But then when it goes silent, you get to meet yourself again, yourself of the past. Even if it's only four minutes in the past, emotions are ephemeral. So most of the time you already sort of moved on and are in a little bit of a different state of mind. So then having this moment where you meet this stranger of yourself like oh yeah oh i just did that wow and that then already makes you externalize it a little bit which is extremely important especially if you've drawn something very heavy like trauma or something like grief and then allowing the player at the end to view themselves and their own emotional reasoning from a third perspective Yeah once we realized we could make that the ending it was like oh yeah that's the beautiful ending because then you leave people with this sort of full circle feeling of yes okay now I can let it go now it's done.
[00:42:08.888] Kent Bye: Yeah, the inspiration of the Freud me the mill Slater, where you have an opportunity to share your problems with a counselor, and then you're talking to Freud, and then you actually embody Freud, and you listen to your problems. And then you, from the perspective of Freud, give back the answer. And so there's this first person, third person shift that happens in that experience that you're really playing with a lot here, where you're starting off in first person, you go into third person, you draw it, and then you re-embody the first person aspect of the perspective of all these emotions you just drew. And then, like you said, then you switch back to the third person and you see yourself again and then are able to kind of listen to all those emotions where, you know, when I went through it, I had this ability to kind of just draw some of the emotions that I was in kind of more of a numb state. And then this allowed me to kind of tune into what was happening across my whole body. and then to step back into that and then move around and have this delay where it's like a reverb type of delay where as you're moving you see like a trace of it so it gives you this dynamism of seeing the emotions in the mirror as I'm moving my body and then you kind of like shake it off and then you let go and so there's a release. You're creating a story of something that is taking something that is ephemeral and turning it into a static object. However, it's not static in the way that it's connected to you in a way that you're allowing it to flow through you and to also shake it off. Then you're able to observe it from a distance again at the end and observe that more static nature of it. I feel like that balance between the nature of emotions being ephemeral and allowing people to really have a process to interrogate what's happening in their body in a way that I think is an extremely powerful use of the medium. I had a really amazing experience. I showed it to my wife. She wanted to make sure that you knew that she absolutely loved it and is looking forward to sharing it with her friends at some point when it becomes more available so that they can go through it and they can talk about it. So I feel like it's a type of thing where you are sharing it at the end to like a public, but I feel like if there was a way to just share it to people that you know or your friends, being able to have that conversation would also be a really powerful use of a tool like this. So I had a really powerful experience of it and I just loved it and can't wait for it to kind of get refined and out in the world so that I can, you know, share it with more people.
[00:44:25.722] Sarah Ticho: Thank you. That's so lovely to hear. Thank you. Yeah. I think all of those things are definitely part of our plan. I think at this point, like we are creating the ability for this installation itself by that we are planning if people want to that we can email it to them but we're working towards a much more robust system that you can access it and I think part of our plans going forward as well of having this as a journaling tool means that we want people to easily access them digitally as well so you can have your own growing archive of stories too. And I think we're so excited for that possibility because I think like a big part of the sort of research behind this as well is really exploring this idea of interception and interception is all of those feelings that you have in your body all the time, it's hunger, it's temperature, it's anxiety, it's like everything that you're feeling internally and that through things like meditation then you can have a more heightened interceptive awareness and that there are real valuable health benefits of that in being able to learn to self-regulate and being able to notice when things perhaps aren't great and thinking about how you can then notice it and do something about it. And so some of the research that we are doing is looking at like, can soul pain improve interceptive awareness in people? And we're not so cocky to think that just by doing it once that there's value, that we can change people's body awareness, but we love it as a way of introducing the concept. And I think part of our plans with the research is that exploring that opportunity for ongoing use, how does that change the way that you start to think about your body and notice sensations and emotions? And through being able to do that over time, does that create a different type of awareness around that? Um, so we'll see how that unfolds.
[00:46:21.191] Kent Bye: Great. And, um, as we start to wrap up, I'd love to have one last question around the installation, maybe elaborate a little bit more as to, as you're showing at South by Southwest, where are the ways that you're trying to create a context that allows people to enter into this magic circle that enables this type of extra vulnerability when you're talking about their emotions.
[00:46:41.875] Niki Smit: Yeah, so we made a set that is... Firstly, it just looks like the game. It has the same sort of soft gradients and soft colors. So it really feels like you're stepping into this other world. And we take care to make it in a way where it's layered, where it's both... open enough to fulfill its role of a set that needs to draw people in and not sort of a closed off black box but at the same time feel secluded enough and safe enough where when you step into the play space that is this secluded area but it's all sort of multiple layers of transparent veils that you sort of sink into.
[00:47:26.932] Sarah Ticho: Yeah, there's this kind of idea that things are increasingly revealed within it. But I think that VR is just part of that experience. And I think that's where installations can really help you step into the concept as soon as you arrive. And so not only do we have, you know, people that can kind of introduce you to the experience and I think start to have that conversation about feeling emotions in your body, but being able to give you permission to explore it at the level that you feel comfortable with. There are some people that don't want to go particularly deep. There are other people that want to share things that are really personal. And I think because there is so much sharing involved in it, then part of that informed consent, I think is really important. important for overall user journey. But I think the thing that we're really excited about with this installation as well is that there's such a limitation in how many people can put on a headset, and that we love the artworks that are created out of it. So we've been working with some really fantastic installation designers from the UK called Creative Giants to think about how we can best present that, including the artworks that are created. So even if you don't have time to have the full experience, then if you consent to sharing your drawing, then it's exhibited on screens as well so you'll be able to witness this growing archive of stories and we have plans for how that could be expanded even more, how could we create a terracotta army of stories of lived experience, how can we take over public squares in that, how can we bring them into libraries and galleries, and that is one way of engaging, but this is also a touch point where people are like, hey, what's this thing going on? Like, oh, this is someone drawing this particular feeling. and they feel it in their toes and their throat and their head and the like part of the experience is just having those artworks available to start these different conversations and that's the thing that I love most about it because there are so many ways that people interpret it and you could be speaking to one person about chakras and another person about neuroscience and how the body works at like a very functional level and that ability to facilitate that conversation is such an important part of the whole piece.
[00:49:44.648] Niki Smit: Yeah, and in that sense, the VR experience is like the nucleus of it, but there's layers around it and that the set design, but also our growing plans for a ever growing sort of library of of human embodied emotions is all about taking something that is very internal and make it visible. Right. And that's what you do with your emotions. But that's also what happens with this traditionally very closed off medium VR that you have it only in your own headset. But what comes out of it is this physical exhibition, but it's not pre-made artworks that we are exhibiting. It's an ever-shifting user-generated exhibition that people that don't want to play VR can also just walk through and listen to other people's stories and explore this forest of embodied artworks that other people made.
[00:50:36.751] Kent Bye: Awesome. And, uh, and finally, what do you each think is the ultimate potential of virtual reality and immersive storytelling in this type of mental health context? And what do you think it might be able to enable?
[00:50:49.822] Sarah Ticho: I am just so fascinated by this ability that we can use VR to connect to our bodies in this way that it can help us completely disconnect or completely reconnect, but I think how it can also transform our sensory experience day to day. It is a form of placebo, it's a ritual, it's this form of belief that you can create and how we can use that as a way of reinventing ourselves, of understanding ourselves and one another and being able to do that through incredible storytelling and creativity but really merging that with the power of science and research to make things that can impact people in a meaningful way and really access the wider public rather than just these small spaces where you're kind of preaching to the converted. So that's my dream for it.
[00:51:49.333] Niki Smit: And I think for me, like on a really functional level, what is a medium or what does it do? We've had a whole string of narrative mediums traditionally, books, theatre, film, and those were all about addressing the big themes of emotions that we have in life. Longing, hate, revenge, love, you know, all stories are about a few of those big ones. And then games came along and that then suddenly turned out to be a medium that is about underscoring emotions that were hard to address in the traditional media. Things like sense of discovery, sense of perseverance. And now with VR, now we have a medium where we as artists can address the human body. And then suddenly it becomes about stories that are not about words, but stories that are about rediscovering and reconnecting your own body. And I think there's huge potential in that.
[00:52:54.034] Kent Bye: Awesome. Well, Sarah and Nikki, congratulations on SoulPaint. I feel like that's a really amazing experience. There's so much potential for how this could be used across many different contexts for friends and family, but in a therapeutic context and mental health or just even tracking your own emotions over time. So really excited to see where this continues to go in the future. Give a little note to the app curators at Meta. Please let Deep and these apps on the App Store. These are amazing applications I'd like to see folks have access to. So yeah, thanks again for taking the time to help share a little bit more of your journey and your story with the intentions behind SoulPaint.
[00:53:31.803] Sarah Ticho: Thank you so much.
[00:53:33.763] Kent Bye: So thanks again for listening to this interview. This is usually where I would share some additional takeaways, but I've started to do a little bit more real-time takeaways at the end of my conversations with folks to give some of my impressions. And I think as time goes on, I'm going to figure out how to use XR technologies within the context of the VoicesofVR.com website itself too. do these type of spatial visualizations. So I'm putting a lot of my energy on thinking about that a lot more right now. But if you do want a little bit more in-depth conversations around some of these different ideas around immersive storytelling, I highly recommend a talk that I gave on YouTube. You can search for StoryCon Keynote, Kent Bye. I did a whole primer on presence, immersive storytelling, and experiential design. So, that's all that I have for today, and I just want to thank you all for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
