Paper Rock Scissors is an interactive story that uses hand-tracking to allow the audience member to play the game of Rock Paper Scissors in the context of a story of how a mother and daughters use the game to make a variety of different types of decisions. It’s an example of an interaction within a narrative that seamlessly fits into the context of the story, and is also really satisfying to play with VR when the hand-tracking without any hitches (which is how I experienced it at Venice Immersive). I had a chance to speak with director Alex Ruhl about the evolution of this project, the catalyst that got her to not just produce other people’s pieces but to take a more active role in directing, and the overall process of designing and building out this experience.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that's looking at the structures and forms of immersive storytelling and the future of spatial computing. You can support me on Patreon at patreon.com slash Voices of VR. So continuing on my series of looking at different immersive stories from Venice immersive Today's episode is with Alex rule of rock-paper-scissors this is a piece that uses hand tracking technology to be able to do the game of rock-paper-scissors, but Built around the whole narrative of a mother and daughter and their relationship and how that changes over time and it's an interactive dimension so as you do this kind of rock-paper-scissors and you're invited to play along then it has these different branches of the story and I Talk about these larger dimensions of the parent-child relationship and yeah Just a really sweet story and and quite enjoyed because I think the interactions of this piece Tied really well into the overall narrative that's being told So that's what we're coming on today's episode of with severe podcast So this interview with Alex happened on Sunday September 4th 2022 at the Venice immersive in Venice, Italy so with that let's go ahead and dive right in and
[00:01:21.787] Alex Ruhl: So I am Alex Rule, I am primarily a virtual reality writer-director. I started in VR probably about 2015-2016, just around that kind of new resurgence of interest in the medium and this was around the time that You know, 360 video was very prevalent, the Gear VR had just kind of hit the consumer market, you know, Oculus Rift had kind of just come out, so there was this really interesting buzz amongst the creative community. And I was working in television at the time and the first time I put on a headset I was like, just blown away by the potential. and at that time I didn't really see anything in the realm of like narrative storytelling so I just kind of threw myself into it and I've been working in it ever since so from creating just simple 360 videos right through to more recently you know fully interactive volumetric captured stories and the piece that I'm here in Venice with Rock Paper Scissors which is a fully animated piece with hand-tracking interactivity. So, yeah, it's been a wild ride, and I feel like I've kind of done a little bit of everything in VR along the way, as most people have, because you have to wear so many hats. But nowadays, I'm very pleased to primarily be kind of leading the creative on projects.
[00:02:33.607] Kent Bye: Yeah, I saw a tweet of yours saying that you were just taking a new role on, maybe give a bit more context as to your background and some of the other things that you're working on in terms of XR.
[00:02:44.612] Alex Ruhl: So, yeah, I mean, very interesting times. Literally, in the week that I premiere this piece of work in Venice Film Festival, I get announced as the head of Metaverse Technologies for PwC, which is one of the big four global consultancy firms. So they're one of the largest consultancy firms in the world, and they work across all sectors, but primarily their job is essentially to solve problems. And, you know, they've got a real lovely human-centered way of thinking about consultancy. And so how that came about is the fact that, as many people know, if you're listening to this and you're a creative, you know that budgets are always thin, especially when you want to fund your own work. It can be quite a painful experience to try and just work solely on your own work. So I've diversified over the years and kind of paid the bills with being a VR creator in general for the corporate and commercial sector. And for the last three or four years, I've been PWC UK's preferred supplier for virtual reality. So with PwC UK I've created everything from cyber security reenactment training where you inhabit the CEO or the CFO or the CIO during a cyber security crisis and you have to navigate how you would react in that situation all the way through to more recently a piece that we did called In My Shoes which is racial diversity training which is the piece that I was referring to, which is fully volumetric interactive capture where you literally inhabit a black employee in the company, PwC, and you go through that experience and you start to learn about microaggressions and how some of the systemic unconscious biases affect your ability to kind of climb the ladder, I guess. So that's how the communication came about with PWC and Jeremy Dalton, who is currently the head of Metaverse Technologies, he is moving into a role in America. And so, yeah, we just got to talking and it's a very exciting time, obviously, with the Metaverse and this being this kind of new term that spurred on this next wave of interest in virtual reality, augmented reality, Web3. And we've had several conversations over the last year about where it's all going, how businesses can use it, and also how creative can still be at the centre of all of that. And so, yeah, I'm very pleased to be kind of taking on that role and for the next couple of years help to shape what the metaverse looks like in terms of helping businesses. So, yeah.
[00:05:04.989] Kent Bye: That's great, yeah. You mentioned that you had worked on a number of different projects in 360 videos, and you have Rock, Paper, Scissors, which I see is like a creative project. Have you had other creative projects that you've been able to work on before this?
[00:05:16.075] Alex Ruhl: Yeah, so actually my first foray into VR in late 2016, I believe, is when we shot it, was a 360 rom-com called Kidalike. And that was the first thing that I wrote specifically that was an original piece for the VR medium. So, you know, I had this lovely little idea about this rom-com, the meeting of these two women who are both experiencing kind of like loss with love and exploring like the different philosophical takes on love, basically. And it's this very simple one take, seven minute rom-com starring Gemma Whelan from Game of Thrones and Natasha Karam from 9-1-1 Lone Star. And so that was my first piece. I was the writer and creative producer on that project. I brought in a really talented traditional TV director to direct that, Chloe Thomas. And since then, you know, I kind of It's funny, you know, I mean, there's so many ways we could go with this, but I had a piece that I exec produced called Playing God that did the festival circuit in 2019. And around that time, a friend of mine, Mary Matheson, who is like a very brilliant VR director in her own right, said to me, Alex, why aren't you directing your own pieces? You know, why aren't you calling yourself a director? And I was like, Do you know what? I don't really know because that's what I'm doing in my, you know, my kind of more corporate work. I'm directing and producing everything myself. And she said, I think you need to, you need to just own that. You need to step into that title and own it. And so that's when the pitch for Rock, Paper, Scissors came about. And, you know, it was the first time I was like, do you know what? I'm going to be the director and writer on this and lead the creative from start to finish just to see what that feels like. So it feels great to have done all that and then ended up premiering, you know, Venice Film Festival.
[00:07:02.240] Kent Bye: Yeah, Mary Matheson and Naomi de la Pena are in the process of about to start their new semester of Arizona State University based in Los Angeles, sort of oddly. But they're starting a new immersive storytelling program there. So I'll be curious to see how that develops. But yeah, there's a little bit of a provocation of being confronted with that. And then so how did the idea come about then after that challenge from Mary?
[00:07:25.380] Alex Ruhl: So, I mean, like you Kent, I'm a bit of a big thinker, right? I'm like constantly thinking about like the bigger questions about the universe and our kind of like the philosophy on how we kind of approach human connection with each other, but also our relationship with just everything in general, and also our relationship to the universe, which is funny when obviously we're going to be talking about a very simple, quite surface-level coming-of-age story. But essentially where the idea came from was I was thinking about the idea of Rock, Paper, Scissors in terms of a game that relies on you having the same choices as someone else, right? It's supposedly a fair game as long as you all have the same choices. rock paper or scissors and seemingly you would have pretty good odds at winning or losing that game but then it was kind of in the depths of the pandemic and obviously there was a lot of kind of like social unrest and obviously the George Floyd murder and we were really starting to have these conversations about privilege and about like yeah like systemic oppression and it got me thinking about the fact that actually not everyone does have the same choices And that's something that you learn when you get older. And so the idea originally for Rock, Paper, Scissors was quite this like deep philosophical narrative around the fact that, you know, you start off with the same choices and then gradually the story evolves. So, you know, you still have those choices, but then the other characters come in and they've got way more choices than you. And you're just always losing as this kind of metaphor for life, I guess, a quite sad metaphor. And then as the kind of story evolved, and my exec producer from the BFI network, she put a lot of trust in me to kind of steer the story how I wanted to. But through conversations with her and with my co-writer Rebecca, we kind of thought, you know what, what does this look like if this is more of like a heartwarming story, you know? Throughout the pandemic, we've had a lot of trauma. Maybe we make something that's actually a little bit more uplifting. Maybe it's a little bit more like, yeah, at the heart, we're talking about the fact essentially about privilege. But could we make it into this actually quite lovely human centred, relationship centred story? And so that's where the idea for this mother daughter relationship and how that evolves over time came to be.
[00:09:39.522] Kent Bye: Yeah, and as I played this experience, I'm struck by how the interactions are both embodied, which feels really good through the hand tracking, but also fit really, really well within the narrative. There's a lot of times where interactions don't directly tie into how the story's unfolding, and so I'd love to hear about the process of getting the hand tracking right and what you had to do to either just use it out of the box, or I'm sure there's probably a lot of other things you had to do to really make it feel as good as it did.
[00:10:07.203] Alex Ruhl: I mean, all of the credit to that goes to my developer, Mia Ramal. She is just an absolute whiz kid when it comes to Unity development. A lot of the time when I see a lot of VR like you, sometimes I think that pieces are trying to do too much. There's too much and you can tell that maybe sometimes... the tech is leading the story rather than the other way around. And with this, even though the idea comes from the game Rock, Paper, Scissors, like you say, I wanted to make sure it didn't get in the way of the story. It always kind of like served it. And so when it came to actually getting the hand tracking right, essentially, that's how it started. It literally started as a completely blank environment with just the hand. And then we worked on that. And that was the prototype. The prototype was just the hand playing Rock, Paper, Scissors with the narration. That was it. And the feedback that I was getting at that stage was people were like, oh my God, I can so imagine these scenes coming to life. I can see it in my head. And that's when the kind of idea to then bring in like the artwork and these kind of more static memories, because we didn't, again, want to take away from the hand tracking being center to the story. But in terms of getting the interaction right, I think it was a lot of optimization, a lot of trial and error as well. Again, when we started to integrate artwork, all of a sudden, certain decisions about the artwork made the hand tracking lag a little bit. And obviously, there's certain environments where hand tracking just doesn't work well in terms of lighting conditions and things. So with a lot of fine tuning, we managed to get to a place, I think anyway, where the hand tracking is pretty seamless. You go through the experience and it does genuinely, you never feel disembodied from your hands. They are definitely, you know, yours and yours alone. But also, I hope at some point you kind of forget that they're CG almost. You kind of, yeah, you fully think that this is just your body in a virtual world. And then once we got it absolutely perfect, you know, hand tracking super smooth, artwork's looking great, then I was like, Mia, maybe we should, could we just have the hands look a little bit more like the art style? Maybe we could change the actual artwork. Which obviously, bless her, was like, all right, fine, back to the drawing board. And again, re-optimizing that to make sure that actually it's not just the blank meta hands that they have. I can't think of the right phrase for that.
[00:12:30.631] Kent Bye: It's like a default set of hands.
[00:12:31.992] Alex Ruhl: Exactly. Instead of the default set of hands, they're a little bit more in keeping with the world to make it feel as well like you're kind of, you are really there.
[00:12:40.607] Kent Bye: And as the theme that you started with, with equality, you're taking that outside of the social political context and putting it more into a parent-child relationship where there is an asymmetry of power and oftentimes for good reason because the parent often is trying to set boundaries that are trying to protect the child. But, you know, sometimes those decisions don't have to be as from that position of authority and power that can be more on an equal level. And so I really liked the structure of how that was able to be used within the context of the story. And so I'm wondering how you took that idea of equality and having something that was democratic in the sense of equal odds, of 50-50 odds of winning or losing with two people playing this game, how you took that from the original context of social political and thinking about the parent-child relationship, and how do you start to then map out the structure of the story as it starts to unfold?
[00:13:33.149] Alex Ruhl: Yeah, good question. I mean, the very first prototype version, there was kind of a little bit more of like a slightly cliched clangor of a, this is the message, where the mum actually just kind of comes out with it in the end and is like, she plays a hand that is not a rock, paper, or a scissors. It's very like Friends, you know, Friends when Joey plays fire. It was very much like that. And again, just kind of like she reaffirms, you know, sometimes this is going to happen in live and you need to realise that you don't have the same choices. It was very like, hammer over the head. And again, after workshopping it, it just felt a little bit too like, I don't know, perceiving your audience is not clever enough to kind of read into it what they want to read into it. So instead, like you say, the way we kind of went about it in the end was like starting with these very like novel, trivial type things that you're playing for, you know. Do you get to watch TV for an extra hour? Have you got to go to bed? Like, who's going to wash up the dishes? Or are you doing homework? You know, we started quite trivial. And then you kind of build to this climax where, again, like, on a surface level, it seems very juvenile, the kind of thing that mum and daughter are arguing over. But the kind of subtext of that, because, you know, without spoiling it, essentially, you get a reference to the daughter's, like, friend who is doing something in particular. the mum's kind of trying to break the news of like, OK, well, that's OK for her and her family, but we can't do that. And that's when the climax, the tension, I guess, kind of reaches its peak and that realisation of like, oh, OK, like I'm maybe not like my friends or maybe I have boundaries that my friends maybe don't have to deal with because of things that maybe hadn't occurred to the daughter up until that point so I'm hoping that you know we've taken something that is a little bit more removed from emotion and we put it into a context of like a parent-child dynamic and it's the same thing we're saying the same thing we're still making that social commentary but just in a much more subtle way that makes you Yeah, it's funny, the parents that go through this experience, they come out of it like, oh my god, that's so my child. Or like, you know, parents with young kids are like, oh wow, yeah, like, that feels so true to what I'm imagining is gonna happen when my kid grows up and they get their own agency and they get their own independence and they have their own mind and all of a sudden they start to see the wider world that's impacting them and their life, not just the little bubble, the four walls that they live in. So I don't know whether that answers your question, but yeah, that's kind of how we manage that transition.
[00:16:12.696] Kent Bye: Yeah, yeah, that makes sense. And I think, you know, there's moments in the story where after you do a round of rock, paper, scissors, you can either win, lose or tie. And so are there ways in which that you are going down different story branches based upon what the outcome of the interaction is?
[00:16:30.401] Alex Ruhl: Yeah, so this was something that we really, really would have loved to kind of explored even more, but just budget restraints meant that we couldn't be as branching as maybe we would have liked to. But essentially, yeah, so there's essentially like three main paths that you can go down, like you say, win, lose or draw. And for each one, the narration, the audio from the different characters plays out differently. And what we wanted to do is we wanted to give it that feel of branching. We wanted to give it that feel that the story is changing no matter what the outcome is, because it kind of is. but also making sure that there is this golden path to where we want you to go. So ultimately, no matter what happens, no matter whether you win, lose or draw, you end up in the same place. It's just that those scenes play out slightly differently. And like I say, you know, it would have been amazing to have enough funding to like maybe branch off into different memories and see how that relationship is different, like depending on whether you're win, losing and drewing. But yeah, for this particular version of the piece, it's purely like the narration difference. But even that was super fascinating because this is my first like fully animated piece. And what was so fun about that is because I come from a filmmaking background and traditionally work with actors in photorealistic circumstances like 360, like volumetric. What was fun about this is like, oh wow, like if it's just narration, it's slightly easier to code different paths for narration than it is visuals. So we had a lot of fun playing with my actors to get them to do like just different takes like oh you know what happens if like someone doesn't play a hand what if they're too busy like looking at the artwork like what do we want the mum to say and in each scene that's slightly different because it's a different context right so in the first scene you're in like a living room so it's like you know are you playing or not or should you know are you gonna head off to bed like it's thinking of every possible interaction that the user could have with the piece and scripting it so that everything feels really personalized and bespoke.
[00:18:28.857] Kent Bye: Yeah, it's the win, lose, draw, or fail, in some sense, of not detected. And I'm curious if each of the rounds of rock, paper, scissors, if it's randomized, or if there's any tricks that you're doing in terms of mirroring the hands at some point to determine a certain outcome.
[00:18:44.653] Alex Ruhl: This is really interesting. So actually, the original prototype, which was much more based on that kind of more higher level socioeconomic commentary, we basically based the coding and the AI plays based on an academic study that was done on rock, paper, scissors, right? which basically was like, yeah, you think this is a fair game, but actually, there's a bit of human behavioral psychology involved. So for example, and this might be giving something away if someone's about to go play rock, paper, scissors, but essentially, for example, if you tie, statistically, you are likely to go to the next one in the sequence. So if you played rock and you tied, you would most likely go to scissors. And so the AI knows that and stays on rock because it knows that it's gonna beat. So actually, we've only, ever had it twice where someone has stuck to that hand no matter what and eventually you know they do like five hands of that and then you know they decide to change finally but yeah and that was something that again we coded from the start because the kind of context was slightly different but we left it in because we found it really quite funny that watching people user test this Literally, it's so funny. We all think we're so random or so unique, but like behaviorally, everyone kind of does a similar thing. It's so amusing to watch. So yeah, it's based on like a study done on like 3,000 games of Rock, Paper, Scissors.
[00:20:05.740] Kent Bye: So yeah, it's not so much a coin flip, which would be 50-50. It sounds like there's actually some human behavior that makes it not so equal. It's very, very interesting. So what has been the reactions here at Venice?
[00:20:18.168] Alex Ruhl: It's been so beautiful. I was literally just catching up with the person who's facilitating it for us at the festival and I was getting kind of like the inside scoop on who are the kind of people that have been doing it.
[00:20:29.327] Kent Bye: This is the the docent who's been managing each of the people coming in and out.
[00:20:32.807] Alex Ruhl: That's right, that's right. And he was saying, you know, everyone from younger adults that are like students that are coming, like who come out being like, oh, wow, like, you know, that was really sweet and sweet and charming is like the two key words that we keep getting, which is exactly what we were going for with this kind of piece. But also right through to like 80 year old Sicilian grandmas who don't speak any English coming in and playing it and again, feeling really quite like connected to the story. From that to like, like I say, a lot of people with kids who have come up to me and said, you know, I almost shed a tear at the end because I just felt so emotional about my own relationship with my own kids and how that's like evolving and, you know, seeing them grow. And there's something about that, I think, that moves a lot of parents when they see it. So the reaction has been phenomenal. You know, more of the kind of like technical eyed people are very much complimenting the hand tracking and in fact that was one thing that Liz obviously one of the curators was like you know it's one of the most seamless hand tracking experiences that she's ever done which is obviously a huge compliment so all around I feel like we we achieved what we set out to achieve which was create this little story that was driven by this very specific mechanic and yeah gives people a nice warm feeling when they come out of it.
[00:21:47.125] Kent Bye: Yeah, the simple story well told, but also the technology integrations, yeah, it was pretty seamless for me as well. And that's not always the case with the different hand tracking experiences. So I thought it was really well executed in the technical aspects, but also integrated into the story in a real seamless way. So yeah, I think it all ties together. And so, yeah, I guess what's next for the project?
[00:22:09.282] Alex Ruhl: I mean, hopefully, I said on a panel earlier, we never anticipated getting into Venice. We were like, we'll apply because you got to apply. But we had no idea that we were going to get in. So it's so wonderful to world premiere here at Venice, hoping now to take it to, you know, one of the other kind of top tier festivals that would be amazing, ideally an American festival. So obviously we'd love to have it in South by or Tribeca or one of those festivals just to kind of, you know, get it on the other side of the world and have like a nice splashy premiere over there. And then after that, you know, really just looking to kind of get it out on the store. And also because this is actually the first VR project funded by the BFI Network, I feel like there's going to be a really strong campaign accompanying this in terms of now what that means in terms of funding regional talent in the UK. So BFI Network, British Film Institute. the main film body funded quite a few pieces already but this is the first time that the network specifically has funded something in VR and with that that's going to open up a lot of doors for you know independent creators that they might look at this medium and go that's too scary because that seems very technical and you need these huge budgets but actually we've proven with Rock Paper Scissors you know that budget was about 20K, just a little bit less, that you can actually achieve something pretty marvellous, as long as you have a good story and, like you say, just really focus on making the technicals as seamless as possible. So, yeah, I hope to do a tour of the UK with that and hopefully encourage new filmmakers into the medium.
[00:23:36.401] Kent Bye: Awesome. And finally, what do you think the ultimate potential of virtual reality and immersive storytelling might be and what it might be able to enable?
[00:23:47.123] Alex Ruhl: This is a podcast all in itself, isn't it? I mean... I mean, I personally, my holistic opinion on the metaverse, as we're calling it, is basically an embodied internet, right? It's the next generation in which we're experiencing the internet. I think that VR will essentially become what the smartphone is today, the vehicle in which we experience the internet. And so I think that right now, we see these very siloed communities, VR gaming, VR for enterprise, VR storytelling, VR theater, you know, it's all separate social VR platforms. But I think the ultimate potential of this is to be simply the glasses of the way in which we access everything and everything will be possible. So you could, like, happily jump into, you know, like a secret cinema type experience, which is, you know, immersive theater, essentially, right? You could do that in VR, but you could also then step into a 360 world and just kind of, like, be led on a passive, more of a passive experience. Or even just like things like right now, We have the ability, obviously, to record everything video-wise with our phone and that's way better than sending a photo. You know, I was literally just like, walked the red carpet and was like, gotta send a video to my mum, she's gonna be so proud. You know, what about in the future when our VR glasses can record that moment in 360 and now I'm sending that to my mum so she can instantly play it back on her glasses. So I think for storytelling, I think storytelling is the heart of everything we do, no matter what sector you come from. And VR will just be the access that we need, really, to fully go to that next level of presence with content.
[00:25:23.288] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:25:28.252] Alex Ruhl: just keep doing your thing and if you're listening to this like I know a lot of people obviously listen to your podcast and I just think so many people they look at especially like a selection like here at Venice Immersive and they feel I know I've definitely felt that in the past like intimidated like my work is not going to be good enough maybe the stories I'm telling aren't relevant enough maybe they don't deserve to be told but I think Rock Paper Scissors is getting into this festival is proof that if you've got a story you want to tell if you're passionate about telling it if you're passionate about jumping in and just learning and being amongst the pioneers that are discovering how to use this medium, then just do it. Otherwise, you will regret it so much.
[00:26:08.011] Kent Bye: Awesome. Well, yeah, I just really enjoyed the piece. I feel like it was integrated and really solid and seamless in its experience and everything else. And so, yeah, congratulations on having it here and thanks for sitting down to help unpack it all.
[00:26:20.262] Alex Ruhl: Thank you so much for having me.
[00:26:22.913] Kent Bye: So that was Alex Rule. She's the creator of Rock, Paper, Scissors. That was showing at Venice Immersive 2022. So if you want more context for the wrap-ups, then I'd recommend checking out the episode 1121, where I talk about all the 30 pieces in competition. And in episode 1144, there's an immersive panel that I did at Venice with some other immersive critics talking about the art of reviewing immersive art and immersive entertainment. I recommend checking that out in order to dig into a little bit of my own process of what I'm trying to do with these larger series and trying to unpack and discuss the art and science of immersive storytelling with a lot of these different pieces that we're showing at Venice Immersive 2022. So that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
