Adventure Time creator Pendleton Ward has been fascinated by the idea of virtual reality since he first read Snow Crash as a teenager. He backed the Oculus Kickstarter, and has been exploring many of the early VR prototypes over the last three years and has started to create interactive stories. Little Pink Best Buds has some VR components and was made as a part of Double Fine’s 2-week Amnesia Fortnight game jam in 2014. Pen is currently exploring the bounds of identity in VR through a new adventure game and story that he’s working on, and he’s been taking inspiration from many different early VR prototypes but especially the out-of-body experiences he had in Robin Arnott’s Sound Self VR experience.
I had a chance to catch up with Pen at SIGGRAPH where he talked about Sound Self, contrasting his ability to express identity in real versus virtual spaces, how VR allows him to get out of his head, recreating a scene at the Black Sun virtual nightclub from Snow Crash, his explorations in social VR, and where he sees the metaverse is heading.
LISTEN TO THE VOICES OF VR PODCAST
For more information about identity in VR, then check out my interview with Mel Slater, who is one of the leading researchers in exploring presence and virtual identity. I’ve also interviewed a couple of Mel’s students including Domna Banakou and Nonny de la Peña who have both explored the virtual body ownership illusion.
Also check out this summary of research from Mel’s Event Lab on their findings about Positive Illusions of Self in Virtual Reality
And here’s another video exploring the Time Travel Illusion in VR
Here’s Pen’s pitch video for Little Pink Best Buds, which was the experience that was selected on to be made as part of Double Fine’s Amnesia Fortnight 2014.
And finally, here’s a couple of cartoons about VR from Pen as well as a Tilt Brush drawing he made using a ladder:
https://twitter.com/buenothebear/status/729216455632289792
https://twitter.com/buenothebear/status/717597740566319104
It's #Tiltmouse time. Come on, grab your friends. (Adventure Time creator @buenothebear used a ladder.) pic.twitter.com/QfTd2js9BV
— Google AR & VR (@GoogleARVR) June 21, 2016
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. My name is Kent Bye and welcome to the Voices of VR podcast. On today's episode, I have Pendleton Ward, who is the creator of Adventure Time, and he's been diving into virtual reality over the last three years since he was one of the original backers of the Oculus Rift Kickstarter. So he's been exploring all the different virtual reality prototypes, especially a lot of the social VR experiences, and is in the process of creating his own adventure game that is stretching the bounds of his identity in virtual spaces. So we're talking to Pendleton about some of his stories of getting into VR and some of his creative process and what he's working on and how VR allows him to get out of his head and feel like he's not on this earth and another reality and traversing the universe. So that's what we'll be covering on today's episode of the Voices of VR podcast. But first, a quick word from our sponsors. Today's episode is brought to you by The Virtual Reality Company. VRC is at the intersection of technology and entertainment, creating interactive storytelling experiences. The thing that's unique about VRC is that they have strategic partnerships with companies like Dbox, which is a haptic chair that takes immersion and presence to the next level. So they're making these digital out-of-home experiences for movies, studios, and original content. For more information, check out thevrcompany.com. Today's episode is also brought to you by The VR Society, which is a new organization made up of major Hollywood studios. The intention is to do consumer research, content production seminars, as well as give away awards to VR professionals. They're going to be hosting a big conference in the fall in Los Angeles to share ideas, experiences, and challenges with other VR professionals. To get more information, check out thevrsociety.com. So this interview with Penn happened at the SIGGRAPH conference happening in Anaheim, California on July 26, 2016. So with that, let's go ahead and dive right in.
[00:02:10.322] Pendleton Ward: My name is Pendleton Ward. I created a cartoon series called Adventure Time, and I want to make a game. I'm trying to learn Blender. I'm looking up tutorials on YouTube. Teenagers teaching me really shitty, fucking long tutorials on how to learn Blender, and I'm working with a friend of mine who's working in Unity, and we're just working together, and we're trying to make an adventure game. I don't know what that looks like yet in VR, but that's what we're working on.
[00:02:35.943] Kent Bye: We were just sitting here talking, and you were just telling me about an experience called SoundSelf. So why don't you tell me about what your experience was like?
[00:02:43.066] Pendleton Ward: I only had one demo set up for me at GDC. It was with Robin for SoundSelf. I went to his hotel room, knocked on a hotel door, and he opened it, and someone was already inside the experience. We were being very quiet. Anyway, flash forward, that person was done, and then Robin made me tea. then we're all talking we're like whispering to each other and then he laid me down in his hotel bed and he put me on top of a vibrating bass pad the sub pack and then he put the headset on me in the headphones and then I laid down in his hotel room at midnight and just screamed all kinds of different like tones I was like oh And I've tried to make my friends experience it the same way I experienced it because Robin's called it this before and I keep calling it powerful, this powerful, like VR is powerful, right? Like the hardware, it's innate, it's transportive, all these words. But my favorite experiences are ones where I don't feel like I'm on this earth anymore, but I'm also not in this reality anymore. And I'm so far from it. I'm in this abstract space. My body isn't what I'm used to anymore. And people say it's this dual thing in VR. And I feel like I'm still in my living room when I'm doing this. But I also am traveling the universe when I was experiencing sound self. I had this moment where I was going, ah! And then there's all this feedback. You can't totally hear your voice, at least the way that I was experiencing it, because everything you're saying is being interpreted by the software and it's coming back at you with low moaning, groaning sounds. So you can't totally hear your voice. I felt like I was communicating with something. And I don't normally talk like this. I've noticed a lot in your interviews people start to sound spiritual when they're talking about virtual reality and they use a lot of words like essence, you know, like literally they start using words that are pulled from like a spiritual like library. It's like it's a it's cool. It's these words that were in the domain of Yahoo's and hippies, and now it's just natural for technicians and people who are engineers to be using all these words, like presence, anyway.
[00:04:51.789] Kent Bye: Well, one thing I would say about that is just that the thing that I find so interesting about virtual reality, especially after getting into artificial intelligence a little bit more, is that they're both reflections of our humanity in different ways. a lot of ways artificial intelligence being, you know, how are we intelligent? What is intelligence? How do we encompass all the different parts of our complexity of our brains? And it's a little bit more left brain and objective in that way. And I feel like virtual reality in a lot of ways is more of the qualitative aspects of our experiences that isn't necessarily so much about like something that can be reduced down to a set of like objectified rational terms, but It's really like this qualitative and very subjective experience of being present. And I think that that's the quality that people are trying to achieve with these experiences, is to give someone this experience of being present, of just giving this sense of essence of taking them into another world that doesn't even exist in this reality. After doing all these different interviews in VR, I do feel like it's like this very qualitative exploration of technology and apply it in a way that, is it fun? Is it enjoyable? Is it something that you want to go in and experience? And I think that that's sort of the benchmark of VR is something like an artist like Khabibo with Loon, some of these kind of transformative experiences or irrational exuberance where you go in and you kind of have like this almost mystical experience where it's like making you ask all these existential questions about who am I and why am I here? And I think that to me is what makes both of those technologies, both VR and AI, so exciting because actuality it's just a mirror of as we look into these technologies that we learn more about ourselves.
[00:06:34.460] Pendleton Ward: I thought of a few different things to say, when you were talking. One was, so I was dancing in my living room the other day, and then my back hurt. And then I was like, fuck you. And then I was like, fuck me. And then I was like, who am I? And then I started thinking about my body. You know, when I think about me, it's what's everything that's inside of my skin. You know, my skin's as far as me goes. And in VR, you can switch all that shit. You know, everything is up in the air because you can have a new kind of body. I could control seven. And then I was thinking about controlling seven. I was thinking about letting me move my hand and then it would move seven other bodies. And does it have to be all within the same skin? Do I need to model, like, a thing that's, like, seven people, but they're all somehow, like, gooped together in this weird, gross way? Or will I feel like me is seven different bodies altogether? And that's stuff we can experiment with now easily. Like, you can make that in an afternoon and decide, yeah, I can be seven things. You know, that's exciting.
[00:07:31.856] Kent Bye: Wow, that's a really interesting idea. I haven't heard of that before, but I think that, you know, there's this thing that happens in VR that's called the Virtual Body Ownership Illusion, that when you go in there and you have one-to-one tracking with your limbs, you start to identify your body. And there's this experience called The Wave, where it premiered at GDC, and then I had a chance to try it at Silicon Valley Virtual Reality Conference this year. You're a DJ and you're up on a stage and there's a button that you can press where you actually teleport into the audience and you're watching yourself on stage be the DJ. And immediately I started waving my hand and I realized, oh, that's me. Like I'm mirroring myself. But then because of that, I started dancing. And that's what Aaron Lemke said is that when you sort of take yourself outside of your body and look at yourself, that like more often than not, like most people start to dance because you start to just mirroring yourself. But you're just talking about like controlling seven avatars It's like I've never really even thought about that But it makes so much sense that you could have that same sort of virtual body ownership Illusion of like you having your identity projected into seven other entities, and what does that feel like?
[00:08:37.818] Pendleton Ward: No, you know the sky's the limit right now with what the things could feel like that's what makes me excited about VR
[00:08:45.504] Kent Bye: So there's a lot of on-the-edge experiments, it sounds like. You're kind of pushing your sense of identity, but storytelling, like, what is your curiosity that's really driving you with VR in terms of pushing the bounds of what's possible to experience?
[00:09:00.297] Pendleton Ward: I mean, it relates to what I was just talking about. Identity, stretching the boundaries of what I think I am. So right now, I feel crazy because I'm being interviewed. And I'm thinking about the pacing at which I'm speaking. I'm thinking about all of my mannerisms as this learned thing that I've stacked up to build my identity. I know that I'm talking a little louder right now. I'm trying to be entertaining because I'm not just talking to you. I'm also thinking about this thing that I'm going to put out to the world. I don't want to be like that. When I'm talking to my friends, I don't talk like this. Sometimes I do, but not all the time. like this, and I don't say much. You know? And I think about this too often. I feel crazy all the time. I think about this all the time. And so that's why I love VR. I'm excited to get out of my head. I'm sort of working that theme into a story I'm writing, and it's taken on a lot of different forms. It's so hard to land on an idea to execute, because the potential is out the door. I'm scared to talk about my game. I don't want to reveal it. Mainly because it's a story and so that's what I ran into when I was working at Double Fine. There was a whole documentary about the project that I did there and I just couldn't talk about it because it was a story and if you start talking about anything beyond the first act it spoils the experience.
[00:10:22.573] Kent Bye: Well, we don't have to, I think it's fair to not spoil it. For you, what are the things that you're finding inspiration from? The influences that you're taking from what's out there in VR now?
[00:10:32.996] Pendleton Ward: It's tough. Everything's inspiring. That's why I troll forums and download every DK1 demo when I just had the DK1, because everything was awesome. It didn't matter what it was. It's interesting now, on the Steam store, there's stuff coming out that's garbage. And everyone knows it. And it's cool now that you can tell the difference, that there's a distinction. Because before, all garbage was beautiful and an amazing experience. Just because of the hardware really because of the power of virtual reality, but now there's a bunch of trash And you can not download some things and I'm okay with that Well, it makes me think of how did you even get into VR then like how did you dive into this new immersive technology? Well, I helped kickstart the DK1 before that kickstarter. I just love Snow Crash. I love Snow Crash. It really had a big impact on me when I read it when I was a teenager. So the first thing I modeled in Blender was the Black Sun from Snow Crash, which is a dance club. and the street. And my friend built these like light cycle like devices, these bikes. We called them Mops. They're mobile orientation platforms. And we rode up and down the street using them, zipping up and down right by the black sun. And it was my dream come true. It was like my fantasy. It was so fun. And I was like, we've got to upload this to the internet. We've got to show everyone this. And he was like, it's not that impressive. And I was like, eh. And then I was like, yeah, OK, let's wait and make something good, and then we'll put it online. Oh, wow.
[00:12:00.593] Kent Bye: Nice. What do you want to experience in VR?
[00:12:04.788] Pendleton Ward: I'm more excited about the different forms of the metaverse and how those are going to come. I've played Second Life for a really long time. I'm excited about social VR more than anything else, I think. It's going to be fun to see how people interact. I know AllSpace, I read an article that they just introduced personal space bubbles, which is really exciting to me that everyone has personal force fields. now, and you need them, you know, you need them, you need that kind of stuff in VR, and I'm excited to see how virtual culture grows. Because everyone right now has the best intentions. Maybe not anymore now that it's actually released, now that it's going straight into the gutter, like really fast. So maybe the best intentions actually that everyone had were maybe a few months ago. and everyone just was talking about making the Metaverse this place where people can come together and there's just going to be like a library of, like I fantasize about this perfect like downtown. I keep thinking about how you can grow a culture. I just wanted to fill like every street corner to have like this hologram. Hello, nice to meet you. What would you like to learn today? You know, this like some philosopher, you know, that's projected out of every street corner.
[00:13:08.380] Kent Bye: The thing that I think of when I think of cultivating culture within VR is I think of something that's constructed in real life like Burning Man for a week is like all these people come together and fosters a certain amount of creativity that I think is being infused into our culture outside of Burning Man. So it's a seed that's planted and then it is brought back into people's communities and then is grown and really takes root. And I think that that to me is what is really super exciting about VR is because you get people who are of like mind and you have these emergent behaviors that only happens when people like share the same values and beliefs and they're kind of able to create reflections of their culture and I think VR is going to provide that opportunity for them to at a very low cost create big huge environments that embed a new vision of what our culture could be because right now we're basically limited by having everything centered around cars and so just that alone like what is a culture that is centered around people moving around in a certain way in a virtual space. It's going to create more of a village like structure rather than like cities with like lots of parking garages. We're not going to have any parking garages in VR because we don't need them. And so just to think about like seeding culture by creating environments and interactions in VR that then from there can be brought back into our real life and then make the world that we really want to see.
[00:14:31.452] Pendleton Ward: Yeah, that's the lofty goals. I mean, what's going to happen is you're going to create this, like, awesome metaverse downtown with, like, libraries and lecture halls where you could go learn anything you want. And you can, like, pop into a room and see, like, real-life doctors working that are being recorded live and broadcast. But what's going to happen is, like, people are just going to hop in a portal and go to, like, casinos and orgy places and just, like, hang out there. And they're never going to go to that beautiful downtown space, because you can go anywhere in the metaverse. It's what's going to happen. It's already happened. And so you can't control it, especially when you give up, when you let people build what they want to build.
[00:15:15.410] Kent Bye: What do you see as kind of the ultimate potential of virtual reality, and what it might be able to enable?
[00:15:20.895] Pendleton Ward: I don't know. Come on Yeah, that's all I'm gonna say about it Okay, well, thank you so much.
[00:15:41.706] Kent Bye: Thank you So that was Pendleton Ward He is the creator of adventure time and is in the process of creating a VR adventure game exploring identity So I have a number of different takeaways about this interview. First of all, I think it's a really fascinating idea to try to see what the bounds are of virtual embodiment of multiple different characters. I think most of the different studies that I've seen about virtual body ownership illusion have really focused about trying to invoke the illusion of body ownership with a first-person avatar. One of the leading researchers in this field is Mel Slater, and I had a chance to interview him back in episode 183, as well as one of his students in episode 251. So those are great episodes to hear a little bit more about this virtual body ownership illusion. But the basic thing that is required in order to invoke this is this one-to-one body tracking of your limbs. Especially if you have at least your hands, you can start to do it. But if you start to have your feet as well, it starts to really induce it. So that's sort of what I think a lot of consumer VR can do. And there's some other ways to really induce this illusion, but that kind of requires having somebody with you there also giving you some synchronous tactile stimulation. So somebody touching you in real life, but also touching you in a virtual world. And that's something that can very quickly induce the virtual body ownership illusion. And anytime you're correlating your visual motor correlations between your real and virtual body, again, that's a limb tracking, but also the head movements of anything that you're moving around and you're kind of seeing what the virtual character is seeing. Now, I was able to kind of have this virtual body ownership illusion with the Wave, where it was kind of like this out-of-body experience where I was in first-person perspective, but I switched to third-person perspective, watching myself on stage, and having that visual motor correlations with my hand being tracked, I was very quickly able to identify that. A lot of my movements that I was doing were clearly me. And so it was kind of like this surreal experience of seeing yourself in a mirror, but within a virtual space. And so just imagine being able to kind of replicate that amongst seven different avatars that you're puppeting at the same time. Now, I like the idea of kind of adding fractal noise between those so you're not doing the exact movements, but it's close enough to you to identify that whatever movements that you're making, you're starting to move multiple different avatars. having the ability to embody many different characters and jump around between them I think is something that's very unique to VR and something that I just talked about here in episode number 408 with Katherine Raewinkle talking about situated knowledges and one thing I just wanted to bring out about that interview again is that And that concept of situated knowledge is you're kind of moving from different perspectives within the scene and you're really focusing on your physical geographic location and that is going to inform what you're able to physically see within the context of a story. But imagine if you take something like Perspective by Rose Joche, who I interviewed back in episode number 286. Essentially what they had to do in Perspective is shoot the same events four different times, and the actors had little slight little variations. And the thing that I really liked about that is that it wasn't the exact same action that you saw from different perspectives. And in a lot of ways, that actually kind of mirrors how our memory works, because we're kind of filtering these events through our own preconceived notions. And you can imagine sort of an extreme version of this, where if you're watching the same story from different vantage points, is that you actually may hear completely different dialogue, or the dialogue may change, or the events may change, or you start to not really able to know what the truth is, because that's kind of more accurately portraying how eyewitness accounts can actually happen is that people have different versions of the story that actually happen. And so in thinking about different identities and jumping between identities and seeing a story unfold, then you can start to play with trying to mimic someone's actual worldview based upon the lenses that you're showing into the world and what they're actually able to see. And you can imagine how you're able to start to play with that. whether someone's in an altered state, whether or not they're drunk, or whether or not they're going to see different things, or there's going to be specific things highlighted from different perspectives. So I think that this combination of the virtual body ownership illusion as well as trying to mimic these subjective perceptions is something that Virtual reality is uniquely poised to really explore, and I think that Pendleton is really interested in trying to explore the bounds of how far he can push the limit of almost having these out-of-body experiences, or not just out-of-body from kind of taking him out of this physical world, but perhaps even giving him a deeper sense of presence through the embodiment of these virtual characters. There's a video from Mel Slater that really goes through all the kind of latest cutting edge research in terms of illusions of self within VR. And one of the clips that they show is from Nani de la Pena's work with Mel Slater, where she was actually putting people into like these torture positions within VR and seeing how they reacted. And people actually started to think that they were being bound up and kind of like on their knees with their hands behind their back. Even though the real body wasn't in that position, they had this perception of being able to see themselves in the virtual world and looking into a mirror and seeing that they were bound up. And then they had this kind of reaction that they were actually being tied up. And so there's lots of really interesting aspects for how we can start to explore our identities within VR. So I'd encourage you to take a look at the video from Millslater to get a little bit more about that research. And the other thing that I wanted to mention is just this concept of time and time delays, because Time Rifters was an example of a game mechanic where you're going into VR, you're going through a scene, and then you actually go back and you go through the same scene and you're playing kind of this multiplayer co-op game with yourself from a previous iteration. And so you just start to layer your experiences on top of each other. And so Mel Slater also had this really fascinating experiment about the time travel illusion, where they invoke the virtual body ownership illusion. You're going into a scene and they give you this sort of moral dilemma where you have to make these different decisions and then at some point you find out that based upon those decisions kind of dictates how many people end up getting killed within the scene. And so then you get come back to the same scene a week later and you are from a different perspective watching yourself do all these actions and people report having this sense of like your time traveling and you're able to actually start to intervene in the previous iterations of what happened in that scene and so that's yet another example of starting to play with this time dimension within VR where you're recording your previous actions and then able to revisit them and kind of watch yourself, which is in some ways, perhaps cultivating this witnessing consciousness where you're actually able to take a step outside of yourself and watch yourself from a third person perspective. And I think this is going to be an area that's going to be really interesting to see what people start to do with playing with this idea of invoking this sense of virtual embodiment and then playing with invoking this illusion of time travel. And I just really appreciate all the different anthropological observations and reflections of kind of the last three years of diving into this virtual reality revolution that's been happening. So that's all I have for today. I am going to be at VRLA later this week, so keep an eye out for me there in Los Angeles. And if you enjoy this podcast, then please do consider telling your friends, spread the word. And if you'd like to help support the podcast, then please consider becoming a contributor to the Patreon at patreon.com slash Voices of VR.
