Within premiered their first real-time rendered, interactive experience at Sundance New Frontier this year with Life of Us, which is the story of life on the planet as told through embodying a series of characters who are evolving into humans. The experience is somewhere betweeen a film and game, but it’s more like a theme park ride. There’s an on-rails narrative story being told, but there’s also opportunities to throw objects, swim or fly around, control a fire-breathing dragon, and interact with another person who has joined you on the experience. You learn about which new character you’re embodying by watching the other person embody that creature with you, and the modulation of your voice also changes with each new character deepening your sense of embodiment and presence.
I had a chance to catch up with Within CTO and co-founder Aaron Koblin at Sundance to talk about their design process, overcoming the uncanny valley of voice modulation delays, how the environment is primary feature of VR experiences, and how their background in large-scale museum installations inspires their work in virtual reality.
Koblin also talks quite a bit about finding that balance between the storytelling of a film and interaction of a game, and how Life of Us is their first serious investigation into that hybrid form that VR provides. He compares this type of VR storytelling to the experience of going to a baseball game with a friend in that this type of sports experience is amplified by the shared stories that are told by your friends. This is similar to collaborative storytelling of group explorations of VRChat, but with an environment that is a lot more opinionated in how it tells a story.
LISTEN TO THE VOICES OF VR PODCAST
Life of Us is a compelling way to connect and get to know someone. The structure of the story is open enough to allow each individual to explore and express themselves, but it also gives a more satisfying narrative arc than a completely open world that can have a fractured story. Life of Us has a deeper message about our relationship to each other and the environment that it’s asking us to contemplate. Overall, Koblin says that our relationships with each other essentially amount to the sum total of our shared experiences, and so Within sees an opportunity to create the types of social & narrative-driven, embodied stories that we can go through to connect and express our humanity to each other.
Here’s a trailer for Life of Us.
The Life of Us experience should be released sometime in 2017, and you can find more information about Within website (which links to all of their platform-specific apps), or their newly launched WebVR portal at VR.With.in.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR podcast. So at Sundance this year, I had a chance to go through all of the different virtual reality experiences that were a part of the Sundance New Frontier program. And probably the most visceral experience that I got to have was Within's Life of Us, where you essentially embody a range of different characters on this evolutionary journey. And it's the story of humanity as it's evolved. As you embody these different characters, you can have your voice modulated, so that helps with the embodiment. But you also get to go through the experience with another person. So you get to discover more about these creatures that you're embodying by watching someone else be embodied by that character. So I'll be talking with the co-founder and CTO of Within, Aaron Koblen, about the process of creating Life of Us on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by the Silicon Valley Virtual Reality Conference and Expo. SVVR is the can't miss virtual reality event of the year. It brings together the full diversity of the virtual reality ecosystem. And I often tell people if they can only go to one VR conference, then be sure to make it SVVR. You'll just have a ton of networking opportunities and a huge expo floor that shows a wide range of all the different VR industries. SVVR 2017 is happening March 29th to 31st, so go to vrexpo.com to sign up today. This interview with Aaron happened at the Sundance Film Festival happening in Park City, Utah from January 19th to 29th, 2017. So with that, let's go ahead and dive right in.
[00:01:59.986] Aaron Koblin: My name's Aaron Koblen. I'm the co-founder and CTO of Within. We're a virtual reality company. Up until now, we've been mostly focused on photorealistic, cinematic, immersive virtual reality experiences and stories. And we're just now branching out to do real-time rendered interactive storytelling with multiple people. So that's a big part of what we're doing here at Sundance, is launching this project called Life of Us, which is Chris Milk, myself, and Pharrell Williams' work to put together this multi-user journey.
[00:02:26.432] Kent Bye: Yeah, and it's exactly that, this multiplayer embodied experience within virtual reality through some sort of narrative that's unfolding. So maybe you could describe the experience.
[00:02:35.739] Aaron Koblin: Sure. So a little bit of background. I think Chris and I both got into cinematic virtual reality pretty early on. He did a project with Beck, which was the 360 video experience. And then when we saw it in HMD, I really realized this is something totally different than what you see on a screen. This is something that you kind of opt out of, not opt into in terms of suspension of disbelief and feeling like you're actually there in that environment. The first couple years of Within, it was originally called Verse, but we renamed it Within, was about photorealistic, real faces, real places, and feeling like you're actually there because we felt like that really delivers kind of the deepest connection. You have more ability to relate to the environments and It triggers all kinds of memory and space and place, things that your brain is used to and it kind of has evolved to become for millions of years. I think where that changed for us recently was when we get into an experience with another person. There's no longer this need to kind of pull the humanity out of the technology. The technology is gushing with this real-life person that you're actually interacting with. So even if it's just a cube, like two cubes for the hands and another for the head, you add voice into that and you see just through those simple motions the humanity of your friend actually reveal itself. And at that point we realized, okay, this is something incredibly powerful. You can connect people from across the planet and multiple friends at the same time, and you can actually use this not just for the new form of video conferencing, but potentially something that's way more guided and crafted and artistic. So our idea was, okay, let's come up with the biggest story we could possibly tell. So Life of Us is basically the entire journey of the history of human life on the planet. So starting from primordial obviously with some artistic liberties taken and quite a heavy dose of Darwinistic interpretation. But anyway, I don't want to give it all away, but hopefully everyone will get a chance to try it out. You basically go through these different stages of evolution and you physically get to become these creatures, which is a very bizarre experience in and of itself. It's like, how do I act as an amoeba? And then once I sprout arms, oh, now I can swim and now I can blow bubbles and do all kinds of other little Easter eggy tricks for each creature.
[00:04:34.085] Kent Bye: Yeah, I think that amount of embodiment was something that was completely unique to this experience in the sense that you're embodying a number of different characters. Not only with your motion-tracked Six Degree of Freedom controllers that you're holding in your hands, but also your voice, which I think is something that I haven't seen anybody really do much yet, potentially because there's a certain amount of processing and latency that is incurred. But I think in this experience it's okay because it's like you are sending something over the wire and other people are hearing it and so there's a little bit of delay that you're hearing but I imagine that's part of the reason why we haven't seen a number of people do that yet is because that latency is there and it's a hard problem. But I'm glad that you went ahead and did it anyway because I think It adds so much to increasing the fidelity of how much you're able to communicate as you evolve, which is a fun and interesting thing. So maybe you could talk a bit about the audio component that you were doing here.
[00:05:24.804] Aaron Koblin: Yeah, so the project was filled with all kinds of technical challenges and adventures trying to dive into things we've never done before. Audio, as you're pointing out, was a huge component to it. We realized quickly on that the other person's voice coming through to you really pounds that human connection home and makes this awesome opportunity. The other strange thing is hearing your own voice. Interestingly, that's where the latency comes more into play, is you're so used to instantaneous latency. You're making the sounds, they're traveling three inches to your earlobes or even just bone resonance. There's really like no delay. I think what we were trying to do was play around artistically and creatively with some of the ways to translate your voice. Your voice when you're an amoeba doesn't sound like your voice when you're a human. So we have a lot of opportunity to play with that delay and to experiment with different types of pitch shifting and vocoding and other techniques. So I think for certain creatures we were able to get an effect that we were far happier with than others. Obviously I think the human is the hardest component to play with, but I think also dealing with some of the environmental aspects like putting you in a space where you have echo and there's certain tricks and things you can play with. I think this was a very solid first stab that we're really proud of and I think there's more that we can do as we dig in further on the technology side. I think with doing a project like this a lot of the challenge is kind of laying the groundwork to be able to do more projects like this. There's a lot of kind of R&D and figuring things out so hopefully the next ones will be a little bit easier to work on and hopefully as a community we can share some of our learnings as well.
[00:06:47.724] Kent Bye: Yeah, there's a number of different components here. There's the embodied component, which we talked about, the social component that we talked about briefly. But also there's an environmental component that is changing. And I think that it just reminds me back at Oculus Connect 1 when they were showing the Crescent Bay demos for the first time. We were put through like 13 different environments in like eight minutes or so. And so it was like this rapidly going through these completely different environments. And I think that as you're going through this story and kind of magically switching the environment and context, I think is something that is a subtle thing that maybe not a lot of people may pick up on, but that actually gives you this sense of being transported into another realm over and over and over again. And I think that's one of the things that I think probably makes it compelling for people to just feel like they have this completely wild experience into these different realms and in these different characters.
[00:07:33.736] Aaron Koblin: That's absolutely one of the things virtual reality is so obviously amazing at, is giving you the sense of space and giving you the feeling that you're there. I think a lot of people are talking about virtual reality as a blend between the opportunity being a blend between movies and games. And I think that's what Chris and I are really looking at. But even more than that, it might be a blend of Disneyland-style theme park rides with movies and stories. And sometimes they get the stories really right there as well. But I think this idea that the environments themselves are conveying so much of the power of the story very differently from, like, theater, for instance, where it's really all about the actors. We're still trying to experiment and figure out the right balance for which types of stories, but there's a lot to explore there. I think the other thing you touched on was the character embodiment, and I know, like, there's been a lot of talk about this. We're certainly not the first folks like Jaron Lanier. He talks a lot about this ability to embody different types of creatures, which was kind of an inspiration to us. The neuroplasticity taking over and teaching you, oh, this is how I fly as a pterodactyl, and this is what it means when I move this way. And potentially even doing things like adding different limbs that you don't have and mapping those onto different motions. That's one of the things Chris and I kind of fell in love with quickly was the ability to become something completely non-human and to really feel like you are that thing. One of the things we were playing with a lot was the seeing yourself in the mirror because it's incredibly intriguing and you can just get lost in mapping yourself to other things in a mirror representation. In a social experience, though, that can tend to take away from it because you see both people just sitting there looking at each other. And in actuality, it's so much more deeply rewarding if you are interacting meaningfully with the other people. So we have moments of mirror reflections, but then back to keep this a social experience.
[00:09:11.494] Kent Bye: Yeah, from the academic community, it's referred to as the virtual body ownership illusion, which means that you're able to trick your mind enough so that you identify with the virtual body that you have, that that's your body. Because the idea is that when you look down at your body, pretty much 100% of your time up to before VR, when you looked at your body, it was your body. And so when you see a direct translation to your proprioception of where you think your body is, and if you see the visual feedback of that, then your mind just gets tricked. And they were doing a lot of research into how to invoke that. Haptics is probably one of the most effective ways to invoke the virtual body ownership illusion, to have a virtual object and a physical object in a mixed reality context touching at the same time. But they found that just moving your hands around and having your legs tracked actually gives even more of an amount of the virtual body ownership illusion. So I think right now with the technology as it is, we kind of get this half level of virtual body ownership illusion. But I think the other dimension that you're introducing here is with the voice and being able to communicate along with your body and seeing other people. You don't mind that your legs aren't necessarily tracked because you can't necessarily even see them all that much. Anyway, there's so much going on in the environment and with the other character you don't really pay attention to what's going down there anyway. So I think you're able to reach a level of that virtual body ownership illusion that I haven't really seen so far.
[00:10:27.670] Aaron Koblin: Yeah, it's an interesting combination because there's absolutely that actual connection that's being created, and then there's this weird surrealist connection when you look down and your legs are running. And you kind of end up embracing that as yourself, even though you recognize that it's not really feeling right. You're like, OK, well, I'm just doing that. I'm going to let that happen.
[00:10:45.467] Kent Bye: You can actually run in place to increase the immersion, in that case, to not have that break in presence, because it is a presence breaker to see that disconnect.
[00:10:52.033] Aaron Koblin: It is, and it's fascinating to watch how different people react to it. It's a pretty 50-50 split between people who actually start running and trying to embrace it and those who just kind of accept it and continue on. The other thing that's really gratifying in terms of that linkage between the body and yourself is physics-based actions. When somebody throws something to you and you catch it, you immediately embrace the body as your own. At least that's what I've experienced. And some of the other things where you're spawning things that have physics, it just becomes so play-oriented so quickly that it becomes second nature. Some lizard aspect of your brain is understanding everything that's going on, and things are clicking, and that feels good.
[00:11:26.621] Kent Bye: Yeah, and I think the other major component, maybe even the most powerful component of this experience, is the social component of this. And so maybe you could talk a bit about the design process that you went through in order to cultivate this as a social interaction.
[00:11:38.288] Aaron Koblin: Yeah, so I'm looking forward to now that we've kind of gotten baseline one, doing even more iterative design. But the best thing to do is like kind of build these prototypes, get in it with another person, and then feel what felt right. And even more interestingly, put people in it who have never experienced anything like this and watch what they try to do. and then figure out, oh, what they're trying to do makes total sense. We should probably consider that and figure out how to play with that and experiment with that. So we certainly don't have all the answers by any stretch, and there's so many interesting contexts, but every time we put somebody new in it, we learn something new. Like, even just the idea of flying. like there are so many different ideas people have about what it would feel like to fly and the difference between flapping to raise yourself up, flapping to excel yourself forward, gliding based off of the twists in your wrists versus the leaning of your body versus the leaning, you know, there's pitch, roll, yaw, it's Some people who are pilots have a very different opinion than people who imagine themselves as birds and starting to play with that and also just the speeds. If you want this to be a social experience, you want people to discover each other and to be able to fly near each other and hang out with each other, but you also want this amazing feeling of soaring and diving and it is kind of a tension there between making it feel amazing and also allowing you to kind of have some control, the kind of control you need to be cohabitating and sharing an experience.
[00:12:56.601] Kent Bye: One of the interesting dimensions of that is that if you know somebody or you don't know somebody, you could have this experience and can learn more about them in some way. We actually did it together, and you've done it a number of times, presumably, so it wasn't your first time. So it's a little, like, asymmetrical in terms of our unique discovery of having this experience for the first time. And so you were kind of, like, guiding me through in certain ways, you know, pay attention to this, oh, look what you can do here. And so, but I'm curious to hear what you were able to kind of discern from me and my personality from going through this experience with me.
[00:13:25.093] Aaron Koblin: Well, you've clearly done a lot of VR. That's the easiest thing to realize is there are those who are just in complete awe that they are actually in another place virtually. And then there are those who are like, OK, how's this tracking working? When I squat down, what happens to my IK rig? Am I maintaining position? So I think you actually had a nice mix because you seem truly interested in the content and context, but also doing a little poking and prodding here at the limits of what's happening. And I think that's often where I and a lot of the developers net out is it's easy to get kind of lost in the technical challenges. But every once in a while you have these moments of just like, holy shit, I'm breathing fire. Like, that is awesome. And that's really what we want to get is like everybody to that moment where you just forget about everything else and you're just soaring through the sky breathing fire and, you know, becoming this creature.
[00:14:09.177] Kent Bye: Yeah, I have so much air element in my body. I just found myself really communicating a lot about what I was experiencing, like, I'm breathing fire now! And I'm just, like, yelling out because I just love to hear that feedback in my voice because it was just sort of like that was, to me, what was I had never seen before. And that was something that I was really playing with in different ways of just, like, starting to tone and scream. You know, Robin Arnett has this experience called SoundSelf, which is all about trying to replicate this peak experience of this audio reactive experience. the visuals are very abstract, but they're actually responding to your voice. And so, you know, in the terms of spectrum of control, there's that one limit where you have complete control over a thing and then it becomes a little bit of a game or an instrument that you can start to play. At the other end, you have this complete chaos where you don't know necessarily, you can't tell anything that you're impacting. And I think he was trying to find somewhere in the middle where he's able to interact, but yet, be in this kind of altered state of not knowing quite what to expect. And I think having that level of voice modulation starts to do that a little bit. Like, as you start to speak and you hear yourself in this voice, that feedback of what you're hearing actually changes how I'm communicating, which was, to me, what I found the most intriguing of the experience was that connection between as I'm speaking and as it's being modulated. Oh, that's actually changing and maybe even accessing a different part of my personality that I may not necessarily access very often.
[00:15:30.587] Aaron Koblin: There's definitely something happening there. There's actually another project by a guy named Ray McClure that I saw at Gray Area Foundation in San Francisco. It's an incredibly psychedelic trip where you're basically modulating your voice and casting basic geometries out of your head. I think that was the first time that I truly felt like, wow, I am on drugs. This virtual reality experience is totally changing the way my brain is operating right now. And one of the things we started to experiment with was exactly that delay. There's a certain amount of delay that you have to embrace. And what we found is there's kind of like another uncanny valley in the delay where there's kind of a like, OK, this feels good. OK, this feels weird. OK, this feels terrible. I can't even think. Like actually sometimes doing presentations, if you get a reverb. It actually shuts your brain down. Somebody told me that there's even a crowd control technique where people monitor the sounds of crowds and play those sounds back at the crowds. And it actually messes with their brains and shuts them up. It causes you to not be interested in speaking anymore.
[00:16:24.664] Kent Bye: Yeah, anybody that's used Skype and had that happen, I definitely have had that, where if it's too far, it's uncanny. I feel like this was sort of on the boundary. Maybe it was still a little uncanny.
[00:16:35.187] Aaron Koblin: What we tried to do was anywhere we could decrease it low enough so that it felt kind of good-ish, we would keep. And anything where it fell into the truly painful uncanny, we'd push it even a little further. Because actually, once it's far enough, it doesn't bother you anymore. It's more like being in a cave or something like that.
[00:16:49.755] Kent Bye: Oh, that's really interesting, because I did find that it was very significant. I mean, it was a long one, but that's really interesting.
[00:16:55.758] Aaron Koblin: The easiest and safest thing to do is make it not your voice. So if it is falling into that uncanny valley, if we pitch shifted it and vocoded it and changed it substantially, it didn't bother you as much. At least that's what I found personally, was if I was speaking in a robot voice or even in a very high pitched amoeba voice that was fluctuating in tone. even if that delay was kind of in the unpleasant spot, it didn't sound like me and it didn't bother me that much. So there's a lot to play with there. Of course, the ideal solution is just get the lowest latency you possibly can get and make it feel great. But no matter what, I'm convinced there's still gonna be a little bit of a druggy reaction if you don't hear your voice and there's any delay on it because you think your brain's trailing, I think is what it is.
[00:17:35.558] Kent Bye: Yeah, actually, as I think about it, there is a certain amount where, you know, on that spectrum of control and out of control, I think you actually kind of go that entire spectrum where you sort of go into this realm where you can't control it, and then that's where I start to maybe use my voice as a musical instrument or try to control it in some way, but just trying to play it and, like you said, just really poking prod at the edges. A couple of other things I wanted to come back to in terms of embodiment was that, you know, there's a couple of experiences here at Sundance. There's chocolate where you're actually looking in a mirror for a good 15 or 20 seconds where you just have a chance to look at yourself like this is you, this is your body, and you kind of move around. And then when you're in the experience, I noticed that that actually did a great job of being like, OK, this is my mental projection of what my body is now. Because when we just see ourselves from the first person perspective, we can only really see our limbs. And so as you're rapidly iterating through these different embodiments, it takes a little moment. And actually having a person on the other side that you can kind of mirror, it actually, I think, increases that level of, like, it's not you, it's not being mirrored, but it's the other person. But being able to see them from that third person perspective actually deepened my level of embodiment with each of the characters.
[00:18:42.239] Aaron Koblin: Yeah, that was very much our solution was because this is a social experience. The initial thought was we need to have a mirror in every transition and every scene to get you reacclimated to who you are. And then, of course, we realized with your friend there, as long as early on we've established you were both the same thing, then that really does take over and it becomes obvious to you that you're now that other creature. Yeah, I think that's a pretty effective method as well, seeing a friend as you.
[00:19:04.318] Kent Bye: And the other experience that makes me think of is Mindshow, which is the process of embodying a character in that you're doing this improv acting with yourself, taking each part separately and kind of recording it against yourself. But with Mindshow, it's really on the other end of the emergent behavior rather than the authored story that you have here in Life of Us, where it's like pretty much on rails. You know, you don't have much ability to change the course of what's happening, and you're just kind of interacting with the limited amount that you can move around. And so I'm just curious to hear some of your thoughts of, you know, this is like the first step that you've done. Are you going to start to experiment with other more global agency interactions with allowing people to really engage? Or if you really see that there's a niche of being able to do these types of social storytelling experiences.
[00:19:49.050] Aaron Koblin: I mean, I'm super excited about the ideas they're playing with. I think my dream is that eventually when we do projects like Life of Us and we want to be telling these stories, we're no longer modeling and creating on 2D screens and then putting on the headset and seeing if it worked. But instead we can do something similar to what they're doing, which is building in HMD. I feel like that's the future, is hop into virtual reality, do your modeling there. potentially embody some of your non-playing characters, create their animations. I mean, I think at this stage, it's still pretty early for a lot of these things, and some of the tools really need to be crafted. Some of the Unity Labs folks are doing some pretty exciting experiments that way, like thinking about how do our desktop-based tools transform and become something else in VR, something that's fast and intuitive. I think there's a lot more to be done there, but I think eventually, I hope that that's where we get. I'm excited to not be crouched over a mouse and keyboard anymore and start using my body again.
[00:20:39.773] Kent Bye: Well, one of the other really big innovations that I think you're doing here at Sundance is in with your installation of being able to do like this projection map of the recording of someone's experience so that when they come out of it, they could potentially sit down and watch what people are going through, which serves a couple of functions, which is to go through a first person experience and allow the person to step back and give this third person experience of what they just went through. But it also gives the audience the ability to have something to look at and pay attention as they're waiting to go through the experience. So maybe talk a bit about all the things that you were trying to do with this installation here.
[00:21:11.954] Aaron Koblin: Yeah. So Chris and I have done installations at Sundance a few years for non-VR related projects. And I think to us, we just, when we're coming back to Sundance, wanted to make sure that it wasn't just people sitting in chairs with headsets on, but was something that had a physical presence that other people could enjoy and connect to and share. I think actually, ironically, part of what got Chris and I interested in virtual reality was doing museum installations and large scale sculptures that had physical presence. And to me, as we exist right now, that's still the most powerful thing you can do is create a physical real world environment that you can live in with your actual real body. But of course that doesn't scale to millions of people around the world and that doesn't connect people across countries. It's so much more limited than what you can do in virtual reality in certain regards. So what this installation for us was, was, okay, let's try to fuse these worlds. Let's bring the virtual into the physical space and let's do something so that we can have, ironically, more scalability for an event like this, where you can have 60 people who are laughing and smiling watching the people participating who are in the deeper immersion inside the headsets. So kind of there's this back and forth play of like scale, connection, reality, immersion. We're just trying to mess around with that. But I think also to your point, we're excited about potentially the media that gets created from these experiences and the ability to get those to people who don't even have headsets right now. People who may have a mobile phone that can do a 360 video or maybe a cardboard. or Daydream or Gear VR. We're still in that phase of trying to figure out how do we get this kind of content to as many people as possible regardless of what their accessibility constraints are.
[00:22:42.238] Kent Bye: Yeah, I think that for that there's a kind of a fixed narrative that you have which I think would maybe be limited in the amount of novelty that you'd be able to generate from watching multiple ones. I think of the other extreme is Mindshow which essentially has no narrative arc that you have the kind of fixed in and so and it feels like it's a little bit of a balance between How do you give people enough control to make it like have their full expression of who they are? I mean right now you still can do a lot with what you're saying and what you're doing and just watching people be like holy shit like I'm a dragon or whatever like that moment when you're watching them and they're just like going through that moment of awe and So I think there's an amount of, as people are new to this and having these types of experiences, there's a lot of really visceral reactions. But in terms of something that's kind of interesting from a narrative perspective, I would wonder, how can you really loosen up the arc to make it interesting so that people's decisions actually have consequence that may be interesting for people to watch?
[00:23:35.105] Aaron Koblin: There's definitely a spectrum, right? And there's a lot of analogies that fall short about, like, going to a baseball game. Nobody goes to baseball games alone, right? You go socially with friends. Is that a narrative? Is that a story? Is that one where the players have agency? Yeah, yes and no. I don't know. I think with things like Mindshow, there's kind of two different ways to look at it, I think. There's the, like, Minecraft, this is fun. I'm building things and sharing them with my friends and reflecting on myself mode. And then there's also kind of what I was alluding to before is this is a tool which will allow somebody to create a super well-crafted narrative that is really interesting to anybody anywhere because they've put in enough artistry and craft. And I think both of those will apply in those spaces. I think what we're doing with the Life of Us context is really a pre-crafted narrative that's first and foremost for the players themselves. And ideally, we're looking at the relationship between those players as kind of a canvas in itself. It's, again, back to the sports game. You don't go to a baseball game alone, and you don't do that because it's actually a means of connecting with people. I think people's relationships are really the sum total of their experiences together. It's about how much time have they spent together, how well have they gotten to know each other via different types of actions. And these different stories and actions that we create deepen those relationships. And that's part of what we're most excited about is providing a context which will allow people to interact in a way they've never interacted. and to have a shared experience that is more intentional and more crafted. And that's not because I don't believe that the less crafted ones aren't worthwhile. I actually am a big gamer and I'm really interested in open-ended things like Minecraft. But I think there's a huge hole and a huge place for those more intentional, super well-crafted, artistic-driven experiences. And I think you'll see some of those come out of this open-ended experience as well. I think you'll see really bright and interesting creators who make things that everybody wants to see and be a part of and share.
[00:25:19.358] Kent Bye: Yeah, and just to come back to the installation part, because somebody was asking me where I saw all this going, because right now we have these applications where, you know, Within has an application. There's lots of videos there. But what does that really look like in a volumetric sense? Because that's really kind of replicating the 2D realm. And I really thought, I was like, well, these types of installations that are here at Sundance are really kind of like a vision of what the Metaverse is going to be like. As you go to the Metaverse, you go to the Within installation piece. You go to some sort of physical architecture that's representing the experience you're about to go into, and then you go into the experience. And then maybe you come back, and there's a lobby, and you decompress, and you watch what just happened or something like that. And so I feel like, as we're moving forward, we're going to start to see more of these types of using architecture and installations to be able to create experiences that come with maybe experiences within themselves, just to go to the lobby and watch and participate, even if you've done it already. And I think that creating that social dimension. So yeah, just curious to hear some of your thoughts.
[00:26:14.293] Aaron Koblin: No, absolutely. I mean, it's something we've spent a lot of time talking about. Unfortunately, we are still guilty like everyone else of using floating thumbnail paradigms for entry points. But ultimately, I see this becoming like fully immersive architectures where each project has its own context. And there's obviously always a trade-off between kind of form and function, like how quickly can people get to where they want to be and need to be. But in a virtual world, you can still have that and have amazing, incredible spaces and scenes and contexts. So that's where I want to get, and I think it's really a matter of time, resources, and energy, but it's pretty high on my list in terms of how can we think of this for what it is. Every medium starts by repeating the past, and we're doing that very much with desktop paradigms when we don't need to be at all.
[00:26:58.180] Kent Bye: So what do you want to experience in VR then?
[00:27:00.980] Aaron Koblin: I mean, I think what we were just talking about is my dream, not to go too Ready Player One or anything, but the context of I'm in spaces that I could not inhibit in real life, spaces that make use of physics and shaders and Easter eggs and trick doors and playing with scale and playing with size. And I think having a virtually physical context for things is just super exciting. Everything I've done that has anything to do with physics, I just find myself lost in. My background's a little bit in generative art and writing kind of procedural code for making dynamic systems. So just the ability to translate that and that kind of level of beauty into a three-dimensional world around you I think we've just hitting the tip of the iceberg Hopefully a lot more people from the demo scene world and kind of the hardcore graphics hackers are gonna start getting involved in VR That's everything I wanted as like a teenage boy and in life, so it's starting to happen, which is pretty exciting
[00:27:52.462] Kent Bye: Great. And finally, what do you see as the ultimate potential of virtual reality and what it might be able to enable?
[00:27:58.426] Aaron Koblin: I think that gets very philosophical and has a really long arc. It depends where you stop thinking of virtual reality as virtual reality. As we get more and more sensors that tap deeper and deeper into our cognition, are we talking about spinal taps, neural taps, and other means of interacting? Or are we talking mostly about HMDs? I think what we're going to see in the next year or two is probably a lot more AR coming onto the scene as well as much better VR and HMDs. You're going to get better immersion, you're going to get interaction with the real world, you're going to get artificial intelligence playing a really meaningful role in how you understand the real world and especially in the AR side of things. You're going to get creatures and non-playing characters and environments that can learn from your actions and respond to you. I think as you look further and further down the road, the potential for it becomes really profound. Kind of how does this change the way we think of ourselves and how does this change the way we work and function as a culture. And I think most of that I'm very excited about. And there are aspects of it that I think we need to think deeply about culturally to make sure that we make good decisions about how we maintain our physical health and our social health. But a lot of those topics already exist, and I don't want to go on a political rant or anything. But I think we are going to have to figure out how we maintain empathy and respect for each other as humans. And I hope that these technologies will help in that regard in terms of being able to relate to people by temporarily occupying their shoes and to be changing the way that you think of yourself, I think is a meaningful and important action. But yeah, there's always pros and cons, right?
[00:29:32.244] Kent Bye: Awesome. Well, thank you so much. Thank you. So that was Aaron Koblen. He's the co-founder and CTO of Within, which was originally named Verse, but they split up into two companies called Within and Here Be Dragons. So I have a number of different takeaways about this interview is that first of all, I think playing with embodiment in the way that they are, I think is probably one of the biggest innovations that I saw at Sundance this year. Partly because you're embodying a number of different creatures, you're doing it across a lot of different environments, but mainly you're doing it with another person. And so usually with a virtual body ownership illusion, you give yourself a mirror and you're able to watch your movements as you move around. So one of the biggest ways of invoking this illusion was to give yourself a mirror so that you can track your body movements. But in this experience, it's a little bit more of a socially induced virtual body ownership illusion. You're watching somebody else across from you in this same embodiment of these different creatures. And you can't see your full body, but you see the other person's body. And so you just assume that you're a tribe of these creatures going through these different experiences. And one of the things that Aaron said that I really loved was that relationships are the sum total of all of our experiences that we've shared with somebody else. And their desire is to create this narrative arc of an experience. It's kind of like a theme park ride. more so than a film or video game. It's an experience that you go through with each other and it's open-ended enough in the sense that you can interact and relate to each other and that sort of becomes the story. It compares it to a baseball game where you go to a baseball game to watch the action that's unfolding but a lot of the experience of a baseball game is the people that you go with and the connections and relationships that you have in the process of watching this game. And it's kind of like that in this experience. You don't really have a lot of ability to change the outcome of anything in this experience. It's really on the rails. But it's also different than a lot of the other open world type of experiences that I've seen. Things that are like in VRChat, for example, they recently launched on Steam and they've had all these different worlds you're able to go through, but there's less of a strict narrative that's associated with them, and it's more about the relationships that you have with the people that you're going through an experience like that, rather than the narrative that's being told through the environment. Although they are starting to have some experiences that start to play with that a little bit. One of the other things that they started to experiment with was the modulation of your voice. And this is something that I talked to Robin Arnett with in the episode that I did on Soundself. But in this experience, you're having a very distinct voice modulation that's correlated to the different levels of embodiment that you're having. And as you progress through the journey of evolution, the fidelity of that audio modulation also increases and being able to have higher degrees of control over what you're saying. So it was interesting to hear Aaron talk about this uncanny valley of voice modulation and for anybody who has been on a Skype call where the other person doesn't have headphones on and you start to hear feedback in your ears as you're speaking, it does this really crazy thing with your mind where it actually makes it super difficult to be able to actually think while you're talking. Now the surprising thing to me was Aaron was actually saying that that is sort of this uncanny valley that happens with the voice modulation that You know when you speak you essentially have bone resonance that happens when you're speaking that is zero latency and whenever you start to modulate the voice you're always introducing some level of latency and when you start to broadcast over the internet that is introducing even more latency and so what they're doing is they actually have to add additional delays such that you hear this huge gap between when you're speaking and when it actually comes out and And whenever you do a pitch shifting, he's saying that that also helps your brain accept the fact that there is that delay and it doesn't interrupt your mind so much. But it did have this direct feedback into dynamically changing how I was communicating and what I was saying, which was a really trippy and weird thing is to be speaking and then as you hear yourself speaking, that feedback actually changes the way that you're thinking and what you actually end up saying. And the fact that you're with another person who can receive whatever you're saying, it just made it a big part of my experience. And so when I was going through The Life of Us for the first time, that's what the thing that I was really testing limits with and playing with a lot was just screaming out different things and just narrating whatever it was that I was doing. One of the other things that Aaron's mentioned with Jaren O'Neill was just commenting on the brain's amazing ability to have this neuroplastic ability to be able to change and adapt to whatever embodiment that you're having within a VR experience. And I've also noticed that is that there's this calibration phase that ends up happening as you're getting embodied into these different VR characters. And Tyler Hurd's Chocolate probably does the best job of this in terms of allowing you to have like 15 or 20 seconds to have a mirror right in front of yourself and you get to watch yourself as you move around and you get to look at your body and you have this identification and that invoking of the virtual body ownership illusion. But without having that mirror there, like I was talking about before, there is that socially induced virtual body ownership illusion, but There's also this calibration phase that is happening as you're going through these different characters. You are moving your body and that is in turn moving how these characters move. And there's an amazing part of your brain that is able to still have the proprioception of where your hands and body are and that as you move it, it's being able to dictate and move these different limbs and these different creatures. I know that I've heard some people talk about some of the research from Jeremy Bailenson where he's been able to add additional limbs and other parts of your body where you're able to start to move your body and control new limbs that humans don't even have. So you're able to control like a third arm. So that's the type of things that the brain is able to adapt and calibrate and change. Any game that you start to play in VR, there is a little bit of that calibration process of being able to move your body around and being able to adjust such that you can match between what you predict is going to happen and what is actually going to happen. That was one of the things that Aaron said is that he felt like physics interactions also really help his embodiment and in my interview with Jeep Barnett he was talking about how in games we like to make predictions about the future state of the world and physics is a perfect way for us to make a prediction for how something is supposed to interact because we have all these real-life physics interactions that we've had in our entire lives and so a lot of our ability of cultivating intelligence is being able to manipulate the world in different ways and to See the correlation between cause and effect and that same process is happening within these virtual environments where we're making predictions about the future state of the world and then as we engage our agency within that experience then we're able to match whether or not a is meeting our expectations, and if it does, then it gives us a deeper sense of plausibility and the reality of suspending our disbelief in the entire experience, but it can also increase the level of embodiment that we have if it's matching what you're expecting with your movement of your hands versus the physics reactions that you're seeing. I think that's probably one of the biggest gaps in terms of where VR experiences go wrong is that if you start to throw something and if it doesn't feel like the physics trajectory that you expected, then it can be a little bit of a presence breaker. But on the other hand, it can also help deepen the level of embodiment if you really get it dialed in. And so just a couple other points about this experience is that you are rapidly going through entirely different environments. And I think that the importance of place and location is one of the very unique affordances of the virtual reality medium. And I think that it's probably one of the things that some people may have not particularly noticed in this experience is just the impact of what it feels like to rapidly change through these drastically different environments as you're evolving through this story. I think that whenever you walk around in the real world, you have this innate sense of your environment and where you are located in time and space. But yet in VR, you're completely get disoriented and you are just thrown into another world. And so there's something about our brain that wants to, to some extent, get oriented to where now you've been thrust into. I think it is possible to do editing in VR and I think a number of experiences prove that it's not only possible but extremely compelling but there is still a part of your brain that wants to get oriented into the place that you're at and so I think that the life of us is playing with this rapidly changing through these different environments in a way that makes it more of a visceral experience. And when you go through it with somebody, and this is definitely an experience that you don't want to do alone because there's just so much that's added for being able to interact with another person and to deepen your sense of embodiment in that way. But because it's that social experience and you're going through it together, you're kind of discovering all these different worlds with each other, which I think is a lot of fun and I expect to see a lot more of as we move forward into blending the storytelling capabilities with the virtual reality medium. And finally just a comment on the installation of Within's Life of Us at Sundance is that it was very highly produced and quite an experience within its own right just to be able to sit down and watch the projection mapping of the recording of what people just went through as people are coming out, they get to watch themselves as they have just gone through this first person experience. And I watched a couple of people watch themselves as they were going through this experience. And I think that was a really interesting touch to be able to. have a first-person experience and then come out and watch a third-person point of view, record it and watch it. And I think that that's likely going to be something that could very easily translate into the metaverse, into some sort of online spatial representation and installation for an experience that you go into. If you can imagine that in the future when you go into an experience like Life of Us, well, what happens when you go into it and come out of it? I think that's where you get into this construction of an architecture that is going to be unique to each of those experiences. It's likely going to be something in the WebVR and a web-based where you go into this environment And that environment is priming you and preparing you to go into this other altered state. It's sort of like the part of the ritual that happens when you go watch a movie is that you watch all the trailers, and then you see the music, the lights go down, and then once the movie starts, then it's part of that ritualistic process that we go through. to be able to be really receptive to that story. And I think there's that same type of ritual that I see being cultivated at different conferences like Sundance, where you see an entire installation that you're able to get this physical embodiment of before you actually go into the experience. And that is something that just translates really well into VR. And it was interesting to hear that part of the original inspirations for both Chris Milk and Aaron Copland was that they were doing these big installations that they couldn't widely distribute and that VR allows them to start to do that. So within just this past week or so launched their WebVR website at vr.with.in And right now it's still just thumbnails, but I expect that in the future they're going to start to build out these thumbnails into entire environments in their own right, such that you would kind of have that similar type of installation experience. And you can imagine that having an installation like that and some of the projected map recordings of the experience as you come out of it, you get to watch that and maybe be kind of like a bar like environment with other people hanging out and watching it. I expect that that's going to be one of the key features of where people are hanging out in the metaverse in the future. Because I do agree that part of the relationships that we have with people are the shared experiences that we have. And I think that this life of us is one of those experiences that I can start to point to of saying, okay, this is something that I would want to do with other people. And there is a bit of novelty of going through it for the first time. But it's still also something that has a repeatable element because you have different relationships with different people. And it's just kind of fun to see what is evoked as you embody these different creatures in the context of this narrative. So that's all that I have for today. I just wanted to thank you for joining me on the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and become a donor. Just a few dollars a month makes a huge difference. So go to patreon.com slash Voices of VR. Thanks for listening.
