#478: VR Fidelity Contract: Cultivating Plausibility & Presence in VR

kimberly-vollCultivating presence is one of the main goals for a lot of VR experiences, but our brains are like a black box of perceptual soup that makes it hard to know all of the right ingredients to achieve this. Kimberly Voll is a cognitive scientist, programmer, and VR developer who is a part of the Fantastic Contraption team, and she has a framework for cultivating presence that she refers to as the “VR Fidelity Contract”.

LISTEN TO THE VOICES OF VR PODCAST

The VR fidelity contract sets up the expectations of a virtual space so that there’s a match between how the affordances are presented to a user and the wealth of knowledge that each person brings based upon their lived experiences. So if there’s a door with a knob on it, then the user would expect to be able to open or close that door. Each time a user performs a successful action in an experience, then it slowly builds trust and the fidelity contract is maintained. If the user takes an action that wasn’t accounted for, then there could be an expectation mismatch that breaks the fidelity contract and ultimately results in a break in presence.

I had a chance to catch up with Kimberly at VRLA where she shared some of her process of cultivating plausibility through user testing, the most common things that break presence, as well as her thoughts on the future of artificial intelligence and storytelling in VR.

Fantastic Contraption recently made a significant Kaiju Update that allows users with limited space to scale the environment down and work in the near field.

Subscribe on iTunes

Donate to the Voices of VR Podcast Patreon

Music: Fatality & Summer Trip

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR Podcast. So one of the holy grails of virtual reality is being able to achieve the state of presence. And researchers like Mel Slater have studied this and came up with two basic illusions that you have to achieve. One is the place illusion, in that you feel like you're transported into another world. And the second is the plausibility illusion, which means that the world makes sense. It's coherent and it matches your expectations of how you expect it to behave. And so on today's episode, I have Dr. Kimberly Voll, who is one of the members of the Fantastic Contraption team, and she got her PhD in artificial intelligence. But for the last 20 years, she's been looking at player psychology and game design. And the way that she thinks about plausibility is in terms of a fidelity contract that you have to make with the person going through your VR experience. So, we're going to be talking about plausibility from the brain's perspective on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by the Voices of VR Patreon campaign. The Voices of VR podcast started as a passion project, but now it's my livelihood. And so, if you're enjoying the content on the Voices of VR podcast, then consider it a service to you and the wider community and send me a tip. Just a couple of dollars a month makes a huge difference, especially if everybody contributes. So, donate today at patreon.com slash voicesofvr. So this interview with Kimberly happened at VRLA that was happening at the Los Angeles Convention Center on August 5th and 6th. So with that, let's go ahead and dive right in.

[00:01:52.037] Kimberly Voll: So my name is Kimberly Voll, I'm one of the members of the Fantastic Contraption team. I did a variety of different things on that project, from programming to design. My background is varied, anywhere from artificial intelligence is what my PhD is in, but I've spent almost 20 years now working in player psychology and understanding game design and how players process experiences, including in VR.

[00:02:12.115] Kent Bye: That's great. Wow. So I just am starting a podcast on AI and my process of going and doing a number of different interviews is that the more that you learn about artificial intelligence, the more you learn about our own brain and how it works and how we replicate that. So given that orientation about all that insights that you have about the brain and how it works, then what's your take of how that relates to VR?

[00:02:34.883] Kimberly Voll: So first of all, not many people draw that connection. They're always like, that's a really weird path. Why do you do all these different things? And absolutely, you're correct. I mean, the one thing that brings it all together is the brain. It's the consistent piece through all of it. So it weaves into VR in, I think, a very natural way in that VR is all about creating a sense of presence and being somewhere else, at least to a certain extent, and creating what I like to call believable experiences, since presence is a very specific thing. It's feeling like you are in a space, but a believable experience is one that you as a player or user, I mean, whatever the setup is, where you feel like your expectations are being met. So every environment that you're in has something I like to call the fidelity contract, which basically sets up the expectations and the affordances of that space for the user. So the things that I think that I can do in a space are because of the way the space is presented to me, as well as the wealth of knowledge that I bring just as being a human being moving about the world. And so when I am in that space, I'm going to try to do things that are suggested to me. And if I can't do them, that's when the space starts to break down in terms of believability.

[00:03:37.067] Kent Bye: Yeah, and Mel Slater's conceptualization of presence, he says that there's a place illusion and the plausibility illusion, and that the plausibility is that the world makes sense, it's coherent and believable. And Richard Skarbez has looked at some of the research in terms of presence. One of the things that he told me is that he sees that the uncanny valley is n-dimensional, meaning that there's many different dimensions of fidelity, whether it's the audio, the haptics, the visuals and that. It's almost a little bit better to pick one level of fidelity rather than trying to go for photorealistic visuals but yet have the audio that's not even spatialized.

[00:04:11.982] Kimberly Voll: Yeah, I think that's really astute. A lot of people I see in this space who are trying to make things that have quote-unquote high presence are aiming for the best of the best in all areas. And the reality is we don't necessarily even know what this giant perceptual soup is made of completely. Obviously, visual cues have an impact. The first time you're in VR, you throw on that rig, you're like, whoa, this is like, holy cow, I'm somewhere else. But as you continue to experience that, all of the other signals that are influencing your brain gradually increase in volume, in a matter of speaking, such that the visual cues stop dictating the fact that you feel like you're there. And so that initial, whoa, I'm here, gradually fades into the background. And now you're seeing all of these little things that are kind of niggly things where your brain's going, wait a minute, wait a minute, I'm not here, I'm not here, I'm not here. and it just keeps pulling you back out of that experience. So one of the common things that folks think is, oh, it's got to be super photorealistic because, you know, if it just has to look so convincing and then it's going to be, that's the key, that will unlock presence. But that's not it. You can draw people into all sorts of interesting experiences, but it's as you say, it's got to be a consistent world, it's got to be a believable world, there's got to be a certain plausibility in terms of how the world is set up for me to believe in it. So if you show me, in some sense, as the creator of this experience, that there is physics like I would expect in the real world, then those physics better persist through that experience, or it will start to become less and less believable. So you can do that with a cartoon world. You can do that even with cartoon physics, if there's a consistency.

[00:05:34.946] Kent Bye: So maybe you could tell me a bit more about what you mean by the fidelity contract.

[00:05:39.282] Kimberly Voll: So when we enter into a space, there is a certain fidelity to that space. There's a fidelity to the audio, to the visual, to the different facets of that experience. And that fidelity is essentially how my brain is going to process my expectations for that space. For example, if you have a box in a space, you can say, this is a box, let's pretend it's a chest of drawers. Okay, it's a chest of drawers. I'm not going to walk up to it and try to open those drawers because nothing about that object tells me that that is a way in which I can interact with it. There are no affordances suggesting that. It's all make-believe at this point. But if you give me a photorealistic, beautifully rendered chest of drawers, and I'm in an environment where, for example, I can open cupboard doors, and I can do all of these other interactive things, and I walk up to that chest of drawers, I'm left with the impression that of course I can pull these drawers open, because I could interact with the other things. This is highly rendered, and so I try to do that, and suddenly it doesn't work. and now I'm pulled out of the experience, and so the fidelity contract in that instance is broken. And that's true for all sorts of things, you know, audio cues, all sorts of ways in which we as developers of this experience have to almost get in the minds of our players, of our users, and anticipate the things that they're going to think in that space, which of course is almost impossible, but it's one of the challenges that we face.

[00:06:57.818] Kent Bye: I think the interesting thing about looking at something both like AI and VR is that you start to see how amazing the brain is at doing things in some ways, like how difficult it is to replicate the brain through AI. But yet, by looking at both of these technologies, it's almost like a mirror so that we're learning more about ourselves. So, I'm curious from your perspective what you've learned and gained insight into us as humans by looking at both AI and VR.

[00:07:22.990] Kimberly Voll: I think one of the biggest learnings for me over the last 10, 20 years has been how much the brain does for us. And that's not to say, you know, I'm not, I don't know all of the nuances of how the brain works, obviously, but there's so much happening in there by way of our brains trying to create consistent experiences, trying to resolve cognitive dissonance, trying to weave a narrative as one of the primary ways in which we store and catalog information and pattern matching and all of these things that our brains are hardwired to do. And a lot of times when we create experiences that break down, that don't feel good, that either land in the uncanny valley or just, you know, those amazingly crappy AI experiences that we're all super familiar with in gaming, a lot of it's because what we've done is we tried to make it too good. We tried to increase the fidelity to a point that we couldn't maintain consistently. And so we end up in a spot where, you know, the user's just disappointed, essentially. And what we created got in the way of what the brain would have naturally done. So to go back to that spot with the box, look at kids, they're amazing facilitators of make-believe, and we can create a narrative around that, a very convincing narrative, and we can buy into it, we can do all sorts of crazy cool things. But if we up that fidelity too much in a way that we can't uphold that fidelity, then our brain is trying to do all these things, and suddenly we're getting in the way, we're causing the brain to stop and go, wait, I was just fine, I was telling a perfectly good narrative here, but now I can see that that's broken, and now I don't buy it anymore. So we just tend to trip ourselves up by overmodeling our worlds, whether it's visually or auditorily. We set players up to feel like this is going to be a consistent experience. And then when it isn't, it rips people out of that experience.

[00:08:56.403] Kent Bye: And so you've been both at GDC and here at VRLA presenting about some of these ideas, about some of these cognitive science insights into virtual reality. So what are some of the big points that you're making at some of these talks that you're giving?

[00:09:09.201] Kimberly Voll: I think the biggest one I can stress is to watch what people do in your space. When people are in VR, it's really hard, as I say, to get into their mind and anticipate all of the things that they're going to do. Through all of my development experience within VR, it never fails, even now when I'm prototyping new things, and I'm surprised what people are doing in that space. So I think if anyone has any takeaways from all of the things that I say around, you know, how we generate these experiences and how our brains interface in these situations, It's, we can't guess that at all. We just sort of know some of the core ingredients of that perceptual soup and we can make some pretty educated guesses, but at the end of the day we can't predict everything. So you need to get a spectrum of people in there. You need to see the things that surprise you so that you can determine, you know, if my environment suggests that people should open those drawers and that feels crummy, then Maybe I need to shift my experience. Maybe I need to take those drawers out. Maybe I need to render them. Maybe I need to think about the ways in which my experience is nudging players or participants in certain directions. So I think that's a really, really huge one is getting people in there and reflecting on what it is they're doing and the subtle affordances and expectations that are being set by the design of the experience itself that you might not even be aware of.

[00:10:17.095] Kent Bye: And I'm wondering if you could talk a bit about some of your observations of people, how they've been playing different virtual reality experiences and fantastic contraption and looking at it through this cognitive lens. What kind of observations or takeaways you have qualitatively?

[00:10:31.821] Kimberly Voll: So I think, you know, there's that saying, the customer is always right. I mean, in a sense, in VR, and maybe this is true of all interactive experiences, the player is always right in that they're trying to do something. Something about this world is telling them to do this thing. You know, they have agency in this situation, which is to say they have the sense of command over their world that is consistent with their expectations, and they don't feel that anything is artificially or unfairly getting in the way of that. You know, one way to think about that is if you put a player in a straitjacket and like, oh, I can't move my arms. Well, that's still realistic, right? I mean, it's just there's a narrative around that that makes sense. So a lot of what we've been observing if we've been doing a lot of playtesting is just some of the interesting ways in which there's kind of two buckets. There's folks that have never tried VR before. So they go into VR and you get the, wow, this is incredible. This is an amazing experience, you know, and it's awesome. And I often talk about the VR giggle, like people just get in there and they giggle and it's just so wonderful and I love it. but that experience playtesting with people that have never been in VR before is incredibly challenging because they're so taken aback and they're so awash in what is amazing and for some people an absolutely life-changing experience that they never get to the thing you're trying to test and I see a lot of superficial testing of that form, where people come out and they're like, oh, that was amazing, and I see the developers, you know, figuratively speaking, but write down, it was amazing, and that's the conclusion that they draw, and it's like, but they never actually tried the experience, that's not, I mean, that was awesome, they had a good time, and I'm so excited for them, and it's wonderful we can bring them that, but you know they didn't test your game, right? And so kind of getting to that and I think we had to learn that too through Growing Pains as well. And then there's the other bucket which is of course people that are experienced and those are the ones that are not to lessen the value of others experiences but they're the ones that are super valuable to us as developers because they're the ones that see through the stuff. They're the ones that have been in there enough times that they're not going to be distracted by just being in VR and have had an opportunity for their brains to start to acclimate in a way that Like I said earlier, the vision part of it, which tends to be the loudest signal at first, has been dampened and settled, and so they can help you get at some of those things that might otherwise be getting in the way. So watching a lot of that and seeing what's been happening with contraptions, like what are they actually trying to do there when they're in the game? What are they interacting with? What are the things that draw their attention? It's been super, super valuable.

[00:12:47.273] Kent Bye: Yeah, and imagine that with virtual reality, and because it's so new, it's a highly iterative process. You have to continually be trying things out and testing it with users. And so I'm just curious in your process of doing a lot of that user testing, what was some of the biggest takeaways that you got from that user testing that then kind of got fed back into the final experiences?

[00:13:06.416] Kimberly Voll: Some of it was not the most exciting stuff, but it was even things like how people connected objects together. You know, just watching them again and again and again. When they felt like they were free to try whatever, what did they try? And kind of cataloging that over enough people to see what the patterns were. What we would do is we would watch people, and then we would throw something in, and then we'd make some iteration around those mechanics, and then we'd throw people back in and see, did it feel more natural? Because at the end of the day, there's sort of two different reactions. One is they don't notice it at all, which is a success, because it was so fluid. It fit right into their brain's narrative that they just continued flowing right through that experience. It didn't break or anything. The other is they were like, oh, whoa, that worked? Whoa. And so, you know, I think of it in a lot of ways, what we're doing as developers and designers of this experience is we're slowly but surely building trust with the person in that experience where they start to actually build that belief in that world. You know, we don't get presents for free. You know, we get it for free for like the 20 seconds that you first put it on. And then everything else after that is us trying not to screw anything up so that the brain realizes it's fake. You know, and sometimes it's already too late because the fidelity of that experience and the way it's set up is failing to meet those expectations.

[00:14:18.352] Kent Bye: Yeah, one of the things that I think about when I think about presence and plausibility is in a movie where you have a suspension of disbelief and that when you go to see a movie in a movie theater there's an entire ritual that you go through that prepares your mind for story and it involves like seeing a lot of trailers and then you have the title card and the music plays and then you kind of get thrust into the movie and you're really present. I think there's a lot of those rituals that we have that really prepare us to be able to receive the story and I'm not sure that we've necessarily figured out all those rituals for VR but one of the things that I think about when I think of presence and plausibility is that it's a little bit like a house of cards in that Just like watching a movie and you see a big plot hole it can be really taking you out of the experience But if there's something that breaks presence and breaks that plausibility then that house of cards can fall and it can be difficult for people to really Feel that same level of immersion and so as you've watched people and been looking at it through this lens I'm curious what you see are some of the triggers that really break that presence

[00:15:18.986] Kimberly Voll: Yeah, I think there's quite a few little things, but if I were to try to group them up, I think there's obviously, I call it the reality tether, so you're not fully giving yourself into that experience, and that can come from a number of different things. That could be a white coat effect, so a lot of people just don't feel comfortable. I mean, they're putting these big rigs on their head, and they know there's people in that space around them in the real world. Some never get to the point where they are actually able to be absorbed by that world, because, you know, they're shy, they want to bump into something, you know, you might be holding on to a representation of the world in your head, because maybe you set up room scale, but you technically know your coffee table's right there, so you're just around. Or, you know, one time I stood on the cord, happened to just roll perfectly vertical, and I stood on it, and that was like a frickin' Lego when I did it! I was barefoot, and I have wussy feet, and I kept thinking about it for a really long time, so I had this over-inflated representation of the cord in my head while I was actually in the rig, and it took probably six weeks before I kind of got over that. But even just that, there is that dance that you slowly develop with the cords where you get pretty good at it, you know, but I mean, we have to be fair. Of course it's taking away some of your attention to that experience. I mean, your brain is in the back somewhere holding a representation of that world so that you can do the little leg twirl to get it untangled while you're still shooting zombies and doing all these other things, right? that definitely can interfere. And then there's things like, I'm hungry, you know, or I'm there are things about my physiology that are high priority and so my brain's reacting to those things. There are things like, you know, I mentioned earlier when the expectations, the fidelity contract is violated. I mean, that can rip me out of that experience. Things like I'm up close to a wall in VR but the air quality is different and something, whatever it is, triggers that in my mind and I notice that and suddenly, oh, this doesn't feel as real. Like the other day I was in an aquarium simulator which was really cool. But I was just actually trying to get clips and different things for my talk, actually, and I didn't throw the headphones on. It's like, oh, this really feels kind of flat. It's like, wait a minute, it's because I can hear the rest of, there's no one home, it was just an empty room, but I could hear the air was just wrong, and I put that on, and having the ambient suddenly brought that in. So it's those sort of things, like almost missing channels, if you will, that can pull people out as well.

[00:17:26.659] Kent Bye: Well, for me, I've been doing a number of different interviews about VR and AI has been a topic that I think has started to come up, especially in the context of narrative and having intelligent NPC characters that you can have these kind of branching narrative type of interactions with. But I think there's also a lot of different dimensions of creative AI to be able to procedurally generate content. And especially within the last couple of years, there's just been a lot of really big breakthroughs in deep learning and VR has had a lot of the confluence of technology, I think AI is also coming up to that tidal wave of innovation that's coming. And so, because you have a background in both AI and VR, I'm just curious if you've been trying to figure out how you see those two worlds of artificial intelligence and VR really starting to come together.

[00:18:09.758] Kimberly Voll: I think from the believability angle is a really huge part, because a lot of people are really looking at AI obviously from an opportunity, like you say, to put artificial agents in that space and to enhance the believability of that space. And it's really, really hard to do that well, especially in VR. It's been hard for 40 years to do it well in video games. It's even harder in VR because there are so many channels to which our brains are paying attention. And that means that the subtlest little thing that actually is repeated in an unnatural way, it just doesn't have that natural cadence, suddenly your brain's like, oh, fake. And everything breaks from that point. And it's hard to create that trust back again when you realize it's artificial. I mean, how do you do that? Of course you know it's artificial because you just put on a rig and you went in there. So there's that pulling into that experience and that almost, like you referred to earlier, the ritualistic kind of priming. I mean, some of that is in the setup of the actual experience itself. It's hard for so many reasons. What is a non-linear narrative? In movies and in games, we have incredible tools to be able to basically make the player do what we want, right? Right from the worst of the worst, invisible walls and things, to just really clever blocking and cuts and different things. But in VR, you can just... proverbially lift up the hood and see everything. You can stick your head through things, you can look around things. So how do you create an experience that nudges the player in the right direction, or the experiencer in the right direction, and that allows it to be consistent with whatever artificial intelligent agent that you're creating? It becomes a lot of content, for example, that you're producing for this. If you're in there long enough, at what point does it repeat? I mean, how do you add those kind of natural saccades to the movement? How do you have it react in a way that anticipates kind of what the player was expecting? I think the key for anyone going down this path and it's really the key for any kind of artificial intelligence that is going to be driving artificial agents is to start simple and to go back to what I was saying earlier about our brains will inject all sorts of amazing powerful attributes to artificial agents so long as we don't violate our expectations. So starting super simple, looking to root out patterns and things that are going to catch people's attention early on and just incrementally exposing people to that. So build something, make it super simple. It may just be a little wiggly thing moving around in the world that you just, it's like a ball moving in the world, but you try to add a natural kind of cadence to that movement and then see what people think and then gradually add the complexity there. But if you start complex, chances are you're not going to get it right, and people are going to just right away get a very adverse reaction or an Uncanny Valley-style reaction to it.

[00:20:39.870] Kent Bye: And so Fantastic Contraption was one of the games that was released with the purchase of the Vive, and so that's been out for a number of months now. I'm just curious, what's sort of next for you in this realm of VR?

[00:20:51.525] Kimberly Voll: Yeah, I mean I'm always experimenting, I'm always playing around with things. I love VR, I love the challenge of, you know, for all those reasons that I said it's hard is one of the reasons why it's so exciting and it is, I think, a super powerful opportunity for us to take people places, to build opportunities for people to empathize and realize experiences that were otherwise never possible before. So I think all of that is super exciting. I'm really interested in questions around non-linear narrative, like how do we tell stories? We're on the eve of a paradigm shift in many respects in how we interact with not only media and how we tell stories, but information itself, the very nature of how we interact and how we learn, of how we take these huge quantities of information and distill it down to something that our human brains can process. When you add a physicality to that, when you add the ability to move into a space, to grab things, to have extension in that world, we only have what Hollywood has taught us that is. And that's got to go. I mean, it was fun growing up in the 80s, but that's not realistic at all. So, you know, I'm kind of really motivated by that challenge of thinking what that is. And, you know, we've got probably 20 prototypes on the go, and we're sort of eyeing them all to see what our next big thing is. But we're definitely going to do something.

[00:21:58.242] Kent Bye: Well, just in looking at a narrative, I've been doing a lot of different interviews about storytelling, and it does seem like one of the most innovative realms of nonlinear interactive narratives have been in from the genre of video games. And so I'm just curious if there's any specific titles or games that you think kind of serve as inspiration, as a really good foundational principles that other storytellers can look to.

[00:22:21.795] Kimberly Voll: You know, just off the top of my head, thinking of different things, looking at things not necessarily from the lens of traditional storytelling, I think is important. I mean, there's lots of really great narrative games out there that have the more kind of traditional format, you know, it's sort of a linear experience. There might be branching, but I mean, you know. complexity-wise, it reduces down to what's effectively linear with a few different choices. But looking for things that build worlds and narratives around those worlds, so games like Journey, for example, tell a story in a very open-worldy kind of sense. I mean, you know, at the end of the day, it's still fairly linear, but the way it's subtle, but the way in which people come together in that game through nonverbal communication, I mean, there's a power there, there's something to understand there in that human-like connection that I think we need to explore. There's other games like Brother, A Tale of Two Sons, which again, still linear fundamentally, but did some really incredible things in terms of building an emotional attachment to your controller stick. And then, I don't want to spoil it for anyone, but messing with that in really interesting ways that just helps us get us thinking in new different ways. You know, we want to keep falling back to the linear pattern, but there's all these interesting tools that I think we've only just begun to discover, and we just need to get super, super creative and experiment with it. We need to stop ourselves from doing input checks and balances, just to not let ourselves to just default back to the linear narrative. There's nothing wrong with it. It's great. I mean, we've got millennia of linear narratives that are incredible, but we want to experiment. We want to step outside of that. We want to see what's possible. What does it mean to be in a world? Are there worlds out there that feel fleshed out? I think back to classic Myst and stuff. I mean, there was a world there. You knew there were things there that were outside of your narrative. In VR, you can go seek those out. You can explore those. You can define your own timeline. It's not about reading the text at the right time or having all of that stuff, right? It's about allowing yourself to explore and have that narrative build up within yourself. And I think that's a lot of the direction that we're going when it comes to narrative.

[00:24:22.547] Kent Bye: And finally, what do you see as the ultimate potential of virtual reality, and what am I able to enable?

[00:24:28.849] Kimberly Voll: I think ultimately, and I don't know in terms of when or obviously how, I think we're just at the cusp of figuring this stuff out. But I think it will fundamentally shift how we as human beings interact with information and how we interact with each other. I don't think it will supplant reality, but who knows? I mean, there's all sorts of cyberpunk out there that would have me be wrong. the stuff that is yet to come, we can't even begin to imagine what that's going to be. We are at early days. I mean, let's face it, it's super, super exciting, but they're not exactly the most comfortable things in the world to wear these rigs, you know, and even to get it in mobile, it's still, the power's not there yet. We're still kind of feeling queasy and there's all these issues to work out. So, you know, I think the big stuff before us, we just haven't been there yet. And the most important thing that we can do is go crazy and wild with our experimentation and create opportunities to question our very core assumptions on how we interact with things that, you know, go all the way back to the Ivan Sutherland days, and we got to toss that out, and we got to start again and figure out what's the language of VR? Like, where are we going? What can we do here now that we have never in our lifespan as humans been able to do?

[00:25:39.780] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say?

[00:25:44.269] Kimberly Voll: Just I'm really excited. I think this is amazing. I'm super excited to see all of the different content that's coming out and all of the different things that people are experimenting with. I think that my word of caution for everybody is don't get too caught up in trying to help people get over motion sickness by forcing them through experiences that you like, oh, you'll develop your VR legs. Again, go back to the roots. Go back to the basics. What's unique about VR? It's not about making people sick. It's about taking them new places. And so we've got to get super creative and I think if we're not careful, the whole nausea problem becomes the reputation of the entire media, which is not going to help us development-wise. So I just want to urge people to stop making people sick.

[00:26:26.670] Kent Bye: Awesome. Well, thank you so much, Kimberly.

[00:26:28.131] Kimberly Voll: Thank you.

[00:26:29.351] Kent Bye: So that was Kimberly Voll. She's one of the members of the Fantastic Contraption team, and she has her PhD in artificial intelligence and for the last 20 years has been looking at player psychology and game design. So I have a number of different takeaways from this interview, is that first of all, I see a lot of the content that Kimberly is talking about in this podcast is kind of proving out a lot of the academic theory within presence research. So there's people like Mel Slater and Richard Scarbez who have talked about presence and plausibility, and Kimberly's actually on the ground floor creating experiences that is trying to cultivate that sense of presence and plausibility. So she talks about this in terms of the fidelity contract, which is essentially this agreement that you're setting up with your users. It's trying to set up the expectations and affordances of that space. So in other words, the fidelity is sending cues to the brain as to what you should expect when you go into this space. And according to Kimberly, that's two parts. One is, how is the space presented? And what type of affordances are you showing that user? As well as the user is bringing the wealth of information that they're bringing from their lived experience of being a human being. And so they're using that information and applying it to what they should reasonably expect should happen if they try to take some action within that space. And Kimberly said that the crazy thing about this is that we really don't know how to predict what people are going to do. And in some sense, you have to just send a large variety of different people into your experience and just watch them and see how they're reacting to your space. And this is just a slow process of building trust with the user, so that each incremental thing that they try in this space, the more that it matches their expectations, the more that they're going to see that it's plausible and the deeper level of presence that you're able to eventually achieve. So if you can't do that, if you don't do that, if there's some sort of break within the fidelity contract, then that's where the break in presence happens, where the suspension of disbelief is broken, the house of cards falls, and that's when they just don't believe that they're actually in that space anymore. And it's kind of like sitting in a movie and seeing a huge plot hole, and it just takes you out of the experience of receiving a story. You just start to question it more. And the same thing can happen within a VR experience. So when I asked Kimberly about some of the things that she sees breaks presence, she talks about it in terms of a reality tether. This is anything that is tethering you to the reality. The most literal example of that is the cords that you have in a VR experience and just the cord management that you have to do walking around a room scale VR experience. It's a subtle thing. Your body eventually starts to kind of subconsciously do it, but actually you're still kind of consciously aware always that you have this tether that's connecting you to a real reality versus what you're actually seeing in the virtual reality experience. And so my own personal experience of going through different tetherless VR experiences is that the level of presence is actually higher and that once we're able to eventually have wireless VR connections, then I think the level of presence is just going to have a huge increase. And other people, when they're going into a VR experience, may just be shy when there's other people watching them and they're in this white coat environment, or they're holding on to some sort of representation of what the real world is when they're going into the VR experience. Or there could be other physiological aspects, like you're hungry, or you could just not be matching your expectations as you go into this VR experience. So the other thing that I find really interesting about Kimberly is that she's got this background in artificial intelligence. And I think in the future, interactive narratives is going to be facilitated by these artificial intelligent agents. But I think we have a long way to go before we actually get there. And that's one of the areas that Kimberly says that she's really interested in exploring. But also there's a common denominator between artificial intelligence and virtual reality, and that's the brain. And that, she says, the brain is really good at trying to resolve cognitive dissonance and weaving a narrative as a primary way of storing and cataloging information, as well as pattern matching. And some of the interviews that I've done on the Voices of AI podcast, which I've done about 90 interviews at this point, and I hope to launch it sometime in 2017, I solicited some feedback from my Voices of VR listeners, and I think about three podcasts per week is about what people are able to reasonably consume. And so I'll be cutting back to about three podcasts a week on Voices of VR, and that'll give me more time to finish up the Ultimate Potential of VR book, and then eventually launch the Voices of AI sometime in 2017. And talking to a researcher, one of the things that I realized is that there's a whole field of artificial intelligence that is knowledge representation. And very interestingly, one of the ways that we actually process and store knowledge in our brains is through the semantic structure of sentences and story. And so there is very much a connection there between weaving a story, connecting all of the dots, and putting a subjective narrative onto what's actually happening within an experience. And that process is part of the reason why I think our brains are so wired to receive story, but it's also a way that we just store knowledge and make sense of the world is through these stories. And so I do think that virtual reality does present the opportunity to tell stories in a completely new way. And I think that The things that Kimberly was saying matches what I've heard from Ran Miller in episode 432 and Denny Unger in 462. It's just that this whole process of environmental storytelling just creates a great mechanism to be able to do non-linear interactive narratives. instead of just sending you on a direct linear path and kind of feeding you the story, you're actually given the opportunity to explore those worlds and to be able to, as Kimberly said, define your own timeline, which means you can go anywhere in that world and you're going to be having the narrative revealed to you in all sorts of different chunks and pieces of information. And there's as many different ways to discover the narrative as there are ways to explore and go through these virtual worlds. So she's really interested in this concept of exploring and letting the narrative build up within yourself. And also, I think that Kimberly's right in terms of like talking about artificial intelligence and VR and trying to go to the highest level of human level of intelligence and interacting with human non-player characters. I think that's just going to very easily fall into the Uncanny Valley. I think the bar is set pretty high in terms of what we expect to have in terms of an interaction with an AI character. and talking to different AI researchers, I think people getting in very early into AI research, they're doing things like virtual pets. And these are pets that can demonstrate certain behavior, but it's not human-level quality of intelligence. And so I think that it's going to be a while before we're actually interacting with these NPCs that kind of match the Turing test, where it just feels like we're interacting with another human. So Kimberly's actually suggesting that instead of rather going from that human level AI to just try to start creating these kind of smaller creatures that have more natural movements. that people don't know exactly what to expect from them, and so they're able to kind of suspend their disbelief a little bit. And I just want to reiterate something that Richard Skarbez said back in interview 130, which is that the uncanny valley is n-dimensional, meaning that the way that you construct the entire world has a certain amount of fidelity from all the different dimensions, whether it's graphic fidelity, the sound, the haptics, it all has to be consistent and matching into what your rental model would perhaps expect within that environment. And so that goes for both the AI characters as well as these different interactive components that you have, whether it's kind of a virtual pet type of object or any type of things that you have the opportunity to express your agency within that environment has to really meet your expectations. And if it does, then it's plausible. And then if you do that, then you're more likely to achieve the state of presence. So that's all that I have for today. I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do tell your friends, spread the word, and become a donor at patreon.com slash Voices of VR. Thanks for listening.

More from this show