Mel Slater’s Theory of Presence includes two major components including the Place Illusion (the degree to which you’re transported to another world), and the Plausibility Illusion (the degree to which you believe what’s happening in that world is believable). Presence researcher Richard Skarbez thinks of these two components as “immersion” and “coherence,” and we previously discussed his presence research where he found that both illusions are vital for a deep sense of presence. The place illusion is largely enabled by the objective details of the VR technology with features such as 1:1 head tracking, low latency, and large field of view. But all of the different dimensions of what makes an experience plausible are still widely unknown, not very well researched, and also difficult to isolate and determine.
Skarbez was back at the IEEE VR conference in March of 2017 showing some of his efforts to break down presence into different factors that included interactions with virtual humans, high-end VR tracking to get fully immersive body tracking, interaction abilities within the environment, as well as whether or not the scenario was coherent and believable. He also deployed a new research method where the subject would experience the full fidelity of the screen, and he’d slowly dial up the fidelity of these different factors to determine which ones were the most helpful in cultivating presence. Presence surveys are not that great at aggregating multiple individual subjective rating the degree of presence on any type of scale, but you can judge relative degrees of presence based upon your own previous experiences. Skarbez was able to determine that full body tracking was one of the biggest indicators of the depth of presence that someone felt within their experiment.
I had a chance to catch up with Skarbez in Los Angeles at the IEEE VR conference where we talked about his presence research as well as how his different components of presence plausibility mapped over to my Elemental Theory of Presence.
LISTEN TO THE VOICES OF VR PODCAST
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR podcast. This week on the Voices of VR podcast, I'll be focusing on a series of interviews that I've done with some of the leading presence researchers in the academic community, where I get to unpack some of their theories of presence, but also compare and contrast it to my own elemental theory of presence, which is broken up into embodied presence, social mental presence, active presence, as well as emotional presence. So I'm going to be starting with Richard Skarbez, who I interviewed back in episode 130. And in that interview, I think Richard does some of the best jobs of really breaking down the difference between the place illusion and the plausibility illusion, which is the theory from Mel Slater. So the place illusion is believing that you're transported into another place, and Richard calls that the immersion factors. And then the plausibility illusion is that everything that you see in the scene is coherent, and it makes sense to you, and you believe it. And that's what Richard calls coherence. So Richard has been continuing the presence research, and he's starting to use Mel Slater's new experimental protocol. People used to do a VR experience, and then after the experience, they'd give them a survey. But then those numbers that they would get from a survey wouldn't be able to aggregate into anything meaningful. You're trying to quantify a qualitative experience. And in that process, it's difficult to judge the quality of someone's presence based upon a survey that they're giving you afterwards. Some presence researchers have looked into, can they get some sort of physiological indication of the level of presence? But at the end of the day, it's kind of an internal subjective experience that you have to compare and contrast to other times that you felt really present. So that's what we'll be covering on today's episode of the Voices of VR podcast. And this interview with Richard happened at the IEEE VR Academic Conference that was happening in Los Angeles, California from March 18th to 22nd, 2017. So with that, let's go ahead and dive right in.
[00:02:06.565] Richard Skarbez: I am Rick Skarbez. I'm very excited to be back talking to Kent on the Voices of VR podcast. I recently graduated with my PhD from the University of North Carolina at Chapel Hill and am now a free agent hoping to continue doing VR research in some shape or form. And what I did today at the IEEE VR conference was present a paper on plausibility illusion and its factors in virtual environments.
[00:02:33.800] Kent Bye: Great, yeah. So it seems like that, you know, there's the place illusion and the plausibility illusion. And with the place illusion, you're kind of separating in your talk, you're talking about sort of the inner subjective qualities of the place illusion and the objective things that you have to do in the VR experience are kind of the immersion factors. And with the plausibility illusion, that's, again, a very subjective internal state. But yet, there's different coherent factors that you're building into the experience. And so After going through over a thousand experiences over the last three years I felt like those kind of breakdown between the play solution and the plausibility illusion were describing it but yet it was kind of lacking for the different flavors of different types of experience that I was having and so it seems like you're now starting to try to do additional research into plausibility and those different fundamental factors and so maybe you could talk a bit about you know what you're breaking up into those different dimensions of plausibility and how you went about actually trying to measure that.
[00:03:30.244] Richard Skarbez: Okay, yeah, fantastic. So there have been proposed three sort of general concepts of virtual experiences that can lead to plausibility illusion. One is that there's correlation between what the user does and what happens in the virtual environment. So what you do has an effect on the world around you. Another is that there's what they call self-reference, so that if there are characters in the virtual environment, they look at you, they speak to you, they do things that are directly related to you and not just sort of repeated animations that are happening to give the illusion of a world around you. And the third thing is, I think the biggest thing, and certainly the broadest thing, which is credibility. Are the things that you see happening in the virtual experience, do they make sense? If you're in virtual Paris, are people speaking French, for example? I think we used that example last time maybe, but I think it still holds. And so in this study, the factors of coherence that contribute to plausibility illusion that I looked at were other virtual humans, the user's own virtual body, the physical coherence, the physical behavior of objects in the environment, and scenario coherence, sort of the overall does what you're seeing happen make sense for the environment that it's happening in. So those were the four factors that I looked at in this study. And interestingly, perhaps not usefully in the immediate future, what I found was that when you give users the option of having a fully tracked virtual body, in this study I had an OptiTrack mode of tracking system with a full body tracking suit. When they know that that's something that they can have, that's all they want. That's by far the most important thing that we saw. which is, I think, perhaps the biggest challenge outstanding for VR right now, because we're getting at where we can track the head pretty well, we can track the hands pretty well, we can do room scale, but when you're trying to track the full virtual body, that's not something that can be done easily or cheaply. these days, so I think that's a big hurdle to get over because it's a huge boon to people's being willing to believe that they're in another space. When they can see their body, when they can move their body and use their body to interact with the environment, it's just a huge shift in how they think about a virtual space.
[00:05:56.005] Kent Bye: Yeah, and I think that with consumer-grade VR right now, people are getting very lightweight levels of embodiment because they're just tracking your hands, and you often don't even have a body. But I think the systems that you're using with the OptiTrack tracksuits, where you're able to actually get all of your limbs in and kick things around and see that expression of your agency. I've had experiences in expos at Sundance where I was able to have that full embodiment. It does give you a completely different level of presence. You know, over the last couple of years, I think that one of my frustrations with the plausibility was that I was starting to see, like, oh, I had this social experience where I had this sense of social presence with other people, or, you know, I would play these highly interactive experiences where everything I was doing was reacting to me in the way that I would expect, and so I would have that level of agency there. or had this invoking of the virtual body ownership illusion, so I did feel like I was embodied. And so when I'm looking at your three different levels, I would actually say that there's a fourth that maybe isn't being looked at yet, but just to go through the ones that you had there, which I think is, for me, if I think of the fire element of interactivity and agency, that there's a sense that you're able to actually engage into that world, and it matches your expectations of expressing your agency in that. And the thing that you kind of referred to is the correlation, which is that your actions that you're taking within the experience are kind of correlated in a way that you expect. And so you're expressing your agency in different ways, and the environment is reacting to you in a way that makes sense. And so the self-reference, so I sort of think of that as embodied presence, the earth element. So you actually have a physical embodiment within the experience in that. this social contact of people recognizing you socially. So there's a social dimension to that as well of people looking at you and looking at your body, recognizing you that you're existing, you're there. But there's also a deeper sense of presence of you actually being embodied within that experience once you have that body there. And then the scenario coherence is a little bit of like, it's both the place that you're in and makes sense, but I think intellectually in your mind, is it a coherent world that makes sense and it matches your expectations given what you have from, let's say, being in a bar? Does it match that expectation of things you might expect of seeing in a bar? And then if you break down these four different components that you're looking at, the virtual humans, I think, is trying to invoke this social dimension of social presence as you're interacting within VR. The physics, I think, is also an expression of agency as you're kicking things around. You're, again, exerting your will into the experience and then seeing if it is reacting in that way. So then the virtual body, I think, is the earth element of you actually being embodied in the experience. So then the scenario, I think, is also kind of like the place illusion, the context, you know, that's both the earth and the air, sort of a combination. You know, any of these sort of elements in my perspective, all four elements are in everything all the time, it's sort of like elemental balance in some ways. But the one thing that I'd say isn't in that, and what I would say is sort of the emotional component. So are you emotionally engaged? Does it have meaning? Is there music that's bringing emotions? Are there facial movements that are invoking emotions? Is there a color that's evoking emotions? So this is something that I think is a little bit You know, emotional engagement is something that I see mostly here in the models of social presence. I've seen that there's some models that are starting to say, like, how emotionally engaged were you in this exchange with this virtual human? So when you're engaging with virtual humans, there's an emotional exchange that can happen between you. But it sounds like you're trying to break up plausibility in these multifaceted dimensions, and you took a unique experimental approach, which is to show people what you thought was the best level of presence, and then start to dial it back into a couple of different degrees so that you have these four dimensions. and you're adding things in one by one until they say, okay, stop. Because one of the things that both you and Mel Slater have said is that trying to do present surveys, trying to quantify the subjective experience is very difficult. It's much more like a lot of companies like Netflix, YouTube as well, used to do like five-star ratings, but a lot of them are just moving to this thumbs up, thumbs down model, which is like, did you like it or not? It's sort of like, that binary thumbs up, thumbs down is a little bit like, okay, that's enough. When you're trying to really guess the qualitative aspect of it, because people have different rating systems and it's hard to combine them. But you're trying to reach that point at which they have successfully achieved the highest level of presence that you think you can have. So maybe you could talk about that process and what you were able to find from that.
[00:10:42.550] Richard Skarbez: Yeah, absolutely. So I think that that's true about questionnaires being not really the right tool to try to measure presence. They serve a purpose, but like you said, they're more like a thumbs up, thumbs down. Because what does a 3.3 average presence score mean? it doesn't have a real value. You know, I feel 3.3 about this doesn't mean anything. But you can use them in a way to sort of say, I like this experience and thought it was good, or I didn't like this experience and thought it was bad. But the experimental design you're talking about, I think you described pretty well. So we had four factors of plausibility illusion that we were manipulating, and three levels of each. We described them earlier, the virtual humans, the user's own virtual body, the physical coherence and the scenario coherence. So we exposed participants to the maximum level of all of them and told them to essentially remember how real this feels. and we want you to feel this real again in the virtual environment. Then we turn everything off, send it down to the lowest level, and say, okay, you can make whatever improvements you want, you can make however many you want, but just tell us when you feel like it's as real as it was in the best condition. And what we found was, again, that users always wanted the virtual body at the second level, at the maximum level possible, and generally wanted everything else at at least the first level of improvement. So that also tells us something. It tells us that everything we manipulated actually is important. If we saw, for example, that no one was turning the virtual humans up at all, it would say, OK, maybe that's something that we don't actually need in virtual environments. But none of the factors we looked at in this study showed that. So everything was important, but the virtual body was most important. The scenario was second most important, which was interesting, because the way we manipulated the scenario was The experiment was that you were in a virtual bar. There were some other virtual patrons in there. We did this experiment at the University of Barcelona, so they were football fans just hanging out at the bar talking about football or talking about drinking, talking about things that you talk about at the bar, and you were able to play with a football, kick it around, so you can have some agency in the virtual environment. And the scenario coherence was either the bar was represented as a bar, or at level one it was represented with a roughly equivalent level of visual quality, but as a fancy restaurant, a place where the conversations and the things that you're doing would be out of place. And then at the lowest level it was just represented by geometric primitives. And so it was interesting that people Wanted to go all the way to level two in that because if the only thing that mattered was visual quality You would think that they would have gotten to visual one and been fine because scenario level one was arguably the most attractive visuals or Certainly approximately equally attractive visuals as the level two bar but people noticed that it was out of place and said things on post experiment questionnaires like It was too fancy a place for two guys in their football shirts to be hanging out. So people noticed this manipulation and noticed that something was wrong. It wasn't anything that could be described as immersion. It's not the quality of the visuals. It's not the number of pixels. It's nothing like that. But it's just that something about this doesn't make sense. So they wanted to go all the way to level two to have it make the most sense as possible. And I think that that's very interesting that that's the second factor that people chose ahead of either the physical coherence, the agency component, or the virtual humans, the social component.
[00:14:34.990] Kent Bye: Yeah, and to me what's interesting is that there's a whole other different dimensions of qualitative, subjective judgments, even the creation of these different spaces. I've seen a lot of consumer-grade VR experiences that have cultivated a deep, deep sense of presence for me. And this is also something that's widely varying in terms of what my preferences are, what people like, what they're into. And so you get into this challenge, I think, of these AAA game studios are creating these highly polished immersive experiences. And then, you know, as sort of the academic kind of programmer art type of aesthetic, I think that, you know, there's certain limitations, I think, of how far we can take the presence research at that sort of level of technical fidelity. But here's what I would say in terms of, like, With my model in terms of trying to go for what I would see is like kind of the highest level of fidelity and presence that I've had. So if we look at the social mental presence, like having other human beings and having social interactions with other actual humans that are also embodied and represented in full embodiment. I think right now in consumer-grade VR, having your hands and moving your head around is kind of like enough. And because the inverse kinematics breaks having a body, it actually breaks presence, and so it's actually better to have a little bit of a ghost. But I'm hoping that over time we're going to get more and more sense of embodiment. That's kind of proved out with your research, that having a body actually was the most important thing, and you're at these high-end systems. My own personal experience also bears that out. But in terms of agency, being able to interact and participate in different ways. So, for example, you could kick a soccer ball around, or you could also engage with other people in a conversation, which then adds other social dimension. If they're actually being puppeteered by another person, and they're moving around, and it doesn't look like they're kind of a robot, sort of uncanny zombie that's moving around, then that's sort of another dimension of that plausibility. But I think the other component here is the emotional presence in terms of like, we're in a bar, but it's not like any fabulous bar that, you know, you couldn't go anywhere in any small town in America. But what if you were in a vast sort of epic, exotic place that is just beautiful colors and really awesome music and just transporting you into the other realm? I think, you know, we'd start to engage the emotions in a different way, but also, having real interactions with other humans, which also have that level of emotions. So I guess, you know, as I'm describing all these things, you know, one of the things that Mel Slater was saying is that, okay, one of the challenges of doing this is that when you start to actually try to isolate this into, like, research, then you have to try to break it down into parts where you dial it back. So I'm just curious to hear your thoughts as to, like, how you kind of continue this line of presence research.
[00:17:14.357] Richard Skarbez: Yeah, well, first up, I'd just like to follow up on what you were saying, because I think it's really interesting to talk about how emotions interact with all this. Because I think that that's an area where there's a huge amount of potential for collaboration between academic VR researchers and entertainment VR and games VR and storytelling. Because research isn't really good at eliciting emotions. And really we don't want to because they're so personal and they're so different for each person. A particular piece of music might not mean anything to you, but it might bring me to tears because it was my grandfather's favorite song. Right? So it's really hard to put into a controlled experiment, can we manipulate your emotions in a way that makes sense? But there are people who are professionals at eliciting emotion. And that's not me. And that's not Mel Slater. And that's probably not anybody at this conference. We're really good at sort of breaking things down and designing experiments and building systems. But for it to go to the next level, you need people for whom storytelling is their business, for whom manipulating emotions is their business. So I think that, you know, if you really want to push that avenue, there needs to be probably more collaboration between sort of people working in entertainment VR and people working in research VR. And there's not a lot of overlap there at the moment. There's some. There's some partnerships, you know, between Oculus and various universities and things like that. And there's Disney Research, where people obviously are experts in storytelling, but also experts in virtual reality. But I think the step forward for me or people like me doing this research is to keep trying to break it down and keep trying different factors in different environments. It may be that people didn't upgrade the virtual humans because the virtual humans I used weren't good enough. So, part of the nature of academic research is that it's slow, and I kind of hope somebody does some of this over again, or I may have to do it over again myself because I want to make sure that these results are meaningful. You know, I believe the results about the virtual body, and especially because I believe that applies in every virtual environment. Every virtual environment that tries to represent something like the real world, you're going to be having a better experience if you can see your own body and move your own body. And I believe that to be true. But the other factors, I think it might depend on what kind of environment you're in and what kind of task you're doing and what kind of story you're trying to tell, whether it's emotional or not. But if there's a story where your agency is more important, you're diffusing a bomb or something, and it's really investing you in the thing that you're doing in this virtual environment, you may see different results than what we saw this morning. So my next step is to design some more of these studies and look at more factors and particularly I want to look at immersion factors and coherence factors together because I think it would be really interesting and really powerful to find out, okay, does it matter more to have more field of view or does it matter more to, you know, spend a little bit more money and get a better modeler? Right? And those aren't questions that we're able to answer right now. You know, the hardware is considered sort of separate from the software, separate from the storytelling. But I think we have the capability now, as presence and plausibility illusion and place illusion theory develops a little bit more, that we can start to ask questions of, do visuals matter more than embodiment or not? Does the quality of virtual human interactions matter more than the quality of their visuals? And once you can start to ask these questions that sort of bridge the gap between place illusion and plausibility illusion, then you can really start to do kind of a cost-benefit analysis of what really matters for a VR experience. And I think once we can start answering that question, there will be more interaction between the academic VR community and entertainment VR or games VR or simulation VR, because we'll start to be able to answer the questions that people actually want answers to. You know, if we can tell you, okay, you have to go spend, you know, $20,000 on a tracking system to get better, or you can hire a modeler and that's going to cost you 50 grand, but they'll both give you about the same improvement, then we can start giving you real advice of like, okay, maybe it's worth it to buy the full body tracking system.
[00:22:01.772] Kent Bye: Yeah, and I think that these cost-benefit trade-offs are going to be context-dependent. I know that, you know, Mel Slater was presenting some research that he was doing on, some research that he and some of his students were doing in terms of having symphony in different contexts. So they, you would put, you're in this small room, and they're changing different factors, and one of the factors was the oralization, so the space and environmental reflections of how convincing the sound was. Did it sound like you were actually in a small room, or if they blew it up to make it sound like you were in a crazy large orchestra hall, you know, that was a disconnect and kind of broke presence in a certain way. And so having that level of oralization seemed to be like the highest one in that specific context, because that's what was actually being presented. And yet, one of the things that he was saying is that, like, basically you have these different tiers of, like, trade-offs between these other components. You can start to make these decisions as to what to really put your energy into. And the other thing I just wanted to say, a couple of other things, is that there's a lot of consumer-grade VR experiences that are out there that could be available to start to be used in these research studies. The thing that comes to mind is Pearl by Google, which is probably one of the most emotionally connecting VR narratives that I've seen. Now, in this experience, you're a passive ghost, and you have the challenge of whenever you go into the storytelling, then you don't have a lot of interactive agency. It's actually less about cultivating presence than having a sense of empathy and receiving. So I feel like the thing that I'm seeing is that these stories are actually not designed for you to have any agency, because the more that you express your agency, the more that you're not able to actually listen to that story that's unfolding. The other thing I just wanted to bring up was that you saw this difference in gender in terms of how engaged women were. And the task was kicking a football back and forth, right? And also I would point out that you're in a bar that isn't all that much. There's no other women. So it's sort of like there's other contextual elements there in terms of if a woman was there in real life, would she be really engaged and want to be there? I feel like there's a dimension of awareness of the emotional dimension and being aware of like, is this actually an environment that someone would actually want to be in and hang out in? And so when you're talking about presence, you're talking about, you know, really being engaged and kind of wanting to participate in this environment. And if you're kind of thrown into what may essentially kind of feel like a frat party at a bar, then it's sort of like not be inherently compelling to them. And that may be reflected in some of the gender differences that you were saying in the level of engagement between men and women.
[00:24:34.114] Richard Skarbez: Yeah, I think that's absolutely true. So the task again was just kicking around a soccer ball a little bit and anecdotally what seemed to happen was that men got really excited about being able to kick the soccer ball. They wanted to do tricks with it and show how skilled they were and got really involved and really interacted extensively with the scene and spent what looked like more focus and more attention on doing the task. where females, again, just as an observer, we didn't specifically collect this data, I wouldn't even have known what to look for, just didn't seem interested. They would kick it once or twice because we told them to and call it a day. And that's obviously cultural differences. You know, if we did this, this study was done at the University of Barcelona And maybe in America, where more girls participate in sports, maybe the gender difference wouldn't have been as marked. But it was very interesting that because the men seemed to interact with the ball more, they considered the physical coherence to be more important. So they generally actually considered it second most important, possibly even ahead of the scenario, for the most important thing for males as a whole. Whereas for females, they clearly considered it the lowest factor, even below the virtual humans. So there's certainly individual differences that come into play with how present you're going to feel in any particular situation.
[00:26:02.545] Kent Bye: Yeah, and I would also say in terms of the experimental design is that I would hypothesize that women are much more attuned to the space and color and the feeling of the architecture of a space, but also the interpersonal relationship. And so if they're just kind of overhearing a conversation from virtual avatars, what would it mean to actually engage socially with someone who's maybe puppeteering it? You get into a little bit of challenges of how do you actually standardize these types of interactions, but I think that's the thing where the expression of agency comes in. If you're interacting with an agent and it feels like you're just talking to a robot, then it's not going to give you that sense of social presence. So my sense is that being embodied and being able to interact with other characters in a more relational way may sort of change the level of presence that you're measuring. So I don't know how you would go about including live people in a multiplayer experience, but From my own qualitative experience of all the different VR that I've had, that's some of the most intense depths of presence that I've had when I am seeing somebody on the other side and I know that's a human being because I am interacting with them and they're responding to me as I would expect a human to respond.
[00:27:21.222] Richard Skarbez: Yeah, absolutely. And there are tricks that you can do to convince people that virtual people are real people. So, you know, you can do what you're talking about puppeteering where a real human is controlling one or more virtual humans. As I remember an interview on your podcast, I don't remember who it was with, but I think it was Charlie Hughes, yeah. Charlie Hughes doing improv acting to control multiple virtual actors. But there was a poster at VR this year over the weekend, or at the 3D UI symposium actually, where they showed that if you're looking at a virtual human, but you have a real human confederate interact with them first, you tend to believe that the virtual human is more real. So you can use a real human, maybe not the whole time, but maybe just to sort of kickstart you into believing that this virtual human is more real than it actually is. So I'm not sure how that would actually work in practice, but maybe you could get away with using one person to have lots of short interactions with other virtual characters rather than necessarily having to puppeteer a character constantly, but to be able to pop in and be like, oh, this other guy, he's totally real. Listen to this conversation that we're having. This is legitimate. All right, I'm going to see you later, but you should talk to him. He's an interesting person. And what the results showed is that people tend to believe the person more, at least over short interactions. So you can trick people into believing that things are more real than they actually are, and that includes social interactions.
[00:28:59.476] Kent Bye: Yeah, and that student actually was from Charlie Hughes's lab, and she said to me, I did an interview with her, and she said that she was just watching how Charlie would introduce these characters to the human experiencer and that she said that he would just talk to these characters as if they were real and it was sort of like this priming of like him vetting and vouching for these other virtual characters and I think that for people who are going into a virtual environment they have a real connection to this person and that person shows up in VR and is communicating with him that I think there is this transfer of social presence and they're kind of giving their their word and saying hey you got it you should really just immerse yourself and I think that so what I see this process of there's a number of different things that I see in terms of this ritual process of deepening presence within a VR experience so a couple of things that have started to see within the consumer realm which is that When you go into an experience and you're embodying a new character that you don't know who you are, Chocolate by Tyler Hurd is a good example of this, where you have a mirror and you just have this kind of countdown of these lights are going down. It's giving you time to be like, okay, who am I? What can I do? And so you have this process of interacting with testing your agency of what you can be able to do. And I also had this experience of Starship Commander where you're basically talking into the experience and you're talking to this computer, but it's essentially Microsoft Cognitive Services that is interpreting what I'm saying. And you're kind of led through this open-ended dialogue where you can just ask all sorts of different questions about who you are, what this mission is, what is this world, who are these alien enemies, tell me about this, you know, tell me about you. and you kind of go through all of these questions and I felt like really primed and like really sold like it was reacting to what I was saying. I was like I believe that I'm here and then they shot me out into space and I was like looking at this globe and I was like oh my god I feel like I'm here. There was a little bit of this ritualistic process, just like when you go see a movie, there's a process of the lights go down, you watch the trailers come up, and then you see all these trailers, and then the pictures of the studios come up, the credits start, and then the movie starts, and then you're ready and primed for story. So I feel like there's a similar process of us deepening our process of really getting embodied, getting present, really getting situated into the environment, where am I at, kind of connecting to the mood of the place, and then if you have any social interactions. I think all these different things, there's going to be a set of kind of best practices of how to induce this level of presence, but I'm starting to already see that.
[00:31:32.943] Richard Skarbez: Yeah, absolutely. I think you're absolutely 100% right. And I think that this is, again, a place where everyone who does VR, whether it be academic or otherwise, can learn a lot from people who do storytelling as a business. And in my mind, the best example of it comes from Disney Imagineering. So when you're getting ready to go on a ride at Disney, you're not just getting in line and going on a ride. From the moment you're getting in line, they are preparing you to believe what you're about to see, to invest yourself as fully as possible in what's about to happen. If you're going on, you know, Star Tours, from the moment you get in line, you're feeling as if you're in the Star Wars universe and you're going to have this experience. So anything that we can do to prime people to be ready for story, to be ready to experience a virtual environment, is going to make that experience more powerful. Because the best thing that we have is still our imagination. Even with all the computing power and all the rendering devices that we have now, it's what's inside our head that makes VR work or not. And the more that you can get a user into the point where they're ready to experience what you're about to give them, the more they're going to like your experience, the more presence they're going to have, the better everything is going to be.
[00:32:58.820] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:33:07.225] Richard Skarbez: So I think with the ultimate potential of virtual reality is it's I'm gonna answer this question a little bit different way. I think it's a different medium than anything else we've ever seen. In a way that, rather than trying to have a single subjective storyteller, as there's been in sort of every means of storytelling throughout history, VR enables a more objective way of telling story, of putting someone in an environment and letting them figure out for themselves what's interesting. And that's something that I don't think we've ever really had before in the course of human history. So I'm really interested to see what people do with VR in a way that's never been seen before. You know, everything else, movies are kind of like novels, and plays are kind of like movies, and everything is kind of like everything else. But it feels to me like VR is a new thing. So I think the ultimate potential of VR is to be an entirely new storytelling medium. And I'm really excited to see what people come up with.
[00:34:17.930] Kent Bye: Awesome. Well, thank you so much.
[00:34:19.650] Richard Skarbez: Thank you.
[00:34:20.071] Kent Bye: It's good to be here. So that was Richard Scarbez. He's a presence researcher who just recently got his PhD from the University of North Carolina at Chapel Hill. So I have a number of different takeaways from this interview is that, first of all, with Mel Slater's theory of presence, there's the place illusion and the plausibility illusion. So the place illusion is really covering a lot of the objective qualities of the experience and have a lot to do with the actual technology, the latency, the field of view, the graphics fidelity, and all these other various physical things that help to lead towards this feeling that you're immersed into another world. It's really focusing on your sensory experience, both your visual, auditory, and also haptics. So Richard thinks about the place illusion in terms of immersion. I think of it in terms of context of the experience, but also the Earth element includes all the different sensory experience as well as the environment that you're around. And for the plausibility illusion, that's what Richard refers to as coherence, he's breaking up plausibility into different subcomponents that he's trying to determine. What are all the different ingredients that you need in order to have a plausible VR experience? So I think there is actually a pretty close mapping between how he's breaking down plausibility and how I think about the elemental theory of presence. So for the air element, that's the social and mental presence. In terms of mental presence, there's the scenario coherence. So does everything kind of make sense within the scene? Is it plausible? It's kind of like this combination of the environmental context as well as that mental presence. But whether or not a scene is plausible kind of goes back to your own personal experience and beliefs of scenarios that you have had before. So you're kind of calling upon your memory and just making sure that everything within that scene kind of makes sense within your mind. Now, the virtual humans is trying to invoke this sense of social presence. And in this specific experimental protocol, they were just using pre-scripted avatars who were talking to each other about soccer. Now, I think if you were to start to have a one-on-one direct interaction with another human where it was clear that that was another human, I think the degree to which you have a sense of social presence just increases exponentially, just because you just know that something is prescripted and in the can, especially if you have no ability to interact with the other people. And in some of the interviews that I was doing at IEEE VR, I'll be exploring more about how there's this connection between emotional presence and social presence so that you can get some transfer of emotional engagement when you're interacting with other people as well. Now in terms of the earth element, that's the body, your own physical body representation. And this is the thing that I think was probably the most significant finding that Richard found was that if you have your full expression of your body within a virtual experience that you have tracked with like these high-end OptiTrack trackers that Once you have all of your limbs and your entire body within a VR experience, and that just increases your level of immersion and presence exponentially. When it comes to consumer VR, I think that this is where the Oculus Rift, I think, in the long term actually has a huge competitive advantage because it's going to be a lot easier for the Oculus Rift to do computer vision through those cameras to do full body tracking than it is to start to put sensors all over your body with the Vive. I've started to see some of the Vive trackers and have my feet tracked, but there's all sorts of issues with having to insert all sorts of new sensors on your body. The advantage to the Vive is that you get so much more precise tracking of your body in those movements. But just the way that they're doing the tracking requires it to be more of an active signal that's being sent back to the computer, which means that it's not just like this passive reflective marker. And that just becomes a lot more challenging to do the level of full body immersion in the longterm. But for the Oculus Rift, as computer vision gets better and better, they're going to be able to turn on more and more features of being able to have full immersion and full body tracking. The problem with the Rift though is that they have the front facing cameras and then they have the lack of a full room scale. And so you have this embodiment, but it's kind of constricted to you not turning around past this sort of invisible 180 degree marker. And then it just becomes a lot more cognitive load for you to have to be able to track. So there's all sorts of different issues both with the Rift as well as with the Vive when it comes to doing full body tracking. But from what I've seen from the academic community as well as some of the experiences that I've had that are high-end digital out-of-home entertainment VR experiences, having your full body within the experience just increases your level of presence and immersion exponentially. So the other thing that Richard is looking at is the correlation between your expression of your agency and the believability of the reaction of the environment to that expression. So kicking a soccer ball or being able to interact in any dynamic way within an experience. And I think that is just the fire element, the sense of active presence of how you're able to explore or express your agency and interact with the environment in some way. And the more that you're able to interact with the environment, the more believable that it becomes. So I do think that there's an element of emotional engagement that is also being invoked in these different VR experiences. And I think that Richard's right in identifying that this is actually the biggest challenge for academia, because a lot of these academic researchers aren't skilled in terms of trying to create these emotionally moving, immersive experiences, whether that's through awe-inspiring art, or from music, or from story and narrative. And so this is where I think there will need to be some sort of collaboration between creatives and the research community. And in talking to different researchers, some of the needs that they need is to be able to break down the different components of an experience and be able to dial it up or down to be able to do these experimental protocols. You know, I had mentioned in this podcast, why don't you use Perl in an experiment? And the reason why they don't is that they don't have control to be able to dial up or down any specific variables within the experience that they can actually use within a test. And so that's where I think that if either developers are able to kind of dial up or down different dimensions of an experience, then there can be more presence research that's being conducted. But I think this is sort of a broader question is what is the limits of researchers doing research into the degree of someone's presence within a VR experience? I mean, there's so much about a VR experience that is totally up to both the content, who you are, what you like, your preferences, your memories, your life experiences. And there's just so many factors that I think go into whether or not you are really engaged in any experience that it becomes a very difficult process for a researcher to come in and try to quantify that, but not only quantify that, but also determine all those things are going to just really resonate with you as an individual. So just as an example, talking about the differences of gender, for example, in this specific experience as to whether or not the 15 or 20 women that went through this experience were able to enjoy kicking a soccer ball or not, I don't think that you're necessarily able to make broad generalizations about all genders or all people in general based upon a very small sample size of a scientifically controlled presence research, where the experience may have not been all that compelling for people in the first place. You're essentially in a bar listening to people talk about soccer. And that may be a huge thing for someone if they are a huge soccer fan and they've never done VR before. But if you've done a lot of VR and you've go into this experience, you're not going to report that deep level of presence. So I think that there is some degree of limits of how far you could take this presence research. However, I also think that there can be very interesting things that are discovered. For example, Richard found that the tracking the body in this research was one of the most important components. So, when it comes down to you as an experiential designer, what are things that you think are going to be some of the most highly leveraged components of your experience that are going to be cultivating that sense of presence? Both based upon your own experience, but also based upon what other people are finding as well. So, as an experiential designer, you have all these different trade-offs of where to put your focus and energy. Do you focus on modeling? Do you focus on sound design? And it really depends on so many different dimensions of the context of what you're doing. And throughout the rest of this week, I'm going to be talking to other presence researchers, what they're looking at, and as well as getting some feedback into how my elemental theory of presence fits in. So that's all that I have for today. Just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a donor to the Patreon. Just a few dollars a month makes a huge difference. And I'm also in the process of looking at different ways to engage the Patreon supporters, whether it's through updates or live streams. I'm going to be going to SIGGRAPH in a couple of weeks. One thing that I know that I really want to do is want to come back, do like a live stream debriefing of the big insights of experiences and interviews that I did at SIGGRAPH. So, you can go to patreon.com slash Voices of VR, sign up today, and I'll be having more information about the different tiers and rewards here probably over the next week or so. So, donate today at patreon.com slash Voices of VR. Thanks for listening.
