Jason Jerald has been working in virtual reality for 20 years now, and he’s celebrating his 20th VR anniversary with the launch of an epic textbook called: The VR Book: Perception and Interaction Design for Virtual Reality. It’s nearly 500 pages and contains over 300 academic references covering the psychological and technical considerations for creating a comfortable VR experience.
The VR Book focuses on human elements of VR, such as how users perceive and intuitively interact with various forms of reality, causes of VR sickness, creating useful and pleasing content, and how to design and iterate upon effective VR applications. It is not just for VR designers; it is for the entire team, as they all should understand the basics of perception, VR sickness, interaction, content creation, and iterative design.
In this interview, Jason gives an overview of some of the chapters and sections that are covered in the book including perception, how to minimize adverse health effects, creating VR content, a list of interaction patterns, iterative design strategies, and where VR is headed.
We talk a bit more about the importance of fidelity in VR experiences, the virtual body ownership illusion, and how he thinks about the different fundamental components of presence.
The VR Book will be launching on August 9th at SIGGRAPH next week, and will eventually be available online for purchase here. It’s not available for online purchase yet, but you can sign up to Jason’s e-mail list to get informed as to when the book becomes available. It’s sure to be a vital collection of all the latest research for what is known and not known about creating comfortable and compelling VR experiences.
If you’re going to be at SIGGRAPH next week, then check out Jason’s VR Book presentation on Monday, August 10 at 4:00pm in the Immersion Dome at the VR Village (South Hall G). He’ll also be signing books on Monday, August 10th at 5:15pm at the SIGGRAPH bookstore where hard copies will be available for purchase.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:12.039] Jason Jerald: My name is Jason Gerald, and I've been doing virtual reality for quite a while now. It's actually my 20th anniversary this year. And on this 20th anniversary, we'll be launching my book, the VR book, Perception and Interaction Design for Virtual Reality at Seagraph, which will be my 20th year in VR and 20th year at Seagraph. More specifically, here at the conference, I'm representing Seagraph. We're putting together a contest at the Seagraph conference. We'll have the best of the best, the top three VR experiences. the developers will be showing that on stage. They'll be showing their VR experiences, the audience will vote for the best, and the winner will be giving some to-be-determined prizes. So I'm really looking forward to seeing some exciting, innovating content at that contest.
[00:00:57.073] Kent Bye: And so with the SIGGRAPH, it's focusing on graphics, and I'm assuming graphic fidelity. So is the more photorealistic VR experiences kind of what they might be looking for?
[00:01:07.223] Jason Jerald: Yeah, so Seagraph historically is very film oriented. It's quite wide in variety of different aspects, everything from website design to film. But film is sort of more of the focus of the conference, at least it has been the last 10 years or so. Back in the 90s, they were big on VR. And now, of course, like everyone, they want to get back in VR as well. They are really pushing virtual reality this year at the conference. We're going to have an exhibit. We don't have an exact number yet, but somewhere around 50 exhibitors. Again, the best of the best. So similar to what we have here at Silicon Valley Virtual Reality.
[00:01:41.446] Kent Bye: And so yeah, maybe you could talk a bit about your VR book and what you were attempting to cover and gather together and who the target audience might be. Absolutely.
[00:01:50.872] Jason Jerald: So I come from very much a technical background. I'm an engineer. But what I realized recently is there's so many others doing such a great job, all the hardware manufacturers as well as the developers, that I transitioned the book more towards content creation and design. help people understand the high-level issues such as motion sickness or how to interact with the world. There's some great content out there right now but a lot of what's being shown is largely just looking around. You don't, for example, look down and see your body or you can't use your hands in a lot of experiences. There's some of that but it's not clear yet how to interact effectively in virtual reality. So one section is devoted purely to interaction. I have something like 16 different interaction patterns like walking as a pattern, then subsets of that individual pattern are specific implementations of walking. So you might have a walk-in-place sort of system, sort of like Stomps had a really good walk-in-place system where you're kind of faking walking by sitting down. Advantages of that are, of course, you're not going to get tired. You can do it a lot longer. But you might have a real walking system as well, where you have a large room that you can walk around. So there's different techniques within those patterns. Or you might have sort of a direct hand technique, which is kind of the most straightforward thing to do, is to just reach out with your hand. You see your hand, and you just reach out and grab things. But then there's extensions of that. There's more specific techniques where you can have sort of a stretchy arm that you can reach off into the distance and grab things further than you can in reality. So there's sort of this continuum from trying to replicate real reality, which is important if you're doing like a training simulation, right? you want to train someone to work in their personal area instead of reaching their hand far off in the distance that they can't do in reality. But in other cases, if it's an entertainment application, maybe it's okay if you reach out in the distance. So none of these techniques, there's no best technique or one answer. It really depends on the specific project. And so I try to, you know, kind of show what are the advantages of different situations or ways of reducing simulator sickness, or what are the iterative stages of design, all the way from you have to define the project, very loosely, you define the project, you make the project, and then you get feedback from users. I call that the learn stage. Then, of course, you know, repeat, repeat, repeat. virtual reality it's so important to iterate more than any other industry because there's so many unknowns at this point we don't know all the answers and so you just got to test test test not only yourself but to get you know real users or even people that you know aren't experienced in virtual reality and so those are some of the things we cover in the book
[00:04:22.441] Kent Bye: Yeah, and it sounds like because you've been doing a lot of virtual reality consulting and creating your own experiences, it sounds like aggregation of all the lessons that you've learned in terms of the biggest problems that you may be facing and the advice that you may be giving, you've sort of consolidated all that into this big, giant VR book.
[00:04:38.778] Jason Jerald: Absolutely. And that was part of the challenge, because if you're a game studio, that's going to be very different if you're, you know, the military and you're trying to do something completely different. The military is going to have requirements, right? You're going to have these formal requirements up front. We have to achieve this to interface with some other system that's already well defined, where a game studio probably isn't going to do that, but they're going to storyboard. They're going to do different things that the military might not do. So on the iterative design stage, it's not like there's, you know, one process that this is how you should do things. No, it very much depends on the project, even within the project. It might be on your first few, you know, iterations, you're not even doing a lot of that. You're just very loose on those definitions and you'll come back and redefine those and maybe even define different processes as you go through. So, you know, this isn't a book that says this is how to do VR. These are just suggestions of what might work for you and you pick and choose what's appropriate for your project.
[00:05:32.735] Kent Bye: I see. And there's also the subtitle of perception, and perception is obviously a pretty huge component of VR. So what types of things about our perceptual system are you covering in the book?
[00:05:43.117] Jason Jerald: Oh, I totally geeked out on the perception section. Human psychology and perception just amazes me. And that's one of the things I love about virtual reality is it brings everything together. We need everyone involved. It's not me having an engineering background, I realize I can't draw. If you see my programmer art, it's going to be a pretty horrible thing. So we need to bring in everyone, especially understanding the psychology and the neuroscience to understand how we perceive reality. If we're going to replicate reality, it probably makes sense that we understand real reality before we understand virtual reality. So all the way from, you know, the obvious of human vision, you know, what is visual acuity and, you know, some of those numbers, but also at a much higher level, how we perceive reality, essentially, even real reality isn't really real. It's still an illusion, right? Because what I'm seeing, the photons reaching my eye, it's not like the objects reaching my eye. It's just a representation of light bouncing off that object coming into my eye. And so there's this difference between objective reality and subjective reality and all these illusions that take place at different levels. And so I discuss different types of illusions and where things go wrong, why do those illusions occur, also how the different sensory modalities go together. So, for example, how do we fuse sound and vision together? you know the ventriloquist effect that you know it sounds like it's coming from where you're seeing it come from even though it's not coming from there or the proprioceptive sense of where are my hands in space you know how much can you get away with where it doesn't have to have this perfect tracking in some situations or the vestibular sense that you have to have that everyone I'm pretty well aware of now of the problems with simulator sickness or motion sickness or what goes by many names. But if those aren't tightly synced up and they're not done well together, then there's problems. It all comes down to perception.
[00:07:29.149] Kent Bye: Yeah, one of the, I think, researchers that I find the most fascinating in this field is Mel Slater and the whole virtual body ownership illusion and all the work that he's doing. So maybe you could, you know, comment on some of the work that he's doing in terms of, like, the sense of self and the sense of body.
[00:07:43.779] Jason Jerald: Absolutely. So Mel Slater is really the king of talking about presence. And presence and immersion are defined in different ways, but The way Slater and others mostly in the academic community define the difference between immersion and presence is that immersion is the objective quality of the fidelity of the hardware of the technology and presence is how do we perceive in our own mind the psychology the perception of those pixels on a screen. So, you know, the resolution on the field of view on a head-mounted display, that would be the immersion. But I can just simply close my eyes. You can give me all the technology in the world, at least visually, and if I close my eyes, it doesn't matter. I don't feel that sense of presence. So presence is in the mind. Immersion is the technical kind of objective nature. And Mel Slater goes beyond that. He talks about the place of illusion, which is many people are aware of that anyone that's tried a great VR experience is that sense of being in this location even though we're at the exhibit hall here you feel like you're in this other imaginary or perhaps replicated reality location and so I divide those things a little bit different than Slater because he kind of takes a little bit more of an academic slant on that but Divided into sort of the illusion of a physical space Which is that sense what now Slater talks about a lot at least part of what he talks about Then I have the illusion of social presence, right? So if I'm an avatar and I'm talking with you can't as if you you know, we're actually in virtual reality I would have that illusion that I was talking to you even that you're in the space even though you're not and so I kind of break down different senses of presence and what are those illusions of how we perceive this artificially created realities
[00:09:21.978] Kent Bye: Yeah, I had a chance to talk to Mel and also Richard Skarbez at the University of North Carolina who's been looking at it. And the way that I understand it is presence is two components of both the place illusion and the plausibility illusion. And the way that Richard describes it is that the place illusion is the sense of being in another place, and that's the immersion. And he actually sort of started to coin the plausibility illusion as the coherence, which is that everything makes sense and is plausible and that you have agency and you can actually have an impact within the world. And so that's how I've been sort of thinking about presence is that you need both of those, the coherence and the immersion in order to have that full sense of presence, but the thing that's the most Sensitive is that coherence if there's any little tweak that breaks then it can the breaks and presence tend to come from the lack of cohesion rather than the lack of Immersion which is sort of the technology of running out of consistent frame way that's sort of getting locked down But making sure that the whole world makes sense to what our expectations are I think is the other key component of that sense of presence
[00:10:19.688] Jason Jerald: Very interesting. Yeah, I know Rick pretty well. I talked to him a little while ago We got a little bit into that but I haven't gotten into what he calls cohesion. It sounds like coherence. Yeah, I'm sorry But yeah, that's definitely another way of looking at it and reminds me I need to get back in contact with him because it might make sense to you know, describe presence and immersion and those things, you know interesting to see how he Sees it and he might have some great ideas as well to include in the book
[00:10:46.077] Kent Bye: Well, it sounds like some of those social presence could be considered a coherence of all those social cues. And I think that his point is that there's been a lot of focus on the immersion, but not a lot of focus on the coherence of making the whole world, making sure that the social cues of the in of any uncanniness of that is eliminated or that there's any rules that are violated that can break those expectations and have the breaks and presence.
[00:11:12.003] Jason Jerald: I see. Yeah. Bricks and presents are obviously it shatters your illusion of reality. Suddenly you realize that I'm in this exhibit hall. I'm not in this virtual world and sort of like, you know, cognitively you realize that we're not quite fooled yet and fully believe we're in that world. But our lizard brain kind of believes that what I call the sort of visceral processes, you know, we feel like our minds fooled at a very sort of emotional primal level.
[00:11:38.393] Kent Bye: Yeah, and I guess the other lesson that I took away from IEEE VR is that the uncanny valley is in-dimensional, is what Richard had said. It's not just avatars, but the sound and everything else, and that you can go low-fidelity and have, like, really sense of presence and that coherence. But then if you start to strive towards that high fidelity, then you have to have realistic haptics, realistic sound, realistic everything. And so you tend to strive for that, but as soon as you start to go towards that, then you kind of fall into the pit of the uncanny valley. So I'm curious, when you're advising different companies about how to navigate this, creating a sense of really deep presence, choosing the level of fidelity, but also trying to maximize the sense of presence.
[00:12:18.569] Jason Jerald: So I actually kind of break it down and give a few different ways of breaking down the fidelity. One is the experience fidelity. How closely does my experience match what the content creator intended? And so you would think, well, you want to get it as close as possible, but no, that's not, you don't necessarily want that. It's not that one end of the spectrum is better than the other, because you might have sort of this free roaming world where the users sort of make up their own stories and you're just providing the context, right? So that's kind of on the other end of that spectrum of the experience fidelity. Or you have interaction fidelity. If I want to represent reality as closely as possible, like I was mentioning briefly at the beginning of the conversation, then I want high interaction fidelity. But if I want to shoot fireballs out of my hand, then that is not realistic. But that's okay. You don't necessarily want that. And then I have another type of fidelity which is really about how abstract the world is. And so we don't necessarily need photorealism, right? I can be in this cartoon world and still feel very, very present in that cartoon world if it's done well. You know, I don't need to be fooled as if it's the real world. Now, immersive film is moving towards that. They're moving towards that high fidelity of the realism of the world. And so, again, it depends what you want to do. You can imagine a very artistic sort of world that is, you know, just a calming experience. It doesn't have to be realistic. Again, one's not better the other. It kind of depends what you're trying to do. As far as specific, an example of a suggestion of the Uncanny Valley, is one thing that's extremely challenging to do is to represent a character moving around in the world. You can pre-animate that character so he walks with his legs, but to match up those animations so that the legs look really good and it looks like he's walking around properly is kind of tough to do. You can do it, it's just going to take a lot of effort. So in that case, what I suggest to do, if you want to get something working quickly, don't try to replicate the legs, use characters without legs. So if you have, you know, a robot that's kind of floating in space, and then you can kind of, you know, maybe you want to add legs later, but as a early in the project, do something simple. And that can be quite convincing. Maybe the robot just kind of leans as he's moving forward and you don't have to worry about the legs because the legs can look better. But if those legs are awkward in any way, You'll have this break in presence like what happened to this guy's legs.
[00:14:34.288] Kent Bye: His legs are broken I don't get it and then boom you pop out of the experience Yeah, and that's the thing about the SIGGRAPH competition that I have a little hesitation in my own gut Is this striving towards super high fidelity of the sense of place illusion? But you know the sense of coherence is sort of left behind and that may actually be a terrible VR experience or maybe half of a VR experience that looks great, but yet isn't as Immersive or engaging as something that sort of goes in the lower fidelity spectrum
[00:15:00.052] Jason Jerald: And that's especially challenging when you put up a picture, you share a screenshot, or you have a video, because that realistic video may look better, but if it has a lot of latency or it has other problems of causing sickness, you totally don't get that from what you do from the actual experience. And that again goes back to what I call a visceral communication, which is not a communication of a language like we're speaking right now, but sort of a human to human level or human to computer level or human to nature level that you can't describe in words, right? If I'm sitting up on top of a mountain and I look down, you know, and have this very sort of Surrealistic experience because I'm one with nature, right? I can't describe I'm trying to describe it with words right now But anyone that's done that, you know, we'll get that words can't do it justice It's the same thing with VR is I can sit here and you know Blab my mouth as long as you know all day and you still won't get it until you try the virtual reality Experience and when you try to communicate that through a video you don't get it
[00:15:59.145] Kent Bye: Yeah, I've definitely found that where it's like I tried to describe my VR experience and I just get to the point be like, you know, you got to just go into it. It's a complicated thing. And I think there's things in life that are complicated like that where you don't be like, no, you actually had to really be there and kind of witness it. What would you call the immersive?
[00:16:16.350] Jason Jerald: I call that visceral communication. And so there's different level. There's like indirect communication, direct communication. And visceral communication is a form of direct communication, right? We're not translating it into words and then conveying in words, then the person has to interpret it. So any spoken language or even sign language is sort of an indirect communication.
[00:16:36.178] Kent Bye: Yeah, I think that's really vital. And I think it's key to the new medium of virtual reality is that you can start to do that that you can't do in other mediums. And so when you think about that, what can you do in VR that you can't do in other mediums?
[00:16:48.868] Jason Jerald: Absolutely. So what can you do in VR that you can't do in other mediums? Well, you know, a great quote from one of my, actually my academic advisor, he said that we create these things, we create these worlds, you know, these other worlds that we walk through, it might be another planet, these things that can't exist in reality today and never can exist in reality. And that's what we're creating, is we're sort of the graphicists or the virtual realistics, or I don't know what the word would be. But we choreograph these experiences through different stimuli instead of just a play up on stage of audio and visual. You know, we go beyond that. We try to bring in all the senses and have people experience things that you just can't do otherwise.
[00:17:32.700] Kent Bye: Yeah, and I think that to me that's what's the most compelling thing about virtual reality is that it does feel like this brand new communications medium that is able to capture human experience that speaks to our subconscious perceptions in a way that we've never been able to do before. And just like the book can capture knowledge and information and that helps spur the renaissance, I do think that VR is going to help capture these experiences that we can't communicate in another way and spur a new type of renaissance. when you look at that what do you see as like on the horizon as for what this is going to enable?
[00:18:03.289] Jason Jerald: Yeah so I the last part of my book of course is you know what may be coming in the future and that's very hard you know predicting the future is hard especially when it's so wide open like virtual reality is right but there's things you know eventually we'll get to the point of direct neural input and output right like there's actually some organizations that have already done that. They can, you know, basically put sensors and as you think something, they can record that and then communicate it through the internet. So someone had a remote location. You have different ways you can have them sense that, even with their eyes closed, right? Direct neural stimulation. So that's obviously very, very early, but that might be where things are going. I think also making the experiences more interactive. Right now it's a lot, you know, looking around, you have a headset, which is absolutely amazing. The developers are doing great on that, but that's only 50% of the experience. I want to put my body in the world. I want to be able to look down. I mean, that can be a break in presence. If I look down and I don't see my own body, Got many, many years of looking down and seeing my real body, right? But to be able to look down and see a representation, getting back to Mel Slater's research of seeing my own body and being able to move my arm and pick things up and, you know, feeling something physically, that just takes things to a whole new level.
[00:19:17.801] Kent Bye: Yeah, the thing that Mel said I think is really interesting is that when you look down at your body, pretty much 100% of the time in your entire life, when you've looked down to look at your body, it's been your body. And so there's something wired in our brains that we look down and we see our virtual avatar, we just accept it. And that's, to me, very surreal, this whole virtual body ownership illusion, to be able to just start to Own this body as if it's your own if you have this high enough and what they say is that it just needs like Close enough body tracking and so as the input solutions are there and we start to have more interactions It's gonna be really interesting to see how the sense of self or a sense of identity is gonna be impacted by virtual reality So, I don't know if you sort of look at that in your book as well
[00:19:59.412] Jason Jerald: Yeah, absolutely. Because when you look down and you don't have to have the same color as your skin or the same shirt you're wearing, right? And I think part of you still you can have that sense that you're in this body that doesn't even look like your body. It's more of the motion that's movement of the arms and such if that's reasonable then that's a big deal or if you take the feather and you stroke it you know visually you see a feather then someone else comes along and does the same thing with a real feather suddenly you feel extremely presence like that level of presence is just taken to a whole new level and so what's interesting about that is you know empathy right you can actually walk in someone else's shoes like literally you can't do that in real life right so that's a great example what you can do with VR and you can't do in other worlds. And I think the reason for that is because I'm wearing this black jacket right now, but I still know it's my arm, right? My body is used to me seeing different types of clothing. So if I look down and my skin's a little bit closer that, you know, from the mind, from a visceral kind of a primal level, I don't really judge, oh, that's not the color of my skin. Of course it's not. I could have any color of shirt on. So we can kind of get away with putting ourselves in other bodies.
[00:21:03.793] Kent Bye: Nice, and so where did you get a lot of the information from your book? Like, what type of sources are you pulling from?
[00:21:08.699] Jason Jerald: Everywhere. So Mal Slater, I mean, you named him there. A lot of people from the IEEE Virtual Reality Conference that you attended that, unfortunately, I didn't get to attend this year because I was so focused on this book, but it's one of my favorite conferences. Of course, with Silicon Valley Virtual Reality. But I have over 300 references in the book, right? This obviously is not my you know, I didn't figure this all out myself, right? I've studied under the greats all the way from you know Virtual reality as well as fields outside of virtual reality like urban planning for example how do you place things in the environment to give wayfinding cues to help people find their way in the world because That becomes absolutely essential in virtual reality if you're doing things like virtual turns, because I may virtually turn 90 degrees, but my body doesn't feel that turn. And so it's really easy to get lost. So some of those concepts from other traditional disciplines are actually more important in VR, not less. it's really easy to become disoriented in virtual reality. So, yeah, just all sorts of books, papers, talking with, you know, great people, you know, getting their ideas. There's several things in this book that I didn't think about. So, you know, I feature a lot of some of the great content that I've tried and talking with different developers and say, man, I gotta, you know, do you mind if I show this in the book? I'd love to, you know, have you featured in the book.
[00:22:22.588] Kent Bye: Awesome, great. And so, what's next for you? What's sort of, you know, on the horizon for as you move forward into this space of VR?
[00:22:29.117] Jason Jerald: Absolutely. So, what's next is to finish this book. You could say it's sort of done, but it's still in rough form, so it still needs cleaned up. I'm having reviewers and such. looking at the book. And it's going to launch on August 9th at Seagraph. And I kind of, you know, get my life back in order. I put in 89 hours this last week of writing, which I was shooting for 90 hours. I didn't quite make it. But 3 a.m. Saturday morning, I actually, the deadline was Friday, but they didn't tell me what time zone, so I figured I was okay. But I figured 89 hours was enough. I didn't want to risk making any mistakes. So trying to get my life back in order and get back to work. You know, people say, oh, you can finally take a break. No, now is not time to take a break in virtual reality. I mean, this is too great of an opportunity to take a break at this time.
[00:23:14.620] Kent Bye: Great. And finally, what do you see as sort of like the ultimate potential for what virtual reality might be able to enable?
[00:23:20.760] Jason Jerald: I think, you know, I actually had a conversation, I think it was last night actually, virtual reality potentially, you know, we don't know if that's going to happen, but becoming an equalizer, right? As costs come down, as everyone has access to virtual reality and you get to the point of interacting in virtual reality in a very intuitive sort of natural way, then anyone can do it. It's not like you have to, you know, take a programming course and learn how to program in Linux, right? Like my grandma can go into virtual reality, someone that's not technically minded, can interact with these virtual realities. So, you know, there's potential. We'll see if that actually happens, because it could go the other way as well. But for to kind of, you know, bridge the gap between the haves and the have-nots. And so I hope that's the path that it takes. Great. Yeah. Thank you. Okay.
[00:24:05.064] Kent Bye: And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
