Jesse Schell has been involved in virtual reality for over 20 years, and he’s also the author of the popular The Art of Game Design: A Book of Lenses. He was able to combine his background in VR and game design at Schell Games to create the very popular I Expect You to Die, which has been the top-ranked experience at Oculus for the last three months. It also took home three Proto Awards including the top prize for Best Overall VR Experience, Best Gameplay and Best Interactive Design. I had a chance to catch up with Jesse at Oculus Connect to talk about some of VR design principles that has helped I Expect You to Die to do so well.
Listen:
One of the key takeaways that Jesse shared is that you have to design for presence first, and that you need to avoid anything that might break presence. He also went against Oculus’ own best practices and designed the experience to work primarily with a mouse interface. He talks about the importance of this decision in terms of feeling engaged in the world, and some of the counter-intuitive insights that they made including that the brain could make sense of looking behind you and pushing the mouse in the opposite direction in order to reach out. Despite the success of I Expect You to Die, Oculus still recommends developers to not rely upon a mouse interface.
One of the important aspects of designing a stand-up VR experience is to avoid proprioceptive disconnects that might prevent the user from believing the scene. Jesse said that a huge inspiration for him was Daniël Ernst’s Blocked In because sitting at a desk was such a familiar situation for so many people. That inspired him to create the setting for I Expect You to Die to be sitting in a car because it’s something that is so familiar to so many people.
When designing I Expect You to Die, there was some internal debate as to whether or not a good VR experience could be designed that didn’t have any motion. They wanted to feel like a superhero, and then realized that there are a lot of tropes about superheroes escaping from impossible situations, and they used these familiar situations to create an engaging and fun escape room type of scenario. Jesse also says that his team got a lot of inspiration from different real-life escape room games, and that Schell Games is actually in the process of creating an escape room that blends collaborative physical puzzles with a virtual world.
Jesse says that social VR is going to be the aspect of VR that is going to be the most compelling. His dream is to build a VR MMO, and he talks about some of his early work in VR in building out some of the technology to make that happen. He says that he’s made a detailed plan, but that it’d take about $10 million dollars worth of investment in order to make it happen so he’s not expecting it to happen unless he’s able to find someone to back it.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:11.884] Jesse Schell: Yeah, I'm Jesse Schell. I'm the CEO of Schell Games, and we're working on a few different VR projects. The one that's been getting the most attention right now is we've done a game called I Expect You to Die, which is kind of a comedy, VR, spy puzzle game that has been up on Oculus Share for a few months, and it's been the top-rated game for a couple solid months, and we're very excited. Last night, we got a few different awards from the Proto Awards for it, so it's been received pretty well.
[00:00:38.483] Kent Bye: So what is it about the game that is so compelling for people? What's actually happening?
[00:00:42.584] Jesse Schell: Yeah, so in the game, the idea of it is you are a secret agent who has to escape from death traps. And in the initial version that we put out there, the idea is you've infiltrated this spy car that the evil Dr. Zor is carrying on his airplane and he's taking it somewhere. And your job is to get this car out of the plane. But the problem is the plane is full of poison gas. And so you have to figure out how are you going to get this car out of the plane. And so it plays kind of like an escape the room puzzle. And I think a lot of the appeal for it for people is it's very much about manipulating the environment, manipulating objects in the environment. And we worked very hard to create a strong sense of presence, of really being in this place, really changing and manipulating things in this place. And sitting in a car is a thing people are very familiar with. And so it feels surprisingly natural.
[00:01:32.538] Kent Bye: And so, yeah, talk a bit about some of the user interaction schemes that you did in order to move objects from place to place in terms of trying to make it feel comfortable in VR with a controller.
[00:01:43.387] Jesse Schell: Yeah, one of the things I think I'm most proud about is what we use is we don't use a game controller, because we honestly think game controllers are kind of antithetical to VR. They are abstract. You know, I make a tiny motion with my thumb and a character runs. Like, that has nothing to do with reality. And you're trying to simulate reality when you do VR. So, against Oculus' and everyone else's advice, we use the mouse as the primary interface. Because the mouse, you actually put your hand on it, you actually can reach into the world, and you can move your hand from side to side. So in the game, you're kind of controlling what we call the telepathic ray. you can just kind of grab objects with the mouse and then move them around quite naturally with your hand. And one of our amazing discoveries is that you kind of have this question like, well, if the mouse is in front of me as an actual physical person, does that mean I can't ever reach anything behind me? Because I can't put the mouse behind me. I can turn around and look behind me. And we realized that if we make the mouse frame of reference travel with your head, your brain thinks that's totally fine. It has no problem with that. So you can turn around 180 degrees behind you, reach out in the opposite direction that you're facing, and your brain thinks it's totally fine to be reaching out behind you. And that's sort of the central aspect of the way this thing works. And then we use the mouse wheel to kind of reel things towards you and away from you. So we kind of get X motion with the mouse and Y motion with the mouse, and then you get your Z motion with the mouse wheel.
[00:03:11.025] Kent Bye: Wow, so yeah, that doesn't seem like an intuitive thing. How did you kind of stumble upon this discovery?
[00:03:16.208] Jesse Schell: At Shell Games, about once a year, we stop everything we're doing and we do a thing called JamWeed. So we got about a hundred people and we just stop and everybody works on passion projects for a week. And Jason Pratt, who's one of our senior engineers, he started working on VR and started just doing experiments and he started playing around. He had this idea for the mouse and he started playing around with it and he was explaining how he was going to try this thing and I was like, that's never going to work, dude. Then he tried it and we're all like, holy crap, it just works. And at first I thought it was just me, but then the more people tried it were like, this actually works and it's really weirdly intuitive. This sort of blend of using your head and your hands to reach out and manipulate objects in the environment. And when we saw that worked, then we started thinking, okay, what kind of game can we make around this interface? Because we felt it was something pretty special.
[00:04:03.280] Kent Bye: Nice. And so, talk a bit about being at the Proto Awards last night and, you know, all the different recognitions that you got last night for the game.
[00:04:11.163] Jesse Schell: Oh, geez. I mean, it really blew my mind. I mean, we were up for, like, three different awards, and I thought, well, you know, we got a shot. This is pretty popular. We might get one of these awards. And we were very excited. First, we got the award for interaction design in VR, and we thought that was really great. We were like, this is awesome. We can now go home and be very proud. And then the award for best gameplay came up, and we got that as well. There was a third award we were up for, and someone else got it, and we were like, okay, that's cool, because this is getting weird. But then, at the end, they announced a surprise award that had not been announced ahead of time, of sort of best overall experience, and to our surprise, we were nominated for that, and then to our massive surprise, We won that. So that was huge for us. But none of that for me compared to the fact that Ivan Sutherland shows up there and gave an amazing speech about the history of VR, the future of VR, and the nature of innovation. And for me, that was just an incredibly special moment. So it was a really, really special night for us.
[00:05:08.764] Kent Bye: Yeah, and you had mentioned in your first speech that you've been working in VR for around 20 years. So what have you been doing in VR? And how did you get into making games now?
[00:05:19.525] Jesse Schell: Yeah, so I fell in love with VR probably back in 1992. I just thought it seemed like a really, really cool idea, and I happened to be going to graduate school at Carnegie Mellon at that time, and there was some VR research going on, and so I started working with the Studio for Creative Inquiry there. We did this thing called the Networked Virtual Art Museum. We basically started doing We had a vision that VRMMO would be a thing. Of course, MMO wasn't even a thing then, but that was our vision. Lots of people in like an online VR world together. So I worked with a group of us. We kind of had this thesis project that was all about what are the... So we were in a computer networking master's degree. So we were like, what are the network structures if you want to make if you actually are going to make massively multiplayer games, how do you build the servers for that? And we built prototypes of our thing and we put it with VR of the day. And that first got shown, there was an exhibit at the Guggenheim in 93, and it was like us and Thomas Dolby, weirdly, was into VR at the time, and a couple other people. And we showed that, we did this ancient Egyptian temple exhibit. Then I got a job working at Disney. I started at Disney Imagineering at the virtual reality studio there. So we built a thing called Disney Quest, which some people might know. And that was, for me, my real education in VR, because we were inventing helmets, inventing experiences. We were working on silicon graphics machines that were way ahead of their time. while other people were like at that point in time the game industry was focused on like the Sega Genesis and here we were doing 7,000 poly characters like in real time in VR at 60 frames a second and so that kind of gave us a bit of a jump and we learned a hell of a lot doing that. Went from there I ended up back in Carnegie Mellon in 2002 where I've been teaching at the Entertainment Technology Center teaching the Building Virtual Worlds class where we've kind of continuously been making VR worlds there for over a decade. So that's been amazing. And I also run Shell Games. We have about 100 people in Pittsburgh. So once VR started going to be something that was going to go to the consumer space, I'm like, I'm ready, man. Let's do this. Let's get some stuff done.
[00:07:17.457] Kent Bye: So what were some of the lessons that you've had from your 20 years of VR experience that you were able to translate into the I Expect You to Die VR experience?
[00:07:25.268] Jesse Schell: Yeah, I think one of the biggest ones is that presence is first and foremost. Game designers tend to focus on gameplay first and foremost. People who are new to VR are often kind of like, well, presence is sort of nice, it's sort of cool, but we need to focus on the gameplay. And you don't. You need to focus on the presence. You need to make sure there's going to be nothing that's going to break immersion. And they're so easy to break immersion. Bad sound effects, bad interaction design. There are so many things that if they just seem a little wrong, you're reminded, oh wait, this isn't real, this isn't real. Because your brain and your body want to believe it's real. And if you can take all the impediments out of the way, they will believe it's real. But you have to focus, focus, focus on that. So that, I think, is the biggest thing. And then after that, the other part of it is you've just got to experiment. You've got to try stuff. The old rules of thumb don't work. You've got to be totally cool with trying new stuff. You've got to strip it back to basics, and you've got to understand the neurology. You need to understand the psychology, how the brain works, because VR is really a kind of brain hacking, right? We're trying to make the brain believe a thing that isn't actually true. And so the more you learn about that, about, well, how does the visual system work, and how does the vestibular system work, and how does proprioception work, the more you can start to understand what will and won't work. But even then, you've still got to experiment.
[00:08:42.766] Kent Bye: It sounds like you decided to be sitting in a car because that's very comfortable. And I know that a lot of experiences with two-handed interactions, there's also standing up experiences. And so if you were to translate this from a sit-down to a stand-up experience, then how would you translate that to maintain that sense of presence then?
[00:09:00.912] Jesse Schell: Yeah, first of all, I would not put it in a car, right? One of the easy ways to have a bad immersion experience is to have proprioceptive disconnect. Like, you know, your body's sense of how your limbs are positioned is something it takes very seriously. And if I'm lying down and I put on a VR helmet and it's a game all about standing up, immediately my body's like, well, this is fake. That's fake. Forget all this. It's fake. But if it all lines up, your body's like, maybe this is real. This could all be real. And so Oculus had made the announcement, seated, seated, seated. We want seated worlds. That's what we're going to recommend, seated worlds. All their demos, they do standing. I don't understand that, but whatever. So we started making a seated world. And then for us, given that we recognized that the mouse interface was a good interface, and well, I actually had somebody from Oculus just today. Tell me, well, we don't recommend you use the mouse. I'm like, we've been number one on your charts for two months, and we're a mouse interface. And you're recommending people like that's a bad idea? I don't know. I don't even understand. Anyway, since we're using the mouse, the mouse wants to be a seated experience. So we're like, let's make a thing that's about being seated. And what really inspired us was Daniel Ernst's demo, Blocked In, because that's an experience where, in the virtual world, you're sitting at a desk looking around a room. That's all you do. And for me, it had been, up to that point, the most immersive VR experience I had ever had. And so we kept looking at that, like, what the hell is it about that? I'm not even doing anything. What is it about this that makes it immersive? And so as we started to decode that, we realized, first of all, no proprioceptive disconnect. Second, it was a familiar thing. I know what sitting at a desk is. And I've, in fact, got a real desk in front of me, and I can actually lean on it. And if that lines up with the virtual desk, that's really good. And then third, lots of interesting things to look at. And when your brain starts kind of looking into lots of interesting things around you, that makes the space and place that you're in seem very, very real. It's a lot of work because it means your world can't be spared. You've got to put a lot of attention to detail in everything around you. But that was when we started to kind of crack the code of getting that done. Standing experience is completely different, and we're doing a standing experience. We're doing one for the Vive right now. We're doing this game, Water Bears VR, which is a puzzle game all about building these kind of pipe structures. And you use both hands to be able to build pipe structures to get water to flow in all these crazy pipes that go all around you. And we're finding it's a great two-handed standing experience, because everything sort of takes place on this virtual table in front of you. And you manipulate things, and some things go under the table. So it's just four different hardware interfaces. You have to design radically different experiences if you want them to be great.
[00:11:38.716] Kent Bye: Yeah, and in the academic community, they talk about near-field VR, or what Paul Bettner would call the sweet spot of VR, which is that if you kind of hold out your hand at arm's length distance, it's kind of like the distance from your fingers to your head is kind of like the sweet spot of where the stereoscopic effects of all the objects that are around you is the greatest. It sounds like you're kind of saying a little bit like you can surround yourself in those objects and that you have things to look at. Did you attempt to do that in some of your experiences in order to create a better sense of presence and immersion for standing up experiences?
[00:12:11.792] Jesse Schell: Yeah, definitely. I mean the notion of sort of stereoscopic vision works the best and the strongest when it is things near your head. And it was something we figured out a little bit late in the prototype development for I Expect You to Die, for example. What we did there was we had just realized, we just for fun kind of locked some objects in the air and we were like, wow, it's really, really kind of fun and immersive to have these things that are near your head. So we made a mouse interface, I don't know, you hold the right button down or something and it locks an object in the air. This is partly cool looking, but it's also partly very practical. Because if I've got to pull out a bunch of tools and I want to arrange them around me, I can just lock them in air where I want them to be. And that's kind of nicely practical. And we thought it would be counterintuitive to people because it's so strange, but it's not. It's kind of like I'm just putting this where I want it to be and it stays there. A weird counter-intuitive part of it, though, was when we first locked the objects in air, we would have people say, oh my god, the world crashed. It crashed. The game crashed. But it didn't crash. It was just strange that this object that they'd been moving in their hand a second ago suddenly was frozen in space. And enough people said it that we said, well, Maybe we just can remedy it in a simple way. So we added a simple bob motion, a simple floaty bob motion on these things in space. It's not very much. It's incredibly subtle. But something about it seems much more natural, and it doesn't seem like the world is broken. The world still seems alive.
[00:13:30.411] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:13:38.096] Jesse Schell: I think when we start talking about the ultimate potential of virtual reality, you can't ignore social. Social experiences are by far, by incredibly wide margin, going to be the most powerful aspect VR experiences because you're going to have experiences in worlds with other people where you will be able to look another avatar in the eye, avatar at human scale, look at them in the eye, have facial expression, have natural voice and speaking, have natural gesture, and all of the things that are part of normal face-to-face human communication are going to be part of kind of an enhanced virtual world and I think the power of those experiences is by far going to be the most intense and memorable part. It's going to be the part that people are going to be most willing to pay for. It's going to be the part that's going to keep people coming back. And we've seen this. It always happens with games. They start single player. Eventually the social comes. The social is where everybody makes the money and the social is what ends up lasting. So that's the part I think I'm most excited about.
[00:14:46.089] Kent Bye: So it would be like a social version of I expect you to die then?
[00:14:50.018] Jesse Schell: I would love a game where the two of us are together. One of the big inspirations for I Expect You to Die was we'd had this argument about motion in VR. I was arguing against motion because it makes you motion sick. Let's make games without motion. Other people are saying you can't make good games without motion. Somebody on the team said, look, it's like you put the headset on, you want to be a superhero. How can I be a superhero if I'm tied to a freaking chair? And we're like, oh my god, that happens all the time. Superheroes and secret agents are always getting tied to chairs. And then they've got to figure a way out of it. What if we made that our game? And there's so many of those where often it's like two people are tied up. Batman and Robin are tied up and they've got to like solve this problem together. I think it would be really cool like a two-player escape the room game where the two of you have to collaborate in order to get out of the room. I would love to be able to do that. But the big thing I want to get to, I am trying to find a way to get VR MMOs going. We've got a whole plan for how it would work. The problem is you're talking at a minimum of a $10 million investment, at an absolute minimum. So that's not something to trifle with, but we're hoping we can find a way to get it done. So if somebody out there's got $10 million, give me a call.
[00:16:01.457] Kent Bye: Great. And I'm just curious if you've had a chance to do actual escape the room, like physical escape the room experiences. I know that a lot of those were inspired by immersive theater, like Sleep No More, you know, get together in a room and, you know, solve these puzzles like real actual physical puzzles in order to get out. Have you had a chance to try some of those?
[00:16:17.627] Jesse Schell: Yeah, no, I've done some and a number of people on the team, we use them as actual inspiration and I can't talk about it a lot, but we are designing one right now that's going to be released somewhere in the United States sometime this fall. and I'll be very excited when we can kind of talk about that because it's not like the traditional Escape the Room games are completely physical and we're making one that is kind of half physical half virtual that we think is going to be kind of interesting and powerful but I can't say much more about it than that right now. Anything else that's left unsaid that you'd like to say? Man, I'm so excited just being here, seeing Oculus Connect this year has been super exciting. There's so much going on. 2016 is going to be just incredibly life-changing in terms of what happens next in VR. And I'm just so excited to see what's going to happen.
[00:17:09.019] Kent Bye: OK, great. Well, thank you so much. Thanks, Kent. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.
