XEODesign‘s Nicole Lazarro has been researching emotional reactions of gamers in VR to discover the unique properties of the medium. You can’t just port a 2D game into VR and expect it to work, and so she’s calling for design leadership in the field in order to create experiences that use the strengths of the medium. She’s come up with 36 different VR design benchmarks across 12 different categories, and she presented some of these at Unite Boston and the VR Intelligence Conference. In this interview, she tells me about some of her findings in audio design in VR, experiments with low-frequency haptics with a Subpac, VR controls, emotion in VR, as well as revisiting her theory on the four keys to fun.
LISTEN TO THE VOICES OF VR PODCAST
Here’s an extended version of Nicole’s talk from Unite Boston:
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:11.992] Nicole Lazarro: I'm Nicole Lazaro and I'm the president of Zeo Design and I look at emotion on people's faces so I measure the emotional response that people have during gameplay and I'm going to give a talk called Follow the White Rabbit which is about design leadership in VR and the need for it. The basic idea is that with any platform change brings in new technology, new ways of experiencing a game and with that has to come new game mechanics and new ideas. And one of the big challenges that the VR industry is facing right now, the first one is of course audience. And that the people that are most motivated to buy VR hardware to give you that just that like incremental increase in the immersion that they have in the games are the traditional hardcore gamers, which is great. But the challenge is that the kinds of games that they want that increased immersion in are exactly the kind of games that don't really currently play on this generation of VR very well. you know, first-person shooters where you're running around, you know, fast targeting, you know, that kind of thing, really are a recipe for, you know, players not having a good time. You might get a little sick there. So then what the next thing is, is like, how can you take these people who are motivated and give them sort of gameplay? So there's a new sort of genres that are going to happen on this platform that'll appeal to the hardcore gamer, but then are still deliverable on the technology. So that's what we're going to be looking for, and that's sort of the first need for it. The idea is that it's a platform shift and what's been happening is that the developers are getting funding, the projects and the teams are getting funding, are the ones that were successful on last generation platform. And so a lot of this first wave of content is inheriting a lot of old design thinking, last generation thinking in terms of gameplay and excitement and how do you get something people want to come back to again and again and again. So that's why we want to, in a sense, follow the white rabbit into new gameplay, new types of experiences, new ways of doing things. So I'll be talking about some design strategies. Probably the most interesting one is some experiences that I've been having. We've been running some low-frequency audio testing in our game. We're working on a game as well as the consulting that I do. and I'm the lead engineer on a game called Follow the White Rabbit that we're doing on Oculus Gear VR right now. And what I was surprised is that I just started to get some audio into the game. So I tried out a few sound effects, and it definitely should create some immersion, right? You click on something, you activate something, and it should make you pull it in. What I was really surprised is that every sound effect I played and every music track that I added to the head-mounted display, everything took that amazing, you know, cafe in Paris in 1889, you know, looking out over the Eiffel Tower that's under construction. Every music I played, every sound I played, completely flattened the experience, completely pulled us out. You put the headset on, you get transported, you know, across time and across geography. If I added music, all of the music that we added completely flattened and it just felt like I was playing a computer game. And now this is including some of my favorites by John Williams and the soundtrack from Myst, which is like my go-to for creating a lot of these things. And so I was really surprised at what was happening. And it seems to me that in terms of design leadership what happens is that not only are whole teams going down the wrong rabbit hole or following the wrong star, what worked on last generation will not work on this generation. Also, it's true in audio design as well. A lot of developers are just treating those ears like eyes. It's like, hey, well, I've got two ears, I've got two eyes, I'll just put in an audio source in the left, audio source in the right, and bang, we're done, we're golden. But that couldn't be further from the truth. Stereo stuff, being able to put monosources inside the world, and then even starting to add reflections and refractions, all really fell short of what I needed to complete the experience my eyes were already having. And so it wasn't until I started playing with the sub-pack that I really started to unlock what's going on in the audio world. Now I'm not an audio engineer, I'm not an audio designer, but what I was experiencing, it has a range of I think like from 5 to 100 hertz, so it's real low stuff. It's a backpack that you wear that's with subwoofers that essentially just vibrate and give your back a tactile impression of the audio that's coming in. And so any music track that's mixed for like a theater release, like a lot of my playlists are, you can feel that sensation in the back. That started to bring the feeling of actually being there. So from that I started to realize that, well, when I'm speaking to someone, like, you know, we're speaking here, it's like when I'm speaking, my voice is not only going into your ears, right, it's also hitting your face. Those air molecules are vibrating against your face. You're getting, as I shift my weight, you're feeling the shifting through your feet as the floor slightly moves. If we have a big roller cart going down the hallway here at the convention, we actually feel it in our bones. And so sound is a much more primitive sense than the eyes. And it's a full body sensation, as well as what you might hear in the ears. And so if you are not getting that, it's like not having stereo. If your body isn't responding in that way. So with enhanced haptic audio, with tactile audio on your body, you actually can feel like a timpani drum. You can actually feel the shape of it. It's like going to a good rave or something like that where they've got the stereo mixed really well. And so you get that enhanced sense of presence and enhanced sense of being there. So I've got a list of about 6 to 10 different experiences you can create with low frequency audio. And then maybe if you don't even have the pack, you can still be designing the lower register of what you can hear to enhance those feelings. So you can get those feelings of a large... Basically, low audio tends to be towards the ground. Low audio also tends to be like large creatures or large events. So you can use that either through the music design or through the sound effect design to enhance those events that happen in your game. So in our game, it's about a magician who's been a charlatan all his life, and then one day his magic actually works. Abracadabra, the rabbit really disappears. As the player, you make magic happen. And so what we want to be sure of as the design is to really design that low audio so that you feel like that magic is happening. You can see in believing and you can hear stuff happening too, but that low audio has got such a powerful emotional impact on it. That was one of the most surprising things. We'll be talking about a couple of others as well.
[00:06:22.662] Kent Bye: Yeah, I know that I've tried the SubPAC back at SVVRCon, and I just remember I put this thing on my chest. I didn't even have a VR headset on. And put on the headphones, I literally felt like I was dancing in a dance club. Like, I felt like standing right next to the subwoofer. And I do think that it's one of the most compelling VR peripherals that I've seen because it has such a primal, visceral, haptic feedback with the sound. It does make it a lot more immersive as experiences, but it sounds like you're really talking about is adding more senses and being able to design for multi-sensory experiences both from the sound and using that for haptic feedback. But also, you know, in your blurb, you said that you had kind of identified 36 different almost design principles. for VR gameplay. And when you look at those 36, how would you kind of break those down into some sort of categories or taxonomy in terms of clustering those different types of gameplay experiences?
[00:07:19.675] Nicole Lazarro: Awesome. Yeah, well to go through all 36 I think is going to be beyond the scope of what we can discuss today. But what we did is we've looked at for over a year, we've been measuring people's faces while they played VR experiences and we've been interviewing them during and after these games. And what we found is that these are 36 best practices, you know, design benchmarks that really work well to help teams create a much more compelling VR experience. I think another one in addition to audio design, that's one of three for audio that I talk about, another one is about just emotion. So that category of emotion, emotion is one of those other benchmarks. And in VR, VR really intensifies, especially now or in the early days, it really intensifies emotion, magnifies emotion. It can magnify, you know, really excitement, it can magnify, you know, fear, it can magnify, you know, all kinds of responses. But just because we can, as designers, turn it up to 11 doesn't mean we necessarily should. And so, if you think about what Hitchcock is known for, he's not known for, you know, jump scares and roller coasters, right? He's not the master of fear, he's that master of suspense. And so the emotion architecture of VR becomes a lot more apparent and a lot more powerful and a lot more detrimental to the success of that experience than it does in traditional games. So you imagine if you trigger fear, you can trigger real fears. If you take it up too far, you can actually trigger a real fear of spiders or heights or, you know, boiling lava or, you know, big creatures. and disgust, you know, when you're in your first-person shooter and you earn some points and you've got some bodies there, some dead bodies there. In a game, that's kind of like, you know, it's not a real dead body, right? But in VR, it's like, if VR is doing its job, it's like, ooh, that's a real dead body. And you might then start to feel disgust. And that's really detrimental to the VR experience because what we're skating the line on is this idea of a sim sickness. You don't want to get motion sick and stuff like that in VR. The emotion of motion sickness is disgust. If you trigger disgust, you know, like blood, feces, stuff that, you know, that you want to throw out of your body, if you trigger that in the player, then any motion sickness they have is just going to be amplified as well. So some of the core things that a lot of traditional game mechanics depend on for creating excitement really get magnified and turned around and twisted against the gameplay in virtual reality. So it's going to be really interesting to take our hardcore gamers and, you know, move them into these experiences. that are quite a bit different than maybe what they signed up for.
[00:09:41.052] Kent Bye: Yeah, and I know that Oculus has released their best practices guide, which is kind of universally pointed to as some of the basic technical things that you have to do in terms of like meeting your frame rate, don't take control over the camera, kind of the more lower level aspects of the technology for creating a sense of presence. But it sounds like what you're doing is also you're kind of creating a set of best practices, a level up in terms of the actual design interaction user experience within a VR game that's going to perhaps make it enjoyable or fun.
[00:10:10.943] Nicole Lazarro: Yeah, exactly. Yeah, what we're doing is this higher level, exactly, design parameters, design techniques, design strategies. And what that allows you to do is because, going back to design leadership, is that we have to innovate. We have to take it to that next generation. We have to take it to new kinds of gameplay. So if you have a set, a good solid set of best practices and design strategies, stuff to think about, you can innovate new ideas, match them against those mechanics, and think about, and it'll give you new ideas in addition. Much easier to innovate than if you are just using the, you're holding on to last generation gameplay because you know it was fun on the PS4. And so you think it's going to be fun, you know, in Sony VR, but who knows, you know. So another one is like looking at controls. So the idea behind controls is you want controls so natural to VR that you can play blindfolded. because you are. You can't see your hands, so putting a traditional game controller in your hands with its 13-button monkey, you know, shenanigans going on, it's about as much fun as putting a transparent D-pad on an iPhone in VR. So the dual-fisted controllers, you know, much better. I really like the stuff from Tactical Haptics. I really like their stuff. Because of the shearing motion you feel on the pads and added that to the gyroscopes, it really feels like you're doing stuff in VR. And that gets your body, you know, really more into that experience. And it's surprising, like, how little you have to do right now. It's really just 8-bit tactic stuff. It's just really big, really 8-bit haptic, you know, being able to use touch in just a very low, low, low, low, super low resolution. It doesn't have to feel like a porcelain teacup. Just the fact that I thump against something whenever it looks like I'm hitting the teacup, that's enough to kind of get me in. And so just these little tricks, finding little tricks to give people some kind of haptic experience will really, really draw people in. On the visual side, you know, you have in the Oculus Best Practices is an awesome starting point. There are a couple things that we feel that are going to get pushed a little bit off that, that need to be expanded on or added to. But one of the things that we're looking at is like the use of, like how do you handle motion? So the idea of using frames just works really well. So we see that both in like the Sports Challenge on Oculus and on EVE Valkyrie, you know, that you see these frames going on. As long as there's a frame, which is why motion inside a vehicle works a little bit better, because we don't, you know, our ears are pretty used to being inside a vehicle, and so we're used to it being kind of off, no problem. Another great design technique is for controls and cameras, just that slow motion. No acceleration, just steady motion through the environment. So using that can increase comfort a lot as well. But I really think that we'll be able to solve a lot of these problems. I mean, everybody's had a sense of real-world sim sickness. which is anytime you're like at the shopping mall and that escalator is out of service, you get that moment where you're actually having to climb an escalator as stairs that are fixed. Your body falls forward a little bit and you feel like a little woozy, just a little bit, but you recover, right? And you're able to use that as stairs. Likewise, when you're on a normal escalator and it's moving up, you have no problem using that escalator because you know exactly how your body's supposed to feel. If you jump into bullet time in VR, you have no idea how that's supposed to feel. So your body goes with it a lot, a lot easier. So I think that's very helpful as well. So if you go into bullet time, you don't know what that's like. You know, going to space, you don't know what that's like. So those are pretty comfortable. But I think if just circling again back to audio, is that, and well, more haptic feedback, is that the body is really good at searching out pattern in noise. And so just having a little bit of vibration in the body during motion experiences gives your body an excuse for what's going on, even if it's just pure noise. And so I think that's a really interesting idea to kind of play with as well. It's that your body is going to interpret, well, what is that, you know, that, whatever that buzz is, is the locomotion that I move forward on that goat cart as I, you know, you know, move towards the castle or whatever that happens to be.
[00:13:59.925] Kent Bye: And finally, since the last time I talked to you, you were at Oculus Connect 2, and you've had a chance to try a lot of demos there, also at CVR, also here at the VR World Expo, and potentially other demos. But I'm just curious if there's been other games that you've seen recently that you feel like are really nailed something innovative in terms of VR gameplay.
[00:14:19.765] Nicole Lazarro: Well, that's a really great, you brought that up. I really love that because I've been really impressed with both Tilt Brush and Medium. So Tilt Brush on the Vive and Medium on Oculus. And, you know, I've been very impressed and people are like, oh my gosh, it's so great. And really, it's like finger painting. The emotions that I'm seeing on their faces as they're describing, it's like, it's novelty, it's fun, you know, because I can just do that for 10 minutes or whatever. And again and again and again. But at some point, you want to have a little bit more than just that, unless you're a professional sculptor, unless you're really trying to make a piece of art. it's still much more like just, you know, doodling. It's fun to doodle. So taking that one step further, I think one game that's really nailed that, or taking it one step further, is I just saw Fantastic Contraption at Day of the Devs this Saturday in San Francisco, and oh my goodness, that looks really, really, really fun. And so it's got a little bit of that, you know, 3D and I can make stuff and move. I mean, I, of course, you know, drew myself a house or, you know, a box to hide in and squiggly sort of things. But in Fantastic Contraption, you can build, like, stuff all the way around you with gears and, you know, or rubber bands and that kind of thing. That's going to be fun. That'll get us outside of that 10 to 15 minute experience that a tilt brush or a medium can kind of support right now. I mean, unless you're, of course, an artist, then that works super well. And why is that important, you might ask? In order for VR to not be a novelty, for VR to not be a plastic guitar. Not that I, I mean they're great plastic guitar games, but in order for VR to really be a complete platform with a whole variety of experiences, we have to get beyond what we call a novelty window, which is the 15 to 20 minute window of what people will do just because. Like I can give you just about any game and you'll play it for about 15 to 20 minutes, just because, but you won't return to it necessarily if it doesn't have what we would call one of the other keys to fun, a little bit of hard fun challenge or some social people fun, something like that, or some kind of really compelling progression system for the serious fun. So we've got to have these things. So I think that Fantastic Contraption looked really good at jumping out of the micro loop of gameplay into some more interesting stuff for the players to do. And for stuff in VR, it's new, right? And that's like, wow, it's like when the iPhone first came out. Anything on the iPhone looked great. But for us to really come back and play it more than once, then you need to have some experience it builds over time. And that's where these 36 design strategies come in for VR and the four keys to fun. That's also still very core to how these systems create engagement.
[00:16:48.108] Kent Bye: OK, great.
[00:16:48.688] Nicole Lazarro: Well, thank you. Thank you.
[00:16:52.009] Kent Bye: And thank you for listening! If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.
