Noah Falstein started as a game design pioneer back in 1980, and lately he’s been collaborating a lot with virtual reality companies like MindMaze, Akili Interactive Labs, StoryUp, and AppliedVR on different immersive medical applications and serious games. He sees that immersive technologies in the medical field are allowing him to collaborate with cutting-edge neuroscientists and researchers, he’s able to create some real pro-social benefit to help people, and it also challenges his skills as a game designer in having to create new embodied gameplay that needs to show empirical results with peer review studies. He’s been helping to create gameplay mechanics for neurorehabilitation for stroke victims, and he collaborated with Adam Gazzaley on Neuroracer, which was featured on the cover of Nature magazine.
I had a chance to catch up with the veteran game designer Falstein at the Canadian Institute for Advanced Research workshop on the Future of Neuroscience and VR in May 2019 where we talked about game design for medical applications, his evolutionary biological approach to game design, how he thinks of mental challenges, social challenges, & physical challenges, and why he created a new special interest group for health and medical games for the International Game Developers Association.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So continuing on in my series of looking at the future of neuroscience in VR, today I talk to Noah Falstein, who is a veteran game developer. He's been designing video games since 1980. So for 39 years of designing and developing video games, spent some time as the chief game designer at Google, and now has since gone off and he's looking at medical applications for different games, especially looking at virtual and immersive technologies and how to use all the biometric data that's coming from your body and how to turn that either into neuro rehabilitation or to find ways to gamify aspects of training or health and wellness. So Noah has been collaborating with a number of different neuroscientists, and I'm starting to look at these different principles of how to do different embodied gameplay and use the principles of game design in order to do medical applications in healing. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Noah happened Thursday, May 23rd, 2019 at the Canadian Institute for Advanced Research, Future of Neuroscience and VR Workshop in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:30.211] Noah Falstein: I'm Noah Falstein. I'm a game designer. Been in the industry since 1980 and worked in a whole range of places. But in the last few years in particular, I've been focusing more and more on the overlap of games and health care and many different aspects.
[00:01:46.659] Kent Bye: Yeah. And so maybe you could talk a bit about like what types of experiences are you working on or what is this cross section between health and gaming?
[00:01:54.955] Noah Falstein: Well, I think it's something that's been in the works for decades, but has just accelerated immensely in just the last three or four years. And just to give you an example, I'm working with half a dozen different companies at this point. The biggest involvement I have is with a company called MindMaze. That's a Swiss company that has raised a lot of money on a larger than billion dollar valuation. And they're working initially in stroke recovery through games, both acute and chronic. So they get people in the hospitals and give them special games they play that can help work with their brain plasticity that's much higher in those first few hours or days after a stroke. And then they have a version that is more for clinics that allows them to work for weeks or months afterwards. A lot of recovering from a stroke, just as one specific example, involves repetitive movements and some very boring long-term exercises. And if instead of just raising your arm slightly higher each time, you're controlling an airplane or climbing a wall or attacking a castle, they have all sorts of different gameplay things that each exercise different muscles, different types of cognitive abilities. And just one of the more positive ways that we're helping the world through gameplay.
[00:03:16.429] Kent Bye: Yeah, this concept of neurorehabilitation that you're able to do these actions that normally are really boring and so people don't do them and so then they don't heal and so it sounds like you're in the process of taking all of your game design experience and trying to take this process as very boring and mundane but make it fun. So how do you make something that's very repetitive and boring fun?
[00:03:37.845] Noah Falstein: Well, you know, I find that each case is unique, but a lot of the techniques are to work with the doctors and clinicians and find out what types of rehabilitation exercises have been successful before and what their biggest problems have been in terms of people being able to do them long enough or well enough or sometimes even restrain themselves. You don't want them to overexercise too much. And then I look for good fits with either classic gameplay or look for ways to apply new types of gameplay to make that all work. And, I mean, it goes well beyond that, though, that there also are games being used now as direct treatment, so it's not just to encourage people to do physical therapy, but playing the game itself is a direct treatment that can be even on the same level as a pharmaceutical in terms of its impact and health improvement.
[00:04:26.153] Kent Bye: Yeah, I was just at the Awakened Futures Summit by Consciousness Hacking and Adam Ghazali was there and he was talking about Akili and this sort of immersion vessel that he announced there in terms of being able to have these completely immersive environments and just this concept of being able to do like real-time biofeedback and what's happening in your body but to start to put that into the gameplay. So I'm wondering if you started to look at that in terms of what can you do with heart rate variability, or galvanic skin response, or EEG, or these kind of more passive emitting things that are happening in our body, but using real-time feedback of a game experience to be able to have this immersive experience of what's happening inside of your body.
[00:05:05.060] Noah Falstein: Yeah, well I actually had the privilege of working with Adam on NeuroRacer about 10-11 years ago when he was just starting to get interested in the idea of applying games to some of the brain imaging work he did. And through a happy coincidence, a lot of the people he was working with were ex-LucasArts people who all knew me one way or another and they were aware that I was working in health and games at that point. And that particular case in point is an interesting example of what you describe in that NeuroRacer was what eventually became essentially the prototype to Akili and it was a game that was meant to help aging adults learn to multitask while they're driving so they can see signs and not run off the road. And it was quite successful. They actually did a big double-blind study and had that published in the journal Nature. The only time that scientific journals had a video game on the cover. But in the process of developing it, there was a really fun give-and-take between Adam and his lab assistants and colleagues and the game developers. He would say, well, we want to make it a little more exciting, but our budget's very low. And we said, well, let's add in motor sounds and all sorts of sound effects. It's one of the cheap things you can do in games. And he said, oh, you know, that would be a real problem because the EEG that we're trying to do, if you engage the audio part of the brain, a whole new area is going to light up and we'd rather keep it silent so that we can isolate this. So we're thinking, well, what else can we do? And I said, how about some flashing lights to signal things? And he almost turned pale and he said, wow, flashing lights are probably the worst thing for sending spikes through an EEG that will totally mess up all of the other data we have. And that kind of give and take was fascinating to me because I love the idea of, as you suggest, applying what I've learned in video games, but learning where the feedback and the technology of modern neuroscience has its strengths and its limitations and working within that. And, you know, the short story of that is that we then look for what we can do and what kind of advantages we can get from having the information about how someone's brain is changing and growing and use that to make the games more effective in whatever particular health aspect that we're aiming at.
[00:07:19.527] Kent Bye: What is it for you that you find exciting about making these games for health? And what is it that draws you to, rather than going off and creating a big entertainment game, what is it about this connection to the health that you're interested in?
[00:07:31.056] Noah Falstein: Well, I think about that a lot and there are three simple things there. One of them is just the fun level of learning something new and I've always been a science nerd and fascinated with how the brain works. Almost all of the game designers I know are really interested in brains and neuroscience because that's at the heart of our work. Another one is just the chance to do something that's so unequivocally good for people. And, you know, beyond treatment, we're training doctors, we're giving caregivers a chance to see what some of the illnesses they're treating feel like through virtual reality. Just a huge range of things that are helping people in really basic ways. And then the final thing is what you've alluded to is that this is something that takes all of the experience I've had making entertainment games and applies it on a whole new level with many new constraints and objectives to actually help people in an effective way and to be able to measure that in both a scientifically valid and cost effective way. And those challenges as a designer I just find fascinating and it kind of is the advanced level of being a game designer for me. I do expect as more people reach larger, you know, I've been a game designer for almost 40 years and you start to get bored with making the same types of games over and over again and bringing in the health aspect makes it that much more challenging and interesting.
[00:08:52.940] Kent Bye: I kind of see the game designers as these digital architects, just in the sense that they have to pull in all these different disciplines and domains and be able to modulate human consciousness through an experience. And I know that within game design, it's a bit of an open question as to what is the model and theory of game design and experiential design. I know Robin Haneke has her own approach. There's different game designers. But I'm just curious for you, how do you think about these equivalence classes of these different types of trade-offs that you have in building a video game, whether it's from agency versus story, or making choices, taking action, a sense of embodiment? How do you think about, as you're designing a game, what the framework that you're using to understand the theory of experience?
[00:09:34.149] Noah Falstein: Great question. For one thing I want to say I often explain to people who don't know much about game development that being a game designer is a lot like being an architect. It's one of the few disciplines where you have an equal balance between the creative aesthetic side of things and the science and engineering side of things and have to somehow be the person setting the plans forth so that all those people can work together. And that said, one of the things I love about this world is that a lot of us, I'd say virtually all of us who've been doing it for a while, get to know each other, and Robin and I are friends. Jesse Schell, who's written what I think is the best book on game design, his Art of Game Design book, I talk to him about this sort of thing. And I would say if I have a philosophy or an approach, it's often evolutionary based. I like to think about how we as humans have evolved, and it's part of what brought me to neuroscience. If you look, for example, at the most popular games, and for that matter, popular entertainment of TV, movies, books, a lot of it has to do with practicing survival skills and it's why games and movies about life and death and you know whether it's a violent thing like a military game or a television show about doctors, we're really drawn to things that set very high stakes and I think that we can make really interesting games that play on that biological need to understand what risks we're taking and use games as a safe way to experiment and learn how to be better at those risks and not necessarily have it be all about violence either because that's only one of many aspects of survival.
[00:11:17.215] Kent Bye: You know, I tend to think of it as making choices and taking action. So you're trying to make some sort of mental model, a prediction of like a model of the world, and you have this mental concept, but then you have to make a choice, but then you have to take action. So you're having all these different behaviors, whether it's the abstractions of the buttons on a video game controller, or actually moving your body within an immersive experience. But then there's a sense of embodiment and all the sensory input that you're getting from just receiving all of the different sensory input that's coming out of the game. But within the VR experience, you have all these other dimensions of haptics and sound and all the different stuff that's actually being processed through your perceptual input. And then you have the emotional engagement, which I think is the music and the sound design and the different ways of building and releasing tension and consonance and dissonance cycles. But I see that there's this existential tension between that narrative and having control of that time-based medium of that drama and that story versus then all of a sudden giving people agency to do what they want. You have this, from the VR industry, you have people that have a center of gravity from the game industry and they're trying to add different elements of the story, but then there's the people coming from more of the film side and they're sort of adding more and more elements of agency. So I see like there's these different trajectories of people where we're trying to create this holistic way of looking at both the mental and social presence, and active presence, embodied presence, and emotional presence.
[00:12:33.668] Noah Falstein: No, it's absolutely true. I mean, you can really, you know, sort of slice the pie many different ways. I tend to break it down somewhat differently, but, you know, I think it's an equally valid way to look at the components that you described. I tend to look at a model of mental challenges, physical challenges, and social challenges. And if you look at a combination of those, you know, pretty much any popular game, or for that matter most popular entertainment forms, focus on at least one and sometimes all three of those simultaneously. When I look at a new technology, and in my career I've had to deal with all sorts of new ones coming in, But with VR or AR, I found that they are great affordances for certain types of experiences. Certainly the mental possibilities are amazing, as we've seen. VR has some really great physical ones. You know, I think Beat Saber is really intriguing because of the way that it may bring back exercise gaming. And not through simulating something real, but simulating something that's kind of uniquely game-like and science fictional. and the social aspect of course is pretty fascinating as well and VR is just gradually moving into that but there are companies like Science Space and High Fidelity that are experimenting with how to do large-scale social VR and AR as a social medium is also pretty fascinating because I love the idea of having an AR headset on and having some people in the room who are physically there with you and other people who appear to be physically with you but are in some other part of the world. So it's really a very exciting time now where sort of all those different dimensions are being enabled and extended by the new technologies that we're seeing here.
[00:14:20.168] Kent Bye: So for you, what are some of the either biggest open questions you're trying to answer or open problems you're trying to solve?
[00:14:28.136] Noah Falstein: Boy, let's see. You know, I'm very open to finding new challenges and I would say that when I think about that, I don't have a particular agenda of my own that way. I mean, the closest I would come is that I'm very interested in interactive narrative. and ways that we can blend the essentially linear experience of traditional storytelling with the interactivity of games, and it goes back to my early LucasArts days and working on graphic adventures, but I really think we're still learning a huge amount about how to tell interactive stories or what an interactive narrative really is. But part of what has drawn me to games and health in particular is just the fact that there are so many unsolved problems, so many opportunities and low-hanging fruit for doing everything from training physicians to helping people directly to diagnosing degenerative diseases before people are even aware that there's a problem. It's just a wide open sort of Wild West feeling. And it reminds me a lot of my early 1980s days. So in some ways I just love the freedom and shared research and cooperation that I see in this field. And that's really more than any particular agenda of a problem to solve. That's really the community sense that I love about this.
[00:15:50.735] Kent Bye: Great. And finally, what do you think the ultimate potential of all these immersive technologies are, and what they might be able to enable?
[00:16:00.126] Noah Falstein: Wow. Well, lots of different directions, lots of different potentials, certainly. I'm a lifetime science fiction fan, and it's interesting for me to see compared to when I was a kid and seeing the movie 2001 I was really disappointed that we didn't have you know orbital Hilton's by that time and I you know even now it's not clear to me whether I'll ever be able to go up into orbit or for that matter even into space but on the other hand in 2001 when they went to play a computer game with Hal they played a really ugly 2D chess game and even by 2001 we had much better than that so I'm encouraged that a lot of the virtual worlds we're exploring, if we can't actually reach some of these science fictional areas, we don't have flying cars right now, we can certainly at least use these simulations to experience them in a way that will make them real. So it's kind of a nice shortcut that gives me a chance to try some of those things out.
[00:16:56.790] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:17:02.238] Noah Falstein: Well, I guess it would just be a plea to my fellow game developers to sort of jump on in the water's fine that I think that health and medicine and neuroscience are particularly interesting, ripe, exciting areas, lucrative areas for game developers and I only wish that more of my fellow game developers were as excited as I am because I think there's a lot of gratifying and useful things that they could get from that area.
[00:17:31.037] Kent Bye: You mentioned something was announced at GDC along those lines. What was that?
[00:17:33.998] Noah Falstein: Yeah, so I'm one of the steering committee of the IGDA, the International Game Developers Association, has a special interest group on health and games, and we're hoping to use that as a clearinghouse to help connect people from the health and medicine and neuroscience communities with game developers of all stripes, VR but also traditional flat-screen gaming. And it's in its early stages, but thanks for remembering that, because I love to plug people to take a look for it. It'll be showing up on the IGDA.org website shortly.
[00:18:08.049] Kent Bye: Awesome, great. Well, thank you so much.
[00:18:09.589] Noah Falstein: You're very welcome. Thank you.
[00:18:11.289] Kent Bye: So that was Noah Falstein. He's been a game designer since 1980, and he's currently looking at the overlap between games and healthcare. So I have a number of different takeaways about this interview is that first of all, well, Noah's been involved with lots of games for nearly 40 years now, but there are certain tropes of classic gameplay of game design. And he sees that virtual reality as applied to doing specific tasks for either medical healing, neuro rehabilitation training. These are all for him a very exciting because it's like requiring him to really up his game as a game designer. And he said there's like three things that are really motivating him right now. For one, he gets to learn something new. He gets to collaborate with all these scientists who are studying the brain and looking at very specific things about health and healing. And so he gets to learn quite a lot by collaborating with these different doctors and neuroscientists. And then also it's just, it's good for people. You're creating something that's actually helped people either recover from strokes or neurodegenerative diseases, or to be able to help train doctors and build empathy. As well as number three, he's able to take all those different experiences of game design and be able to apply it on a whole new level and be able to actually produce experiences that are then backed by scientific research that then are able to verify these different levels of game design that he's been cultivating as a craft for the last 39 years. So he also looks at game design as very similar to architecture. And, you know, my previous whole series was on talking to different architects. And so you really do have this blending of art and science and really have to take this interdisciplinary approach and taking all those different aspects of human cognition and perception and anthropology, as well as with all the different technical aspects of what it actually takes to pull off an actual game where you're working with these different technologies like unity and virtual reality and all these biometric sensors. He said that he's been collaborating with MindMaze, and I have an interview with one of the founders of MindMaze here coming up. But MindMaze is essentially this VR headset that has an EEG built in, and I think it's really targeting these different medical applications so they can start to do these therapies that are people who are just having a stroke. He said that there is a window where if you're able to play with the different plasticity of the brain very early on, then you can actually do a lot of good very early, just after a stroke. And so just seeing how these different virtual reality technologies with a company like MindMaze can start to be applied to the medical field. Just in talking generally about game design, he looks at from a very evolutionary biological approach of seeing what are the things that we actually are interested in, in terms of what are going to help us to survive, you know, drawn to different things that are very high stakes, and how we can start to simulate that within these games. But there's certain biological needs that we have and these different risks that we're taking. And so he's tends to see his game design through that evolutionary biological lens, which I think is very fascinating and interesting, but it kind of, uh, boil it down to these three major different challenges, the mental challenges, social challenges, and physical challenges. For me, I kind of combine into different mental and social dimensions of presence as well as embodied presence. And for him, he says, mental challenge and social challenge and physical challenge. And he also mentioned later that he's interested in also trying to combine different aspects of interactive gameplay with traditional linear storytelling. And so for me, I think that's that tension between that emotional presence and different aspects of agency and being able to actually express your will within an experience. And so there's fundamental tensions that he's also trying to figure out as well. And that for him, looking at traditional sci-fi, he had this vision that we'd have flying cars, but that maybe with virtual reality technologies, he could look at something like Kubrick's 2001 and see that by the time 2001 actually came around, gaming were actually way more advanced than what Kubrick had imagined back in the 60s. And so he was thinking about, well, if we can't actually have physical flying cars, then what would it be like to actually be in a VR experience and start to simulate a lot of these things that we had dreams of? and he mentioned briefly at the end that he's a part of the steering community for the IGDA and there's a special interest group on health and games that he's helping lead up and he's hoping that there are going to be a lot more game designers who get more and more interested in this cross-section between game design and health and healing because there's certainly a lot of applications and then from his perspective it's really making him really up his game and really looking at new types of embodied gameplay, especially of these different rehabilitation exercises, which are often very boring, and then trying to add different levels of gameplay and narrative and story on top of that, so that it just gives people more of a reason to be able to do tasks that are otherwise very mundane and boring, and be able to figure out how to translate these different movements that need to be done incrementally, and to be able to measure that and feed that into these game design and these different gameplay mechanics. in order to actually create experiences that help people on their healing path or to be able to rehabilitate after an injury or to do some sort of neuro rehabilitation after a stroke. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then there's a couple of things you can do. First of all, just spread the word, tell your friends, this is a podcast that really relies upon word of mouth. and the more that you can share these podcasts with people that you know, the more that the podcast continues to grow. And also to help continue to bring this podcast to you and to the rest of the community for free, I do rely upon donations from my listeners. And so if you'd like to become a supporting member to help support and sustain this podcast, to be able to allow me to continue to survive as an independent journalist, to continue to document the evolution of these spatial computing platforms, then please do consider becoming a member of the Patreon. Just $5 a month goes a long way to be able to allow me to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
