Aldis Sipolins says that brain training is broken, and that VR can help to fix it. He suspects that how our brains work while being immersed within virtual environments more closely resembles how they work within real life. But at this point, it’s really difficult to prove that doing brain training tasks within a 2D context would “transfer” to improving overall cognitive skills. Aldis hopes to change that with his VR brain training game called Cerevrum, which has a tagline of “Not training. Learning.”
Aldis is finishing his Ph.D. in Cognitive Neuroscience at the University of Illinois Urbana-Champaign, and he’s been researching videogame-based brain training to enhance cognition. He was giving demos of his Cerevrum game at SVVRCon, and he hopes to eventually be able to scientifically show that doing these types of brain training exercises within VR will have benefits that are transferrable to our every day lives.
Aldis was hesitant to hype up any capabilities of Cerevrum because it’s at this point largely unproven. He’d actually prefer to not refer to it as a brain training application, but rather a game that will be able to captivate other hardcore gamers like himself. If it’s not fun to play, then it’s ultimately not going to succeed within the initial audience of gamers. He identifies as a hardcore gamer himself, and so he wants to create a game that’s both cognitively challenging and fun.
He says that given the choice to do something that we’re good at versus something we’re bad at, then we’ll usually choose to do what we’re good at. By using advanced machine learning on the backend of Cerevrum, he hopes that the game will be able to detect the area where we’re weak and then help us improve on it. Eventually we’ll be able to quantify our abilities in these different cognitive areas and be able to compare yourself with your friends.
To me what Aldis is doing with Cerevrum is one of the most exciting possibilities of the potential of virtual reality. He says that our brains display the most neuroplasticitiy while we’re in a flow state, and so being completely immersed & engaged within a game within a virtual environment might have the capability to rewire and expand the capacity of our brains in a way that transfers into our everyday lives. The potential cognitive improvements will be different for every person, and he’s looking forward to continuing to develop the game and do the research necessary to scientifically validate it’s effectiveness.
Here’s a video trailer from the Cerevrum site:
Aldis said that Palmer Luckey tried out the game and made it to Wave #14 in the game, and that he really enjoyed playing it.
We're happy to see @PalmerLuckey and other guys from @oculus at @SVVR2015. They checked out @CEREVRUMGAME #indiedev pic.twitter.com/i8QQ6EeQ6e
— VRARlab (@VRARlab) May 20, 2015
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.958] Aldis Sipolins: My name is Aldous Sipilans. I'm currently finishing up a degree in cognitive neuroscience. I've been studying brain training for the past four years, and it's just broken right now. It just doesn't work. And I'm convinced that what's missing is VR, a feeling of presence and immersion that you get in VR. It makes you engage the same cognitive skills that you use in everyday life. And that link to the real world is what's missing from brain training right now.
[00:00:30.950] Kent Bye: Great. So what type of insights from neuroscience are you applying into cognitive training?
[00:00:36.274] Aldis Sipolins: So we've been studying brain training in all sorts of different iterations. We're working on this huge study where we made a brain training game on Android. And it's all right. People put up with it. But it doesn't lead to real world benefits. It doesn't make you better at skills you use in the real world. And that's because looking at a tiny smartphone screen and hitting buttons only goes so far.
[00:00:53.386] Kent Bye: Right. So what type of experiences do you have in VR that makes it a lot more immersive then?
[00:00:58.155] Aldis Sipolins: So in VR, right now we have a playable demo that puts you in a spaceship destroying enemies, and every weapon on your ship is a different cognitive skill. And so by integrating these cognitive demands into the game makes it a lot more engaging, and it puts players in this flow state. Flow is kind of this catchphrase, but it actually has been scientifically shown to promote brain plasticity.
[00:01:14.924] Kent Bye: Right, so talk a bit about the mechanics of what type of things that are happening in the VR experience, and what type of skills that you're trying to train in that way.
[00:01:22.053] Aldis Sipolins: Right, right. So we're taking cognitive demands from brain training games, but rather than have them be isolated minigames that you play on their own, we integrate them into an overall game structure. We're going to have an engaging story with kind of malicious VR narrator robot, and we're going to have unlockable weapons and environments and social media integration so you can compare your brain to your friends to see what they're bad at.
[00:01:42.188] Kent Bye: Right, so I guess this is really interesting in terms of, like, we have a working memory, usually you can remember, like, seven chunks of information, you know, and what's exciting to me is the possibility that something like VR could actually help support expanding our capacity of our brain, our working memory. So, what do you see as sort of the endgame of this, of what might be possible through training and doing repetitions like this, what a human's able to do and expand their mind?
[00:02:06.731] Aldis Sipolins: So the endgame is to lead to noticeable, demonstrable benefits in cognition. That has to start with research showing the benefits. And we're actively planning research collaborations centered around an open database that we're going to give the software away for free to research labs in exchange for anonymized data from them. So we're going to upload this data so anyone, any researcher or any person can access it, analyze it, and publish our data. We're putting our money where our mouth is. And so if it works, those papers are going to get published. If it doesn't, those are going to get published.
[00:02:34.282] Kent Bye: I see. So for your own personal experience of playing with this game, what kind of before and after have you noticed? Maybe you could describe, like, this is what I was able to do before, and then after playing this game, this is what I can do now.
[00:02:46.451] Aldis Sipolins: That's not exactly how it works. As a scientist, I'm not going to hype this to doing anything. I didn't feel different before and after. But the whole premise of brain training is that if you do this regularly and it becomes a part of your life, that it hopefully will improve your day-to-day cognition. And that's going to be up to the research labs to show that it does, and I'm confident that it will. Well, how do you measure that then? So the traditional way you measure that is you test people on a bunch of cognitive abilities, like memory, attention, multitasking. Then you have them play this game for a period of, I don't know, days to weeks. And then you test them again to see how much they improve. So the term for it is transfer. If you do anything over and over again, you get better at it. But that's just practice. That's training. What you really want is to improve on other things. And so you want the game to transfer to benefits in real life. And transfer is kind of the holy grail of cognitive training. And it's elusive. And very few people have shown it. And I'm convinced that the reason they haven't is because so far it's just been playing brain mini games on a browser window or a smartphone. In VR, it's a totally different experience.
[00:03:41.647] Kent Bye: So are there numbers that are associated with this? Or how do you even begin to quantify this?
[00:03:46.368] Aldis Sipolins: So the way we quantify it is we have a robust machine learning back end. One of our co-founders is a PhD in computational neuroscience specializing in machine learning. And so what machine learning lets you do is we just assemble this huge mountain of player data, and we let this machine learning intelligence figure out how it structures itself, what connects with what. And so we're not coming at it with any a priori hypotheses of we think this is a memory game, or we think this is an intention game. The data will tell us what it's measuring. And so with this kind of machine learning back end, we can know what's connected to what. And when you improve on one thing, if that leads to improvements in other things, that's how we know a transfer is happening.
[00:04:19.624] Kent Bye: Now, you have different levels of increasing difficulty. You talk about waves. How do you describe the difficulty of each wave?
[00:04:29.633] Aldis Sipolins: So difficulty is one of the things we're doing kind of different. So traditional adaptive difficulty, it assumes that everyone finds the same things difficult and this just flat out isn't true. And so rather than just having like a traditional step iteration of difficulty, every single minigame we have has several different tunable difficulty parameters. Like we have a missile game where we can change the number of missiles that you have to keep track of, the speed at which they fire, the radius in which they fire. And every one of these parameters taps into a different cognitive skill. And so machine learning will tell us what these different things are measuring. And also, for each player, it's personalized what they're good and bad at. So maybe increasing the number of missiles is really hard for one person, but making them fly a lot faster is super easy for them. So we'll be able to, using machine learning, to kind of play with these parameters to discreetly force people to get better at the things they're not very good at.
[00:05:13.322] Kent Bye: So you're saying there's kind of an interactive machine learning, which is kind of like very customized to an individual, trying to see where their limitation or threshold is for where they start to break down and then slowly adapt the game? Where's this information going? Is it like interactive and dynamic in that way?
[00:05:29.235] Aldis Sipolins: Yeah, exactly. It's very interactive. So we'll know what you're good and what you're bad at. People have a tendency to only want to do the things they're good at. Given the choice between something you're good and bad at, you're going to pick where you're good at. So the whole point of the machine learning is to have it happen so under the hood that you don't even notice it. And this machine learning is personified in this robotic narrator who's going to guide you through the experience. And the whole premise of the game is your narrator wants to make you smarter. And as you play, you give her data, which improves her ability to make you smarter.
[00:05:54.875] Kent Bye: And so I know that there's a number of different like neuroscience principles like neuroplasticity that I've heard sort of thrown about in terms of like VR being able to kind of rewire your brain. I'm curious from your perspective, how would you describe that of what's possible of how we can kind of like either just change how our brain is wired together, connected, or how neuroplasticity principles play out using VR?
[00:06:17.248] Aldis Sipolins: So, I mean, everyone in the neuroscience field agrees that the brain is plastic, it changes, and it's changing all the time. It changes based on experience. The term in neuroscience is neurons that fire together, wire together. So if you're doing the same task over and over again, those neurons fire together, and so you establish those networks. Now, I'm not going to overhype this at all. I'm not going to say anything about the networks we're changing. Up until a research lab collaborates with us to run a brain imaging study, that's when we can start saying things about how the brain is changing. In the meantime, it's a fun game that I firmly believe will make you smarter.
[00:06:47.348] Kent Bye: Okay, yeah, I just think of, like, James Blaha's Vivid Visions, formerly known as Diplopia, but being able to kind of, like, not be able to see in stereoscopic 3D, but playing kind of a 3D game that is able to allow him to see in 3D for the first time, and something like that, he's sort of building upon neuroplasticity principles, so is it sort of theoretically possible to kind of, like, change the way our brain is wired through just VR input of using HMD?
[00:07:14.940] Aldis Sipolins: Oh, absolutely. And those kinds of experiences, they really speak to the power of VR in shaping the brain. I mean, any experience you have changes your brain. And the more immersed you are, the more kind of different sensory stimulation you're getting, the more your brain is going to change in response. So having fully immersive audio, having fully immersive visual sensation, and even working motion sensing, we're really excited to develop for the Vive. Because as soon as you can see your hands, you get that extra feeling of presence. And being able to interact with objects kind of naturally and intuitively, it minimizes the jump from the game to real life. And I think those kind of experiences are what's going to lead to really noticeable improvements.
[00:07:48.322] Kent Bye: Yeah, and I guess you would sort of put this into the category of a serious game. And maybe, do you consider it a serious game? Or how do you even describe what you're working on?
[00:07:57.428] Aldis Sipolins: So we really don't want this to be kind of written off as a brain training game. The whole goal is we want to make something so fun you forget it's making you smarter. So the early market for VR is going to be early adopters. People who like brain training aren't going to buy a VR headset just to play this game. So we want to make a game so fun that hardcore gamers like me would play without even knowing it's making them smarter. That'll just be kind of a side benefit.
[00:08:18.340] Kent Bye: And so do you find your game fun, or do you feel like you get really immersed and engaged, or do you feel like there's a lot of other things to make it, like, as a hardcore gamer, make it really sort of long-lasting and engaging for you?
[00:08:28.189] Aldis Sipolins: So we have a very early playable demo, and I do find it fun just because it's the adaptive difficulty makes it really kind of challenging. And that's the thing I always really like in games, is I want to be challenged and I want to get this feeling of mastery over something. Thinking about all the cool things we're going to add in the future, my mind just boggles. I can't wait to get achievements, unlockable environments, to customize the game. We're hoping to do co-op at some point. I want to compare my multitasking ability with my friends to show them how stupid they are. Yeah, the mind boggles.
[00:08:52.691] Kent Bye: And so when you talk about comparing it, it goes back to this whole being able to quantify where you're at. So what does that even look like in terms of is it just an abstract bar that is a number from 0 to 100 and you're comparing it? Or how do you start to put numbers on some of these different tasks?
[00:09:09.919] Aldis Sipolins: So the way we do that is we have a home base that we call your nexus. And in your nexus is what we call your cognitive cloud, which is the 3D visualization of your cognitive skills as determined by all those different game parameters. So the machine learning intelligence, this shapes the cognitive cloud. That determines the structure of the cloud, where every different node in the network is, how bright each star is, how connected everything is. So the whole point is when you play a game and come back to your nexus, you can look at your cognitive cloud and a whole new part of it will light up. And as we add new games and new mini games and new weapons, new parts of that cloud will light up. So machine learning will tell us, oh, you've added something totally new that's testing something different. Or if we add something and, no, you've already got something that's measuring that. We won't add that. We'll take it out.
[00:09:49.950] Kent Bye: And you're developing on Gear VR as your primary platform? Or is this going to be on both mobile and DK2, all the different headsets? And what's the platform spread that you're thinking about?
[00:09:59.218] Aldis Sipolins: We're currently developing for Gear VR because I think it makes the most sense right now. It's going to be the first one to market. And I think this is going to be a lot of people's first experience with VR. And we want to be people's first experience with really comfortable VR. We're really looking forward to developing for other platforms like the Oculus Rift and the Vive. We're already planning out brain training games that make use of the Vive's motion controls. There's going to be some really exciting stuff about that coming out.
[00:10:19.973] Kent Bye: Oh, wow. That sounds really exciting. To me, this is one of the most exciting parts about virtual reality is this whole concept that I could actually improve myself by, you know, become a better person or smarter or more adept at just playing these games that, at the same time, be fun, but at the end, come out of it being able to do things in the world that I wasn't able to do before. So yeah, I guess I get caught on this before and after. What is the thing that I'm going to be able to do that I wouldn't be able to do? Is that going to be totally dependent for each person? Or are there examples from literature of after people have played brain training games, they're able to do this or that? Or they haven't been able to show any of that at all because, like you said, they haven't been able to prove transfer?
[00:10:59.591] Aldis Sipolins: Yeah, so there aren't examples like that because current brain training just doesn't really work. It's going to be different for every person, the kind of differences you see. So the whole point of the machine learning is that it adapts to each player. So if you ever, me, I have a terrible memory. I forget, I walk upstairs and I forget what I went up there for. I forget where I put my keys and my cell phone. So for me, the kind of differences I would expect to see are I somehow, I have a easier time remembering those things now. For someone else who has trouble multitasking and doing several things at once, they might notice they struggle a little less with, you know, walking and talking or cooking and listening to music.
[00:11:31.431] Kent Bye: And so what are the things that you're looking, really looking forward to do in virtual reality then?
[00:11:36.250] Aldis Sipolins: Honestly, it's playing with the Vive and getting motion sensing. We have a really cool idea for a game coming out. It's basically Fruit Ninja with cognitive demands, where we're going to 3D print a holder for the Vive motion controller to make it feel like a samurai sword. More to come, but this is the thing I'm most excited about in VR right now.
[00:11:52.933] Kent Bye: Yeah, it feels like that, you know, the intersection between neuroscience and VR could be pretty one-to-one in terms of being able to do neuroscience research within the VR HMD. Do you see that at all in terms of like experiments they could start to do with, like you said, like having different cognitive demand loads and then seeing how people behave by tweaking different variables within a VR environment?
[00:12:15.116] Aldis Sipolins: Oh god, yeah. I've been running a VR research lab at the University of Illinois for the past year, and this is exactly what we're doing. We're taking these conventional psychological tests of memory and attention that are usually done either on paper or pencil or on a monitor, and we're turning them into 3D VR tasks. And it makes them so much more ecologically valid, because we live our lives in 3D, and so it doesn't really make sense to test something on a 2D screen.
[00:12:35.059] Kent Bye: And so have you found that a lot of these studies are just reinforced and even more effective within an immersive environment? Or have you found ones that actually go against some of the previous findings?
[00:12:45.226] Aldis Sipolins: So no one's really done that research yet. My lab is actively pursuing that exact question of directly comparing conventional 2D tests and training and comparing that to these new VR experiences. And the first step in this research is validating VR as a platform. And that takes direct comparison of these things. And this is what we hope to show in research in the future. I see.
[00:13:02.597] Kent Bye: And finally, what do you see as kind of the ultimate potential for virtual reality and what it might be able to enable?
[00:13:07.421] Aldis Sipolins: I mean, we all want the same thing. We all want the matrix, the oasis, the metaverse. That's where I think it's going, and that's what I'm most looking forward to. And I think navigating that kind of complex VR environment, you're going to have to be pretty smart to do it. So hopefully we can help you get there.
[00:13:20.809] Kent Bye: Awesome. Well, thanks so much. Yeah, thank you. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voices of VR.
