Larry Hodges is a professor of human computer interaction at Clemson University, and he was one of the co-chairs of the very first IEEE VR academic in 1999. Hodges also co-founded a start-up named Recover, which originated from a successful research project into stroke recovery done by his student Austen Hayes. Inspired by the latest research into neurorehabilitation & skill relearning, they create a Kinect-based experience that gamifies the rehab exercises that 85% of people don’t do because they’re either too boring or painful. They’re able to inspire stroke recovery patients to do extended rehab practices while also progressively increasing the difficulty of the tasks over time as they slowly get better.
I had a chance to catch up with Larry at the IEEE VR academic conference in March 2016, where we talked about the history of the IEEE VR conference, his work with stroke rehabilitation and virtual therapy along with some of the results that they’re seeing, as well as some of his research on treating PTSD with virtual reality.
LISTEN TO THE VOICES OF VR PODCAST
Stoke victims may lose the full range of movement within their limbs, but Hodge says that this is more of a brain problem than a physical problem. Recovr created a game called “Duck Duck Punch” that amplifies the movement of their arms, and so they’re able to feel like they’re accomplishing something with their limited movements. Rather than being frustrated in not being able to accomplish anything of significance, the amplification of movement within the game restores a feeling of robust agency within the patient and inspires them to do up to 600 repetitions per day. This is the level of extended practice that is required in order to relearn a skill, and to effectively rewire their brain through these many repetitions of progressively harder tasks. Hodge says that they’re able to see actual physical progress as soon after five sessions in some people who have been paralyzed for years. The supervising doctor can change the amplification multiplier as they start to regain more and more of their range of motion and motor control.
Medical applications and virtual reality therapy are one of the clear use cases where the benefits are clear, and the next big challenge for Recovr is convincing insurance companies to pay for this virtual treatment. Hodge says that there are some people who would pay just about anything to get access to this treatment, and that he’s received more feedback from the lives he’s changed than other domains of computer science. Perhaps eventually, for some treatments it’ll be cheaper and more efficient for insurance companies to pay for the installation of virtual reality systems within homes to help usher in the era of distributed telemedicine.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to The Voices of VR Podcast. On today's episode, I have Larry Hodges, who is a professor of human-computer interaction at Clemson University. And Larry is one of the original co-chairs of the IEEE VR Academic Conference, which started back in 1999. And so today Larry talks to me about one of the research projects that got converted into a startup called Recover. And what they're doing is they're gamifying the process of stroke recovery using a Kinect body track controller. So if there's a body part that's paralyzed from a stroke, there's nothing actually wrong with that body part and it can actually be retrained through the principles of neuroplasticity. So you can rewire your brain through the process of repeating the actions. And you could use virtual reality technologies to amplify the progress in order to help encourage the patients, but also be able to track their progress for insurance reasons. So we'll be talking about that as well as using VR to recover from PTSD on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by the Voices of VR Patreon campaign. The Voices of VR podcast started as a passion project, but now it's my livelihood. And so if you're enjoying the content on the Voices of VR podcast, then consider it a service to you in the wider community and send me a tip. Just a couple of dollars a month makes a huge difference, especially if everybody contributes. So donate today at patreon.com slash voices of VR. So this interview with Larry happened at the IEEE VR conference that was happening in Greenville, South Carolina from March 19th to 23rd, 2016. So with that, let's go ahead and dive right in.
[00:01:58.618] Larry Hodges: I'm Larry Hodges. I'm a professor of human-centered computing at Clemson University and co-chairing IEEE VR this year. My research is in all aspects of virtual reality and looking at applications for healthcare, for simulation and training, and a lot of things that have to do with just evaluation of different techniques for display and interaction in virtual environments.
[00:02:24.233] Kent Bye: You were just mentioning to me the IEEE VR. Maybe you could talk a bit about the history of this conference and then your involvement in that.
[00:02:32.948] Larry Hodges: So the first IEEE VR was held in Houston in 1999. Bowen Loftin and I were the co-chairs. Bowen was the main chair and I was the co-chair with him. It's been held every year since then. It's the primary academic conference in virtual reality, so it has Actually, two parts, 3D UI, which is a 3D interaction conference that's always held Saturday and Sunday as a fully reviewed conference. And then the virtual reality conference proper, which is Monday, Tuesday, Wednesday. It's all highly reviewed in terms of the papers so that the quality of the papers here is probably the best in the world. And then there are lots of other aspects. We do a doctoral consortium. We have contests for building systems and looking at 3D interfaces. We have workshops and tutorials. And historically, about a third to 40% of the participants come from either Europe or the Pacific Rim, so it's a very international conference. And it's also sort of a gathering where a lot of us who know each other, because virtual reality is not a large community, get together and sort of renew our relationships to each other and exchange research ideas.
[00:03:41.677] Kent Bye: And so, yeah, maybe you could tell me a bit about the nature of research that you've done in healthcare and virtual reality then.
[00:03:47.061] Larry Hodges: Okay. So, actually, 20 years ago, I co-founded a company that used virtual reality for behavior therapy. So we started out looking at things like fear of flying, fear of heights, and it's called Virtually Better. I'm no longer part of the company. I've sold out all my stock, but it's still in existence in Atlanta. I was at Georgia Tech at the time. Probably the best-known product is they actually install and support systems that are used for veterans returning from war for post-traumatic stress disorder for treatment of that. Now, besides being a professor at Clemson, I have a startup company that's doing rehabilitation games. And not only games, but the games also collect information that help clinics be able to show the patients the progress they're making and use it also for reimbursement for insurance companies to show that they're making progress. and can show how much therapy they're getting. So that's a new start-up. It's here in Clemson, South Carolina, and working with primarily skilled nursing centers at this point. We started out working primarily with stroke rehab, and now anything is upper limb rehab, no matter what the cause, where you need to exercise. And the primary thing is exercise that people will do, is what we advertise. And the biggest competitor for that is you get sent home with a piece of paper that says, go do these exercises, which no one does. And science says that most people who are trying to do like stroke rehab get only about 10% of the repetitions they need to recover use of their arms. There's nothing wrong with their arm, it's a brain problem. So you need to relearn that use. So we developed with the Medical University of South Carolina games that actually looked at the science of the motions they need to make, the repetitions, and what would keep them engaged in order to do that kind of work.
[00:05:31.687] Kent Bye: It's interesting. So most stroke victims that may have like portions of their body that are essentially paralyzed are not forever paralyzed if they are able to kind of rewire their brain through neuroplasticity principles and just through repetition. And is it something that helps inspire them to like show them something in VR, like even if they can't move their arm and extend their range of motion? And so tell me a bit of how VR plays a part in this specifically and what you found.
[00:05:58.595] Larry Hodges: Yeah, that's exactly the right idea. So we are using the Microsoft Kinect as a tracker so that they see the game on the screen. They play the game by moving their impaired arm. Of course, you've hit upon exactly the right problem. They might have very limited motion and there's actually a high number of patients, especially from stroke, who don't have enough movement to do any of the exercises that they need to do to recover. So our game actually scales the movement so when you see a virtual arm on the screen, if you have limited motion in your real arm, then we scale it so your virtual arm actually has a full range of motion so that you can play the game no matter what your level of impairment is. As you get better, the therapist actually can change the scaling. So it's designed so that if you have any movement whatsoever, you can still play the game. As you get better, we can redo it to make it more challenging. And we put the control for all of that with the therapist. So we build an intuitive interface where the therapist makes all the clinical decisions in terms of how much to challenge your patient, how many repetitions. And our job is to make it fun and make it track the data so the therapist can actually look back and look over time at how the patient's doing, how the scaling has changed, how many repetitions they need to do.
[00:07:11.696] Kent Bye: So maybe you could talk a bit about the element of motivation in terms of seeing progress. It seems like if someone's not seeing progress when they're doing these exercises, then they may just sort of like, oh, what's the point? And they give up. Talk a bit about that motivational aspect of actually seeing your body get better and then how that impacts your motivation over time.
[00:07:32.688] Larry Hodges: Yeah, I think there are two things. I've talked to several people who have done rehab of different types. The problems are, one, Normally, what you really get is a piece of paper that says, go do these exercises. So they're boring. They're no fun. They sometimes hurt. And also, the comment I hear is like, well, I couldn't tell it was doing me any good, so I just quit doing the exercises. So part of our goal is to do two things. One is to give them a game that's actually interesting to do. So we do a lot of game things, like you get a score. You have different levels so that you get different visual things. You can level up to a different level of play. But we also have to make it simple, especially for stroke rehab, sometimes there are also cognitive problems. So, you know, there can't be a long list of rules or instructions you need to do. For our first game, the instructions are, when you see a duck, hit it. It's called duck, duck, punch. And that's it. And that's very intuitive. And we're also given that the age group for a lot of stroke survivors is older people. So, you know, 60 or older. We also thought a lot about, well, what type of game would they like to play? So, one of the original games was modeled after an old-time shooting gallery. You would see it at a fair where, you know, essentially geese pop up and you have a gun and you shoot them. So, the artwork for the game actually was intended not to look realistic, but look like an old-time carnival game. And that was an easy connection. Now it turned out, the more we worked with real patients, we found out there's a fair number of young people that are having strokes now. And the stroke belt is through the southeast United States. South Carolinas like to belt buckle the stroke belt in terms we're right in the middle of that. So it's a very serious problem not only for the U.S. but also for this part of the country. So one reason we wanted to address that. And we see, you know, it's not unusual for a young woman to have a stroke after childbirth. I've seen several of those, and we've had kids, actually teenagers, that have had strokes maybe from accidents, like one person who essentially went off a snowmobile and hit his head on a tree and, you know, had a stroke. But the research says that a large number of these guys can regain use of their arm if they get enough repetitions. Again, they're relearning neuroplasticity, just like you said. They relearn the movement, but only about 15% of them ever do. And one of the problems is that in a therapy session, you might get 30 to 60 repetitions of whatever the movement is, which might, and usually it's as simple as reaching out and picking up a Coke can. getting the elbow extension, the shoulder flexation, and the control of the arm back. You need about 500 to 600 repetitions a day according to the research. And the research comes often from animal studies because nobody's getting that much rehab. So the goal was to actually build a game that in the therapy session they could do closer to that number, but also it's simple enough that you could actually take it home and do therapy under direction of your therapist every day using the game. And we've done several pilot studies at the Medical University of South Carolina where if we send it home to a patient, they were actually doing 300 to 600 reps a day. In five sessions, we can see actually measurable progress for people that had a stroke sometimes years ago in terms of their improvement, which is just amazing considering that they've been living with this for years and pretty much what they do is they just quit using that arm. In some of our studies, we would put trackers on each arm so we could track how much they use their arm. In some cases, they play the game so much they use their impaired arm more than they did their unimpaired arm throughout the week.
[00:11:11.649] Kent Bye: Talk a bit about the neuroscience behind it. This concept of being able to rewire your brain is really interesting and fascinating to me. What do we know from the research about that?
[00:11:22.750] Larry Hodges: I think that the simple idea is that it's repetitions and not just any kind of repetition. So the idea that you can play any game and improve is not the correct idea. It's actually movements of a certain kind. So if you go to an occupational therapist, they're very good at having all kinds of ideas, very simple ideas, reaching out and picking up a Coke can. And, you know, different types of movements are harder than others, so movement across the body is usually easier than a movement out away from the body. Reaching up is harder than reaching down. Lots of stroke patients have their elbow extension is very limited, and so they have to really work on that reaching motion. Some of them actually have full extension, but they have real problems with control. So it's putting targets out there so you have the reaching movement the therapist thinks they need to work on and making it engaging enough that they're willing to do enough repetitions and challenge them with like a score or you know getting to level up to the next level. And also I think with our scaling techniques to make it so that they can see success, because if you're very impaired, it's very demotivating. I can't reach out and pick up a Coke can. I can't do anything with this arm. So the fact that we can actually adjust the game from the beginning so that they can see success from the beginning is very important. So there's no magic here except motivating the right movements and engaging the patient so that they do those movements. Sounds like a very simple idea, but it's something for most rehab if you've ever rehabbed any body part People hate doing this because it's boring and they don't see progress and the fact that we actually give you a score Contract your progress over time and show you that you are getting better. Also, I think it's a very important aspect of this So the company that we just started that's commercialized this, it started out as a lab experiment with the Medical University of South Carolina. We built a prototype. One of the sort of adventures along the way is that we entered it into the Microsoft Imagine Cup competition, which was like 500 teams worldwide trying to build applications for the Kinect as a tracker. In the finals in Sydney, Australia, we came in second in the world in that competition, which made us look at each other and say, this might have been a pretty good idea. And so we've commercialized it now. One of my former students is the CEO of Recover, which is the name of the company, and we're trying to get it out to rehab clinics, skilled nursing centers that actually work with patients who are doing upper limb rehab, which is what we're focused on. And to continue building new games on top of this that have the same scientific underlying principles of making the correct movements and also really working with the interface design so that both the patient and the therapist don't have to have a computer science degree to play this game. Everything we can make intuitive and all the clinical decisions leaving to the therapist so they can design the game specific to this patient's needs.
[00:14:14.263] Kent Bye: So one of the things that this brings up is that having these motion track controllers like a Kinect or perhaps other motion controllers in the future, you're able to get more specific data to track the progress of the increase of range of motion of people over time. I'm curious of how you see that's going to eventually change the type of care that they're given, if the system's going to be able to adaptively measure that and actually change their therapy sessions based upon getting information that doctors never had before.
[00:14:45.452] Larry Hodges: Yeah, we have the potential to do very precise measurements with fairly inexpensive equipment now, the Kinect being one of the prime examples, to actually show progress, to document. The big problem that you have to overcome if you have any kind of clinical idea is someone's got to pay for it. So reimbursement in the U.S. is a big deal, and often you've got to show either that the patient is making progress, but they haven't plateaued in order to get insurance to continue covering, or you've got to document the progress they're making in some way. So a lot of the data, besides being motivating to the patient, because they like to see they're making progress, so they keep doing the exercises, but also when you get to the admins who run the clinics and look at the bottom line, it's like, okay, we need to be able to document that the progress made in order for reimbursement issues. I personally, I've had other ideas, which we tested in hospitals, all the clinical people love, but nobody was willing to pay for it. So when you're an academic, you just think, well, this does good for the world. If you try to actually get it out to the lab and to the real world, you also have to talk to the guys who pay the bills and say, this is how it can actually increase your reimbursement or make you money or save you time. or else nobody is going to use your product. So to make the impact, you have to do the business case also. And for people like me who spent most of my career in academia is learning those lessons that everybody's got a good idea. Nobody cares unless you can also make a difference in the world and figure out who's willing to pay for it. That's the real world if you want to make a difference.
[00:16:16.473] Kent Bye: Yeah, it sounds like because this is effective, it could actually save insurance companies money, so they would have an incentive to covering it.
[00:16:22.295] Larry Hodges: Yeah. And we're moving from a system where it's a payment for care to a payment for results. It's kind of more and more important to be able to document results. Lots of hospitals, they don't get paid now for how many days somebody's in the hospital. They get paid for getting them out of hospital and keeping them out of the hospital. So we need to be able to track things and show that what we're doing is effective. And that's collecting a lot of data.
[00:16:46.544] Kent Bye: Do you have any stories of people who used this and it impacted their life in a way?
[00:16:52.246] Larry Hodges: Yes. We've been using this because we are, in terms of FDA, we are a medical device. We were doing joint research at the Medical University of South Carolina until we got our FDA approval, which we do have now, so we can sell systems. but a lot of our early people who are part of our studies came back and you know the one that was one of the first ones is that myself and my two partners for Recover were in actually a digital health accelerator program and we got a call from a therapist we were working with and says you know get on Skype I got somebody wants to talk with you. And she had a stroke survivor who had been working with her who just got so excited over the game and the fact she was making progress. She just wanted to personally tell us what a difference it's making in her life. That just simple things like being able to brush your teeth or put on your jacket without your husband helping you. or putting on arm deodorant yourself was a huge deal. And this particular person who was a fairly young woman who had small children, she had been like a master's level teacher, and the fact that now she was actually being able to recover movement and be able to do a lot of these things, just simple things that we take for granted. And she just wanted to tell us about it. You know, at the end, she was crying, we were crying, we were all crying. If you build a better operating system, nobody calls you up and says, this changed my life. Windows 10 changed my life. The fun thing about doing this is people actually contacted and say, this was a huge change in the quality of my life, and I just want to thank you. That's a pretty cool thing. Computer scientists don't get that all the time. So we really enjoyed that and had several people come back and said, this makes a huge difference. And, you know, we go to great lengths. We had one person in one of our studies who drove down, has use of only one side, would drive down from Raleigh and go to Charleston where we were running the studies just to be part of the study and check in a hotel to spend a week, you know, using the game and doing rehab, which there's some indication that this is therapy they would do and they're seeing results and nothing else has done that before. which is just, you know, a mind-blowing idea that there are something like 800,000 people in the U.S. that have a stroke every year. There's something like 8 million people who live with arm impairment because of stroke in the U.S., and some of which from strokes years ago and a potential to recover even years later, and it's still not happening. So, yeah, we want to make a difference there.
[00:19:11.169] Kent Bye: Is this something that you would be able to deploy into people's homes, or is this something that will have to go into a clinic in order to use?
[00:19:17.991] Larry Hodges: We're targeting clinics now, but clearly we've done actually home studies where we sent this home as research to understand how to do that. So if it goes home, you send home a Kinect, you need a computer, and we hook it into the TV. We've even hooked it into old CRT TVs in people's homes. So it clearly could be used at homes. The idea is that you'd be just like any other prescription. Your therapist and your doctor have to write a prescription that you need to be doing this at home. The data would be uploaded to a portal in the cloud so the therapist could actually see what you're doing, monitor progress, and maybe make changes to the difficulty of the game. So that could be done remotely. The problems there is that a software download would be the least expensive way to deliver to homes, but you need a laptop and a Kinect. We would rather deliver the laptop to Kinect and the software, because then we know everything works. If you use your own laptop, there's some subset of people who are sophisticated enough using their computers and the Kinect and hook that up and have it at home. But you know they're easily people now we could sell it to who would just spend any amount of money for that but there's a large segment of people who can't afford like a laptop and a connect and Till it gets the points where insurance and Medicare will reimburse it then that would make it Yeah, the homes would be the big area where it make a huge difference because now I'm getting this enough exercise, enough repetitions every day to recover. So that's going to be key to making this hugely useful for outpatient care, we think. But financially, there's a lot of documentation we still have to do to get insurance companies and Medicare on board for that. So they probably will do some limited to people who can do private pay just to get the feedback and understand how to do this. But our goal long term is to put it in homes and get it covered by insurance and have like a game in a box where we know everything works. When we sell it to a clinic, we sell not only the system, but it's a monthly fee and we send in therapists to train you. We send in computer guys. If it breaks, we fix it. That's a model we can do with clinics. For individual homes, it has to be bulletproof. So we'd love to give you, you know, rehab in a box to make it as widely applicable as possible. But, you know, there's a financial cost there that still has to be worked out.
[00:21:32.772] Kent Bye: And you said like 600 repetitions is kind of like the ideal number over the course of a day. Is that in one sitting or is that over like five or six different sessions?
[00:21:41.475] Larry Hodges: It could be either. Again, this is a clinical decision that we leave with the therapist so they can set it up so they could freeze the game and come back, which a lot of patients do. We had one guy who was doing it at home who actually was a therapist who had a stroke. And he did like 1,000 or so the first day and was sore the next day. It's like, you idiot, you're smart enough to know you shouldn't have done this. You're a therapist yourself. But he was so excited because he had something he could do. So, again, anything that's a clinical decision, we try to leave it for the occupational therapist, physical therapist working with them. But, you know, even for us, for you or me, it's a very simple game. It's like, oh, this is easy. I could do this all day. But, you know, a couple hundred repetitions later, you begin to like, you know, my arm's getting tired now. But you want to set it up so if they get repetitions a day, that's good. If they do it in multiple settings or a few settings. We were collecting data about how long it takes them to do it, how many sessions it took them to do it, so we can attract all that data for the therapist and understand how to give it to the therapist so that they have the data that they want. Because it's a connect. We can collect all kinds of movement data. More than the therapist wants to see. So part of our research has been talking to therapists and saying, We've got this information. What part do you really care about? Because they can't spend an hour analyzing data. They need to look at something, get a visual representation, and understand where the patient is and what needs to happen next and do that quickly. Because they have a fairly limited amount of time before they have to go on and look at the next patient also. So we're trying to work with that to make it easy for them to understand what's happening.
[00:23:13.303] Kent Bye: I've heard that range of motion is kind of like one of them. What are the big metrics that they actually want to look at?
[00:23:19.975] Larry Hodges: So elbow extension, which is part of the range of motion shoulder flexation the control for different parts of the body So the games are all set up so they can control what areas you're reaching towards so you might be working continually on just the left side or right side our lower areas so the screen is divided up so therapists can control where the targets pop up and whether or not they're just reaching out and Well, we say punch, and we're actually just reaching out and touching things, or if they're tracking things, in some cases we might want something moving where the idea is to move in and hit it up. But, you know, our job is to make a game that has those switches on it, the therapist can actually determine those things. But the fascinating thing to me is that, you know, on the face of it, there's nothing complicated that you need to do. You need to do simple things, but you need to do a lot of it, and you need to stay motivated. And you need to be able to play the game, even if you have very, very limited motion. And you could be sitting in a wheelchair, so we've got to track you sitting in a wheelchair. Often the therapist might be working with you, so we've got to track your arm movement and know it's your arm and not the therapist's arm, and be able to tell the difference. Some of the previous games that have been used for rehab, they had all those problems. And we keep working with our software with the Connect. Early on, if you're sitting in a wheelchair, sometimes they thought the wheelchair wheels were your legs. That really messed up the connect. Now we've programmed around that. We can pick who you're tracking and it'll stay on that person so the therapist can work with you. We've even had people in a sling because they couldn't move their arm at all unless it was counterweighting. Play the game and actually be able to successfully do therapy with it. which amazed even us because the sling, you know, okay, I've got lots of other things on my arm, yes, can it connect, recognize this is an arm. Turns out we could get it to do that and to make it as accessible as we can to as many people.
[00:25:14.610] Kent Bye: Is there a VR component to this or plan in the future?
[00:25:17.823] Larry Hodges: So it depends on your definition of VR. So we do this on a big screen display. We do it for Kinect, so we're doing body tracking, which is part of the definition of VR. Doing this in a head-mounted display and things doesn't make sense because you don't need that. There's also feedback in terms of what you see on the screen is corresponding to how your body is moving. And there are secret sauce things in terms of what we recognize and what kind of movements we actually reward you for and what kind of movements we don't reward you for in terms of playing the game to get you to move correctly, which we've spent by looking at a lot of literature for especially stroke rehab, but rehab in general, and a lot of talking with therapists and building systems. And luckily, we've had just excellent group at the Medical University of South Carolina working with us. So even though physically there's like four hours difference between us, we do a lot of Skyping back and forth. They'll use our system with patients and send us videos and do a post-game analysis. These are ideas we had that really worked. Here's some things you did that really are not working at all. That wasn't our best idea, meaning our best idea. And actually a lot of feedback from therapists who really have been on board with us and helped us. learn the vocabulary and understand what's useful for the therapist and the patient as opposed to what just seemed like cool ideas that we had. Because the computer guys should never let them do the interface design, right? Because we see the world differently from the real customers that we're trying to reach out to.
[00:26:46.173] Kent Bye: Has there been any resistance from the people in this age demographic to the idea of playing a game or a video game in terms of their recovery?
[00:26:54.320] Larry Hodges: You know, not at all like we thought. So we had quite a few older people play the game, and once they see the game and play it, they're enthusiasts. Now, early on, when we said, we're going to play this game, one of our very first studies was an older gentleman. He was going to the clinic. He was going to play the game in the clinic. And he has like a grandson that tells him that, oh, you're going to play a game. He says, you'll be shooting people. And he was actually a war veteran. The idea of shooting anybody in a game was completely abhorrent to him. So he called up the therapist and said, I can't do this. I don't want to shoot people. And she had explained to him, no ducks were harmed. That's not that kind of game. But once they see the game, and we designed it specifically to correspond to types of games they might see at an all-time fair and be familiar with, and that it's actually Simple, there are no complex rules, but it's visually engaging. There's audio and there are different levels to try to keep you interested. So once they saw it, we've had people in their 70s playing at home on an old-time CRT TV that's hooked up to a laptop. and doing their exercises and thinking, this is great. And at the other end of the spectrum, we've had some very, very young kids with strokes. It's not a very interesting game, yeah. But after a while, they started playing and realized they were getting better. They came along. Our best story for the engagement of the game, though, is we had a woman who had young children, old enough, and they were really jealous. Mom gets to play the game and we don't. So Curtis, there's no reason why they shouldn't. So now there's something they can do with mom. They could sit down and play the game and they could compete, because we can make it harder for them. And the fact that that was actually, all of a sudden, there's something normal we can do with mom. We're all doing the same thing, we're competing. And we thought, well, that's actually a great idea that we never thought of. And so we set it up so the kids can log on, or the parent can log on for the therapy. But the kids can also play, and they can compare scores. That's something that was not even in our head when we started. Later on, we thought, this is great, because it's a leveler, and we're all in the same boat here.
[00:29:03.930] Kent Bye: So how do you make it harder for them?
[00:29:06.174] Larry Hodges: Oh, we can scale. So we make it so that they have to be much more precise in how they reach out. They have to do the full reach for mom might, you know, get it scaled. So she has limited movement. So she hasn't moved so much. So that's all built in. So the therapist can actually scale the level of difficulty to the level of challenge the patient needs. We can challenge, you know, normal people too, by making it, you know, you got to do it faster. You got to do it more precise. You got to do a longer reach.
[00:29:31.480] Kent Bye: It sounds like you're creating this whole collaborative social game that, you know, rehab used to be very individual. People never wanted to do it. But now, if the family's involved, actually, that could be an additional motivating factor, I would think.
[00:29:43.271] Larry Hodges: I think that's pretty important for sending it home and working with it. And we're still doing that on a research basis. But hopefully, if we work out the issues that have to do with reimbursement and insurance coverage, that will be a normal thing. That'd be awesome.
[00:29:57.256] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:30:04.580] Larry Hodges: Okay. So my opinion is that there are especially immersive virtual reality since we're having a lot of head-mounted displays come along. I purchased my first head-mounted display in like 1992. So I've been doing this a long time. But from my point of view, niche areas where immersion does things that you can't do any other way. And the applications that have been successful up to this point have been fairly limited. I mean, there's what I call expensive systems, rich customers, so training astronauts, a lot of military training where you need to be immersed in the environment. That's been very, very successful. Virtual reality therapy is one of the few areas where head-mounted display has been commercially successful in terms of working with veterans returning from wars with PTSD where the therapy involves essentially putting them back into the environment and expose them to certain stimuli and get them really to tell their story because part of the way you deal with PTSD is you need to get them to talk about it and this is the last thing they talk about. You need to talk through it and you know pretty much over and over until it loses its power over you. And the immersive virtual reality has been very successful in doing that. some types of training, especially sort of social interaction training. So here you'll see immersive virtual environments, but also virtual characters that interact with you. And one of the projects one of my earlier companies did is that we built a system for the CDC to train them in aspects of the culture for when they went into places where maybe there was a plague in some country where they were Historically, you sent sort of noxious stuff to CDC and they analyzed what it is. Now they put a lot of boots on the ground. So they said, well, we send medical people into this environment where they understand the medicine, but they might not understand the culture. Like it may be the culture that if someone dies, the family gathers around and washes them and dresses them and buries them. But if they die at a plague, you can't do that. You've got to deal with the cultural issues, which are very, very strong. So we built training systems for CDC personnel to understand the culture of interacting with different people or maybe interacting with different agencies that have different points of view of how to deal with things. And this was before some of the plagues that actually hit Africa just a few years after that. So anything that has training in social interaction, you know, from job applications, asking a girl out from a date to going into another culture, because you need to be immersed and get rid of that. And, you know, the interesting thing that's happening now is you see news reporters with, you know, sort of the inexpensive head-mounted displays. I'm going to take you back so you understand what was happening. So what if you could be in a crowd when Martin Luther King did I Have the Dream speech? You know, not just hear about it or hear, but I'm standing there and there are people around me and I'm seeing their reaction. So I think, you know, the historical reenactment and understanding the news in other countries in immersive environments is a pretty cool idea because you get this subjective impression, which is very, very important to reality that you don't get by hearing about it or watching it on TV. Games would be interesting to see if it's a lasting thing because we've had it come and go with games. Because the thing that makes virtual reality really, really effective is that it takes you out of the physical environment you're in and puts you in a different environment. That's also the biggest problem with it because I'm no longer connected to this world I'm in. And some people just, okay, I want to see what's going on around me in the real world. So the trick is to look for applications where putting you in a different environment and immersing you in it is actually an important aspect of what you're trying to learn or understand or do. Those are the really sweet spots. An early example, years ago, there were several, you go to arcades and stuff, there were virtual reality games. Many of them were unsuccessful because people took games that were made for a screen and put a head mount display, all the action was in front of you because it was made for just a regular computer screen. So the fact that you could look around and you had like 360 degrees that you could look around, there was nothing there. You know, there's nothing interesting about the game because it wasn't designed to take advantage of the fact that there could be stuff going on behind me. You know, it's not just on a screen in front of me. So you have to also design the software to realize, okay, I need to take advantage of this field of regard and field of view that I have to work with. and also thinking about, you know, subjectively, what am I trying to learn? If I can learn it looking at a big screen, VR is a waste of time. But there are applications, I think, and we're finding them where, you know, we need to be surrounded, and not only visually, but with auditory cues, which actually is a very, very strong cue. One of our early research projects, we built a virtual Vietnam for Vietnam War veterans with PTSD. So when we demoed it to visitors to our lab, if it was someone that was old enough to be in Vietnam, I'd tell my students to come and get me when we do this demo because even guys without PTSD would get strong reactions. We built some of the early environments that they would experience if they were in Vietnam at all, like going somewhere on a Huey helicopter or being in a landing zone. And I did one demo, and the guy took off the head-mounted display, and we had a lot of audio cues, and he was talking about how good it was. And I asked him about the visual cues, and he looked at me, sort of blankly in his face, and said, there were visuals? The head-mounted display visuals was turned off. All he got was audio cues. But he thought it was great, and he found out he could have seen something also, because it brought back what would happen, because we'd been very careful to try to design the audio cues to correspond to things you would hear.
[00:35:42.243] Kent Bye: Just to follow up on that PTSD like that seems like there's the process of exposure therapy How do you kind of dial in the specific triggers? I guess because it's such a broad Range of way that people get traumatized So what what have you seen in terms of like what does that process look like and then how do they like recover from that?
[00:36:00.779] Larry Hodges: So for any type of environment you build for PTSD, the things you have to take into consideration are that one, you can't build just any environment. Some may not be appropriate and it can't be so specific to one person that you have to design it just for that person. And the most successful applications have been things to do with war veterans. As you look and see, what are the common experiences that almost everyone that participated in that conflict went through? So if it was Iraq. It would be like driving down the road and possible IED blowing up. So you reconstruct, you know, to drive down the road, you could have, you know, maybe an IED go off and blow up the vehicle in front of you. It's the things that, you know, could really happen to almost everybody. Even if it didn't, there was the potential because That's how that war zone worked. Our early work with Vietnam, it was riding the Huey helicopter because that's where you went everywhere. It was landing in a landing zone because as soon as you landed there could be snipers in the trees and you needed to get under cover quickly. The other one that we couldn't do at the time, the technology might have caught up to us now, is that being in the bush where you could hear voices around you, some of your buddies and some actually of the enemy, but the vegetation is so thick you can't see where everybody is or where they are. We couldn't do that with the technology when I first started this 20 years ago. So we never built that one, but the Huey helicopter and landing zone were common, riding in a Jeep. If you go to virtuallybetter.com, they've got videos of some of the scenarios that are used now in VA hospitals. So the trick is understanding enough about the war zone that say, what are the common experiences that all these guys had to go through? You don't have to do the specific experience. You know, often we found that once they got talking, they would start seeing and hearing things that we never put in the simulation because all of a sudden, It's a storytelling environment. In fact, I think that's the best description of how the therapy works. You want them to tell their story, and you give the therapist control as they tell their story. Anything we did put in the environment, they can actually trigger. They said, then we started hearing the mortars come in. You can trigger mortars. My buddy in the truck in front of me all of a sudden blew up. You know, so you want to put in enough that there's something in the common story that gets them going. And from then, they may tell a story completely different from what you're doing visually, but they're there. And part of the therapy is to get them to talk and in some sense re-experience this till it loses its hold over them. The other important thing is like virtual reality doesn't cure anyone. It's having a therapist who understands how to use it. And this is part of a larger therapy regime that they put them through, but it's actually can be very key to get them to the point that they want. So, yeah, our job is actually support, you know, for, you know, of our rehabs work, our early work with phobias and anxiety disorders is try to understand enough about what the therapist is already doing and what they want to do to support them. And I think that's the whole idea of HCI is that we need to understand our customers. And to be successful and not just talk to each other about technology, but actually get it out into other disciplines and make a difference, we have to understand what their needs are and be responsive to them.
[00:39:17.067] Kent Bye: Good. Great. Awesome. Well, thank you so much. You're welcome. So that was Larry Hodges. He's a professor of human-computer interaction at Clemson University and one of the co-founders of Recover. So I have a number of different takeaways about this interview is that, first of all, I didn't realize that it was possible to recover paralyzed parts of your body from a stroke and that it was more of a matter of the brain than the actual body. So, just by using the principles of neuroplasticity, they're able to rewire the brain to be able to recover functionality of that body part, even if it had been dormant for a number of years. So, this is similar to other virtual therapy podcasts that I've covered with James Blaha and Vivid Vision back in episode number 11, as well as 346, as well as with Michael Ayrto of VR Recover back in episode 273. So, VR Recover is different than Recover, which is what Larry Hodge is. was talking about, but they're essentially doing the same thing which is using a Kinect body tracker to be able to track someone's motions and be able to project it onto the screen and you have this gamified experience. And the big thing is that they're amplifying the movement so that people can start to get a feeling of having the full expression of their agency with that body part. And that actually is very encouraging for people to be able to see that they can actually do something with that arm. even if it's very subtle. And so they're taking that and being able to also just track progress over time, which I think is also a huge motivating factor for people who are trying to keep motivated to keep on doing these exercises that are, first of all, pretty boring most of the time, but can also be painful. So the boring part is being solved by creating into a game and they're able to inspire people to do the number of repetitions that they need to in order to actually start to have the recovery of the mobility of their limbs. So I expect to see this type of application expanded to many other different types of rehabilitation, using virtual reality technologies to really gamify the process and to encourage people to make it interesting, but also to really track their progress as subtle as it might be, not only just for the insurance purposes, but for their own sort of intrinsic motivation to be able to do the exercises. So in talking about using a Kinect camera in conjunction with a flat screen, that is one definition of virtual reality. I think a lot of people right now with the consumer VR think that it has to be, you know, head-mounted displays and that's the only type of VR that's there is but there's been a long history of other types of virtual reality systems like cave systems which is essentially taking like a corner of a room and using projectors to project a stereoscopic scene onto that and you're wearing these shutter glasses so that you can actually like see 3d depth but just even tracking the full expression of your body and translating that into a virtualized experience is one form of virtual reality and i think that we're going to start to continue to see more and more of that especially here after ces this year when we start to have More and more opportunities to track more of our body parts at this point inverse kinematics with just your hands It's not enough to actually get the full expression of your body and so you need way more than just motion track controllers to be able to hold on to and if you have like a paralyzed hand and you're not going to be able to actually like grip and hold on to something and so I expect that, you know, with the Kinect camera, that's something that's kind of the first generation of this tracking technology. We're going to start to see a couple of things. One is that the Microsoft HoloLens is going to have a lot of those sensors that were the thread of Kinect is the most advanced part of those tracking sensors are now going to be in the augmented reality headsets and to do inside out tracking. Perhaps they'll be able to actually track other people as well and do something similar, but that's not really practical in this use case where you need to actually have it steady and just use the sensor. And so I think that something like the lighthouse and having more bodied parts tracked on your body, or perhaps it'll be more camera based like Oculus is going with, that they'll be able to add more artificial intelligent algorithms to be able to do more sophisticated full body tracking in the future. So as the VR technology starts to get into the second generation here of the consumer VR, we're going to see more and more ability to be able to track more parts of our body. That's going to be great for invoking the virtual body ownership illusion, which really needs to have very specific tracking of your full body in order to actually feel like it's your body in VR. But it's going to have a lot of medical applications, like Recover, where you're going to be able to track movements of your body and be able to provide all this data to doctors. And it's going to change the way that doctors are treating patients, having that access to the data to be able to be more specific to the things that they need. but I think inspired new rehabilitation practices and regimens for people, these spaced repetition. With Larry, he said they needed to just get kind of a flat 600 repetitions in a day. It didn't matter if it was in sequences or all at once. But I think the important thing is that day in and day out, it's like a daily practice that you get all these different repetitions in. And perhaps we'll discover more nuanced ways of trying to space different types of movements in order to really optimize that healing process. So as they get more data, I think they're going to be able to be more specific for the type of practices that people are doing in order to recover. And to kind of take this as a principle of neuroplasticity and to look at what's already being done with rehabilitation and with James Blaha being able to see in stereoscopic 3D from essentially curing his lazy eye, I think we're going to start to see potentially this expand out into latent human potentials of things that we don't even know that we are Capable of because it has been too boring to try to do that daily practice every day That's the theme that I see is that these are things have always been possible with a lazy eye or stroke recovery They give you a sheet of paper and say do these exercises and people are like these are really boring I can't see any progress but when you start to have a capability of track subtle progress over a long period of time you you apply that to all sorts of other different dimensions of what the human body may be even capable of. And I think that we're going to start to see VR open up new human potentials that we don't even know that we're capable of doing just yet. And to me, that's one of the most exciting unknowns about the future of virtual reality that I can think of. And finally, just to say a few words about PTSD recovery, Larry was saying is that there are some veterans who have had these experiences and they're kind of archetypal experiences of either flying on a Huey helicopter or driving down a road in Iraq and potentially having an IED explosion. So these are types of situations that many, many people have been through. And so they're able to provide a VR experience that gives them that experience. And whether or not they went through that exact experience, the point is that it's trying to evoke these different memories that perhaps they have not fully remembered or articulated. And being in the virtual experience, they're able to go back to that place and have this embodied cognition, which the environment is triggering all these different memories. And so From there, they go into this storytelling mode where they're able to tell their stories about what happened to them personally and potentially do it in a way that is able to not be so charged and to be able to use the principles of exposure therapy to slowly build up to the point where they can tell more and more of their story such that once they are able to articulate the narrative of their experience, then the charge of the PTSD is lessened. And so that I think is a really super powerful use of virtual reality and that they're starting with veterans because they have these clear archetypal experiences that they're able to give to people in order to remind them of the different traumatic events that they've been through. And so moving forward with virtual reality, I expect to see more and more of these archetypal templates that are able to potentially in a safe way allow people access to be able to be in the emotions and the memories that they went through but to be able to articulate their story without being overly triggered within this hyper vigilant state where they're just completely overwhelmed. So I think that this is something that is for sure needing to be worked in cooperation with trained therapists who are able to be there and to be able to listen to that story, but also to help guide the person through that process of going through that experience without being too overwhelming or being too much. So that's all that I have for today, and I just wanted to thank you for joining me here on the Voices of VR podcast. And if you enjoyed the podcast, then please do tell your friends, spread the word, and become a donor to the Patreon. Just a few dollars a month makes a huge difference to allow me to continue to do this work for you and the rest of the VR community. So go to patreon.com slash Voices of VR. Thanks for listening.
