#134: John Quarles on using Virtual Humans to improve learning & teach joint attention to adolescents with Autism Spectrum Disorder

Dr. John Quarles is an assistant professor in the San Antonio Virtual Environments lab at the University of Texas at San Antonio.

He talks about some research that his student Chao Mei did in researching the impact of customizable virtual humans for hand-eye coordination training game with adolescents who have Autism Spectrum Disorder (ASD). They expected the adolescents to be more engaged and play for longer, but they didn’t expect that they would actually perform better when they’re able to customize the virtual humans within their Imagination Soccer training game.


John talks about their findings as well as some of their future research that they’ll be looking into how to use eye tracking technologies in order to better train adolescents with ASD to improve the abilities of maintaining joint attention. He talks about using Tobii eye tracking along with a Kinect sensors. They’re not using VR HMDs yet because the eye tracking technology isn’t affordable enough to be accessible to all of the therapists who could use it.

John is skeptical as to whether or not virtual reality technologies will ever be able to fully replace human therapists. Even though adolescents sometimes prefer to interact with virtual humans over real-life humans, being able to successfully navigate social interactions with real people is something that they’ll ultimately need to be able to learn how to do.

The interesting takeaway that I got it that there’s something powerful and potent in allowing the users to customize the virtual humans that are in virtual environments. It seems to make people more invested and engaged, and as a result could actually enable them to perform better at specific tasks. There’s further research that needs to be done investigating this, but it adds another incentive for virtual reality developers to allow for the customization of specific elements within the experiences that they’re creating.

Theme music: “Fatality” by Tigoolio

Subscribe to the Voices of VR podcast.

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast.

[00:00:12.136] John Quarles: My name is John Quarles. I'm from the University of Texas at San Antonio, and today we're presenting a poster that one of my students, Chow Mei, in collaboration with Lee Mason, who's a behavioral psychology autism therapy expert also from UTSA, What we're trying to do is create a virtual character that kids with Autism Spectrum Disorder can customize to be to their specific interests. Because one of the problems that kids often experience in Autism Spectrum Disorder is that it's not really a problem, it's just a lot of kids with Autism Spectrum Disorder will have very specific interests. And so it makes it a little bit harder to get them motivated into therapy and things like that. What we wanted to do is, could we create a virtual character that plays to anybody's interest? So we have lots of different variables that you can customize, like if you want a really strong character, or a character that's more your age, or a character with a specific hairstyle, or even customize a t-shirt so you can go on Google Images and find whatever thing that you're interested in. And the idea was if we could do that, maybe, and integrate it into a game, maybe we could motivate these kids a little more to spend more time in therapy and just have a little more fun with the therapy. And so we created this soccer game where you customize a virtual character to the way you like it and then the character kicks a soccer ball at you and the idea is that you have to, you know, wave your hands and block the soccer ball. So we're also kind of training hand-eye coordination at the same time, which is also a problem with autism spectrum disorder. We ran a study where we compared playing the game when you had customized a virtual character or playing with a virtual character that you had not personally customized. And what we expected was going to happen was that, okay, they're going to be more interested with the character they had customized. They're going to play longer. And that did happen. But what we didn't expect that it was going to happen was that they actually performed better. So even if you take the same amount of time with a customized virtual character and a non-customized virtual character, they still performed better. So they actually learned the hand-eye coordination skills more effectively with the customized virtual character. Which we, I'm not entirely sure why. It may just be like a motivation thing. Like they really just tried harder with a customized virtual character. But yeah, so in conclusion, in general, if we can create more customizable virtual characters that really play to what these kids' interests are, then we can create a better therapy experience and potentially have more effective therapy. But we have to sort of test this on a more longitudinal study to really verify that. But yeah, so that's future work.

[00:02:55.149] Kent Bye: And so what other type of things can you do with kids who are on the autism spectrum disorder on that spectrum? What other types of skills may you be able to train them within a virtual reality context?

[00:03:07.382] John Quarles: That's a good question. So actually the next part of future work that we're working on is we're focusing on joint attention. So joint attention is, and this is a common problem that kids with autism spectrum disorder have, where if I gesture to something, like I'm talking about this picture, or if I just look up at the picture, then a lot of times kids with autism spectrum disorder are going to have difficulty being able to understand what it is that I'm gazing at, or what it is that I'm gesturing to, or paying attention to it. And so they go through a lot of training and therapy to try and improve this joint attention so that we're both sort of, so when you're communicating with somebody you can understand what the other person is talking about and learn more effectively and things like that. So right now we're working on creating a customizable virtual character, actually a customizable virtual teacher or a customizable virtual therapist that the kids can customize to the way that they, based on their interests, and then the hypothesis is that it will improve their overall efficiency in training their joint attention skills by having these different tasks that the virtual character teaches you, you know, how to play the drums, for example, or what the different sounds the drums make, or, and there's several different scenarios, but that's, yeah.

[00:04:21.850] Kent Bye: Yeah, it sounds like because you would have head tracking within a head-mounted display in virtual reality, you may be able to actually track to see what they're actually physically looking at.

[00:04:30.056] John Quarles: Well, what we're doing, actually, I mean, interestingly enough, that's not enough. So the trick with joint attention is that it's really about eye gaze. So the next thing that we're doing really is we're taking into account eye gaze. So the virtual character knows whether or not you are looking at their eyes or what you are looking at. So it's not just head gaze, because I can do this, right? I can move my head around. But it's really about where specifically I'm looking with my eyes. So that's one of the things that we're looking at is using that as an interface in these kind of interventions for autism spectrum disorder.

[00:05:07.924] Kent Bye: And so what type of setup would you use, like in terms of eye tracking or HMD, or how is the virtual character interacting with them? Is it on a 2D screen, or how are you able to kind of interface with a virtual character and still kind of track all these things?

[00:05:20.069] John Quarles: So we're trying to kind of keep it low cost. so that in the future if people want to be able to practice these skills in their home, then they could. So we're actually using a Tobii iX tracker which is only about $150 or something. It doesn't fit into an HMD and it really only works on like a desktop size, like 22-inch display or something like that. I mean, later maybe we would go into putting an HMD on them, getting an eye tracker on the HMD, but that's of course still, even with an Oculus, that's still a pretty costly, because I mean, those kind of eye trackers still cost like $10,000 or something like that, right? For interaction in this one, we used a Kinect in the soccer game. But in this next one, we're going to be using a Razor Hydra just because we don't need to do full body, basically. And it's a little more reliable for just tracking the two hands. It's all the orientation and position of the two hands. And it's not a huge space. So the Razor Hydra kind of made sense with that.

[00:06:19.416] Kent Bye: And finally, it seems like these types of virtual technologies have the potential to train a lot of social skills with kids who are on the autism spectrum disorder. And so I'm curious what you think the ultimate potential is for using these technologies in a therapeutic context with people with ASD.

[00:06:37.291] John Quarles: The way that I see this is that you're never going to replace the therapist. But the therapist in some sense is a bottleneck, right? Because you've only got a certain amount of time with a therapist and you have lots of kids with autism. So the way that we see this is that it's going to be more of a supplement. So outside of therapy, they'll be able to practice these same skills in therapy, plus getting the extra benefit from being able to customize the characters, because you can't exactly customize your therapist. I mean, the therapists are smart and they'll try to play to the interests of the kid that they're specifically working with, so in some sense they'll try to customize themselves a little bit. But they can't change their facial features, but you can do that in virtual reality. So yeah, there's some added benefits, but I think it'll end up being, we'll end up being able to make therapy more effective, but it's not going to replace the way that therapy is done right now. Because at the end of the day, you still got to interact with real people. And one of the interesting things about autism is that I've read a paper before that a lot of times, and I don't remember the citation, but there are a number of papers that reference this where A lot of times kids with autism will actually socially be more comfortable with interacting with a virtual character or a cartoon-looking character than a real person. And I think some people have done some studies where you can kind of slowly add more realism into it to kind of get them up to training with a real person. So I can imagine this being used in that way as well, yeah.

[00:08:22.423] Kent Bye: Great. Well, thank you so much. Yeah, sure. No problem.

More from this show