Saadia Khan is an adjunct assistant professor of psychology and education at Teachers College, Columbia University. She has research is with looking at how using avatars in virtual worlds can improve learning and how they can make you feel better.
Saadia explains how Embodiment Theory shows how experience things with more than one sense can improve learning, and that virtual world avatars can also provide that type of multimodal learning. Avatars can increase interest, focus, motivation, engagement as well as making a more emotional connection to characters and periods in history.
She describes some of her research in using virtual worlds for education, and the importance of identifying with your avatar in order to have a self-image in the virtual world which can provide stronger sense of embodiment. There a lot of potential for using virtual worlds and avatars for education, and Saadia is definitely on the cutting edge of researching this field.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.896] Saadia Khan: My name is Dr. Sadia Khan. I'm an adjunct assistant professor of psychology and education at Teachers College, Columbia University. The kind of research that I do is in immersive technologies and my particular focus right now in my area of interest is immersive technologies which are virtual. So I've been looking at how avatars can really help people learn, how they can help people feel better as well. Because feeling better or feeling positive has been found to be related to improving your learning, improving memory comprehension, also your focus, and also your motivation. So, at Columbia Teachers College, I have been doing research for the last several years, and this started when I was actually doing my doctorate as part of my dissertation. And I was told by everyone that, well, nobody has ever done that before. So I was like, yay, all right, let's go for it. But I never imagined that nobody had actually worked consistently towards actually establishing a whole branch of research which only focuses on how avatars really work in improving learning gains and improving motivation. So initially I have divided my research into two main areas of focus and one area of focus deals with just experimental studies. So as a psychologist, since I'm a psychologist and an educator, I design different kinds of studies, They're not lab experiments like in a science environment, but it's a social science. So the lab experiments that I have basically involve giving people different kinds of learning tasks, having them embody avatars. and then seeing how they learn from that, getting feedback from them, measuring their learning gains. And what I've found is, in all these various experiments, that the majority of participants, which is a significant number of participants in these experiments, actually report that they feel better. that they have higher motivation, that they have more enjoyment, and that's not it. It seems also to be related to their learning gains. So, they score higher on memory tests as compared to those who don't experience embodiment through their avatars. Also, transfer of learning, as I was talking about in my talk, they can actually learn in one domain, for example, history that I tried, and then that transfers onto another domain, which is, for example, I tried English literature and it transferred. Also, for example, they can learn something like psychology of thinking course teaches them problem solving, and they can transfer that into different domains. So they can transfer that into a real-life scenario, or they can even transfer that to something like math. And so, other than that, once I had gathered all that data, and it consistently showed that these avatar embodiment experiences were actually helping people learn, the next step and the next question was obviously that, well, you know, how do you apply this in real life? So we have this data within controlled environments. That's what lab experiments are. So then the next step that I took was to take that into the classroom. So I teach these courses at Teachers College Columbia and these are education and psychology courses and I teach both online and in class. One of the major concerns of the online courses that I teach is that students find them more difficult than in-class courses. They find them quite difficult, in fact, because these are two quite dense psychology courses. They have huge books and they find it difficult. So I brought in the avatar experience for them. And if you look at the theories of embodiment as they exist right now, these theories suggest that bodily movement, whether you actually do that physically or whether you imagine the movement or think about it or whether you use certain devices or whether, as I spoke in my talk, use a surrogate or a deputy to perform actions for you, All of these have, over time, been shown to help people improve their memory, to help them understand things better. For example, I'm talking to you, I'm gesturing. We all gesture all the time. So even that helps with cognitive abilities and cognitive processes. So I brought in that avatar experience and I asked my students to create avatars and then I gave them different activities during their online classes which they had to complete as avatars and after the activities I would ask them to report their levels of enjoyment But that's not enough where learning is concerned, right? So they enjoyed themselves great, but did they really learn? So then I gave them tests which were open-ended. So we coded them because we were not testing for their recognition memory. We wanted to see if they actually remembered things. We also wanted to test for comprehension, so we asked them questions. And we had a control group where certain people were put in a section where they got no embodiment, no avatar experience, and then we compared both the groups. And when I compared these groups, the avatar experience group, or the embodiment group as I like to call them, did much better. So they had learned more, they had enjoyed more. The only group which seemed to enjoy the experience a little less were, as I said, people who are above 40. But if you look at it from an experimental point of view or from a scholarly point of view, it could have something to do just with the sample. And the sample was restricted only to graduate students at Teachers College, Columbia University. And so therefore a good thing to do would be to repeat this with a different sample so that we get a more representative sample of perhaps that age group. And it was a wide age group because it was like 21 and the oldest student was 56. So it's pretty wide. So that is something else that I have done and that basically takes the theoretical part of it and the theory that I've developed with this into practical application because it's not just important to come up with theories and you know write about them. It's more important and equally important to actually go and try it in the field and see if it works or not. So I've tried this with my students, and my goal right now is to encourage other people to go ahead and try distance learning with the use of avatars. And I'm a graphic designer, so I also know programming. I'm not an expert, but you know, it's good enough. So these things are easy to me, the technological part of it, but I understand that a lot of educators find it very difficult, and they need support, and they need training. So another thing that I'm very interested in encouraging others to do is to have some kind of training, teacher training, so that teachers adopt these technologies more. And, you know, not just get impressed by them or look at other people doing it, but do it themselves. And once they do, they'll find that, you know, as Zarin was saying, for example, it does change you in a better way. in a good way. So that's basically some of the things that I do. So that's the crux of my research right now. Another thing is, apart from the learning aspect as I just mentioned, I'm looking at how these immersive technologies actually help you improve your mood, improve how you feel, make you feel more positive. And that's the positive embodied affect that I was talking about. So, you know, physically smiling, infectious smiles can make other people smile. You know, you feel a positive person, you feel more positive. Similarly, I tried that with my students also in my research, asked them to smile, they did report that they felt more positive, and you know what? Their learning gains were higher. So now that could be another variable that we didn't consider because that's what happens, right? Some extraneous variable in there that, you know, maybe made them feel better and not really the smiling aspect of it. But, you know, when you have subjects or participants who are going towards, you know, six, seven hundred, then that reduces the percentage error to quite a bit. So I feel that it is worth exploring and it is worth looking at. And I'm hoping that, you know, more people catch on to this and get into this field and do more good work in it.
[00:08:50.560] Kent Bye: And so from the perspective of virtual reality, a lot of times when you're an avatar, you're in a first person perspective where you look down and can see your body. And I know in some virtual worlds, you can have more of a third person perspective where you actually see your full avatar. And so maybe you could talk about what perspectives you're using with your avatar research, as well as what virtual worlds you're doing this research in.
[00:09:10.560] Saadia Khan: Okay, so initially what I did was I started with Second Life and the reason being that first of all I was working at Columbia and Columbia has an island, it's very well developed and it's there for research purposes and they maintain it, they pay for it. So that's something that I started using and I found that students and learners that I was working with found it very easy to use. They have support if you need it and it's pretty well maintained. OpenSim I didn't go for because OpenSim requires much more work in the sense that it's not as well developed. Nobody else is doing it for you. You have to do it yourself. And assistants that I had were not very well versed and you know since I do so much I don't have that much time to do that. But that's an option and that's a good option to have. Same scripting language as well. And open source is always good. So that's basically what you saw today. That's what I have been doing. So in Second Life, when we were doing these experiments and also using them in classes, we're using all perspectives. So if you look at the views, you can view yourself from the rear. You can view yourself from the front and also from the side. So I make sure that everybody uses all three perspectives. And here's why. When you think of yourself, you have seen yourself from different angles using a mirror, right? Everybody has. So you have a self-image of yourself. If you just use an avatar from behind and never get to see the avatar ever, you will not be able to relate to it at a psychological level because you don't know what you look like, basically. And I have asked a few people to do that. and they don't seem to relate that much to the avatar as much as if they get to see different angles. Now when they're actually using the avatar, after they've viewed the avatar, moved around, they're free to do what they please. In my research what I do is I keep track of what they're doing. So there's definitely a difference when you use perspectives and there is research available on that. So that's a good thing. You know, a substantial amount of research is available on perspective taking and how it really helps and how it doesn't help sometimes. Another form of perspective taking apart from the physical aspect of it is also putting yourself in another person's shoes, for example. So the research that I was talking about, I had two characters from the Indian subcontinent and, you know, historical characters. Oddly enough, they were both Muslim characters, right? One was from present-day Iran, then Persia, and one was the Mughal Empire. They only had, you know, these Mughals, and that was their religion. So the perspective that people were taking in the United States were of two historical characters who were completely different from them, and also from regions which are in conflict with most countries nowadays, right? Not India, but you know, Afghanistan, and because it was part of India at that time. So that was an interesting experience in itself, and I didn't have time to talk about that, but you know, that kind of perspective taking. How does that affect someone in understanding, in conflict resolution? Or, you know, does it or doesn't it? That's also something that needs to be explored further, and some people are doing it, so that's another thing that I was looking at.
[00:12:43.405] Kent Bye: And what do you think are the underlying psychological principles that are contributing to an improvement in learning or engagement when you're embodied within an avatar in a virtual immersive environment versus not?
[00:12:56.395] Saadia Khan: So one of the main things that happens apart from the enjoyment aspect, which is a huge aspect of this, which is if you don't enjoy something, you will tend to not learn as much because you will find it dull or you'll find it boring and it becomes tedious. Learning has to be fun. There has to be an element of play in it, but not too much, right? You shouldn't get so aroused that it distracts you. One good thing about virtual worlds is you're sitting in one place. And from a cognitive or psychological point of view, you can focus your attention through your eyes rather than moving around. In one of my experiments, I had people move around versus sitting down. So the ones who were sitting down showed better memory. They didn't show better comprehension, but better memory, because the others were too distracted when they were moving. So that's one of the things. But other than that, one of the main things is embodiment theory. And it's not just theory on paper. It's something that is supported by neuroimaging data. It is supported by research by other researchers and scholars. and especially scholars who are into embodiment and embodied cognition. And that basically states that any kind of movement that you perform actually helps with cognitive processes. And a lot of research in gesturing, for example, supports this. Because gestures do help people talk. Gestures can help people sing. Or they help people with math. So, gesturing. Next comes dance. Dance has been known to help people. So, what if we bring all of that into the learning environment? It does help people. So, as I said, physical movement might not always be the best option because it has to be goal-oriented physical movement, not just, you know, going crazy and running around. So in that case, then you can bring in virtual worlds. Avatars help because they provide embodiment, but they also provide you with a second self. And gradually I've seen when people use an avatar over and over again, that second self merges with the primary or the first self, so there's a merging going on. And then my student has a cat, as I said, but the cat is no longer my cat avatar. It's me. One of my students chose a male avatar, but she really didn't see the gender. She just saw that as her. And we had a great debate in the class because the guys were very interested. Well, why did you choose a male avatar? You know, you have some power issues or things like that just for fun, but also because they were very interested as to why. Also, when I give a male avatar, like I sign it, to a female student and I ask them, did that affect your experience? They say no. And same vice versa with my male students getting a female avatar. So these are very interesting issues and they are also bypassing any kind of separation between the avatar and you. Because it's a part of your personality that you're expressing. Like we were talking about in the panel, it can be used for good purposes, for people to sort out their differences or, you know, talk to each other. So I think that the connection is the strongest thing. And how does it happen and when does it happen I think really depends on people's personality, the amount of usage, the kind of usage. So in that I would say individual differences and subject variables or individual variables do play a part.
[00:16:54.870] Kent Bye: And is embodiment theory coming out of virtual reality research or is this something that's coming from the broader psychological field?
[00:17:01.164] Saadia Khan: It's coming from the broader psychological field. So it's coming from the field of cognitive psychology. So we had behaviorists and they basically looked at stimulus and response. Something happens to you and then you respond. And they were not interested in what happens within your mind or your brain and what cognitive processes are involved. So that school of thought was basically replaced in modern times with cognitive psychologists. That's where people like me fit in. where we're more concerned with, okay, there's stimulus, and then there are mental processes that go on, and then you respond. So it's very important to know what's happening, what the mental processes are. So embodied cognition is a part of that, which is again, as you can see, bringing in neuroscience, looking at how your brain works, what areas are lighting up, which are not, and also how it makes you feel. So you're looking at different kinds of multimodal elements, not just unimodal anymore. So emotions, vision, you're combining all of them together and seeing. And, you know, I don't want to lecture you about embodiment theory, but basically the concept of multimodal is that When you go through an experience, for example, you're asking me questions right now. So you're taking in all this information. So you're hearing what I'm saying. You're looking at me. You're seeing me. You're feeling the breeze, right? All your senses are at work. And what happens is you store all of this in your memory. And at a later time, for example, maybe when you have to put this up, or you have to tell somebody about it, or you're just thinking about it, you recall this experience. So when you recall this experience, it is all of these multimodal representations that you are reactivating. When you reactivate them, then you're actually feeling the breeze that helps you remember better. Or if you are more into visual, some people are, then you remember my hair, my face, my nose, anything, right? What I'm wearing, you remember that. Or maybe even the nice trees at the back. So this helps you remember things better. And it will also help you connect more with the experience that you're having, and therefore you will understand it better. So with embodiment and avatars, the same thing is happening. Because of this whole experience which involves multimodal elements, you will remember things better, you will understand them better. And because of that, sometimes certain elements will get activated and others won't. You will be able to transfer that learning to other areas where adaptability comes in.
[00:19:46.973] Kent Bye: I see. And so it sounds like you're creating these virtual worlds to be able to get students out of the classroom, to give them a unique context that they feel more immersed in, and that is somehow giving them more pegs in their memory to kind of grab onto and form memories, it sounds like.
[00:20:03.687] Saadia Khan: Right. And so therefore learning is not learning. Learning is real life. Right. That's exactly it. And you said it very well.
[00:20:12.313] Kent Bye: When you're putting people in an immersive environment, where are you putting them? And if you're teaching history, maybe some examples of what you've done that seems to work well.
[00:20:20.593] Saadia Khan: Okay. So, for example, when I was teaching history, as I told you, I took completely novel text. We tested everyone. They hadn't read it before. They weren't aware of the characters or anything. And a majority of them, a significant majority, reported that they absolutely hated history and they could never remember anything and they couldn't understand why. I think it also had to do with age again. As you grow older you realize that history is very important, but you know in my younger years I used to dislike history too, so I completely understood where they were coming from. So in that sense again, having an experience where you're putting yourself in another person's shoes, experiencing the emotions, and embodied emotions rather than just thinking, oh, he must have been happy, oh, he must have been devastated, but actually feeling those emotions. And how did I know that they were actually feeling those emotions other than they're reporting it? Because sometimes they might just be reporting it to please a teacher or an experimenter. is we asked them to write down, you know, what they were feeling, their experiences, or what they did, and in these reports or diaries that they wrote, they always referred to the main character, which was a real historical character, as me. I did this, I did that, I felt very sad because of this, I was jubilant, you know, so that connection was established. Another thing that I'm using these virtual worlds and avatars for is to teach psychology courses. So in these psychology courses, for example, I gave an example of problem solving. So problem solving theory is a theory which students find hard, it's dense, you know, you have a problem space, you have to visualize it. So if you're not good at visualizing, you sometimes have trouble with it. So, you know, you have a problem space, there's a start space, there's an end state, there's intermediate states, what operators do you choose? Now how would you possibly remember so much theory? So in comes embodiment, in comes you as an avatar who walks around in a space, is given an assignment and is also then told after doing that embodied assignment to actually examine the whole activity. and to see, well, how did it really parallel the theory that I was studying or I was reading about. And so that helps them too. And similarly, any theory that they've been reading, I've been using avatars and embodiment, and it helps them. And my whole point is I'm using psychology or educational psychology classes because that's what I teach. But this can be done in anything. So you can definitely do this in math. You can do this in English. You can do this in any domain. and this will help people learn.
[00:23:01.717] Kent Bye: Are there any examples of things that you experimented with that tended to not work or be detrimental to learning? And I'm just curious if there's some worst practices that people could be aware of, things to avoid.
[00:23:14.252] Saadia Khan: I actually didn't, and I'm not saying that just because I was doing this research, but the first thing that you learn as a researcher and as a responsible human being is to stay away from things that might harm people. That's unethical, could be psychotic. So I didn't really do anything that would harm them in any way. So I would say I didn't find anything.
[00:23:39.155] Kent Bye: What were some examples of things that would be harmful?
[00:23:42.457] Saadia Khan: Sure. So an example of something would be, for example, if I'm doing some research on emotions and I induce a negative emotion, which a subject or a participant or a student is not able to recover from. I don't, but if a memory triggers or something. So for example, I did some research in autobiographical memory, and I asked them to recall a happy memory. I made sure that I didn't ask them to recall a sad memory. But in research like this, although I didn't encounter it, you have to build it in a certain way to ensure that, you know, some depressed student or a student with problems does not end up recalling a happy memory and then it's in the past and then becomes sad. So what you have to do is debrief them afterwards. So debriefing would involve talking to them, explaining things, making sure they're okay. Some researchers just, you know, I think some of them are lazy. They just give students or learners or participants something sweet to eat, thinking, well, you know, this will elevate their mood. But if you're doing research which involves emotions, then you have to make sure that everybody's okay. So my research in emotions is about positive emotions. about smiling and being happy and you know so I haven't really come across that but I've stayed away from having them recall anything negative and if I did encounter that then I would take immediate action to help
[00:25:16.625] Kent Bye: Yeah, what that reminds me of is right now in the virtual reality community, it's a bit of the Wild West where people are creating virtual reality experiences that are very visceral. And there's no sort of disclosure as to whether or not there's going to be like some traumatic jump scare in these experiences. And so I'm just, you know, it feels like there's going to be a lot of ethical implications of virtual reality development and providing people these types of immersive experiences. So.
[00:25:41.532] Saadia Khan: Yeah, absolutely. And I agree with you. I mean, these are huge, huge ethical issues that need to be considered. And I think designers really need to pay attention to that. And the whole community needs to come together to establish some kind of guidelines. And also with younger audiences, there has to be education in terms of how much to use and how to use virtual immersive environments. And, you know, we've all heard of those cases in certain Asian countries, Japan, for example, where things get a little out of hand and, you know, they make people sick or ill. I think somebody died once or twice, right? So I think a lot needs to be done in this area. And I tend to really stick with the educational implications and education side of it, because I feel that if people really focus more on these constructive positive things, then the negative is automatically lessened in the sense that, you know, there's more focus on the good and less on the harmful aspects. And then with education will come the need to focus more on reducing the harmful effects, if any. Right? You know, like, they didn't have television in this area of Canada at one time. And then the Annenberg School did some research, they introduced, they didn't, but you know, television was introduced, and violence went up immediately. So the question was, hey, you know, all these violent shows on TV, did they do it? There's also research which shows if you watch too much television, they say that it probably affects you more. And same with video games and stuff. But then as I think Rich pointed out, and I agree with him, it's not necessarily the programming, it's also the individual. So not to say that the burden is on the individual, but if somebody has something going on, right, has a problem, or maybe doesn't, something happens in his or her life, which is a trigger event, and then this experience with technology coincides with it, right? So if you look at it from a psychological point of view, then the person might mention the technology or the blame might be put on the technology, whereas we really need to also study where the source is. So there's never really one source, right? I mean, people can pick up knives and use those as well. So, it's a very complicated issue and I feel that we need more research in it, we need more dialogue for sure. And, you know, constructive dialogue which results in, you know, some kind of getting together as a community and doing something about it.
[00:28:29.566] Kent Bye: And so with the Oculus Rift coming out and fully immersive virtual reality, you start to have things like Leap Motion and the Kinect cameras and Hydra and STEM controllers to be able to track body limbs and even data gloves are on the way. And so I'm curious if you started to look at what the implications are of being in a fully immersive first person perspective virtual reality and the effect of being in an avatar in that context.
[00:28:55.665] Saadia Khan: So I haven't compared the two because to compare the two conditions they would need to be comparable and at the moment they're not comparable because if you look at just immersive environments you know you're just looking at a computer screen at the moment but then you can also you know use projections and stuff and you make it into an environment which is, you know, less restrictive. But the theory that I was explaining to you, which is embodied cognition theory, basically, according to that, all kinds of these immersive technologies and their use would fall under embodied cognition. So it should work better than not using them. The only thing would be, as I mentioned before, for example, attentional factors. So how much of it do you use? you know, is there a goal? Or do you just leave students or learners in there and then see what happens? Because that's also a form of learning. You know, all learning is not really directed by somebody else. It could be self-directed. There's also stealth learning. So that's something that I would be very interested in. For example, 3D printing. Sure, somebody printed a gun, but not everybody would do that again. So there are lots of positive aspects to it as well. And something like 3D printing would definitely fall under embodied cognition as well. And everything else would too. Yeah.
[00:30:26.554] Kent Bye: And finally, what do you see as the ultimate potential for some of these virtual worlds and fully immersive technologies?
[00:30:34.154] Saadia Khan: Have you seen the Geminoid? Alright, the Geminoid is a cyborg, which is a real cyborg, created by this Japanese professor, and somehow he gets very little coverage in the US. But the Geminoid looks exactly like the professor, I forget his name, but if you look him up on the web, you'll find him. And he has started actually sending the Geminoid to deliver lectures when he can't go or maybe he doesn't feel like going. Ultimate dream, right? And he's also created more Geminoids. So some people find it a little freaky because when you... and do go and take a look at the Geminoid. It looks exactly like him. And perhaps in person you can still tell, but if you just see videos and stuff, you cannot tell if that's a real person or if it's really a cyborg. So I would say that anything and everything that we imagine And, you know, people write about the time machine came out in the last century, right? People going to the moon, although that's highly contested. You know, it will happen. Because when we think of something, it's the human condition and our brains can actually figure out and problem solve how to make it happen. So I would say anything and everything is possible and keeping in mind your question, the big question is how do we keep it all ethical and how do we keep it within bounds and have a balanced life when we indulge in all these new technologies and how do we use them positively.
[00:32:20.636] Kent Bye: Just a quick follow-up on the Geminoid is because you get the principle of the Uncanny Valley is once you get closer to looking more and more like a human and if it doesn't fully replicate the expressions of a human then it just becomes creepy and so I'm curious if there's a drop-off when you're looking at avatars that are crossing that threshold of the Uncanny Valley.
[00:32:42.496] Saadia Khan: I think with avatars that will not occur because there is always that feeling that they're in a virtual environment. And so we're constantly used to as kids, you know, we watch cartoons as adults, we do too. So, I mean, that does not have the same creep value as an actual person who kind of looks like you, but the Geminoid is very good. The Geminoid looks exactly like him. and he's controlled the facial muscles and everything. It is either creepy, people find them either creepy or they find them brilliant. I find it a little bit of both. You have to check it out.
[00:33:23.415] Kent Bye: Cool, well thanks.
[00:33:24.456] Saadia Khan: Sure, you're welcome.
