One of the recurring themes that was coming up at the IEEE VR academic conference was how VR is starting to catalyze interdisciplinary collaborations between different academic fields including computer science, social science, neuroscience, and cognitive psychology. I talked with Dr. Slyvia Xueni Pan at the IEEE VR 2017 conference after she had organized a panel on virtual social interactions with Dr Antonia Hamilton, Prof Anthony Steed, Dr Laura Fademrecht, Prof Jonathan Gratch, and Dr Marco Gillies.
Pan has specialized in generating expressive virtual characters in order to create an empathic social interaction in virtual immersive environments. She talks about how VR creates ecological validity and experimental control in order to research social interactions ranging from mimicry, gaze, racial bias, subconscious reactions to blushing, and exploring variations of the rubber hand illusion. She also talks about how VR is facilitating cross-disciplinary collaborations between computer scientists and social psychology researchers.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So virtual reality is going to be enabling all sorts of new interdisciplinary collaborations. At the IEEE VR in 2017, there was a panel on virtual social interactions where there were computer scientists and social scientists and neuroscientists all thinking about how virtual reality technologies could be able to do new empirical studies and research. So I had a chance to talk to the organizer of this panel, Dr. Shwini Pan. She's an assistant professor of virtual reality at Goldsmiths, University of London. So she talks about some of the different fields in social science where virtual reality is able to do new types of empirical research. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Shwini happened on Monday, March 20th, 2017 at the IEEE VR Conference in Los Angeles, California. So with that, let's go ahead and dive right in.
[00:01:11.391] Xueni Pan: So my name is Srinipan. I'm also known as Sylvia by many friends and researchers, collaborators of mine. I'm a lecturer in VR in Department of Computing of Goldsmiths in London. So I've been working in VR for the past 12 years. I started coming to the UK, did my master's degree in VR. in UCL and then followed by a PhD in the same area, and then a postdoc in morality and VR, and then I moved to social neuroscience and did two years in mimicry and VR until I got a lectureship at Goldsmiths University.
[00:01:45.034] Kent Bye: Great, so there was just a social interactions panel here at the IEEE VR, so maybe you could talk a bit about what you were talking about and presenting here.
[00:01:52.951] Xueni Pan: Yeah, so for me this panel really summarized what I've kind of been doing for the past 12 years. So basically I organized the panel, we had in total six people and I think half of us are psychologists, the other half are computer scientists, including myself being half-half and I guess Jonathan Grash, Professor Jonathan Grash from USC also being a half-half. So it was a really exciting opportunity to put these two groups together because I think virtual reality has been a very exciting research area recently but actually it has existed probably for a good 20-30 years in terms of research and the history of this conference. But I think very recently VR has really matured enough for us to be able to implement realistic and expressive virtual characters which are really useful in social interactions. And I think only recently we were really able to deliver some kind of experience in VR with social interactions that are really useful in all sorts of areas including training, education, therapy and also research for social science. And I think there are lots of exciting things that could happen between social scientists and researchers in VR.
[00:03:04.606] Kent Bye: Yeah, and I'm curious to hear these different perspectives of both the computer science perspective, but also the psychologist perspective. What are some of the open questions that you're actively investigating right now from both of those perspectives, the computer science side, but also the psychological side?
[00:03:18.994] Xueni Pan: So I guess one of the problems computer scientists have is that we want to be able to use VR in training and education. So we want to model expressive virtual characters who react towards the user realistically. But the problem we have is that we don't actually understand how social interaction work in real life and as I said in the panel in the beginning because a lot of the interaction we have actually subconscious. I don't plan my action when I'm talking to you and you know as you're doing this interview we're doing it face to face and that's incredibly important because I can see how much you understood what I say and this is not the kind of same thing we would have if we have a sort of audio interview but again psychologists don't exactly understand how this social interaction actually happen to the sort of extent, after this interaction, we will have some kind of impression of the other person, but we can't actually explain. So even psychologists can't explain, but they have some kind of theory, they have lots of experience in trying to explain this, and that is incredibly valuable for computer scientists. And at the same time, computer scientists are also becoming, because of this power we have in modeling realistic virtual characters, animating them and programming them to be interactive also became very useful for psychologists because they realized that they can use virtual characters in kind of social interaction experiments where you have both ecological validity and experimental control, which are the two things they're obsessed about. So this is not a new concept, but I think Now it really becomes one of the very important concepts in both social psychology and VR research.
[00:04:56.583] Kent Bye: And I've often heard people talk about the percentage of how much communication happens non-verbally. And there's a certain amount of content that we're saying in these abstracted languages. And yet, there's all these other subtle things that, as we're perceiving people in an embodied experience, whether it's face-to-face or whether or not you have different degrees of embodiment and fidelity of emotional expression, whether it's your head gaze, your hand movements, your eye gaze, your facial movements eventually. So that's going to be getting more and more higher fidelity. I'm curious to like how you make sense and quantify or describe with some sort of framework this nonverbal body language as an element of communication.
[00:05:35.057] Xueni Pan: Yeah, I mean we're still kind of at the infancy of this study because as you know this is interdisciplinary and it requires people from two different, very different areas to talk to each other. It's incredibly hard. But maybe I can sort of give a small example of, this is going to be a very simple example because that's what got me interested in doing research in this area. became interested in studying blushing like when we sort of embarrassed our face turns darker that's the understanding of the time but nobody really understand why do we blush and Darwin even said blushing is the most human of all expressions but nobody really had a clue of what exactly social function does blushing serve because you can't actually study it with an actor Because if we get an actor to come in and tell them to blush, they can't. But then with virtual reality, it's really easy to control the blush on a virtual character. I can program the virtual character to be doing something else, but then you're testing the participant's reaction towards their blushing. So that's one of the very first work I did. And again, it's just the really beginning of this sort of big journey of that. And what we found is that although people don't realize the avatar is blushing, but they give higher ratings to the avatar who is blushing. So it's really proved that it is something that happens at a subconscious level. And there are many other things that recently we've been starting to do is we try to study social mimicry, like how we kind of automatically copy each other in conversations and social interactions. And again, this is something otherwise really difficult to study, but in VR you can program the virtual characters to copy the head movement, for instance, of the avatar. I guess it's not something incredibly new, but it's something that we can do really well now with the technology we have available. And this is something really, really important for social psychologists to know because that is linked to the missing puzzle for people who have certain kind of neurological disorder like autism. So in order to understand and help those people really to understand how we mimic each other, and if they are mimicking each other to the same extent as we do, and if they're not, why?
[00:07:40.292] Kent Bye: What type of questions and answers were you able to find with the process of adding mimicry within a virtual environment with virtual characters?
[00:07:49.195] Xueni Pan: So for instance, again, I'm not the expert that was in the room here just now. Antonia Hamilton, I think she has a paper talking about the difference between mimicry and emulation. So mimicry is like, when I copy someone, I would copy not only what I'm trying to copy, but also everything else that comes with it, right? So for instance, when I learned English, when I was at school, the teacher would pronounce a word, I would try to copy that word, but I would also inevitably copy the accent of how she said it, and maybe even the facial expression she had when she was saying that word, right? Emulation is basically just copying an action without copying everything else that comes with it. So you copy the goal rather than the kinematics of a movement for instance. And the theory is that people with autism do more immunation than mimicry and normal people will do more mimicry than emulation. And in VR we can basically study that by getting both types of participants to come in and get them to copy the movement of an avatar and get them to repeat it a hundred times so we can get something actually meaningful. Which is incredibly difficult to do with real actors or video stimuli because they're not as interactive real actors They're too random. You can't control exactly what they do
[00:09:02.508] Kent Bye: Yeah, one of the things that I've also been really interested in is the emotional and affective dimension of social interactions. I know that because you're in an embodied experience and you're able to have a full expression of your body language, and eventually with facial expression and eye tracking, I think that it's going to start to communicate more higher levels of emotional information to each other. And yet, I see that there's different social presence theory models that have a whole dimension of how emotionally connected you are to the characters. And so, from your perspective, how do you start to think about or create a framework around emotion when it comes to these social interactions?
[00:09:39.492] Xueni Pan: I try to stay away from emotion because they're too high-level and complicated. And by the way, the professor in embodiment is in this room, Professor Mel Slater, who was my PhD supervisor. You probably should definitely interview him. And so, yeah, I try to stay away from emotion because my experience in social neuroscience is that we're better off, because we're still in the infancy of this venture, so you're better off studying the most basic kind of low-level brain reaction. rather than going something that you have so many different factors that could contribute to the emotion you feel. Your personality, your experience, whether you're tired, whether you're hungry, you know, all these things could sort of influence on your emotional reaction. So it's better to take a step back and just sort of see The more basic instinct reaction you have. So that's my yes. I'm not an expert in motion Some people might disagree with me. That's fine.
[00:10:33.389] Kent Bye: But you know, I stay away from it as much as I can Well, I think because it is hard to quantify I could understand why you might want to stay away from it however, I think from a perspective of presence and social interaction, I feel like it's a An important component that, you know, I think it's a part of the holistic way of looking at it. But if you're not looking at presence, or are you looking at social presence, or what are the certain things or the goals of maybe the gold standard of social interaction? Like, what is the function that you're really looking at with the research that you're doing?
[00:11:03.518] Xueni Pan: Yes, so there are two types of research I do. I do either use VR to study social interaction, which basically I work with neuroscientists, social neuroscientists, and to understand what their research question is and try to use, help them to use virtual characters to answer their question. So the kind of research theory is pretty much from them. And another type of study I do is basically I try to use VR in training, education, and therapy. And for that I need to make sure I create a situation that really generates presence, or social presence in that sense, and I use Mel Slater's theory in plausibility illusion and place illusion. So that's kind of the framework I'm currently using.
[00:11:42.779] Kent Bye: For you, what are some of the biggest open questions that are really driving your research forward right now?
[00:11:47.922] Xueni Pan: How to get computer scientists and psychologists to work together, and that's one, because the culture difference, for instance, I feel there are huge amount of expertise in both areas, but people don't speak the same language. because I try to do a bit of both myself and I found that incredibly challenging. Sometimes I want to create social interactions that are really real and emotional, but then when I started doing programming, I found it's almost like I have a switch in my brain that I have to think like a machine and I lose all the kind of emotion that I'm trying to express so it's really hard. So maybe like one of the interesting things would really to see that in the future we can have students who take both modules from quite early young age so that they will be able to have expertise in both areas so they won't have the kind of struggle I have now which is trying to understand lots of frameworks in psychology and I really struggle in that because I didn't go through the proper training for that.
[00:12:46.166] Kent Bye: And so when you're working with virtual humans, I'm curious if you ever have a live puppeteer driven by an actual human, or if you're coding these humans in certain ways. Because one of the things I've just anecdotally noticed in watching some of the videos of these virtual humans is that they feel very robotic. They don't feel like they're lifelike. And they kind of feel uncanny in that way. So I'm just curious if you've looked at also a live embodiment of these virtual humans.
[00:13:12.653] Xueni Pan: So there are sort of two different virtual characters avatars or agents, but normally what we do is something in the middle So one of the paradigm I've been using is called visa of odds So basically I record a little VR sort of animation motion captured clips with sort of sentences as well so I click a button and avatar would say something with the body and movement that really goes with this thing they're saying because we really use our body a lot when we talk and if it's not really synchronized it doesn't quite work. So that's kind of one layer and then we normally have an experimenter kind of pressing buttons to trigger these kind of interactions. But at the same time, they're another layer running, which is basically kind of the agent part of this avatar. So they can be sort of, while they're triggered different sentences, they can be looking at a participant, you know, and this is kind of driven by an algorithm, they can have different percentages of how much should I be looking at you, because when we're talking, we're not looking at each other 100%. And you might want to have some kind of variables in that depending on whatever measure you're taking from the participant. So yeah, again, this is really sort of tricky when you design a study, how much percentage do you want each method to use?
[00:14:28.125] Kent Bye: Yeah, and because you are having people from both the neuroscience and computer science background on this panel today, I'm just curious if you have any examples of where you see that collaboration really working really well, where they're kind of on the same page, or also sort of areas where they both have blind spots in terms of what they don't know yet.
[00:14:44.308] Xueni Pan: Yeah, I think they totally want to work with each other because they see really the point. Social scientists now, I mean, probably since about three years ago, before then, when I go to see a psychologist and ask them, do you want to use VR? They were just like, no, nothing is working. But now they really see the advantage of using VR and they realize they can actually afford I don't mind spending 2,000 pounds, even if it doesn't work, it doesn't matter. But it used to be it has to be 200,000 pounds. And that's really changed their willingness to work with VR. And obviously there is a big advantage in terms of ecological validity and experimental control. So they're now really willing to work with us. As a computer scientist, I also see that people now more or less solve the most fundamental problems in VR in terms of computer graphics, rendering, delay, frame rate, illumination, realism, we can also interact with the system to a natural realistic way, not 100%, but you know, we made little steps. And lots of people realize actually the next thing we want to do is to be able to interact with another person in this environment, an agent or an avatar. And then when they start doing it, they realize nothing is working. They realize there's so many things they don't know, and they ended up implementing all these characters you just mentioned. They just look like a zombie. And that's when they turn towards social psychologists. And there are obviously problems in this area. So first, people don't speak the same language. They often use the same concept, which describe really different things. So for instance, when psychologists are talking about perspective, perspective taking, they're really talking about there's a whole lot of theory behind what perspective taking really mean, what is theory of mind, and all that. And everybody kind of studied all that. But then when you talk about perspective taking with a computer scientist, they start thinking about, oh, first person shooter game. So there's some overlap, but the language is incredibly different. And secondly, lots of people in VR are now trying to start running user studies, and they try to have some experimental design and statistics in their paper. But I've reviewed quite a lot of papers, and lots of them are a disaster. Yes, I actually said today that I reviewed someone used a paired t-test for between-group study. I know the psychologist is just like, oh, but then all the computer scientists, what's the problem? So you see, I don't need to explain what it is, but it's just like there's a computer scientist, if they wanted to use the studies, if you want to do social interaction, they really need to talk to psychologists or read books in experimental design and statistics. I'm speaking as a computer scientist, so that's what I think we can do.
[00:17:24.170] Kent Bye: And I'm curious to hear, since you said that you have these neuroscientists and psychologists that are starting to maybe start to use virtual reality in their own use cases, I'm curious to hear some of those early examples and use cases that you've seen in terms of these psychologists starting to adopt this consumer VR technology and start to use it in their research.
[00:17:44.304] Xueni Pan: Well, I kind of gave one example on mimicry, and another one, blushing, is my own work, but it's not from psychologists. To do that work, I read lots of tons of psychology paper. Oh yeah, there are some psychologists trying to study gaze, so they put people in an fMRI scanner, and then they have avatars, either looking at them or not looking at them, and they scan their brain. And there are probably also psychologists trying to study, not using VR, but they are looking at using VR now to study racial bias. So there are already some study in VR has been done to study this. So psychologists now are also interested because in VR you can be somebody else. You can be someone from a different ethnic group and that's really an experience you won't be able to have in real life. And psychologists has been trying to use a rubber hand illusion to recreate this illusion. But in VR it's so much easier to do that and you create a much stronger illusion than what psychologists can do by just putting a arm in front of you and make you think that's your arm, right? In VR you can actually move your arm and seeing another arm moving exactly the same as your arms, you really have a really strong illusion that this other arm in VR is my arm, and then through that, you can then become somebody else.
[00:18:51.189] Kent Bye: Yeah, that virtual body ownership illusion, I think that Mel has worked with a lot.
[00:18:55.665] Xueni Pan: The thing you said about the fMRI usually you're immobilized right so you're not able to move around and so Yeah, and so being able to use VR to actually give them visual stimulus I'm not a neuroscientist But I think they were basically looking at ways to have some kind of tracking system in the fMRI machine because in that machine you can't have any metal in it. But there are some kind of computer vision based tracking system you can put in there which means that it is possible if there's enough demand from all areas then you should be able to maybe move your hand in a certain way to have the illusion and then have the brain scan where you have your head kind of fixed in a fixed position, so. Yeah, sorry, can I just add one more thing about fMRI study? Because there's also another way of measuring brain activity, or many different ways of measuring brain activity, like EEG and fNIRS. They're both similar to putting things on your head and you don't need to go into that monster machine. So, yeah, obviously VR probably will work with these mechanisms rather than fMRI.
[00:19:54.285] Kent Bye: Interesting. Yeah. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:20:02.668] Xueni Pan: That's a very interesting question. I think for me VR is something that makes things that are impossible possible. So I'm not so much in there. Lots of talk mentioning about VR in the area of entertainment, but that's not my interest. I'm really interested in using VR in training and therapy so that we can make people who otherwise just lock themselves at home won't be able to sort of get out there have a normal life so we can use VR to help them to overcome their psychological issues and just for them to like enjoy themselves like everybody else so that's my hope and also another sort of lots of people are doing this as well with 360 degree video is using VR to create understanding of somebody else's perspective. So for instance, I did some work in creating virtual environment, which represents the sort of visual perception of someone with autism. Because we come into this room, we instantly look at the people in the room, because that's what our brain is basically doing automatically. But people with autism, they come into this room, they are probably going to be very overwhelmed by the carpet and the wallpaper. And for them, that's a lot more interesting than the people in this room. So they will behave in a weird way to our eyes. But actually, they might be a really amazing experience they're having. So I think it's interesting for us to also understand how they would experience in their world, so we can then accept they're no longer the odd person. They're just another different kind of personality. In order to understand that personality, we need to experience how it is, because it's more difficult to describe with language.
[00:21:41.882] Kent Bye: Awesome. Well, thank you so much.
[00:21:42.983] Xueni Pan: Thank you. Thank you very much.
[00:21:45.339] Kent Bye: So that was Dr. Sweeney Pan. She's one of the coordinators of the Virtual Social Interaction panel that happened at IEEE VR in 2017, and she's an assistant professor in virtual reality at Goldsmiths at the University of London. So I have a number of different takeaways about this interview is that, first of all, I did see that one of the trends at IEEE VR was how virtual reality is going to enable all sorts of new types of interdisciplinary collaborations. In this case, it's between computer scientists and social scientists and neuroscientists. And one of the challenges is just coming up with common language and that there's many different contexts that are out there. And within those different contexts of those different disciplines, there's going to be different languages and different ways of describing similar things, but they're going to have different meanings depending on that context. And I think that Through the lens of human experience, I think that VR is going to provide this mechanism by which all these different disciplines are going to start to be able to collaborate in new and different ways. And as you're able to create these virtual environments, and as they become more and more realistic, as they are able to have these virtual humans that are much more convincing, I think one of the big problems that I've seen, at least in the research realm, is that there's a certain amount of uncanniness when it comes to interacting with virtual humans within the virtuality context. Over time, we're going to get a lot better with trying to mimic all the very subtle nuances of body language and communication. One of the things that Sweeney said is that one of the things that social scientists have had a huge difficulty in trying to do within these empirical studies is deploying these experiments within an ecologically valid environment as well as having experimental control. I think that when it comes to social science and VR research, this is going to be an area where you're going to be able to essentially create a simulation that you can repeat over and over and have many people go within there. You know, one of the things that Sweeney said is that in terms of emotion, it's just something that she tries to avoid just because there's so many different variables and it changes day to day. And so it's really hard to control all the different elements. And I think that is going to be one of the things that I think is consistent amongst all psychological and social science research is that there are so many different variables when it comes to what we're as individuals bringing into the experience and what you can and cannot control. And so it's basically like, how do you come up with models to be able to take into account all these various different aspects of consciousness as a human experience? I think This is one of the most difficult problems in science and it's possible that over time there's going to be able to have different models and to be able to create these experiences within virtual reality to be able to both model human consciousness but also test it in different ways to be able to actually empirically test some of these different experiential design frameworks or models of operationalizing consciousness. I think over time, we're also going to see a lot more of EEGs and actually doing this neural correlates of what's happening within your body and being able to correlate that into subjective phenomenal experience within a virtual reality environment. And so whether that is being able to control blushing and virtual avatars, whereas you can't really do that within an actor to have them blush on command, you're able to actually have these virtual avatars that are blushing. And it's at the subconscious unconscious level. So it's things that they're not able to actually articulate why they are feeling these certain ways towards the different characters, but there's these subconscious and unconscious cues that we have. And I think that within these virtual reality technologies, they're able to potentially recreate some of these things that you wouldn't be able to do with a real human within the context of these virtual environments, but also to potentially have new ways of drawing new neuro-phenomenological correlates into what's actually happening at the perceptional level, at what is essentially kind of like this sub-symbolic level that we can't even articulate or put words to. It's just part of our human perception that is happening below our level of conscious awareness. And so they're trying to figure out all these different ways to be able to either experimentally control and test at these different subconscious processes. But I think over time, there's going to be some really interesting research opportunities when it comes to trying to blend these two aspects of social science research and neuroscience, as well as in the context of these virtual reality environments. whether it's through blushing like Sweeney Pan is doing, through mimicry, through eye gaze, through racial bias, through the rubber hand illusion. These are just some of the early social science types of experiments that have already been doing within these virtual environments. And I think over time, we're going to see a lot more, especially if feeling like you're actually there, like you have this place illusion and plausibility illusion that you're able to be in these virtual environments and then start to do these types of ecologically valid environments that are able to have some sort of experimental control that can be standardized over time. Now, I think one of the other things that comes up for me is that I think there's a difference between you interacting with a real human and you interacting with an AI avatar within a virtual environment. I think There's still a little bit of uncanniness. There's a lot of different social cues. I think that most people can really start to tell whether or not they're interacting with virtual humans or real humans. But I think in specific contexts, I've seen, especially like in medical training, for example, even if they are repeating these stock lines, there may be situations in a training context where it doesn't really necessarily matter that you fully believe that who you're interacting with is a real human or a virtual human, especially if the context of what you're trying to do is a training scenario where you're just needing to have lines be fed to you by a team environment where there's other virtual people there, and you have to actually make a decision and take action, and it's more about you knowing the information and being able to properly respond to the conversation than for you to believe that you're actually engaging within a social interaction and having a deep emotional connection with one of these virtual humans. but I think this is one of the things to look out for in the future is the trade-offs between having a Wizard of Oz human interactor that is embodying and puppeteering some of these virtual humans versus having to rely upon something like AI or some of these branching narratives and what I've seen a lot is sometimes a virtual interactor or Wizard of Oz being able to interpret the different social interactions and being able to perhaps trigger different branching narratives of these simulations, especially if the AI isn't at the point of being able to pick up all the different natural language input and conversations and dialogues and to have a human be able to do that interpretive aspect, but then to also be controlling different aspects of the simulation. So I think that in terms of the ecological validity and experimental control when it comes to social science research, I think that has been typically one of the things that has been very difficult to do. these types of standardized research, especially when you're hiring actors and you have these sort of variants when it comes to the types of experiences that are happening for people. So I think this is an exciting area to see what type of more interdisciplinary collaborations are going to come out of virtual reality with social science as well as with neuroscience. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do consider becoming a member to the Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So you can donate today at patreon.com slash voices of VR. Thanks for listening
