Play episode

#596: Researching Empathy in VR with Cognitive Science & Phenomenology

November 20, 2017by kentbye

lynda_joy_gerryCognitive scientist and phenomenologist Lynda Joy Gerry got into virtual reality after seeing the body swap experiments by Machine to Be Another as well as Shawn Gallager’s A Neurophenomenology of Awe and Wonder research. Gerry’s master’s thesis was on empathy in VR where she did a survey of the established theories on empathy from cognitive science, social science, phenomenology, and virtual reality. I had a chance to talk with Gerry at the IEEE VR academic conference in March where she was presenting her research on empathy in VR, and advocating for a more holistic framework for empathy research that bridges the objective theories of empathy from cognitive science with the more subjective, intersubjective, and experiential perspectives from the philosophical branch of phenomenology.

LISTEN TO THE VOICES OF VR PODCAST

SHOW NOTES

This is a listener supported podcast, considering making a donation to the Voices of VR Podcast Patreon

Music: Fatality

Share this
FacebookXRedditEmail

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So at the IEEE VR Academic Conference this year, one of the major takeaways that I had was that VR is being used to facilitate all sorts of different interdisciplinary research. And one of the people that I had a chance to talk to was Linda Joy Gary. At the time she was a graduate student at the University of Copenhagen and she was looking at empathy and how you could use virtual reality to be able to have embodied simulations and perspective taking in the body of another. And so Linda was actually combining cognitive science and phenomenology, and she was trying to bring together the objective and the subjective in the process of looking specifically at the issue of empathy. And in her master's thesis, she does a great job of just recounting all the different theories that are out there from all the different perspectives, and then also virtual reality research that both Mel Slater and Jeremy Balanson have been doing into empathy, as well as the direct experience perspectives of the phenomenologists. So this interview that I did with Linda actually got me inspired to be able to dive deeper into the different philosophers of phenomenology because You know, her take was that there is a certain amount of perspective that you get from reductive science, but that you need to have some way of combining the subjective perspective with the objective perspective. And I think that combination of cognitive neuroscience with phenomenology, I think is this really interesting mix to be able to see how they're showing different perspectives. So we're going to cover all of that and more in today's episode of the Voices of VR podcast. So this interview with Linda happened on Wednesday, March 22nd, 2017 at the IEEE VR conference in Los Angeles, California. So with that, let's go ahead and dive right in.

[00:01:59.309] Linda Joy Gerry: Hi, I'm Linda Joy Gary. I'm at University of Copenhagen. I work with the Center for Subjectivity Research and the Multisensory Experience Lab at Aalborg University Copenhagen, as well as some other labs. But I work specifically with sense of self, self-awareness, agency and empathy, and ways that virtual and augmented realities can actually change our mode of access to one another as a communication tool.

[00:02:25.835] Kent Bye: Great. Yeah. So this sort of tension between agency and interactivity versus sort of empathy and storytelling, I think I've seen come up a lot in interactive games as well as people who are telling stories. So for you, maybe you could talk a little bit about the experiment that you did and the talk that you were giving here at IEEE VR.

[00:02:44.609] Linda Joy Gerry: Yeah, so I conducted an experiment where I'm actually having a person step inside the shoes of a painter, seeing from the painter's point of view while she's painting on a canvas, she's painting a face, a portrait. And the painter is wearing binaural microphones, so you can actually hear from her point of view as well. And then the user is seeing a tracked rendering of their own hands using leap motion as a layer over the video. It's sort of a semi-transparent hand. And then we later added tracking for a 3D mesh of the paintbrush. So it's sort of an extended of the leap hand. And essentially what they're doing is they're listening to the painter tell a story about her creative process while following her movements synchronously and seeing from her point of view So I was trying to explore in practice applied concepts that are talked about in theories of empathy. So we have theory of mind which says that perspective taking is really critical and that empathy is really about sort of imaginatively projecting ourselves in the shoes of another person. we finally have a technology that basically allows us to directly do that. So I use stereoscopic filming in a pre-recorded virtual environment, but you can also do the same thing in a live stream. I have reasons that I went with a pre-recorded versus live stream, but we're now kind of playing around to see if we can do this in a live stream. So yeah, it's using perspective taking and then also sort of aspects of motor resonance related to kinesthetic empathy. There's something about moving synchronously with someone that kind of allows us to maybe get a better sense of their experience. But the literature on simulation theory, I think, is not exhaustive in terms of how this works. And I'm trying to get a sense of that more richly and deeply. And then narrative practice theory, which says that so much of our ability to understand our own experience and our own historically situated being and the experiences of others is actually through telling stories and narrative in an almost literal linguistic sense.

[00:04:45.181] Kent Bye: Yeah, and so, most of the times when I think about empathy or, you know, the different experiences that are out there in terms of, you know, what would be typically thought of as an empathy VR piece, like Gabbro Aurora's Waves of Grace or Clouds of Residra, where you're starting to actually see the other person and they're talking, and so you get to see the facial expressions, you get to feel the emotions, and, you know, one of the things that Gabbro said is that the thing that makes it such a compelling transfer of empathy is that story, and so, What he was trying to do is show them doing ordinary things so you could relate them to when they're eating or, you know, helping their children get ready for school. But I'm wondering if this perspective taking is, you know, from the first or third person perspective, you know, because, you know, as you're listening to somebody tell a story and you're seeing their face and their facial expression, it's sort of triggering those mirror neurons, which is like really triggering those empathy centers of our brain. So I'm just kind of curious if you're going into the first person, if you're still getting that type of transfer.

[00:05:42.019] Linda Joy Gerry: A really excellent question because from the first person you're also lacking a lot of information, right? I mean, I think what I love most as a frame for empathy is interactive second-person conversation where there can be this mutuality, reciprocity, coordination. I think all of those elements are really crucial for our ability to react and adapt. And I think that most of our experiences with others are within this interactive context. I mean, obviously technology is changing that so we don't have all of those layers of interactivity possibilities for information transfer there. But there's something really unique about the second person perspective that I think most theories of empathy leave out. And in a way, I think my project is almost kind of criticizing those theories because it's saying that if we try to assume that empathy is really about understanding the first person experience of someone else, then really that's missing a lot of the value of interpersonal transfer within a second person type of situation. Yeah. And then the other option is sort of this third person observation. You know, I watch someone in the world and I kind of get a sense of what they might be thinking or feeling, but there's no possibility for interaction, which I think is best applied to sort of characters in a story or in a movie or something like that. You know, I mean, of course you can always watch someone from a distance and maybe they don't notice you or something like that. But I would say, yeah, the highest efficacy for empathy is probably situated within second person interaction.

[00:07:06.951] Kent Bye: So I've heard about first-person perspective, that you're seeing the world from your perspective. Third-person, you're kind of maybe a disembodied entity looking down on your body. What does the second-person perspective look like in VR?

[00:07:19.368] Linda Joy Gerry: That's a good question and I don't know that anybody's actually ever done anything like that. It's something I've been sort of thinking about. So when I think second-person in terms of filmic experience, the first thing that I think of is Errol Morris, who ironically did a television series that's called First Person. It's excellent. And what he does is he uses a filming technique that's called the Enteratron. And I don't know the exact technical details for how it works, but it basically projects a video onto the lens of the camera of his own face while he's interviewing his subjects. And it creates this really eerie effect where subjects are basically like looking him into the eyes, but when you as an audience member watch it, you have this sort of you relationship, right? And that's what second person is all about. So you feel like the audience member is talking to you. So what I would like to do in VR, and one of the things I'm sort of exploring as an application for social cognition skills training for autistic adolescents probably, would be to use first person. I want to use foveated imaging so that they have a pre-rendered video first personally with image foveation showing what a person would be looking at during a social interaction. So it's sort of these gaze cues that direct their attention while they're in conversation with someone else. And then somehow also use like something like the Interitron camera filming technique so that the audience member would also be looking them or it would seem like they were looking them in the eyes as a way to help these individuals become more comfortable and understand how social interaction would occur for someone who was unafflicted by some of the issues that they have in social cognition.

[00:08:47.078] Kent Bye: Yeah, and maybe before we start to dive into more of what you found, I'm curious to hear a little bit more context in your background of where you're coming from. It sounds like you're coming more from a psychological trajectory than computer science.

[00:08:58.223] Linda Joy Gerry: Yeah, absolutely. So my background's in psychology and neuroscience for my undergraduate. And right now I'm in a program that's technically called Cognition and Communication. So it has this cognitive science and media technology, communicative technology. interdisciplinary mix and then within that I've been long obsessed with phenomenology so I've basically just taken every single phenomenology course that I could and I've been really pursuing that trajectory and I'm going to be pursuing a PhD in cognitive science and phenomenology.

[00:09:24.895] Kent Bye: Yeah, so it sounds like there's the objective and the subjective there, but with this sort of cognitive science, you're able to get objective data from the neuroscience and being the major brain data. But yet the phenomenology is a lot about your subjective personal experience. Is that right?

[00:09:38.619] Linda Joy Gerry: Yeah, absolutely. So there's a research program that was started by my hero, Francisco Varela, that's called Neurophenomenology. And essentially, this is getting at one of the main issues with cognitive neuroscience specifically, which is that we have all this objective data about neural activity, and we're trying to correlate this to like a battery of cognitive tasks, but we don't ever really ask the subjects what their first-person experience is, and it's a data access point that we're missing. And so it just seems like the analysis doesn't go all the way down to what it can, and usually this is conceived as being like the hard problem in cognitive neuroscience, but I actually think that it's not that hard of a problem. I've studied with Claire Petit-Magin, in Paris using the elicitation interview, which is a micro-phenomenological interview technique to get people to attenuate to their experience in a very detailed manner. And what Clara said, and I agree with this, is that it's not that we don't have the ability talk about our experience. That's not the issue. We can actually do that quite well. The issue is that we don't have neuroimaging techniques that are advanced enough to be at a micro enough level to be congruent with what we're observing on a micro level with experiential awareness and attention. But that's very controversial. I mean a lot of people think that like all self-report is biased and it's not very good data and all these things, but I think that in order to really move forward in neuroscience we're going to need this type of approach.

[00:10:59.955] Kent Bye: Yeah, well I totally agree with that, and actually I've had quite a number of different discussions with people here. Just as an anecdote and example, I've done over 200 hours worth of first-person anecdotes of stories about people having experiences within VR, and a lot of that is from that phenomenological perspective of their direct experience, and they're telling stories about that. And yet I talked to the academics here at IEEE VR, and some of them who listened to my podcast, and they told me, you know, we can't cite your podcast in any of our work. It's not peer-reviewed. And I was just sort of like, it just made me realize how much of the philosophy of science is sort of stuck in this kind of objectified, peer-reviewed approach, and yet we have these silos and such, so you have these things where you have the neuroscientists doing their thing, you have the phenomenologists and the game designers and the people who are storytellers and you know people are actually looking at things holistically and so That's sort of the thing that I'm sort of running into is that there's you know, just as an example I just had an interview with David Engelman who's this, you know big neuroscientist and you know, I asked him about you know What he thought about embodied cognition about the body and he said well here's my metaphor like the brain is a city and the body is just a suburb and And I was like, wow, okay, like that is really interesting because I think it could actually be the opposite. Yet, you know, from the objective data of what he's looking at from within his worldview, it's hard for me to argue that. But I see holistically that there is this other side of our subjective phenomenological experience, but yet it's sort of devalued.

[00:12:30.520] Linda Joy Gerry: Yeah, I mean, I have a few things to say in response to that. So I'm just going to go with the first, which is the notion of the barrier between objective, evidence driven, empirical research in the sciences, which is still kind of suffering from a notion of, I think, inter-theoretical reduction, you know, scientists still kind of want what they're talking about to somehow be as reducible and simple as the laws of physics and a lot of those sort of equations. And a lot of times I think it can't be. Or maybe that doesn't make sense to try to, you know, sort of put it in those terms. That has a lot to do with like the history of science and logical positivism and things like that. I have learned to be very patient with research in cognitive science and realize that there's value to all of these views and that embodied cognition isn't so much a divergent, you know, sort of third wave paradigm shift, but it's adding things to this perspective, and they're all important. But what I was going to say is that Husserl had this notion of, and Varela, Thompson, and Roche write about this in The Embodied Mind, they call it objectivity in parentheses. But the idea is that we can never really be truly objective, right? Because, you know, our own experience is always like an aspect of what we're studying. But we can do what they call objectivity in parentheses. which is sort of like having a relationship to an object with this like kind of noble goal of objectivity. But Husserl's notion of intersubjective verifiability, I think, is actually closer to what science is actually doing, which is saying, OK, I'm going to give you a design and a setup, and I'm going to get your first-person bodily orientation and then tools and measurements to be more or less the same so we can reproduce and find the same phenomenon in the world or in our experience or in some combination thereof. And I think that's really key. Again, I think the technology is actually kind of playing on our ability to do that, or at least that's the aspect of the technology that I get the most excited about. I don't care about reproducing reality. That's boring to me. I feel like any reproduction is always going to be kind of lame in comparison to reality itself, because experience in the world, I think, is already pretty great. But what got me interested in VR is something I call virtual alterity, which I think is a term that nobody else has ever used, but is the ability to sort of step into modes of experience that are varieties of experience as experienced by or for someone else in the world. And I think I'm surprised about David Eagleman's comment because I see him as actually doing a step in that direction. With sensory substitution, you're actually able to change your a priori sensory mode of interaction and re-pattern it to interpret stimuli in a new way. I mean, he's not directly doing this, but like you could do something to allow a person to have the type of sort of haptic, tactile attunement and awareness as a blind person, for instance, if you give them the appropriate training. And it doesn't have to use VR, but it can. And I think VR is a great tool for it or augmented reality. So, yeah.

[00:15:34.072] Kent Bye: Yeah, and one of the other things that I've been really looking back to the ancients and looking at neoplatonic thinking and just using the four elements as primary metaphors to be able to describe these different dimensions of presence. And if we look at embodied cognition, for example, and the work of Lakoff and Johnson talking about the importance of metaphor and being able to have like these primary metaphors to be able to understand these complex concepts and that in some ways there's this interdisciplinary issue which is that there's different disciplines that have different language and they describe essentially the same thing but they use different words to describe it and so it's very difficult to have this kind of transfer learning language to be able to interact with people so I what I am sort of putting forth is like okay let's go back to the ancients and look at what Plato was saying with these four elements and say as a holistic model of all four of these elements are in everything all the time, so you have these elemental balances, but you have the earth element that's embodied cognition, and the virtual body ownership illusion, and the sense of self, and these body schemas, all these other different parts of your body representation within a VR experience. And then the fire is agency, interactivity, your willful presence and how you're actually interacting with the experience. And then the air is a lot of the mental, social, cognitive dimensions. It's anything that's abstracted but also communication and being able to speak and language and being able to objectify things. And then the water element is the emotion. And that's the being able to actually be fully engaged in your emotions and all the things that are emotionally engaging you. So whether that's music or narrative and story. And so, you know, if you look at all of these parts of all these elements, that they're all happening all at the same time, but yet when I find myself talking to presence researchers like Mel Slater, like, okay, well, how does emotion fit into your theory of presence? And he's like, it doesn't fit in. And I'm like, then that's not a holistic theory of presence. So I'm going to sort of come up with a higher meta framework that incorporates these all four elements. So I'm curious to hear your thoughts on this kind of framework, but also specifically emotion, because it's something that I feel like is the thing that I have the most trouble telling the story of why the motions might be as equal part of all these other dimensions.

[00:17:46.509] Linda Joy Gerry: Well, first, I'll just say that there's a quote that all modern philosophy is just footnotes to Plato, and I think that that's fairly accurate. Plato was extremely prolific and ahead of his time, and I'm constantly impressed by some of his insights. A little bit about just like language, sort of transcription, fluency between disciplines. I think that people are a little too hung up on that. They're sort of, I'm using this term and I'm defining it in this way and you're defining it in another way, therefore we can't talk to each other or something like that. And it's an issue, I mean philosophy is such a linguistic discipline in general, right? So much of your argument is in language and through language, with language. It's your tool. And I see a lot of philosophers, you know, sort of criticizing one another about their interpretation of another author. And it's just, I think that's, it's important, but also ultimately we just have to reach some sort of consensus and some sort of agreed understanding about the object at play. And I think that it's a huge issue for the mind sciences as they emerge. And I would even say that your model and the notion of emotions is really a central, this more holistic model of really understanding what the mind is. and all the relations in our existence that occur as part of what we conceive of as mind, right? I mean it's really interesting to me that these theories of cognition are increasingly kind of getting more and more further from the brain, right? So there was first like embodiment, so we're embodied beings and then we're embedded in a world and then there are these notions of ecological psychology and ecological being and then you get situated cognition, you know, with others. You have distributed cognition and understanding of sort of objects in space. And it's sort of like at some point, you know, everything's cognition. That's a bit extreme. That's not, I don't mean to go in that direction, but I think that one of the reasons why these theories of cognition are becoming more and more sort of branched out and saying that all of the, you know, the remote aspects of our being are part of our cognitive system is really important. I mean, and there have even been studies, I'm sure you're aware of this, on sort of how the functioning of bacteria in the stomach actually influences like that connection through our brain into our nervous system into our cognitive being in the world. In Western science, and medicine specifically, it's very reductionistic, so we only study things on a system-by-system level. That's why I worked in a neuroimmunology lab with multiple sclerosis studies, and they were just starting to do neuroimmunology, because that disease is, for some reason, immune cells are getting through the blood-brain barrier and causing a lot of problems for these patients. I was just surprised though that like after I mean neuroscience is a relatively young discipline but still like it's just like we've just gotten to understand the interaction between these two systems but what about all the other systems of the body? And I think that neuroscientists tend to be a little bit too confined in their view of just the nervous system one independently of other systems which has to do with sort of holistic notions of health in the body, but also the notion of like cognition as just being something that's sort of behavioral and in the brain and practiced through verbal report. It's not even really interactive. Social cognition, you know, research in social neuroscience and psychology doesn't really account for for interaction, and also like the notion of the mind. I mean, you have the relationship between the mind and the body, you have the relationship between the mind and the body and the environment, or the body and the environment, the body and other people. I mean, you have all of these sort of like dimensions to our minded being. And we haven't really figured out how to study all of that together. And we need to be interdisciplinary in order to do that. And if we're going to be interdisciplinary in order to do it, we have to be able to be more comfortable talking to one another. I can't tell you how many times we have guests come to present at Center for Subjectivity Research who say like, I am terrified to be here because I don't know how to talk to philosophers and I will also see for myself like philosophers are really the toughest audience because they want to know why you're making the assumptions that you are and they don't really let you get away with anything along that process which is great but also really intimidating.

[00:21:49.915] Kent Bye: I think the trend that I see is that actually a lot of these thoughts that have been talked about for a long, long time in the philosophy circles are actually, people are able to have a direct experience of this, of like, you know, oh, well, maybe my construction of reality isn't actually an objective reality. And they start to, you know, have a direct experience of what that actually feels like. I sort of see this fusion with virtual reality. It's sort of like this nexus of all these different things coming together where VR may actually be the transfer learning language where all these people can actually collaborate and talk to each other through VR and direct experience.

[00:22:20.423] Linda Joy Gerry: Well, what's really interesting about this is that We're actually using virtual reality technology to rediscover extremely obvious aspects of, and simple aspects, I believe, of our experience that have always been there all along anyway. That's what I find with a lot of my research, is that you're just getting people to be aware of one aspect of their experience that is already there. And I think a lot of great art is also doing this, you know, and it's these small movements for it, but it just, it amazes me that we're using literally the most advanced technology that we have available today to do something that's like, you know, so basic and so simple and so primary in our experience and, you know, and to connect with other people in this way. It's interesting, but it allows people to get excited about these things that are distal from the way that we generally talk about being human. So, you know, Alan Wallace talks about how in Western science, we started out by looking at objects, astronomical objects, very, very far away. So we developed all of our tools of scientific investigation to do that mode of inquiry, right? Studying these very, very far away objects. And it took so long in the history of Western science for us to finally get to psychology basically and lived experience in the mind versus if you look at, this is a little bit reductionistic, but if you look at, you know, more Eastern cultures, they started out practicing a cultivated awareness attenuation to the lived body and mental experience and how to orient attention and really starting at home with oneself and with one's lived experience in human mortality.

[00:24:06.394] Kent Bye: Yeah, and just reading Rick Tarnas's Passion of the Western Mind, he kind of puts this Renaissance time period as kind of the beginning of the modern self with this differentiation and being able to turn inward and actually study the sense of self. So I looked at his work in studying the evolution of Western thought since Plato. But I just wanted to go back to this research that you're looking at in empathy and also this question about emotions and, you know, what can you tell me that you've seen from the literature and all your studies as to why emotions are an important part to really try to consider? Why do you think it might be worth considering?

[00:24:42.638] Linda Joy Gerry: Yeah, yeah, and I'm sorry I didn't get to that in my previous answer. I think emotions sort of filter the way that we attenuate and perceive and are able to orient ourselves and acquire new knowledge and interface with other people. There's sort of like a valence is a little bit too simple of a term, but it, you know, it's kind of related to that idea that there a general sort of indicator for, or you know, sort of spectrum for how we are predisposed to engage with something. I think that there's an important distinction between mood and emotion that I'm blurring a bit here. I'm talking a little bit more about mood. But in relation to empathy, a huge part of empathy is either emotional recognition, inferring the emotions of another person, or emotional sharing. It's interesting. My professor, Dan Zehavy, sort of distinguishes different concepts. Empathy is one of them. But he talks about emotional contagion, empathy, and then emotional sharing, and says that emotional sharing is actually something separate from empathy. And I've contested that. I really think that emotional sharing is quite key to empathy. And what we really need to get at is what we mean by sharing. My thing is, you know, maybe even what I'm studying is slightly different than empathy. I mean, empathy, the thing is, we talk about it as though it's one thing and one process, but it's many things and it contains many different aspects and can take many different forms, but I guess what I'm really interested in is how we come to share experiences with one another in a communicative way as well as like a simultaneous kind of synchronous way of shared semantic understanding and shared emotional experience. So I think that's really, really crucial for empathy. There are a couple of things that come to mind. So one is that some of the cognitive neuroscience research on empathy that kind of pushed it forward was Divigabond and Singer did some studies with rats on sort of what's called pain empathy. And it's interesting because in the philosophical community and some of the criticisms of this is that, well, pain isn't really an emotion. And I've talked with a pain researcher and she's, you know, pretty much convinced me that there's a huge emotional component to pain. And that actually pain could be largely emotional. And I mean you could even try to break it down as like, oh there's physical pain and then there's like emotional pain when you're hurt by someone. But I think it's actually, the two are even more contiguous than that. I think that this emotional sort of aspect of pain is really what makes the experience what it is. So, yeah, that's something that's interested me. And then another thing about simulation theory that kind of has to do with emotions that I find really fascinating is that simulation theory is saying that we first personally experience vicariously what another person's feeling. And I find that really interesting because, again, there's research that like personal distress is an inhibitor to empathizing with someone. If you're in a lot of pain or there's something, you know, if I witness like a motorcycle accident or something like that and I'm too distressed by that or I'm too like, I feel their pain, then I, you know, don't have as much of an ability to really extend caring behavior and to be there for that person. And I think that's a huge, really valuable aspect of empathy. So that's important. And then The other is that, well, if I'm first personally feeling something, what allows me to do several things that I think are important for empathy? One, to somehow realize that another person is a source of the emotion that I'm first personally experiencing. And also that the other person could be feeling something different from what I am first personally experiencing. And, well, I mean, the other question is the criticism of simulation theory in general is to what extent are we actually doing this all the time in our lives? Because that would be really, you know, what would be, we'd be exhausted by, I mean, I think we would be exhausted by the end of the day. And the reason I say that is there have been some studies with actors and, well, this is a little different, but actors with facial expressions and micro-emotions where they were doing all these different facial expressions and they found that they were transiently feeling all of those emotions when they're doing these facial expressions and that they were just exhausted by the end of the day. So, yeah, I mean, I don't know. I think that simulation theory is on to something, but I don't know if I would say that, like, every time we experience empathy or every time we even look at someone's face or have direct perception of, you know, sort of their body, that we're somehow automatically simulating their emotional states.

[00:29:00.952] Kent Bye: Just to clarify the language of simulation theory, I know within VR circles, simulation theory could mean that you actually live in a simulation, but I think you might be actually talking about a specific empathy simulation theory. Maybe you could elaborate what the empathy simulation theory, because there may be a number of people listening who are thinking that you may be talking about, do we live in the matrix?

[00:29:18.588] Linda Joy Gerry: Yeah, sure. So I'm referring to a conversation in philosophy of mind and phenomenology about social cognition, which differentiates theory theory and simulation theory. Theory theory is more that we understand other people by inferring their emotional states and simulation theory, by contrast, is saying that we, you know, we have this like direct sort of when we perceive other people in the environment, there's some sort of it's kind kind of related to mirror neurons, but it's, I've always said it's kind of a tamer version of mirror neurons because I think that its explanations are a little bit simpler. But it is pretty consistent with the mirror neuron literature theory, you know, right? When we, when we see someone perform an action, we actually have similar neurological events. I don't understand exactly how that gets to emotion. to me there's like a weird gap there that doesn't seem to make a lot of sense because where does emotion arise within motoric performance and even with kinesthetic empathy? I mean I think this is a great question but I haven't been able to wrap my head around it and I have not been able to parse through the literature to really get a sense of this. You know, there's notions that like you can perceive someone as more like me. Glazy talks about that quite a bit if you're moving similarly and we know that if someone's imitating your movements that we perceive them as more likable. But again, what does this have to do with like emotions and understanding that other person's emotional experience? I don't know entirely. And I think some of the most sort of transformative theories in philosophy and science have been the ones that were really, you know, taking things a little bit too far. You know, mirror neuron, again, is a really good example. It's very extreme, and it has a lot of points to be criticized, but it's definitely on to something. And I think that, you know, we're still parsing through that and trying to understand, like, okay, so there's something going on here. How does this actually work? And what are its implications? And I really admire Sean Gallagher's work. And I think he's another figure who's been a little bit very, very bold and a little bit extreme in some instances, but it's extreme in a really valuable way. And I've heard Debra Toffelson said that so much of philosophy is sort of like, you know, these divergent branches that just gradually move close, you know, to the middle. And I think that that's just happening increasingly and that that's an important move. There's a really excellent book that I've always loved by Leonard Schlein, who is a neurologist, neuroscientist. He wrote a book called Art and Physics. And a lot of his books are sort of focusing on the relationship between art and science. He wrote a book called Language Versus the Goddess. And I'm really interested in the discovery of written language or the transition of society into using written language and how that really transformed our thinking. It has a little bit to do with Malaforos' material engagement theory. So, you know, what Leonard Schlain is saying is that, I mean, it's kind of a bold argument, but he defends the argument in quite a beautiful way. And even the anecdotes that he's using in the process are just extraordinary. I highly recommend it. But he's saying that we move from a matriarchal society predominantly where we've lived in goddesses to a predominantly patriarchal society. And that that shift is largely, he attributes to the advent of written language. You know, before we were telling stories and it was all about this like oral practice of, you know, this ritualized sort of transmission of knowledge, understanding, human experience, and what we had learned in our wisdom in our lives, right? I mean, think of Plato again, you know, obviously that was written, but I mean, so much of his conversations with Socrates were, you know, in dialogue, and Socrates was all about that, right? So, Schlein says that we put the pen in the fighting hand, you know, and then we're inscribing language. He gives this anecdote about this man who had never- he'd grown up in a culture not exposed to written language, and he was in a library somewhere in Western culture. and he sort of grabbed the books and he said like, what have you done with the words? Why are they trapped? You know, why would you do this to words? And I think that, I don't know, just somehow like that as a metaphor really gets to something that we are doing or have done or in a long process in human history of like really trying to like grasp the intangible and like break it down and like pin it down and say like, I understand it now, you know, but it always evades us. And I think that a lot of people can outright contest that there's something intractable about experience, or avoid it, which many people do, or be scared about that, because it causes a lot of uncertainty, or just accept it. And I guess for me, it's just, I find it kind of magical. One of my favorite things about communication is that I can never fully understand another person's experience, that they're always somehow outside of my range, yet there's some things, you can have this true moment of, Kim Crezon in Waking Life, she describes it as this moment of almost spiritual communion that we feel that we are really understood. And she says that those moments may be transient, but she thinks that they're what we live for. And I agree with that.

[00:34:19.207] Kent Bye: Yeah, I've actually been looking at Leonard Schlein's work and kind of like this idea that the written language was something that was more separating and disconnecting, whereas when I talked to Charlie Melcher, he said that Socrates said that it's now the dead word, you know, when you actually write it down, rather than the living, the embodied emotional communication of it. And I think that there's been this resurgence photography, film, the visual mediums were kind of bringing back the right-brained perception and that with VR I think it's really you know this communication medium that's really trying to equally marry both sides of the objective and the subjective and so it's like this new medium that is this sort of like the new language that is able to allow people to have that direct experience because I think that With the abstractions, it's sort of like you've learned so much of the world based upon what other people tell you, and now we have the capability to have your own direct experience. And from that direct experience, then you're able to use all those unconscious processes that puts information in your body in a new way.

[00:35:23.408] Linda Joy Gerry: Yeah, what you said about VR being both subjective and objective immediately reminded me of Merleau-Ponty and his notion of, I think it's called reflexivity. It's the ways that our lived body is both interior and exterior. So he actually describes this as sort of being preemptive for our ability to have empathy and other awareness. But he says that it's the awareness sort of like when we're touching our own finger of, you know, having an internal sensation, but also a body that's external and in the world. And that this experience allows us to understand that that same state is possible for others. I just, I'm not explaining this perfectly, but I always found that really, really intriguing as a notion. Yeah.

[00:36:17.790] Kent Bye: So the thing that I find really fascinating about your approach is that you're basically coming from this completely out of left field in terms of like, I think you may have been like the first trained philosopher who's using VR in an applied way of really asking these deep philosophical questions to really kind of learn about, you know, what it means to be human, but also these deeper philosophical questions. And so you're coming at a perspective that is able to see and critique different theories of embodiment and other sort of approaches that the VR community has had from, more of the grassroots kind of bottom-up of direct experimentation. So I'm curious to hear some of your insights on embodiment or other things that you think might be kind of blind spots within the VR community.

[00:36:55.868] Linda Joy Gerry: Yeah, absolutely. I think there are a lot. I really realized, for me, I mean, virtual reality, like I said in my presentation, is really the first tool that phenomenologists have had to sort of practically explore and apply different theories of the structures of experience, because it is itself in an experiential medium and a much broader scale and scope than any other technology. So I have been trying to get a lot of my philosopher friends interested in virtual reality and it's happening. There's a conference in Belgium at the Husserl Archives that's on phenomenality and virtuality. So it's definitely starting to become a conversation, but also I should humbly cite and admirably cite Sean Gallagher as really being, you know, the first philosopher that I know of, at least, to really be working with virtual reality with his project, The Neurophenomenology of All in Wonder. And, you know, that project hugely inspired me and I think is really impressive. And then also The Machine to Be Another that was The mission to be another is basically the whole reason I got into virtual reality. I've been fascinated by first-person experience and the structures of what's called agentive self-location. This notion that I look out from this view and I always associate this with I. What does that mean to be able to share that with another person for the first time or experience someone else's first personal modes of access to the world? It's subtle, but I think it's important in some way. It's interesting, too, because we were at this conference all about cognitive neuro-enhancement with the Extech Expo. And I realized that so many people are sort of going at it in this way that I think is kind of counterintuitive, where they're like, oh, we'll do neurostimulation and we'll enhance your neural processes that way. And I think the most plastic and flexible realm is experience. And with VR, you can really build experiences that will drive neuroplasticity. And I also think that attention is really important. So if you guide people's attention to new aspects of their experience, then that will really create the major shifts that I think a lot of those. researchers and developers in virtual and augmented reality environments are looking to achieve. But yeah, I mean, I think the other thing for me, and this is just my personal view, but I think that one of the biggest tools that we have is really attenuating to the varieties of experience as lived by or for another person. And if we can really learn to do that and create tools to do that with virtual environments, and I think we're doing ourselves a huge service. One, for being able to understand others, but two, for being able to understand varieties and maybe aspects of our own experience that might have been there, but we just didn't know how to pay attention to them.

[00:39:39.881] Kent Bye: Could you share a little bit of your major takeaways from your first kind of inquiry and this balance between agency and empathy, but also some of the big open questions that you think are kind of left that's going to be driving your research forward?

[00:39:53.868] Linda Joy Gerry: Yeah. Yeah. So I feel like nobody's asking this question and I don't know why. Hopefully people will start to, but when I ask people questions like, what is the relationship between agency and empathy? You know, they sort of say, well, I'm not an expert in that, or I don't know. It's a very bold question. I wrote a project proposal for Thomas Metzinger on a project that got cancelled, but it was on first-person methodologies and his notion of mind-wandering and mental autonomy. So they're basically two different extremes. What Metzinger sort of reports for mind-wandering is that if you do random experience sampling on a person and have them report their cognitive activity throughout the day, their thoughts, then it's generally not related to what's directly in front of them. They spend most of their lives mind-wandering. But he says that we have this illusion of having sort of this great, what he calls mental autonomy, this notion that we can harness our mental energies for mental and behavioral actions. But really, that's an illusion. So I just sort of distinguish between unreflected and reflected notions of mental autonomy. And what I've kind of hypothesized is that within these experiences, specifically within the machine to be another body swap. So you're seeing a live video stream from another person's point of view. and seeing their hands and moving together. There's a performance art installation where you're applying synchronous tactile feedback that they can see, so visual tactile stimulation. And then you tell them to move together very slowly, but also synchronously. And in this process, there's a turn-taking initially, but in a lot of cases, that turn-taking breaks down and they experience this sort of illusion of sort of continuous movement moving together. And I think that there's something that happens there where one aspect of our unreflective mental autonomy, it's like, Oh, wait, okay, I normally like perform actions, and they're just how they are in the world, or I'm interacting with someone else. And I have a choice about, you know, whether I'm initiating or following, or, you know, I can choose whether to mimic their movements or whatever. But in this instance, like that agency, and the association that that our sense of agency has to our physical spatio temporal in body location is disrupted. And then I think that that is related to a new type of attenuation for really what self-experience really is and its limits and a deeper attenuation for another and another's experience.

[00:42:14.424] Kent Bye: Yeah, this tension between agency and empathy is something that I feel like I've been covering a lot in the context of both storytelling, but also gaming. Because, you know, when you're playing a game, you're expressing your agency within the experience, but yet you will have storytelling elements. And so it's often a context, which so that you are playing the game and then, okay, now you stop and now you're going to receive the story. And, you know, from a filmmaking perspective, it's like you have a certain amount of narrative arc that is a time-based medium that you are creating tension and that narrative tension is dissolved if you are in an open world and you can do anything that you want. So there's sort of a time-based nature of that. So I think that experiences like Facade, for example, with Andrew Stern and Michael Mateus back in 2005 started to do these natural language processing so that you could actually participate in the story and it would have a drama manager there so you were able to actually interact with the narrative as it was unfolding and it would have like a number of different branches that were unfolding in that process and so I've been thinking about this in the context of VR and you know one of the things that was told to me is that you know Jesse Schell has said that you know Looking back to film, once film had sound, then we had sort of this holistic experience in that once games are able to listen, then we're going to have a lot more richer experiences. So with that, what I'm seeing is that an example of empathy and agency is a heartfelt conversation between two people who are listening and who are completely present. They're able to both listen to the content of the words, but also feel the words. And when they participate, they're able to reflect back to them and have this balance. It's a turn-taking of a conversation. And what I'm seeing in VR is that kind of metaphor of a conversation is that it's very difficult to have that authentic conversation when the game can't really understand or listen to you at all, right? but with artificial intelligence we're starting to see the first steps towards being able to use natural language processing to have the intent extracted from that to be able to be put into some sort of like drama manager or a branching narrative that is giving you the next sort of piece of that story and so you're able to construct a narrative that has an arc but yet you are able to fully participate so it's sort of like the ultimate essence of this that is done right now is Dungeons and Dragons where you have someone who's a dungeon master who's facilitating a collaborative process, but he's actually really deeply listening to the emotional content and he's creating an experience and driving the narrative so that people can feel like they can participate, but he's also giving them challenges. And it's a little bit less about, you know, cultivating empathy, but it's that same sort of interaction. And I think if you change the context from away from exploration, but more inner depth vulnerability and really, you know, sharing of your deep soul process that I sort of see that's where the empathy interaction takes place and that you're either like really listening or you're kind of speaking from the mind. And that's sort of like, that's how I think about it at least in some metaphors.

[00:45:17.086] Linda Joy Gerry: Yeah, you know, we try our best to do something that's both or in between. Yeah, definitely. Yeah, just in regards to interactive narratives and the importance of interactivity and participation. So I mentioned earlier that I had chosen to do my experience as a pre-recorded experience. And the reason I did that was because you can add augmented layers in a pre-recorded experience. I haven't figured out how to do that easily in a live stream. And also visual resolution is a lot stronger. but that has a lot of disadvantages. One is that it's like a one way channel of communication. And as I said, I really value the importance of interaction, especially around empathy. So one of the things that we're sort of looking into doing is trying to figure out how to do this video in a live stream set up and also to do it interactive on both sides so that the painter can actually see what the user is doing in some feedback and the user can see what the painter is doing. and then the painter can sort of respond to the user and guide them along the way. I think that would be really beautiful and I think the move that's going to need to make that happen is the use of augmented reality technology as a headset hardware rather than virtual because I think it'd be extremely difficult to do with virtual. But yeah, we've kind of joked about, and I don't know a good way to do this, but I think it would be really fun to have both a painter and a subject. So we were thinking about like, okay, can you have them both like not know that it's interactive and then test like a Turing test to see when they realize that it's live stream and it's interactive. And we had sort of said that, okay, you could tell the user to start drawing something and tell them to draw what the painter is describing as she's painting something. But it becomes a little tricky, and basically what you end up with is that you have mutual stumbling. You know, the other person's waiting for the other person to do something, and they can't really figure it out. And then it becomes quite obvious that it's interactive, because you're like, OK, I guess nothing is happening. No one's doing anything. But if we can figure out a good model for doing that, I think it'd be really fun. Just because I think that's really important too for them to come up with that moment of like, oh wow, I thought this was pre-recorded but this must be another person because they're responding to what I'm doing. And then also we've kind of played around with the notion of having two different painters sort of paint simultaneously. This is something we've noticed in the experience already is that users are reporting paintings or their experience is something that's like imagined. using the painting that they're actually seeing physically created by the painter in the video, but then they're sort of, you know, adding layers of what their haptic feedback is giving them and what they think they're doing. So they are imagining this painting that is on neither canvas, right? And I think that's kind of cool too, because then the whole experience is really in the mind and imaginative. So, yeah, I'm sort of playing around with different combinations like that, and I think they're all really important. And essentially, too, like, if you're really studying empathy, it should be something that is experienced by and for the painter as well. And the painter did to some extent, so, you know, obviously she's aware that she's creating this experience for a user. She has a teacher kind of position, but I think it should be bi-directional. And then another thing I want to mention is something that I think is really important to empathy. You hit upon this a little bit, but it's this notion of being seen. And it was something I met a man I was talking with on the plane. He was from the UK. And we were talking about how, again, I hit upon this earlier, but so much of when people talk about empathy, they usually say that it's most likely to occur in face-to-face interaction. I agree with that and I think face-to-face interaction is wonderful but having grown up as a very shy kid who didn't have a lot of people I could talk to in my immediate environment, I was interested in philosophy from about when I was like 13 or 14 and I couldn't talk to anybody about that. So I used forums to basically ask questions and I shared a lot of my writing online and that helps me realize that there are other people out there like me and I had a sense of being seen. and being understood. And I was thinking like in an extreme case, if you have maybe someone growing up in a remote area in West Africa or something like that, who's gay or something and is not aware that that's a possibility of being in the world, because that's not something they are aware is possible within their environment, connect to someone remotely in another part of the world, then again they could have this deep sense of being seen, you know, really being connected to someone. So I think that's important to consider when we're talking about empathy is that Yeah, face-to-face communication is really great, and there are so many potentials for understanding another within that type of face-to-face interaction, but there are also other ways of connecting with people that are still incredibly impactful.

[00:49:38.777] Kent Bye: Great, and finally, what do you think is kind of the ultimate potential of virtual reality, and what it might be able to enable?

[00:49:46.936] Linda Joy Gerry: Yeah, again, this greater intersubjective understanding of one another's experience is really where I go with that. I really think that, you know, we can sort of wire ourselves into sensory, bodily, motoric, cognitive and perceptual aspects of one another's experience. And I think that doing that will really open us up to possibilities and transform the way that we understand and communicate with one another. That's just my personal, like, that's what I would love to see in the world. I would love to see people really being able to share with one another in a new way. Even with my project, I realized that I was having a conversation with subjects after the experience that I've never been able to have with people before about just like what it means to have an experience. And all of a sudden, just this experience gave them a platform as a jumping off point to be able to have that type of conversation. And yeah, I'm hoping that it will cultivate a sort of respect, admiration and appreciation for the wisdom of varieties of experience for other people. Yeah.

[00:51:00.193] Kent Bye: Awesome. Well, thank you so much.

[00:51:01.534] Linda Joy Gerry: Yeah, thank you. This has been great.

[00:51:03.695] Kent Bye: So that was Linda Joy Gary. And at the time of this interview, she was finishing up her master's thesis on virtual reality and empathy and embodied simulations and perspective taking on the body of another at the University of Copenhagen. So I have a number of different takeaways about this interview is that first of all, the thing that Linda said that as a philosopher, she had been having some of the most interesting discussions about philosophy after people having a virtual reality experience, and that those VR experiences were actually bringing a level of awareness and attention to the dynamics of human experience in a way that people had never been able to recognize before. And I found that to be true as well. I think that there's a lot of these philosophers and phenomenologists have been talking about these dimensions of human experience for well over 200 years, but it is giving people a chance to have a direct experience of some of these deeper insights that they have about the nature of our experience. I also really appreciated Linda's approach of trying to balance what the cognitive science can be able to bring in as well as the phenomenological perspectives and how these different disciplines are trying to, in some ways, be combined together, and that in some ways, virtual reality is able to do that, to be able to start to holistically measure some of these topics like empathy. Empathy is something that is incorporating all sorts of different dimensions of our awareness. And at the very end, one of the things that Linda says is that she's trying to see how Empathy is a part of our sensory experience, our body experience, our perceptual experience, our sensory motor experience with agency, our cognitive experience, and there's also social dimensions of empathy as well. And if you look at her master's thesis, she goes through and kind of accounts all the different perspectives of empathy. I think within the popular media about virtual reality, Chris Milk's TED talk about VR as the ultimate empathy machine is something that has gotten a lot of press and attention. But in her master's thesis, what she's saying is that there's a lot of buzz about it, but when you actually look at the literature, there's only like, you know, 20 or so different actual research articles within virtual reality looking at empathy. I know that Mel Slater, as well as Jeremy Balanson have been doing a lot of research with looking at how VR can give us insights into empathy. Beyond that, though, there's all sorts of other research that's been going on to empathy and to all the different branches. And she starts to bring all that together and start to see how VR could be used to get some unique insights into what's happening with empathy. But this concept of neurophenomenology is something that also I thought was super interesting that came out of this interview. And that's just being able to find the neurological correlates to what's happening in your brain, but also to combine that with the first person perspective. And in the discussions that I've had with Mel Slater and looking at presence research and surveys, there's this difficulty in being able to actually take somebody's first person account of their narrative experience either during or after the fact that sometimes it's not always super reliable in terms of the story that people have about their experience matching their behavior or other dimensions of what's actually happening. And so there can be a disconnect there. But at the same time, sometimes there could be a lot of insight about what is happening with someone's first person perspective that is going to give so much more context and depth into where people are actually at. And I think the challenge in the current sciences is that there's not a really great way to be able to fuse together someone's first-person perspective with some other more objective data. And I think that is the goal of something like neurophenomenology, is to find some of those neural correlates of someone's experience but also matched with what their first-person perspective is and that's what Sean Gallagher was looking at and specifically with awe and wonder. Awe and wonder is something that is such a subjective experience and that by simulating people going into space and having the overview effect he was looking at this experience of awe and wonder and seeing how he could simulate it but to do that neurophenomenological approach of trying to combine together the objective data with that first-person subjective experience. And that, I think, is something that is largely an open question, but I think moving forward we're going to see a lot more of that. One example that I see that's happening in Silicon Valley is Tristan Harris's Time Well Spent, which is trying to takes someone's phenomenological experience of an app, specifically whether or not they're happy or not, or if they're filled with regret from using it, and he's able to show that basically some of the most popular applications that are out there for the iPhone, people are happy with their experience about one third of the time and not happy about two thirds of the time. So there's all sorts of different phenomenological techniques, whether it's the elicitation interview technique or some of these more micro phenomenological techniques. But the difficulty that Linda is saying that there's not a lot of advanced neuroimaging techniques that are able to make a correlation between what they're observing at a micro level with the neurology inside of the brain with the experiential awareness and attention that is being reported by the subject. So I think that with different technologies, like perhaps Neurables, where they have this interconnection between being able to measure the brainwaves with the virtual reality technology, that maybe they'll be able to get a little bit more of the neurophenomenological techniques down, where we'll have new epistemological processes for being able to actually measure somebody's brain activity combined with what they're reporting of what they're saying. That and combined with like artificial intelligent natural language processing it might be easier to both capture and be able to analyze dynamically on the fly what other people are reporting whatever their experience is. And finally, the other thing that I'm really taking away about this interview is just the whole range of emergent ideas about the theory of mind and what cognition is, as well as the phenomenological approach. And so Linda lists a number of the different theories of cognition that range from embodied cognition to looking at how you're embedded within a world, at the ecological psychology, the situated cognition and distributed cognition. looking at kind of the whole body or the environment as being part of that cognitive processes. And that Varelia's embodied mind is putting objectivity in quotes, and that the more phenomenological approach from Husserl is that there's more of an intersubjective verifiability. It's basically the consensus realities of what people are agreeing to of what is actually happening, that that is another way of looking at objectivity through the lens of subjectivity. And so I think that there's certainly a huge amount of value of reductionism and materialism being able to reduce things down to individual component parts, but that at this point, it feels like the movement within the sciences is to move back up into using metaphors of organisms and interconnections and how things are actually related. And so whether that's Linda's approach of making this combination of cognitive neuroscience with something like phenomenology or neurophenomenology, or finding other ways to be able to have interdisciplinary collaboration so that they're able to look at things in a holistic way. And that's where I was seeing virtual reality was really being that platform and mechanism to allow people from different disciplines to start to collaborate and talk to each other. And that even if they had different words for talking about the same thing, that they were able to kind of focus it on an experiential perspective through the medium of virtual reality, and that they're able to look at these systems in a much more holistic way. And that was some of the criticisms that Linda was bringing from looking at the existing research into empathy and saying, hey, wait a minute, there's not a lot of intersubjective perspectives or phenomenological perspectives. And maybe this experience of empathy is about, you know, having a shared experience with somebody else and being able to really tune into what their experience is. And so she was really bringing some different criticisms that there wasn't a lot of the intersubjective or phenomenological perspectives when it came to empathy. So if you want to dive deep into a lot of the different references and theories about empathy, I'd highly recommend checking out her master's thesis. It's called Virtuality and Empathy, Embodied Simulations and Perspective Taking in a Body of Another. It's on ResearchGate. I'll have a link to it in the show notes here. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you'd like to support the podcast, there's a couple things you can do. First of all, just spread the word, tell your friends. And secondly, consider becoming a member to the Patreon. This is a listener-supported podcast, and I rely upon your donations to continue to bring you this type of coverage. And just a few dollars a month makes a huge difference. So you can donate today and become a member at patreon.com slash Voices of VR. Thanks for listening.

Tagged as:

More from this show

Menu

#596: Researching Empathy in VR with Cognitive Science & Phenomenology