I talk with Joey J. Lee & Elliot Hu-Au about a paper they published in Frontiers of Virtual Reality called “E3XR: An Analytical Framework for Ethical, Educational and Eudaimonic XR Design,” which looks at the intersection of XR ethics, best practice approaches for applying immersive XR technologies for technologies, and how to create experiences that promote human flourishing and eudaimonia. Their paper provides a great survey of a broad range of citations on the intersection of technology and ethics, and a first take of separating between the range of must-have ethical considerations as well as some of the more nice-to-have aspects of eudaimonia. Where exactly that line is between issues that are more ethical and ones that are more eudaimonic is a question that I believe will depend on the specific context that will change from project to project as there are different tradeoffs in mitigating harms across different intersectional identities and demographics.
Lee cautioned that some of the research into the efficacy of immersive EdTech apps can have a “hand waviness” that assumes that VR is a silver bullet that will obviously lead to better learning outcomes. Lee & Hu-Au lay out some of the contexts where VR probably isn’t a great fit, and where it really shines. Lee also said that the underlying applying immersive tech to education is relatively under theorized for creating systems and frameworks to help navigate how to best leverage the affordances of XR for education.
Lee & Hu-Au’s paper was published on October 6th, which was an auspicious time for me as I had a couple of related panels and talks I was giving to XR education folks. I gave a quick a 10-minute primer on XR Ethics for Education for a webinar put on by the IEEE’s Global Initiative on the Ethics of Extended Reality focusing on XR for Education ethical and policy issues on October 14th I also was invited to give a talk on October 20th for the University of Michigan’s Center for Academic Innovation on “XR for Higher Education: Experiential Design Affordances, Ethical Considerations, & Production Strategies” (the video should be coming soon). So this paper also provided a great launching off point to check in on some of the latest theories and research for how to best leverage XR tech for education.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
UPDATE: The video of my October 20th talk on “XR for Higher Education: Experiential Design Affordances, Ethical Considerations, & Production Strategies” that I mention in the intro/outro of this podcast was published on October 21st:
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So on today's episode, I'm really excited to look at some of the intersections between XR ethics, XR education and learning theory, as well as eudaimia, or human flourishing. So this conversation came about because on October 6, there was a new paper that was published by Joy Lee and Elliot Huao. It was called E3XR. It's an analytical framework for ethical, educational, and eudaimonic XR design. So, I was actually pinged by Joey, because he had come across some of my work on ethics and just wanted to give me a heads up. And I had also actually already had on the books a couple of XR ethics and education discussions. One was with the IEEE that happened on October 14th, and one that just happened today, October 20th, where I was giving a talk at the University of Michigan's Center for Academic Innovation, where there's lots of different folks trying to figure out how to deploy XR technologies throughout all the different university departments. strategies to start to think about how best to go about that. Providing some thoughts on that. This paper came at a really great time because it was really an awesome survey of a lot of different learning theory and the latest for how to apply that to immersive technologies, as well as a bit of a survey and trying to understand the ethical landscape when dealing with XR technologies within the context of education. but also in these larger fields of human flourishing and eudaimonia. So that's what we're covering on today's episode of the voices of VR podcast. So this interview with Joey and Elliot happened on Wednesday, October 13th, 2021. So with that, let's go ahead and dive right in.
[00:01:44.764] Joey Lee: My name is Joey Lee. I am a lecturer at Teachers College, Columbia University, where I've been teaching courses on technology and design and ethics, virtual reality for a little over 11 years now, almost 12 years. And yeah, that's great to be here.
[00:02:02.395] Elliot Hu-Au: My name is Elliot Hual, and I'm a doctoral student at Teachers College, Columbia. I research virtual reality and how it can be useful in science education, as well as learning abstract concepts.
[00:02:16.937] Joey Lee: And I should say that over at Teachers College, we have a couple of MA programs that have been wrestling with these kinds of issues having to do with ethics and more meaningful design and using game design and virtual reality, augmented reality for not just educational purposes, but also social good kinds of purposes. Like our relatively newest MA program is the MA in Design and Development of Digital Games. And because it's Teachers College, you know, it's quite strategic in terms of its proximity to the New York City public schools and really trying to make an impact in the education space.
[00:02:56.607] Kent Bye: Okay. So yeah, maybe you could each give me a bit more context as to your background and your journey into XR.
[00:03:02.891] Elliot Hu-Au: So I started in the master's program at Teachers College in 2015, and VR was still in the like Oculus TV one stages, I think around that time. But it was really interesting to me. And I kind of went back to school, because I had previously been a high school physics teacher for a long time, over 10 years. And I just had noticed that students when they were Specifically, I guess learning through like online means, because I previously taught at a public school where students can make up their credits. They had to do this virtual learning thing and it was just like so dry, boring. I felt like this was not the way that education should go. And I liked using games, video games even to teach. So when I came back to school, VR was still feeling kind of brand new. And so then I thought, oh, this would be a great way to introduce students to science concepts, especially ones where we can't always like see them with our eyes. So yeah, that kind of led me down a path and Joey really helped me mentored me, I would say towards like how to become a researcher in that area, how to just get information, how to learn things, listen to things like your podcast. And so, yeah, now here I am.
[00:04:21.293] Joey Lee: Yeah, for me, it's been a long journey. So I graduated with a computer science degree, and I worked as a software engineer for a few years, but I really didn't find it rewarding. Like I wanted to do something more meaningful with technology and make more of a societal impact. And so I went back to graduate school, got my PhD from Penn State University. And my dissertation topic actually had to do with using games to raise awareness of racial microaggressions. And so at that point, I was very intrigued by the idea of using interactive media and game-like technologies to try to make more of a social impact. And around that time, I went to my first Games for Change conference. So every year in New York City, there's a pretty big conference, pretty big festival called the Games for Change Festival. And a couple hundred folks who are not just interested in games for fun or VR for fun, but they're interested in using it to enact some kind of social change or raise awareness of injustices or promote more empathy or to get us to take action on climate change. That was very transformational for me. It definitely opened my eyes as far as the power of games and VR as designed experiences. that can really make an impact on one's self-identity, but also getting a person to think about an issue in a new way or to see something from a new perspective. And around that time, I was also very interested in how VR could give firsthand perspectives and experiences, basically do impossible things that you can't do in real life, right? Like things like time travel and see possible futures. I've done a bunch of work over the last several years having to do with climate change education and VR as a way to envision possible futures we want to avoid, right? Like what happens if sea level rise, you know, continues to have a negative impact, you know, fast forward 50 to a hundred years in the future, how might coastlines be impacted by the effects of climate change? You know, traveling back in time, like historical history-based education opportunities, and as Elliot said, a lot of interesting new possibilities with abstract scientific concepts. That situated learning opportunity and that constructivist learning, the affordances that are provided by XR, to me, there's really not much like it. And there's so many tremendous opportunities for educators especially when they have a good grasp of learning theory and research to allow for better learning. Because there's a lot of really bad ed tech that's out there, it's hard to make effective software that supports learning, but it's also fun. In my games class, we talk a lot about chocolate-covered broccoli. Why is it that so many games are like worksheets, but adding a layer of sugar on top? That's part of the motivation, which we'll get to a little later, I guess, behind our framework that tries to bring together ethics, education, and eudaimonia. So just to kind of wrap up with my journey over the last, I would say five or six years, I definitely sensed the need for not just better learning through games, but also really wrestling with ethics and just seeing a lot of things in the news that made me feel very uncomfortable, right? We see, obviously, with social media, a lot of negative effects on the attention economy and the race to the bottom, kind of like the time well-spent movement and some of the ethical challenges and issues with monetization and surveillance capitalism and these kinds of challenges. But just see other kinds of things depicted in the black mirror in some of these popular shows, where if current trends or if the trajectory of where things are going continue, It's like, do we really want to have that kind of future? It's kind of like, you know, things are moving so fast and we're not really pausing and wrestling with, should we be doing this? Should we be designing software in this way? You know, what about all this new tech like AI and facial recognition, all this other kind of stuff that's entering the classroom? You know, a lot of stuff that's dramatized in movies and TV shows, but I kind of feel like we're in the middle of wrestling with a lot of this kind of stuff, you know, every day now. So that's the motivation behind an ethics course that I created at Teachers College about five or six years ago. and really thinking about what is most important. And to me, it comes down to social justice and human flourishing and how do we design games and XR experiences that are not just about hedonism and temporary pleasure and that kind of thing, but more importantly, how do we help people achieve their full potential and how do we achieve that kind of human flourishing through these kinds of experiences. And that's kind of where I'm at at this point today.
[00:09:19.993] Kent Bye: Yeah. And so you recently published a whole paper in the frontiers in virtual reality. Yep. It says E3XR, an analytical framework for ethical, educational, and eudaimonic XR design. And so, you know, this ethics is something that in the course of doing the podcast, I've just been talking to well over a thousand people over the last seven years. And I think a lot of these issues kind of organically have come up. And, you know, I tried to do an initial mapping of what I thought some of those contexts might be in 2019. And it looks like you are taking that and also building a three-tiered approach where there's the ethics, then there's the educational framework, and then there's the eudaimonic. And I guess another way of saying is that the human flourishing aspects So yeah, maybe you could just tell the story of how you evolved to that point to be able to look at what the existing landscape of different frameworks are that are out there and how you came about this E3XR approach.
[00:10:12.944] Joey Lee: Absolutely. Yes. And definitely, you know, we want to thank you for a lot of your foundational work in this space and definitely the domains of human experience and a lot of these, you know, the stuff from a few years ago were definitely very influential for us. for sure. And to me, it was very exciting to see folks from various disciplines wrestling with some of these questions about how do we make sure that our designs are ethical? You know, how can we even determine if something is good to do and what are the unintended consequences of this? And also, you know, are there absent voices in the process, you know, whose voices are being represented and who are the decision makers, you know, who is part of the negotiating table and all this kind of stuff as related to policy and what tech ends up being used, you know, within schools or at home or whatever. certainly all. You know, I think it's like a very timely, very important moment in time, I think, to definitely continue to build upon this kind of work. And it's great to see folks everywhere kind of contributing to this. And definitely you're at the top of the list. So definitely want to thank you, Kent, for your work in this space. regarding the E3XR framework, you know, part of it I think was motivated by getting really frustrated, I think, with how people would make generalizations, a lot of assumptions, right? Like oftentimes we'll see the silver bullet kind of mentality of VR will definitely lead to wonderful learning, right? Or games are going to be great for education or VR will definitely be a wonderful, magical technology that's great for everybody, right? And just not enough nuance, not enough careful attention to how do we ensure that learning is successful? How do we account for learning theory and findings from empirical research, accounting for cognitive science principles and multimedia learning theory and all these kinds of things. So psychology as the foundation to ensure learning, but strong learning is not enough either. If something is not enjoyable or not fun or not a good experience, then it's also not successful. And then the ethics side of things, For sure, as I mentioned before, dramatized in some of these sci-fi films and TV shows, a lot of ways things can go wrong. I was struck by data collection. Was it like the 2 million points of data that's collected in one short VR experience? And just all the ways that data could be misused. Let's say like the insurance company or somebody could totally twist that in ways that we wouldn't anticipate. Or just a lot of ways where we're not really thinking about where this is all headed. and some of the unintended consequences or intended consequences in some cases, right? Where we have people trying to serve their own interests, but definitely not the interests of the user, but kind of like that dark UX pattern sort of thing. So we definitely wanted to consider the ethics side of things. And then the Eudaimonia angle, human flourishing kind of perspective, This is definitely something that's ingrained more so, I guess, the social justice and equity emphasis of our institution at Teachers College, where there's plenty of ed tech that's out there, but there's often concerns and challenges with access that's not equitable or other kinds of issues where it exacerbates some problems, right? You know, definitely I guess that's the spirit of what we were trying to address at Teachers College. But we wanted to integrate that as part of our framework as well, because a lot of tech makes the privileged more privileged and more powerful and worsens some inequities that currently exist. And, you know, we definitely get into that in some of the sections within the human flourishing section. Elliot, do you want to add to that?
[00:14:07.868] Elliot Hu-Au: I mean, just like a very specific event I remember that pushed me to want to contribute with Joey was when Facebook started requiring people to link their Oculus accounts with their Facebook account. And I just remember thinking from a former teacher's perspective that it just really made no sense because in schools, they don't allow kids often to check Facebook for many good reasons. And then just making that decision without really thinking about the users, both for children who might be wanting to use it or just marrying this social media thing with this technology didn't seem like a very good way to really roll it out. And so for me, I was like, okay, yeah, something needs to kind of hopefully push people in the right direction. If we want to get this to be an educational tool that can actually be useful and used, then it needs to be done in a way that can protect kids especially.
[00:15:08.385] Kent Bye: Yeah, well, I feel like that there's a good survey of a lot of the different learning and learning theory. And I want to get into that. But before we do that, you kind of start with the ethics. Well, first of all, you have a pyramid that you have here where you have like, the ethics do no harm, education, effective learning, and then eudaimonia, which is ascending to your full potential, that kind of flourishing, the human flourishing, kind of like the Maslow's hierarchy of needs in some way.
[00:15:32.578] Joey Lee: And I get that parallels the Maslow hierarchy in some ways. Yeah.
[00:15:37.263] Kent Bye: It's kind of like that self-actualization is, you know, take care of the basic things. I guess the thing that I wonder is, you know, the difference between eudaimonia and ethics, because a lot of times eudaimonia is seen as an ethical framework. Some ways the ethics is like the core privacy, autonomy, your body, but then there could be other aspects of eudaimonia that's the collective, but there's also agency in there. So for me, I guess I have trouble differentiating when you say ethics and eudaimonia, how do you draw the line between what's ethics and what's flourishing in eudaimonia?
[00:16:07.525] Joey Lee: I mean, certainly one could make the argument that there is overlap or that eudaimonia could potentially be viewed as a subset of ethics or something like this. I definitely think that's potentially a reasonable way to categorize that. For us, I think it's partly to place some extra emphasis on the idea that eudaimonic kinds of values and support one's human flourishing and these kinds of values of inclusion and representation and equity, these sorts of things. I guess, I wonder if it's possible to have something be ethical, but not eudaimonic? Like, I would say probably yes, right? So it's like, maybe there's a way where we want to play some extra emphasis to try to promote more of that human flourishing kind of notion.
[00:16:54.376] Elliot Hu-Au: I mean, when I first it as like you done when it's like the broader goal, I guess, like the ethics part is in some ways like the very specific actionable things that we can do, like protecting privacy and data and all of that stuff. And then do those things contribute to helping people either generally or also as an individual, does it contribute to them achieving their full potential?
[00:17:24.428] Joey Lee: Yeah, that's a good way to put it too, right? Because the learning is also in service as sort of like another step to kind of help someone reach that full potential kind of the top of the pyramid as well. Like again, that similar to that Maslow model, transcendent to the highest level of reaching one's full potential.
[00:17:42.667] Kent Bye: Yeah, I think that makes sense. And you know, one of the things that I appreciate is, you know, with any framework, you have certain trade offs that you have to make. And I, you know, sometimes I feel like when I did my extra ethics manifesto, when I finished it, I was thinking about it. And I was like, Oh, wow, this would actually be impossible to achieve every single most ideal suggestion that I had made there, just because there are some of these different contextual domains that are in contrast with each other. And so you almost have to pick and choose which aspects of things that you want to do. And in some ways, Facebook has started to do that with the responsible innovation dimensions, which is also kind of looking at these sort of existential trade-offs. And just as an example, like for surveillance capitalism, on one hand, that is potentially taking away someone's autonomy or agency, But at the same time, it's also bridging the digital divide, meaning that there's a business model that is making their services more widely accessible. So there's things that are making the technology more accessible because the price is coming down, but it's at the cost of privacy. And so there's some ways in which that some of these ethical frameworks, for me at least, they have these inherent trade-offs between each other that are in more of an ecological context. you know, rather than something that you could fully optimize and do something perfect, because I feel like in some ways, part of the ethical dilemma is, is that you can't do it perfect. And so you have to figure out where you're going to cut those corners.
[00:19:01.547] Joey Lee: Yeah, I think that's a great point. I mean, you know, business realities and, you know, I was thinking about like Bacha Friedman's value-sensitive design, right? You have different stakeholders and some stakeholders win and others lose. And there's trade-offs as far as, you know, who do you prioritize? Like which stakeholder gets the benefit when you make a certain design choice? right? And so part of our motivation behind this framework too, in trying to create an analytical framework, it was actually kind of neat how we could see almost like a mirror image, you know, down in like figure four of the paper where we have almost like three continua with like low ranging to high. you can definitely come up with examples that don't do a good job accounting for ethics, right? And so it would be more kind of the stuff on the left side of the spectrum, whereas there may be deceptive, dark UX kinds of strategies. So they may be relying on addiction or surveillance capitalism and dopamine hits and getting a person kind of on that social validation feedback loop. So trying to provide a way to analyze existing XR experiences, like where on the continuum would something fall, right? And by trying to make things a little more transparent or more overt in this way, hopefully that provides a valuable tool that can be not just like an evaluative assessment kind of analytical tool, but maybe could lead to design principles or could lead to something a little more prescriptive as far as if you're setting out to create something new.
[00:20:35.050] Kent Bye: Well, maybe it's worth going through some of these polarity contrasts, both in ethics and eudaimonia, and then we'll dive into some of the educational stuff. But I do think it's interesting to look at the polar extremes of not taking an ethical approach or not optimized for eudaimonia, you know, what ends up happening. And so maybe we could go through some of these different contrasts that you have in this figure four and just talk through them a little bit.
[00:20:56.446] Joey Lee: Yeah, I mean, so in the paper, we have several bullet points where we go through and contrast. I don't know if you'd call them like trade-offs, but it's kind of like contrasting. We call it the double-edged sword where it's kind of like there could be certain benefits of XR, but it's like if you don't do things the right way, then the mirror image potentially could also be true. And there's been some interesting papers that I've read imploring folks to think about what are the ways that VR could be used to deliberately cause harm, like torture in VR or military use of VR or AR to dehumanize the enemy or what would a sexual assault In VR, you know, there've been some interesting papers that we've mentioned here as well, kind of like depicting virtual rape and things like this, like what would that entail? We also talk about the Proteus Effect, which is a phenomenon studied by Jeremy Bailenson over at Stanford and Nicky and others, in which people tend to conform with the expectations of an avatar that they use. So there were several research studies that they did where taller avatars, when they stepped into them, they ended up behaving more aggressively in the virtual space. But also afterwards, there was a sense of transfer in the real world where they also continued to act aggressively. And then they did similar experiments with attractive avatars and other kinds of avatars. And they ultimately found that people tend to conform to those expectations of what that avatar represents. So that's both an opportunity for positive behavior change, I guess, but also negative, potentially problematic, maybe reinforcing stereotypes or other kinds of issues with that. know, from the more positive angle, we were thinking about how could these kinds of experiences make an impact in terms of self-identity? Like if we want to, for example, promote more science and technology and engineering and math for underrepresented groups, you know, certainly meaningful and relevant STEM experiences where you play the role of someone performing a scientific discovery and you meet Marie Curie doing these amazing kinds of things and making the science more personally relevant. that could certainly be an opportunity to create those more positive identity experiences. But I'm kind of talking about, maybe I'll pass it on to Elliot, maybe if you want to continue on talking about some of these other ethical points that we were discussing.
[00:23:24.119] Elliot Hu-Au: Yeah. I mean, actually I was just thinking of an example again, where it's sort of like a double edged sword was again, going back to Facebook, it seems, but that event where Mark Zuckerberg and then someone else who worked with him, they did this like slum tourism almost kind of thing, right. Where they did a Facebook spaces demo and visited Puerto Rico right after they had the hurricane. And on the surface, it seemed like, Oh, this is great. We'll expose people to see kind of like how much the devastation is. I'm in this place and we'll give them a sense, you know, that's more immersive by having it be a 360 experience. And so while there's an opportunity there, it turned out to be very poorly done because in a way it just made it look more like they were kind of just ogling, you know, what was going on. And there's this very large disconnect with this billionaire guiding through this place where all these people just lost their homes and things like that. So I think, again, it's like designers and developers have to be very careful with the experiences that they make as best as they can. No one's going to be perfect and do it right the first time, but to be able to think about as they're designing it, what could be the possible ramifications of us doing this? Who are the people that we're going to be involved? Like Zuckerberg said, I don't know, maybe let's try and have someone who's actually there in person to talk about what is going on in this 360 experience, rather than just me sitting in Menlo Park or wherever I am, just jauntily running through this place.
[00:24:56.547] Joey Lee: Yeah. I think it's a great point. So like, you know, Carter and Eggleston talk about that, right? Like who sets up the camera, do you call this slum tourism? But yeah, like these empathy based experiences, right? Where Chris Milk and others who've talked about the power of VR to deliver a very powerful empathy based experience, oftentimes there's no danger, I think, of promoting that kind of hero complex and trivializing and really not doing justice by putting in that kind of savior-oriented perspective and really not telling the story. It's like, whose story do you really tell and whose reality are you sharing and who is part of that storytelling process? Like the way that Carter and Eggleston talk about, ideally, we would want folks to tell their story and to empower folks to share their perspective as opposed to us framing things through our egocentric perspective. Yeah.
[00:25:52.318] Kent Bye: Yeah, when I read through the descriptions at the bottom, I'm just going to read through both the ethics and eudaimonia, because I think it gives a good survey of all these different aspects from ethics do no harm with privacy, transparency, responsible data use, risk minimization, autonomy, disclosure, consent, and psychological transference. And so on the negative side, you have negative transference, addiction, data misuse, surveillance, capitalism, exploitation, deception, and unnecessary risk. And on the positive side, you have positive psychological effects, self-regulation, transparency, privacy, autonomy, and safety. You know, it makes me think of the neuro rights, which is like, you know, the right to yourself, your identity, the right to your mental privacy, the right to agency. And there's kind of these fundamental things that you're talking about, the privacy and autonomy. There's aspects of the eudaimonia, which are also in that in terms of inclusion, empathy, equity, accessibility, representation, identity, self-regulation, and persistence. There are still aspects of autonomy identity there that are sort of eudaimonia. You know, for me, again, it'd be hard for me to say identity is more ethics versus eudaimonia. We have the lack of eudaimonia, the lack of diversity and inclusion with absent voices and loss of agency, data exploitation, limiting stereotypes and perpetuation of inequalities. And then on the positive side, identity, support, positive selves, inclusion, accessibility, empathy, self-regulation and human flourishing. So I feel like that there's a lot of those contextual domains that are there. You know, what's, I mean, at first it was like the individual versus the collective or You know, if you're thinking about Maslow's hierarchy of needs, declaring that these are the baseline essentials versus the other ones that are nice to have are not towards self-actualization. So for me, it's harder to separate those into those two groups, because I think they're all similar in some sense, but yeah, it's just interesting to see how you kind of broke it down there.
[00:27:38.763] Joey Lee: I think that's fair. I mean, definitely, you know, like I said, I think there's definitely a lot of overlap. Maybe it's a limited framing to kind of be like ethics simplified down to do no harm, as opposed to eudaimonia being almost like building up something that wasn't there before and removing barriers and allowing growth of something that would not have existed before. Like actually proactively planting the seeds and watering the plants and allowing that to grow. But to me, at least in my mind, there's something a little more active about supporting that as opposed to just making sure that something is done the right way. Yeah, which maybe that's not necessarily a fair characterization of that, but one feels a little more wanting to do it the right way. Of course, you have to be active in guiding that ethical designs do not happen naturally, right? We have to set that as a goal and be very careful about it. And as you said, there's always trade-offs too where Yes, sure. We would love to give full data control and ownership to the user, but maybe then we can't do personalization or maybe we can't have these other features or whatever. So I can definitely see some challenges, but yeah.
[00:28:50.178] Kent Bye: And if you have any other thoughts on the whole list of all those things, anything else that sort of jumps out?
[00:28:56.167] Elliot Hu-Au: No, I mean, it is a difficult thing to figure out how to separate those two concepts in ways that it's very, very easy to bifurcate it. And I think, again, in my mind, I like to think of it as, you know, the actionable steps, sort of like what we're reaching to or trying to get to in the broader context of humanity, of individuals. But again, I mean, I'd say we designed it to be like a model and, you know, all models are limited, but, you know, they hopefully lead us in a direction that will actually be helpful. So, I mean, it would be great to have someone else in the future take this on and then for something that is even more exact or more comprehensive. But we hope it can be a starting point.
[00:29:43.624] Joey Lee: Right. And I think there's a lot of unintended consequences or stuff that maybe designers don't really anticipate as they are creating XR. Like for example, with Pokemon Go, right? Like certainly there was a lot of writing about how there were accidents and people would wander around the streets and maybe like not pay attention and maybe, you know, they may cause issues with oncoming traffic or these kinds of incidents. But beyond that, in terms of politics of place and space, folks who have never had to worry about physical safety and danger. Let's say if they're wandering to a certain neighborhood, as opposed to a person of color, where it's very different, where they may be seen as trespassing or a threat or something like this. There are a lot of unintended consequences and danger that can arise if a person doesn't really take into account all the different kinds of people who will be playing the XR or the AR game in this case. so the need to anticipate some of these kinds of situations where these kinds of unexpected issues can arise. And I think we already kind of talked about the surveillance capitalism and the dopamine-driven addictive models and some of the ways that leads to some ethical concerns as well, that kind of exploitation of one's data, things like that.
[00:31:00.622] Kent Bye: Yeah, in some ways, what I think about is when someone creates an experience, they may not have the resources or the momentum to take care of all of the things in terms of inclusion or accessibility. It takes resources to be able to take into account all those things. And sometimes you have to start with a baseline and then maybe expand into that in terms of of making things more and more accessible as an example. I think of like an immersive theater show that may not be fully accessible, but then once it gets big enough, then maybe they could start to find ways to support and do the hard technological transitions that haven't really even been fully developed yet. So I feel like that's one way that you've kind of broken it out is here's the baseline of things that we should all be taken care of. And then the sort of nice to have you doing money as although, you know, for some people who are on the other side of that, they don't have fully abled bodies for them. That may be a baseline ethics thing, but that for the creator, it's not possible for them or feasible for them because there's only limited amounts of time and money to be able to produce these experiences that it becomes something to sort of add on top later, to be able to make it as diverse and as inclusive as it possibly can be.
[00:32:11.111] Elliot Hu-Au: I would say too that for designers, just to even be aware of these things and that they're aware that they might not be addressing all of the issues is still a great step in the developer process.
[00:32:22.165] Joey Lee: Right, right. I mean, like in terms of co-design or having more folks at the table who would be able to raise the point, hey, I don't really know if this is a good idea for folks to actually wander around in this neighborhood where potentially could be dangerous, you know, just to have those voices represented in the design process or to have a set of design principles laid out somewhere where a person could reflect upon their design and to make sure that it would not cause any unnecessary risk. I think that would be important.
[00:32:50.714] Kent Bye: Yeah. And I wanted to move on to the middle part, which I think is the big heart of a lot of this, which is the education and effective learning and the learning theory. You have a nice table that describes different aspects of behaviorism, cognitivism, constructivism, and connectivism, but also in this paper, you do a nice survey of evaluating what's known about how effective XR is in terms of you know, like you said, you don't take the silver bullet approach, just throw XR at it and expect that you're going to get better learning outcomes. So maybe you could set the context a little bit in terms of being your background, being at Teachers College and looking at a lot of this learning theory, you're kind of pulling in a lot of the latest of the learning theory. So maybe you could give us a little brief overview of what does XR change or modulate or what's the best approach to sort of think about different learning theories, and if there's one of these that, like you said, you know, constructivism, or if there's one of these that are particularly well suited to be able to use the contextual nature of XR to be able to use the affordances of XR to its full potential.
[00:33:49.353] Joey Lee: I mean, I feel like to start that discussion, I feel like it should be noted that a lot of ed tech is made backwards where it gets built first and then they later try to bring in consultants to then get them to say that this is great for learning. Or ed tech, in many cases, the values of let's say like ed tech startups or a lot of ed tech that's out there are not necessarily on the same page with actually supporting good learning. We kind of see this with business models of a lot of ed tech where, I mean, I guess there's just a conflict in values in many cases. That is to say that there needs to be a lot more high quality, in this case, XR that makes sense in a formal educational context, like actually works well within classrooms and can be valuable and something that educators can use their context for if it's kind of like blended learning kind of model, or if it's like an at-home model, if it's more about the lifelong learners or folks who are just downloading an app at home and just trying to play with something and they learn some science or whatever it is. There's a lot of bad ed tech out there that may treat VR or AR as kind of like the latest gimmick, like the latest kind of hype and therefore does not really take into account solid learning theory. Right. We see this a lot. Yeah. I mean, I published a couple of papers on this kind of topic, like five years ago, I published a paper on language learning games and how so many of them are about like flashcard-y kind of drill and practice type you know, tap the right picture kind of stuff. And that's highly behaviorist, right? And highly drill and practice oriented, and not really taking advantage of what learning scientists view as potentially a better way to support deep learning, you know, the 21st century learning skills that are needed today. And that's where VR becomes very exciting, right? Because well-designed VR naturally is about being immersed in an environment and active learning and learning through experience and learning through exploration. And it seems like the natural affordances of the environment lends itself very much to a constructivist learning experience or even a social constructivist learning experience. where you can actually interact with people all over the world and share expertise and share knowledge and, you know, model behaviors and a lot of procedural knowledge and sharing like tacit knowledge and what we call like cognitive apprenticeship, right? Like modeling behaviors as part of a community of practice, right? Elliot, do you have any?
[00:36:34.807] Elliot Hu-Au: Yeah, I agree. constructivism is a learning theory that can support a lot of the affordances of virtual reality. And a lot of those are based upon like sort of open ended, explorable kind of things where people are able type of things out there are easily supported by constructivism, as well as, I would say, the cognitive theories out there. And body cognition is a great way to see how learning can be really helpful or can be very effective through virtual reality. And body cognition is this idea that it's not just sort of your mind that is doing all of this sort of thinking and knowledge building for you, but it's your senses, all of your senses, your ears, you know, your hands. They're not just sending things to your mind to be processed up there, but they're actually in some ways holding some of that knowledge in what they are. It's kind of like muscle memory, if you think about it that way. Mirror neurons, you know, when they see somebody doing something, it you're doing that same thing, even though you're not actually doing it. And so there's some kind of connection between our physical bodies, as well as the things that we're thinking of that in virtual reality, you know, if you're learning about some topic like math, for instance, that you're able to involve your hands, at least for now, right? Your hands. And maybe like, if you have feet trackers on, you can use your feet. In some kind of learning experience that could help forge more connections to that concept. than would be able to be done if you're just like sitting in a desk, you know, writing on a piece of paper. So things like that are very exciting to me, I think, with VR and what could be done sort of in the future with how we can help people learn about these kinds of things with these new tools.
[00:38:24.533] Joey Lee: One thing that I kind of worry about, and I see this a lot with educational games, not necessarily VR, but when educators oftentimes want to use games or VR, AR, they often start with transmissionist kind of model, right? Like too often they try to take a textbook type content and then gamify it or Like, I've seen a few VR experiences like this too, where it's really like a ton of content is really just sort of like, you're like drowning in this content and you just hear a lot of voiceover or even worse is like a ton of words and you're really just like reading walls of text within VR. And it's really a huge mistake, I think, to try to make like a textbook in VR format. and really not taking advantage of the benefits of VR and the affordances of a much more spatially oriented, immersive kind of context. That's part of the motivation or the reason why we put in those learning paradigms in that table in the paper, because it has to do with one's assumptions of how learning works or like, you know, one's assumptions of what is the nature of learning and what is the role of the learner and how can these kinds of learning experiences be delivered in ways that will lead to other kinds of outcomes, right? So maybe there is a time and a place for like drill and practice flashcard type stuff, you're trying to build automaticity or learning basic primitives if you're learning a foreign language or something like this, right? But that's far more limited compared to giving a person a context where they're transported to a foreign country and you're already interacting with locals in a different culture and being immersed in that kind of context. So we wanted to contrast that and show you some examples of how the learning process looks different and how it takes place differently depending on your worldview of how learning takes place.
[00:40:27.895] Elliot Hu-Au: I was just gonna ask you that one of the sort of fun things about thinking about VR and what you can do with educational experiences is that it gives you such control over the environment that you place the learner in, that you can really fine tune what you want them to learn and have it be supported by different things that they can interact with in the environment. You know, whether it's NPCs that you might use or just actual things that you can put in the environment. you can really make something be less distracting than maybe if they were trying to do it in the real world, you know, just a lot more effective.
[00:41:03.123] Joey Lee: You know, what that reminds me of is Jeremy Bailenson's study of like, he did a bunch of experiments where he tried to see what happens if you give students the best seat in the house, like the best seat in the classroom. So he did some experiments where, if I recall correctly, basically he made it such that it was as though the student were sitting in the second row of the classroom, like very close up front and had like good eye contact. with the lecturer. And that's what's cool about VR, right? We can give these optimal, like every single user can have that optimal learning environment. And we're not bound by a lot of the constraints of traditional formal education context, but even better than a VR classroom is actually putting them on site to institute, you know, actually doing, let's say like data collection in nature to learn the scientific method or whatever it is, or to test hypotheses and to see why is everybody getting sick and learn science through very realistic, authentic experiences in VR.
[00:42:04.892] Kent Bye: Yeah, as I look over the different learning theories and approaches, it seems like potentially like the experiential design, if it's really well done, could actually incorporate all aspects of these. So just as an example, you talked about the textbook and, you know, just translating things over into a spatial experience. That's the challenge, which is like how to create an immersive, embodied, interactive experience that unfolds over time. But this constructivism, which is the discovery, the active sense maker, so it's active learning through experience. So being able to take the content and create an experiential dimension where people do have the opportunity to kind of tinker or discover or play with things and have a prediction in their mind what's going to happen, but actually have an opportunity to see what happens. Almost like a chemistry lab as a good example there. But you know, those hands-on experiments that you mentioned here,
[00:42:51.963] Joey Lee: You know, speaking of that, so Elliot, as part of his dissertation that he's working on, he's actually created his own virtual reality chemistry lab. And maybe speak more about some of the experiments you've run and also the different variations of the lab and different modes and things like that.
[00:43:09.009] Elliot Hu-Au: Yeah. Well, it's funny because I'm actually writing on a response from the Oculus app lab because I wanted to have them hopefully send it out to people to try it. But yeah, it's a lab that I designed. I taught myself C-sharp and used Unity to create this lab that was sort of a hands-on, it was a simulation actually at first of the chemistry lab that students could do chemistry experiments in typically. And the idea was that it would be an experience where students who didn't have normal access to real chemistry labs, they could still get an experience doing chemistry experiments that way. in these sort of funny COVID times that actually became a little more relevant because, you know, for a whole year, students couldn't go into school. And so while I didn't get to actually send it out and have people test it as much, it still became this like relevant idea of like, oh, well, this could be one way where at least I wouldn't want it to necessarily replace science experiments. You know, as a former teacher, that goes against kind of what I was hoping for. At least it would give students a better experience than nothing, being able to try out science experiments. you know, allows them to do, for instance, that rainbow flame experiment, where if you remember, I don't know how long ago it was, but in Detroit, I think a kid actually got burned because his teacher sort of did it in a wrong way. And so, you know, that kind of idea where students are able to do experiments that could be very dangerous, at least in VR, you would be able to experience it without those really permanent consequences. So yes, I think it's a great idea.
[00:44:44.938] Joey Lee: And you recently published a journal article based on that project as well, right?
[00:44:50.545] Elliot Hu-Au: Yes. Yeah. And I got published, I think earlier this year, some of the results from my pilot study, which were beta that students who were doing the virtual reality experiment learned basically the same amount of content that students who did it. in real life. So it was helpful to see that at least establish a baseline that a virtual reality simulation of a chemistry lab could still provide a lot of the same knowledge potential that a real life science lab could.
[00:45:19.458] Joey Lee: Yeah. And when, when folks have been using your demo, they also enjoy like the more fantasy based game modes where you zoom into the size of a molecule. And there's also like the stress relief mode where people destroy beakers and glassware and it's highly satisfying to destroy glassware in VR.
[00:45:38.283] Elliot Hu-Au: Yeah. It's quite funny how much people want to break glass and things, I guess, in an environment where they maybe previously were restricted from doing those kinds of things. And then a lot of times, one of the first things they want to do is just destroy everything. And I'm like, go ahead. Cause it's very easy to clean up. Just, you know, hit reset.
[00:45:58.289] Joey Lee: Yeah. But it was interesting, definitely, you know, comparing a very realistic VR chem lab and then just seeing how people would play with like the abstract or the more, um, you know, making yourself the same size as the molecules kind of mode and more game-like kind of modes. doing more of the impossible stuff that you can't do in real life is also interesting.
[00:46:20.461] Kent Bye: Yeah, well, I wanted to go through these four different types of learning theory and just give some associative links that at least I have as I'm looking over it, because like I was saying, that does feel like that a really good, robust experiential design could actually include all of these. So the constructivist aspect of the hands-on experiments that you're able to do, we talked about that a little bit of that embodied experience to be able to actually have the hands-on and express your agency and have an action and see what the reaction is. Cognitivism. You talk about a VR memory palace or mind map to visually connect ideas. And that's something that's a powerful part of being able to connect the dots between different aspects and use the environmental design in that sense, or just create a context under which you're able to make these different connections. The behaviorism, it sounds like there's different aspects of gamification and the point system, the rewards could create an external motivation for people to keep them interested and motivated to keep going. So there's certainly a lot of behavioristic aspects of game design. And then the Connectivist, you had mentioned different ways of interacting with virtual beings. People having some social connections can create learning chains for their more of a social dimension that we're just kind of social creatures. So remote collaboration with experts and holoportation. So it feels like there's elements here of like either the social presence and a body presence and active presence and the environmental presence and creating overarching motivations for people with gamifying things and putting points and unlocking different things.
[00:47:46.645] Joey Lee: Definitely. I definitely think, you know, I could envision a lot of these learning paradigms working in concert to achieve desired results. Right. And especially, you know, some of the, um, what you just mentioned with like connectivism, because, you know, learning takes place within communities, right. And the scientific discovery certainly within conferences and communities and various disciplines I'm a big fan of, for example, science VR, Jackie Lee's company, where he has some pretty interesting projects interact with famous people throughout history that have meaningful scientific discoveries. But as VR becomes more and more social, as there's tons of stuff nowadays that we've been seeing in alt space or I guess like Facebook Horizon and some of these other platforms, I could definitely see subject matter experts dropping into your VR space for class, right? And having that be a really powerful class experience, it's like, Hey, we brought in this like NASA engineer and he or she is going to come in and give a talk for a few minutes here in VR. Like that's amazing, right? Just being able to seamlessly drop in and out and share knowledge, you know, I think everything's kind of moving that way. Like with Zoom being so dominant in the past year or two, there was like this hunger and demand for something better than Zoom, something with more social presence, like playing with Spatial and some of these other platforms. It just felt like holographic XR made a lot of sense. as far as more natural kinds of experiences, more intuitive kinds of experiences that were social. And my guess is things are going to trend more and more this way where XR for education is going to be a bigger and bigger deal. And stuff like I just described with really powerful guest speakers as lecturers to pop into class in VR or simultaneously broadcast to all kinds of places. or interactive rather, not just passively broadcast. I think there are a lot of really cool possibilities with that.
[00:49:48.473] Kent Bye: Yeah, well, there's a one paragraph in here that really jumped out to me, at least in terms of all the different surveys of different learning theories. And I'd love to read through some of this and have you comment on some of this. You said in a systematic literature review of educational VR research, Ridianti found that most studies on educational VR focus on usability rather than learning outcomes and that the learning theories don't actually guide the design. And then Loke identified the most commonly adopted learning theories, including experiential learning, situated learning, social constructivism, constructivism, presence theory, flow theory, projective identity. In addition to those theories, you have the multimedia learning theory, cognitive load theory, embodied cognition, which we talked about briefly, preparation for future learning, and provide some important theoretical perspectives for strategies and guiding principles. That seemed like a pretty good survey of a lot of different learning theories. So I'd love to hear some reflections on that.
[00:50:44.159] Joey Lee: I mean, my personal take, I don't know if you agree with this, Elliot, but I feel like this space, especially VR for education, is relatively under-theorized. That list you just mentioned there, there's a lot of hand waviness. where people are like situate learning, social constructivism, presence theory. In that study, basically the vast majority of research just cited those theoretical lenses and didn't really get too much more specific beyond that. And that's part of the problem, I think, that we're wrestling with here is Again, that hand waviness is related to that silver bullet kind of mentality of VR automatically or automagically will be good for learning, right? And so, that's why I really like the studies like Markanski and his colleagues that found in their study, the VR experience actually led to lower learning because of extraneous cognitive load students found that to be distracting and they actually didn't learn as well compared to a control. They found it enjoyable. They found it a lot of fun. They reported high levels of engagement and enjoyment, but they actually didn't learn as well as effectively compared to the other group. So that's really problematic because Ultimately, we want to get to a point where we have solid design principles and a solid body of research that can lead to XR for learning that's successful and not just like VR is magic. It's definitely great.
[00:52:16.515] Elliot Hu-Au: I would say, yeah, I think we're getting to the point where the research is starting to parallel with the development of VR headsets and maybe as VR headsets become more accessible to consumers and therefore also like researchers, researchers can probably do more research that is very specific in terms of like what kinds of learning theories do they actually look for, you know, using virtual reality as a tool. Whereas before, you know, with the Radiante Literature Review, even though it was published in 2020, you know, it might've been started like in 2017 or it takes years sometimes for these things to happen. And so then it could have been earlier on that they were looking at VR research that is still just sort of trying to establish like a rationale for using VR in educational spaces. So then they're shooting out, you know, a bunch of different learning theories, you know, to say like, look, possibilities.
[00:53:11.062] Joey Lee: I think that's a great point, right? Because when you think about usability and cost of VR headsets, for example, I remember not long ago, quite recently when Elliot and I had to do demos at Teachers College, it was such a pain to set up those demos, just to cart a very heavy, powerful desktop machine, have it tethered. It was not long ago, right? The quest has not been that long. standalone six degrees of freedom, VR headsets have not been around all that long. And therefore, to be able to use VR in classrooms, it really isn't that widespread. It really takes a savvy, dedicated educator, or researcher to make that happen. And it would have cost a ton of money. And oftentimes, you know, teachers don't really have a high budget or even like a decent budget to put this kind of technology in the classroom. So I think things are gradually changing, right? As the cost goes down and, you know, certainly you have a lot of enthusiasts. They're in love with VR and AR and AR certainly because everybody or many people, most people have smartphones that are AR capable, certainly that's bringing a lot more of these experiences to the classroom and making it way more within reach. But still, we can't take for granted or assume that people can afford a $500 smartphone and things like this too. But all that to say, I would imagine there is now in recent years, there has been much more of an explosion of AR and VR for education kinds of applications, even like Google and some of the big tech kinds of companies have dabbled in it. It was a little disappointing when Google expeditions closed down, but there are creative folks who have been making things happen as far as taking the existing tech and still providing some pretty meaningful experiences for learners. So if there were new systematic literature reviews, I do still think the theory basis would still be the same. There's really not going to be a lot of new stuff. And that's why we were suggesting a few additional perspectives that could lead to different research questions. It may lead to different kinds of designs for the actual AR, VR experiences too. For example, with projective identity theory, if VR, AR could be a tool to support or construct the kinds of identities that are more necessary for the 21st century, like you mentioned things like STEM identities or new possible selves, how could that lead to a different kind of XR experience to get a person to value the same kinds of values from that epistemic frame, right? So whether it's like science is obviously the go-to, that's very obvious, but maybe it's something else. Maybe it's fake news and journalism and how do we train people to think critically through an interactive experience in VR, AR, right? Or then they start to view themselves as a journalist or as a critical thinker or something like that. And by giving these experiences, how do we shift identity in that way? That's very interesting to me. And that wouldn't happen if you only stuck with like a certain theory and didn't really consider other possible ways to lead to some more interesting research questions.
[00:56:22.512] Kent Bye: Yeah. As I hear you respond to that paragraph, what immediately came to mind is like, well, what is the good research that's out there? And I actually table three as the low consideration of educational efficacy and the high consideration of educational efficacy. So it does look like you've filtered down to at least six or seven or eight different studies or concepts or ideas in terms of germane cognitive load and constructivist learning experiences or social learning or dynamic visualizations, just-in-time exploration and surroundings and situated learning. perspectives that transcend space, time and place and being able to spark student interest and engagement in preparation for future learning. So it looks like some of these are the ones that you were pulling out in terms of these are the things that seem to be working well in the context of using XR for education.
[00:57:10.500] Joey Lee: Yeah, absolutely. I feel like it's neat that we were able to articulate some of these contrasts. It's a little bit of a caricature in some ways. It's easy to criticize traditional learning, traditional education as being transmissionist, didactic, and very isolated facts and this kind of stuff. The reality, in some cases, is not so extreme. at least for the purposes of this table, it was useful to contrast kind of like what would be not really accounting for learning theory and what do we consider to be poor learning experiences or poor ed tech, and then how would one take advantage of XR to then provide what the learning sciences research literature would describe as better learning or the kind of learning that's more needed nowadays, like higher order thinking skills, et cetera.
[00:58:03.757] Kent Bye: Yeah. I don't know if you had any other thoughts on those contrasts there.
[00:58:08.598] Elliot Hu-Au: Um, nope. Uh, I, I just agree. I think, I mean, this is the area that we specialize in, so it's always good actually to just like, you're a fresh on them and just how helpful XR experiences can be. And, you know, again, these are just things we can try and aim for and hopefully they be useful to educational XR experience designers. They can use them to make effective. experiences for people.
[00:58:35.350] Joey Lee: Or to take from existing XR experiences and to use them. If they're not a designer, at least they can better evaluate existing XR experiences, you know, at least have some way to consider, you know, is this going to be good for learning or not? I mean, I feel like teachers sometimes intuitively can get a sense, but it would be great to have some resources out there. Be like, well, this is good for learning for this reason, or this can help with this learning objective, or use this for this educational standard, right? Like if teachers have to align to those particular standards.
[00:59:10.808] Kent Bye: I'm going to be giving a talk about XR ethics and education, as well as kind of just some strategies for, you know, what should higher education places be thinking about in terms of starting to adopt XR technologies. And I'm going to be trying to take what I know about experiential design and sort of pick up from what I understand, at least from some of the work that you've been laying out here, these different surveys and the best practice in learning theory may be, but be curious to hear your take in terms of what advice would you give to other higher educational institutions that may want to start to pull in different XR technology and where would be a good place to start in terms of like here would be a good easy win that would grab some learning benefits, assuming the technology is available and that it's worth all the logistics to getting it there. But The aspects of presence and all the other affordances that we've been talking about here in the course of this conversation, everything from body cognition to situated learning, what would be some tips that you'd give for other schools or higher education institutions to be able to start to adopt XR technologies in the context of using it for education?
[01:00:15.672] Joey Lee: That makes me kind of think I should send you our paper from a few years ago where we focused especially on VR and its opportunities for education. That one's a little more big picture, but yeah, I don't know. What do you think, Elliot? Do you think that would be irrelevant?
[01:00:31.817] Elliot Hu-Au: Yeah, I mean, definitely. I think, well, a specific area that I have interest in is just any topic that deals with like abstract concepts, science and math, like concepts that are often difficult for students because there's no real tangible representation. And if you can find like a VR experience or even AR experience that can give a visual or again, tangible representation to these abstract concepts, then it makes learning those things a lot easier for students.
[01:01:05.206] Kent Bye: What were some of the other takeaways in the paper that you wrote, Joey, in terms of the suggestions that you were including there in that paper?
[01:01:12.362] Joey Lee: I think we kind of built up this argument that a lot of the existing challenges of education kind of are naturally addressed through well-designed VR experience or AR experience. You know, so like a lot of education, traditional education, there's challenges with personal relevance and you have folks who struggle with abstract concepts like Elliot was saying, or that possible selves identity angle, I think is really important, right? So it's like, you could try to teach something on science, right? But if a person just reads from a science textbook, it's sort of detached from why that matters in that student's life. So by giving more powerful kind of hands-on firsthand experiences through VR, you know, you could definitely deliver much more relevant experiences that align to a person's academic identity. There's a lot of great work by Nayela and Nasir and others on identity and how that relates to games. I mean, definitely could be extrapolated to VR for sure. But VR creates those meaningful learning contexts, right? And it's not just a person's identity in relation to that context, but it's the constructivist learning opportunities and the situated learning opportunities. But definitely, I think the engagement angle, certainly very important too, right? And making the subject matter relevant. Also thinking about John Bransford's idea of preparation for future learning. You know, sometimes it's not about like hitting people with a textbook and having them understand all that content, but sometimes it's about sparking new interest, which then provides that very strong motivation for a person to then want to learn more and then see the relevance and then, you know, go off and It provides that foundation to then spark a lot more class discussion, or you can give it like a shared experience for folks to then reflect upon that can then lead to a lot more germane, fruitful class discussions and lectures and whatnot, or things like that. So that preparation for future learning kind of idea is actually pretty powerful as well.
[01:03:25.499] Elliot Hu-Au: Can I add to, I think actually a very easy win with VR experiences is often ones where you transport the learner to someplace that they can't normally go. And we saw this a lot, like when Google expeditions was out, but sending kids down, you know, underwater and they get to experience like being at a sunken ship or something like that. It has that kind of like wow factor that often schools just want to see, you know, and see their students like kind of in awe of something, and then being able to build off of that with good learning design. But yeah, there are time experiences where you can transport them back into a historical period. It's always something kind of cool if there's something out there that was designed for that. But going further than our physical reaches of our time and place are one of the good ways that VR can have a pretty effective educational experience.
[01:04:17.710] Kent Bye: Right. And, uh, and finally, what do you each think is the ultimate potential of these immersive technologies and virtual reality and the context of education and what it might be able to enable?
[01:04:30.540] Joey Lee: Ooh, like in the future, you know, like, uh, where this is heading or, or like, how big are we dreaming here? Or like what timeframe are we thinking as far as you want to go?
[01:04:40.228] Kent Bye: The ultimate potential. So whenever we reach that.
[01:04:42.989] Joey Lee: I mean, people love to talk about like ready player one or the matrix or the metaverse. I mean, you know, on a side note, you know, some of the stuff there's intriguing convergences with like NFT and blockchain and some of this other kind of stuff where gaming experiences or VR experiences are leading to. more consequential or real life implications or monetizable kinds of opportunities and ownership opportunities and stuff. But think of what that can mean for education as well. How could there be much more powerful experiences that I'm kind of thinking about taking Pokemon Go to the extreme for education, right? Where you're not just collecting Pokemon and then you just add them to your collection just for the sake of leveling up and earning points. But it's all in the spirit, almost like a gamified educational experience, almost like building up your own memory palace, but it's almost like your museum of stuff that you've mastered and learned, right? So you can imagine like, I don't know, maybe you're like Indiana Jones or something, and you're going off on different adventures and kind through these adventures in the real world, it could be tied into the gig economy too, where you perform these different tasks. You could actually earn money or earn rewards for actually learning things and mastering stuff. And maybe what's really happening under the hood is you might actually be contributing to a knowledge economy, or you're teaching people coding skills, or you're doing other meaningful things as part of this kind of like XR metaverse. But learning would feel natural and fun. You wouldn't even realize that you're learning and you are constructing knowledge just naturally as you are living out these different adventures in the real world and sharing that knowledge with others and easily, seamlessly hopping into different environments and communities, like different portals that jump you around. I think there are a lot of possibilities and I think it's exciting to see that convergence in these different areas. And there will be certainly a lot of bad stuff too, a lot of crappy experiences, but I think eventually there will be something pretty interesting. At least that's my limited vision for where that could go.
[01:06:56.747] Elliot Hu-Au: Yeah, for me, I have two things I would love to see eventually come to fruition with VR. One is just on the social aspect of having a really realistic embodied experience of like feeling like I'm in the same place as like a significant other or a loved one who's like across the country and most of my family live on the west coast and I'm not here on this coast and just having you know zoom conversations is great but being able to have like a real embodied experience in a virtual world would be great. You know, more so than what we now with VR chat, you know, with these avatars that takes a lot of effort if you want to actually make them kind of look realistic. But even if they were ones that could sort of show the facial reactions that people have as, you know, they interact and talk, that'd be pretty great. And then my second one is kind of more on the knowledge construction level is, I don't know if you guys remember the movie Contact with Jodie Foster. And if you remember the way they figured out how to build that alien mean in the end was instead of looking at all the One of the things where I think like virtual reality could be a really potentially useful tool is to help people sort of think in a new dimension, especially with like abstract concepts. And so, you know, in sort of like my fantasy world, VR could be actually this tool that helps people expand their ways of thinking of certain concepts and enables us to either come to concepts more quickly, but also just expands the way that we're able to see things in the world. help expand knowledge that humans can have.
[01:08:43.031] Joey Lee: Yeah. Like one more idea that just came to mind too, I was thinking about like virtual humans technology, the artificial intelligence you can imagine in the near future, you could go into various contexts and you could be like talking to Thomas Edison, you know, like an AI generated avatar of that and ask them questions and sort of like learn alongside them in a cognitive apprenticeship kind of model or talk to Abraham Lincoln in the context of the 1860s or whatever. I mean, they could almost like mentor you and it could be this really interesting kind of thing where it's this social learning in action with these like real people throughout history or these innovators or inventors or scientists or whoever. Yeah, a lot of cool possibilities where this is all headed for the context of education in particular.
[01:09:29.545] Elliot Hu-Au: that can go really wrong too.
[01:09:30.886] Joey Lee: A lot of these things can't go wrong and that's why we wrote these papers, right?
[01:09:36.909] Kent Bye: Yeah. One quick thought, kind of an associative link from what Elliot said was that a number of years ago, I went to the joint mathematics meeting and they have a whole philosophy of mathematics, special interest groups of the Palm Sigma, which was talking about mathematical notation and how mathematical notation itself is like a language that is very constrained by the 2D media. and that how in the past, there was more spatialized ways of doing notation, but then we're even limited to what the printing press could actually print. And so in the past, there's already been mathematical notation that was more spatialized, but limited by the printing technology. But now that we have fully immersive technology, then what's that mean for things like the philosophy of mathematical notation? Are there new spatial ways of even notating and communicating with each other. So adding additional dimensions to the way that we communicate with each other, what does that mean for things like mathematical notation and how we express different abstract concepts? And it's not only just the dimension, it's also how that unfolds over time. So having things dynamically move and what are the ways that we use those abstractions to be able to convey different concepts or ideas, you know, going back to that concept of the embodied cognition that gives us a spatial metaphor that allows us to really have a deeper understanding. And so we're the ways that the language itself can start to evolve into this more spatial language. And I think that's still the frontiers in my mind where I go when I hear something like that.
[01:11:02.065] Elliot Hu-Au: Yeah, very cool.
[01:11:04.427] Kent Bye: Awesome. Well, is there, is there anything else that's left and said that you'd like to say to the immersive community?
[01:11:10.275] Joey Lee: Um, not really. I mean, I think that if folks are participating or, you know, listening to the podcasts and it means that they are probably doing really meaningful, important work. Right. And we definitely value that and appreciate that. And we want to just encourage folks to keep going with that as well. I mean, we are too, we're trying to keep going and. You know, I think it's wonderful to meet other folks who are asking these important questions and addressing these kinds of issues and thinking critically and thoughtfully about where this is all headed. And so it's great stuff. Hope folks continue to do it.
[01:11:45.413] Elliot Hu-Au: Yeah. I would just say that each of us is working on our own sort of sliver of the XR space. You know, ours is education and XR and everyone working together, like you said, as a community. I think we can help realize a lot of the dreams that people have for XR.
[01:12:04.328] Kent Bye: Awesome. Well, Elliot and Joey, thank you so much for joining me today. And if people want to get more information, I highly recommend check out the article that you wrote, The Frontiers in Virtual Reality, E3XR, An Analytical Framework for Ethical, Educational, and Eudaimonic XR Design. published on October 6, 2021, and it'll be linked down in the show notes. Go check it out and follow up on all those different references. And yeah, just thanks for giving us an overview and a survey of what's happening there with both the educational learning frameworks, but also some of the different ethical frameworks as well. So thanks again for joining me and unpacking it all.
[01:12:38.686] Joey Lee: Yeah. Thanks again. Thanks for your time.
[01:12:41.033] Kent Bye: So that was Joey Lee. He's a lecturer at Teachers College at Columbia, teaching on technology, design, ethics, and VR for 11 years now. And then Elliot Huao, who's a doctoral student at Teachers College at Columbia, who's researching VR for how it could be useful for use in scientific education, but also learning abstract concepts. So I have a number of different takeaways about this interview is that first of all, Well, it's a really great paper to be able to launch off and to look into lots of different resources on this topic. It was particularly helpful for me to look at all the different aspects of learning theory and how it applies to virtual reality. But it's also very interesting to be able to have a conversation with the co-authors, because Joey said, you know, that he feels like there can be a lot of hand waviness when it comes to connecting the dots between the underlying theory and the practice of what they're finding in terms of applying immersive technologies to education. It was striking to me to hear Joey say that he feels like overall that this is a field that was relatively under-theorized. I guess I took that as a bit of an invitation to continue to look into some of these different aspects. The talk that I just gave today at the University of Michigan Center for Academic Innovation was really a synthesis of a lot of just diving deep into a lot of the resources that were listed in this paper, but also just generally trying to wrap my head around how to apply all these different learning theories and how they all combine together. They are somewhat fragmented in the way that they think about things, but at the same time, with immersive technologies, I do believe that there's going to be this synthesis and combination of all of these different things. I'll maybe link off to the slides that I presented today and then update the post with the video once that's up online. Yeah, just really interesting to be able to listen to the development of this framework and lots of different references, again, to different discussions that are happening within ethics and XR. For me, the biggest challenge was this differentiation between, what do you consider ethics and what do you consider human flourishing and eudaimonia? I think at the end of it, after talking to them and thinking about it, I really do think it's going to be up to the context. There's not going to be a universal answer to that question, because it's all going to be about who the stakeholders are, who's involved, what the intention is, and what kind of time and resources that you have to be able to produce a project. No matter what you do on anything with XR, there's always going to be some compromise and trade-off that you have to do. It's more about how do you prioritize what is going to be the baseline of what you're covering, and then what is going to be the nice-to-have, things that if you have the extra time and resources to really think about. The thing that's tricky about this issue, I think, if you think about it even in that way, is that if it's for you, an aspect of eudaimonia, for people who are on the opposite side, that may, for them, be a baseline ethical thing. where they're an underrepresented minority, where they are having undue harm based upon the decision that you made from maybe more of a utilitarian argument saying that, well, we don't have the time and resources, so let's try to make something that's going to be the best for the most people, which when you start to do that, then the people who are already in these oppressed or marginalized communities then feel the brunt of some of these different decisions that are made at the design level. I think that's the challenge of how do you know what is something that should be a baseline ethical thing for anything that you do versus what's the more eudaimonic, aspirational, human flourishing aspect that is nice to have but may not be there at the first launch of what you're doing. But I think it's also important to realize that a lot of times, these different types of projects do have a process of evolving over time. The baseline of the ethics that I see at the bottom in their pyramid, which I'm going to be referring to a lot because I do actually think it's very useful to think about it in this way, because you can't do everything, and so you have to make some choices as to what you're going to focus on and prioritize. some of the baseline decisions will have to be, by necessity, things that you're required to do by law. So, like in the United States, you have to follow FERPA and COPA, which means that you can't necessarily have kids under 13 using a Facebook device. They're not supposed to be on there because it's not really COPA-compliant to be on Facebook. And there's certain aspects of what happens to the data, and are you able to track that, that these devices aren't even FERPA-compliant, either. So, there's ways in which there's kind of a baseline that, in some ways, the FERPA and COPPA are providing a bit of a shield and a cover for protecting the students' privacy, because otherwise you'd be having these surveillance capitalism devices advocated by the schools and the teachers and whatnot. Again, as I'm talking about this, I sometimes default to thinking about high schools versus thinking about higher education and universities, just because, in a lot of ways, there's more agency as a university student. In an ethical sense, there's more potential harm to be done when you're thinking about it in the context of these primary education folks. But the point being is that there's going to be certain laws that the teachers are going to have to follow, and knowing what either the local or state or federal regulations are going to be, and making sure that that becomes a part of the ethical design at a baseline. There may be certain provisions in terms of accessibility that, in order to really deploy something, it has to be fully accessible. I think, again, by only looking at things, if you're looking at the small population of students, what happens the next year if you may have students come in that may not be fully able-bodied? How do you really address all those other accessibility issues? So again, going down to this question of what is any given issue, is it an ethical issue or is it about human flourishing and eudaimonia? And like I said, it's going to be up to the context and up to people to kind of figure out on their own. So, you know, just some other quick thoughts. Lots of different things in terms of different learning theories, multimedia learning theory, context-based learning, and preparation for future learning. So even if there's nothing that you're able to measure, it's setting a context of an experience that then becomes a foundation of where you go in the future. When I think about all the different other major learning theories, everything from behavioralism, it's a lot about how your actions are instigated by different stimuli. It's a lot about your behaviors. I think of it as this agency and active presence and ways that you're able to modulate that, potentially through game design elements. There's also active-based learning, which is a lot of inquiry-based learning, active-based learning that's all about putting the agency within the hands of the students. Project-based learning is another one where it's all about creation and making stuff. That's probably at the higher end aspects of the Bloom's taxonomy that has a hierarchy of different things, where at the low level it's more about memorizing and recall. The higher level is all about analysis and creation and being able to really have a mastery over a topic. I do think that there is going to be things with the immersive technologies that are teaching these 21st century schools, as Joey says, this type of critical thinking and other higher-level skills that students need. In some of the other different learning theories, you have the constructivist, which is a lot about the direct embodied experiences that you have and be able to tinker with things and experiment and have an action and see what the result is. I think that process of both context-based learning and experiential learning, in terms of the embodied presence and environmental presence, a no-brainer to look at things like experiential learning or constructivist type of learning theory. Cognitivist learning theory, really looking at ways in which you can make connections, but also the mental abstractions and understanding at a conceptual level. I think of that as the mental presence aspects of virtual reality. There's also the social learning or the social constructivist learning, meaning that you're with other people and making the meaning together with other people. But the context is, I think, where a lot of the other contextual domains and how they relate to each other, I think that's where a lot of the ethical aspects come up as well. So it was just interesting to see how they started to map some of those different issues out by looking at XR technologies and reading some of the other literature that's out there. So lots of good ideas to kind of see what the landscape is like here in this paper. You know one of the things that Joey said is that there's a lot of bad ad tech That's out there meaning that a lot of stuff is just making it to use the technology for technology's sake and then there's almost like afterwards trying to make sure that it's Improving learning outcomes, and so there's a couple of papers that he was pointing to that We're actually showing that the virtual augmented reality were providing higher levels of cognitive load meaning that it was actually bleeding to a worse learning outcomes and Depending on the topic and the experience everything else It may be harder for them to really focus or have a good balance of cognitive load when it comes to like cognitive load theory Value-sensitive design just looking a lot of different stakeholders and who the stakeholders are I think that's a big part of really understanding the different relationships and dynamics and and For me, I'm very drawn to Whitehead's process-relational philosophy that is really trying to look at things in terms of these relationships and trying to map out the different contexts. And for me, as I read over some of these papers that have been done, sometimes they're drawing these sort of empirical numbers that are trying to assess the efficacy of this or that. and someone has centered in experiential journalism, I would love to be able to actually have the experience of whatever they were showing, and then be able to match it based upon my own direct experience of whatever they're trying to show, and then see what the paper found. Because I think there is a bit of challenge, which is a lot of this research is done with programmer art, or what's the quality of the experiential design that's there. It's hard for me, at least, to really assess the efficacy of something without really seeing the broader context of what the experience is. I think a lot of the different research that's been done has been, for people, this may be some of their first VR experiences. But as VR continues to grow and expand out there, then the level of production design is just going to get much higher and higher and higher. I do think there's going to be a need for more interdisciplinary collaborations just to up the game when it comes to producing the levels of experiences to be able to actually test this level of immersion and the efficacy of some of these different experiences. But generally, there has not been as much of an ethic of sharing the source material of whatever experiences that you're creating. On the one hand, you could upload it to, say, SideQuest or App Lab if it's on Oculus hardware. Steam doesn't really distribute a lot of that stuff. You could maybe go to HIO, but that's really still about games. WebXR, I think, is something that I'm really excited about, because I do feel like that's going to be a native medium. A lot of things that Oculus is focusing on is much more on gaming. I couldn't even find the list of educational games listed on the website anywhere for the Quest. On the go, when you search for educational experiences, there's like 292 experiences that come up on the go, but I was only able to really track down seven on the Quest. There certainly may be more, but Oculus has not made it easy to find those experiences. There's already not as many experiences on the Quest already. But they're just harder to find. And like I said, the quest is not FERPA or COPA compliant, meaning that even if educators wanted to, they really probably shouldn't because they're not really COPA or FERPA compliant because they require the Facebook login ID. So there's things that Oculus have done that have not really made it easier for educators to be able to use their technology. And yeah, there's a part of needing to wait for other competitors to come along. Maybe they have policies that are more in line with whatever the baseline compliance is for some of those. So. Last thought is, some of the different things that I was sharing with the University of Michigan Center of Innovation talk that I gave about XR ethics was looking at not only the ethics of education, the learning theory, but embodied cognition and the theories of presence, but also just some tips in terms of strategies for understanding the communication media and how it evolves, and how would you go about doing the next iterative step. How do you start to make sense of how to combine all these different learning theories and experiential design and start to mix and match ways of combining the theoretical background, but also putting it into practice? And the thing I recommend folks to do is to really check out the stuff that's already out there and have your own experience of stuff. And then if you feel inspired to be able to create your own experiences, then start to put stuff together. You know, there's starting at the baseline 360 video, and then you have uploading a scene and be able to look at it by yourself within an application. And then adding different social layers, either through Mozilla Hubs, but there's Engage platform, there's Altspace, there's Rec Room or even VRChat. But I'd say Engage and Altspace is probably where a lot of the different educators are hanging out and spending a lot of time. But also more and more, I've seen more stuff within Mozilla Hubs, which is probably the most open and cross-compatible platform out there. And yeah, adding in different levels of that social dimension to really tap into that social constructivist aspects. Being able to have those live transmissions and conversations and interactions in the context of a spatial context can be super powerful. And then the last part of adding agency and interactivity, that's the thing that's probably the hardest. There's some great experiences by Shell Games with the HoloLab and really pushing the limits of what's possible with recreating a chemistry lab and making it into a full-fledged game. really brilliant experience that I highly recommend folks checking out. For me, that's kind of like the pinnacle of interactivity and agency matched with the experiential design of what VR as a medium can do. Super powerful. I think that's also the hardest thing to do, that type of interactivity. When you look at something like hubs, it starts to mean that a lot of different experiences in order to do that, you also have to either Download unreal engine or unity to start to have that type of interactivity There are things like in rec room to start to do types of interactivity within the game and also within engage So definitely check out some of those in-world games, especially if you're trying to create socially created interactivity But yeah, I do think this conversation that I had with Joey and Elliot and a lot of the work of a lot of the resources were super helpful for me to start to think a little bit more about the underlying theory of how to mash up things like experiential design and immersive technologies and the learning theory and adding all the different aspects of embodied cognition and perception and the new affordances of virtual and augmented reality. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listed supporter podcast, and I do rely upon donations from people like yourself in order to continue to bring you this coverage. So, you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
