I believe that the principle of Embodied Cognition is probably one of the most significant and important concepts to understand about virtual reality. Cognitive science researchers have been connecting the dots the importance of our bodies when it comes to perception, the subjective construction of reality, and how we process and think about information. We use our entire body and surrounding environment in our cognitive processeses, and virtual reality is bringing our full bodies into computing in a way that takes full advantage of the insights coming from embodied cognition research.
EleVR is a VR research collective that has declared 2017 as the “Year of the Body.” “Mathemusician” and virtual reality philosopher Vi Hart was a self-proclaimed body skeptic seeing it as an inconvenience to take care of in the persuit of higher forms of beauty with math and music, but after some preliminary experiments with embodied visualizations of physics she started to have a direct experience of the power of Embodied Knowledge.
I had a chance to catch up with EleVR’s Vi Hart and M Eifler to hear about their VR experiments and research into embodied cognition from creating interactive math museums built around 3D Venn Diagrams, visualizing hyperbolic space, and exploring the boundaries of container schemas and metaphors for understanding the concept of home and a place to rest.
LISTEN TO THE VOICES OF VR PODCAST
Venn Diagram Museum
Hyperbolic Space in VR
Real Virtual Physics
Check out my previous episodes about Embodied Cognition:
- Embodied Cognition: Using Dance to Teach Computational Thinking
- Immersive Analytics & Embodied Cognition
- Saadia Khan on Embodiment Theory & how using virtual avatars can improve learning & how you feel
- Embodied Cognition & Using Social Structures for Collaborative Learning
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR podcast. On today's episode, I'm featuring a concept and idea that I think is probably one of the most fundamental and foundational concepts that virtual reality is revealing to us. There's a lot of different applications and uses of VR, but this concept and idea of embodied cognition is probably one of the ones that has the biggest amount of philosophical implications about how we even understand the world. So the basic idea about embodied cognition is that we use more than our brains to think, that we actually use our entire bodies to process information and to make sense of it. Not only that, but we also use the environment as well that helps us situate us into a specific context which also can help us remember things. And so on today's episode, I feature a couple of VR researchers from Ella VR, that's Vi Hart and M. Eifler. They've been researching virtual reality topics for the last four years. And so this year they've declared the year of the body where they're going to be creating these rapid prototype experiences, exploring this concept of embodied knowledge, making interactive math museums and walking around hyperbolic space and generating 50 different environments that are in the same theme, but allow you to experience it in different ways. So we'll be covering Ella VR's journey from body skeptic to full-on believer and evangelist and pioneer of embodied knowledge on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by the Voices of VR Patreon campaign. The Voices of VR podcast started as a passion project, but now it's my livelihood. And so if you're enjoying the content on the Voices of VR podcast, then consider it a service to you in the wider community and send me a tip. Just a couple of dollars a month makes a huge difference, especially if everybody contributes. So donate today at patreon.com slash Voices of VR. So this interview with Vi and Em happened during the week of GDC on Thursday, March 2nd, 2017 in San Francisco. So with that, let's go ahead and dive right in.
[00:02:24.396] Vi Hart: I'm Vi Hart, and right now we're working a lot on embodied understanding of complicated concepts such as hyperbolic geometry or abstract algebra and mathematics. So we're having a lot of fun with that.
[00:02:41.429] M Eifler: I'm M. Eifler. I have been mostly working on some bed related projects because beds are where you have the most body doing and rolling around and stuff. The most recent two projects were about getting you to lay down in beds that I had made and instructions about rolling over and how that related to the content that you were watching and basically trying to think through instructions for how to get people's bodies to do the thing that you want as a form of content and also sort of responding to their body position as a form of content.
[00:03:11.994] Kent Bye: Interesting. Okay. So let's dive in first to the embodiment of abstract mathematical concepts. So maybe a talk a bit about how did that come about of adding the sense of embodiment to things that we necessarily can't get a direct experience of in the real world?
[00:03:28.623] Vi Hart: Well, I would say I was a body skeptic for a long time. I think that's pretty common, especially among the mathematical community or computer scientists, where the body is often seen as an inconvenience. And certainly I felt that way for a long time. Like, I want to be thinking about these beautiful mathematical things or musical things, but I have to feed this thing and take care of it. And if I don't have the right posture while I'm programming, then I feel terrible. But recently, doing more stuff on the art side, and especially with the influence of the rest of my team, I started to realize, well, there's some hardcore research that has to do with embodied understanding, not just of things that aren't math, but also of math. And that kind of hooked me, that there's been a lot of research over the years, since the beginning of computers, about embodied understanding of math. So looking at that research convinced me there was something to it. And in the past few months where we've been diving in, it's there. It's really there. So it's exciting.
[00:04:25.487] Kent Bye: Yeah. Maybe you can expand on that, your direct experience of math. Like what does that feel like or what does that look like?
[00:04:32.996] Vi Hart: Well, the first papers I looked at were about just understanding graphs and derivatives. And I'd have to get you a reference, but Alan Kay made me look at some papers that they were looking at in the 60s about how you have a simple thing that can just tell what distance you are from it. And by moving back and forth, they showed that students could learn better the idea of acceleration and motion by seeing a very simple one-dimensional motion graph of what they were doing. And I saw this paper, and they did some actual research that showed this helped students understand the idea of graphs and motion. And I thought, well, now we can do that really easily. We don't need to set up this whole complicated, expensive thing. And we can do it in three dimensions. And we can do it in real time. And we can see what we're doing as we do it. So I made this little prototype called Real Virtual Physics. And I thought it was pretty effective, where you can move the controllers. They're tracked in real time. And you can drop a controller. You can see the acceleration. You can see it graphed as you drop it and get this real embodied sense connecting what you're doing to what you're seeing. It was way too compelling, and then I realized a lot of other people have been saying this for a long time, and so it was about time that the VR community got on it.
[00:05:49.110] Kent Bye: Yeah, and it sounds like you're also looking at embodiment. Maybe you could describe your approach of unpacking embodiment.
[00:05:56.927] M Eifler: We started reading Philosophy in the Flesh and one of the concepts that really stuck with me from that research was about basic categories and how they relate to your physical existence. A basic level category is essentially something that you can imagine using one silhouette or you can interact with using one physical schema. A great example is a car. A car is not necessarily any one individual car. It's not a Honda or a Civic. There's a certain way that you use your feet. Maybe it's a manual or an automatic, but when you move up in the category to vehicle, then you no longer have like one silhouette or one physical schema in order to interact with that object. That was super fascinating. I had never read anything about that before. And so I started diving into like, okay, so can I hold the same physical schema for something at vastly different scales? And can I then use your body's understanding of those two objects as having the same motor schema as to put them in the same category? Meaning that I could make something tiny that you would interact with in the real world and then make it huge in the virtual world and then change the motor schema? Does that change the objects into two different categories? Sort of like manipulating motor schema as a way to manipulate category itself and basic level category itself. And one of the reasons I think it's so fascinating is because I really think that VR's like special talent or whatever is that you can manipulate this very strict like human categorical thing that we love doing. We love putting things in like one bucket or another bucket but actually you can like mix the buckets really effectively using VR and AR too. We've been doing a lot of experiments with that, like blending those categories together.
[00:07:44.407] Kent Bye: Yeah, and just to maybe clarify and just make sure I understand some of the terminology in terms of schema is that I'm starting to look at artificial intelligence. And I was at the International Joint Conference of AI talking to these AI researchers about knowledge representation and how the brain actually stores information and knowledge. And one of the things that he was saying is that we actually learn through metaphor. And he said they were doing these studies where they would test people. And they would say, my life is a prison. And then they would say, my prison is a life. And the second one is wrong grammatically, but yet they were able to measure some metric saying that the brain understood it if you were to cut it off at a certain point. So in other words, they got the concepts together, and even though they were backwards, it still had that same idea is that this metaphoric way of understanding was saying the same concept. So is that kind of like what you're saying in terms of the schemas and the different types of learning?
[00:08:37.189] Vi Hart: Yeah, and Philosophy in the Flesh by Johnson and Lecoff definitely talks a lot about that exact kind of embodiment of metaphor. So, if your life is a prison, there's an embodied sense of being in a prison and you know what that means because you know what it feels like to be trapped in a space. Even if you've never been in a prison, you still know what it's like to be trapped in a space like that. And that makes that metaphor work, unlike my prison is a life. There's like, how do you embody that? There's no motor schema about walking around in a small room trying to get out that has to do with that. So now we can make lots of spaces and get our bodies involved and track it and be computational about it if we want to. And AI could certainly learn a lot from it now that we have It's easy to train AI on images now that we have a vast library of images. Maybe someday we'll have a vast library of motor actions when we have great tracking and then we can train AI on motor schema.
[00:09:36.418] M Eifler: The prison thing is a great example because the prison cell was one of the examples I used in Making the Bed specifically because, okay, so Making the Bed is this project in any land where you click on a bed and you are teleported to its location, you can lay down in it, but they're all in these vastly different spaces and one of them is in a cell and one of them is on a fire escape and one of them is this very plush 18th century bed. And there's something about the way that beds are connected to identity and the way that your body interacts with that space that is directly related to your identity that I thought was super interesting. And how you could, in this project, become the thing that is not any one individual human, the thing that could be separate from identity, and yet go out and experience these different spaces as identities and how your body relates to those spaces as being your temporary identity in that space.
[00:10:27.096] Kent Bye: Yeah, one of the, just to kind of expand on what you're saying there is that when I went to the IEEE VR last year in 2016 in Greenville, South Carolina, I had a number of interviews about embodied cognition and looking into it, the way I kind of describe it is that a lot of people when they think about our mind and our cognitive thinking, we think about it in terms of the brain. The principle of embodied cognition is saying that is, and actually more than just the brain, it's in the entire body that we use to think. But not only that, it's our environment, such that when we see a unique architecture, like we're in this room right now with all these little crazy sculptures, and when I recall this conversation that we're having right now, I'll remember where we're at. part of the reason why I wanted to be in here, because it'll make it such a much more vivid memory. And so you can start to do that in VR, is to create these architectures that are metaphorically representative of the concept that you're trying to really talk about. And by doing that, you can actually accelerate the type of learning and give people a sense of embodiment and some concepts that, you know, you're making like art museums and different ways that you're just rapidly prototyping them. It would take a lot of time and resources to actually build it, but you can just prototype it in VR and you can have that direct embodied experience of that and to show people how compelling it is, then perhaps even build it. So, but that concept of the way that we think actually changes depending on the environment that we're in, is that we can actually... I mean, we had a concept that like memory palaces has been a concept for a very long time.
[00:11:54.220] M Eifler: People have been using this idea for a long time and that you can augment your memory by imagining spaces. Well, why can't you do it like the other way around? Like build spaces in order to augment the things that you're learning in them.
[00:12:05.328] Vi Hart: Yeah, it's amazing how our sense of where we are in space and where things are has been totally underutilized just because we didn't have the technology. And we augment our memories and our thinking by sketching diagrams, by writing things down, by writing lists. Writing is a technology, and we think of it that way. And we think of, for example, graph representations or different representations of data. Data representation is a kind of technology to help us augment our understanding and use what the brain is naturally good at to let us hold more concepts in our head at the same time. So if we can make a picture of a thing, a representational picture of a thing by drawing, a Venn diagram is a kind of technology for helping us create this picture in our heads that uses some things that we are very naturally good at when it comes to seeing things organized in space in a certain way. and we could do new kinds of that, that we haven't thought of yet, when we can quickly sketch, the same way we might sketch our notes and our lists in different sections of our notebook, if we could quickly sketch things in 3D that take full advantage of our ability to know, oh, that thing's right behind me, and this is over here, and I keep this in my virtual pocket, and this in the little left virtual pocket, and I can reach all these things without looking, and I know where they are, because I evolved to be very good at knowing what's in reach of my hands and knowing what's in reach of my feet and knowing what I can walk around and get. And that's just there without the extra step of, oh wait, what file was that called? How do I open that? Do I remember this off the top of my head? Can I find the right book? So all these things we can bring closer to our ability to understand instantly.
[00:13:52.383] Kent Bye: Yeah, the thing that I find really fascinating with this trajectory of where you're going of really giving a sense of embodied presence with math is that when we talk about metaphor, we're kind of limited to our direct experience of things that we can actually experience. And yet, you know, in order to understand some of the more advanced concepts and science, you have to have a mathematical background such that it's like a metaphor for you to really start to comprehend these higher order levels of complexity when we start to try to use math and statistics to describe how the world works. And so you're starting to use VR to explore concepts that are not even available to be experienced within reality. So maybe you could the best you can describe like this VR experience of hyperbolic space and what your direct experience of that and what that has taught you about these abstract constructs.
[00:14:45.439] Vi Hart: Yeah, well, the theory of metaphor would say that all understanding is metaphorical. So it's not that these things are necessarily further out of reach, just further out of our experience. So if we can bring the experience, then that makes it not abstract anymore. So it's not like these things are in a different category. We just haven't brought them in yet. And so I think we can bring hyperbolic space in because the computer knows what hyperbolic space is. Well, it knows if you program it. So we program the computer to know what hyperbolic space is. The computer doesn't care. It's like, all right, we can do this. I'm a computer. I don't know anything about reality. This works. You know, the math works. It's fine. So we have hyperbolic space programmed in, and you can walk around in it, and it doesn't act the way you're used to. In some ways it does, and in some ways it doesn't. So as someone who has a lot of familiarity with hyperbolic space, I have these metaphors and concepts in my head already. I kind of knew what it would be like. And for me, it's just really cool to be able to be in it and be like, yes, this is what it should be. Oh, this is so great. I'm there. But for people who didn't have the experience to be able to come up with how to imagine it, now they can skip that step and maybe just be there and then get that embodied understanding of when you walk around the room and get back to where you started in real life, well, in hyperbolic space, you're not back where you started. You're, you know, not all the way back because space is bigger around you in a very hyperbolic way. And once we can get in hand controls, which we're working on, you'll be able to manipulate objects and just get a feel for the things that make hyperbolic space so hyperbolic-y, which I know from imagining it for so long in my head, and which people will just be able to experience. Like you don't need to know the calculating number math to get an intuition for how these spaces behave. And mathematicians often talk about like how a thing behaves. And that can mean all sorts of different kind of things that sound abstract. But we can make those behaviors part of how we behave. We can see the behavior of there being more than 360 degrees of terrain to get where you started. Because we know how to walk around in a circle. People know how to walk around a room. They know how to tilt their head. You just know that your body is doing these things without thinking about it. So it's exciting.
[00:17:05.687] M Eifler: It's interesting as the metaphor is like a mechanism, because you're mapping Euclidean space, like the space that we're walking in, onto this other thing that you're only experiencing with your eyes, which your body then like. It's just interesting to me that your feet are in Euclidean space and your head is in hyperbolic space. And at some point, there's like this, I don't know, this weird thermocline in the middle where you have to transition from one to the other.
[00:17:29.807] Vi Hart: Thermocline, nice.
[00:17:32.714] Kent Bye: Well, I think one of the things that we've seen from VR is that actually the visual system can dominate the other systems. So, you know, things like redirected walking just shows that you can give a visual feedback that's different from what your body's feeling. And your body just kind of trusts your eyes because the light travels the quickest and, you know, you just have a more reliable orientation to that space.
[00:17:53.601] Vi Hart: Yeah. And for the hyperbolic stuff, you very quickly get that same sense that, okay, I trust my relative motion. I know that if I'm taking a step forward, I'm taking a step forward because hyperbolic space locally is pretty much like Euclidean space. But as you walk around larger distance globally, you get all the curvature and it depends on how curvy you make it. But for what we're trying to do, we're at least trying to keep some of our Euclidean sense. And so you have that same thing as redirected walking where like, You locally think you're walking straight, and whether space is curving or you are, you realize that that part isn't really what matters. And you can let go of that Euclidean part of your sense and get the hyperbolic sense of like, oh, well, relative motion is fairly Euclidean. But globally, we don't need to worry about that anymore. We're having something else going on.
[00:18:46.090] Kent Bye: Yeah, and I'm curious to hear more about your own direct experiences of embodied cognition and this principle of embodiment and some of your maybe favorite stories of embodiment within VR and some of the big open questions that you're really looking at.
[00:19:00.680] M Eifler: I think the most obvious one was when I tried to put my hand on a counter that wasn't there and then I fell and then my body immediately learned. I never made that mistake ever again. I never ever tried to lean on anything ever again. It was amazing to me like how fast I learned that like instantly. One mistake, never do it again. But I think my favorite thing that I've been learning about is how my body likes to make stuff in VR. I wrote in one of my recent blog posts about the hug range, where I actually like things to be in this space around my belly and just slightly in front of me. and how when I have to reach my arm up and back behind my shoulder to grab something in the virtual world, that means that thing is less valuable to me than the thing that's in front of me. That I gather and place and organize things in a very particular way that might not be true for everyone, but that I place value judgments on the muscular effort that is required to get something. Like, oh, I guess I'll go get that thing. But actually, everything that I really want is right here and you know you can do some artificial things where you like artificially turn yourself or something without having to move your body but mostly I try and focus on like how does it actually feel like what does my body want to be doing when I'm creating like I like to sit on the floor I like to not wear headphones I like to be in an environment where people are around me so that I don't feel isolated inside of the headset and then I can focus on like the way that my spine feels when I'm creating something and like where a pain starts in my shoulder blades when I'm making something and all of that gets incorporated into the meaning of the thing. Not because I'm avoiding pain necessarily, but because the way that your body makes the thing is great. An example is when I was making the bed, I made all of the beds originally as teeny tiny little models, something that would fit between my shoulder blades, something very small. And then later on I blew them up and made them more detailed. But I didn't work at full scale because that would have required me to exit the hug zone. Like I didn't have the thing all right in front of me. And the scale of the object also then, like my body's desire to have the object to that scale when I originally created them also led to there being like this mini map where you could like sit on the ground and see all the things and you could click on a little one and go to the big one like that. Teleportation system inside the project was all created because my body wanted to make things that were just like between my heart and my belly button.
[00:21:25.794] Kent Bye: Yeah. And you have this whole Venn diagram immersive museum that you created and maybe you could tell me a bit about the origins of that and what you were trying to really create there.
[00:21:37.880] Vi Hart: Yeah, still in progress. But I guess I've been thinking about the idea of how can we augment our understanding of things if we're not stuck with the old tools. So Venn diagrams was kind of a jumping off point of trying to think, well, what could be a 3D diagram? And I never got to the something beyond the Venn diagrams part because I'm having so much fun with Venn diagrams and riffing off of them. So I've just been building piles and piles of things related to Venn diagrams, including, as soon as I made this tiny model, as M said, I made this tiny model of like a Venn diagram. And immediately I thought, I want to be in this. And all I had to do was scale it up. And now instead of looking at the categories from the outside, I can actually be in them. And it made such a difference because all of a sudden, instead of just looking at, there's the things in this part of the Venn diagram and the things in this one, it's like, oh, I am in a room with these objects. I'm familiar with this. I've done this my whole life. I know what it's like to know that there's these objects in a room. Now my body knows it. And then as I can walk from different rooms, it's a different kind of memory. So I've been working on a architectural scale Venn diagram museum of mathematics, I suppose, this week. And so it's a three circle Venn diagram except it's building sized and you can walk around and I have different interactive exhibits in different categories. And it's been great because I like to kind of work when I'm sleeping or getting up in the morning, and I start imagining what I'm going to do about my project today. What will I add? What will I put where? How is it going? And it's awesome because when I think about this thing I'm building, I know where everything is in such an embodied way that I could never really have imagined. Like, it's really as if I were designing a house and just putting real things in real places. I just know, oh, I have a sense of, oh, I want this wall, maybe I'll put an exhibit of combinatorics on this wall, because that'll look nice. On the left, there's the thing about graph theory and geometric group theory, so that'll look nice. I know where everything is, and I can remember all the things I've done so far. Which is a problem sometimes. It's like, oh, did I do something that is both topology and geometry? I can't remember. I need to look at my notes. And when I was just designing on pen and paper, I would have to look at my notes and say, oh, do I have something in this category? But now I just know. Is there something in that room? I just know. It's kind of weird.
[00:24:09.013] Kent Bye: Yeah, it's in your body. It's like a direct experience of it. And I think that's the thing that's so compelling about, you know, my own explorations of embodied cognition, which for me really, I think, took a deep dive when I got access to Google Earth VR, when I was able to go back to my hometown and then Google Earth VR, they have 3D geometry in some towns and other towns don't. And I was lucky enough to all the places that I've lived, have like the 3D geometry where I can actually go and get that sense of embodied cognition. And what I found was that I would go to these places in Google Earth VR and start to have these memories come up that I hadn't thought of in a really long time. And so then I started to do this mapping of the emotional architecture of my life and I started to like zoom out and to like see all the hotspots of the peak emotional experiences that I had growing up. And so then when I went home from Thanksgiving, then as I was driving along, I was like, I had a embodied sense of all of these different locations that as I was driving home, I just was like, able to really remember those memories and so the connection between memory and place has been something that I Have been thinking about a lot because you know I go around the world doing interviews with people all the time and then I can just recall where I was just thinking about the context of me walking there and remembering all of the buildings and the where I was standing and then the conversation and then and So when I'm editing it, I get transported back to that place and then it's in my body in a way such that I almost want to like give people access to that same level of embodiment as they're listening to it. So then I started to think about, okay, as people are listening to this podcast, they're like driving in their car, maybe they're walking or whatever, but what if they could go into VR? What if I could like somehow translate these words and ideas into like some sort of nodal graph bubble that somehow was able to link all those concepts such that with this aid of artificial intelligence we could start to map out that architecture but start to even map out an architecture of the metaphoric ideas as you start to listen to it and imagine going into a VR experience and listening to that podcast and you just like see this amazing experience that it's in your body in a way and I think that's kind of where I see all this going is like being able to create these spaces and understand these principles of embodiment and how much the space can allow you to remember more things. Because, you know, as I was walking around this Museum of Mathematics, you know, I'm curious about math, but it's not something that I've like really wanted to just like interact with and play with. And this felt like an interactive museum where it was structured in a beautiful, elegant way that was making a map of all these different domains in the cross section. And if you add geometry with algebra, then it comes up with
[00:26:42.538] Vi Hart: Algebraic geometry.
[00:26:43.819] Kent Bye: Algebraic geometry. Yeah. So then, you know, so you have all these combinations of all these things so that, you know, you go into these rooms and you're in this architecture of a Venn diagram, which, you know, I've never been in before. So it's like, that's a memory I have now of that, but you create these learning environments such that you just create this curiosity of exploration of like interacting with the spatial medium, which math is a, you know, very spatial and you're able to actually see those three dimensions and have these objects.
[00:27:11.405] M Eifler: I think the manipulation of real-world place cells is what Unscannables is all about. You come into a room. It's a physical room. Your brain gives you all these place cells. It gives you the positions of objects and how you got into that room and everything. And then you make a virtual duplicate of the room. And you can activate someone's place cells by giving them a visual representation of the place they were just in so that eventually, with enough exposure to both the physical and the virtual environment, mapping the two on top of each other until what you're seeing where your hometown is recreated in this virtual space, like being able to manipulate the room until it's both a physical room and a virtual room because you've manipulated someone's place cells into thinking they're the same thing. And that means that you can make a physical object have virtual categories and make a virtual object have physical categories, like pokeability, for example. Virtual objects don't have pokeability yet. And physical objects don't have the ability to change at any scale. And the entire history of sculpture is essentially like, these sculptures are this because they're this size and these sculptures are this because they're this size and you physically interact with them in this way so they're all in this category but none of that has to be true when you can manipulate someone's place cells into telling them that all that stuff is the same.
[00:28:21.286] Kent Bye: So what are place cells? Maybe you could describe what place cells are in your brain.
[00:28:25.008] M Eifler: Place cells are in your hippocampus. They activate in like sort of patterns that chunk space into different cells. So my whole office probably has 15 or something place cells. And when I come here via the train, you know, my house has a place cell, then outside of my house, and then my street, and they activate in a row to tell you like where I just was. and where I currently am, and the route that I took to get here, that's all mapped by these special cells in your hippocampus. And you can reactivate them using virtual reality and by giving visual feedback of the same physical space.
[00:29:01.671] Kent Bye: Wow. And are those connected to memory?
[00:29:04.120] M Eifler: Yeah, they essentially give you spatial and geographic memory. They're not for, I am in one particular room, I moved three feet to the right. They don't do that as well. They're more like route based. Oh, I see.
[00:29:17.390] Kent Bye: So I think that's an important thing because I've noticed this. I have been here and sometimes I take a cab and I go from one place to another and then I get out and I'm kind of disoriented or I take the subway. But when I walk to a place, I maintain that integrity of those place cells in that route. And so I found the same thing in VR. Sometimes when you're in VR and you start teleporting around, then you can start to get disoriented. But yet, if you're walking through a space, and then if you get comfortable enough to actually locomote in a way where you're not teleporting around, I think you can actually kind of preserve the integrity of the place illusion that you're actually tricking your mind that you're in another place.
[00:29:50.389] M Eifler: And yet, with a skilled city dweller can go in the subway and come out of the subway teleportation, Like, the space that they were in didn't change. Your play style didn't get changed because, you know, when you're on the train. But when you come out, you have this gap and then your brain's like, oh, okay, we're doing this thing again. So there are teleportations in the real world that your brain manages to deal with. But that's mostly because you have this overlay, this like, I'm going to say augmented reality, even though it's just in your imagination. This map that you're like, I went in here and then there was a fancy teleportation and I came out here. And I think the more you integrate that into virtual reality experiences, the less disoriented people's spacefulness would be.
[00:30:26.892] Vi Hart: Yeah, especially when in-betweens in virtual reality get to be more place-go-like, which what I mean by that, if you teleport along the same route and you're kind of going along, and Anylands is kind of nice in that way because it only lets you teleport a short distance. So you can teleport yourself anywhere, but you can't go all the way across the field. You kind of go in a few chunks. And that really helps you retain that route memory, even though it's still teleportation and you only spend a moment, but you have that path built into your brain somehow.
[00:31:00.963] Kent Bye: Great. So what do you each want to experience in VR?
[00:31:06.054] Vi Hart: everything I'm making right now. But what I'm really excited for is when we get to this stuff that is going to seem so obvious in 20 years, but that right now we just didn't realize, oh, of course, when you want to remember to feed your dog, you just make this hand motion and this bubble pops up and that's like the easy reminder that Like, these mental tools is really what I'm excited about. I think they're there, and I don't know how to describe them because they don't exist yet. But the same way that, like, the list was an invention. A to-do list. So simple and so powerful. I use to-do lists every day, and at some point, those were invented. So we're going to have all sorts of, like, spatially categorized, oh, this and this, oh, and here and there, that are going to just help us so much.
[00:31:57.027] M Eifler: What do I want to see? So last week, it was raining so much here, and there were just dead umbrellas everywhere. And I started just stopping and making photogrammetry models of dead umbrellas on the sidewalk, because why not? And what I want to do is I want to put dead umbrellas in the HoloLens, then position them where the dead umbrella was in the place, and then just walk around and look at some dead umbrellas. That's just what I want.
[00:32:24.375] Kent Bye: That's like a visceral memory of that. It was a pretty intense storms here in California that, you know, so it's sort of like taking you back to the, that experience of how intense the flooding and everything was here. Yeah. And, um, you've done a number of scouring of the research of embodiment and pulling from all sorts of different sources. I'm curious if you could kind of pick out some of the highlights and key takeaways that you've seen in your research into this topic. Cause personally, I think this is one of the huge applications of VR.
[00:32:54.439] Vi Hart: Reading Mindstorms is always inspirational, Seymour Papert, and a lot of people in the education world know this, just the idea of building these play spaces for learning mathematical or computational concepts. And Mindstorms, either you know it and you already love it, or you should read it. But bringing those ideas to VR is gonna be powerful.
[00:33:18.558] M Eifler: I've mostly been reading art-related stuff. I highly recommend Experience. It was an MIT press book. Things about embodiment that I don't generally think of as experiential, like my immune system being a centuries-old device to define what is yourself and what is not yourself, work like that that I found super interesting. Also, currently reading Art as Experience by John Dewey, which is a great book about how looking experiences and sort of performative experiences for audience members change their relationship to the work. There's a giant book list on our website.
[00:33:57.343] Kent Bye: There's also Embodied in the Flesh, I think.
[00:33:58.964] Vi Hart: Philosophy in the Flesh. So that's a, if you're interested in any of these embodiment concepts, yeah, that's been a huge inspiration for us recently. So Philosophy in the Flesh. Johnson and Lakoff. It's got stuff about the cognitive science of all this stuff in addition to hardcore philosophy stuff if you read ontology for fun. I do. Yeah. Yeah. It's a really fun book and has a lot of inspiration in it.
[00:34:25.001] Kent Bye: Awesome. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:34:33.908] Vi Hart: I kind of think of it, as I said, like writing or like literacy. Once we are all able to sketch and take notes, just as part of our daily lives, we're going to really be able to think better when we can use these tools of our bodies, which we've evolved for so long, and they're so good at so many things that we no longer really care about. Like, we no longer care about some of the amazing things our bodies can do. We don't need that. But we can still take those skills, those body skills, and when we can bring them into our mind, it's going to be so great. We're going to be much better at being people.
[00:35:11.073] M Eifler: new ways of thinking, new ways of developing mental flexibility from a very young age, developing ways of changing the category that you currently have someone placed in. We're in this very sort of intersectional state right now, intersectional feminism and people with non-binary genders and like this kind of thing and people with mixed race backgrounds, the categories that we've lived with for a long time in human culture are sort of starting to fade and get a little more flexible. And I think that VR is a great tool and AR is a great tool for developing that mental flexibility for a wider range of things.
[00:35:51.038] Kent Bye: Awesome. Thank you so much.
[00:35:53.140] Vi Hart: Thanks for having us. Yeah, thank you.
[00:35:55.215] Kent Bye: So that was Vi Hart and M. Eifler of EleVR, and they are doing some experiments in embodied knowledge and embodied cognition. So I have a number of different takeaways about this interview is that first of all, I think the concept of embodied cognition is probably one of the most significant ideas and concepts to really understand as to what makes virtual reality such a powerful medium. The fact that you can transport someone and to trick their minds into believing that they're in another place allows you to create these learning environments and metaphoric ways of externalizing our thoughts into these situated environments, such that we can improve the way that we think, but it can also teach ourselves these concepts a lot better. To really break down topics into these higher level archetypal ideas, such that you can give someone an embodied and direct experience of that. There's this underlying philosophical idea that I think we have from the Enlightenment. It's from Descartes, which is kind of like this mind-body dualism, where you think in your mind and you have your own body experience. And those are two separate things. But yet, I think what we're finding is with this philosophy in the flesh book that came out in 1999, as well as a long history of philosophical thought of the postmodernists and Kant and with Freud talking about the unconscious and Jung talking about this collective unconscious and you can even go back to Plato and the Neoplatonic thinking of these fundamental forms and archetypes. These are all just many different ways of talking about the same idea which is like this metaphoric way of our brain thinking and processing information. And that if you can understand the fundamental archetype or metaphor, that you can have a direct embodied experience of that, then you can start to really understand that concept. I think this is going to just completely revolutionize how we learn. It's already how we kind of learn already, but to really incorporate our bodies within that learning process is just going to not only accelerate the process of learning, I think it's just going to open up all sorts of new ways of exploring concepts that we could never really have an intuitive grasp of before. So if we just take this example of being able to walk around hyperbolic space. So normally if you were to think about Euclidean space and you were to put like four cubes and you could walk around that space, But in hyperbolic space, you can put like six cubes because the space is actually kind of bending around. It's kind of simulating this redirected walking sense where you're seeing a different visual sense of this hyperbolic space, but yet your body's in this Euclidean space. But just being able to walk around and get this intuitive sense of what that means and what that feels like gives you access to that metaphoric concept that then unlocks all these other mathematical concepts. The Venn diagram museum that Vi has created is super fascinating to be able to walk around and play with these different mathematical concepts. I was just super surprised as to how really truly compelling it was to take this concept of these schemas and then a Venn diagram is these different categorization buckets and seeing where those two things overlap was just a great way of creating an architecture that you could really have this embodied experience of these abstract mathematical concepts. that are actually really spatial in their nature, but we've never had an opportunity to really actually engage and interact with them and to express our agency with them. And when you're able to have that dynamic interaction, that was some of the early tests that Vi had with these real virtual physics was that you're able to move your hand around and you're able to track the position of your hand, but not only that, you're able to see the velocity of your hand and the speed so that you're able to do calculus and do the integral of that and be able to look at the slope and then see the connection between the position and the slope of the change versus the speed and the velocity and to really just Put that in your body in a way that you really just are able to understand some of these complex mathematical concepts But once you have the experience of doing that experience, it's now in your body in a new way So the philosophical implications of embodied cognition is going to start to, I think, challenge our ideas of the separation of the objectivity and the subjectivity. And I think if you really study how our perception works and read about that in Philosophy of the Flesh, even as you're listening to me say these words, you're doing all this unconscious process of you're listening to the string of phenomes, you're having all the context, you're adding the meaning, you're being able to connect that to your previous memories, and then You just hear me speaking and you understand it, but yet there's so much of that process that are happening on an unconscious process. And so when you start to think about, okay, what is like rational objectivity and what is true and what is reality, you start to see how much of the construction of reality is just this really fluid, subjective, unconscious process. Then I think it's a lot harder to start to make those firm boundaries between what is objectively real and what is subjectively your perception. So for me, as I'm trying to start to understand and start to put these different frameworks of how I'm navigating this new world, I'm really finding just a lot of insight of going back to the ancients and going back to the Neoplatonic thinkers and to go back to a lot of that revival that was happening during the Renaissance and to see that recontextualization of this Neoplatonic thinking of these fundamental archetypes and forms and how can we start to think about how that really applies to our perception and our understanding of the world. And that the more that you study these fundamental archetypes and principles, that the more that you could start to have a direct experience of being able to understand your own experience, but also understand how the world works. So I think that the applications of embodied cognition are going to be so vast. It's a little bit difficult for me to be sitting here now to really extrapolate all of what's going to happen, but really looking at these concepts of like a memory palace, you know, what was Giordano Bruno doing back and inspired to be able to take all these different esoteric traditions and to be able to overlay that on top of his conceptualization of these different celestial spheres and as you are able to walk around this physical place that is connected to the natural movements of the heavens and how the world is working then you can start to make this connection between these abstract principles and constructs that are this whole set of places that you can go in your memory, in your imagination. And functionally, what he's doing is to be able to combine and break down sentences and to have these metaphoric representations that are located in specific places in his memory palace, such that he can connect these things together and kind of stream together these sentences of these stories that are happening within his imaginal idea of this memory palace. So just really emphasizing and using this concept of our place cells, being able to use that loci or that being able to locate our memory in a physical place, and that when we learn something and have a very distinct architecture, we can recall that memory by going and seeing that place again. So how can we start to do that to improve the way that we think and communicate? I think that's some of the questions that I'm really interested in thinking about. It just starts to really give a lot more power to these 360 video Google expeditions of being able to take guided tours of these ancient cities. And you can get this embodied experience of these different physical locations, such that when you're studying that part of history, you can actually go there in Google Earth. You can watch a 360 video. Then you can learn about it. And then all those things you can start to really put into your body in a new way. So this idea of the connection between place and memory and the place cells and memory palaces and all these deep ideas from the ancients I think are being recontextualized today and so that the more that we study about what these ancients were saying the more that we can start to create these VR experiences where we can create our own memory palaces and to be able to start to create environments that help us really think deeply about specific topics because our environment is an extension of our thinking and our brain just like you use a whiteboard to be able to write down different notes or if you're taking notes on a piece of paper and you're drawing pictures all that is an externalization of your thinking and that If you start to extrapolate that into an immersive environment, then you're starting to get to this idea that Viya is alluding to here, is that there's going to be all sorts of different really sophisticated tools that are going to improve our thinking. And this is a topic that I've covered before with Chris North saying, you know, how can some of these ideas of embodied cognition be connected to space and to help the process of sensemaking for intelligence analysis? But I think these are the types of tools that we're going to start to see more and more in 2017, 2018. These thinking tools that allow us to go into VR, to be able to have these conversations, to map out these conversations in real time, and to be able to take our level of thinking and understanding in an embodied way beyond what we've ever seen before. And I think some of the implications of this range the full scale of what you can do in augmented reality, what you can do in virtual reality, but I think there's just a huge application between embodied cognition and the future of education. So. That's all that I have for today. I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and become a donor. Just a few dollars a month makes a huge difference. So donate today at patreon.com slash Voices of VR. Thanks for listening.
