During the Enlightenment, René Descartes declared that the mind and body were split and that we should think about them as separate dualistic entities. But more and more evidence is pointing to the fact that our bodies are much more involved in cognitive processes than we ever thought before. One of the most interesting theories along these lines is called “embodied cognition,” which asserts that we learn about the world by manipulating and interacting with it through our body and all of our senses and that the context turns out to be an extremely important part of learning as well.
LISTEN TO THE VOICES OF VR PODCAST
I went to an Embodied Learning educational workshop at the IEEE VR academic conference in March where “embodied cognition” was the hottest buzzword throughout the entire day. I had a chance to meet and talk with Clemson education student Nikeetha D’Souza who worked on an interdisciplanary team where they used dance to teach middle school girls concepts of computational thinking.
They used a visual scripting program called “Virtual Environment Interactions”, which is abbreviated as VEnvI. Students could choreograph a dance routine by using the visual scripting language to do move sequences, conditionals, and repeating loops. This would control a virtual avatar who would be executing the dance. And then they would go into a virtual environment and then dance with the digital avatar.
They wrote up their results in a paper titled “Programming moves: Design and evaluation of applying embodied interaction in virtual environments to enhance computational thinking in middle school students.” They found that “students enjoyed the experience and successfully engaged the virtual character in the immersive embodied interaction, thus exhibiting high telepresence and social presence. Students also showed increased interest and excitement regarding the computing field at the end of their summer camp experience using VEnvI.”
For more information on Embodiment Theory and Embodied Cognition, then be sure to check out my previous interviews with Saadia Khan and Chris North
Homework for Immersive Analytics: Read Margaret Wilson's "Six Views of Embodied Cognition."https://t.co/1kQffI3Y2Z pic.twitter.com/kbXzS7QLKK
— Kent Bye (Voices of VR) (@kentbye) March 20, 2016
Unity plug-ins like Infinite Gesture will enable the next generation of embodied cognition applications:
OMG! VR INFINITE GESTURE @UNITY3D PLUGIN LOOKS AMAZING!!!
Uses #MachineLearning by @edwonia: https://t.co/nWF7ZlAwYF pic.twitter.com/pXmJSk6tyM— Kent Bye (Voices of VR) (@kentbye) July 1, 2016
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye and welcome to The Voices of VR Podcast. So when I was at the IEEE VR Academic Conference this year, I went to one of the educational workshops to see what were some of the big topics and themes that were coming up in terms of how virtual reality can be applied to education. And one of the buzzwords that kept coming up over and over again was this concept of embodied cognition. So embodied cognition is essentially this thought that we don't just think with our minds and our brains, that we're actually using our entire body, that we can think of our body as an information processing machine, not just our mind and our brain. And not only is your body a part of your brain, but also your surrounding environment. So what that means is that you can create virtual environments and be able to do specific things using your entire body, and that'll actually help you learn better. So that's the basic concept that we'll be covering today with Nikita D'Souza, who was collaborating on this project where they were actually teaching kids how to do computational thinking through dance. So that's what we'll be covering on today's episode of the Voices of VR podcast. But first, a quick word from our sponsors. Today's episode is brought to you by The Virtual Reality Company. VRC is at the intersection of technology and entertainment, creating interactive storytelling experiences. The thing that's unique about VRC is that they have strategic partnerships with companies like Dbox, which is a haptic chair that takes immersion and presence to the next level. So they're making these digital out-of-home experiences for movies, studios, and original content. For more information, check out thevrcompany.com. Today's episode is also brought to you by The VR Society, which is a new organization made up of major Hollywood studios. The intention is to do consumer research, content production seminars, as well as give away awards to VR professionals. They're going to be hosting a big conference in the fall in Los Angeles to share ideas, experiences, and challenges with other VR professionals. To get more information, check out thevrsociety.com. So this interview with Nikita happened at the IEEE VR conference in Greenville, South Carolina, happening between March 19th to 23rd. So with that, let's go ahead and dive right in.
[00:02:32.709] Nikeetha DSouza: So my name's Ankita D'Souza, and I'm in the School of Education at Clemson University. And we work with an interdisciplinary team along with the University of Florida and my own school's computer science department. And we have this software called Venvi, which uses dance to teach computational thinking concepts to middle school students. That would be the one-line definition.
[00:02:55.520] Kent Bye: So you're using dance to teach computational computing principles. So how does that work?
[00:03:01.532] Nikeetha DSouza: OK, so to make this big word, computational thinking, simpler, it's basically just the basics of what programming and coding is based on. Like you would make a sequence of codes, or you would loop a code, or you would use a variable. So these basic concepts is what we're trying to teach students in the earlier grades to get them interested. And what we're doing is, the reason that we're using dance is because we firmly believe in this theory of thinking called embodied cognition, where we believe that we use our bodies to think. And so there are a lot of work that does use like hands on activities and gestures. But what our project is unique is we have the software that uses dance and programming. But also we have a curriculum around it that is also embodied. And that's why it's so specifically dance. So how a classroom would look like what we were doing is we'll first get the kids to just get up and dance and it could be a simple song that would give you instructions to like step to the left or step to the life and turn around and that's easy to follow and so we use that analogy or that metaphor you could say to teach the computational concepts.
[00:04:10.545] Kent Bye: Great so maybe you could give me an example for what this looks like and how are you using VR to actually sort of do this beyond just a dance?
[00:04:19.281] Nikeetha DSouza: So like I said, the software right now is a simple PC software and it's just a drag and drop interface. Basically what the software does is the kids can program the virtual character to dance. So you just pick a few steps and you get the virtual character dance. So what the virtual reality does is these kids, once you've programmed this character to dance, you can actually be in this environment with that virtual character and dance with them. And that just, how cool is that? Like you just, you're dancing in this virtual world with a virtual character that you programmed. And so, I mean, beyond the obvious, it gets kids engaged. They're extremely excited. And we have interview data. saying how cool this is or how wonderful this is and how they would use this more often and it also gives them the idea of what computer scientists really do, you know. They just think these computer scientists are people who just sit in front of a laptop and just like code all day but they really don't know what the work looks like and so having these kids play with all of these software not only teaches them the concepts but also gives an idea of the world of computer science and that's how moving is what we're hoping to do in our project, I guess.
[00:05:28.746] Kent Bye: So it sounds like they're in a virtual environment with this avatar that they are having to write a computer program in order to give the instructions for this avatar to dance. And so maybe you could kind of step through some of the first steps, first moves, and the concept you're trying to teach, and maybe kind of walk through a little bit of the progression as how they move forward.
[00:05:49.340] Nikeetha DSouza: Okay, so as soon as you open the software, you get the character customization page. Right now it's at basic, so we probably have like just a male and female and you can probably pick two or three outfits. We're hoping to do more so that kids could get, that's also another area we're interested in, to expand how the character looks. But for now, we have this avatar that they work with. And you have a sequence of moves that have been made into blocks, you can say. So it would be a clap, or a hop, or the cha-cha. And you'll have all of these blocks available. All you need to do is drag and drop the blocks that you like and create a dance sequence. So you've choreographed your character into dancing the way you like. Our software doesn't have the music right now inbuilt in it, so we use external software. The kids can pick their songs from YouTube. And you'll have kids programming these characters to dance to their favorite song. And they often dance with these characters. They just put it up on the large screen and they dance with them. And then the next step is connecting the head-mounted display and then putting this on. And they just do the same thing. They dance with the characters in the steps that they had programmed.
[00:06:59.960] Kent Bye: And so how does that teach computational thinking then?
[00:07:03.320] Nikeetha DSouza: So like I said, a lot of this is also the way we have conversations with this. So when they're doing this, they're also dancing. And so let me break that down. So the first time is, like I said, the hustle when we dance. So they're dancing to the song that says, hey, step right and step left and kick your leg out and turn around and do it all over again. And then we talk to them and we ask, what did y'all just do? And it's simple as dance. If you think about it is, hey, you picked these steps and you put them in an order, which is a sequence. And some of these steps you repeat, which is a loop. And sometimes there's a condition, oh, when this part comes, you do that step. And so it's a conditional. So we use dance as a metaphor to teach this. And the fact that they get to do it on the software, They pull these blocks together. Now for them, making a sequence makes sense to them. Using a conditional makes sense to them. Using a loop makes sense. So it's more context-based. And what is really cool is that we've taken something so simple and something so everyday, like dance, and just dancing to a song. to saying hey coding's not that hard like it's as simple as putting these things together. Obviously it's a really simplified version and it is done so for middle school kids to get interested in coding and also removing the barrier of oh my god you have to be really intelligent and Computing is really hard, and so unless you fit in that category, you'll never be able to code. But we've realized that coding is so important, and so we need to bring in more people, and so that's the aim of this project, to make it more approachable, so that we can get more people involved in coding.
[00:08:42.564] Kent Bye: Yeah, and so maybe you could take a step back and talk about these theories of embodied cognition, and what kind of studies have been done, and what they've found, and how that's informing what you're doing.
[00:08:53.418] Nikeetha DSouza: So I think the biggest names in embodied cognition, like without the technology aspect, a lot of work has been done in Berkeley. And there are these famous others, George Lakoff does a lot of work, Dodd-Abrahamson does a lot of work, and we've based a lot of work from their work, and the big idea of constructivism, which is in education, where we believe that kids, or anyone actually, construct their own meaning, construct their own knowledge. And so taking it from there, you move to embodied cognition, where you're thinking of constructing this knowledge through your body, you're thinking through your body. And I think if you remember from the workshop, The other educator pointed to Rene Descartes. It was because he was so important in splitting the mind and the body. Before the earlier thought was like the body is just a vessel for the mind and nothing else. But now we've realized like there's a lot of sensory motor learning. We learn by touching and feeling and seeing. I think the biggest work is, let's start with the go-go boards. I think they're like blocks that kids can put together. It's like the Sesame Street blocks. When you take these blocks and you're working with them, there is more learning than just passively sitting in front of a screen and watching something. So the GOGO board is basically developed by Sipitakiat, Blickstein and Cavallo. Basically these are programmable bricks that you could really construct models of anything. So they're just made as blocks like how Lego blocks would be and it would be up to the teacher or the instructor to utilize it and I think the one study that they have done was they made kids use these blocks to build the ideal city. and explain through that and these blocks were used for that. And I think that's also coming back to the point of what I was trying to say that you don't just learn computational concepts in isolation, you learn what they could be used for or where they come from maybe. So the idea that these GoGo blocks, of course you could just play and build something, but also you're building for your community. What does your ideal community ask? You're asking these big questions like, How do you want your world to look like? You're asking these little children these big questions but using simple blocks. So let me tie it back to embodied cognition is basically the idea that use manipulatives to construct your ideas and something tangible and visible to you. I think that would be the difference.
[00:11:17.232] Kent Bye: Yeah, one word that gets thrown around a lot is multimodal. So maybe you could sort of expand on what that means and how that applies there.
[00:11:24.903] Nikeetha DSouza: So I think multimodal, first of all, it could just be different types of modes of communication. So it doesn't have to always be embodied. It could be if you're talking and writing and watching TV at the same time. That's multimodal. So a multimodal classroom could be just where the teacher has used a movie and she's asked the kids to write something and then they probably draw something. It could be like two, three modes. So I think embodied cognition brings in this very mobile idea to this multimodal aspect. And that's what we've been talking about. We strongly believe that thinking is no longer just in the brain. A, it's obviously through the body. And B, it's through the body interacting with the environment. And C, it's context-based. So all of this encompasses embodied cognition. And that is how you learn. So when you're talking about social learning and maybe talking to someone else and learning, It is because your body is interacting with the environment and it's in a context. And so all of these aspects of embodied cognition enhances that multimodal learning. You're suddenly opening up these options to the ways you can learn. And it's not just writing and watching a screen anymore. It's moving your body, it's using gestures, it's using manipulatives, building a model. or even dancing the shapes for your geometry. It could be any of these. And embodied cognition is heavily, there's a lot of work done in math, and I think it's led by Rafael Nunez. There is work done in computational thinking, a little bit of physics. And I think there was one work done using language Chinese characters. So the field is just about getting started. And so you probably have just one or two research projects being done. But the hope is that you would move all of learning into an embodied context because that is how we naturally talk and naturally learn and interact with the environment.
[00:13:19.341] Kent Bye: Yeah. And maybe you could go a little bit more into your background from the education side and then how you ended up working in these immersive technologies.
[00:13:26.962] Nikeetha DSouza: That was actually just luck and coincidence. My background is actually in science. I was a biochemist. Also, I dabble in art a bit, and I always found the two to be separated and done in isolation. And I think I was looking for a way where we could work the art and the science together. Like, why should science not be beautiful was my idea. And so when I came to Clemson and interviewing with professors that I was interested to work with, this was the ideal project because they integrated something so scientific like computer science and programming with something so what we would consider artistic and have no science in it like dance and I liked that combination and getting into it was this other layer of embodied cognition which is what drew me in where you're using your body it's a completely new type of thinking nobody ever tells you that hey your body is another aspect through which you learn And the VR is such a natural fit for embodied cognition, and I think that's why there's a lot of work related. Because we can talk about embodied cognition and how you would use manipulatives, but the VR opens this world where you're in this world and moving your entire body in a virtual environment. I think the world is heading to a way where most things now are going to be online or virtual. But we shouldn't lose this aspect of embodied cognition because we firmly believe that we need the body to learn. So I think VR can now bridge that gap, like when we're moving from the real to the virtual. virtual reality technologies give us that advantage of using our bodies in a virtual space because right now virtual space in my head looks like Facebook which is an online forum for discussion and that's how most online classes are, right? Just a video and you probably have to post a discussion or type in an answer and send it. And you're completely taking away this whole interactive environment that you could have. And I think VR is that answer when we're moving from the real to the virtual world. It can really push learning because it's utilizing something so basic and so instinctive, like embodied cognition.
[00:15:32.762] Kent Bye: One of the things that you said in the workshop is that you can qualitatively see that concepts are being taught, but it may be a little bit more difficult to quantitative measure what is actually happening in this type of learning. And so maybe you could start with, tell me a little bit about how can you qualitatively tell that this is effective?
[00:15:50.329] Nikeetha DSouza: So qualitatively is when you remove the number aspect, right? So you can either discuss it in detail, you can actually see this. And so for example, we have this one setup where we worked in a dance school. We worked with biology concepts in the school. So we couldn't give them a paper and say, define the nucleus and define the cell because we didn't really get to that point. But how we see this qualitatively is the moment you got these kids to say, hey, like, This is what the cell is like. And do you think you can make this into a dance show? You're pushing kids to think of the definition of something like my favorite example of all, the nucleus is the brain of the cell or controls the cell. So you're pushing them to think about what that actually means through this embodied cognition. And so you can see these kids act out these. And oftentimes in classrooms, you're ignoring all of the knowledge of these kids. because they cannot write it on a piece of paper and you cannot give them a score. And so what I mean by qualitatively is like that student who could twirl her partner and say, I'm the nucleus and so I control her action. She's completely understood the concept. But if you write that answer in a standardized test, you're not going to get anything. So I think that's a struggle that we're trying to see. She's obviously learned what it is and that's what you want her to know. But in a standardized test, like she doesn't know anything technically. And so I think that's what we're struggling with, is how to assess the learning that they can. And embodied cognition is so body movement. How do you assess what the body is doing? It has to be done through video or probably modern technologies. So we'll have to really up our game in terms of assessment. Again, I think VR is possible because now with the Kinect, you can track movements. So maybe, maybe analytics embedded in these VR systems could be a better way to judge learning. But that's like so off. That's just something you can think of, but I don't know how it would work in the real world, you know.
[00:17:51.816] Kent Bye: And so what do you see as some big wins in terms of topics or areas that make perfectly good sense to take an embodied cognition approach to teach?
[00:18:02.137] Nikeetha DSouza: I don't know. I think like maybe certain aspects that are spatially organized, maybe like chemical molecules could be embodied because you need the idea of concept of space. But if you see research, there's a lot of abstract concepts like mathematics and what we did like the cell in biology that it fits because like I said, it's very natural for us to think through our bodies and it could just be gesturing what the molecule looks like or the cell looks like. I don't know if there's a limit to what topics could match with embodied cognition. But are you saying immediately what could work is what you're hoping?
[00:18:37.252] Kent Bye: Either that or I'm also just kind of curious like what do you want to experience through embodied cognition? Like what type of experiences would you love to have to learn topics through your body?
[00:18:47.793] Nikeetha DSouza: Because I think I'm partial to chemistry. And as a chemistry teacher, a lot of students struggled with organic chemistry and biochemistry. And I think it's because they just couldn't see how these molecules interacted in the environment and in their own bodies. And I think VR would be amazing. Imagine being able to play with these molecules or a simulation of these molecules and see how they work in the environment, and then put them in your body and see how that works. a lot of that part of chemistry is spatial and so ideally I would like to see whether those possibilities are. Right now my work is in programming and that is also very close to my heart because I think now whatever you be in the future you need to have some sort of programming or coding skills and I'm just excited by the fact that you can learn coding through embodied cognition. And by the way joining this project is when I learned the basics of the programming and actually after that I did take a programming class in Python. And so I know this works because I had the first-hand experience because I've never ever thought of myself like a computer person at all. But now I can tell you basics and I can probably write some basic Python code is because of being interested and being told like this is what coding looks like and you can learn it basically any way you want that comes naturally to you. Like dancing, yeah.
[00:20:13.280] Kent Bye: So what's next with this project then? Where does it go from here?
[00:20:15.506] Nikeetha DSouza: So the VR aspect itself is next, because we just only ran like one test drive to see if, I think the word that they use is presence, like how much of a presence you feel with the character. So I think we're probably going to delve deeper into that. And the next is actually gamification, because right now it's just program your character and you watch it dance. But can we gamify this, like points, or can we embed some data systems in here where we're assessing them? Can we make it collaborative? Can you have two people in the same virtual world? And so probably those are the next directions that we're going to.
[00:20:51.613] Kent Bye: And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:20:57.254] Nikeetha DSouza: Wow. I mean, this might be a little rebellious, but it might even eliminate the need to come to a physical world, like in the sense, get up and leave your home and come to a university class. You may still need that interaction, but I didn't know it could give access to, I'm just thinking like a kid in India, if you had that internet system though, a kid in India could access a university in the U.S. virtually and not just watch a video and write a forum. And that's exciting, like I can be in the U.S. being in India and that completely changes actually what it means to be embodied, like where your body really is. So yeah, it could break more barriers to where you travel, where you can go, what you can learn. You could learn about the ecology by being in an ecological environment. You could learn about the rivers by standing next to the river. Like, I think it made learning so much more fun. Right? Yeah.
[00:21:55.820] Kent Bye: Okay, great. Well, thank you so much.
[00:21:57.461] Nikeetha DSouza: Thank you so much. Yeah.
[00:21:59.042] Kent Bye: So that was Nikita D'Souza. She's in the School of Education at Clemson University working on an interdisciplinary team looking at how to teach computational thinking through dance using the principles of embodied cognition. So I have a number of different takeaways from this interview is that first of all I really am fascinated by this concept and idea of embodied cognition and This is a pretty cutting edge idea, this theory that our way that we process information, it goes beyond just our mind, but it's actually incorporating our entire body. And not only our entire body, but our environment as well. And so you can start to think about, well, how could you construct virtual environments to be able to optimize learning specific topics and be able to externalize our thinking in these different environments, but also inspire us to move around into different positions. There is an Infinite Gesture plugin that came out for Unity within the last two or three weeks and it essentially uses machine learning to be able to both learn and detect different movements that you're using with a motion controller like with the HTC Vive or the Oculus Touch. So essentially what that means is that you'll be able to kind of encode specific gestures that maybe are trying to be some sort of mnemonic to help you learn different concepts. I think there's going to be a little bit more of a kinesthetic language that evolves for people to perhaps encode information into their memory. Starting with this idea of using dance in order to teach computational thinking, I think there's going to be a lot of different other applications and I think it'll be interesting to see what specific topics are really well suited or if it just happens to be all topics are well suited for this type of embodied cognition and it's just a software platform that somebody develops where you're able to really take advantage of some of these concepts. So I'm excited to see where some of these educational applications go in the future, really integrating some of these concepts. So I've looked at embodiment theory before, back in episode 73, looking at how you can use some of these principles for learning. But also back in episode 375 with Chris North, who is really an expert on immersive analytics and talking a little bit more of some of these high-level concepts of embodied cognition. And specifically, there's the six different views of embodied cognition from Margaret Wilson. And I've included a tweet that I sent out there that you can dig more into the concept. So another point that I just wanted to really bring out is that a common theme through all different educational applications for VR is assessment. That's the big open question. And I think the thing that is really probably the most difficult problem is to really find some sort of way of proving that this sort of immersive technology is better in the long run than doing other techniques. I think that there's some sort of an impact and effect that the novelty of virtual reality has when people are first using it. So it's such a memorable experience that it's hard to tell whether or not it's the technology and it being new and novel, or if there's actually something more deeper going on within the different techniques that are being implied through the VR technologies. So I think that's something yet to see but this is something that is an issue with all of education and trying to prove the efficacy of one technique over another is something that I think is still pretty much a large open question and individual kids have their own ways of learning specific things and any type of standardized approach is going to be kind of an oversimplification of what type of education is going to be kind of universally applicable And I think that the real future of education is going to be integrating a lot of these artificial intelligence techniques and to be able to have these tutors to be able to actually teach you. In the process of doing the Voices of AI podcast, I was able to talk to some people about the future of Turing tests that people have, moving beyond this kind of simplistic chatbot type of test for the Turing test. And I think that One of the Turing tests is being able to create an AI that's going to be a scientific tutor, to be able to take somebody who's a child and be able to coach and teach them everything they need to know to be able to come like a professional scientist. And so you can imagine what type of sophisticated artificial intelligence would need to happen in order to make that a possibility. But part of that would be able to both be able to assess where a child is at, what they know, what they don't know, and kind of the learning process to best help them along their way. So AI is going to be a huge part of that, but I think that VR with these different environments that are tuned to do specific lessons, I think personally that there's going to be a pretty huge role for VR in the future of education. But in order to really bring this into mainstream education, you not only have to prove that it's effective, but you have to do like these longitudinal studies. So you have to look at it over long periods of time, which I think that's sort of the biggest challenge for doing any type of educational research. So just wanted to throw that out there. And so that's all that I have for today. If you enjoy the Voices of VR podcast, then please do tell your friends and sign up on my email list. I'll probably be having some more virtual events coming up here soon and give you more information as to when the Voices of AI podcast actually launches. But also I'll be at VRLA this Friday and Saturday, so I'll be roaming around. So if you're gonna be there, keep an eye out for me. Drop by, say hello. And if you'd like to support the podcast, then please do consider becoming a donor to patreon.com slash Voices of VR.
