Ryan McMahan is Assistant Professor of Computer Science at the University of Texas at Dallas, where his research focuses on the effects of system fidelity for virtual reality (VR) applications and systems. He had an interest in VR training applications and was presenting at IEEE VR about a realistic ladder climbing technique called “march-and-reach.”
Climbing a ladder sounds in VR like it’d be a fairly straight-forward problem, but yet this interview shows all of the various nuances and design decisions that had to be made in order to both accurately replicate the feeling of virtually climbing a ladder, but also do it in a way where they could teach ladder safety.
There were previous ladder climbing techniques that would either be purely hand or feet-based methods for controlling vertical locomotion. Ryan decided to use the feet for controlling the vertical locomotion, and that there needed to be at least two points of contact at all time otherwise the person would fall off the ladder.
Part of why VR training can be so effective is that you can show people what a failure condition looks and feels like without putting someone’s physical safety in danger. They can make people fall of a ladder in VR and show that they’d break their legs, and having this level of realistic fear can actually help create episodic memories that can help people more effectively remember to always keep two points of contact while climbing a ladder.
There were other nuanced changes that they had to make in order to make the simulation more realistic. While you’re climbing a ladder, you’re holding on to the rungs of the ladder and leaning backwards so that when you look down you can see your feet. But yet if you’re walking in place and look down to see where your feet are, then you actually have to lean forward if the feet are tracked accurately, which would make people loose their balance and potentially fall down in real life. So Ryan had to create an offset the feet by 10 degrees forward so that people would be able to actually see their feet in VR while still maintaining their balance.
Here’s more information on the march-and-reach paper that Ryan presented at IEEE VR.
March-and-Reach: A realistic ladder climbing technique
In most 3D applications, travel is limited to horizontal movement. A few 3D travel techniques allow for vertical travel, but most of them rely on “magic” abilities, such as flying. We sought to develop a realistic vertical travel technique for climbing ladders. We have developed March-and-Reach, with which the user marches in place to virtually step on lower ladder rungs while reaching to virtually grab higher rungs. We conducted a within-subject study to compare March-and-Reach to two prior ladder-climbing techniques. Results indicate that users consider and treat March-and-Reach as the most realistic ladder climbing technique.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:12.019] Ryan McMahan: I am Ryan McMahon. I'm an assistant professor of computer science and arts and technology at UT Dallas. And my research mainly focuses on using virtual reality for more effective training.
[00:00:24.789] Kent Bye: Okay, great, and so maybe you could talk a bit about some of the talks that you're giving here in terms of the march and reach and locomotion and virtual reality and how that kind of came about.
[00:00:33.858] Ryan McMahan: Okay, so one of the areas that we are focusing on is basically safety in the construction and mining industries. One particular aspect is ladder climbing, so basically how to properly climb a ladder and safely. So one of the issues here is there weren't really any realistic techniques for climbing a ladder in VR. So before we can get to the point of training people, we really need to have the ability to allow them to have the realistic motions of it. So that's what we focused on and that's what we presented here this week. Our technique, March and Reach, where you basically use your arms to grab the rungs above you and then use your feet to go up or down the ladder rungs.
[00:01:10.571] Kent Bye: And so without the presence of an actual physical ladder, how effective is it to kind of pretend that you're climbing up a ladder when you're not actually moving up? So maybe talk a bit about that disconnect without having that real haptic feedback and how effective that actually is within the context of a VR simulation.
[00:01:29.113] Ryan McMahan: Sure. So I would say for people who are very familiar with ladders, our technique offers a very realistic aspect of interaction, at least in terms of the cognitive load of thinking, I need to place my hand here, I need to place my foot here. In terms of the psychomotor skills and the biomechanics involved in our technique, We obviously can't match an actual ladder because we are missing the force feedback from the actual physical ladder. There's a few balance issues with a head-mounted display. When you have people stand on one foot or step in place, they kind of sway from side to side, which isn't necessarily realistic. But again, I think cognitively it has the same demands as real ladder climbing and that's where we think that we can improve Ladder safety is by focusing on the cognitive demands and focusing on Teaching the workers to focus on where to place their hands and how to move their feet I see and so, you know within the context of virtual reality, you know, the visual feels pretty dominant and so I'm curious within the context of doing a ladder climbing simulation if
[00:02:31.518] Kent Bye: you're kind of tracking the limbs and then trying to actually move and vertical scale a VR simulation in order to kind of match that. Whether or not when you're in a VR environment that it kind of actually feels like you're climbing a ladder even though you're not even moving up or down at all.
[00:02:45.588] Ryan McMahan: Sure. So basically what we did for this initial work was we were tracking the hands and we were tracking the feet. So you could see your virtual hands move and you could see your virtual feet move. We didn't do full body tracking, though we could. In terms of feeling like you're actually climbing up the ladder, you very much get that sensation because visually it appears as if you're climbing up the ladder. You see your hands and your feet go in the proper rungs. You see your entire point of view lift off the ground. Some people really responded when they're near the top of the ladder and they forget and only have like maybe one point of contact so they fall off the ladder. You can kind of see them jar a little bit. So I think visually we're very compelling. We can convey that action of climbing a ladder.
[00:03:27.975] Kent Bye: Yeah. What were some of the other previous techniques in this realm in terms of vertical locomotion in VR?
[00:03:33.876] Ryan McMahan: Sure. So there are really two categories of vertical locomotion techniques. One would be what we call the magical techniques. So flying or what's called hand-directed steering, where you basically use a six-degree freedom device to point in the direction you want to go. And then you press a button or press up to fly in that direction. So you can go in any direction you want to. This obviously isn't very realistic though, right? So you could fly up in the middle of the air and just stop and hang there and there's nothing around you. The other category of techniques were more realistic techniques. And the two that we had found in literature, one was walking in place, where Slater and his colleagues used the walking in place technique that basically when you ran into a ladder, you would start climbing up the ladder. The problem here though is you don't have the realistic motions of the hands. And then the other technique that we came across was last year there was a video demo with ladder climbing. And they used this reaching technique where basically you grab a rung. Once you grab that rung, if you push it down, you go up. If you move that rung up, you go down. very realistic in terms of arm motions, but now we're missing the leg motions. So initially what we thought, well, we'll just take those two techniques and smash them together, but that actually doesn't work very well. There's a lot of nuances, and that's really what we presented in the paper were the small nuances of putting those two techniques together.
[00:04:50.507] Kent Bye: Yeah, and so, I guess, how do they fuse together? I guess, you know, maybe just kind of walk through the motion that you have to go through in order to actually kind of realistically simulate the process of climbing a ladder in VR.
[00:05:00.279] Ryan McMahan: Sure. So, what we had to do, though, by smashing the two techniques together, we kind of had to pick which one was actually controlling travel. So we decided to allow walking in place to control the vertical travel. So what you do first is we have you use the kind of reach and grab technique to reach out and grab a rung. Now what this represents is it represents a point of contact. If you move your hand down or up, nothing's going to happen. You're not going to travel up or down. But now, because our technique knows that you're grabbing wrong, when you step in place, it allows you to push yourself up. Now, the way that we differentiated between going up and going down was we were using Wii remotes, so we had the user press A and B button to indicate that they're really trying to grab and go up, and we had them just use the B button to indicate, okay, I'm going down now. And that changes the direction. In terms of the nuances, though, One thing that we realized was, again, in a head-mounted display, when you have someone stepping on one foot, they kind of sway a lot. So at first we were trying to have the users keep their hands exactly co-located with the virtual rung, but as soon as they start to step, their body and their hands sway quite a bit. So we had to create this bounding box around the original point of contact. And basically, as long as their physical hand was within that bounding box, we assumed that their virtual hand stayed in the same location. And that dramatically improved the usability of our technique. And then the other issue, too, was normally on a ladder, when you have your feet on the ladder, your body actually kind of leans backwards a little bit, so you can kind of see your feet. But with this, you have nothing to lean against. So in order to provide feedback to the user about the position of their feet, we actually had to position the feet about 10 inches in front of their physical feet so that they could look down and see it. And what this does, it kind of gives them the simulation of leaning back, but not exactly.
[00:06:47.325] Kent Bye: I see. Did you find that that would affect people's balance in terms of when they're actually standing straight up, but when they look down, it may sort of appear as if they're leaning back?
[00:06:55.151] Ryan McMahan: So it actually had the reverse effect. It improved their balance. Before, when we weren't pushing the virtual feet 10 inches in front, people were trying to look at their feet, but they couldn't. And they would try to physically lean back to look at their feet. And then they would step completely off the ladder. So by visually displaying the virtual feet in front of the physical feet, we've actually improved balance with the technique.
[00:07:14.084] Kent Bye: And did you actually simulate, if you were to step backwards, like falling down off the ladder? Or would that just induce sort of motion sickness if you did that?
[00:07:20.850] Ryan McMahan: Yeah, so we simulate falling. Basically, either if you step backwards off the ladder, or if you don't maintain two points of contact with the ladder, we just basically have you fall. So all the vertical travel you've done, we just basically reverse it and let the gravity take effect and bring you down to the ground. Now, we don't do anything gruesome, at least at the moment, but, you know, it's obvious that we could do things like show your broken leg or something, right?
[00:07:44.214] Kent Bye: So I guess that's part of the training that there are consequences that if you don't do it correctly, then you fall down. And so maybe you could talk a bit about that process of training in VR and some of the other higher level concepts that you're trying to integrate in these types of simulations. Sure.
[00:07:59.236] Ryan McMahan: So one of the things that we're really trying to leverage with virtual reality and training is this concept of episodic memory. So episodic memory is very much kind of like your first person point of view memory. So if you ever think back to a memory and you kind of see yourself there in first person, that's most likely an episodic memory. And episodic memories are normally closely tied to emotion, especially intense emotions like fear or extreme joy. So with the training, especially with safety training, if we induce states of extreme fear, so for instance if you fall off a ladder and you look down and your virtual leg is broken and it's moving really weird even though your physical leg is fine, we imagine that we can induce a certain level of fear, and hopefully that will create an episodic memory of the event, and then later on when they're on a real ladder, maybe they'll think twice about not using the points of contact that they should.
[00:08:52.131] Kent Bye: I see, yeah, that seems like a tricky balance in terms of like creating this sense of contrived fear, but yet use the fear to actually help with learning. And so, you know, how do you strike that balance in terms of, you know, how much is enough fear, and how do you know when you've gone too far?
[00:09:06.882] Ryan McMahan: Right, so that's actually a very interesting future research question. We're just now beginning to get into those details. There's actually a very interesting paper here that was presented yesterday and they had some very interestingly gruesome scenes from like not doing what you were supposed to do. I don't think that they've investigated really what's the proper level of fear either, but this is definitely an open research question that the community needs to look at.
[00:09:30.047] Kent Bye: I see. And so what other type of factors of training are you looking at at your virtual reality research lab?
[00:09:35.609] Ryan McMahan: So in addition to the construction mining industry, we've also been looking at robotic surgery operating rooms. Not necessarily in training the surgeons themselves, there's plenty of simulations out there, but the nurses often get left behind. There's almost no training at all for the nurses on how to prepare the surgical robot, how to do the interoperative skills that they need to, and how to clean up the room. So we're looking at using immersive virtual reality to show them things like how to drape the robot with sterile drapes, how to change out different tools, and how to clean up the room. In addition to that, we're also dealing with some very rudimentary training tasks. We've been using Lego a lot. It's really nice because after the person completes the virtual reality training, we can bring them back at some point later and give them a real Lego model and have them rebuild it, right? So it's a very nice way of assessing how well our virtual reality techniques work.
[00:10:26.735] Kent Bye: Yeah, and I guess that's the challenge is that, you know, with a virtual simulation, you're trying to emulate the real world. And so I guess there's this trick of knowing when is the right time to actually do the physical training in the real world, because it's maybe easier, faster, cheaper, more effective, versus going into a virtual environment. So what are some of the big factors that you look at in terms of deciding whether or not this would be a great use case for doing a virtual reality training?
[00:10:51.378] Ryan McMahan: Sure. So I think one of the things really that we're kind of focusing on is doing things in virtual reality that you can't do in the real world, right? So I could give you a real ladder and I could tell you exactly how to climb it and you could climb it all day long, but I wouldn't be able to give you the experience of falling off that ladder without you actually getting physically harmed with virtual reality. We can give you that experience of falling off the ladder, but we don't physically harm you. So we're looking at case scenarios like that, places where we think that we can dramatically improve the training by using virtual reality, not just replace real-world training.
[00:11:26.353] Kent Bye: I see. And so what are some of the other scenarios that you've seen that would be impossible or difficult or too dangerous to do in the real world, and so they end up doing it in VR?
[00:11:35.923] Ryan McMahan: So, there's quite a few. The military is a huge user of virtual reality technologies and simulation. Obviously, you wouldn't want someone on a real battlefield for training, you know, having bullets whizz by them. Of course, they can do paintball and things of the such, but you still wouldn't be able to simulate things like explosion and whatnot very well. The medical field is obviously a very important area for training because in the real world you would have a patient's life at stake, right, for them to train. So that's one reason they use simulation mainly. So any case scenario where life or bodily harm is a potential factor, virtual reality really should be used.
[00:12:14.208] Kent Bye: And how did it come about that your lab at UT Dallas is sort of focused on virtual reality training?
[00:12:20.192] Ryan McMahan: Well, to be honest, I've always wanted to focus on training in virtual reality. I didn't quite have that opportunity as a PhD student, so now as an independent professor, I get to kind of choose what I want to dabble in, and now I get to finally play with training in virtual reality.
[00:12:35.021] Kent Bye: And what was it that was drawing you to the VR training then?
[00:12:38.046] Ryan McMahan: Well, I think virtual reality has been around quite a while. Granted, it's really had a big boost in the past couple of years, but we really need more killer apps, right? More apps that show that virtual reality is a huge, important thing that we should focus on, not just for video gaming. And I think training is a great context for that.
[00:12:56.004] Kent Bye: Great. And finally, what do you see as the ultimate potential for virtual reality?
[00:12:59.728] Ryan McMahan: Ooh, that one's hard. In the near future, probably just something that would be easy to set up, head-mounted display most likely. If you could afford multiple users that would be great in the same space, then you could have collaborations. Obviously, it needs to be high resolution. Long term, though, I actually expect something like the Matrix to pop up. There's a lot of brain-to-brain research going on right now in neuroscience, where people are able to take the signal from one brain and send it to the other brain for interpretation. That, to me, indicates that the core concept of the Matrix is probably very feasible. I think it's a very long ways away. There's a lot of roadblocks to that technology, but I think eventually we may get there.
[00:13:43.535] Kent Bye: Yeah, it's interesting. I've been just talking to people here, asking about the killer app, and to me, I think neuroplasticity may be the killer app of VR in terms of being able to rewire your brain. But you had mentioned, you know, some of the brain-to-brain communication, but what other types of insights of neuroscience or that type of neuroplasticity do you see that would help kind of lead towards this vision of the matrix or being able to kind of rewire our brains to be able to do things that we couldn't do before?
[00:14:09.223] Ryan McMahan: I think the biggest thing is understanding how the neurons are mapped out in the brain and understanding how information is stored in the neurons. I think that's where neuroscience is really making a lot of strides right now and that's one reason they're able to connect two brains and have one brain kind of control the other brain's interpretation. So yeah, I think just that mapping of the brain is really huge.
[00:14:29.343] Kent Bye: OK, great. Well, thank you. Thank you. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
