#627: VirtualSelf Full Body Tracking in Cloudgate Studios’ “Island 359”

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So at CES in 2017, HEC announced their Vive Tracker. These are the tracker pucks that you're able to put on your body and to be able to essentially do more tracking of either different objects or different parts of your body. So then last year at GDC, Cloud Gate Studios had their Island 359 experience where they were able to do three different Vive trackers and to be able to do a full tracking of your embodiment within virtual reality. They come from a background of motion capture and so they always wanted to have the ability to track your full body within virtual reality. And whenever you have more of your limbs tracked to this higher fidelity, then it does a much better job of evoking this virtual body ownership illusion. It just deepens the sense of embodied presence. So they just came out with their full release of Island 359, having all these different features of this virtual self, full body virtual reality. So I had a chance to try it out last year at GDC, and I had this interview with Steve Bowler about the process of how they created this virtual self system that is flexible to be able to deal with one, two, or three different trackers. And so we cover all that and more on today's episode of the Voices of VR podcast. So this interview with Steve happened on Wednesday, March 1, 2017, at the Game Developers Conference in San Francisco, California. So with that, let's go ahead and dive right in.

[00:01:40.012] Steve Bowler: Hi, Steve Bowler. I am president of Cloud Gate Studio, and we are currently doing what we call our virtual self system, where you'll be able to see your whole body in VR. We're using the HTC trackers with the Vive, and we have an innovative little clip system. You usually put them on your shoes and your belt. You can see your whole body when you're in VR now.

[00:01:59.648] Kent Bye: Great. So yeah, this is the first time that I've had my legs in the game and be able to actually do something with them. I've had full embodiment before where I've had my feet tracked, and it gives an extra sense of presence. But maybe you could talk a bit about how this came about and what you were really trying to explore here.

[00:02:16.255] Steve Bowler: Sure. So, a little long-winded, but back when we first got our hands on a Vive dev in November of 2015, we have an extensive motion capture background, and we set it up in a motion capture volume, and we saw how accurate the tracking was in it. We call it sub-millimeter accuracy. That's good enough for professional mocap. And we were like, you know, if we just had more of those sombrero hats that came with the original Vive dev controllers with batteries on them, we could track a whole human body in VR. This is like an incredible system. And they were like, oh, that'll never happen. And we kind of just dropped it, right? It was a dream we could hit someday. And then we were working, and HTC announced at CES the new tracker program that they were going to show. And we were like, oh, wait. Remember that thing we talked about? That's the thing we wanted, right? We needed those pucks now instead of the sombreros or pucks. So that was the week before MLK weekend, and MLK weekend, we had both said we weren't going to work and we were really crunchy. And so I was going to play a game and I was bored and I couldn't find any games. And I couldn't stop thinking about the project that we wanted to do over a year ago. And I was like, I'm just going to start prototyping the rig now. And I told Jeremy, I'm like, I'm just going to start cranking on it. And so I did the double Vive hack to get four wireless controllers right. I don't recommend anybody trying it. It's really cumbersome and not fun, but it works. So we prototyped two feet and two hands with wireless controllers and we were like, this is proof that this works. That's our first experiment video we did, but I really wanted to see hips on it. So then I realized we could run a USB from the headset down to a hips controller. So we bought more controllers, right? I got a hammer holster to stick the controller into like carpenters use and we could track the hips and we're like, this looks really good. And that's kind of when everybody was getting on fire about it, like, wait, this could really work. And HTC agreed to send us three trackers in the program. I think they said we got the first three, so I think they were excited about it. And so we kept working with those as soon as they sent them to us, prototyping the calibration thing that you saw. It scales to the user. We have male and female avatars in there. And as soon as we got it in there, you know, we turned physics on so we could push stuff out of the way with our arms and our legs. And we were like, kicking all the things is like so fun that we've realized people ages ago said, I hate the compies. I just want to kick those little bastards. Right. Cause they get really low and they're by your feet. And we were like, let's just give weapons to the players feet and see what happens. And sure enough, kicking copies is really fun. And then we realized we could kick raptors. And as you saw, you actually went after an allosaur and picked a fight with it and started kicking it in the legs, which I was like, don't do that, Ken. He's going to eat you alive. And you somehow kicked him to death. So hats off. We haven't even done that yet. So I was pretty brave.

[00:04:50.599] Kent Bye: Well, it's a new mechanic, you know, I have a gun and a knife and I've shot a lot of things in VR, but I've never been able to take on. It was actually really fun because it was like this huge dinosaur that you just, okay, I'm going to sneak up behind him and like kick his legs. And like, yeah, I think it's a very, it can be a very satisfying mechanic to kick things. It almost feels like you need an entire game designed around just that mechanic. At the same time, how many people are going to have access to three tracker pucks? So it's a little bit of a dilemma as a game developer. Are you going to design something that maybe a very small section of people can do and have a very intense amount of embodied presence or have a game that you're also developing and add that as a bonus?

[00:05:33.187] Steve Bowler: So our goal right now is we want to put this in Island 359 first. We have an arcade mode that's fun for home people to play, but it's also sort of like, hey, VR arcades, here's a mode that's easy for people to spin up and play. So adding it for VR arcade use makes a lot of sense, because those are the places that are going to want that kind of super premium experience. And we agree, it's probably really expensive for the average Vive owner. So what we're doing is we're going to add scaling to it. So if you only own one Vive tracker and you have our mounting thing, you can put it on your belt and we'll give you, like, your whole torso down to, like, your hips, like, your pelvic cradle region. No legs, but you'll actually be able to see, you know, your stomach moving and your hips moving according to where you're moving, which is really nice, because then we can play around with things like giving you holsters to put the guns on on your chest and stuff. If you have two trackers, what we'll do is we'll take the, we won't use the hips one, put them on your feet, and we'll procedurally solve where your hips are. It's not as sexy, it's not as cool, you can't use the Shakira stuff, right? That's what our first experiment was, and we were like, this is fun, because you can now go back to kicking things. So you could kick things with just two trackers. And then if you have the third one, obviously you'll get the full experience, right, with the hips and the two feet. And it's just going to scale. We're just going to auto-detect when you start the game. Do you even have one? If so, hey, here's a calibration step. If not, just play the game. We're trying to make it as easy as possible, right? We're pretty sure a lot of the enthusiasts will at least buy one tracker. And we actually have some fans who are like, I am going to go buy three trackers for this right now. We don't expect everybody to do it, which is why it's scalable, right? But we're just excited about it. We think it's really fun. And we've seen people already just today who've played through it. And they're like, now that I've experienced it, they're like, I don't know if I can go back to playing without seeing my whole body. And we're not even done yet, right? It's not retail. It's not finished yet. So we're just trying to push the medium a little bit and see how far we can go with it.

[00:07:19.581] Kent Bye: Yeah, the thing that I was really struck by was being able to dance for the first time in VR, you know, like, and to have a mirror, because I haven't had that before. And, you know, I danced around a little bit too vigorously and it kind of moved my hips and I got a little disjointed, and so I was like, okay, I'm not going to dance too vigorously, but... I think that's an issue of moving around and potentially having the pucks move around too much such that you kind of lose the tracking. Can you talk a bit about what the constraints were with what you need to do to both constrain it to your feet but also have it communicate back to the computer?

[00:07:51.538] Steve Bowler: Right, so what we're working with now are, these are like our original prototype clip mounts. We just needed to get them on your body, it was like mission one. The whole issue of it like kind of flopping off because you're moving too violently should go away with our actual production clips that we're gonna sell. Those are gonna grab you a lot tighter, they'll sit on your belt tighter, they'll sit on your shoelaces tighter, so we shouldn't see that. But the whole idea of tracking the body parts comes back to our motion capture experience, where we know that you need to have a, we kind of nickname it like a hard mount. point, right? It can't be a soft mount. A soft mount is something that would be attached to your abdomen. It would wiggle and it would flop around. But we want to attach it to bone if you can, which is why we go for the belt, because the belt's really tight on your pelvic cradle, it's on your hip bones, right? That's why we want to go for your shoelaces, because the bones of your foot are directly underneath it and they don't really move. So you get a really nice, hard, fixed mount. And what that means is that we get really tight one-to-one tracking with what your body's actually doing, without vibration, without wobbling. Like we said, we see it now with the ones we have now, because they're super prototyped. They're not meant for production quality use. But we're pretty confident that we're going to get really tight one-to-one tracking with the actual clip mounts come Q2, whenever we actually see these things go to retail. So it definitely comes down to seeing that one-to-one tracking that we learned from our motion capture background. And that's what we're aiming for, and I think we can hit it.

[00:09:10.230] Kent Bye: Yeah, most VR games that are out there have made the decision because it's so tricky and difficult to track the elbows that you just don't track them at all because it can break presence if you don't have that just right. But if you're adding the feet you kind of need an upper body embodiment so you're kind of doing both that but also adding an additional one and the middle of your body. So maybe you could talk about figuring out if actually those five points are like the minimum that you need or if it's still like you feel like you need to have more to really get that extra full embodiment with everything.

[00:09:42.687] Steve Bowler: Right. Yeah. The five are definitely the minimum, right? It's your two hands and your two feet and your pelvis. Like we can go with less, like we described with the scalable system. Right. But to get the minimum actual, Hey, I'm being fooled into thinking I'm seeing, you know, really good facsimile of my human body movement. We really need to get your core, which is where your hips are coming from in your feet. And the IK solution for the knees that we've got working is pretty good. It's pretty robust. It turns out you can't do too much with your knee that we can't already guess from your foot to your hips, you know, through human biometrics and stuff like that. It turns out elbows are really complex, like they're a single hinge joint on the human body. But the problem is, is that your wrist is this like really complex, it's not quite a ball joint, it's like almost a double hinge joint. and then your single hinge elbow connects to a ball joint on your shoulder, and that ball joint on your shoulder connects to another, it kind of floats like a two hinged axis to your clav, right? So everybody thinks it's just an elbow, it's a one joint solve, and it's really not, because your wrist is one, your elbow's two, your shoulder's three, and your clav is four, but we're trying to make it look as solid as you expect it to be, and There's a trick, obviously the home people can't see this, but we can lock your hand in place, and even with your chest in place, you can put your elbow in a huge range of positions, and so we can't even guess this, right? So what we're gonna go with is we're gonna go with the best possible prediction of where a natural human elbow at rest or in an active pose should be in that area, and it turns out our elbow solve is about four times as complex as everything else that we have, and we're still not done yet. It's good, but it's not great, but we're pretty sure we can make it better.

[00:11:17.738] Kent Bye: It's definitely one of the better that I've seen. And of course, when I get in there, I try to break it a little bit. And it's still not perfect in that sense. But do you feel like the pelvis actually helps with the elbow tracking in any way, having that extra point?

[00:11:31.421] Steve Bowler: Yeah, it actually does. Because what we're doing is, when the hips are being tracked, what it does is it allows us to try and accurately predict where your chest is rotating from. Again, nobody's going to be able to see this, but I'll try and describe it. That if I stand hips forward, face forward, arms out like this, My arms are at exactly the same length, right? They're always going to be the exact same length. But the shoulders are at the exact same points on this axis, right? But if I rotate my hips 90 degrees, can you see, Kent, what's happening to my arms right now? My torso is doing a 50% blend between my hips and my forward-facing head, and my hands have changed position, right? It forces your shoulders to sort of in between the rotation between your hips and your head. And so we get better arm solving when we're tracking this because otherwise what happens is it like it doesn't know what you're really doing and it's just trying to guess and it'll totally put the torso in the wrong position. And because when I actually do this, my hand is physically being dragged back in space and we're tracking that. Like, if my torso doesn't roll, what happens is you get this. And now this elbow's bent, and my shoulders are square, and it looks wrong. So it actually does help in a really strange way. So.

[00:12:42.480] Kent Bye: Yeah, and what has been your experience of experiencing embodiment in VR with this higher level of fidelity now? What kind of experiences and stories do you have of that?

[00:12:51.845] Steve Bowler: I mean, really just what we've been doing now with this, right? We weren't even sure we could pull it off, so we surprised ourselves by pulling it off. And we're just really excited about it. Everybody goes like, don't go there. But when we put the female gender in, because VR is for everybody, and we want to make sure everybody can experience this stuff, we have to test it. So we're like, well, let's see what it looks like. And suddenly we're a woman in VR now, and we're seeing our body view from the point of view that women have. And we actually had to ask our friends, female developers, our wives, like, can you not see your navel? Because we can't see our navel when we're in the womb thing. They're like, no, we can't. That's what it really looks like. So we're like, OK, we're learning things now that we didn't know about, about the other sex and stuff. So that's really interesting.

[00:13:35.657] Kent Bye: But you know yeah, it also includes your feet a little bit more as well because I was like Oh, I actually want to see my feet because this whole demo is about foot presence But for women if you're actually correctly modeling the body it could be a dampened experience of actually having that foot presence

[00:13:51.422] Steve Bowler: a little bit, but it's still accurate to what that person experiences normally every day, right? So one of the things that we really want to do is, again, we keep saying VR is for everybody. We want to make sure everybody can at least see their ideal virtual self, right? In VR is we're going to try and get some creative player options in there, and not just skin color and hair color, which are needed. but body type modifications, right? Like, I want to be skinnier. Like, you're skinnier than I am, so we need a skinnier feature, right? I'm heavier than the model we have, so I need to make it a little bit heavier. And we want that across both genders. We want people to be able to be comfortable with who they see in VR, right? Either they want to see themselves as they really are, or see their ideal self in VR. And we think that's really cool. So we're also going to go with some other clothing options, too. We've got some fun experiments that like, perfectly emulate how you would look at clothes, right? And you go shopping, like, hold something up and look at it, and then just tap it to yourself, and now you're wearing it, right? Which has crazy applications outside of games, but it's fun to just go, hey, we have a game, we're gonna prove this stuff out now, and kind of see where it falls after that.

[00:14:50.595] Kent Bye: It's interesting as you were saying that I kind of realized that yeah the guy was a little bit more broader shoulder and heavier than I am in it. I think that actually kind of dampened my embodied presence a little bit because it's like not what I'm used to but at the same time I think because VR in that level of embodiment you are really wanting to have a range of identities so I think that that is important. So what do you want to experience in VR?

[00:15:16.114] Steve Bowler: You know, it's a really hard question. It's like the most open-ended possible question. It's such a, like, undiscovered country frontier thing, right, that it's so wide open that we just like doing things that are fun experiments and see how far we can push the medium. You know, we were one of the first companies out there to try doing the sprint technique, right, the linear interp movement, where there's no acceleration, like you saw it in the dash through the world. It's now become sort of an industry standard movement, and that's fun. Like, we're happy that we're able to contribute to it. And now we like innovating on this front. And we're like, hey, this is something we've wanted for a long time, since November of 2015 at least. And it's funny, as soon as people see it, it's kind of like that, oh yeah, this is what, like, Stevenson and Gibson were writing about in The Street, in The Matrix, and, you know, like, all of the, like, cyberpunk novels and stuff, right? That they want to see that avatar presence in VR. And we kind of haven't had it yet. Everybody's experimenting with it, and we're not perfect, We're definitely getting there, and it's kind of fun getting to show people that that is possible. And yeah, they were pretty visionary, and people really do identify with this stuff. So I think our strength is in doing fun mechanics in VR and doubling down on human body presence and human kinematics and animation, because that's our background. So we'll probably keep experimenting in this for a while. And we've got some other fun stuff we want to play with, but we can't show that hand just yet. But we've got some plans for the future that should be a lot of fun.

[00:16:45.318] Kent Bye: Awesome. And finally, what do you see as the ultimate potential of virtual reality and what it might be able to enable?

[00:16:54.621] Steve Bowler: From the movies and fiction and stuff that we've read, I think the ultimate potential is that idea of the virtual street, that you're going shopping in VR, maybe you want to go working in VR. It's all up to the individual user, right? I always like to say that as good as VR is, still nothing quite replaces a human face-to-face connection, right? Whether it's us talking or whether it's being on a date. And I think even Neil Stevenson in Snow Crash wrote that VR really took off when they could perfectly emulate a human face one-to-one to the point where business people would do across the table meeting in VR and have that same level of connection, like decipher if they might be lying or not, right? Like all of these weird like offline processes that human brain does during a conversation that you don't even realize it's doing and you actually get it from VR, I think we can get there. It's no longer science fiction, right? This is like day one. And maybe that's thousands of days down the road from now, but we'll get there. And I think we'll see people shopping in VR. I think we'll see people telecommuting to work in VR. I think we'll get there. I think it'll happen. Awesome. Well, thank you so much. Hey, thank you for coming and checking it out, Kent.

[00:18:06.133] Kent Bye: So that was Steve Bowler, he's the president of CloudGate Studios, and they just released Island 359, which has implemented this virtual self full-body VR. So, I have a number of different takeaways about this interview, is that first of all, having your full body tracked in VR makes a huge difference when it comes to this sense of embodied presence. Now, their solution that they have does track between three different trackers that you could get from HTC. And each of the trackers are now available. They were available for pre-order back in November of 2017. But for the three trackers, if you only have one, they put them on your torso. If you have two, then you put them on your two feet. And then if you have three, then you put them on your feet and then also on your torso. So because they come from a background of motion capture, then I think that they have some of the most sophisticated inverse kinematics solutions that I've seen. There's still, like he was saying, a lot of hard problems when it comes to the elbow and if you know how to kind of move your arm around, you can break it, but the more that you do that, you actually kind of break the presence. But their background in motion capture allows them to kind of think about these things and really push forward what's even possible with embodiment within virtual reality. So each of those trackers are like $99 each. So that's another like $300 that people would drop to get the full level. And in thinking in the future of where the two technological stacks are going with the Oculus Rift, they have a camera based solution. So I expect that some of this full embodiment types of functionalities and features may actually be a little bit easier to do within Oculus Rift. and just the camera-based solution is a little bit more flexible in order to track the full fidelity of your body. I haven't actually seen anybody do that yet, that's just a little bit more theoretical. The fact of the matter is that the HEC tracker pugs are actually out there and available, so you can start to either experience games like Island 359 that actually start to experiment and see what it's like to actually have access to that level of tracking, But you can also do some of your own experiences with those extra trackers, which I think is something that we haven't seen something equivalent from Oculus. In other words, being able to open up their tracking solution for third party developers to come in and have different accessories and whatnot. And yeah, and usually in the virtual reality experiences, because it is so difficult to track the elbows, most of the advice that's been out there up to this point is just to have floating hands. And so there's a level of disembodiment that I think people have been experiencing within pretty much any virtual reality experience that's out there. It's very rare that you get that full embodiment because tracking the elbows is so difficult that when you get it wrong and it sort of flips out in a way that it doesn't match your proprioception of your body awareness of where your body is and when that disconnect happens, that proprioceptive disconnect just breaks presence and it's actually better to just have your hands than to have that full body. But I think that what they're trying to push forward there at the Cloud Gate Studios is, you know, really experimenting with this level of embodiment and create an experience where you're kind of running around an island and getting to kick different dinosaurs. And I will say that it was a lot of fun to kind of sneak up on this giant dinosaur that was probably like, you know, almost six or seven times larger than me and to just kind of sneak up behind it and then kick it in its legs and eventually kill it from just from kicking it. So it is a fun mechanic. And I expect that, you know, especially when it comes to a couple of things. One is social VR. So having your full body tracked within things like VR chat, I think actually makes a huge difference. I know some people that were in some of the virtual worlds that I was in that actually had their hips and feet tracked. And it makes such a big difference to see how people move their full bodies. And so within social VR, that's going to be a huge thing. I think also within like the more exercise virtual reality experiences where something like soundboxing or audio shield or some of these other more active exercise VR types of experiences that having your full body will allow you to do actually more gameplay with exercise as well. And so I think that's another big one. You know, and one of the things that Steve says is that overall within virtual reality is that we're kind of moving into this metaverse vision of having a street where you'll be able to walk down. And he cites the Neil Stephenson snow crash saying that, you know, once we get to the point of being able to track all of the facial expressions, the point where you're able to discern whether or not someone is telling the truth or not. So being able to capture all those different micro expressions is a threshold that once we get to that point, then you're able to get all of the even the micro expressions and everything. I think that is going to make such a huge difference when it comes to social VR as well. And I think that once that's possible, then, you know, all bets are off in terms of what types of experiences people want to have, because the social interactions within virtual reality will be such a higher level. Even having eye tracking within virtual reality, some of the demos that I saw last year at GDC makes such a huge difference. And so it'll be curious to see over this next year of 2018 what other new types of tracking solutions that we'll see either from eye tracking or when it comes to body tracking, if there's something that Oculus is going to be able to use some more machine learning to be able to unlock some of the actual visual feedback that they're getting from those cameras. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, there's a couple of things you can do. First of all, just spread the word, tell your friends. Also, consider becoming a member to the Patreon. This is a listener supported podcast. And so I do rely upon your donations in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.