Tobias Baumann is the director of game development for Perception Neuron, which is an IMU-based motion capture system. It might eventually be used as a VR input device, but Tobias acknowledges that the current price for the full system is a bit steep to be used for anything than a motion capture system at first.
The 32-sensor based full-body suit with finger-tracking runs at $1,499, and the 18-sensor full-body without fingers has an academic price of $799. The single hand version is more affordable for $100, and a two-handed option available for $200.
The Perception Neuron kickstarter more doubled their funding goal of $250k, and are getting ready to ship out their rewards within the next couple of weeks.
In the interview Tobias talks about:
- The mechanics of how their wireless IMU-tracking system works and how it’s mapped to an inverse kinematic skeletal model
- A bit of the history of Perception Neuron & how he first got involved
- Some of the preliminary prototypes and design issues in grabbing virtual objects
- Space requirements for roaming freely and cautions about Gear VR locomotion issues
- The production pipeline for using Perception Neuron for motion capture and the Unity plug-in & Perception Access data capture software
- Rigging considerations for getting Perception Neuron to work with motion retargeting
- Hopes that it might eventually be cheaper to be used a viable VR input device
Finally, Tobias talks a bit about how he first got into VR and how he got involved with the Perception Neuron project.
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.936] Tobias Baumann: My name is Tobias Baumann. I'm director of game development for Noytem. I'm working on the Perception Neuron motion capture system. And I'm also in charge of the SDK and the whole game demos. And all the game demos that we're showing here at GDC are made by me.
[00:00:27.226] Kent Bye: Awesome. So is this primarily a motion capture system, then? Or would you imagine this being used as an input controller for virtual reality?
[00:00:34.662] Tobias Baumann: For me, because I'm a game developer, it's more an input controller. But I think with the price right now, it needs to be cheaper for it to be a viable option. So a lot of people will buy it, and it's attractive for developers to use.
[00:00:47.165] Kent Bye: I think it's both. And so maybe you could tell me a bit more about the actual technology of the Perception Neuron. Describe to me what this system actually looks like.
[00:00:56.882] Tobias Baumann: Okay, so the system is, each sensor has a gyroscope, an accelerometer, a magnetometer inside. And each of the sensors measures its own orientation and its own movement in space. And then all the data gets sent to the computer over a hub. And you can use it wirelessly over Wi-Fi or USB. Or you can also record it on a SD card on the hub. And then our software takes all the data and combines it into a skeleton based setup. That means it's mostly inverse kinematic based. So we measure like this bone belongs to this bone and then we calculate the stuff in between all the bones. And then you have a pretty solid setup.
[00:01:37.630] Kent Bye: And so how many sensors are we talking about then that are part of this entire system?
[00:01:41.544] Tobias Baumann: The standard setup is 30 sensors, so that's with fingers, and you can do like 18 for just the whole body, and no fingers.
[00:01:51.909] Kent Bye: I see, and so you have two different levels, one with 18 and one in 30 then?
[00:01:55.311] Tobias Baumann: Yeah, well you can also, what we show here is like the one setup is only eight sensors, and that's one arm with fingers, or with the other game it's like only three sensors, so it's only one arm without fingers, so it's very adaptable.
[00:02:08.446] Kent Bye: So tell me a bit about the history of this project and how it came about and where you guys are at now.
[00:02:13.981] Tobias Baumann: So I've been only with the company for six months now. When I went there, it was still a secret project and they were like working on it for a year or something, using it for a glove and finger tracking. So I went there and helped them do like game prototypes, you know, doing like, what can we do? What kind of action can you do in the game? And I did a lot of experiments and they liked what I did and they liked how I saw a lot of possibilities. Yeah, that's how I got started.
[00:02:41.400] Kent Bye: So what type of rapid prototypes were you doing to prove it as a proof of concept then?
[00:02:46.378] Tobias Baumann: So I did a lot of work on like how to do finger interaction, right? So like, when you decide to grab an object, when you decide to release an object, like criterias that work and that work naturally so people can just use it. Because the problem is you don't have any feedback, you don't have any constraint. So you can grab through anything, which is unusual, you know? So when you grab a glass, you know from experience, you know how to hold it and how to grab it and you know how to When you lose friction, you know it will fall, so you grab closer. All those kind of things you don't have. So it's all about trying out what works best and what doesn't.
[00:03:26.305] Kent Bye: And so what were some of the things that you tried that did not work at all?
[00:03:30.050] Tobias Baumann: We did something with some location-based thing where we didn't have enough space to do it. So that didn't work. You need like a lot of space if you want to walk around in a room. You need like a big room with nothing in it. We did in the office, we did some tests with GearVR. Like hooking you up with GearVR and then you can, you don't have any tether anymore. And you can walk around very freely. That works pretty well, but the thing is you don't see anything. You walk very easily, like in a table, in a chair, fall over, or somewhere else. So yeah, that's a danger that you need to consider.
[00:04:02.564] Kent Bye: And so, was there a Kickstarter that was kickstarted?
[00:04:05.370] Tobias Baumann: So yeah, there was a Kickstarter, and we doubled our funding goal. And it's like the second most successful Kickstarter from China.
[00:04:14.732] Kent Bye: So what is the timeline for when we can expect to see some of these first kits delivered to the Kickstarter backers, but then also when it might be available to the general public?
[00:04:25.508] Tobias Baumann: So the thing we have here is that we received 10 days ago. So it's like the first batch from the factory. And we're going to start mass production, I think, in one or two weeks, if I remember correctly. And then we're going to ship to the first backers. And after we've fulfilled all backer orders, we're going to ship to pre-orders. And the website for pre-ordering should be up, I think, in a week or something. Yeah, there's some issue with payment, but it's basically ready.
[00:04:52.583] Kent Bye: Yeah, and so walk me through a little bit of the pipeline that you would imagine people using this and they get the perception you're on the mail and then you know what do they hook it up to on the software side and then to be able to start actually using it.
[00:05:05.715] Tobias Baumann: I think for the price right now most people will do it for animation. So they will like set it up and set up the system on their body and do like a walk cycle or do like finger animation or for whatever you want to use in your game. Record it and export it to your 3D Max or Maya or Motion Builder, whatever software you want to use. And then do some data cleaning and use it in your game. And the other part is virtual reality, which I believe will a lot of people use. They will use our SDK and use it in real time to do real time interactions.
[00:05:38.753] Kent Bye: I hope. I see. And so is there like a Unity plug-in that's going to be delivered with this as well?
[00:05:43.855] Tobias Baumann: Yeah, yeah. There's a Unity plug-in which already is in beta status. You can download on the Kickstarter website. And we will do Unreal and probably some other engines too.
[00:05:53.558] Kent Bye: What is the process that you have to go through to be able to correlate this incoming data that's coming from these 18 to 30 sensors? And how do you connect that to a rigged 3D model? What's that process there?
[00:06:05.813] Tobias Baumann: So the SDK gives you, like, what you receive from the software, from the tool that's... I mean, we have a software, it's called Perception Access, that runs in the background. So you start the Perception Access, you connect with the sensors, and then you do a calibration. And after that, you're good to go. It will start streaming. And then you load your game or your game engine, and it will use a BVH data reader, so binary data stream. And you get all the rotation values, all the position values as a raw data list. It's just about applying it to the right bone. Or you can do also some other stuff with it. So you don't want to use that data or ignore that data or modify it.
[00:06:47.255] Kent Bye: So I would imagine that if somebody's rigged up a 3D model, they may not have gone through and wired all 30 points for fingers if they didn't have something before the perception neuron to be able to actually do that. And so what do you recommend in terms of are there different models out there that are already pre-rigged that you would recommend? Or is it something, a process where they just have to make their model a little bit more nuanced?
[00:07:10.303] Tobias Baumann: We did some tests and we need to do some more work on that, that it will work with stock models or models you can download. For some models, the thing is, it's the way they're rigged. If your model is not rigged with the zeroed out rotations, it will not work. So your model has to be in a T-pose, with no rotations applied, and then it will work. But if your model does not have that, you have to re-rig it. And that takes a lot of time, depending on your model, so it's not really... And we're trying to solve this, but it's a very hard problem and it's called motion retargeting. Yeah, so I cannot tell you like when we will have it ready, but we're working on it.
[00:07:48.416] Kent Bye: Is there a way to just sort of capture some raw data for the perception neuron and then take that sort of skeleton and then easily transpose it onto an existing model?
[00:07:58.393] Tobias Baumann: Yeah, when you do a recording, you can export it as a BVH or RAW or FBX file. And then if you use MotionBuilder or any other tool, you can load it in, and then it's an animation already. And then you can do modifications to it.
[00:08:12.104] Kent Bye: And so where do you hope to see this go in the future as you guys move forward?
[00:08:16.187] Tobias Baumann: I hope it will be in the future with a lower-priced version. It will be a viable input solution for VR, a natural solution.
[00:08:24.693] Kent Bye: What's the price point now for getting the system?
[00:08:27.641] Tobias Baumann: The 30th Sensor Neuron package is $1,499, and the 18th Sensor is $800. Great.
[00:08:36.346] Kent Bye: And what do you see as the ultimate potential for virtual reality and what it might be able to enable for people?
[00:08:44.650] Tobias Baumann: I don't know. I think that's yet to be explored. I don't know. I think it's hidden somewhere. Somebody will find it out. And then everybody will say, oh, yeah, why didn't we think of it?
[00:08:54.385] Kent Bye: Great. And maybe you could just to just kind of wrap it up here. Tell me a little bit about like how you got into virtual reality and maybe some of your first experiences.
[00:09:02.907] Tobias Baumann: maybe two years ago, something like that. No, one year ago. I had some ideas combining, you know, a motion capture system with Oculus and then having like a perfect body, a replication in VR. And I tried to do some of my own stuff with using Xsense technology and doing some tests with it. And it was pretty interesting experience. And I figured, yeah, this could be something cool. and there are like three companies in the world that do IMU based motion capture systems and one is based in China and I wrote them an email and asked hey guys I saw you're doing something in Unity and you know have some similar stuff and then I they said yeah we do why don't you would you like to come over and you know work on some secret project and I was like yeah why not let's try it out and so I got into this whole business and it was pretty really really interesting Great. Well, thank you so much. You're welcome. Thank you.
