Oliver “Doc_Ok” Kreylos is one of the most prolific VR hackers and very popular participant within the Oculus subreddit. He talks about his quest to roll his own Linux SDK for the Oculus Rift since there hasn’t been an official one released yet. In that process he’s been building support for different input devices for his Virtual Reality User Interface (Vrui) Toolkit.
Since Oculus Connect, he’s had a number of epic follow-up posts on his blog (Intro, Part 1, Part 2, Part 3) about this quest towards a Linux SDK. He also discusses some of his biggest takeaways about the VR video rendering pipeline as well as some possible limitations of the graphics API in the Windows operating system that came out of this process as well.
He also talks about some of his findings of hacking the Microsoft Kinect v2 depth sensor towards the end of using it as a VR telepresence tool. He talks a lot more about this in my first Voices of VR interview with Oliver back in May.
Oliver then talks about some of his reviews of the VR hardware and input devices from SVVRCon, and some of his WILD SPECULATION™ around what type of sensor fusion combinations may make the most sense within a VR input device.
Finally, he talks about moving from the hard sciences to the soft sciences in his VR research by moving to the UC Davis Technocultural Studies department in the process of creating a VR research lab as a part of the UC Davis ModLab.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. I'm Oliver Kreilos. I am a researcher at the University of California, Davis. I've been doing VR for a long, long, long time. And now I'm sort of looking at it from a different angle again, which is now with the main stimification, with the Rift coming out and so forth, and all the new devices. And essentially what I'm working on right now, and since the last time we talked, is primarily to get all the new devices that I can get my hands on supported in my existing software. The bit of background the software has been developing over the last decade, or more than that actually, runs on all kinds of VR environments. From the very high end, from a cave, or projected screens, to a regular laptop, but then also to the Rift of course. And the wrinkle there is that me being on Linux right now, there is no official SDK for the Rift developer kit version 2. So I've been working with that and I have to say, I can't say I just got it working as of two o'clock this morning, minus positional tracking, that's the biggie of course, but everything else, orientation tracking, lens distortion correction, chromatic correction is working. So that's been keeping me busy for the last probably two weeks or so. Apart from that, I have been working quite a bit in telepresence, which is something that I've also been doing a lot over the years, and I've gotten back to that. I made that video end of May, I think, showing just a video capture with Kinect, and so I've been trying to make the follow-up video showing two people interact remotely, and I just haven't gotten around to it. It's just one of those crazy things where too many other things happening at the same time, and I'm multitasking too much. Well, that's basically been it. I'm also building a new VR lab at my university. It's going to be a large track space, head-mounted, for multiple people to be in the same space, to work together, play together. It's part of the humanities initiative that we have there, Digital Humanities. So it's going to be a very different application than the stuff I've been doing so far, which is more like hardcore science. And now we're going into social sciences, humanities, soft sciences, to do some theatre. and those kind of things. It's going to be very exciting, but we are literally building that lab right now. I mean, not today, obviously, but yesterday and on Monday again. That's been it, basically.
[00:02:11.618] Oliver Kreylos: So let's jump to the Kinect, since there's the Kinect V2 that's come out, and have you been able to get it working within a Linux environment, or is it kind of locked down to be only Windows 8.1?
[00:02:22.540] Kent Bye: I can get it to work, but it has been a real slog so far. I forgot to mention that. It's another thing I'm working on on the side. The thing is that the Kinect 2 is... Well, let me backtrack a bit. Initially, when it came out, I thought it would be locked down just to protect Microsoft's investment, to protect Microsoft SDK, and said the same thing about the Kinect 1. I was wrong then. I said the same thing about the Kinect 2. I was wrong again. They did not intentionally lock it down. But as it turns out, it is much, much harder to work with on a low level than the Kinect One. And that is just because they really, the interface that allows you to talk to it at the low level is much, much raw than it was on the Kinect One. On the Kinect One, if you were able to get the USB protocol figured out, the device would give you a stream of color frames and a stream of depth frames, where you have a pixel grid, 640 by 480, and a bunch of those pixels have a depth information in them. So, you know, so far, so good. It's easy to work with. The new Kinect has an amazing sensor. It is extremely advanced in terms of sensing technology, what it can do, because it can really measure a distance for every individual pixel very precisely. But the thing is, and I don't want to go into the details of how it works, but what the device sends over USB is the raw, raw, raw data collected by the sensor. So there's not a depth map in any sense of the word. You're getting a bunch of different images, and you have to extract the depth map from that by doing some pretty advanced video processing. And there's also a lot of secret ingredients in there, parameters that are not really known, and the algorithm is somewhat described in the pattern that Microsoft has, but of course what parameters you have to use, what scaling factors, all that is unknown. So there's a lot of things going on like that, which is why it's taken me a long time. I got two Kinects. beginning of June, and I have them working now that I can get the color stream out, no problem, that was easy, and I can get the raw depth images streamed just fine, and I'm yet now in the middle of the process of converting them into a usable depth map, which is going to happen at some point, but it's been sort of my side project because it was so slow going and other things going on, but I am optimistic that it can make it work, and then it will work really well. The image quality we are getting compared to Kinect One is just class is better. I mean, it's a totally new device, totally different technology, much, much better results. I'm very much looking forward to working with it. I wish there was some official support from Microsoft, but I'm just doing my thing. I mean, they did kind of support me by sending me two of them, so that was nice of them, but it's been hard.
[00:04:41.543] Oliver Kreylos: Yeah, it's interesting that when you talk about telepresence, it seems like a pretty key component of using that as a device to be able to bring these types of both 3D depth and the image from the camera as a texture map onto it to be able to put that into a virtual reality environment and to have these telepresence rooms. That is correct, yes.
[00:05:00.432] Kent Bye: I mean, we talked about this briefly last time and I said then that I think telepresence is for many, many people going to be the killer application for VR. It's the one thing that you buy a Rift or whatever to just do this one thing, which is still surprising me that not too many other people are really working on that. There's now two, maybe two and a half other projects I've seen that do similar things. And they, of course, having the benefit of being on Windows, can use the Kinect 2, and I'm very jealous that I can't yet. So that's... I think once I come back from here, I'm going to really put some more effort into that to get it going, because it's going to make... Once I have the low-level stuff going, it's going to drop into my software without any further changes. It's the same thing. and it's going to make everything just look so much better. And the biggest complaint so far has been the really fuzzy, low-resolution look of those 3D video avatars. The thing is, if you're really in the space, it doesn't matter that much, because you ignore that. You just see beyond it. But if you see a picture or a video of it, it looks really, really bad. And with the Kinect 2 or a similar device, it just looks so much better. It's so much more convincing. So I'm really, really excited about that and I hope I can find the time to get that going and put some serious software engineering on that to push it out the door and make a product that people can download and just play with at home and talk to their friends and then maybe, you know, go somewhere with it. It would be really cool to do that, of course. Yeah, I'm definitely hoping to be involved in that as we go forward.
[00:06:19.618] Oliver Kreylos: And one of the things that you wrote about recently in the process of doing this Linux investigation of kind of rolling your own SDK, in some sense, is the video rendering pipeline. And you did a big, long blog post, yet another wall of text, as you called it. So maybe you could describe to me what you were trying to communicate with that and some of your lessons that you learned in the process of kind of just digging around in the guts of trying to implement the Rift on Linux.
[00:06:43.542] Kent Bye: Yeah, I apologize in advance or in reverse rather for that article because I wrote that at I think two o'clock in the morning and it shows. It's probably my worst structured one that I've written so far. I mean people have been really nice about it and nobody has yet come to me and said you got it all wrong. This is all BS, this is all wrong. Nobody has said that so far. So either they're being polite or actually got some info out of it. So the idea was that once of course I got that thing and I got my DK2 very late. I don't recall right now how long I had it but probably about a month or so tops. And it's been sitting in the box for about three weeks of that because there was no software for it. And I finally took it out and started playing with it. And so the thing is, I've been, you know, following along on Reddit, on the Oculus subreddit, what people are doing, what people are talking about. And there was a lot of discussion and there were these tweets from John Carmack who were forwarded, talking about the things he's been doing with Gear VR and reading all of that. It kind of, it didn't form a picture in my mind. It's like I had all these different tidbits and it didn't really make sense. It's like, what is he talking about? What is going on here? This doesn't make any sense. And so I figured out, okay, I'm going to look into that. And then the other thing I was going to do is, I have been noticing recently a bunch of latency issues I was having with my software on some displays. And I didn't really have a way of measuring that. And then a friend of mine actually had an official genuine Oculus Relatency Tester, so I borrowed that from him. and just made a little driver for it and just plugged it into my software and started measuring things. And so I realized that my monitor at home has a built-in 60 milliseconds latency, just the thing, which is ridiculous. I got to replace that one, but it's almost 10 years old, so okay. And then, of course, I started measuring the latency with a fast display like the Rift DK2, which has a very, very fast OLED, and I was still getting very bad latency, and I didn't understand why. Then I started tinkering with that and looking, okay, how can I crank down on this latency? If I turn off frame synchronization, okay, then I can reduce it a bit, but I get tearing. If I use VSync, okay, then I get at least one frame at least. But if I use VSync and delay my rendering to the end of that, then I can cut down on it. But then, oh, I run into the danger of missing the VSync period and missing an entire frame. So all of these things that, you know, people have been talking about, I had never really looked at them in that much detail to admit it. And suddenly then it started making sense that the reason why we do the rendering pipeline the way we do is that normally we can't control the things we render. They get rendered depending on what your view parameters. So you have to do your rendering in the background and then when you're done rendering you push it to the graphics card which induces latency but it's the only way to do it. So now with the Rift and other HMDs that do lens distortion correction, it turns out the last step, the lens distortion correction, is not random at all. It's an extremely predictable thing. You know exactly how long it's going to take. You're drawing your pixels in a very prescribed order. You're drawing every single pixel only exactly once. You never overdraw. So there's suddenly no reason anymore for doing this background drawing. So now you can go back to what we used to do, well when I say we, what my forebears used to do in the early 80s, is to draw the things out to the video system, to the TV, the moment they are getting drawn by the TV without any framebuffer at all. They called that chasing the beam back then and they're still calling it that. And so I suddenly realized that I had not thought about it at all, and that you can do it. And so then when I started really experimenting, how can I set it up to crank down on the latency? And that is why, where I just did this experiment and realized what I can do, just to really get it down to a few milliseconds, and then suddenly all the things that Oculus had been talking about made sense. And all the problems they were running into with Windows, because Windows doesn't support some of these things, which I'm not an expert on, I don't know Windows very well. But I got a lot of input from people who know Windows extremely well, telling me, oh, you can't do this, you can't draw to the front buffer, you can't do a just-in-time buffer swap, you always get one frame of latency. And so that's why I figured, okay, I need to, this whole thing that I just sort of learned for myself, I need to write that down because it's, I guess it's important and people need to know it. I don't know if I'm explaining it very well, but I'm going to try. And so I wrote this article and, you know, it got a bit rambly there. But that was the idea, I was trying to get across why exactly is it that we are doing things the way we are doing them, why is it that this does induce the kind of latencies it does, and how can we work around those properties of the video pipeline, of the very low-level video hardware. to reduce that. And I hope, at least I got a couple of ideas how I can do it, so now I'm actually working in my code and my SDK to apply those methods to hopefully get the next version to have below one frame latency across the board. Maybe getting it down to just, the lowest you can really reasonably get it down to is about, on a 75 hertz display, you know, 8 milliseconds for the middle of the screen because it takes so long for the for the virtual beam to trace along your pixels, so you can't go below that. But if you can get that sweet spot, then you're doing as well as you can. And I found now all the little tools and utilities and hooks and knobs and so forth in Linux to make that happen, which would be extremely cool because that's something that apparently doesn't work on Windows, period. So that would be very nice to be able to show that. So again, I don't know if I did a good job explaining that apparently I didn't get it totally wrong at least but I wrote it mostly for myself just to dump down all these things that just sort of dawned on me about playing with this just so I would know it in the future and that's where that basically came from and a lot of people looked at that actually so I think maybe there's something to it.
[00:11:49.578] Oliver Kreylos: Yeah, I think the thing that I saw that it was starting to explain the guts of what is happening when you see tearing or judder, but there's also the limitation of Windows itself as an operating system. It sounds like it's sort of getting in the way in some ways of that, you know, perhaps eventually at some point Linux could be a more optimized way of using virtual reality is what I'm hearing in some sense.
[00:12:12.059] Kent Bye: Like I said, I don't really want to say anything because I'm not by any means a Windows expert. I'm only relaying second-hand what people who I don't want to name at Oculus and other places have told me. But it was surprising to me. I didn't expect that there would be these very weird decisions in the Windows graphics API that induce latency. because someone said, well, you know, you can't blame them for that. They wrote this in 1980 or something, so back then we didn't have VR, but then the sense, of course, is that they redid the whole thing with Arrow, and Arrow was designed to be low latency, and they still kind of didn't go all the way. I don't understand why they did it. They might have a good reason, but this is now we're in a nice position where Windows is apparently, and again, second-hand knowledge for me, not the ideal operating system to do VR. And now that Oculus have been working on the Gear VR, which of course is Android, so it's Linux-based, and they have been able to do all these things and actually are able on a... And I haven't tried the Gear VR yet, so this is again second-hand. I'm hopefully going to try it today. But apparently they were able to create a smoother VR experience on low-powered hardware than they can on a pretty high-end Windows PC. And that indicates that there's something in Windows that is not optimal when it comes to latency. Now, as it turns out, Nvidia has just, was it yesterday, announced a new graphics card line with all these VR-specific features, so they are waking up to the fact that there's a problem there. And SLI never worked for VR due to their alternate frame rendering, so they're ditching that now. So maybe there's now Microsoft getting off their butts and addressing latency in their system, which, of course, they can. I mean, if Linux can do it, they can do it. They just need to want to do it. And I was very surprised to find out that so far they apparently haven't wanted to. Which, given that Windows is such a multimedia system, almost by design, it was really surprising to me that they have these strange latencies in there.
[00:13:52.683] Oliver Kreylos: Since last time we talked, SVVR also happened, and you were there and were able to try out a whole bunch of different virtual reality input devices. And I think your blog post is probably the authoritative kind of deconstruction of where a lot of these inertial measurement units sort of fall down. Can you describe to me a little bit in terms of when you were testing some of these VR input devices, what types of things that you're doing to kind of put it to its edge and to its breaking point?
[00:14:21.453] Kent Bye: Oh, yeah, that's a little bit hard because there's sort of a black art to it, what I was just pushing them. The thing that really got me to write that was that I was very curious about specifically the PryOVR suit, which is, you have a bunch of IMUs all over the long bones in your body, lower arms, upper arms, legs, and so forth, and they are claiming that they can do drift-free positional tracking using those things. And the moment somebody says drift-free tracking with IMUs, it doesn't work. It is fundamentally impossible. says me right now. So I was trying to figure out, okay, what can they mean that they're not lying? Because I'm believing that they're not lying. So what are they doing and how does it work? So the answer, of course, is that they're not tracking the position of the entire body. They are tracking the position of the joints with respect to each other. And that you can do because you can do that based purely on the orientation of the IMUs. Now, I'm doing this gesture here, which you can't see, of course, and it's kind of important. The thing is that IMUs can measure orientation without drift because there's gravity which you can use as a reference frame. So if you know the exact orientation of your lower arm and you know the exact orientation of your upper arm because you have one IMU on each, then you can calculate the position of your wrist with respect to your shoulder without any drift because it's all based on orientation plus forward kinematics. And that is precisely what they're doing. They were very obfuscating. They never mentioned this. They were actually slagging some other people for doing IK, which is somewhat hypocritical, if you think about it. But that's what they were doing, and it works. So that means that it breaks down the moment you start walking, because then they have to track you in a larger space, and there's tremendous drift in there, obviously. And they're not lying about that. They're admitting it. It's nice of them. But the tracking of the hands and the feet with respect to each other and the hands to the upper body, that actually works as advertised, because it's all based on kinematics. So what I was looking at when I was at the showroom floor, I was probably creeping out the guy who was demonstrating, because I was just watching him for half an hour, just seeing what he was doing. The reason for that is he didn't let me try it myself. He was like, no, you can't try it, because it's calibrated for myself, which was actually not true, because it was totally not calibrated for him. But that's another story. And so just by observing him, what he was doing and how the system reacted to that, I was able to kind of get an idea of how it exactly works. And then when I described that in my blog post and then on Reddit, then someone from the company came and said, yeah, that's exactly how it works. So you nailed it, which made me happy, of course. But I was really happy about that because it showed me that they were not lying about what they can do. It's just they were communicating rather poorly what the thing actually does. And I think now that I exactly understand how it works and hopefully other people understand exactly how it works, it's actually better for them because now people know it's not a sham. Now they know it works. And the same with Control VR, by the way, which is using the same principles to do their tracking. And they haven't tried it, but they're apparently doing a very good job with it. And so one of the reasons why I went to the SVVI Expo was to be able to try all of these input devices that I normally don't get to try. Because I think that the frontier going forward in VR is going to be input. And I'm kind of almost betting money on hearing something from Oculus today or tomorrow about input. I have been harping for years, years, years about how VR software should handle input devices that you should not write to a device type but to middleware and so now this all is coming to a head with all these things coming out and so trying all these different devices and getting a real feel for how they work and what their shortcomings are and how I would use them and how I would write support software for them was for me a very big thing of going to the expo and again being here today as well. So the reason why I wrote about all that is again to write down the things that I had learned and hopefully tell other people what I think how they work and in retrospect I think I got almost everything right what I wrote. There were some minor little things I got wrong but nothing serious and I was happy about that. Seeing the stem, how well it works, seeing the prior VR suit, how well it does what it does and how not well it does the thing it's not supposed to do and I was really happy about the PlayStation Move because I have one at home and I haven't gotten around to it, but seeing it so well in action was just a real motivation boost for me to get going on it, still haven't managed to. And what else did I see? The leap motion, which I had seen a long, long time ago when I visited the company, which I'm still somewhat on the fence on about. There are some fundamental issues with it. But the way I approach these things, I guess, that you try to form a hypothesis of how is it that it works, how can it possibly do what it's doing, and then based on that hypothesis do an experiment. That's where the scientists come in and say, okay, how can I break the hypothesis? If it's based on orientation tracking, then if it's not like the prior VR suit, then if it's not well calibrated and you put your hands together, the system will not actually recognize your hands being together because your lower arms are maybe shorter than the system thinks they are. And so there would be a gap. And I saw that very clearly just observing the guy texting on his phone. I wrote a sentence about that. And so I knew it was not calibrated for him. It was actually, they should have done a better job at that. And so that's basically how you try to reverse engineer these things. And I mean, that's how I go about understanding how they work, but then also about writing drivers for them. I must admit I'm not very good at that. There are other people who are much, much better at that kind of stuff. I usually piggyback off them, but I'm getting better just because I have quite a bit of practice with it now, just having seen so many different devices. It is very exciting. I mean, I would say that what really has revolutionized the whole input thing is just suddenly the cheapness and quality of IMUs. I'm coming to appreciate more and more that there's nothing they can't do. If you have any problem, just throw an IMU in there and it's probably going to help in some way. That's sort of the bottom line to take away from that. And so seeing what people do with them in very clever ways and how they combine them with other sensors, like STEM even has IMUs in there for noise correction and maybe even field correction. I'm not sure about that yet. It's really cool to see.
[00:19:42.978] Oliver Kreylos: You've also had the Razor Hydros for a while and you were also trying out the latest STEM iterations at SCVR and talking about points of what they were breaking down as well with the magnetic fields interfering and whatnot. Have you had a chance to try some of the latest iterations and what are some of your thoughts on the STEM and where that's at?
[00:20:01.240] Kent Bye: Yeah, I haven't tried anything since SVVR. I have always been in kind of loose contact with Sixth Sense. They have been talking to me, I've been talking to them, but for some reason we never set up any kind of collaboration. I have no inside knowledge about the system, I have no inside access to any of their hardware, so I only see what they show publicly. And the thing is electromagnetic tracking has been around for a long, long time. It was the go-to technology in the early 90s because there was nothing else to track a 3D position. And so we know what the shortcomings of the technology are ever since then. And the thing is just no matter how well you do in designing your device, no matter how correctly you wrap your coils and no matter how Awesomely, you engineer the thing, physics gets in the way. The moment there's any kind of magnetic, paramagnetic material in the environment, the entire electromagnetic field gets warped. And that is something that is completely external to the device. It cannot know it because space itself is warped basically. It's just like gravity bends light. magnetics bend electromagnetic fields. If you only have electromagnetics, there's no way to correct for it unless you do calibration via some external means, which I do in my lab when we use electromagnetic tracking in Sirius. I have, like I said, done that a lot during the late 90s, early 2000s when there was nothing else around, where I had this calibration rig where you would take your electromagnetic tracker to a known position in space and then just record what the system says and you build up this grid of distortion coefficients basically. And you can do that and it works, but it's not something that a home user could ever do. You need some special hardware, special devices for that. And even then, you just come in and you have a set of keys in your pocket and that just messes everything up again. So it's just fundamentally difficult. The benefits, of course, are it's easy to do, and you can do it at very high frame rates, at very low latencies. So what Sixth Sense have been trying to do, and again, this is based on purely external observation, is to combine the electromagnetics with other sensors. And they're again using IMUs, and it's not exactly clear what they're doing with them. The obvious, of course, is that IMUs reduce noise a lot, because they are very sensitive to instantaneous rotation and acceleration, which is a problem in electromagnetic tracking. So if you combine the two, you can get rid of the noise, but still get the position accuracy. There's also potentially a way, and I haven't really worked it out yet, how you can use the IMU to iteratively, dynamically map out the magnetic field as you're using the system. That is very esoteric. If they're actually thinking about that, I do not know. It is potentially possible, but I have no idea if it's feasible. It's an interesting idea. Again, I have no idea if they're doing that. So that's where I'm seeing them going, to try to pound as much as they can on the sensor fusion to get really the best quality of tracking out of their device. And I have to say that for an electromagnetic tracking device, the one that I tried at SVVR was really, really good. It had the typical problems. I had my keys in my pocket, and I brought them up to the antenna, and of course the thing went to hell. But there's nothing they can do about that. That's just the name of the game. And it is clearly much, much better than the Razer Hydra. And I don't have the one-to-one comparison, but I would say that at this point, the STEM system is probably better than the high-end electromagnetic system that have been around for a long, long, long time, the polymeses and the ascensions, that are extremely expensive and very, very finicky, and I would say that they have it at that level at that point. So what they're now facing is not so much that they can improve their electromagnetics at this point, they have to get around the limitations of it by doing sensor fusion, basically. I'm very curious to see what they're going to come up with in the end. It is definitely a promising device for the reasons I said. Latency, frame rates, and drift-free positional tracking. That's definitely something that you have to have.
[00:23:30.347] Oliver Kreylos: Yeah, you mentioned sensor fusion a number of times there, and just thinking about something that potentially is combining the best of everything from camera-based to something that's a 3D depth, like a leap motion and electromagnetics and the IMU, sort of fusing all of those together to get the best solution, it sounds like.
[00:23:47.295] Kent Bye: That is a good question. I mean, there are a couple of really winning combinations. And the combination of optical and inertial, that is a winning combination right there. We have known that for a long time. It's not exactly the principle of the PlayStation Move. It doesn't really fuse between the two because they're treated separately, position and orientation, but that's a detail. And that's, of course, what the DK2 is doing. Another really good combination is inertial and ultrasound, which is some of the high-end tracking systems that we are using in our systems, in the Cave and on the large display wall. which is basically a sonar plus inertial. It makes for an extremely robust system with very few line-of-sight issues, but also a very expensive system. Unless somebody commodifies it again, hopefully, but right now it's extremely expensive. So if you combine these two, you can see that the common denominator there is the IMU. Like I said, these things can do everything. So then electromagnetic, how to combine that with an IMU. Again I said it's very esoteric but might work. Combining electromagnetic with optical, now that might be another winner. I haven't seen anybody do that. Well actually I've experimented with it a bit in my lab because I happen to have an optical tracking system and of course I have the Hydra. So I made a little utility that allows you to map out your tracking space like I explained earlier to get the distortion coefficients. by backing up your electromagnetic with optical tracking and then during use you would not use the optical tracking. You would just use the magnetic and use the distortion correction and that worked very well. But it's something that is not easy to set up at home. The thing is that the moment you have two absolute positioning devices like electromagnetic and optical, you have to calibrate them with respect to each other and that is not a great user experience. Let's say you can do it. I'm happy to do it in my lab. I'll do it any day. but it's not something a Joe user would necessarily want to do, that sounds really condescending, but would know how to do, because it's really not a straightforward process. I'm quite impressed by the way how Oculus got around that issue with the DK2 tracking camera, how it self-calibrates. I did not expect them to do such a nice job of it, to be honest. Well, they did a really nice job. So how to combine those two, I'm not exactly sure. How to throw all of them into the same bin? That would be interesting. That's something that somebody has to try. I'm guessing, hypothesizing, that at that point you're going to see diminishing returns because you're going to see these things fighting each other. Like I said, having an instantaneous system like an IMU and a laggy system but precise system like optical, that's a logical combination. But having two that almost have already the same benefits and combining them that might only add to the complexity but might not give that much in performance benefits. This is wild speculation as they like to say around here. I think somebody has to try it and see how it works. But generally I think sensor fusion is really where it's at. So I'm kind of regretting that I never took an official class on all of this. I need to now learn all about it to figure out and to understand it all. So it's for the future for me.
[00:26:27.084] Oliver Kreylos: Yeah, and maybe just to kind of wrap things up, it sounds like you're moving from the hard sciences to the soft sciences in some ways of moving into going from data visualization and doing a lot of stuff with research with geospatial information and now moving into the humanities with this VR lab and presence. And maybe you could just expand on that a little bit more in terms of what you hope to see come out of that.
[00:26:51.967] Kent Bye: Right, it's not so much a migration as we have been doing that on and off over time. For example, many people don't know that, I was part of the production of a modern dance performance in 2007, where we used body tracking and 3D visualization, stereoscopic visualization in a theater to augment the dance performance. I thought it was awesome. You can check it out. It's called Collapse, Things Falling Down. So we have done it in the past. And this new collaboration is because at UC Davis we have this institute called Technocultural Studies. It's an interdisciplinary department between humanities and sciences, specifically engineering and computer science. and so they do all kinds of things from film production techniques to special effects to 3D modeling to modern performances and all that but always with a technical component to it and so this new lab is an offshoot of that and I don't want to go into too much detail because I'll just get it wrong and the PI is going to be angry at me so Colin forgive me is a joint project with the oh my god I think University of Waterloo in Canada That is about gaming sciences, gaming technologies, meaning from the impact on society point of view, not the writing a game point of view, even though they're doing some of that. And so that gave us the money to build this new VR lab, where we're going to look into how people interact with the VR space, more from a human perspective, and how people interact with VR games specifically. And we are also going to use it to see if we can digitally augment traditional performance. And I have also not talked about it very much, but we did a prototype. We did a virtual augmented reality mixed media theater performance in mid-May, I think. where we had a normal performance space, and then three VR sites, each one with an Oculus Rift, DK1 at the time, a Razer Hydra, and a couple of Kinects to capture the person in that space, and so these three spaces were separately, they were in the same room so the audience could see them, but they were in different corners of the room, but the actors in these spaces could, via our collaboration infrastructure, via the telepresence stuff, could act, literally act with each other. and then there were also actors walking around physically just going into and out of these spaces and there was a big projector so the audience could see what was going on in the shared virtual space and it was really awesome in the way how it worked and I think we are going to explore a lot more about that in this new VR space where we are going to have multiple people wearing let's say head-mounted displays in the same physical space and interacting with each other physically and remotely at the same time. One thing that blew my mind during the rehearsals for this performance was that we had two actors who were in different spaces and they were trying, can we remotely hug? And so they did a remote hug, just pretending. The interesting thing is, at the same time in the other space, there were two actors physically hugging because they were in the same space. And if you looked at that in the 3D video feed, you could not actually tell the difference. It looked exactly the same. And that was for me like, wow, this actually works. I mean, these people who are not experts, they have rehearsed a bit, but not nearly as much as I hoped. were able to do something that is very personal, close space, intimate thing and were able to do it over not a long distance but you know a long enough distance that they couldn't have done in reality and it just worked and you couldn't really tell apart. That was for me sort of this that put the idea on that yes this is really something that I should be doing, this is really something that's going to be a big deal which is why I've been moving more into the telepresence area and why we are building this new space.
[00:30:16.742] Oliver Kreylos: Great. And finally, is there anything else that's left unsaid that you'd like to say?
[00:30:20.504] Kent Bye: Oh, I think I've already chewed your ear off. Stay in touch. I mean, you're seeing what I'm working on. I see that you follow me on Twitter. So I guess that's probably the best way of staying in touch with the kind of things that I'm doing in my neck of the woods.
[00:30:32.749] Oliver Kreylos: Great. Well, thank you so much. Oh, you're welcome.
