Amir Rubin, CEO of Sixense, the distributors of the Razer Hydra controller and manufacturers of the wireless STEM controllers, which have been described as enabling a 3D mutli-touch interface and will could become the first “mouse of VR.”
Amir talks about how the intention of Sixense is to create presence in VR by bringing your body into VR in an intuitive fashion. He talks about the differences between a physical-based solution like the STEM controllers versus using gesture controls with a camera-based system.
He talks about the history and timing of the Razer Hydra, and how it was a tough sell to convince developers that this paradigm shift was needed for interacting in 3D spaces — especially before virtual reality came along. He talks about the challenges faced with shortages of the Hydra and how they’ve been supporting exciting VR projects.
He talks about the modular design of the STEM controllers, and how developers will be able to bring all sorts of different peripheral objects into VR by using the STEM pack component.
Amir then goes into what first inspired the creation of Sixense, and how he wanted to be able to provide VR training experiences where the technology didn’t get into the way, and that people could have as much immersion and presence as possible in order to tap into their deeper intuition. He believes that almost every experience in reality could be improved with virtual reality, and he wants to help bring that sense of immersion and presence to people through Sixense technologies.
Finally, he talks about the lessons learned from the failed VR revolution of the 90s, and how that has influenced the collaborative mindset that Sixense has as well as others within the VR community. He’d like to help make the VR revolution happen this time around, and help provide incredible educational and training experiences as well as allowing people to step out of reality and have boundless entertainment experiences where they can live out their wildest fantasies.
Reddit discussion here.
TOPICS
- 0:00 – Intro Sixense proving presence in virtual reality. Presence requires bringing your body into VR. Partnering with Razor Hydra. Working on the wireless STEM controllers.
- 1:36 – Timing of Razer Hydra and being out of stock. Not manufactured by Sixense. Providing Hydras to exciting VR development projects. Hydra designed to enable people working with 3D applications as “3D multi-touch” technology. It was nice to have, but hard to convince developers. Valve was supportive, but Hydra was hard technology to justify adopting before VR. Carmack’s contribution to resurgence of VR. Hydra became the mouse of VR. Creating a wireless version with STEM, and the stoppage of manufacturing of the Hydra
- 5:29 – Modular design of the STEM System controllers – Three components: STEM base. STEM controllers with lots of buttons and joystick controller. STEM pack that can be attached to other objects to get 6 DOF control with them. Trying to make it as simple for developers as possible.
- 8:04 – How do you see the difference between VR input differences between camera-based gestures and controls with physical buttons? Camera-based gestures can be great if line of sight and you have an intuitive use case. But simulating using objects in VR doesn’t work as well because it’s not natural. Sixense sensors and input devices in order to be able to use your intuition rather than learning a new sign language. Power of bringing hand and eye coordination into VR. Sony & Oculus using a combination camera and physical controllers. Sixense SDK supporting other 6 DOF controllers like Sony Move and Intel RealSense camera for perceptual computing.
- 11:28 – What was the inspiration for the creation of Sixense Entertainment? Amir was inspired by VR in the 90s. Saw how much gear that military soldiers had to wear to have their VR training exercises. General wanted to have soldiers have a VR experience without mentioning all of the technology that was getting in the way. Creating a system that’s 100% experience delivery where the transition between reality and virtual reality is seamless and fast. Focusing on the consumer VR version, and wants to create VR training experiences where you forget about the VR tech and easily forget reality and step into a fantasy world and experience all of the experiences you want to. Don’t worry about the constraints of technology because it breaks VR immersion.
- 15:01 – What is driving you now? Is it a business desire or is there something in VR that you want to experience? Reality isn’t as good as as it should be. VR can improve on nearly everything we have in reality. Step into entertainment experiences and transform into a character and live out your fantasies. Explore resort vacations with his family. Go anywhere together in with your family. Everything that you need to can come to you digitally.
- 17:11 – How have you seen the VR industry evolve since you’ve been involved with it? Learn from the failures of the 90s. The technology was not there yet, but the business execution was also lacking from too much competition and not enough collaboration. He’s seeing a lot more collaboration this time around from Oculus to the open collaborative mindset of Sixense. Can’t support professional applications. Focusing on the has market at the moment. Integrating with the Sony SDK. Sixense wants to collaborate with other hardware manufactures and distributors.
- 19:57 – What is the ultimate potential for virtual reality? Every use case of reality can be improved by virtual reality. Education and training implications are huge.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.974] Amir Rubin: My name is Amir Rubin. I'm the CEO of Sixth Sense Entertainment. Sixth Sense is focused on enabling presence in virtual reality. So when you talk about presence, we'll break the virtual reality experience to two components. You have the HMD, as an example, we'll use the Rift. The Oculus Rift is providing you a window into virtual reality. When you talk about presence being in virtual reality and experiencing the world of virtual reality from within, it requires you to bring your body into it. your hands, your body, and experiencing it with your intuitive gestures, intuitive motion. And therefore, Sixth Sense was established back in 2007 to bring presence into 3D worlds. We worked very hard on developing a wireless technology, started with wired because it was easier and cheaper. partnered with Razer and released the Hydra back in 2011. And since then, we were able to develop a viable solution for wireless for multiple applications in VR and outside of VR and announced the STEM as our first VR developer kit and early adopters enthusiast system. And that's basically where we are today.
[00:01:36.887] Kent Bye: And so yeah, with the Hydra, it seems like it was almost a device ahead of its time in some ways. The timing was before there was a viable HMD, and then once the Rift came out, then it seemed like there was a lot of people in virtual reality who were like, oh wow, I see the potential of this, and then they're kind of out of stock or so. Maybe you could talk about the timing and how that unfolded.
[00:01:55.230] Amir Rubin: So yeah, so first of all, we apologize for the fact that the Hydra is out of stock. We also have to explain the fact that not only it's out of stock, is we still have units that we were able to purchase. But to just clarify one very important point, Sixth Sense do not manufacture the Hydra. The Hydra is manufactured by Razor. And from the time Razor stopped supplying it, we were going out there and purchasing the Hydra and making sure that especially for developers, Hydra is going to be available. Now, on every one Hydra that is being sold, we provide developers Hydras for free. And basically, you know, big developers or exciting projects that are working on VR applications sometimes don't have money and sometimes, you know, you can't ask them for money because they're too big and too complex to get you a purchase order. The bottom line is we support them. So right now, we still have some Hydras. We're still making Hydras available. Now, take back in time and understand the Razer Hydra, it was really designed for a new and a different way to interact with what we refer to as virtual worlds. But again, it was not virtual reality. It was designed from the mindset of 2D applications has the mouse as a motion controller. It's a 2D motion controller. has multi-touch, you know, which was introduced by Apple. The multi-touch was at the point today's the best way to interact with 2D applications and was a good way also to a certain extent to interact with mobile 3D applications. But the Hydra and our motion tracking technology was really designed to enable people interacting with 3D applications. Many in the media and analysts are referring to our technology being the Sixth Sense motion tracking technology as a 3D multi-touch technology. And with that, we went out to the market. The problem that we faced is that it was at the point where it was a good-to-have input device. It was hard to persuade developers that they need to change the way that they go about designing applications. And it took some time. You know, Valve was very supportive, and others that supported it, but still, it was hard to justify because it was a nice and a good-to-have, and a variation on a mouse and keyboard and an Xbox Dual Analog Controller. But then came Palmer with his Oculus and suddenly John Carmack says, okay, we can't see the keyboard and mouse. We can't touch the hands to a controller. We need to make it a one-to-one experience. It needs to be an intuitive experience. It needs to be motion tracking for your body and starting with your hands. And of course, with your head, position tracking for both head and hands and arms. And that led to Hydra becoming the mouse of VR, because you are using 3D multi-touch to interact with the 3D world that you are in. And unfortunately, that transition happened before Oculus was taken over by Facebook, and therefore the Hydra went out of manufacturing the moment Sixth Sense announced that we are releasing a wireless version of our technology. We always believed in wireless as a better solution. It was just always too expensive. And now our engineers were smart enough to come up with some amazing solutions to many of the original problems. And it's going to come to the market later this year.
[00:05:29.185] Kent Bye: Yeah, and I know that seeing a lot of demos in virtual reality space that the Hydra, a lot of people getting that extra level of immersion and interactivity. And it does seem like the STEM sounds like the leading solution in terms of a physical button based of positional tracking. And I'm curious, it sounds like you're doing a modular component so that you have your STEM controllers, but you're able to also take out the components and kind of attach it to other things. Maybe you could talk about that modular nature of the STEM controller.
[00:05:57.935] Amir Rubin: So the stem system is made of three components, three main components. You have the base, which includes the reference point that basically takes all the data that comes from the tracking devices and sends it to the computer. The controllers are basically the second component where you are able to hold controllers in your hand and because it's designed as a developer kit, we added as many buttons, a joystick, a trigger, a bumper that we believe is the maximum that will be required by developers that develop all sorts of applications. MMOs and RTSs all the way to a no button application that will need usually in VR you'll need very few buttons or even a joystick. You need a trigger. The bottom line is that at this point the controllers are designed to support many many applications all the way to the point that we don't even know what people might do. So we had to enable it with the current controller design. Now, we also added a component which we call it the Stamp Pack. The Stamp Pack is you take all the tracking, you put it into a small box, You enable that box to be attached to anything. So that box can be attached to the back of your iPhone. It can be attached to the back of your tablet. It can be attached to the back of anything you want, including your shin, your leg, your arm. And at that point, it gives you full six degrees of freedom motion tracking at the highest level of resolution that the stem can do it. And the important thing is that at that level, you can turn any object to become part of your experience. You take it, you attach it to a sword, it's there. You attach it to a toy gun, it's there. You attach it to your fist and you attach it to gloves, you have boxing gloves in the game. It's mainly designed to be as simple as possible. And more importantly, it's a one package that you don't need to add any power to or electronics to. Originally, we thought about doing it differently and split it. But then we tried it with some developers and it became very complicated for them to integrate. So we decided to give them one complete package.
[00:08:05.659] Kent Bye: How do you think about the difference in terms of a camera-based motion tracking of just kind of natural gestures versus something that has more of a physical component and buttons? Like how, in your mind, do you distinguish the two?
[00:08:19.064] Amir Rubin: So I don't believe in, first of all, buttons. Some places are necessary, some places are not. But that's not the differentiation between camera and physical objects, meaning a camera is a great solution. If line of sight is not a problem for you in the application, A camera is a great solution. As long as you're using the camera with an intuitive setting, an intuitive use case, meaning you are the camera seeing you, seeing your face, seeing your face expressions, you can communicate with the camera what the camera was designed to. But the moment you're trying to simulate shooting a gun and you're posing with your finger as you're holding a gun and you're doing as a gesture of pointing or directional, All that doesn't work because it's not natural. It's definitely not going to work in VR, but it doesn't even work outside of VR. Teaching you a sign language, all sorts of gestures, is not the way to go. The way to go is use sensors and input devices for the purpose of enabling people to use their intuition. We call the company Sixth Sense with exactly that objective. Deliver you the level of presence in the virtual world So you can use your intuition that you developed all those years of your life. All that experience that resulted with you having that intuition, that sixth sense, is what is needed to have a full, complete experience. And it starts with the most basic one. The eye and hand coordination. The fact that you take your hand, you bring it in front of your face, and it's there. VR or not VR. If you take your hand, you bring it in front of your face, but it's here, or suddenly it's flickering because there's an interference between this hand and that hand. and you have line-of-sight issues or you have sun issues, all those issues are problematic. But the way Sony does it is not an issue, less of an issue with line-of-sight, but overall it's a great solution. They have a camera, they have a controller that's being tracked, and they have gyro accelerometers and a magnetometer to correct each other, so they got this fusion between the two. The way Oculus did head tracking, have LEDs on the Rift, on DK2, having a camera detecting it, all that is great. I expect Leap Motion to, soon enough, come on also with introduction of a whole list of peripherals. In their case, they have a great technology, but in their case, if you just have a steering wheel being detected, which they already have all sorts of solutions, but push it ahead, I think their technology will also be great. RealSense of Perceptual Computing. We support them too. Our Sixth Sense VR SDK supports all six degrees of freedom input devices. We of course support the Hydra and the Stem, but we support RealSense Perceptual Computing. We will support soon enough the Sony, both head tracking and the Move. We always support the Move. And we're developing also, we're developing on the interperceptual computing on real sense, we're developing on the move, we released a title on the PlayStation 3, on the PS3 move. My point is, I believe all those technologies needs to work together to deliver the one and only objective, the one and only goal. Give me real presence in virtual reality.
[00:11:29.739] Kent Bye: I've heard you mention that you had experienced some level of virtual reality in a military context or, you know, what were the things that really inspired the creation of Sixth Sense Entertainment?
[00:11:42.224] Amir Rubin: I was inspired about VR back in the 90s. You know, when I licensed, exclusive license to Sega, at the time I had motion tracking technology based on IR, infrared. And then I licensed to Roland Electronics Dimension Beam technology for keyboards. I continued into the professional world, and I took our technology then to the only places that I did not license, which is military and medical, and medical I didn't like. I didn't do much, but sports. I did the Callaway simulator, golf simulator. I did a baseball simulator with Louis W. Slager. And the bottom line is, what inspired me was way back, one of the first time that I went to a demo and then put on myself and watch soldiers being trained with virtual reality. And when I saw the amount of hardware, the amount of gear they have to carry on their backs, on their heads, to be able to be trained, the gun armed with all sorts of sensors, but still all wired. And then I tried it and I said to myself, if I can make that for everyone, it will by far be the best way to experience content, the best way to really get that total recall type fantasy world that you can go and offer people. And then what inspired me on what direction the development of the product needs to be was a discussion that I was having with a general as a customer. The question that I asked him, what's going to make you happy? What's my success criteria? What if I come to you, did what we made met your expectations? as he referred to a success criteria, he says, if my soldiers can come back to me and discuss the experience they had, without mentioning the technology, you met the success criteria. It is all about developing a system that is one, is close to 100% experience delivery system, and that transition between reality and virtual reality, specifically with an objective of being trained for a specific purpose, as quickly as they are in training, as better is the system developed. For virtual reality entertainment that we are working on, consumer level, is even more important. Because you're not telling them just move away from reality. You're telling them into reality training of becoming a better soldier. You're telling them move away from reality to the level of forgetting about reality. You need to put it behind you, put it outside of the world. You just stepped into a fantasy world. You are one of the seals that conquered and got Bin Laden. You can live through that experience. You can do all those amazing, you know, you can drive a Daytona 500 and win or compete. All those experiences that people want to experience, they deserve to have it without reminded all the time, wait, wait, wait, I cannot go too far. No, no, I cannot cover one hand with another hand because it's not going to be good. You want the developers not to be limited with the constraints of technology because then it is not going to be reality. It will not be immersive. It will not be virtual reality that we believe will sustain as the iPhone, as the mobile phones and as the tablets are.
[00:15:01.957] Kent Bye: It sounds like you had a small taste of being in those virtual realities and having that sense of presence, but that the technology was not invisible enough. And so I'm curious if now you're driven just from a pure business perspective of like, oh, there's a market here and a demand that needs to be met, or if there's something within virtual reality that you are really trying to also experience and have that sense of presence.
[00:15:25.173] Amir Rubin: So I've always been driven by a very simple, the motive is, reality is not as good as it should be. We deserve to enjoy it better. Anything you do in reality, in my opinion, based on my experience, can be improved in virtual reality. We have a limited time on this planet, in this world, and we deserve to get the best out of it. So I believe everything you want to do, you want to be entertained, instead of going to a Spider-Man movie and expecting to be watching Spider-Man all over the place, doing this, doing that, saving people, shooting web, I want to be Spider-Man. I want to be Spider-Man. I want to be Iron Man. I deserve to be able to have that few minutes of fantasy of becoming that. And virtual reality can give it to you. Now, I want to be able to meet my family even though I'm away and go with them to explore resort locations together. I want to meet them in a virtual reality, be in Maui together and decide do we want to go this summer to this resort or do we want to go to Cancun or do we want to go here. We should be able to go anywhere together. I should be able to go to Amazon and shop for Christmas with my daughter and my son and together we'll explore her dolls and his video games and test it and play it and decide all together. I'm on a business trip, he's in college and she's at home. No reason for all those not to be made available. Not to mention I'm in a digital world, therefore I can get all the data coming in and not to be disturbed by noise. Everything that is relevant comes available to me. Everything is close, everything is there and I can experience it exactly the way I like things and be offered ways that I might like it and make choices.
[00:17:12.544] Kent Bye: Because you were involved in virtual reality for so long, and then within the last couple of years, it's been this real renaissance and explosion in consumer virtual reality. And so I'm curious in terms from your perspective, how you've seen this industry evolve since you've been involved with it.
[00:17:28.428] Amir Rubin: So what I've seen back in the 90s is a lot of mistakes done on the business execution of VR. Now technology was not there, the displays were not there, the processing abilities were not there. So that's the real reason why I didn't make it then. But also what I saw is that there was not enough collaboration between the big companies. And we are not a big company, we're definitely a small company, definitely compared to Facebook and all those who are a tiny company. But I believe that one of the things that I see today, and a good example is the conference at the Silicon Valley Virtual Reality Conference that we're all in, is the fact that you can see that all the companies are working together. Even Oculus is working with everyone. Even Oculus is working with everyone, opening it to everyone, working with academia, working with smaller and bigger developers, and educating. Everyone is working with everyone. And if you take it at Sixth Sense, we are not only making our controllers, but we're also making our controllers meaning our reference design of our hardware available to anybody that has a good distribution, is a good channel. We will work with anybody that make peripherals. They want to make it bigger, smaller. We support every, if you want to make gloves, come to us, we'll give you the tracking. You want to make, as long as you're focused on enabling the mass consumer market. We have no interest. At this moment, we cannot be focused and cannot support professional applications. We care about enabling every individual to have the experience that they deserve, and we want to go from the mass, from the millions. There's enough companies to support the professionals. So going back, we feel we are in a great place to enable mass adoption. You look at our, again, VR SDK. VR SDK supports all platforms. We released it on the Mac, we released it on We will announce soon Android. We have released it obviously on Linux and Windows, but also we're going to go for the PS4 with our SDK. So developers that develop an application on the Oculus, on the PC or on the Mac, will also be able to easily port it to Morpheus and Sony PlayStation 4. There's no reason not to, and whoever else is making any peripheral that has motion tracking, position motion tracking, we will support it. And that's what's required. At this moment, for the VR resurgent to be successful and not to have the same unfortunate death of the 90s, you need more collaboration between the different players.
[00:19:57.890] Kent Bye: And finally, what do you see as the ultimate potential for virtual reality?
[00:20:03.235] Amir Rubin: I believe virtual reality will enable you to experience almost every, and I'm saying almost just because I'm sure I didn't think about all, almost every use case in your reality that you want to do, you could do it better in virtual reality. starting with the most basic. You can be better entertained in a game because there's nothing to disturb you. You're fully immersed on the one hand. On the other hand, you're fully focused. You cannot be... Nobody passes across your eyes. Nobody can step in front of your face when you're playing something in front of a 15-inch display. The bottom line is because of the focus and because of the immersive, you can do anything and enjoy it. So if you're doing it for entertainment, you'll be more better entertained. You do it for education, you can definitely be better educated. You can be... In Rome, you can be in Greece, you can be in the different places around the world, you can experience from lab chemistry experiences to others. We work with a company that is doing all sorts of training, you know, for people at home, for different type of very necessary activities that will provide them better access to jobs. I believe it's much better for them to be able to do with the VR, where they can feel fully engaged and they can get better training. Now, go to anything, go to shopping. I believe VR will improve every part of our life.
[00:21:28.195] Kent Bye: Great. Well, thank you so much.
[00:21:30.336] Amir Rubin: It's a pleasure. Thank you very much for taking the time.
