I talked with Cix Liv, the CEO of Liv, about their mixed reality hardware and software solutions. Mixed reality shows what it feels like to be in VR better than any other depiction, and Cix is really interested to see how mixed reality is going to drive innovation in VR eSports. We also talk about the Beat Saber phenomena, and how so many of the viral videos were born out of their liv.chat Discord community.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So of all the different trends that I'm seeing in VR right now, I think one of the biggest one and interesting one is embodiment. So with Beat Saber coming out on May 1st it was all about putting your body into the experience and using your body as a gameplay mechanic and One of the key successes to Beat Saber has been the mixed reality system that is integrated into it called live and live allows you to set up a green screen in your home and to be able to do what used to be only possible in a mixed reality studio with a lot of really expensive equipment you can Start to do that now in your room with virtual reality and start to live stream your own mixed reality experiences So people who are doing that within Beat Saber are the ones who are creating these videos that are going viral so you have the original video with Swan and other people that I'll actually be having an interview with Swan to sort of unpack the viral dynamic of what happened with the videos that they were creating at live and But I had a chance to talk to the CEO and co-founder of Liv named SixLiv. So Six talks about some of the trends that he's seeing within virtual reality, but also just trying to tackle this very intractable problem that comes to being able to set up a plug-and-play mixed reality system. They have some systems that they're working on in terms of the cube and they're also working on specific hardware but they're also doing a lot of software solutions for people who want to just do a lot of the manual tweaking in order to get their mixed reality system up and running. So we'll be talking about mixed reality, where it's at, where it's going, on today's episode of the Voices of VR podcast. So this interview with Six happened on Friday, May 4th, 2018 at VRLA in Los Angeles, California. So with that, let's go ahead and dive right in.
[00:02:04.885] Cix Liv: So my name is Six, I'm the CEO of Liv. Our catchphrase is, your window into VR. And we believe it's a very unsolved problem to be able to spectate within VR. And what we mean by that is, as people are experiencing VR, the average person, and even people in the VR space, have a hard time understanding the context in what they're doing. So we want to solve the inherent problem, which is spectating for people outside the headset.
[00:02:32.078] Kent Bye: Great, yeah. So you have a mixed reality solution. And I just got it set up for the Portland Community Media. Matt Henderson helped me set up with the whole studio, a green screen. Maybe you could just talk a bit about what are the basic requirements for what people need to have, what kind of gear they need to have in order to get the live system up and running with mixed reality shoots.
[00:02:54.937] Cix Liv: Right, so on the absolute bare minimum, some of our communities do something called partial mixed reality, which is they render things only in the foreground. This doesn't necessarily require a green screen, but there's limitations to that, which is essentially anything in the foreground Like, for example, if they're playing in FPS, they'll be able to see their gun and, like, you know, whatever is in their immediate foreground. That is what our community has coined partial mixed reality. And for that, you just need a camera. On the hardware stack side, you would need a VR-ready PC, but a little bit beyond that, because we're also rendering an additional viewpoint. So, generally speaking, we recommend at least a 1070. and like an i7 or Ryzen equivalent on the CPU side. And then to do proper mixed reality, you would require a green screen and a higher-end webcam, or we use a GoPro internally, which is a really good midway between a webcam and a DSLR.
[00:03:49.683] Kent Bye: And does it have like an HDMI out, or how do you get the GoPro into the computer then?
[00:03:55.128] Cix Liv: So the GoPro Black 5 and 6 is what we use. It has a micro HDMI that pushes out to the computer. And it's good not only because it has a decent level of keen and color quality, but also it has a high FOV. And you can correct the GoPro so it's in its linear mode, it doesn't have a fisheye, which is a problem with a lot of the high field of view cameras.
[00:04:18.509] Kent Bye: Yeah, there was a big release of Beat Saber that came out this past week and I had early access to it for a week and I played it about an hour a day for up until the point where I could actually like finish some of the expert levels and I wanted to create a video with the podcast interview that I did with the two creators and I didn't want to just have the first-person point of view because I wanted to have people really see me, see my embodiment and see what it looks like. to be immersed into this environment. And I just felt like the first person point of view didn't really capture what my body was doing and how it felt like. Even after I watched the mixed reality footage, I was like, well, that still doesn't capture what it feels like to play the game, but it's a lot closer than doing a first person point of view.
[00:05:02.600] Cix Liv: Absolutely. I think one of the biggest things we're trying to say is that there's no replacement for the nuances of how we move as humans. Where we're at right now with content creation is we still have the issue of the face being occluded with mixed reality and even its current sense. But as we see, people are using mixed reality and, you know, the actual like element of how we move as people and People who don't understand VR connect with that type of media much more deeply than they do an avatar that is rigged, you know, in kind of these janky ways where we're at right now in VR. To have the nuances of human movement is a really good way for people who don't understand VR to understand it. And we've seen things like, you know, in our community we've had people who wear a T-Rex costume and play Beat Saber. We've had... We've had someone who connected two controllers to a rod and became Darth Maul. So we have these things and these type of media that is being created that is literally not possible in any other form of media creation.
[00:06:09.550] Kent Bye: Yeah, and I think that, you know, as I was going through and setting it up, I had the help of people at the Portland Community Media Center that were really trying to get their own mixed reality lab set up. And the key things that conceptually that I got from that process was that there's a calibration phase of knowing where the camera is at so that you can render that camera within the virtual scene and have a specific angle. And then you have to kind of get everything sort of set up in a specific order. It's a little like alchemical in the sense of like if you messed up one of the orders then you know it doesn't work. But it's somewhat temperamental but also like dependent on each person's setup what kind of problems they're gonna run into which I guess from your perspective is hard to sort of create a universal solution, but you create the camera, the calibration, then you fire up the live, and then you open up the compositor, you make sure that you're seeing your composited self, and then you sort of launch into the game, and then you can have a screen that you can capture and do the mixed reality screen. I don't know if there's any sort of thing that I missed, but if that's sort of conceptually the major kind of points that you have to go through.
[00:07:15.885] Cix Liv: Yeah, so on an actual onboarding process, basically what you do is the first thing you need to do is obviously set up, you know, VR setup that's appropriate, the green screen, the lighting, and then you have to calibrate where the camera is relative to the virtual world. So we use something called Viver, and basically what you're doing is you're saying, okay, the camera is here, this is where it's located in the physical space relative to the game, This is how large its field of view is. And in some cases it's either a virtual angle or you can define it to where a tracker is. So the Vive has an infrared tracking puck that people use. So you could use it to calibrate to a virtual angle if you don't have a puck, you don't have one of these trackers, or if you're using a rift or a physical tracked object. And then essentially you would take that offset that value and you put that in our compositor and then essentially what we're doing is we're matching that viewpoint and rendering that viewpoint in the virtual world and compositing you as the player in the game.
[00:08:18.179] Kent Bye: Yeah, the issue I think that we were running into is that like we would try to calibrate the camera and then it wouldn't be completely right and then we'd have to kind of tweak it and you're essentially broadcasting a 2D vision of what you're seeing on the screen but then you have to kind of spatially aligned relative to where the camera is, the controllers in your hand, and then when you turn around or move around, sometimes that calibration is off if the field of view isn't right or whatnot. So, like, there's a challenge of trying to correspond, like, where the location of the camera is, and then where you are at, and then based on, like, what you're seeing visually with your hand in real reality broadcast on the 2D screen, and then trying to align these 3D dimensional objects up to your controller. And I guess it's that alignment process that, is that something that you would imagine that at some point you'd be able to just kind of algorithmically figure it out without having to sort of tweak the X, Y, Z coordinates of this sort of alignment? Because I found that it's sort of like, we can never really quite get it perfect.
[00:09:17.787] Cix Liv: Yeah, absolutely. So, the two ways we can ultimately solve this. The true way you solve this is a camera that has the actual tracking built into it. So, essentially a camera that has lighthouse tracking integrated into it. That way, no calibration would be necessary. You'd already have the divine, you know, field of view of what the camera is. It already know where it is tracked relative to the position of the lens itself. Truly, that's the end, like, most simple plug-and-play solution. The intermediary to that would be something in the sense of a physical QR codes or various different visual markers that you could essentially present to the camera on a, you know, printed A4 or something. But that has additional levels of complexity, like asking our end users to print on a sheet of paper and putting it on a board, you know, which is technically more simple, but again, there's other expectations there. Now it's not only a software solution, right? Now we have to expect people to have some level of, like, a physical object to be able to solve that, right? So that's kind of the two. The ultimate solution is literally, once we get to a certain level, we could do a camera that has lighthouse tracking integrated and or, you know, whatever tracking the headset uses.
[00:10:38.520] Kent Bye: So in the absence of people having this integrated solution, Liv, as a software company, has to figure out algorithmic solutions to be able to use all these different systems to be able to figure it out, ranging from a whole range of different cameras, of camera field of views, types of camera. I mean, I guess that's another dimension of even just the camera field of view and different types of cameras. How do you handle that and calibrate it such that It's got some sort of perspective that it's shooting you, but it's going to look right within the virtual scene.
[00:11:10.237] Cix Liv: Cameras are a whole other thing by themselves. What we're dealing with is a hardware problem, an audio-visual problem, an optic problem. It's literally like trying to solve that entire stack. And the short answer is the only way to fully solve this is an end-to-end hardware solution. And by solving it, I mean something that's simple enough that an LB would want to jump on this right away and press a single button and, you know, hey, here's content of you experiencing this game. You can share with your friends and family. Or what we're most excited about is the eSports vertical. You know, if we just look at this on a high level, there's no replacement for the nuances of how the human body interacts. And this is especially true in VR eSports. And we think that mixed reality will be what we're building will be a mandatory component of that as it gets built out.
[00:12:04.209] Kent Bye: Yeah, I mean, when I go to GDC, I am doing demos and I'm like, I want to be able to, like, shoot my camera at the screen and have a mixed reality video. But after setting up the mixed reality, just set up, I was like, wow, I could imagine being a developer and then trying to get everything ready for, you know, the demo at GDC, and then all of a sudden also have to do and manage the mixed reality setup. I think it's like having a plug-and-play system is what we all want. And so for Liv, is this something where you're kind of relying upon this magical integrated solution that's going to be in the pipeline to come, and that you're going to be continuing to focus on the software? Or what's sort of the next steps for you at this kind of like this dream of having a plug-and-play mixed reality system?
[00:12:46.416] Cix Liv: Yeah, so we've already built something to that level. I don't like talking about it a whole lot because we don't necessarily want to just become associated with a cube company. So what we've done is we've built essentially a green screen cube around all the different caveats that currently exists for where to place the lighthouses, to make sure the lighting is proper, to make sure the room scale is large enough. All these little nuances that don't seem that complicated, but if you set it up, would know they are pretty complicated. So we're going to be actually starting to sell that this next week. And that in combination with the hardware presets will essentially be an end-to-end solution for mixed reality for LBs and locations. Ultimately that's not necessarily our end goal as a company. Our end goal as a company is to literally solve the very large overarching problem which is watching people experience VR right now is lame. And we're trying to solve that and make it awesome to watch. And that evolves into much more than just mixed reality in its current sense. We've been testing things like voice commands to be able to change to virtual cameras where you have your own avatar. This is how we've kind of addressed the highest level of prosumer, which is people who would be willing to set up green screens and do this type of thing. And we think it's also very important to have the nuances of the human body, like as a proxy when you're watching content like this. But we also think that it's limited. And as hard as mixed reality is to set up, volumetric is even harder. So as much as we'd love to have full body volumetric in everybody's house, that's a few years minimum away from being anything even remotely consumer.
[00:14:32.475] Kent Bye: Yeah, I can imagine that going into the Locust Shame-based entertainment with one of these live cubes and doing a run of Beat Saber and being able to capture it, film it, and maybe share it on your social media account, this is sort of like, people want to, like, capture themselves looking like they're a ninja in this game. And just also capturing the level of embodiment and fun that they're having in it. Is that kind of the idea, is that these location-based entertainment would set up these mixed reality setups so that they would be able to create this media that people could then sort of have access to to be able to then promote themselves, but also to promote that location of where they're at, and as well as the game and the live mixed reality studio solution.
[00:15:11.311] Cix Liv: Absolutely. So I was just speaking to someone a little bit earlier who mentioned that there was a mall that The Void opened in, and the majority of the people who started to enter that mall were not people who had ever been to that mall before. And I think that what we're building is not only something that, you know, would increase throughput of locations, but it also, you know, help those locations communicate to people what exactly they're experiencing. Because when I talk to a lot of people about LBEs right now, I don't have anything beyond a lot of just vocal communication. How was the experience? Oh, I touched this, I experienced this, versus having a piece of media that would demonstrate to me directly, oh, this was me fighting against a Sith or something like that. And seeing that and being like, wow, OK, this is what I'm expecting when I go there, is a very effective way to increase the people who are interested and also the social anxiety that some people have of wanting to even put on a VR headset in the first place.
[00:16:15.483] Kent Bye: What are some of the other big titles, VR titles, that have live integration?
[00:16:20.718] Cix Liv: under NDA right now, yeah. So we have a few coming out, the biggest one that I can't talk about. But yeah, we have some games in the pipeline and I think that a lot of developers come to us not only because of mixed reality as a technology platform, but also the community that we kind of push towards games. I think Beat Saber was a good example of that, that a lot of the videos that media publications picked up, you know, and kind of went all over the place on all these different VR media sites, as well as gaming sites in general, were literally born in our Discord. which is, I'm going to say here, it's live.chat is how you get directed into that. And we see a lot of people inventing new ways to create this type of media. So when we integrate with a game or we work with a game developer, we're not only providing them the technology, but also this kind of like promotion and community that starts pushing their game. And that's super important, especially for multiplayer titles, that it's difficult to organize people to play with the limited amount of headsets out right now.
[00:17:25.627] Kent Bye: Have you seen an uptick of interest in Liv since Beat Saber came out?
[00:17:29.028] Cix Liv: Oh, hell yeah. We had three of our community members on release day and the day after just handling the inbound of our Discord all day for 14 hours. So, yeah, it was quite an uptick. Yeah, and it's fascinating to see that this type of media that I think a lot of people didn't take too seriously because it was stuck in studios before, to be able to actually have this accessible to people who don't necessarily have these massive studios, it's a new thing and it's exciting. Great, so what's next for Liv? What's next for Liv is, there's still that much larger spectator problem, which is, how do I communicate what it's like for me to experience this? Which, all of us in the VR space, we know how amazing it is to be in a Vive, in a Rift, and have 6DOF experiences. The average person doesn't or you know, just like in the general metrics itself. So we still want to solve the spectator problem which is going to increase the amount of people who who want to adopt VR and the amount of people who are willing to even create that content in the first place. You know, we're building things like a chat integration into Live so you can actually communicate while you're live streaming on Twitch. And then we're actively looking into virtual angles and avatars to be able to capture yourself in other ways that are limited by mixed reality. And I think one of the advantages that we have is that we have a community kind of directing us instead of us kind of Assuming, what do people want? We have a very active content creator community that's telling us, this is how we want to create content. This is how our audience wants to watch it. So that's great.
[00:19:22.781] Kent Bye: Yeah, and what do you personally want to experience in VR?
[00:19:27.808] Cix Liv: So my co-founder is named AJ, or we call him Dr. Doom. I want to have a competition where I beat him terribly and livestream it to at least 10,000 people at some point in my life. So that's my life goal, and that's why I co-founded this company. It's all about your ego. It's all about my ego. I have to beat him on a large scale. No, I mean, I'm just super excited about, like, personally, I'm super excited about the convergence of physicality and gaming for the first time in history. I actually came, my name comes from World of Warcraft. Six was my rogue character. It's actually what I, you know, it's what I personify right now in some limited sense. But I think that gamers have become kind of a footnote in the sense that their physical bodies are dying but their minds are fascinated. And I think this is the first time that we can have our minds being fascinated by what technology can provide as an entertainment medium while not letting our bodies die. Instead of the idea that we're just carrying our brains around, you know, we actually have our bodies are active as well, right?
[00:20:36.506] Kent Bye: Yeah, and there was actually a big discussion that I had on Twitter over the last couple of days of like talking about Beat Saber and could it be an eSport? Is this going to be like a new genre? And a lot of people were referencing the Dance Dance Revolution and Guitar Hero as kind of equivalent people sort of competing against themselves or competing against this objective score, but having also a performative element. So like perhaps doing sort of a mix between the objective score and as well as the Subjective judging that's I think a genre of eSports and then some people were like, you know, that's that's like bowling You know, you're like you're competing against yourself and you know, it's like there's a performative element but you know It's not as exciting as going head-to-head against somebody else where there's some things that you are doing that's actively against somebody else and what they are doing and so I'm just curious to hear your thoughts of like this As VR is coming in, there's something that's clearly very compelling to watch people play Beat Saber, especially if, you know, they have this performative element to it. But at the same time, I can see that it's still a single player and there's not this kind of like, what does that look like to actually have multiple people in a VR environment in mixed reality?
[00:21:44.172] Cix Liv: Yeah, absolutely. So, Beat Saber is working on a multiplayer aspect of their game, Beat Saber specifically. So, the thing that we've been trying to, I mean, the thing that's true is that live streaming, there is no replacement for live streaming. And the fact that it's instantaneous, the fact that you can kind of build a community that you respond to directly, there's no replacement to that. When it comes to multiplayer things, especially in mixed reality, it's an additional level of challenge, but it's something we are actively looking into. I don't have a short answer for that, but the truth is having it live-streamed and having it done on a tournament level would be something more compelling than necessarily competing against yourself in a scoreboard. But what we see already to some level is that the top people who are playing it, they are recording footage of them doing that, And especially if it's done in mixed reality, it's something compelling and something that people definitely watch.
[00:22:42.022] Kent Bye: Great. And finally, what do you think is kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:22:50.891] Cix Liv: Oh, virtual reality in general. Well, there's the dark side, and then there's the optimistic side. The dark side of it is that we all become so immersed in virtual reality, we don't give a shit about our real lives anymore, and we have divorces and neglect, and everybody's life is terrible on a high level, but we are so engrossed in virtual worlds, we don't give a shit. That's the negative side. The positive side is if we think of VR not as escapism, but as a way for us to reimagine what our real world could be in a best-case scenario, I envision some ways that we can kind of materialize and conceptualize a real world in a virtual world without the type of restrictions of what we would do to model that and kind of conceptualize what a better life could be like in the real world and almost manifest that. So that's the dark side and the positive side. It depends what movie you want to watch in 10 years.
[00:23:49.897] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say?
[00:23:53.452] Cix Liv: Yeah, so if you want to follow us, we're at LIV on every single social platform. Don't ask me how I got all those. And then our website is LIV.tv. And our Discord is LIV.chat. And we have a very active community that will help you out if you want to take this challenge to set up a mixed reality setup.
[00:24:15.815] Kent Bye: Awesome. Yeah, and it's worth it once you get it working. I think it's so satisfying once it finally works and you get it working and just to be able to see yourself in VR and have a better vision of what it feels like to be in VR. So yeah, I just wanted to thank you for joining me today on the podcast.
[00:24:30.606] Cix Liv: Thank you so much, Kent.
[00:24:32.236] Kent Bye: So that was SixLiv. He's the CEO of Liv, which is a mixed reality software solution and hardware solution that is trying to bring embodiment and physicality back into computing. So I have a number of different takeaways about this interview is that first of all, embodiment in general, I think is a huge trend within virtual reality, and it has been since the beginning, but I think there's something that's shifted and changed where it's becoming even more of an emphasis to have embodied gameplay, but also being able to show what you look like in VR. When people watch these videos of people within Beat Saber, they get this intuition like, okay, that looks amazing. And then they go into VR and they actually experience it, and it is amazing. And so there's this correspondence between like the first time we're able to really start to much more clearly communicate the experience of being in virtual reality. And one of the things that Six is saying is that live streaming is going to be absolutely critical to the future of mixed reality, but also like esports. And so that's a challenge. So first of all, Beat Saber, it sounds like they are actually working on a multiplayer mode. Because, you know, usually when you do a Beat Saber run, it's pretty fixed and you're just kind of reacting to whatever has already been pre-programmed. But what does it look like to be reacting to the agency of some other player, and what does that actually feel like? I'm very curious to see what that is going to look like. So, you know, to then be able to translate that multiplayer experience into a live stream event, especially if it's distributed, you know, I think there's issues with like the Battle Royale type of experiences where there's essentially lots of people who are playing. How do you live stream that event when there's like 100 different people within a single experience? you know that's like the extent of the battle royale problem in live streaming but then if you sort of scale that back to just even two people playing against each other then in order to watch that then do you need to be able to see both people and how do you logistically do that right now without being at the same co-located place to be able to be able to broadcast that it becomes very difficult to actually like watch some of these types of experiences. So does that mean delays? Does that mean like, you know, how do you actually create the software to be able to do that? So that's a whole other challenge that I think that Live is, you know, working towards. But at the same time, I think the VR experiences that are demanding that are also in the process of being developed. So the other thing is that it did actually take a lot of technical tweaking to actually get the live mixed reality system working. I think it's a challenging problem because there's lots of nuances for every individual system, but also like the camera and the calibration phase. I mean, it's not an insignificant problem. It's like takes like some technical chops and a lot of patience and They have their live.chat community to be able to have people help you out. And once you get it running, it's very satisfying, but it wasn't like an easy thing. And I think that's part because it's not an easy solution to come up with. And so they're also coming up with other things like the live cube, which you can basically get this mixed reality cube that's already pre-configured, all set up. You just sort of plug and play once you get all the hardware that is being specifically shipped from them. But it sounds like they're also working on these mixed reality cameras that have the Lighthouse solutions like integrated into it so that you can just be able to put that in with a Vive and not have to worry about all this calibration stuff. They know where the camera is at and they know what the camera is and so from there they can just do a lot more integrations more quickly and be able to have the alignment and everything just like look right. And so that's something that sounds like it's likely going to be for like the location based entertainment type of like in the higher end companies. I mean, it's going to be. Likely not a cheap solution to get this camera. Although we'll see you know over time I think there's gonna be more and more people who want to do mixed reality streaming and I think that There's a turning point that I think is gonna start to happen because you know, there's people that are in VR chat They're having these social interactions that are really driving a lot of like some of what's been happening with live streaming in VR But there hasn't been really any other title that has been like, you know found huge success and being able to do live streaming of that I think Beat Saber is maybe starting to see that, but still like relative to all the other games that are happening on Twitch, VR is not like a huge thing yet. And I think that mixed reality and being able to put your body and your physicality directly within the scene that is then somehow translated into a 2D depiction, I think that is going to shift things. And I think we're at this turning point where the mixed reality videos that are coming out from Beat Saber from, you know, people like Rage Sack doing the Darth Maul and just people, you know, doing different wacky things and having just a lot of style within their experiences, but also people who are making the perfect runs that if you're able to do a perfect run and just look like a badass while you're doing it, then that's also the types of videos that are going viral within Beat Saber. So embodiment's a huge trend within VR, and Liv is trying to bring back that physicality within computing. And we don't have to just use our brains and let our bodies die, but that we're actually doing experiences that are invigorating for our bodies. And I think if there's any large trend that I saw, both from VRLA and all the different recent trips that I've gone out and seen what's happening in the VR industry, I think haptics and embodiment and being able to do mixed reality streaming and stuff like that. That's one of the hot topics and trends that I'm seeing within VR right now. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a donor to the Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So, you can donate today at patreon.com slash voicesofvr. Thanks for listening.
