Soundboxing is a VR rhythm game that has found a community of people who use it for exercising in VR, with some people reporting that they’ve lost up to 50 pounds from playing it. Soundboxing is similar to Audioshield in that you punch orbs set to the rhythm of songs streamed from YouTube, but rather than using an algorithmic approach Soundboxing allows users to record their own runs, which means that all of the content is user generated. Soundboxing allows users to record and edit their own runs by playing a song and punching an invisible wall, and the scoring system encourages streaks, which results in helping to cultivate and track flow states.
I’ve really enjoyed playing Soundboxing, and it’s an engaging game that has a lot of options to allow you to follow creators, curate playlists, and customize your gaming experience. For example, you can record a run with you dominant hand, and then flip the recording so that you can train yourself to become more ambidextrous. The official Soundboxing website also has user profiles, with an impressive set of archive and search integrations compared to other VR game websites.
Soundboxing was created by solo indie developer Eric Florenzano, who was working on a VR browser for Reddit and discovered how compelling it was to record your embodiment in the process of trying to figure out what comments look like in VR.
https://twitter.com/DannyBittman/status/936330712805978112
Florenzano has thought a lot about the larger economic systems and architecture within the VR software ecosystem, and he shares a lot of ideas for how to cultivate and design systems that would allow people to make a living within VR. Florenzano’s Soundboxing relies upon user generate content, and he was surprised to find that about 50% of his users were creating and sharing content. He sees Soundboxing a 3D immersive web browser, and takes a lot of inspiration for how the Brave browser is attempting to create blockchain-based digital advertising with their Basic Attention Token.
This is a chat that I had with Florenzano back in May talking about how people are using Soundboxing for exercise and fitness as well a lot of deep thoughts about the future of virtual economies & paying arists, and how he’s thinking about what immersive fitness experiences will look across different immersive platforms and mediums.
LISTEN TO THE VOICES OF VR PODCAST
This is a listener supported podcast, considering making a donation to the Voices of VR Podcast Patreon
Music: Fatality
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So soundboxing is one of my favorite VR experiences that I've done in virtual reality and I've put quite a number of hours into it. It's a rhythm game where you can record your own embodiment and play it back and you can get into these different flow states. You can start to pull in all sorts of different videos from YouTube, and so there's all sorts of different variety where you can record your own embodiment or play other people's songs. And so it's a game that's really caught on within the fitness and exercise applications of virtual reality. People can go in and get some exercise, but also lose quite a lot of weight by playing an experience like soundboxing. So I talked to Eric Florenzano, who's the one-man independent developer of Soundboxing, and he talks about his process of how it started as this web browser for Reddit, and he was trying to think about ways to allow people to comment. And so he did these experiments of recording embodiment and just found this whole other application of being able to use VR to be able to exercise and expressing yourself through recording your embodiment. The other thing that I really appreciate is Eric's ability to be able to take a step back and look at the larger ecosystems and to look at the system architecture of the larger virtual reality ecosystem and what are the things that he can do as a developer to start to support ways for people to start to make a living within virtual reality. So we'll be covering all that and more on today's episode of the Voices of VR podcast. So this interview with Eric happened on Friday, May 19th, 2017 in San Francisco, California. So with that, let's go ahead and dive right in.
[00:01:54.653] Eric Florenzano: I'm Eric Florenzano, and I'm an independent game developer right now, and I work on a game called Soundboxing. Soundboxing is a music video kickboxing game, or actually it's not really kickboxing right now, it's a music video boxing game, where you can load up any web content, so right now it's only YouTube, but you can load up web YouTube content, and you play the video back, and while the video is playing back, the system records sort of your body movements. And when you actually are finished recording the sort of whatever body movements you do, which you kind of end up playing patty cake with a wall. Essentially what happens is when you load up the song, a wall comes up in front of you and you basically you punch through the wall. And every time you punch through the wall, it creates an orb. And the orb goes out and sort of shoots out from where you are. And then basically when somebody later comes to play the challenge, they'll play the same YouTube video and the orbs will sync up to how you punched the wall, how you played patty cake. And what happens is the orbs will come and sort of attack you and you have to punch them before they hit you. So the thing is, it really creates this dynamic where you have to really mimic the movements that the creator of the challenge did. When I originally set out to create this game, I didn't think that it was going to be anything more than a fun game. But what's happened is, since there's this active punching component and there's all this user-generated content for the YouTube videos, it's really taken off in the fitness community because there's sort of new content every single day to log in and to get really active and to play these musical, rhythmic games. And so people are having a lot of fun actually getting fit, which has been a real surprise for me and has been really awesome. So I've been really leaning into that use case and really excited about it. And I want to make soundboxing sort of the best place to go do that. And so that's what I've been focusing on is trying to make that really, really great.
[00:03:38.572] Kent Bye: Yeah, and I had been playing AudioShield for a long, long time, and that's a little bit more algorithmic in terms of how it's interpreting the song and putting the balls at you. And I feel like with soundboxing, it's a little bit more like I can go in there and record my embodiment, and then I can play that back to myself. And I can shut my eyes and go crazy and then just struggle to keep up with what I was able to do in that first. Pass because while I'm intuitively moving while I'm recording it when I'm actually playing it then the balls are coming at me and I have to you know kind of really do a number of different things of being able to Predict where balls are coming and if I'm crossing my hands But it is a very active game and I've noticed that people either treat it as a puzzle game so they're not moving around a lot and it's more of like a puzzle for how they're gonna move their hands around and but I personally like to treat it like a capturing of my embodiment so it's like a full-on dance experience where I'm like moving all around and then yeah so maybe you could just talk a bit about your thoughts on that.
[00:04:38.931] Eric Florenzano: Yeah, so it's interesting. When you look at this game, it's hard to not look and see the similarities to AudioShield, because there's orbs coming at you and it looks, on the surface level, it's very similar. But when you actually get into it, you get at the core of it. It's less about the orbs, it's less about the things coming and attacking you, and it's more about capturing your embodiment and playing that back and doing interesting things with that. I capture your embodiment, but then you can decide to do different things based on that capture. So when I was in development of the game, I experimented with a bunch of different ideas. So right now the game, when you punch through some threshold, it creates this orb and it goes directly out in front of you. But I didn't need to do that. I could have taken that same embodied sort of recording and made it so when you punch the wall, it goes out in the direction that you punched. Or that, you know, various, any number of things could change depending on just the interpretation of how you interpret the body movement. So my thesis on VR is that it's all about this. This is the future of VR, is all about capturing these new sort of input mechanisms we have, storing that, and then enabling creative outlets on top of that that we can share. because now that we have these new input mechanisms and now that we have this ability to capture essentially a motion-captured recording and replay that motion-captured recording, that that is the new unit. Right now we share videos, we share pictures, we share all these different things. We share apps, you know, I'll tell you, hey, I've got a great app I like, I just found it, you know, you gotta check it out. But what we don't do right now is we don't share performances. We don't share embodied versions of ourselves doing things. And so the way that this actually came about in the first place was I was building a Reddit app. I've told this story a couple of times now, so I apologize if people have heard this. But yeah, I was building a Reddit app, and I was thinking about how are people going to comment? How are people going to leave their comments? No one's going to type it out. No one's going to sit there and voice dictation. I didn't really believe in at the time, which now I completely believe in. There's been a lot of advances in that. But I was thinking, how are people going to actually leave their mark? How are we going to get user-generated content in VR? Because certainly while VR is interesting, everybody's not always online at the same time. So my thoughts are always about how do we create worlds where when people log on and log off, the world is now more rich. The world is now better for somebody else to come in and leave their mark. And so, yeah, I think that's basically where I'm going with all this. And I think that that's really what this all creates.
[00:07:01.517] Kent Bye: Yeah, the thing that I discovered about myself and playing soundboxing after I had been recording a number of different songs, it was suddenly so clear to me how much stronger my right hand was than my left hand. How much more dynamic range, how I was doing more, and how it was basically kind of like, if you think about drumming, you just have, you know, the left hand just in the same spot with maybe the right hand kind of playing all the different drums, kind of as a metaphor. And you have a feature in there to actually switch it. So then I started to record the extent of what my right hand can do, and then I switch it, and it forces my left hand to train my arm. So I've been really into this idea of trying to train myself to be a little bit more ambidextrous. which I think that, I feel like there's other changes that's doing to my body, but as somebody who's playing VR games, I think that's gonna happen more and more is that like a lot of these games and these gameplay mechanics require you to have like a good control of both parts of your body and to be balanced in some way. And still when I record myself doing it, I see this imbalance, but the more that I flip it and switch it, I feel like I'm slowly gaining more ability in my left arm, which I think is really cool side effect of this.
[00:08:08.775] Eric Florenzano: Yeah, that's actually something that's really been interesting is, so I've always been a clumsy guy. My whole life, you know, I'm a big guy, I've been clumsy, I've got elbows, I bump into things, you know, that's just part of me. But one thing I've noticed is that in the past sort of like three or four months, when I knock something off a table, like, I can catch it before it hits the ground. And like, there's just these like Superman moments that I've had lately where I'm like, I think the only answer that I can think of is that I basically day in and day out, I'm training my hand-eye coordination. And now that we've got millimeter accuracy on these hand tracking things, you know, yeah, you can. You're doing a training routine on yourself. And I actually ended up trying your routine, which is really cool, because you kind of start off doing both, and then you transition to kind of just doing one arm. And at first I was thinking, oh, how long is this going to last? And I'm thinking, oh. I'm getting really tired this is hard and you know it's it's cool so I like the emergent behaviors on this thing I had no idea anyone was going to do something like that train their one arm like that's so far away from the thoughts that I had when I was building this thing it's so cool to see I mean I literally was thinking when I was building this thing because you know I don't know when you build a game when you build content you don't know if people are going to take to it And I literally thought I would be signing on every night, creating content, creating challenges, and I was worried that no one would be creating challenges. But I've been flabbergasted because, and this is something I want to talk about too, which is that I think VR, so in other mediums, the engagement level that you get with other things. If you've got, let's say you've got an email, the number of response, the response rate on that is going to be, you know, maybe 1%. If you've got a social application, you know, most people, 95% or higher than that, probably 98% are going to be consuming the content and only 2 or 3% are going to be creating that content. And the thing that's been surprising to me as the creator of one of these sort of user-generated content things in VR is that I see up to 50% of the users that play the game create content, which is absolutely off the charts. And I don't think that's because my game is so amazing. I mean, I think the game is fun, obviously, but I think there's something happening here. There's something that is happening with VR that demands this sort of engagement, demands interactivity, and it demands this kind of interaction that everybody, all the brands, game makers, you know, social developers, everybody's looking for more engagement and VR is the way that we're gonna get it, in my opinion. At least VR or one of these spatialized computing environments, which I think that's where I'm sort of going at next is taking a look at all these different platforms that are coming out and thinking about how can soundboxing work on that platform and how can we get 3D spatialized exercise going for essentially every platform out there. And so that's sort of where I'm at now. And so it's the stage that the company's at as well. So if there's anybody out there that sort of has a headset or wants to sort of build this use case up, definitely contact me, eric at maxint.co. And let me know because I'm trying to get this really spread this use case of exercise and embodied capture and doing all this as sort of a bigger thing. Because I think it's bigger than it is right now. I think this has got a chance to be a huge use case for VR and essentially the thing that takes spatialized computing to a bigger place.
[00:11:07.950] Kent Bye: Yeah, the great thing about being able to record your own song is that if you don't like any of the challenges or you don't feel like you're enjoying either how you're moving or the song that's playing, then you can create your own. And I think that's the thing that's also interesting is that you've got a lot of just ways of almost creating a playlist of things that you're creating or you're favoriting so you can start to find somebody that you may like as a creator and follow them if you don't want to create everything, if you want sort of a challenge of a different style of embodiment. I personally found that a lot of the styles of moving was very static. I like to dance, I like to move around a lot. So I'm more focused about how I'm dancing than the fidelity of how I'm moving my hands. It's a little bit more abstracted in the mind if I'm just moving my hands and not my entire body. because of that you allow people to basically just record whatever both song they want but also style and they can play against themselves or they could you know if they don't want to create then they can play other people.
[00:12:05.615] Eric Florenzano: Yeah, definitely. So that was one of the things is everyone has their different taste in music. And beyond that, everyone has their different taste in exercise and dance. And there's so many different ways you can go about this that I wanted to leave it open-ended. But I also wanted to give it, you can leave things open-ended. And you can leave things to the users to figure out and to gamers to experiment with and get to a cool place. But I think you also need restrictions. And you need to put a lane on it and say, this is going to be focused on this one type of experience. There's no question when you load up Soundboxing, there's no question about what you're supposed to do. I mean, it could use maybe a little better tutorial and things like that, but what I'm trying to get at is it's not just dumping you in a room. You have a guided ability to create these sort of challenges. And yeah, I think that's the sweet spot. I think it's about creating these focused experiences that allow you to be creative and share that creativity with other people. And so, in addition to expanding beyond the platforms that I'm currently on, which is actually only the Vive and the Oculus Rift, the next step beyond that is also in, like, for example, the Vive trackers came out the other day, which are really exciting for me, because up until now, you've only been able to track the neck and the two hands. But with the Vive trackers, you can attach them to your feet, and you can do kickboxing. And kickboxing has always been something I've been very interested in. But you start to get into these use cases, and I was starting to look at the Leap Motion and some of these other input devices, and you start to get into these use cases where, how is it all going to work together? If there's people with feet, and there's people with hands, and there's some people with different capabilities, how is all of this going to go into a shared pool? So, what I'm starting to think about now more is creating different game types. So, maybe there's a kickboxing mode, maybe there's a mode where it goes out in the direction you punch in, maybe there's a 360 mode where it's all around you. And what I would love to be able to do is, for example, make that user extensible and make that user-generated content too. So people could essentially download React VR, put together an app, and that's an app, it's like a sort of a music rhythmic app that can take advantage of all the things that soundboxing does, like recording movement. I'm getting way ahead of myself. I haven't built any of this yet, so don't expect any of it. But that's what I'm thinking about. I'm thinking about how to extend it out into different apps and how things can be more extensible in that way. I've got a bunch of environments right now, and I would love to open that up. Okay, here's something I want to talk about. I'm sorry, I'm just ranting on all these different subjects. When I first launched the game, I just put in some very basic Cubemap backgrounds. There's just some backgrounds that are sitting there. I went on an asset store and I downloaded them, and I think I spent, I don't know, $60, $70 on these assets, and they're great assets. But I only spent $60 or $70 on them. I paid that guy, some guy from the Unity store or whatever, and now every time a user buys my game, they're getting the benefit of that, right? And so I've been thinking a lot about this and how can we create ecosystems where like there's some incredible tilt brush artists out there and they are creating art that should be paid for. They're creating art that is incredible and I think people really enjoy. So is there a marketplace or is there an ability to create a marketplace maybe where artists can create some incredible art and then put it out there for gamers, not game developers, but gamers to buy and get environments in that way. Because, for example, me, as an individual developer, I'll never be able to create all the environments that people are going to want to see in soundboxing. But I would love to hit two birds with one stone, for example, and create an ecosystem where people are actually benefiting from the art that they're creating. And then also, users are benefiting because they're getting all this great art content. And so, I don't know. There's all these different things I'm thinking about on how to increase the extensibility and how to build. I really want to build communities of jobs. I think VR is really where we're going to see all the jobs happen in the future. Like we're building these new worlds where we're all going to work in. And so how does that look? And I think it looks a lot like marketplaces and it looks a lot like creative work and it looks a lot like application development and a lot of the stuff that we've already been that we sort of see in Silicon Valley, but coming to everybody and being more sort of for everybody and gamers and anyone can do this stuff. Anyone who can pick up a six degrees of freedom motion controller can make art, which so I don't know that that's the stuff I'm thinking about.
[00:16:15.900] Kent Bye: I think you're asking a lot of questions that I've actually been talking a lot both on the podcast and I've recorded interviews that haven't released yet. It's this idea that there's competition and cooperation and that the competition of paying this guy 60 bucks and that's the end of it. There's a guy named Charles Eisenstein and he has a book called Sacred Economics and he says this thing which I think is very profound. He says that anytime that you pay somebody any money, you're essentially saying our relationship is over. You're giving me your services and we're done. Now, in a gift economy or something that's more yen currency, you may have a barter system where it's a give and take, or it's something that there's a relationship that's there over time. But you can also think of it as an open source community of like, what if people were chipping back into this art style that one guy created, and instead of all the money going to you, the developer, which you are enabling it, you know, what about the artists? And is there other people to get involved into this ecosystem of creating? Or if as me, as a creator who loves to create like different challenges, is there other ways to send either financial money back or other reputation back? So it sounds like that's kind of what you're thinking about is like how to create these ecosystems such that it's not just you that's profiting, but you're actually creating entire ecosystems that people could potentially be their full-time job of either creating challenges and soundboxing or art.
[00:17:37.987] Eric Florenzano: Exactly. And so that's step one. Step one is figuring out how do we create, like, kickstart an ecosystem. And I've got ideas on how to do that. But then I think step two that is imperative that we do after that, and, you know, I don't know if it's me or just, it's, I'm saying we have to do it as an industry, is we need to make these things federated and able, like, if there's a single ecosystem in a single game, that's, like, not good enough. It needs to be an ecosystem that works across games, across platforms, across all these things. So, I know there's a lot of work going on around OpenXR, and there's a lot of good work going on there, but I think that we need to be thinking about it from a much more holistic point of view. Like, I think that we need to think about how does payments work in a system like that, and is the unit of sharing an app? Is it an app? Is it a hologram? What if I want to share a captured embodiment of myself? Is that a GLTF thing? I think there's lots of different questions, and we don't have all the answers yet. And I think that another big piece of the puzzle is probably going to be the blockchain or some sort of distributed system along those lines that has a trustless sort of shared federated ability. I'm not sure how that's going to look, but I think that that's super important. I think that Tim Sweeney has been great about talking about IPFS, for example, and some of the capabilities of that and some of the opportunities that that opens up. And I think that that's opened my eyes, and I think a lot more people should be looking at solutions like that, and distributed systems, and how can we create something bigger than the sum of all of its parts, essentially, as an ecosystem and as a community. And I think we can do it. We're there. Basically, there's two game engines, and so if you can support two game engines, if you can support OpenGL, you pretty much get a good swath of all the games out there. And so after that, it's just a matter of figuring out all the long tail. So I don't know. I think I'm optimistic about all of this stuff. I think that we have a bright future for spatial computing, for VR, for AR. I think it merges. I think this is where all the jobs are. And it's just a matter of how do we actually create, how do we kickstart and create these ecosystems where we can live in the future and have economies and do all these cool things that we want to do in a new economy.
[00:19:39.767] Kent Bye: I think right now, because microtransactions are basically like, anytime you pay someone a small amount, the transaction fee to be able to do that kind of cancels it out. So, we haven't had a vibrant microtransaction economy. And I think that these cryptocurrencies, speaking of a blockchain, could potentially enable that. And right now, we don't necessarily have a metaverse ecosystem that's able to have that economic question solved that has everything integrated in. I did this interview with Cory Doctorow and he was talking about his book, Walk Away, how there's this group of people who essentially walk away from our existing systems of capitalism and go off and start their own completely separated and new gift economy. Now that's sort of like in the metaphor of IT, or like information technology, that's like going from a brownfield environment of legacy systems and starting an entirely new greenfield system. And I think the challenge is that we're kind of in this hybrid mode where we still are living in these old realities of these legacy systems, but with VR we have the potential to create these greenfield realities. and start to tie in some of these systems, like the blockchain, like distributed web, and maybe microtransactions such that there's these marketplaces where you could start to participate in. In some ways, ultimately, it'd be great if you're paying attention to, there's some sort of thing that's happening. that is kind of like right now you have Netflix where you're paying monthly and it's a subscription fee, but yet if you were to go and say for the next 10 years it's gonna cost you like $1,500 for Netflix, that's a different cost from doing the slow subscription base. So I feel like we have that models that are being proved out and that like if you go into that like you know slow, almost like pocket change, but in aggregate it's able to actually support people.
[00:21:29.972] Eric Florenzano: Well, and then there's a really cool thing that's coming out soon. It's called the basic attention token. I don't know if you've heard about this. Basically, it's a brilliant idea and it's made by Brendan Eich, the guy who created JavaScript and started Mozilla. And so the thing that's super exciting about basic attention token and the way they're doing it. is that what they're trying to do is create a cryptocurrency that is trackable by the browser. So the browser would basically have an auditable trackable piece of code that would keep track of what the user is actually looking at. What is the user paying attention to? Because the browser is the only thing that can reliably know that. And then they've created this system where it's a closed system between advertisers, publishers, and consumers. And then consumers' information is all anonymized. And then the idea is that whatever time the consumer is spending their attention on, that publisher gets some small token of this currency. And so there's a publisher, there's an advertiser, they've worked it all out, so theoretically this ecosystem could actually happen. So that ICO, the initial coin offering, is happening actually at the end of this month, on I believe May 30th. This is happening right now, and it's a collaboration with the Brave web browser. And so I'm looking at this as an outsider, and as somebody, I mean, if you think about what soundboxing is, it's a web browser, for browsing YouTube videos that also has a VR app attached to it, right? And so I'm thinking about, okay, I've got this web browser. People are spending their attention on these YouTube videos. You know, I'm thinking about maybe I can participate in this ecosystem. Maybe soundboxing could be another basic attention browser. Or so, I don't know. The cool thing about what they're doing is that they're taking the internet and they're retrofitting a new monetization method onto it. So while this augmented virtual reality metaverse is all greenfield and it's such a great opportunity for breaking from tradition, there's also something out there that's showing us how we take something and retrofit on top of it. And I think there's a very short distance to go between a brave enabled basic attention token enabled web browser that all of a sudden maybe if it has web VR, maybe if it's attached to a blockchain that can distribute apps, We're getting really close to a place where there's not a whole lot of changes in the technology that we need to retrofit the existing internet to being able to be this mecca that we all want. And so, I don't know, I'm optimistic. Like I said before, I'm optimistic, I'm excited. There's all these projects that are happening right now. Basic Attention Token, Ethereum, you know, there's these VR projects that are happening, the proliferation of all these APIs, OpenXR is going to help a lot. This is like a stream of consciousness at this point. I'm just saying words, but I mean, I'm excited. I think this is really, it's all going to start coming together now in the next two, three, four years. And I'm just so excited that I get to be part of this whole process.
[00:23:59.887] Kent Bye: No, it all starts with this fitness game that was a Reddit browser for commenting. I think that the journey of where you've started and where you're going and kind of iterating, but also the thing that I've just been super impressed with what you've been able to pull off with soundboxing so far is that you have a a website, you're capturing all these movements. While you're playing, you have robots that are showing the original embodiment of how the person was actually moving when you recorded it, such that you can actually record a performance and have a performative element of not only just creating a capture of your embodiment, but also a performance that they can visually see and dance with you. So it's like a collaboration in that way. and also just the leaderboards and participation. All in all, the reason why I'm so drawn to it is there's been a lot of abstracted expressions of agency or sort of mental gameplay. It's like doing things that were done before, but just in VR. But I feel like something like soundboxing would just literally be impossible to do in anything other than VR, and it's kind of architected from the ground up. focusing on this sense of body presence but also active presence where you're physically moving and people are losing up to like 60 pounds by playing it.
[00:25:16.425] Eric Florenzano: Yeah, it's absolutely crazy. I mean, I just, I know that from the past, you know, I've been around for the industry for a while now. I started doing websites and then apps and then did the, you know, the YC thing. And so I've seen the cycles and I've seen, you know, how this all kind of works a little bit. And the thing that always doesn't work is taking an old idea and trying to sort of like, do it on the new platform without changing anything or changing minimal amounts of things. So I knew approaching this new medium that I was going to need to think from the ground up, from the start, around what does this now mean? And then I think beyond that, it's all about following where does it lead you. And there's no way I thought I would be building a fitness embodied repeating game but hey like that's what happened and I think that who knows what soundboxing or whatever this is right now turns into you can tell I've got all these crazy ideas who knows how many of them will get done I'm only a single developer right now so I've got some contractors helping on art and things like that but it's just a matter of how much can I get done how much do I want to grow the company how fast do I want to grow it and how big of an impact can this all have
[00:26:19.737] Kent Bye: Yeah, I wanted to dive into a little bit of the actual scoring system of soundboxing, because I think it's different and distinct than something like AudioShield. So AudioShield, you have sort of a percentage that you have of balls that you hit, and you also have an artistic style, which essentially amounts to how much you're moving your arms around. So, you know, the more that you're punching and moving around, you can have both an artistic and an accuracy scale. So, I think with soundboxing, though, it's a little bit more brutal in terms of, like, rewarding people who are consistent rather than people who have artistic flair. I'm someone who has a lot of artistic flair, and so I often will miss one or two, but yet, you know, I feel like I get unevenly penalized for being artistic in that, by some ways, the way that you've designed the scoring system is constraining the way that people are playing it. And if it's an embodied game, I feel like you're penalizing people for being too artistic. You're rewarding people for being precise, but precision I don't think is necessarily, like, the thing that I play for, but that's just my style. But I'm just curious to hear your thoughts as you're doing the game design of that, making these trade-offs.
[00:27:25.526] Eric Florenzano: Yeah, no, that's a really good point. And I worried a lot about this when I was implementing the scoring system, because, I mean, I've talked about this a little bit before, I think, and I oftentimes don't even look at the score. I don't really care about the score. I'm just having fun when I'm playing the game. And I know that's not the way that most people play games, because I'm just weird like that, and it sounds like you're that way, too. I do have people come up to me and they say, I don't care about any of these score things, whatever, just dance. And that's great. I don't want to make that use case not possible. I do it that way. So that's definitely a use case I want to keep. The thing was, in development of the game, I had a big problem with people not realizing that they had missed a note or people had not realized that they had made a mistake. And so over time, I created all these different ways of giving you subtle and unsubtle clues about why, like, yeah, you did just lose your performance. But I didn't want to make people feel horrible, like they just died or something. So that was the balance I was trying to strike. And this has been a relatively consistent point of feedback from a lot of people, is it's very streaky. So depending on how long your streak goes, that will really affect your score. And the system may lean on that a little bit too much right now. I've been hesitant to change the scoring system just because everyone's scores are already locked in. I've done a few changes here and there. For example, when I added the ability to change the scale and the offset of that, if you scale it up, you don't get any extra points. But if you start to scale it way down, all your punches are going to get, like if it's 60% of the size, your punches are only going to get 60% of the score. So I have played around with it a little bit just in terms of trying to keep it fair. But I haven't made any major breaking changes to the scores yet. And it's a matter of just of time. I'll do it, but I want to try and get the most feedback as possible, so I only have to do it hopefully once, and then I'll do that.
[00:29:10.393] Kent Bye: Well, I think, you know, so after playing the game, and so this is a thing that just to kind of reflect back as I'm playing the game, and there's a couple of different pieces of information that I'm getting from the score that I get. And I don't think that, let's say, the quality of a run necessarily is reflected in the score. If I get, like, let's say, half of the entire first ones, I go a streak where I don't miss one at all, and then I miss the rest of them. That could give me a bigger score than if I miss a couple intermittently, equal distance, but get, like, 95% of it. And so I think the ultimate goal in some ways is achieving flow and flow states. And, like, I don't know if, like, encouraging people to focus on that precision is, you know, there's a certain degree that you are actually kind of measuring the flow states of whether or not you're able to do a streak, because a streak could be, like, you're in that flow.
[00:29:59.499] Eric Florenzano: Yeah, and there's other problems with the leaderboards, too. Like, for example, if you score, like, the more times you do it, it keeps adding new scores to the leaderboard instead of just showing your top one. So there's definitely things I want to do. But another thing that I've been really just incredibly impressed and surprised by is You talk about the performative aspect of this, and you talk about how some people are very static, and some people stay still, and some people really move around, and there's some serious rock stars in there I've noticed. There's this guy named Yerbal, and there's another guy named Hiyoromo, and these guys are performance artists really at this point, and some of them have actually gotten quite a big following. I think Hyoromo has like over 400 followers now within soundboxing, so every time he posts a new challenge, like there's 400 people who see that, which is crazy. That blows my mind. But yeah, no, that in and of itself has been a really incredible thing to see. It's like a new kind of user. They're like a super user, but they're also like a rock star too. They really are. They're putting performances on every day.
[00:30:59.631] Kent Bye: Well, one of the things that I was surprised to find is that as you do a run, all of your movements are recorded so that you can actually watch somebody who's at the top of the leaderboard and how they played it. And you can step back and actually watch them. And also, as you're actually recording it, that's being recorded. But just to kind of take a step back, because I've been giving Oculus a lot of grief about this potential of recording physical movements. So you're starting to record these physical movements and save them. And in some sense, you could say this is starting to record biometric data. And I have a lot of deeper concerns about privacy. But yet, in the case of soundboxing, I'm like, totally OK with it, because I feel like what you're able to do with it is amazing. So we're kind of in this period where we have this exploration with people doing stuff like this. But I can also see, in the future, where that could potentially be like, hey, you've got all this data of how I've moved. Let's get some machine learning to be able to determine how people are moving. And now, all of a sudden, we can tell that, yes, this is definitely Kent Bye's movements, because we can tell this is matching his profile of all the features that were extracted from all the runs that he's ever done. So maybe, I don't know if you have any thoughts about this process of now capturing biometric data and how you're using it within soundboxing.
[00:32:12.411] Eric Florenzano: Yeah, I was super worried about this. I was super worried about this when I launched, because I thought, especially on the playback side, when you're playing back the recordings and it's recording you, I think that the key is, it has to be a core part of the game. Like, the fact is, if somebody doesn't realize that, you know, they're being recorded, it's just essentially my fault as a game developer. It should be so abundantly clear that they're being recorded and that that's, like, the game is to be recorded. that there's no qualm because if you didn't like it, you refund the game or you get rid of it or whatever. It's the concept of the game is something you don't like. So I think the thing is, is just being explicit. The ultimate nightmare scenario is somebody is being recorded, they don't know it, and that recording is used for nefarious purposes. So, we never want to get into that situation. So, as far as there's anything I can do, I try and make it as explicit as possible. On the other hand, so, you know, this is an amazing amount of training data, which is for good or for bad. On the good side of things, I've been experimenting with machine learning and essentially training an algorithm to take the performance and classify whether we think you'll like to play that based on the performance data. So using that for things like recommendations, where the song title might not be enough. You don't really know based on the song title, but you'll know based on all the different punches and all the performance and the movements. You might really like it when somebody's dancy and jumping up and down, and the algorithm could find out that you like that and tailor your suggestions towards that. But somebody else might really like the super, you got to place every punch in exactly the right place and very methodical and not dancing at all, you know, very precise. Some people really get into that style of game. So I think there's a lot of opportunity for machine learning to help with that problem. There's a lot of opportunity for machine learning to Right now, everything is user-generated content, literally everything. You can't play a game if it hasn't had somebody come in and play a song. But some of these new generative adversarial networks and some of these new convolutional neural networks and all these different things are enabling the ability to generate one-shot examples that look a lot like the distribution of the real... Basically, I'm looking into ways of automatically generating content that's very good and that people will really enjoy to play. And I think that's only possible based on doing all these recordings. That's literally not possible unless I do all these recordings and I'm able to do those correlations and able to do all these things that help people like it. It's a trade-off, and I think more often than not, it's exciting and it's great. It's just a matter of setting user expectations, not going outside of those user expectations, and always leaving the user in control. If you do all those things, I think we're going to be in a good place.
[00:34:47.600] Kent Bye: Yeah, as I've been recording things, there's a couple of different things I've noticed is that I can shut my eyes and just see what feels good to move around, but then when I play it back, it's just utterly boring, right? And so there's certain design patterns of different ways of moving my body, such that it's interesting to both do it in the original recording, but also as I play it back, I personally love the rhythmic aspect of it and love imbuing my humanity into it and so I think rhythm detection is actually a very difficult problem that is sort of somewhat unsolved and so being able to do beat detection on top of like the game design patterns and stuff on top of like what kind of music do I even like so for me I'm like like skeptical for how far that can go but I also am optimistic for like where you can take that but There's a certain amount of like design patterns for what actually makes an interesting song and I think that is the thing that actually is like almost learning how to both speak and read a language at the same time because you're recording it and playing it back and then you're getting this feedback as to okay what are the things that I actually like to record and what's fun and interesting for other people to do.
[00:35:52.950] Eric Florenzano: Yeah, I think it's a little bit like learning to speak Pig Latin, because you already know how to dance, you already know how to, you know, punch, but you don't know how to do it in this context with all this stuff. So, yeah, I think it's about taking such a... Okay, this is one thing that I think people don't really understand. When you think about creating a challenge and creating all these things, it seems like a very daunting, difficult, like, it's gonna be a super user that needs to do this. But really what it is, is I hack your familiarity with dancing and your familiarity with punching, You play the song, you play the music video, and you punch while that happens. And so the length that it should take to make a recording or make a challenge is the length of time that it takes you to go through the song. Unless you want to go back and make some changes, which now I've enabled that. There was a long time when you had to do it all in one shot. And if you screwed up, well, you had to do it in another one shot. But now you can edit, now you can change, you can go back, you can, and I'm gonna add the ability, it's not out yet, but it's coming soon, where you're gonna be able to remix, so you'll be able to take someone else's challenge, load it up in the recorder, and change just the parts that you want to, and remix it and re-save that out as a new challenge. So I think that's gonna be another vector for incredible cool challenges and things. I don't know, I'm just, I think there's a lot more places for this to go in this genre, and I think, yeah, I mean, I think I'm the first game of, really, of this genre. I don't want to discount some of the other rhythm games that are out there, but the embodiment idea, the idea of recording and playing that back, I think there's going to be a whole broad range of games that follow this pattern. Like, I was actually working on a charades game with a team where you asynchronously go online and you record some charades recording, and I think that was a good idea. It didn't actually end up coming to Fruition yet, maybe it will someday. There was another idea, the Reddit idea. I mean, there's so many things that can take advantage of this concept. But then on top of that, I think multiplayer is also important, too, and the social embodied presence with somebody there and being able to actually have a social communication, that is important. So that's coming, too.
[00:37:44.954] Kent Bye: Yeah, there's a couple ones just that I've seen. There's a mind show, which is more of both moving your body but also acting. Tavori is a way to be able to record things. There's an experience that was just at Google I.O. called Dance Tonight, which you're able to actually, like, Record yourself and loop yourself and you know, that was really amazing to be able to like actually record your body It's amazing how much little information from moving your hands around and head, you know Just basically two hands and a head you can tell there's so much humanity that's imbued into that So I do think that embodiment is gonna be a whole genre of much more of these games it's part of the reason why I'm so drawn to this because it's like I That's kind of the types of games that I tend to be drawn towards. But for you, what specifically do you want to experience in VR?
[00:38:31.979] Eric Florenzano: Me? No one is going to agree with me. I'm going to go off and I'm going to say something that no one agrees with, because I'm just that way, I guess. I want to see an experience where you get in a VR virtual theater, and instead of having film or 3D content on the screen, and by the way, like a real theater with a model and everything, and then outside, instead of having a screen be there, have there be a window, and outside the window is rendered content. I think that there's a whole world of essentially like flat content that we can create now. It totally flies in the face of everything that people are talking about right now, but I think we need more sit-down, passive, interactive experiences, if that makes any sense. Like the ability for you to sit down after a day of work or whatever and relax while playing VR. You don't need to get rid of controllers for that. You don't need to necessarily stand up for that. I think there's a whole lot of opportunities for interactive, passive things, if that makes any sense at all. Do you have any example of that at all? No, not really. I have ideas, but I don't know how to articulate them very well. Okay, I'll give another answer. I also think that I want to see more asynchronous experiences. There's nothing where you can go in and you can build a house and have other people come explore that.
[00:39:49.294] Kent Bye: I think there's Anyland. I think they're starting to do that. Anyland is an experience that does that.
[00:39:53.720] Eric Florenzano: I just downloaded Anyland. I haven't tried it yet. I'm excited to. But that's the kind of thing that I want to see a lot more of. You jump in the world, you do something, and when you leave, you know, somebody else can come in and enjoy that thing that you did in the world.
[00:40:06.950] Kent Bye: I think that it gets back to what you were saying earlier, which is like you go and spend some time in VR, but that when you leave, you've actually added something to the experience in some way for other people to enjoy.
[00:40:15.653] Eric Florenzano: Yep, yep. So, I mean, like, I've been doing this for a while now. I got the DK1. I've seen the Mars experiences. I've seen the Apollo experiences. I've seen the incredible things that there are out there, and they are incredible, seriously. I mean, everyone who's listening to this is bought in, at least in some way, to the idea that this VR stuff is truly impressive, at least visually. I'm starting to go beyond that and starting to wonder, what I crave is more social experiences and more about what are we gonna do with that now, and not necessarily all in the same time and the same space in there together.
[00:40:50.528] Kent Bye: Great. And finally, what do you think is the ultimate potential of virtual reality and what it might be able to enable?
[00:40:58.270] Eric Florenzano: Well, I kind of talked about this earlier, but I think the ultimate potential is jobs. I think this is where all of our creative outlet and work outlet is going to go increasingly in the future is moving into these virtual spaces. And so I think the ultimate potential is this is where all the jobs are going to come from. So that's ultimate potential. This is how we keep society going because the other jobs are going away, right?
[00:41:22.168] Kent Bye: I just was at Google I O and they said that every day there's 1 billion hours of YouTube content that's being consumed and I just that's an impressive number but then at the same time I'm like wow that's a thousand hours of like what cat videos and like What kind of productivity is that? Is there a way to harness into that so that those one billion hours a day are actually going in and generating something that is being shared? I mean, there's a lot of legitimate uses for that, but I'm just saying there's a good portion of it that is just being passive consumption.
[00:41:52.537] Eric Florenzano: Well, I think let's talk about the legitimate uses. I learned machine learning from YouTube. I learned machine learning from YouTube. That's crazy. That is insane. And I learned it through Stanford from YouTube. We are living in an age where there is access to everything. And I think the only barrier now is the will and the entertainment of it. It's not very fun to learn a new thing always, especially if it's a dry subject. So yeah, OK, going back to other things I want to see in VR, I think Educational content, the ability to interact with it, and the ability to understand and grok something from a, like, hack your hippocampus and deal with the spatial awareness and, like, maybe remember things better. This kind of stuff is what it's all going to be about, is how do we get users creating content? Like, how do we get users creating educational content that they share? Right? That's what the YouTube of VR will be, in my opinion. It's not going to be a 360 video. It's going to be people creating creative narratives. Something more like MindShow is going to be the YouTube of VR, in my opinion.
[00:42:47.232] Kent Bye: Yeah, or the Google Expeditions, which is, you know, they're starting to both start with, like, the 360 photospheres and have narration and stories, but, you know, they're moving into having more and more things being driven by WebVR, WebAR, the open web, and so just having the behaviors of the open web tied into these immersive experiences so that we get this feeling of the metaverse that we all want.
[00:43:09.355] Eric Florenzano: Yeah, I think that's where I'd like to see things go, is tying in WebVR, tying in these technologies, using the social things that we've learned from the previous generations of apps and games and social things, and creating something that is a bigger community, a community that we all enjoy being part of, and a community that grows and expands and can encompass the whole world eventually. We want everyone using VR and AR, we want everyone using spatial computing. Thinking of ways to grow and expand the whole ecosystem and community is what we all have to be doing. And that's what I'm thinking about.
[00:43:40.624] Kent Bye: Awesome. Well, thank you so much. Thank you. So that was Eric Florenzano. And he is the developer of Soundboxing for the Oculus Rift and HTC Vive. And I highly recommend it if you haven't had a chance to play it out. It's one of my favorite VR experiences. And I think that Eric's also just done a lot of really amazing stuff on the back end, both on his website and just all the architecture of the game. I think that the gameplay and the interactions are the things that is the most compelling. And I think it's just got some gameplay that is super engaging. And if you're looking to use virtual reality as a form of exercise and get into different flow states, then I'd highly recommend checking out what is going on with soundboxing. So, I have a number of different takeaways about this interview is that, first of all, the thing that Eric said about thinking about his virtual reality application as equivalent to a 3D immersive web browser, and that he was allowing you to have a new experience with the YouTube content, but that it's within the context of this experience that he's created with all these gameplay interactions. And so, that is going to be the equivalent of any website destination within the immersive web. And also just trying to think about these deeper questions about how do you architect a system that's going to allow people to contribute and participate and have not just the creator of the VR experience get all of the revenue, but are there ways to have the creators and the artists participate within these ecosystems and then actually get paid for the work? This was recorded back in May before Google has released a lot of the tools that, you know, I think have since started to fill in the gaps for some of the things that Eric was talking about in terms of being able to share art. Google's Polly, you can both upload Tilt Brush sketches as well as blocks, drawings, and everything on there right now is done with Creative Commons, and so it's by Creative Commons attribution. You can actually change the attribution and just have it show up there without having people download it. But it's a system that is designed to help promote and get content out there. It's kind of like a clipart for people to be able to have access to to be able to do other applications within VR that weren't possible before. Now, the only thing that I would say is that it wasn't necessarily designed from the beginning to be able to actually pay those creators and have different microtransactions in there. I think that what Google wants is just people to be able to use the technology, not have any blockers in terms of not having access to content or 3D immersive content, because that is for a lot of people the biggest blocker to actually making different VR experiences. So the fact that that's out there is huge. However, there's not something that's built in from the beginning and the inception that is going to allow those creators to get paid for that work. And I think that is some of the things that Eric has been thinking about because he's created this experience and some of the largest value for participating in an experience like this is having access to this different content that other people have recorded. One of the things that was surprising was that Eric said that there's about 50% of the users that are playing soundboxing, which happened to be, you know, early adopter of VR people. So it could be, you know, somewhat of a skewed sample size as you go into other people adopting VR. But the point is, is that people want to record and play their own embodiment of these experiences. And, you know, some people who don't do that, then they can play other people's sessions that they're creating, but that a lot of people are finding a lot of enjoyment out of creating their own content. And that Eric's really thinking about, well, if that's a big part of the value that he's bringing to this game is what the user generated content is doing, then what are the different systems that can be architected within the fabric of this software ecosystem that are going to actually kick back and allow people to get paid for their work? So the Brave web browser has got this attention token, you know, using this sort of cryptocurrency to create this closed ecosystem that can start to harness that attention that you're paying attention to without doing a lot of surveillance on your specific, like, who you are and tying that back to your identity, but to just have it participate in this open market. So that's certainly one aspect of the overall technology ecosystem that is still developing. And since over the last year, throughout the course of 2017, cryptocurrencies and Bitcoin and Litecoin, all these things have really jumped into the mainstream. And there's been a lot of hype and excitement around the potential of these new cryptocurrencies. And if we really want a virtual reality to succeed, then we are going to have to think about, are there other ways that we can create these economies and ecosystems that make it such that people can actually make a living? The stuff that High Fidelity is doing with their High Fidelity coin is really interesting. They just launched recently a island where you can buy avatars. And instead of something like Bitcoin that is capped, the amount of Bitcoins that are ever going to be created at like 21 million, there's something that's kind of equivalent to the Federal Reserve within high-fidelity systems so that they can start to add more cryptocurrency into the system as it's needed. And so there is going to be this inflationary aspect. So as you hold on to the high-fidelity cryptocurrency, the idea is that it's going to be worth less and that there's going to be a little bit more of an incentive to actually spend that currency. To me, that's an interesting idea. I'd love to talk to Philip Rosedale and High Fidelity more at some point, whenever I run into them again, to talk more about that. But it's just that idea that right now a lot of people that are using and buying Bitcoin, they're kind of treating it more as an investment asset rather than an actual currency that you're using. Because if you're starting to use it for payment, then the value of it is going up to, at this point, it's over $17,000 per one Bitcoin. So because the prices are fluctuating so much right now, it doesn't make it actually good to use it as an actual currency. So I expect that there's going to be more and more of these different cryptocurrencies that are out there and that we're going to need some sort of integrated way to be able to make payments within virtual reality. And finally, one of the things that Eric says in this interview is that he sees that the future of VR is going to be able to share your embodied captures with each other. And this was before the Animoji was released over the summer by Apple. And I think that is starting to get into a little bit more of a consumer way of you recording your embodiment and recording like an emotional reaction to something. But you are embodying this virtual avatar and sending it off into the iPhone X. It's the phones that have that front facing camera that are able to do that. And so I do think that we're going to start to see a lot more of these abilities to record your embodiments. And I think that Snapchat has been doing that for a long time, being able to have these different filters, be able to embody these different virtual characters. And it just brings something out in your own personality that's different than had you not been embodied into these different characters. And then I do think that virtual reality as a overall thing is going to be a lot about, you know, recording your embodiment and sharing it with other people, both with like something like Snapchat or with Apple's integration with Animoji. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you'd like to support the podcast, there's a couple of things that you can do. First of all, just spread the word, tell your friends. Grassroots marketing is one of the most effective ways that people hear about the Voices of VR podcast. Even just sending out a tweet or leaving a review on iTunes helps out a lot. Secondly, any financial donations that you can commit to as a monthly donation ensures that I can continue to bring you this coverage, and so I encourage you to consider donating and becoming a member. If you want a suggested donation, $5 is a great place to start at and that it just helps ensure that I have a healthy financial grounding to be able to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices in VR. Thanks for listening.
