Geoff Skow is a co-founder of Fish Bowl VR, which has over 200 early adopter, VR enthusiasts available to do user testing for VR experiences. VR developers can get a monthly subscription to get on-demand, user testing with let’s play videos as well as quantative and qualitative feedback on their experience. Fish Bowl VR provides feedback ranging from the framerate and performance across a spectrum of different hardware, ratings on the GUI and game play, as well as open-ended survey questions talking about what types of things could be improved or added to the experience. Some of the biggest open problems that Geoff sees VR developers face is how to train users how to play their game, and he talks about some different tutorial approaches that are embedded within the VR environment. Fish Bowl VR has over 250 VR enthusiasts on-hand who are paid to play and record their playtest sessions, and are always looking for more users to get paid to play VR experiences and offer their feedback. Getting objectively detailed feedback from people experienced and familiar with VR is certainly filling a market need, and Geoff says that VR developers can use their service in order to track their development progress over time.
Listen:
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.978] Geoff Skow: I'm Jeff Skou, co-founder of Fishbowl VR. We offer usability, playtesting, and analytics for VR. So we've got a panel of now over 250 early adopters that own DK2s primarily, but also gear and cardboard and eventually the Vive and PlayStation VR. And they're kind of available on demand to do playtest recordings, like a split screen of what they're seeing in VR and then what they're doing. And then they vocalize kind of where are they delighted, where are they confused, where are they, God forbid, nauseous. And then they also fill out post-playtest questionnaires with questions like, on a scale of 1 to 10, rate the menu or the UI or your level of comfort. And also freeform stuff like, And if you had a magic wand, how would you improve an experience? So really, the goal is to provide unbiased feedback that I think is really critical at this stage in VR.
[00:01:02.915] Kent Bye: So what are some of the most common mistakes that you see people making VR experiences in terms of the feedback that people get?
[00:01:08.837] Geoff Skow: I mean, you see most of them written about by developers. There are obviously best practices around not getting people sick with what you do with the horizon or not forcing the point of view. But I think those things people are very quickly catching up on. And I'm much more interested in the things that people are trying out about how to educate people about how the game is played and how to kind of get people up to speed so that they're comfortable in an environment. And figuring out, you know, the more important things, like how do you get people to come back and play? How long do they play for? How do you really get them stoked?
[00:01:40.290] Kent Bye: And so what are some of the other findings that you found? Because having that many users that are testing and having different playtests, what are some of the other takeaways that you've had so far in that venture?
[00:01:50.999] Geoff Skow: Well, part of it's just really fun to see the experimentation. I mean, that's a theme you hear over and over again here at Kinect, and I think for good reason. So we were just doing some playtests of, have you played Marble Mountain? It's super fun. I mean, I grew up playing Marble Madness, which was on NES, so I'm clearly dating myself here. But I had many a fun frustrating hour playing that game and this is game reimagined in VR They're actually coming out with a new build that we'll be doing some tests for but it's kind of this marble that rolls or that you navigate down a mountain and make sure that it doesn't fall off and They bring the instructions and kind of bring you up to speed on on the dynamics of the game and how it works as you're rolling through these kind of beginner levels and And I thought that was a really clever way to kind of bring the instructions into the environment. So seeing those kinds of things that people are trying is cool.
[00:02:40.502] Kent Bye: Great. And so how did you get into this? How did this get started?
[00:02:44.097] Geoff Skow: You know, my co-founder Josh was one of the early employees at Justin TV and worked at Twitch. We met at a startup in the Bay Area. My background's in kind of usability, analytics, but more on the web. I worked for quite a few online startups dealing with online marketplaces and I was a big user of sites like usertesting.com and I always just found that this idea of putting an app in someone's hands and seeing how they react without any coaching or any kind of preface is ultimately the best way to get feedback and know whether you're on the right track or not. So I came, I guess, back into VR. I've been interested in VR for 20 years. I guess after reading Ready Player One and then being in the Bay Area, just going to meetups and demoing as much stuff as I could. And it became pretty clear to me that that kind of unbiased kind of objective feedback was something that could serve the community well. And I felt like I was in a position to try to do it. So trying to make a real company out of it now.
[00:03:39.565] Kent Bye: And so what are some of the other analytics that you're looking at? Because I know in traditional usability testing, you have the ability to track somebody's eyes and do eye tracking and be able to see what people are actually looking at. But in certain aspects, you have that data that's theoretically available in VR. not necessarily readily available. And even if it was, it may not be digestible in a way that's even useful to be an overload of numbers. And so when you look at analytics and usability of VR, what is it that you're actually looking at?
[00:04:11.053] Geoff Skow: I mean, some of the obvious things that we're able to grab at present is duration of play, frames per second, and then you can correlate that to the different system specifications. So we spit that out for each tester. So you can say, okay, what graphics card do they have? I mean, GPU, et cetera. And I could see a world where we go more towards some like analytics plugins. But I think at this point in time, it's really one of the values that we provide in terms of data is just go get five, 10, 20, 25 people to play it and see how they score the menu in the UI. See how they score their level of comfort and then provide yourself a benchmark off of which to both gauge those scores against other similar applications that we're doing testing for, but also against yourself as you do additional builds. Cause it's always nice to kind of have an objective scoring for how you feel like things are performing before you get it to the end consumer.
[00:05:00.637] Kent Bye: In terms of the surveys, are you collecting both quantitative and qualitative feedback, and how are you digesting that?
[00:05:08.479] Geoff Skow: Yeah, I had mentioned some of the things we asked at the top, but yeah, so some of it's very structured, kind of 1 to 10. So on a scale of 1 to 10, rate the menu, the controls, rate the auditory experience, rate your level of comfort, rate your overall experience, some NPS or net promoter score type scores, like would you play this title again? Would you recommend it to a friend? But then oftentimes the best feedback that we deliver is the free form kind, where especially our panel, having played often as many Rift or Gear VR titles as anybody, you know, just as much as anyone here at Kinect, is the answers to questions like, if you had a magic wand, how would you improve this experience? Or what elements of the gameplay frustrated you? They're really candid and thorough in the answers that they give. So, to the extent that we're able to help people make better games with that feedback, I know our panel loves to give the feedback, and I think it's, for the people that we've worked with, it's been really helpful.
[00:06:03.484] Kent Bye: And so if there are virtual reality enthusiasts out there that are interested in being on the panel and providing more feedback, what's the process and are they getting paid?
[00:06:12.365] Geoff Skow: They are getting paid. Yeah, yeah. We pay a tester per test completed. And you can sign up on the site. It's fishbowlvr.com. We typically, you know, we do vet the panel. So we want you to be comfortable recording yourself, because I think it's very important to have both. It's critical that you not only see what someone's seeing in VR, but also seeing what they're doing with their body. And then there's also kind of a, there's a skill around vocalizing what you're feeling and thinking. So we'll provide some feedback there. But provided that all sounds good, we, we onboard you. So we've got again, over Over 250 people now, but yeah, the more the merrier.
[00:06:45.959] Kent Bye: Great. And so yeah, what's next for you guys?
[00:06:48.741] Geoff Skow: So we're currently going through the Techstars Seattle Accelerator Program. Everyone in Seattle has been incredible. We're based out of the Bay Area. So we've been up there. We're up there for a three-month stint. That ends at the end of October, where we'll have our demo day. And our goal is to raise some seed funding so that we can continue to do this. So that's the plan.
[00:07:08.258] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what that might be able to enable?
[00:07:14.541] Geoff Skow: I don't know if I mentioned Ready Player. How cool is it, A, that there's a signed Ernest Cline Ready Player One book in the gimme bag? I mean, my mind runs wild about the possibilities every time. Just doing the Toy Box demo today, I hadn't thought all that much about the importance of a second party. I mean, I've done all of this VR. I understand that social applications could be really cool, but that there was something really neat there that already my mind's kind of racing. But I think the opportunities, we're still just scratching the surface, right? A lot of it is just porting over, I think, concepts that we've seen elsewhere. But you have a chance to reimagine stuff that we haven't thought about doing ever.
[00:07:54.515] Kent Bye: So it's super exciting. Anything else that's left unsaid that you'd like to say?
[00:08:00.492] Geoff Skow: cool, I mean everyone from like Carl Krantz, everyone in the Bay Area has been super supportive of what we've been doing and it's just such a cool community and I'm glad to be a part of it and hope to be so for a long time.
[00:08:11.642] Kent Bye: Awesome, well thank you. Yeah, thank you. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voices of VR.
