Sony has been looking into virtual reality for a long time, and Sony Magic Labs’ Director Richard Marks tells me about this long history and the development of PlayStation VR. He helped to create the PlayStation Move, which is a 6 degree of freedom controller that enabled VR enthusiast developers within Sony a way to experiment with motion controllers within a VR experience. Richard’s division was in charge of the capabilities of high-end tracking technology and then creating prototypes for game developers to explore the game design possibilities. I caught up with him at the VRX conference where he shared more information on the 120 fps framerate, how the PlayStation VR tracks the headset and Move controllers, and what he sees as the vast potential of social gaming.
LISTEN TO THE VOICES OF VR PODCAST
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:12.198] Richard Marks: I'm Richard Marks. I work at PlayStation. We have a PlayStation VR coming out soon. And my group is currently working on looking at different interaction possibilities in using PlayStation VR. Previously, our group worked on the PlayStation Move and the DualShock 4 and tracking those things. So we're pretty familiar with tracking in general.
[00:00:32.997] Kent Bye: And it seems like at Sony that there was probably a long interest and research into virtual reality way before the Oculus Kickstarter really launched. And so from your perspective, when did Sony really start to investigate and get into researching virtual reality?
[00:00:48.810] Richard Marks: Yeah, I started in 1999 and the entire time I've been there, there have been people interested in virtual reality in the background, kind of. And then when the PlayStation Move launched, it gave all these different people a really easy way to experiment with VR. So a lot of different groups independently started making VR projects where they put a move on their head as a tracking system and then got some kind of display system. And some people were more thorough about it and were looking at optics and things like that. All of this energy got noticed by the company and said, you know, we need to consolidate some of this work. There's some great things going on and we should consolidate that. And that was all back in 2010 and end of 2010, beginning of 2011. So quite a long time ago. And also we felt like the PlayStation 4, you know, the graphics that would be needed to drive the experiences that we would want to make, it's kind of aligned best with PlayStation 4, not so possible to do on PS3.
[00:01:42.418] Kent Bye: And wasn't there also, I forget the name of it, but it seemed like there was some sort of 3D viewer, I don't know if it was from Sony, but the start of the meant-to-be-seen forums, there was a lot of people that were kind of playing around with that. Was that one of the technologies from Sony that was doing some stereoscopic experimentations before the initial Morpheus slash PlayStation VR?
[00:02:01.905] Richard Marks: Yeah, so in England, in particular, we had a lot of expertise in 3D games, like stereo gaming for stereo television sets. And so that technology is very overlapped with what you need for rendering VR stuff. And then in America, we had the tracking technology for Move, which is very overlapped with what tracking you would need for the head. So we had these complementary expertises that we combined. And then in Japan, we have a lot of expertise in display hardware and optics hardware. That's why it's a very global project for us, because we have expertise in each different part in different places.
[00:02:36.880] Kent Bye: And so when the Oculus Kickstarter went out and started to really blow up, where were you at at that moment in working on this type of stuff within Sony?
[00:02:44.766] Richard Marks: Well, we had already released PlayStation Move, so we were already very interested in what you could do in 3D interaction. We already knew what we were doing with our VR side of things. We're still not sure when our product would release at that point. You know, we're still figuring out and it was still in the R&D phase, I would guess you'd say. It was an official project, but it was still in the development phase. So it was exciting to see, you know, that energy happening there. It's a little bit frustrating not to get to talk about it at that point, but it was, I think it was good to see how energized the space got.
[00:03:18.319] Kent Bye: And one of the things that I've really noticed about the Sony VR demos is that there seems to be a very strong story element in trying to make this combination of the narrative of the experience, but also have it somewhat interactive. And so maybe you could talk about some of the things that you've seen experimented with in the different demos where you're trying to blend this combination of agency and narrative.
[00:03:40.102] Richard Marks: Right, I think there's different games that do that. Some do it very strongly, some do it not at all. You can have a very simple, spatial puzzle game that is just about doing stuff and there's no story to it at all. And that's fine, those are fun to do. Like a crossword has very little story to it or something. But on the other hand, maybe something like Journey, which is a very strong feeling like you're moving towards some kind of end story, yet you still feel like you're involved in what you do matters. That's kind of the holy grail for some people, is to create an experience like that. Or maybe like we were just talking next door, The Last of Us has that feeling, where it's a very strong storyline, but yet very interactive. Those are challenging because it's a difficult thing to tell a story and also give the player agency. But I think a lot of our game teams are interested in that and feel like pushing that art forward is a great thing for us to do as a company.
[00:04:33.957] Kent Bye: And so what type of things have you personally been working on moving this forward in terms of the R&D and what sort of big problems are left yet to be solved within the VR space?
[00:04:44.118] Richard Marks: I'm more of a technology person usually and we take our technology and see what experiences it might enable. So we don't actually make the end content, we make a lot of little prototypes. So we never do have a full narrative to anything that we work on. But we try to give those tools to the game developers so that they could make those things. And lately, you know, we've been looking at how you can interact with the virtual world in a way that makes sense. So how you can move locomotion through the world is an area that's still, I think, an interesting topic to explore. Kind of the interaction that you have and whether things like snap or how your hands, whether you render a body or not, whether your avatar is just these floating hands or you see your whole body. Those are all things that are interesting and still have technical challenges associated with them, so we're trying to see which ones of those are easy to solve and which ones are worth solving. Sometimes the way we do it is we kind of wizard of Oz it. We just assume that there is a solution and then say, if we have that solution, does it matter? Because we don't want to spend a whole lot of time trying to figure out something that if we solve it, it still just doesn't matter at all. So that's probably what our group does the most is try to figure out what it is that matters. And sometimes by not even solving the real technical problem, but just sidestepping it, making a prototype, seeing, oh, if you have that, it does matter a lot. OK, then let's figure it out now.
[00:06:04.860] Kent Bye: I see, so it's sort of like using additional peripherals or other tracking stuff just to kind of play out, see what the mechanics actually feel like and then that would drive the actual development of the technology?
[00:06:15.985] Richard Marks: Right, so like an example that we worked a lot on 3D cameras, which we never actually productized. But you know, we started with a $50,000 3D camera. And we saw all the things that could do and we knew that eventually that could be made affordable and maybe the specs wouldn't be good. So you can artificially dial the specs down after you have a very expensive one and see what a less expensive one would behave like and see how much you lose when you dial it down a little bit. And so those are the kinds of things we've done that. That's a case of where we ended up not productizing once we decided what we could do with it.
[00:06:50.096] Kent Bye: I see. And I noticed there's some sort of 3D camera on the top of the monitor over there. Is that also something that is required for the Move in order to work, or is that just a supplement for the demo that's playing here?
[00:07:01.541] Richard Marks: When Move launched, it was with PlayStation 3, and we had the PlayStation Eye camera, which is a mono, regular RGB camera. And it's possible to track the Move in 3D with just a mono camera. With PlayStation 4, we have a stereo RGB camera, so there's two RGB cameras, basically. and you can track the move better and also you can track the PlayStation VR in 3D with that. because you can triangulate using the two cameras. So whereas before we had to use a single camera and use the kind of size of objects to know how far away they are, with the stereo camera, you can triangulate. And that thing you saw is the regular PlayStation camera. So it's been on sale already. You know, when we launched PlayStation 4, it was available in the Playroom was one of the things that came pre-installed on the PlayStation 4 and you needed the camera in order to enjoy that.
[00:07:51.880] Kent Bye: Are there any lighting considerations, like bad lighting or sunlight is going to be terrible? What sort of conditions are optimal and what are the thresholds when it starts to break down and not perform as well?
[00:08:04.669] Richard Marks: Lighting is probably the biggest issue for computer vision and tracking with a camera. So, yeah, it's something that we spend a lot of time working on. With the original EyeToy, when we did it, we were trying to track motion of the body and then lighting was very, very important. And so one of the things that we decided at that point is in order to sidestep a lot of the lighting issues, we would put glowing markers like this glowing sphere on the move and the glowing lights that are on the headset. And so what that lets you do is you can lower the exposure way down of the camera so you can basically see in the dark and you look for these glowing lights. So for the PlayStation VR and move the thing that's the worst case is when you have Way too much light like the Sun shining on things the Sun is so bright that it overpowers everything else That's the worst case actually if you're in total darkness our system works great because there's no competing light I Say that's interesting
[00:08:56.967] Kent Bye: Yeah, so I guess the other aspect of that is that it seems like you're going to be designing for forward-facing virtuality experiences, sit-down. Or have you considered adding additional cameras to do full being able to turn around and being able to stand up? Or are you recommending that all PlayStation VR experiences are kind of sit-down experiences?
[00:09:18.632] Richard Marks: Sit-down is definitely the easiest to do. We have a single camera, like you said. The range of motion, the headset actually will track a full 360 because it has markers behind on the backside of it as well. But if you occlude the PlayStation moves, then those wouldn't track or the DualShock. So pretty much mostly forward facing, you can spin and look backwards and things like that, but mostly forward facing. and more standing or sitting, but not really moving around the space too much. It all does work based on camera-based tracking, so you have to be within the volume of the camera for anything to work. Depending on how far you are away from the camera, that's how much space you have. And it's actually pretty good. I mean, you can hold your arms out completely to the size and be pretty close to the camera. It's a very wide field of view. It's like 72 degrees horizontal field of view. So you can figure out what that means. But however far away you are from the camera, you have about that much space wide. But I'd say for us, we're definitely encouraging sitting down or maybe standing, but not moving too much.
[00:10:23.968] Kent Bye: Yeah, and of all the 6-degree-of-freedom controllers that I've used, I feel like the haptics that are in the Move controllers is probably the most immersive, and actually I think adds a lot to the experience in terms of giving that extra haptic feedback to increase that immersion. So maybe talk a bit about what you've seen people being able to do with the haptic feedback within the controllers, and then how it kind of is different from other solutions that are out there.
[00:10:48.027] Richard Marks: There's different kinds of haptic feedback, actually. So there's the rumble, which is kind of traditional feedback that's in the controllers, and the move has rumble feedback, so it can vibrate when you want it to, when the game developer wants it to. But then another kind of haptic feedback that's there is actually just this feedback that's like the springs built into the trigger and how much throw the trigger has. And I think the hardware guys that made the trigger and the move did a really great job. It feels very intuitive. When you squeeze it, it has a nice amount of throw to it and you can really kind of squeeze it the right amount that you want to squeeze. So that's very passive, but it matters a lot when you're holding it and it's like the ergonomics of it basically. And then the active haptics can be triggered spatially. So when you move to a certain spot or you do some event in the game, then it can, you know, rumble the thing for a short or long period of time. That also is a really important part. And it just takes a little bit of that in VR to really trigger your mind to believe something. And you can combine that with audio. And then so you can have a little tick feeling, but a kind of a bigger audio experience. And then you'll just kind of assume that it was all a big haptic experience when really the haptics was a very small part of it.
[00:12:00.758] Kent Bye: And what's been your experience of kind of playing VR experiences with a gamepad versus doing it with the Move controllers? Because Oculus isn't shipping with six-degree-of-freedom controllers, a lot of VR developers have to make a decision as to whether or not they're going to... The way it was put to me by Akrima El-Hassan was that you have to choose two out of three. Either innovative game design controls, support for the gamepad, or support for six-degree-of-freedom controllers. You can't do all three at the same time because of just the difference. So from your perspective in moving forward, what has been your experience of the difference between the gamepad experiences versus the move controllers?
[00:12:38.992] Richard Marks: I think it really depends on what game it is you want to make and what agency you want to give to the player. Some games are far better with the DualShock, and some games are far better with some kind of hand controllers. It's very much about what the game is supposed to be about. If it is a, I want to fly a spaceship through space, that feels a lot better usually with a controller, a more traditional controller, because it feels like it should be used to drive something. If you want to actually be able to see your hands and reach out and pick something up, then you would want something like that. And I really think that the game design, what it calls for, drives the choice of what controller you would make. It is a challenge to try to make a single game that works with both. I can see why developers would give you that feedback.
[00:13:22.377] Kent Bye: Yeah, and in terms of, you know, from your position of, like, innovating on the technology and then handing it over to the game developers to design stuff, what have been some, like, examples of things that you created that then enabled this new mechanic or gameplay?
[00:13:35.587] Richard Marks: I guess we've done, um... Well, like the PlayStation Move itself, and iToy, and PlayStation Eye, and the new PlayStation game. All those things are things that we work on. We experiment and see what kind of prototypes we could make with them. We show them to the game developers. Then we go one more step, and the game developers work with them, and then they feed back into our product groups what things should be changed and modified and extended. So all of those are examples. And PlayStation VR is another solid example where Early on we built some prototypes of it, and it's not just my group, we participate, but there's other research groups like I mentioned in London and Tokyo working on these things. And then we make prototypes, then we refine them by seeing what things that they would like to have different. So for example, having markers on the backside of the PlayStation VR, turned out to be important because people do want to rotate and look behind them and they don't want to see a tracking glitch when that happens or something. So those markers were, you know, something we're not sure would be important and it's decided, yeah, that's pretty important to have.
[00:14:39.867] Kent Bye: Now there's been some talk about like the frame rate in terms of 60 frames and then getting up to 120 frames per second. Is it some sort of trick that you're doing in order to get to 120 frames per second or is it able to do that natively?
[00:14:52.777] Richard Marks: The PlayStation 4 can just render native 120 frames per second rendering and the headset can display 120 frames per second. That's possible and there's no tricks. It just can all do that that fast. However, you know, there's a finite amount of graphics horsepower in any given system. And so if you want to do something where you need more graphics horsepower, you can instead render at 60 frames per second, and then you have twice as much time for your rendering. And then what we do in that case is something called reprojection, where it takes that thing that you've rendered at 60 and it outputs it But then the next 120th of a second later, it hasn't gotten a new rendering by then, but it will slightly adjust it based on what you've done with your head motion in order to show it to you again in a way that's basically corrected for however little much you might've moved your head during that time. And what it effectively does is it makes it feel like it's responding at 120 frames per second to your head motion. And so it feels smoother with your head. It feels a lot tighter connected to your head. It's not exactly the same as doing a full 120 native rendering, but it gives a lot of the benefits that you would get if you did relative to what's happening with your head.
[00:16:04.647] Kent Bye: Now there may be some experiences that you can't talk about, but I'm curious to hear about what have been some of the most compelling experiences that you've seen in VR.
[00:16:13.239] Richard Marks: No, I can talk about, I mean, I've gotten to see so many different things. I think I'm very biased maybe, but I still feel like if you have five minutes, our demo that we show often from the London heist is like the best intent, most intense five minutes you can have to give people a glimpse of what VR's promise is. But there's a lot of other kinds of experiences that are very different from that, that I think are amazing. Some of the creation experiences, like I've gotten to try tilt brush. I haven't gotten to try medium, but I'm sure it's somewhat similar. And those kind of feeling like I can make things in 3D, that's a really awesome experience. I just got to try CCP's kind of tech demo inside where you look down and you see your whole body. because they're using a Kinect blended with an Oculus and you can look down and see your whole body in VR. That's really a cool feeling and seeing other people walk past you as they go in this kind of 3D ghost-like way. I mean, I've played some of the kind of cockpit shooter games. You forget that you're not actually sitting in the cockpit of a spaceship for a while. And that's a pretty neat feeling when I've played those games a lot and they're very similar to games I might have already played, but you start to actually feel like you're sitting there and flying those things. That's a little bit different than you have other games. But, what else? Other ones is my interaction with other people in VR. Some of the social things are really powerful. One of the most powerful things I've ever had in VR was very low fidelity graphics at the time. It was with some 3D spatial audio and it was a kind of chat thing. And somebody just leaned their avatar really close to me and whispered in my ear. And that was like, you know, they're in my space right now. That's really overpowering. Like, I'm not sure I'm comfortable with you being that close. But of course they're physically not even close to me at all in the real world. And similarly with virtual characters, not even in a kind of social space, When you have some of these virtual characters move into your physical space, that can be a very powerful feeling for good or for bad. I mean, horror experiences are powerful because of that. And also, other experiences can be a very positive feeling when you have these little cute things running around. It's just good to be feeling like they're next to you.
[00:18:24.999] Kent Bye: Yeah, and one of the experiences that I think has been one of the most compelling for me has been the little dollhouse demo with the little tiny robots dance around has been one of the most compelling things that I've seen. It's just something about the magic of the sweet spot of VR with all that stereoscopic effects. But yeah, sort of you're surrounded in a circular room and you kind of lean in and each time you lean in it sort of triggers an animation. Is some of that being created in Japan from their design studios?
[00:18:52.920] Richard Marks: Right, that's from our Playroom VR team and they're in Japan and the designer of that team, he worked on iPad before that and he worked on the Playroom, which the Playroom is an augmented reality experience where you have little robots and you can interact directly with them. And I agree, I actually really like the near world VR, kind of like when it's surrounding you or close up to you. And the only thing, I love that experience and I love peeking in and looking like I can explore that world with my head, kind of. And I just want to pick them up and play with them though. It's like one step more that I want. And I actually, I'm looking forward to when somebody gives me kind of like a, maybe a toy box mixed together with tower defense game where I can build something and then turn it on and like watch it play out in front of me. think that will be a really cool VR experience like when it's just all your toys all piled around you.
[00:19:41.620] Kent Bye: Are there multiplayer capabilities within a PS4 and is that something that you expect to see like social games within the PlayStation VR?
[00:19:49.452] Richard Marks: Yeah, there's two different ways really that you can have social with PlayStation VR. You can be online together in a virtual space. And we have a game called Rigs, which is like a mech combat game where it's three players versus three players in a team based thing. And so that's a social VR experience. And then we have a very different kind of thing, Playroom VR, where one person is in VR and then the other people in the same room as you in the real world are not in VR and they're using the regular control scheme. And they play the game looking at the television set, and you're seeing what's on in VR. So it's kind of an asymmetric situation. One person has this VR view, and everyone else has a traditional 2D television view. And you can have a lot of cool game experiences with that. And so they're exploring that space.
[00:20:34.245] Kent Bye: Yeah, and I did have a chance to try out the London Heist demo for the first time at SVVRCon, and I actually had a really visceral reaction to it, that it felt like when I was in the game with my hands, you know, it was probably one of the earlier experiences of actually having my hands in a game and shooting somebody in VR, and having all this blood fly everywhere. And it was really disturbing to me, just because of my own sort of history of seeing different traumas in my life. I felt like I knew in my cognitive mind, my rational mind, that this wasn't real, but yet there was something in my deep self, my limbic brain, that really felt like I had just shot somebody and I was deeply disturbed. But I feel like with these types of games and first-person shooters, it's a kind of a sensitive area with virtual reality where something that may not be rated that high within a 2D experience, I think it's going to be much more intense within virtual reality. So as you develop these experiences, I guess what do you foresee is kind of how that is going to play out in terms of like the level of violence that people may be able to take within VR?
[00:21:37.050] Richard Marks: Yeah, I think that the rating board will probably take into account a little bit, whether it's in a VR situation or not. We've had a similar thing in the past with EyeToy. You know, we had very stylized animated characters, but you're using your own physical body to interact with them. And so it made people question, is this the same now? Because, I mean, normally this would be an everyone game, but because you're using your own physical body to interact with them, is that somehow changing the rules of what you think? And there's been a lot of studies of this stuff, actually, in an academic space of how VR can be used for psychological understanding of people and whether the same rules apply in VR as they do in the real world. So I think there's quite a bit of knowledge in that space that will get used.
[00:22:25.549] Kent Bye: Yeah, I also think there's a lot of stuff that we just don't know, partly because of the technology and the content that, you know, a lot of things that if you would have tried to put past like a university review board for psychological studies, it may be not passing. But since it's in a game context, it's sort of more available. So I think there's going to be a lot of experiences in virtual reality that people never had before. And we just sort of don't know. So it's something that we kind of have to look out for.
[00:22:51.684] Richard Marks: Somebody mentioned yesterday that it's very Powerful. It's a very visceral thing. I mean so it has a lot of Power, which is why it is such a interesting field to study I think and so finally what do you see is kind of the the ultimate potential of virtual reality and what it might be able to enable You know, it's I have pretty much agree with everyone else that it's going to expand beyond the initial markets I really believe that initially gaming will drive VR and will be very much the leader in the space because the people who make games are already very familiar with the technology and the tools that are required to make it, but I definitely see it expanding out into a lot of other areas. The first slide I had I listed the ones I could think of off the hat, but there's training and education and visualization and design and virtual travel and just being able to look at things and view things in a way that is spatially correct is something that we haven't had before. So just putting on a headset and looking at real estate, just something like that, and being able to see, oh, what does this house look like? Do I want to bother driving over there to see the house in real life? Those kind of things, I think, will happen once it becomes more ubiquitous. Great. Well, thank you. Thanks.
[00:24:08.322] Kent Bye: And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
