Tim Sweeney is the founder of Epic Games, which has been collaborating with Oculus over the last two years creating a number of different tech demos with their Unreal Engine 4. I had a chance to catch up with Tim at the VRX conference, and ask him about the evolution of VR at Epic, cinematic VR, whether Epic is working on a full VR game, and where he sees the biggest potential of VR is going to be.
LISTEN TO THE VOICES OF VR PODCAST
Tim says that after John Carmack and Palmer Luckey were showing some of the early demos of the Oculus Rift, that people at Epic Games started to investigate VR. He had a conversation with Michael Abrash, and then ran some numbers to see that there was a confluence of both the hardware being ready in combination with some design inspiration that pointed towards the time being now for VR to really take off.
Whenever Oculus has revealed either a new iteration of their head-mounted display or touch controllers, Epic has been there with a new tech demo that pushes the limits of what’s possible within VR. The most recent tech demo for the Oculus Touch controllers was the Bullet Train demo, which explores using motion controllers to have gun fights with some teleportation locomotion techniques.
When asked whether Epic was working on a release title for the Oculus Rift, Tim said that he is not able to reveal any secret projects. However, he said that if you look at the history of Epic when it comes to new technologies that they have a history of doing a number of different experimental tech demos that usually lead to a full game of some sorts. So it seems very likely that Epic is indeed working towards a full game based upon some of the Bullet Train game play dynamics.
Tim said that Epic has been experimenting with different gameplay modes, and that they’ve broken it down into three major categories.
- first-person and full immersion that feels as if you’re there, but it’s also the most constraining due to motion sickness issues.
- Tabletop experiences like Couch Knights where they’ve only scratched the surface with input control and what’s may be possible with finger tracking with force feedback
- an experience where you’re surrounded by a “badass 270-degree monitor” that can deal with rapid motion.
The last gameplay mode is the one that’s the most vague and on where there hasn’t been a lot of public demos of it yet. Tim said that it’s feels more immersive than a screen, but perhaps is more tolerant to motion than the full first-person perspective type of immersion. Tim believes that VR will have a much larger dynamic range of experiences than the traditional video game market and screen-based media has so far supported.
The sequencer within Unreal Engine has also picked up a lot of traction within the cinematic VR community as Oculus Story Studio has adopted it as their primary toolset and have started to extend that toolset. One of the more incredible cinematic tech demos was when WETA Digital did the Smaug demo at GDC.
Finally, Tim sees that any chicken-and-egg dilemma that VR has faced so far has been due to a lack of clear business opportunity, and that we’re well past that point now. He’s looking forward to how the multi-user games will blend aspects of social media and move towards the visions of the metaverse. He cautions that a lot of sci-fi visions of the metaverse were created before either the world wide web or the gaming industry had fully taken off. He foresees that the metaverse will look a lot more like the current Balkanized gaming platform ecosystem than any unified Internet. VR will eventually be indistinguishable from reality, and that now is a great time to be a part of the industry.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.934] Tim Sweeney: I'm Tim Sweeney, the founder of Epic Games. We make the Unreal Engine, and we're doing all sorts of crazy stuff with VR nowadays. You may have seen the recent bullet train demo, which combined this new locomotion mechanism using teleportation to let you travel large distances across the world without the nausea, together with motion controller-based gunfights. We've been building VR experiences starting with some small tech demos and growing up from there for the last couple of years working really closely with Oculus and everyone else and really excited about it.
[00:00:43.894] Kent Bye: Yeah, maybe you can talk a bit about like at what point did you kind of see a contemporary VR experience and you knew that there was something there that was worth investing more energy into?
[00:00:55.722] Tim Sweeney: Well, a bunch of things came together all at once. You know, John Carmack was off doing VR experiences starting about three years ago or so. And Palmer Luckey was out doing the same thing. And they were starting to get some industry attention. And so I started digging in and looking at the underlying technical fundamentals that supported the whole idea and talked to Michael Abrash. You know, it became very clear looking at all the numbers that finally hardware had gotten to a point where VR was now not only possible but practical on a large scale with enormously high quality consumer experiences. You know, the whole VR idea has been circulating since the 1960s. It made a false start in the 1990s with some little 320 by 200, 30 frames per second of VR devices that attempted to be marketed. But, you know, it's one of these things like the smartphone revolution. It's like the idea sucked until a critical mass of hardware capabilities and great design sense came together to make it revolutionary.
[00:01:57.083] Kent Bye: Yeah, and right now it seems like in the early days enough that the language of VR is still being cultivated, you know, what makes a good VR experience, and you know, I know that there's been a number of different tech demos that Unreal has been working with and in cooperation with Oculus that a lot of times with these tech conferences they'll be revealing a lot of these new experiences, but what do you see is kind of like the thing that makes a real compelling virtual reality experience?
[00:02:22.271] Tim Sweeney: Well, we are still learning that as we go. You know, why did Digital put together this smog demo in Unreal Engine 4 using content that they created for the Hobbit movie? You know, they're the effects house that did all of the visual effects for the movie and they took those and put them together into this demo. Seeing that come together was really impressive. What was more interesting was watching how people react to it when they hadn't seen it at all and were encountering it for the first time. And we saw people who were really terrified and uncomfortable with this huge dragon slithering around them. We saw a wide range of reactions from people that indicated that they really felt they were there. And it wasn't like watching a movie or seeing a computer monitor. It was something entirely different, an entirely new type of experience. It comes in little phases here and there when you see a glimpse of the future. They're rare, but they're becoming more frequent every day now. Seeing the Oculus Henry demo for the first time, they've really been experimenting with how to build narrative techniques into VR experiences that go way beyond film. One of the points in the Henry demo, this little poor hedgehog, at one point the action kind of pauses and he looks over at you. the viewer like wherever you're standing wherever you're looking he's there like he glimpses at you and he has these sad eyes it's like they're kind of tearing up and you feel this incredible emotional connection with them that you really never experience in a movie. In a movie you're watching this static thing you can tell it's the same experience every time but here they're customizing the experience to the viewer. It's not a game, but it's more like a movie than a game. But I feel like we're really seeing the invention of a completely new form of media, from stage plays to radio to movies. And this is the next thing. And it's a different thing than those previous forms of media.
[00:04:09.970] Kent Bye: Yeah, and I'd say that, you know, because Oculus has gone and used the Unreal Engine for a lot of their demos, but also the Oculus Story Studio seems to have adopted the Unreal Engine for kind of like the most cutting-edge cinematic VR experiences. And maybe you can comment upon that in terms of some of the tool sets that they've been developing to make these types of cinematic VR experiences with the Unreal Engine.
[00:04:31.680] Tim Sweeney: Oh, yeah, sure. In Unreal Engine 4, we have a cinematic framework that's existed in the engine for a very long time. It was used for the Gears of War cinematics, which were revolutionary for their time, but it's grown kind of dated. And so in Unreal Engine 4, we've built a new framework for live motion editing called Sequencer. It's just like a nonlinear film editing environment that's used in Hollywood productions, but it's all real-time 3D. So you have the ability both to composite clips non-linearly and to completely control all the details of 3D shots, and to manage them as shots where the lighting might change from scene to scene, you know, according to the particular needs. So it's really a a system that's been designed from the beginning by folks who've been long-time participants in Hollywood, who come from that real cinematic background and are building the future right there. And then Akio Storia Studio adopted it, and they've been pushing it further and further, and really finding the fundamental new narrative techniques that work in VR. You know, if you watch kind of the history of movies from the 1920s to present, you see that a lot of the old narrative techniques really weren't developed until several decades into the medium, like the JumpCut or these other devices. And we're seeing now this entirely new repertoire of techniques being reinvented for VR. Some of those storytelling techniques, like the big camera zooms, they make you barf. So you can't do that, but you can do other things. And those other things turn out to be even more powerful, since everything is interactive and can incorporate the viewer into the action to some extent.
[00:06:01.746] Kent Bye: Yeah, and I think that's To kind of follow up on that, you know, when you're in a virtual reality experience, you do have a sense of embodied presence that your body treats it as a memory and as if you're there. And when you're in first person shooter situations, I've had a personal experience of kind of almost being triggered with PTSD, of having experienced certain trauma and then experiencing shooting someone for the first time. It's just something that's qualitatively different to be immersed with the environment. So I'm just curious, you know, because Epic has a history of first-person shooters, the bullet train's obviously one of the more elegant first-person shooters that I've seen in VR. If you see that, is that something that's going to continue to go down that route, or if you think it's going to be, you know, a little bit different? Maybe it's like we're replicating the 2D medium of video games, but yet maybe there's the new kind of genre of video games that we haven't even discovered yet.
[00:06:52.300] Tim Sweeney: Yeah, well, I think VR is going to prove to support a much wider dynamic range of products, if you want to call them that, than the video game market has historically, ranging from everything from largely non-interactive film-like storytelling experiences, all the way to fully immersive, fully interactive games, and a lot of things in between. You know, these tabletop game experiences where you're not immersed in the world, but you're sitting at a table manipulating some cool virtual stuff. using the motion controllers. I think it's got far more potential than any sort of screen-based entertainment before it because you can create anything that your brain is capable of perceiving there. I think it's definitely going to lead to a burst of creativity from game developers is its four different techniques. You know, the fully immersive shooter experiences that simulate being in a military gunfight should be expected to have the same sorts of characteristics as real-world combat because it does feel like you're there. And though your higher-level consciousness might know that the bullets aren't real, your brain's short-circuited control pads are telling you it's real. And it'll be really interesting to see how that evolves. I think you're definitely opening up a Pandora's box of possibilities. You know, a horrific game experience could actually be really bad and traumatizing there. So we have to be very considerate about how we approach this.
[00:08:13.058] Kent Bye: Yeah, and did you have a chance to actually experience the Valve room demo? I'm just curious, some of your first experiences of VR of this latest generation.
[00:08:21.845] Tim Sweeney: Oh, yeah, yeah, I did get the chance to try Valve's demo at GDC. It's really eye-opening as to the possibilities. When you feel like you're there and you can move around at least short distances, interacting with things with the motion controller, It feels completely real, and the latency and the fine-tuning of the hardware, both with the HTC product and Oculus, are so awesome that it really does feel immersive as if you're there. And what really struck me about Valve's Portal demo was just the combination of the technical realism of it with brilliant artistry. you know, at one point you open a drawer and there's some cutesy little characters in it, you know, in this portal game and it's just awesome to see that, like, such brilliance. It's the sort of breakthrough that I only recall seeing, you know, maybe a dozen times in my whole life.
[00:09:10.281] Kent Bye: Yeah, and I've definitely found that that tabletop and the miniaturized components of VR tends to have that highest level of stereoscopic effects. So I know Sony had, you know, a demo with like little dancing robots and also the Portal demo with those little characters. But, you know, I'm curious if you've seen anything within Epic that is starting to experiment with that near-field VR, that kind of miniaturized tabletop approach to doing gaming.
[00:09:33.465] Tim Sweeney: Yeah, you know, our first VR demo on Oculus was this Couch Knights demo where you sat down opposite another person in the real world and we recreated in the virtual environment a table and couch situation that was just the same as physical. So when you put on the headset, it's like, oh wow, I'm actually here. And then you're engaging in some tabletop gaming with another player. It's clear that that's going to be a big part of this thing. There are at least three major modes of gameplay that we've identified so far that all work really well. One is full immersion, as if you're there, which is the most visceral, but it's also the most constraining because, you know, if your motion in the environment doesn't agree with what your inner ear sense is, you start to get sick, and you have all sorts of secondary game design constraints that imposes. Then there are these tabletop experiences. I think we've really only scratched the surface of what's possible there. Like, just wait till we have The ability to do that with a much higher resolution and a much finer degree of input control. You know, motion controllers are a quantum leap above game joysticks and keyboards and mice, but really I feel like there's a much, much longer way to go when we get gloves or the ability to track individual hands and fingers with really precise detail. you know, potentially with forced feedback, it's going to feel like you're really interacting with those things. And then finally, we've done some experimentation and had some good results with the kind of experience where you put on the VR headset and we're really just surrounded by like a really badass 270 degree monitor. And so you can get away with more of the, you know, rapid motion and head turning and things that exist in a traditional shooter. And you're just getting a better and more immersive experience, but it's not trying to fool you into thinking that you're actually there. Those are just the possibilities we've tried so far. I think there's a lot more to go. And then you add an augmented reality and the potential to integrate computer images into real-world scenes, and that adds a whole new dimension to it.
[00:11:22.607] Kent Bye: And have your efforts within virtual reality been primarily to support the integration within the Unreal Engine, but also experimenting in collaboration with Oculus? Or are you also starting to develop some games that you would try to target for a launch date for the release windows for the headsets this coming year?
[00:11:43.946] Tim Sweeney: Ah, you're asking about secret projects. Well, I can't answer the questions about the secret projects, but if you look at how Epic has traditionally approached every generation of new hardware, from the Unreal Engine 3 generation to the UE4 generation, We've started with a relatively long period of experimentation where we show off increasingly elaborate tech demos, each one proving out several new techniques in isolation and a fairly self-contained experience. But those are all leading up to something much bigger, actual games which incorporate our learnings in a major way. We've now gone through three major, actually four major tech demos with VR so far, and we're incorporating those learnings into everything we do.
[00:12:29.085] Kent Bye: Yeah, and talking to Nick Whiting, the thing that I was really struck by was that it really felt like it was kind of his personal pet project to do integration within virtual reality, within the context of Epic, and that, you know, there was kind of an internal effort to kind of build support within Epic to put more resources into it. He was kind of like doing it on his own time, really, up until, at some point, E3 or something, that there was an award that was given. But maybe you could talk about, from your perspective, you know, how that kind of organically emerged from the grassroots from within Epic.
[00:12:59.848] Tim Sweeney: Yeah, you know, every big thing originally starts small. The first generation Unreal Engine started with me tinkering around with a little software render and grew from there. Unreal Engine 4's Epyx VR efforts began with Nick Whiting integrating some VR support into the engine, kind of as a funny experiment on the side, and it's grown into a much, much bigger project that now spans many people and many teams across the whole company. It's pioneering work that shows what was possible and as soon as the potential became clear, then there was the pylon stage where Epic was still like, okay, that's a kind of neat thing. I guess people can work on it if they really want to. And a whole bunch of people would join in and start contributing awesome ideas and work to the projects. And now it's developed into a major aspect of our engine strategy where Every major new feature and system we're building for Unreal Engine 4, we're thinking through VR really, really thoroughly. From user interface to 3D to photorealism to input, it's all a major part of this. And we recognize in 10 years from now, we're not going to have any of these legacy little 2D monitors sitting around our desks anymore. It's all going to be virtual reality and augmented reality. It's going to be a far, far more impressive experience than anything we're seeing now.
[00:14:15.944] Kent Bye: And in talking to Leila Ma, she was at AMD and doing a lot of really advanced rendering technologies and what she said was that it was a bit of a chicken and egg problem right now within the game engines and the hardware where she wants to do something as Extreme as say 16 GPUs in parallel, but yet there wouldn't be the support for an engine to be able to handle that many GPUs. So there's kind of like the engine may be waiting for the hardware to be there, but yet the hardware can't really do that until the engine is there. So as things move forward, how do you see some of these things playing out in terms of throwing more and more resources in order to give the level of visual fidelity that we really want to get to?
[00:14:54.566] Tim Sweeney: Well, when you see people Complaining about chicken and egg problems is usually the result of kind of a lack of opportunity or business opportunity to encourage people to actually solve the problems. I did not think we're going to see any of these chicken and egg problems with VR and augmented reality from here on. We realize where things are going and so does everybody else and we recognize that there's a demand for many tens of teraflops, maybe hundreds of teraflops, which today are not even achievable with four GPUs running in parallel. And so there's going to be a lot of effort across the whole industry to run incredibly high-end GPU configurations for advanced uses. If a filmmaker now is building a movie today that's going to be available to consumers in three years, then they should be working with hardware that's two or four times faster than today's hardware. That means moving to those advanced solutions. And if you look beyond the game industry, Like an architecture firm that's designing a building for the client and wants the client to have a completely photorealistic pre-visualization experience for previewing their building. There's a budget of tens of thousands of dollars available for hardware to make that sort of experience possible. And so there are a lot of areas across the industry where there will be complete justification for really high-end hardware solutions, solutions which go way beyond any sort of SLI, multi-GPU solution NVIDIA makes available today. It's not going to be your gamer rig, but the content creators, the enterprise users, the theme parks, all these experiences will have a demand for that, and that's going to define the leading edge, and that hardware will rapidly become consumerized from generation to generation.
[00:16:34.670] Kent Bye: And so what type of experiences do you want to have in virtual reality then?
[00:16:38.640] Tim Sweeney: Well, there's an infinitude of possibilities, right? My goal with Epic is to make technology available that lets teams everywhere do anything they want and have no limitations that hold them back from that. But, you know, I think a lot of the most exciting areas of use here are going to be around multi-user experiences of some sort. You know, how do you take the best of massively multiplayer games today and combine them with sort of social interaction that people have on social networks and, you know, add in elements of the metaverse that have been developed in fiction? to develop that sort of experience where it feels like you're there, but not only you are there, but everybody else is there too, and you're having a big experience together. I think that's going to be really the most game-changing of all of the things that are coming up. And nobody's really figured out the rule set yet. You can read the fiction references for the Metaverse, but you know, those were all written before the modern game industry, largely before the Internet developed into its modern form, and so I think they're not taking a lot of modern learnings into account and we have to take their interpretations of the future with a degree of skepticism and look more to what's happened in the game industry for guidance.
[00:17:49.288] Kent Bye: Are there things specifically that you think, from the internet, from the game industry, what are they missing or what type of things do you think that they didn't get, for example?
[00:17:57.868] Tim Sweeney: Well, you know, futurists have always mispredicted things in kind of funny utopian ways. Gosh, if you look at early visions of a hyperlinked system giving people access to all the information in the world, that's kind of what the internet is. But more than that, it's a way for people to share their lives with their friends, or to share stupid cat pictures, or do any number of other activities going from high-minded down to the downright seedy. And that's what the internet actually is. I think if we look at what virtual reality is versus what it's predicted to be, what we see in the real world is a very wide variety of different experiences. You know, you might have, in the 1940s, have predicted that someday we'd have bigger computers and that there would be a game that everybody in the world would be able to play. But the game industry isn't that. There are hundreds of popular games, thousands of games available entirely. Different players choose different experiences. There are different genres. There are a lot of platforms, an unfortunately large number of platforms, and things are somewhat balkanized and complicated, and things get kind of messy when they're translated to the real world. I think that will be the case with conceptualizations of the Metaverse as well.
[00:19:10.201] Kent Bye: And finally, what do you see as kind of the ultimate potential of virtual reality and what that might be able to enable?
[00:19:17.120] Tim Sweeney: Well, you know, ultimately it's going to be indistinguishable from reality. It will provide 3D entertainment experiences that you can't tell apart from the real world, and it will be able to scan in what's happening in the real world with such fidelity that it can be recreated anywhere else so anybody else can experience it. What we do with that is kind of up to us. I mean, think of what mankind could do if we could just exempt ourselves from the laws of physics. That's kind of what virtual reality will enable. Anything else that's left unsaid that you'd like to say? It's a cool time to be in the industry. I feel like everything we've done over the last two decades is leading up to this.
[00:19:52.393] Kent Bye: OK, great. Well, thank you. Yeah, thanks. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
