When Ted Schilowitz was looking for what to do after traveling the world as the first RED Camera employee, he happened upon an opportunity to serve as a futurist for 20th Century Fox for looking at how to use emerging technologies for storytelling. Over the past three years, he’s had a lot of early access to hardware from all of the major virtual and augmented reality companies ranging from Oculus, Valve, Sony, Google, Magic Leap, ODG, and Microsoft.
LISTEN TO THE VOICES OF VR PODCAST
He’s been exploring what’s possible with VR and AR, and he says that “the abilities of a new medium start to define the demands of a new medium.” He’s worked on a number of different VR experiments to discover how to best blend together narrative and interactivity within the context of these new “spatial mediums.” One of the first and most ambitious experiments was a half-hour long Martian VR experience that was one of the hottest tickets at Sundance. It integrated the D-BOX 4D effects chair and Oculus Touch controllers, and put you in the first-person perspective of many key scenes from The Martian movie.
I had a chance to catch up with Ted at VRLA where he told me the story of introducing VR and AR technologies to a lot of Hollywood studio executives and storytellers. He shares some of his favorite interactive narrative experiences ranging from Pearl to Valve’s Aperature Repair to The Gallery as well as polished interactive experiences like NVIDIA’s VR Funhouse and the Valve Lab demos. We talk about the balance between global and local agency in interactive narratives, what can be learned from storytelling in theme park rides, the emerging language of storytelling in VR, and what it takes to become a viable practitioner of these future technologies.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to The Voices of VR Podcast. So in today's episode, I'm going to be featuring the story of Ted Shilowitz, and he's a futurist at 20th Century Fox. So Ted has been involved with all the major VR and AR players over the last three years, ranging from Oculus and Valve, Sony, Google, Magic Leap, ODG, Microsoft. And he's been looking at the future of storytelling and interactive media when it comes to Hollywood studios and how they're going to start to integrate their storytelling techniques within a lot of these new emerging technologies. So I had a chance to sit down with Ted at VRLA where we talked about interactive storytelling, the future of narrative, and some of the best examples of interactive narrative that he's seen so far, including his Martian experience that he did for 20th Century Fox in collaboration with the virtual reality company. So that's what we'll be talking about on today's episode of the Voices of VR podcast. But first, a quick word from our sponsors. This is a paid sponsored ad by the Intel Core i7 processor. You might be asking, what's the CPU have to do with VR? Well, it processes all the game logic and multiplayer data, physics simulation and spatialized audio. It also calculates the positional tracking, which is only going to increase as more and more objects are tracked. It also runs all of your other PC apps that you may be running when you're within a virtualized desktop environment. And there's probably a lot of other things that it'll do in VR that we don't even know about yet. So Intel asked me to share my process, which is that I decided to future-proof my PC by selecting the Intel Core i7 processor. Today's episode is also brought to you by VR on the Lot. VR on the Lot is an education summit from the VR Society happening at Paramount Studios October 13th and 14th. More than 1,000 creators from Hollywood studios and over 40 VR companies will be sharing immersive storytelling best practices and industry analytics, as well as a VR Expo with the latest world premiere VR demos. This is going to be the can't miss networking event of the year with exclusive access to thought leaders of immersive entertainment. So purchase your tickets today while early bird pricing is still in effect at vronthelot.com So this interview with Ted happened at VRLA, which was happening at the Los Angeles Convention Center on August 5th and 6th. So with that, let's go ahead and dive right in.
[00:02:41.133] Ted Schilowitz: My name is Ted Shilowitz. I'm the futurist at 20th Century Fox. I've been doing that for about three years since I retired from a movie camera company called Red Digital Cinema where I was one of the first guys involved in that. I was the first employee and helped bring that camera to life and then decided after seven years of wonderful goodness and traveling around the world that it'd be time to do something new but had no idea that this is what it would end up being and then good friend of mine at 20th Century Fox who runs post-production also his name is Ted asked me to come and be the futurist at the movie studio which was this very odd job but they had enough foresight to say we want someone with some eyes and ears toward the future that will just kind of help us look around the next corner with no real defined description other than that. And I started looking around the next corners and through some interesting introductions got introduced to the head guys at Oculus super early on, well before the Facebook thing, you know, a good year plus before that when they were just starting to raise investment and building stuff in Orange County and we spent some time over there and I got really excited and I brought some of the other studio executives over and they got really excited then of course subsequently made connections with HTC and and Valve and Steam as that thing started going and of course deep connections with Sony and deep connections with Google and you know so forth and so on and before you knew it We were really enamored and embraced by what the future could hold in these headworn media. But also take a fairly broad view that headworn is not the only way to start to push out into the future. So simultaneously I'm involved with a company called Barco, a big movie projector company, and we're doing immersive cinema. And I'm the chief creative officer there, so I actually do sort of two things at once. And we did the first two movies, Maze Runner 1, Maze Runner 2, in an immersive cinema experience called Barco Escape. And now we have Star Trek Beyond running in a little over 30 theaters all around the world doing this immersive cinema thing. So lots of different tentacles that all sort of started from this essence of, let's try and figure out what the future could be from a big movie studio perspective.
[00:04:39.105] Kent Bye: Interesting. So both of those threads, with both the film as well as the Dome, seem to be emphasizing kind of a passive ghost. You're looking at an experience that you're not actually dynamically interacting with, especially with the Dome content. But coming from the film world, just talking to a number of different people about storytelling and the future of storytelling, having sort of a passive experience is one dimension, where story is really emphasized. And then on the other end, you have agency and interactivity, where it's more about your own sense of cultivating presence, right? It feels like there's this blend that's happening and tension almost between this interactivity and sort of passive. And coming from Hollywood and storytellers, they're used to having control of the frame, being able to direct attention through either their shots and really directing. And so maybe you could talk about like how you think about storytelling is evolving with kind of like this new emerging language that's happening with immersive technologies.
[00:05:32.863] Ted Schilowitz: Well, one of the things that I've sort of been thrown into by being so bullish about let's look at the future and let's figure things out is I tend to give a lot of presentations and speeches around the world and these things come up a lot. And one of the things I refer to is leaning forward or leaning back. And I also refer to the abilities of a new medium start to define the demands of a new medium, right? Now, anybody that's been in this for any length of time knows that none of this is actually new. that this kind of interactive, immersive concepts and experiences have been around for a long time, mostly living in sort of theme park environments and things where you do want to sort of interact with the story. It's not necessarily passive, but there are passive moments through it. There's a deep lineage to this and this new fervor around it, I think, is sometimes misconstrued by people that think, oh my gosh, this is a totally new thing. I'm like, it's not a new thing at all. This kind of entertainment has been thought about and executed by a lot of people in a lot of ways very successfully, but not really taken to a mass distribution in every home or every mobile device in the world. That's the fundamental difference, right? So when you go back to this idea of the abilities of the medium become potentially the demands of the medium. It's kind of a big cornerstone for me. So I refer to virtual reality and augmented reality as a spatial medium, meaning that when you create entertainment, when you create information, when you create any sort of experience, in a spatial medium, you have the ability to use that spatial medium, which is completely different than a flat screen, whether that's a giant movie screen or a plasma television or an old CRT tube television, which were essentially boxes with restricted amount of space that you would look at, right? Now you have a medium where you don't just have to look. you actually have the ability to engage if you choose to. And if the creators and developers of the material that will be going into those headsets choose to explore that, right? So if we were in virtual reality right now, you know, you have this microphone sitting right up against my face. I could potentially move it and it would probably go off. access audio, which I just did. So we did a little audio virtual reality there, right? I couldn't do that on a television, right? But I could do that if we were sitting with two headsets in a social VR experience. That is actually, to me, extraordinarily powerful. And when you look at the touch points of what I think make the most instrumental and transformative and powerful virtual reality storytelling moments, It's ones where the developers and creators are starting to embrace the idea of narrative plus interactivity plus spatial sense. So I'll give you a perfect example. And it's not the ones that we've done yet because I'll point out to those two. But one of my favorite experiences is something that Epic Games did called Bullet Train, which I presume you've done like I've done a number of times. It is so powerful because they've created space. You're in a subway train, then you go out into a subway station. They've created the ability to create a story arc without a traditional narrative, let's just watch a story play out. You actually participate in the story. You can pick up weapons, and a lot of these things tend to have a militaristic sort of shoot-em-up kind of theme to them because it's an obvious logical way to interact. But by far not the only way, which we can talk about that in a second. But in the world of bullet train, you're in a subway train, you learn how to navigate and use teleportation. You learn how to pick up a cell phone call. You learn that there is going to be some bad things that can potentially happen and you should be prepared to deal with that. Then you go out in the subway train, you can pick up different guns, you can transport around the different station areas. you can, as the bad guys, which are sort of robot-like, are firing bullets at you, you have the ability with the touch controllers to grab those bullets and throw them back to kill the bad guys. So it's a spatial, it's real. It's like, that's what, you know, you couldn't really do that, but if you could really do that, that would be something that would be really cool to do. And then later on in the experience, this big robot thing starts to shoot RPGs, you can grab and catch the RPG, and because they have thrust to them, you sort of hold onto them and you can turn them around and let them go and blow up the bad guy. To me, that's a new form of storytelling, which I think video game creators have been trying to do for a long time, but they're always restricted by the fact that they don't have spatial space. They just have flat space that they can sort of pretend to be spatial, and you navigate around with a controller. When suddenly your hands and your body get into the action, The dynamic of what makes it what we call virtual reality starts to become real, right? There's another piece that I saw, and I saw it in the Vive, it's by the ATAP, Google Storytelling Group. So this is a totally different vein, right? I talked about a first-person shooter just now. Now let's take it in a whole different direction. But just as powerful, in fact, even more powerful. So it's by the director of an Academy Award-winning short from the Pixar world, I believe, called Feast. Do you know what I'm talking about? So Patrick Osborne's Pearl? Yes. So they have it for cardboard, which I think is fine, but I actually got a chance to see it in the Vive. And in the Vive, they actually allowed the use of spatial. So you're in a car and a story plays out over time and space. And the car itself becomes a protagonist in the story over a period of time. And it's so wonderful and it was so meaningful to me that when I saw it I immediately had to go back and do it again to see the little nuggets that I missed or might have missed. And I was able to talk to one of the sound designers on the project and it was just an extraordinary special thing for me and I can't wait for other people to see that and absolutely see it in mobile if that's the only way to see it. have any chance as they start to proliferate this out into the world to see it in a headset that actually allows full spatial tracking and movement. So one of the things you can do as you're sitting in the car is you can pop your head out the sunroof. and look around because very often the car's not moving. It's actually static and it plays out a part of the story and time and space goes on as a little girl sort of grows up through the car. It was extremely emotional. I actually cried at the end of it the first time a little bit in the headset because I was moved by the storytelling and the sort of natural nuances of the grasping of the story. So I recommend people see it any way they can. It's extraordinary. And it sets the stage for some light interactivity. It's not designed to be interactive like bullet train is. It's just designed that if you're in a car and all the windows were open and the sunroof were open, what would you do? Well, you could, if it was a real car, you could pop your head out the sunroof and see that stuff might be going on. That's telling the story outside of the car and time and space happens is amazing. Audio spatial audio. And I think a lot of people. don't count audio enough in their equation of what makes virtual reality virtual reality. We hear a lot of things. You know, we're in a trade show environment right now at VRLA, so there's a lot of sort of spatial stuff going on, but we are very paying attention because we're tracking in a little microcosm world of sound. The more that people pay attention to sound, the better off we are. So now I can hop into one of our projects, which is the Martian VR, which will be hitting Oculus Vive and the Sony PSVR very soon. It became sort of a tastemaker thing. We had it at Sundance, we had it at CES, it became sort of a viral hit and probably one of the most ambitious pieces in VR so far, so we did this through the Fox Innovation Lab. It's almost 30 minutes in VR. You get to live the part of Mark Watney from The Martian. And you get to actually interact and do all the things that he would do on his way of hopefully getting rescued. But you get to play that part. So because of the spatial nature of VR, if you know anything about the movie or the book by Andy Weir, potatoes play a very big orchestral part in his survival. And you get to actually experience in sort of a kitschy sort of fun way. It's not nearly as serious as what happens in the movie. of actually manipulating potatoes, trying to get them in the right bucket, trying to keep the hab alive, and even exploding the hab. And then there are rescue moments where you're on your way, and they mirror what happens in the book and happens in the movie, but it's not exactly the same thing, where you have to sort of figure out how to use the plutonium, you have to figure out how to get the solar panels on to the rover, while there's a big Mars dust storm, which is very dangerous, coming, and you feel that spatially, and we use sound and visual. As you turn your head to the right, it's like, oh my gosh. this stuff's coming, and now I've got peril, I've got time, I've gotta figure this out. And that was actually an interesting story point, I can tell you a little back story. When we first created that moment in the VR piece, we didn't have that. We just had, we're gonna put the solar panels on, and then you're gonna drive away, you're gonna achieve the next mission. And part of what I felt was, it works really good, but there's no peril, there's no reason to do this fast. So in our creative meetings, I said, what if we add a dust storm? And now we have the ability to add sound and energy to that, and we have the ability to, create like the pellets of the Martian landscape banging on the side of your helmet, spatially with audio and visual. And suddenly it's like, holy shit, I gotta figure this out. And now it becomes story plus interactivity on a timeframe. That if you don't get it done in time, you're gonna be in trouble, right? And then as we move through, I won't give away all the secrets, but as we move through it, you get to drive the rover around to get to your rescue moment and eventually try and find your rescue moment. So Rob Stromberg and Ridley Scott worked very closely hand-in-hand. Rob directed the piece, Ridley executive produced, and had a very deep hand in all those creative moments that wanted to become the story. So now in the Fox Innovation Lab we're working on four or five new pieces that will come to light when we're able to tell you about them. A lot of them are based on our big movie IP and we continue to always try and push the envelope and figure out new ways to achieve that spatial sense of storytelling and not just that passive sense of storytelling.
[00:14:51.122] Kent Bye: Yeah, I just also had a chance to try the Vive experience of Patrick Osborne's Pearl, and it also was very emotional for me because it felt like the car actually kind of becomes a character within this whole experience, but they are also, because you have this fixed frame, they're able to do kind of more of a cinematic editing, which you don't tend to see a lot of within other VR experiences, and so it's kind of like this really great blend of cinematic storytelling, but also, like you said, you have this little bit of agency. Now, I would like to make the distinction between local agency and global agency, because local agency is that you're able to sit up and look out through the sunroof and kind of look around, but yet, you're not actually changing the outcome of the story. And I think that, also, if you think of theme park rides, you're also having a bit of a local agency, but you're not really able to really change the outcome of some of it.
[00:15:41.202] Ted Schilowitz: It's on rails. At the end of the day, it's a story that is tracking whether you do things with it or not. But everybody has a slightly different experience, which is great. To me, one of the original virtual reality moments in theme park lore is in Pirates of the Caribbean, as you're moving through the ride, and one of the pirates is cut off, sort of pirate garb on. and you see his leg dangling. And as the boat moves under, you get closer and closer to it, and you kind of get a sense of how realistic that leg is. And as you get closer to it, you see that the leg actually has hairs on it. And you have this little moment like, oh my God, that really looks like a real leg. And it's because you passed in a certain proximity to it that you got that, right? So that's spatial because it required moving through the experience to do it. So that's your point on sort of global interactivity, which is mostly passive, but has a feeling of interactivity. And I think Pearl did that extremely well.
[00:16:34.853] Kent Bye: Yeah, and because the thing that VR offers is this ability to actually do some global agency with artificial intelligence and kind of more branching narratives. But specifically with The Martian, I had a chance to try out the abbreviated 15-minute version and Sundance within the D-Box chair. And the thing that I was really struck with was that you're kind of really blending a lot of like video game components where it feels like, okay, is this a story or a video game? with kind of like traditional storytelling and sort of a mix of both of those. And so as you're kind of thinking about creating these interactive experiences, how do you draw the line between what's a video game and what's a story?
[00:17:08.967] Ted Schilowitz: Yeah, and for me personally, and I think the rest of the team at the lab too, I try not to actually draw a line. I try and look at what are the abilities and capabilities of all the tool sets that we have in our reach to try and pull this off in today's technology, knowing that today's technology is just a moment in time and it's going to improve vastly, especially as this market starts to cultivate and catch up and become something important, right? Which I think already has, and it will continue to do that. But in the world of the Martian VR, we used Epic, the game engine. We worked very closely with them to really sort of figure out what could we do that had some of the tenets and logic and choice-based, goal-based things that a game has, that a video game has. But it's told in more of a story arc way. And it's told in more of a way that when Rob created the story of Martian VR, that it's different than a video game. But it feels sort of like a video game. the kind of agency you have, as to your point, is somewhat global and somewhat personal. It's actually a blending of the two, right? Because certain things you definitely feel like you're influencing the outcome of, but underneath the covers, we're going to let you achieve that regardless of how good you are at it.
[00:18:21.503] Kent Bye: I would actually call that local agency, not global agency, because you really don't have a choice in some ways. You only have one outcome, really.
[00:18:28.450] Ted Schilowitz: And there are certain sections where you absolutely do have a choice in The Martian, and it does affect things. You know, what things that you throw and change will actually change some of the interior of that scene. But it won't really affect the entire us moving you through the story to achieve what happens at the end. which I think is actually like Pearl, and probably why it resonated so well with me, right? I think some of the stuff that the Penrose Studio group is doing is like that as well, where there's an endgame in mind, but you move through these stories, like the Rose and I and the other piece that he's doing, Alumet, I think it's called. Eugene is the guy's name, he's also a friend of mine. And I think that they're creating really interesting story worlds that you can kind of migrate and move around, which is exciting and interesting, but it's different than playing a video game. I think what Oculus Story Studio is doing is exploring those boundaries and figuring out how to connect those dots in different ways. I also think it'll be interesting to get your feedback on this too. Every time I hear people saying that there are limitations to the medium, that you can't really do cinematic cuts, that you can't specifically direct what you want the viewer to see, I find it completely not true. I find that you can absolutely do Almost anything you could do in a cinema language in the world of traditional cinema or traditional television But you can do it better. You can do it with more engagement points And it doesn't always have to be passive, but you can absolutely do cuts. And you can do quick cuts in virtual reality as long as you know what you're doing. You can guide people to a very focused point in time, which is what we did with The Martian, and you can craft that directly into what you want the viewer to achieve. And what I kind of like about that is even when I said all that, you still within the world of VR, just like you have in the world of reality, you can still make a choice to look away, to look at something different. Even with all the cues that the director or producer or craftsman gave you to pay attention to this moment, you may actually be absorbed by something else and you may want to go back and do another visit. to see what the director chose. So, like, the first time I saw Lost, which was one of the Oculus Story Studio things, the first time they brought me in to take a look at that, I was so enamored by how they created the moon shining through the big, sort of, sequoia forest. that I paid more attention to that in my first run of it than I did on the story arc itself, on what they wanted me to watch. And I talked to Sashka Asper and I asked, you know, is he okay with it? He's totally okay with that. He knew I was going to watch it again. And I was just so fantastically amazed by how well crafted the light coming down from the moonscape was that I missed probably 70% of what he wanted me to see. So is that a good thing or a bad thing? In my world, I think it's a great thing. Because I think it's one of the abilities of the medium. So why don't you embrace it? Don't fight against it. Know that, you know, if I was going to go out into the real world and something was happening, like, let's just say across the way here at this convention, there was something really engaging to my eyes. And I caught that for a minute, but then something else was more engaging to me and I moved over to that. And then I was going to go back to that. That's something that television can't really do, right? But VR allows you to do with aplomb. So I think it's an interesting area to discuss.
[00:21:38.319] Kent Bye: Yeah, my feedback on the editing is that in talking to Patrick Osborne, I had a chance to do an interview with him at SIGGRAPH, and I asked him about the editing and the cutting, because I said it did feel like probably the most fastly paced experience that I've seen in VR. And he actually said that he tried to export it into a 2D, and that it was super slow, that if you tried to do the same type of cutting pacing within 2D, it just feels like it drags on for too long. So the amount of breadth and spaciousness that you have to give those cuts is a lot longer cuts that you need to have within VR. The other thing that I'd say is that Sonic VR is probably one of the most innovative techniques in terms of using both first-person perspective and cutting into third-person perspective that allows you to do more aggressive cinematic things because you're going away from being an embodied perspective and you can do a little bit more camera movements and other things when you're kind of switching between first and third person perspective within editing and so that's that was another interesting approach that I've seen in terms of editing and and kind of creating a cinematic feel within a virtual reality experience but overall it feels like there's going to be other guidelines in terms of like you know, not moving, having too much motion, and wherever you end up looking at something that's moving, then the next cut has to kind of pick up from that. And so I know that Jessica Brillhardt talks about like circles, that she thinks about that, where it's sort of like circles within each other so that as you move from one scene to the next, you have kind of like this map of if you're looking at this, like there's an example where you're looking out a window, an actual trolley going up a mountain, most people are going to be looking out the window. So then the next cut, you're looking directly into the face of a horse. And so those types of, you know, languages in terms of depending on what you're looking at may kind of symbolically be connected to the next thing that you're looking at in the next scene.
[00:23:20.967] Ted Schilowitz: And I think just like with great craftsmen within the world of cinema, we're at the early stages of starting to see great craftsmen emerge in the world of virtual reality and augmented reality that really grasp the medium, that aren't afraid to take risks, that will put that risk tolerance at full throttle, and then throttle it back if they don't feel like it's working for them personally as they craft it, rather than playing it safe. Rather than sort of relying on, well everybody's told me I shouldn't move the camera. Everybody's told me I shouldn't do these kind of things within the engine, within a game engine. Everyone's told me X, Y, and Z. The more you can feel that you have artistic license and freedom to just try because the newness of the medium at its current state allows you to try without much damage to that, you can try and sometimes you'll discover something along the way. that becomes way more meaningful than you had thought. But I think a lot of what is happening now is you need the cycles for craftsmanship and it's just starting to emerge but largely gets ignored. People just try and rush things out to market, put it out as fast as they can and try and garnish an audience for it when you need to actually craft things. The Martian took a long time with a lot of people that put their energy into it, right? So I think I'm starting to see things like, you know, we talked about bullet train and things that these really obvious genres that sometimes there are special moments. There's a VR experience called hordes that I like a lot, which I don't know if you had a chance to do, but it's a zombie shooter, but they float you continuously at a slow pace through the, the architecture of these dungeons and different things. There's many levels and the physics are really good. Like they spend a lot of time on the physics and the, the choices of the weapons and so forth and so on. So it has a lot of sort of logical touch points to something like Bullet Train, but a completely different genre and a completely different way to go. It's simple but it's powerful and it's scary and it works really, really well. So I give them credit and kudos for not trying to say, well, I guess I just have to stay in one place and I'll bring the environments to you. What if you can sort of float and they probably, although I haven't talked to them specifically about it, they probably did a number of testing rounds on how fast or slow and what kind of geometry should you move in to minimize or try and eliminate any kind of nausea. and create this continual moving world that sets a new style of gaming and storytelling. And while I wouldn't call it any specific story that goes on, there is a story that gets created by the levels you play, by the backgrounds you see, and by your emotions through it. My friends that are doing a VR experience for a paranormal activity that is extremely ambitious, that is something I've done a few times and brought my family out to do, and there are real terror moments, but they're taking a very broad scope of it and they're trying to build it with some real scope and depth, which I think is really great.
[00:26:09.085] Kent Bye: I think it's a little bit of a double-edged sword there of innovation in terms of trying new things, but also respecting the kind of standard practices for how to create a comfortable VR experience in terms of how to minimize motion sickness. But then on the other hand, I've done an interview with Brent Litter, the director of Lawnmower Man, and he was on a panel talking about the big cultural shift that's going to have to happen from traditional filmmaking and thinking about things in 2D. And it's quite a different paradigm shift and almost like moving from storytelling to story worlds and actually kind of creating these interactive experiences where, in some ways, it's the video game developers that are more familiar and comfortable with this type of interactive storytelling. And so I kind of see this tension of the traditional film and kind of the old paradigm and then this new paradigm, but kind of bringing them together and really getting to collaborate. So I'm just curious to hear some of your thoughts on those cultural things that are happening between those communities.
[00:27:03.847] Ted Schilowitz: I would agree and I actually have a very positive view of that. So one of the things I've talked to many people about is the challenge I've always seen with console games, platform games, on trying to blend cinematic moments with interactivity. It always feels a little clunky, it always feels a little like it's not the right mechanics because you always know when you're sort of sitting back and watching something and waiting to interact versus the interaction. Now, over time, there are better and better examples. So if you look at something like Uncharted, or you look at Heavy Rain, or you look at some of these iconic video games that have blended story, Left 4 Dead, these things that actually found new ways to do that, different than a Call of Duty or a Grand Theft Auto, which is a little more obvious as to, now I'm laying back, now I'm going forward. They try and obfuscate the obviousness. But they've never really had the right spatial environment to do it. This goes back to me, that spatial environment. The minute I put on a VR headset, now years ago, in its very primitive form and then, you know, saw the abilities of where it could go, I immediately felt that the days of blending cinematic moments and interactive moments would actually come together in a much more naturalistic way, in a much more organic way, where you wouldn't feel the difference between now I'm trying to tell you a story moment versus now you better interact or you're going to die or bad things are going to happen or good things are going to happen. It just happens to work really, really well because you're able to tell a naturalistic story without having to break up the action. So in the world of the Martian... The interactive moments are part of the story. It carries you through the story, right? In the world of bullet train, in the world of hordes, in the world of Pearl, you don't feel like there's a break point, right? In my mind. So the medium itself becomes the success point of what can blend story and interactivity. And I think you're going to see things like there was a developer. You can download it on on Oculus. It's called Kismet. And it's a fortune teller world. It's interactive, but you're living inside this world, and story and interactivity play really well. Ubisoft showed us something about a campfire game thing, where they're working with really interesting mechanics and clever ways to interact and game simultaneously, which I thought worked really well. And I could give you lots and lots of examples. The Vive Lab, their sort of showcase piece that they put out, I think is an excellent example of being able to navigate into worlds. That magic shop where you bop around from place to place is sort of part of the Dota 2 world that they sort of brought into VR. So I think that world's great and there are five or six worlds that you can navigate in there and of course the portal experience, the robot repair experience is one of the, I think, iconic pieces that we talk about a lot because it brings you into a world, you interact, it's game-like but it's not a game and it tells a story arc in a fairly short period of time. And then the other really amazingly powerful moment is the robotic dog that happens to wander around and you think, OK, so what would I do if it was a real dog? Well, I'd pick up something to throw to the dog, like a ball or a bone, and he'd bring it back to me, which happens. And then he kind of tracks with you through the worlds that you go in. And the mechanics they built of picking up a sphere and putting that to your head and transporting to a world that way. That's a game mechanic, but it's not clunky anymore. It's not like this weird sort of, why are you taking me out of the game to do this? It's just part of the natural ability of what we would do in real life if we were sort of in a museum. If that was a real museum and there were these big globes and you could pick them up, and it would tell you how to navigate to the next thing, right? So I think all those things are really valuable. My friends at Cloudhead Games that did the gallery stuff, where they navigate you through a world and tell a story, where you're picking up clues and you have a backpack and you have to take things and listen to the tape in the little old-fashioned tape recorder. All those cues from the real world are extremely powerful. The Everest VR thing I think is incredible, using photogrammetry and bringing you into a world where it would be hard for you to get there, sort of like Mars. It's hard for you to get to Mars, so we could bring you to Mars, right? That Everest experience is so incredibly powerful to me, I can't wait for people to do it. The VR Funhaus that NVIDIA put out as sort of a test and provability of the power of their GPUs and what could be done with stuff is adorably fun and interesting, and you can see that it sets a world of You just play things. It's like going to a carnival in virtual reality. So I can give you lots and lots of them. But I think you get the point already that there are many, many things and some that I'm not really sort of reacting to right away. But I've got hundreds of more things because as part of the practitioner of the future, you have to actually do this stuff. You have to come to these conventions like VRLA, which has been a real lightning rod for people to figure things out. You have to put the same time and energy in that someone that works in the cinema business in a movie studio goes to screenings three, four times a week, interacts with people three, four times a week to actually start to learn what they're doing and help build things. I think that's important. You have to practice what you preach a little bit as best you can.
[00:32:00.789] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:32:08.335] Ted Schilowitz: So, I think the potential is actually pretty limitless. I think certainly the entertainment genre is powerful because of its economic force and the fact that a lot of people want to do it. But well beyond that, therapeutic reasons, educational reasons, the idea of taking people to places that they couldn't go if they were immobilized or in a hospital or in some sort of immobilized state is extraordinarily powerful. Education has tons of tentacles. To be able to take people to places, I often say, what if I could take you to the Great Wall of China without having to put you on a 15-hour plane flight? And the abilities to do that relate beyond sort of CGI. So a big part of what I'm working on and I'm involved in a number of companies on is volumetric capture, is the ability to put you where only CGI could do today. And the effort to make something photorealistic, which never, of course, gets truly photorealistic, is extremely hard. But if we can actually have cameras, which there's a company called Hype VR that I'm working with in a very, very early stage that we're involved in, they're raising capital and they're doing really interesting things. have a camera rig that can actually see the world, that you can actually be part of that spatial experience that I keep going back from over and over again. Today I can take you on a walk on the beach. That's not CGI, and it's not just photogrammetry with a bunch of still pictures that we glued together. It's actual live video. It's pretty powerful. But really early stage, so those are the things that set the stage for the next like three to five years. The other thing is the headsets themselves and what we can do with those headsets will get extraordinarily more valuable as they become more clear, as they become more lightweight. And one of the companies that Fox is deeply involved in there is a company called ODG that builds a augmented reality headset that we're very close to and working on what we hope will be amazing experiences. We also have a very close relationship with Magic Leap and we continue to see where they're headed in the future And I have a very big belief that they are on to something extraordinarily powerful that will set the stage for what's next.
[00:34:02.932] Kent Bye: I just had a chance to try the ODG at SIGGRAPH with OTOI with a lot of the digital light fields, and it seemed like a super high resolution. It seemed like a virtual reality, but it was actually an augmented reality. Has it passed through as well?
[00:34:14.881] Ted Schilowitz: It can be both. That's what's so interesting. The most important part is the resolution we can get on your face now. People are like, when will it get out of that day where it's still pixelated, when it's still just like this kind of fairly low-res experience? And I said, that day is coming, and it's coming faster than you think. It's why artificial environments in VR today, with Oculus and the Rift and Sony stuff, actually work better than true photorealistic environments. Because your brain can let it go if it's not built to be super photorealistic. But the minute it's built to be photorealistic, you need a better expectation of what the headset can do. And that's why what we're doing with ODG we think is so powerful. So you saw what Otoy, which is a talk about a forward-thinking company, is doing with this crazy high-res volumetric moment. And you can imagine where we're going with that, right? And how when we get to the next stages of that and get it out to the world, how powerful that can be. and no longer are you tethered to a computer to get a high-resolution experience. It just feels more like some sort of eyewear that you would put on.
[00:35:13.549] Kent Bye: I'm presuming that you've had a chance to try out some Magic Leap technology, and I'm curious if shooting photons directly into your eyeball, if you think that there's going to be limits with approaches like ODG or using LCD screens, and if you think that, like, in order to get this sort of real sense of tricking your mind into mixed reality, that we're going to need to have, like, digital light fields in this virtual retina display technology that Magic Leap is using or if there's going to be kind of an inroad towards what we can do with the existing cell phone screen technology and that it's going to reach a limit and we're going to have to move it to this other completely different technology stack.
[00:35:47.572] Ted Schilowitz: I mean, I think all things advance in a very robust way, right? I think what we're seeing with mobile phone technology as the displays get better, the optics get better, what Samsung continues to do to keep proving the envelope and what Google continues to do to keep proving that they can do more and they can be more advanced, fantastic. And I think you're going to see massive improvements there. And I think simultaneously to that, at the real cutting edge of optics, groups like Magic Leap, ODG, and a few others, Microsoft with their HoloLens project, are starting to push the other side of that envelope, right? And show that these are ways that we can actually get to a pixel-free universe, that we can actually bring objects into the real world that feel real. So I don't preclude one against the other. I actually think that they both have really strong viable paths to success and that what's important for us from a movie studio standpoint is to be open to all things and to be open to what makes the best sense here. What is economically viable, what is creatively viable, what will make people excited about things and continue their journey, you know, whether they're a developer or an end user. and what makes what can really happen next possible and what are the things that we as a movie studio and other developers and people with investment capital and people with new ideas need to keep pushing on and not be afraid that just because this is already working that we shouldn't do something completely different. So I actually don't sort of see it as a competitive environment. I see it as a landscape of technology and creativity that will create the best moments over the next five to ten years.
[00:37:20.150] Kent Bye: Awesome. Well, thank you so much, Ted. My pleasure. So that was Ted Shilowitz. He's the futurist at 20th Century Fox and looking at how to use all these new emerging technologies for the future of interactive narrative and storytelling. So I have a number of different takeaways about this interview is that, first of all, I just wanted to revisit this idea of local and global agency, since I think it was kind of a continuous thread that was going out throughout this entire interview. So one of the things that Ted said about the Martian experience is that we're going to let you achieve that no matter how good you are. So essentially what he's saying is that you have no real global agency in some of these experiences where you're able to actually do some things within the experience that then kind of pop up and flavor the experience in small ways, but there's nothing of significance in being able to actually control the outcome and direction of the narrative. In a lot of ways, it's just kind of a fixed linear path where if you were to do it over and over again, you'd experience the exact same thing. You could argue that even if you had like five different branching outcomes, that some of those would still kind of be like no real global agency. But I think you get a little bit more variation in terms of having a replayability of an experience that if you were to do it over and over again, the big question is, would some of your actions drastically change the outcome of what actually happens? And I think in most cases, the answer is no, it's going to be kind of a fixed linear path. And so that is what I mean by global agency is that you really don't have much global agency at all. But just by the very nature of VR, you have the ability to kind of look around and experience different aspects of an experience. And so in that ways, that's what Ted has been talking about when he's talking about that it's a spatial medium where you're able to allow people to control what they're looking at and what they're focusing on. And so I think the new thing that VR is enabling is that you're able to actually potentially take some of these small local agency interactions and completely drastically change the outcome. Ted was trying to make the point that there are theme park environments where you're interacting with the story and that there's a lot of these established practices for having these interactive narratives. But again, I would go back and kind of disagree with Ted in saying that while that may be true, it's not really giving you any real global agency as to deciding how the whole experience is going to play out. And I think that is the major difference between like a theme park ride and what you are able to do now with virtual reality, where you can have this real choose your own adventure, where you could send people off onto all these different specific customized outcomes for however they're interacting with the experience. I think one really good example of where you're getting a little bit more of this ability to exert your agency with an experience is with Rick and Morty Simulator by Alchemy Labs. Now, granted, even that experience, you're still on a fixed trajectory on Rails where it's a story. This was in the context of the five-minute demo. We'll kind of see how they end up overall structuring some of these experiences. But at least in that experience, you get the idea that you're able to interrupt and interact with just about everything within the experience. You know, when Ted was listing off a number of different experiences that he was really looking at, he mentioned the VR Funhaus from NVIDIA. that's not really necessarily a story at all you're just kind of in this highly dynamic interactive experience and so to me i kind of go back to my model of the four different types of presence where both the alchemy labs job simulator as well as the rick and morty simulator as well as the nvidia vr funhouse are really focusing on active presence where you're able to interact with everything in the room, you have a certain amount of physics that you're expecting, and you're able to kind of interrupt whatever is happening within the experience so that makes it feel like you have a little bit more control or that it's matching your expectations for your interaction and what is actually happening. I think a lot of times when you're in these VR narratives you kind of have to sit back and just watch whatever's happening and you have no real control as to actually engaging with these characters or you know if it was a real life situation and you were to try to interrupt or maybe not even pay attention to what they were saying then you'd get some sort of reaction. And I think in the absence of having a lot of those reactive behaviors within these NPCs and these experiences, then it can actually break plausibility and break presence. So I think it's one of these things where you're able to, in a 2D game, get away with a lot more of just having these non-convincing NPCs just spout off the narrative and story, whereas when you're in an interactive environment, you're kind of expecting a lot of these social cues and highly interruptible types of behavior. So I think that when Ted says the abilities of a new medium start to define the demands of a new medium, I think these are the things that I start to think about is the four different types of presence, whether it's active presence, where you have a lot of high agency, whether or not it's about embodiment, whether you're trying to invoke the virtual body ownership illusion and perhaps give some haptics or other things that really give you this deeper sense of sensory presence, but also just the sense of being in another place. And then the social presence is one that we didn't talk about a lot within this experience but I think that storytelling with other people and having kind of like this collaborative adventure I think is going to be a big part of what is the new affordances of VR that wasn't possible in any other medium before. So you're able to kind of take an adventure with other people where you're able to have the story emerge with the reactions of the people that are going through this tour and adventure in another world. And then finally the emotional presence is probably the type of presence that is most present within a lot of these narratives where it's trying to really hook you in emotionally and trying to evoke these different feelings of either empathy or just to have some more nuances of the emotional life of the characters and have you to really participate in that arc. So it really resonates with what Ted is saying about, you know, if you want to be a practitioner of the future, then you have to actually go out and do a lot of these different types of experiences. And for me, after doing a lot of these different VR experiences, I've started to see a lot of these different categories and types of presence. And by looking at these types of presence and start to be able to categorize things in different ways, and I What I'm really trying to get down to is trying to really define what the abilities of this new virtual reality medium are so that we can see what the demands of the new medium are. So I think that what a lot of people will find is that when they go into an immersive story and if they aren't able to participate in their agency in a way that they expect or they think is plausible, then it's going to start to break presence and it's not going to be as believable. I know that I just saw an experience at TechCrunch Disrupt, which was Westworld's HBO VR experience, where they had this kind of blend of where you're able to have a gun and shoot at things. But then when it came down to it, that was the only real way that you were able to kind of meaningfully interact with the experience. And so you're just kind of standing in an immersive environment receiving the story but yet without this ability to kind of interact and really test to see how much control you had over the situation then you kind of just have to sit back and it kind of turns into more of a theater performance where you're kind of standing on set with these actors that are performing but yet you can't really do anything to kind of change the course of any of the story that's happening. So I think it's a really interesting balance and challenge as we move forward and we try to see what works, what doesn't work, what are the abilities of this new medium that are going to define the demands of this new medium, as well as to try to blend these things of, like Ted says, this narrative and interactivity and this spatial sense of being transported into another world. So that's all that I have for today. I have just actually decided over the last couple of days that I'm going to, instead of going to the Steam Dev Days, I'm actually going to go to this VR on the Lot event that is sponsored by the VR Society. They are one of my sponsors and I was kind of conflicted as to whether or not to go to the Steam Dev Days or to go to VR on the Lot, but after talking to Chet and kind of hearing that they're not really wanting to have a lot of press or other interviews, I I feel like there's going to be just a lot more people that I'm going to be able to meet and talk to at this VR on the lot meeting that's going to be happening at October 13th and 14th at Paramount Studios. So looking forward to going there and just talking to a lot of people from Hollywood, a lot of storytellers that have been working in these other mediums of film and theme parks and other entertainment realms where they've really focused in on the craft of storytelling. And just looking forward to going there and learning more about how they're trying to discover and work with a lot of the affordances of VR, some of their experiments, as well as some of the kind of premiere experiences that I think are going to be premiering there. So definitely looking forward to checking that out and talking to lots of these big Hollywood players and real innovators when it comes to early explorations within the medium of VR and what's happening with storytelling. So that's all that I have for today. I'd like to just thank you for listening to the Voices of VR podcast. And if you would like to support the podcast, then please tell your friends, help spread the word, and become a donor at patreon.com slash Voices of VR.
