Jessica Brillhart is the principle VR filmmaker at Google working with the Cardboard team to field test the latest 360-degree Jump camera technology. She was working at Google’s Creative Lab as a filmmaker when she was invited to start doing test shoots in VR. She quickly saw the power of the VR medium, and has continued to do experiments to learn the language of VR storytelling. She’s been sharing her insights through a series of essays on Medium, and I had a chance to catch up with her at Sundance to talk more about the language of cinematic VR & her thoughts on storytelling in VR.
LISTEN TO THE VOICES OF VR PODCAST
The first VR film that Jessica produced was called World Tour, where she traveled the globe trying to show the extremes of earth conditions and trying to capture the essence of the following Kurt Vonnegut quote:
Hello babies. Welcome to Earth. It’s hot in the summer and cold in the winter. It’s round and wet and crowded. On the outside, babies, you’ve got a hundred years here. There’s only one rule that I know of, babies – “God damn it, you’ve got to be kind.”
Jessica has continued to experiment and film around the world in order to better understand and help inform others about the emergent medium of virtual reality. Some of her essays on Medium are about the rebellious nature of VR users, who try to push the medium to it’s limit to break the experience. And she’s been exploring the nature of editing in VR where it’s less about cutting from frame to frame, and more about nesting worlds within each other.
She started to explore the nature of relationships within VR with her second VR film called Resonance, where you follow the violinist Tim Fain around various different sonic environments. There’s a series of doors and portals, and it turns into a bit of a ‘Where’s Waldo’ type of experience in trying to locate where the sound is coming from. It encourages you to look around the space and really notice the differences in how sound reverberates depending on the location:
Jessica is continuing to explore and experiment in VR and asking questions about how narrative is going to work in VR. What the right pacing for editing? How will timing, motion, location, and voice overs work together within a 3D landscape within VR? How does presence fit into the equation of passive cinematic experiences? How can you use the drive towards discovery to choreograph and orchestrate compelling VR experiences? And how will artificial intelligence eventually be incorporated into more dynamic and interactive VR experiences?
To keep up with Jessica’s progress on her cinematic VR experiments, then be sure to follow here on Twitter as well as on Medium.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.956] Jessica Brillhart: I'm Jessica Burlhart. I'm the principal filmmaker for VR at Google. I have a filmmaker background, mostly live action. So I've been filming a lot of live action stuff, mostly with the prototype for Jump, which first camera's going to be Odyssey with GoPro. And I've been using that just to film a lot of stuff, random stuff, working with really cool people to sort of go to different events in different parts of the world and film and see what emerges from what I film.
[00:00:38.291] Kent Bye: So how did you get into VR? How did you transition from being a filmmaker to now working at Google, kind of being the principal filmmaker at Google?
[00:00:45.869] Jessica Brillhart: So I've been a filmmaker at Google for six years. So before then I was working at the Creative Lab. I was doing a lot of films that were mostly meant to sort of being like the gateway between an ambassador between the people that make technology and the people that use it. And I was really passionate about that. I still am because there's something really beautiful about that relationship. There's something very powerfully human about technology that we miss because we use it as a tool but we don't necessarily see ourselves in it. But I've grown up with it and been involved with it for so long that to me it was a very important thing to break apart and try to make art and film about that stuff. The VR thing was a bit random. It was something I thought about for a while. I've known Chris Milk for a long time and Aaron Coleman for a long time and Sashka for a long time. So for us it was like, I knew they were doing stuff and I'm like, it might be something cool. And I got an email very kind of randomly from someone on the Now Jump team who said, hey, we built this camera, and we noticed you're a filmmaker that works here. Do you think you could come hang out with us and find some cool things to make with it? And it's very rare. I mean, maybe it happens more often in Google, but at that point, usually it goes through somebody else. And it was never directly at me. It was such a really awesome thing. But I was like, yeah, I always want to work more closely with engineers. Let's do this. So I went and I was worried that the camera would be like this thing with like five arms or something. You never know. You're like, OK, like people, you know, engineers making a camera. It's interesting. And the first time I saw it, I knew it was going to be something great, something special. And they did this. They filmed a test on it in the office. And it was of all the engineers sort of like smiling at it and be like, hey, you know, like just fooling around, just being themselves. And it was that one thing that I couldn't quite get like, you know, where it's like these engineers are human and they have wants and desires and dreams and passions and like feelings. And and here it was like I was hanging out with them. And it was the coolest thing out of all the things they shot. You know, it was just really cool to see where they thought it was important. And they hit the nail on that one immediately. And that's when I knew I was like, this is something there's something here. So then Clay Beauvoir, who, you know, is the VP of our team now. I mean, he has always been, but now he's like dedicated, which is great. was like, go, film something with this, make something with this. And so, I was like, okay, let's figure that out. So, I took the prototype rig with my producer, Nick Cadner, and we went all around the world, and I used prose from Kurt Vonnegut. It's called Hello, Babies, Welcome to Earth, just because I wanted some kind of spectrum of human experience. And I used that as like, so what's busy, what's hot, what's cold, what's wet? So we went to Iceland, Japan, Puerto Rico, you know, California, just in case something went wrong. We need to get more GoPros, or we need to get another rig. And we filmed what ended up becoming World Tour. So, you know, I was still at Creative Lab, and I wasn't really a fully member of Cardboard or the VR initiative. But once I made that, I started understanding all sorts of things. My brain began to rewire. I sort of discovered, like, what we could do to edit it, what editing was like in the space. And it was such a compressed amount of time, but I learned so much in that amount of time. I was hooked. I couldn't go back to just making you know, the films that I made before. There was something really powerful and something that I knew I needed to do. There was an evolution of the craft that I was doing and something that was really important. So I knew I had to do it. That was it. And I think they knew that I had to be involved with it as well. And so, yeah, then I switched paths and here I am, you know.
[00:03:57.557] Kent Bye: And so you've been writing some really amazing posts on Medium, kind of digesting some of your findings of what it means to edit and make films in VR. So what were some of the big insights that you've had so far that you've been sharing through that medium, but also just digesting all these experiences and takeaways about how to work with this medium?
[00:04:16.572] Jessica Brillhart: Yeah, I mean, it's been very helpful for me, also. Because, I mean, it's really tough to dilute a lot of that stuff into stuff that's, like, easily understandable. And one of my biggest things is to try to help other people understand the medium more as well. You know, I don't want to just keep this all for me. I shouldn't. And I think that we'll all make better content if we're much more communicative about what's actually happening here. One of the things I'm most passionate about is the editing part. Because for a long time, a lot of people were saying, you can't edit in VR and you shouldn't, you know. And at this point, that's sort of a buzz thing comment on it, and we can laugh about it a little bit, because you can. But it's a lot more about understanding what a viewer is going to do in a space, the probability, like how likely a person is going to do something in a space, and sort of the coordination of that. And being able to understand that each scene that we film is like its own little world of experience, and so editing is less frame-to-frame and more of these worlds that build upon each other. And that was actually a very simple understanding of like ultimately what the format's like has completely changed the way that I've seen the medium, and I still feel like It's really hard to find these insights and understand them and then try to do them all the time because I'm still set in certain filmmaker ways where I can't quite break it yet. I have to do it more and more and more to really teach myself to do these things right. I almost still do the other things wrong so that I can know that it's the right thing to do, if that makes any sense. But some of the other things are This idea of rebellion, which I actually learned just from the nature of talking to a lot of video game folks, you know, what people do as soon as they put this stuff on is they try to break it, right? And the first thing they do is they turn around. And then I started doing that too. And the idea that they're actually going to almost want to identify where you want them to look and then like be jerks about it and like look the other direction. Or they're just completely oblivious and they're like, what else is here, you know? And I think utilizing the entirety of the space and actually kind of using it both sides to communicate to each other, to communicate towards a common thing is important. Which led me to believe that because we have very limited control of what a person's going to look at, I mean we have some control but we can't guarantee anything, it's almost better to accept that the viewer is the person driving the storytelling or the story. And if that's the case, then it's more about the spirit of the story and less about the linearity of the story, or less about trying to get them to see what we want them to see. It's not going to happen. It's never going to be guaranteed, and we have to stop. We should talk about elements that can help. corral certain parts of the story or aspects of the story to make the whole thing an experience that is beneficial to the story, but the story has to work a lot harder than it's ever had to work. And that's our job, is to sort of figure out how we can get it to work for a viewer instead of having the viewer work for us. And I think that's something that was really important. And then I guess the first thing I always talked about was this idea of presence. And we actually have a panel that's coming up about that too, where we have some really great people who've worked in VR for a while who are going to talk more about that, which I'm really excited about because I actually want to learn from them. But this idea of actually being there and what that psychologically does to you and how a lot of this relates to the mind and how there are things about the mind that we don't know and don't completely understand. And a lot of that is evolutionary, how we've evolved over time, and it's also stuff that we've been conditioned to feel growing up based on social impact, based on things we've been taught in schools or the ways our parents raised us or whatever, experiences we've had on our own, what we bring to experiences. But yeah, fundamentally presence has a lot to do with VR and that affects a lot of how we experience it and how we take it in and I think understanding more about and crafting more for that is going to make VR better. It's going to allow people to make much better content down the road.
[00:07:45.743] Kent Bye: And one of the things that I found striking is, you know, talking about editing and pacing. And like, it seems like when we watch a film, we have a certain expectation that there's going to be cuts at a certain time if there's nothing dynamic happening. And yet in VR, it seems like you need to give a lot more spaciousness to let people like really settle in and then take this scene in.
[00:08:05.476] Jessica Brillhart: Yeah, I think it's a combination because both could happen, right? It could be shorter and could be before the expectation that it will cut or after. or at the exact time. And in film, like, Walter Murch is a huge hero of mine, and like, for him to talk about, you know, the blink being something that's like a mechanism of the mind, how we parse out information, actually, in the way that we take in the real world is, you know, it's how he edited, and how he found edits to work for him. And I think it's a really great, I mean, for VR, it's more directly linked to the mind, but the mind still makes that same parsing of information. So, But I think there's some way to use the uncomfortableness of before or after edit to work for your advantage or not. Like maybe cutting earlier from something, it tells a different story or it creates a different feeling or a different vibe that might actually be useful to the story. I don't know how. I haven't really experimented that much with it, but I imagine it to be the case. motion sickness as much as we try to avoid it could still be something useful in an experience, oddly. I mean, it's like that uncomfortability, that weirdness. It's also about like understanding what the format wants, but then also seeing what happens if you don't do it. But very like, you know, nuance that, like not make it so blatantly a mistake, but just use those mistakes to help craft something. But yeah, I think timing, yeah, of course, it's coordination, it's timing, it's motion, it's layers of experience. It's like, if I'm not paying attention, is the location something wonderful? Or maybe it's like, is the location telling its story? Is the voiceover telling its story? Is the music telling a story? Is the person in front of me telling the story? Because I'm going to pay attention to a mixture of different elements. And how are they harmoniously working together? How are they different? Why are they different? I mean, it's really seeing it as like a three-dimensional object instead of this really flat window.
[00:09:46.620] Kent Bye: Yeah, it's an interesting idea that no matter where someone may be looking at any particular scene, there's kind of like a theme that is consistent that you're kind of jumping from those scenes to scenes to tell a story, which to me is like a completely new and different way to try to tell a story. So as you think about that, How do you structure or think about, like, you know, for example, the world tour? Like, how did you kind of plan out, like, this is the intent and this is the story we're going to tell, and then this is how we executed it?
[00:10:15.010] Jessica Brillhart: So, world tour was actually more of a montage, I guess. I mean, it still is. It doesn't really tell any sort of story. You could probably extrapolate a couple of things from it, but, I mean, for me, it loosely follows that Vonnegut prose, you know, like, hello, babies, welcome to Earth, it's hot in summer and cold in the winter. At the end of the prose, he says, God damn it, you have to be kind. And the first shot is these rocks that surround you. So it's like, here's Earth. This is a very simple representation of what Earth is. And it goes through all the different elements, like hot, cold, wet, crowded. At the end, I like trip and fall. And if you catch it, and I try to put it right at the side of your peripheral view, if you follow the breadcrumbs and you get to the end, just to the left of where you're looking, I fall. It wasn't intentional. I was trying to throw a mini glacier in the ocean to save it, because I don't know why. I was so delirious. did it and I tripped backwards on a glacier and I just laid there for a long time because I was so tired. And my producer like puts his hand out to try to make, you know, he's like, come on, you gotta get up. And then it cuts on me like grabbing his hand. So that to me was like a very loose interpretation of Vonnegut's piece. And someone actually, someone saw it and was like, this is reminds me a lot of like that Vonnegut thing, bizarrely. And I was not, I had done a voiceover for it with the Vonnegut thing, I got an engineer to do it. and it didn't work, like there are all these things that I had presumed would work and they didn't work and the thing that worked was this really loose but well-intentioned edit where I was like it just had the spirit of that prose and it worked, which is crazy and it was wonderful to see that. But it was things like that where it was like trying to think of like where were points of interest in an experience, what was the ebb and flow, what was the vibe and finding that the next world would be able to sort of gently lead a person to it. So, one of my favorite parts of World Tour, which I was so happy when I realized it worked, was like you're in a tram, you're in Japan, you're on this tram, and you're climbing this mountain range, and you could look at one of the four, generally I was like, okay, most likely people are gonna look at one of the four windows. So, it cuts to a corral in Iceland, this horse stable actually, and if you're looking through the front and back windows, and it cuts, you're looking down or back through the corridor of the stable, If you look out the side windows, you're met with a horse face. And the funny thing is, people generally are looking through the front and the back, and then they hear a sound of a horse, and they turn right, and this horse is trying to get at you. And you just see this, like, oh, it's a horse. And it was one of their favorite scenes. And that and the tripping. They're just like, what happened to her? One of the things that I just remembered that I thought was really important was this idea in video games and even computer games where it's like you take the first five minutes or like the first level that you play is always the one that gets you acclimated. So it's something that I'm really I feel is really important to VR experience just like even the rocks even though they were like the first moment that was supposed to match with that prose it's also about like Everywhere you look, it's very calming. It's like, OK, this is stereo. There are rocks. They're coming at me. This is really nice. And you're letting it happen. And then the next shot, generally people don't move too much in VR at first, is the glacier climber who's behind you. And so then you're kind of forced to turn around to see, well, who is that? And the discovery element, the want of discovering something, the one of understanding what's going on, motivates you to turn. And suddenly you understand that it's 360, you can turn, there's audio. And then you're suddenly watching this glacier climber climb up this mountain. And then I know you're looking at him, because there's nothing else to look at or listen to. I know I can make a good guess. Let's say I don't know for sure. And then I can cut to something else that I think is important. And I just started, things started to click. It all became like this really intricate cipher that began to happen. And I still am continuously finding new things that people are noticing in it and be able to build other insights based on that.
[00:13:56.782] Kent Bye: Yeah, the thing that comes to mind is a couple of things. One is that, like, the description that you said, the intent of the story of the montage sequence is fairly abstract. And, you know, maybe one person in the world got it. But, you know, when you watch a film, you get it and people get the story. But with VR, it feels like this kind of weird, abstract poetry where how do you use this medium to actually tell the story?
[00:14:18.793] Jessica Brillhart: Yeah, yeah. No, I think that's something we need to accept. I think that that's something that's great because we all kind of, again, like people go into that experience with many different, like some people have gone to Iceland before and they're like, oh my God, it was here, but, and I'm here again, but it's so weird to be back. Like, you know, and or like, they've been on that same tram in Japan. But the second film that we made, so that was the first thing I did. I had, you know, to be very completely honest, I had no idea what I was doing. Like, I was just like, okay, we're just going to go to these places and I'm going to film based upon kind of roughly what I would do if I was making an actual film. But the other thing, Resonance, which is the second piece that I made, that I think we had a little bit more, I'm kind of adding different elements, like throwing something in to see what would happen. And then the second with Resonance, one of the things I wanted to do is see like, can I train someone to kind of follow and look for and seek out a character and kind of build a relationship with them such that in every scene they're kind of trying to figure out where he might be and then could he cut to something else and so with Resonance in the beginning, Tim Fain who's the violinist, you're kind of following him around and he's figuring out the song and then in sort of the second part He cuts to doors. Something really interesting about doors, right? And you kind of piece together, okay, like, he's walking through these doors, and then as you're moving from the door to him, you're almost like, in your mind, creating this, like, pan. It's almost like you're going on this journey with him, even though you're not necessarily watching him the entire time. and then other times he cuts to characters, other characters, and then suddenly he's gone, but he's there embodied as this song, and you're suddenly caught between, whoa, this is a piece of a story, or the room changes entirely, and it becomes more about how the song is affected, and then he's everywhere. It just sort of breaks itself into chaos a little bit, and I really like that. It kind of made me think, okay, well then maybe you can almost build these relationships with people, And then that is actually an element in VR that can allow you to experience the rest of the story because you want to be alongside this person. And that's an incredible thing. It wasn't like there was a voiceover telling you, like, I am this person, you must find me. It was like, visually, how do I do this? And how can sound and music motivate a person to do that as well? And there's something interesting psychologically going on there, and that's the stuff that I'm really interested in. It's like, what's the mind doing here? What are we trying to figure out here? What's interesting? And I think that's the stuff that I'm really excited about. And that's where narrative is going to come in, I think, is really trying to parse out, OK, this is how people really experience the world, and this is what they expect. That's the key to unlocking a lot of stuff. And it's stuff that we need to experiment with more in order to fully understand or grasp what's possible here.
[00:16:52.558] Kent Bye: I think the challenge with virtual reality is twofold. One is that first, the content creators have to create and structure a story in a certain way. And then the second is that the viewers have to learn how to watch virtual reality content. So even if you created this, then there has to be some sort of like, you know, we know how to watch a film. We know how to kind of like piece it together. But we don't really know how to either watch or even we may not even know how to create it yet.
[00:17:18.886] Jessica Brillhart: I know it's our job to help them. I mean, that's a plain fact. Like, we can't expect someone's gonna put, like, I can't expect my mom or my dad or any, I mean, that's even baseline. I can't expect anyone to just put on a headset and just do it. Like, it's like no video game person, I mean, some do, but, like, the best games, the ones that go down in history as being the best games are the ones that, like, really, at the beginning, tried to get you to kind of understand or grasp what was going on. Otherwise, no one would play it. Everyone would just be like, I don't understand this. I'm not going to bother with this. It all started somewhere with this understanding what the rules are. What is this universe that I'm in? And I think that there's a fine line between really overdoing the like, this is VR. Welcome to VR. Look everywhere. Look, it's a 360 environment. which is obviously not the right way, versus just like, I don't care what you think, this is what it is, and if you don't get it, I don't care. That obviously is the wrong thing too, but there's some really beautiful conversation that we can have with a viewer that doesn't over-explain it, but gives them clues that helps them discover it on their own, or seemingly on their own. And that's the cleverness, right? Where it's like, I can put the glacier climber behind you and suddenly you understand, you've been trained to understand that there's ambisonics, there's spatial audio there, and that you can turn. I didn't tell you you could turn. I didn't have a one minute explanation about what VR was, but it was enough that I used the idea that you like to discover something, that you're curious, to help you figure this out on your own. And that discovery thing is so important. It's such a huge part of the spirit of the format that I think, when tapped into more, will be the way to get people more used to the format without over-explaining it to them, without dumbing it down for them.
[00:18:57.882] Kent Bye: And I think with VR, we're experiencing something that we could never really experience before, unless we were blindfolded. So if I blindfolded you and took the blindfold off, you may have a tendency to want to, like, look all around, but we never actually do that. We kind of, like, have a way of taking in a space and we don't need to do that. But in VR, you're kind of thrown into a situation and you... almost have to take it all in but yet if you're trying to continually take it all in you may kind of miss what's happening and so I've kind of like gone through this evolution of just picking a part to look at and not trying to take it all in because it's just gonna be too much.
[00:19:32.359] Jessica Brillhart: Yeah I think you have to be very simple about it too right like you have to really like and I keep going back to merch but I love the guy and I think that like something you said was like you can't over complicate it. You just gotta think simply about it. You're asking a lot for someone to first of all put on a headset and trust you. And then trust the experience or want to be involved with the experience. So I think it's an understanding that there's someone that's going to be fundamentally connected to your experience when they put that thing on. It's kind of like, weirdly, them stepping into the film. Literally. I mean, they're there. They're a part of it. They don't necessarily be a character, but they're psychologically connecting themselves to it in a way where it's their experience now. And so it's about really being selfless. And you're inviting them in. You're chaperoning in. You're taking them through this thing. So it's, yeah, it's that understanding of that relationship that I think is going to be very helpful for creating really powerful experiences.
[00:20:29.017] Kent Bye: And virtual reality is a new communications medium, and it kind of feels like the Wild West of trying to figure it out. And there's an excitement there, but it's also like just a lot of learning and things that we don't know. And so for you, what do you think some of the biggest open questions about VR and 360 video and storytelling are?
[00:20:46.631] Jessica Brillhart: That's a really good question. I think, I mean, there's a lot of the open questions are like what narrative looks like. I think a lot of us are really kind of like from Felix and Paul to Chris and Aaron over at Verse to Sashka, Oculus, like, we're all kind of trying to be like, okay, well, what is what is this storytelling narrative thing, you know, and we're all trying to kind of figure out what that is. For me, it's more about basing upon, like, trying to get some of these experiments underway and really think about, like, what the mind does when it's there and what we can do to hopefully, you know, make that better. But yeah, like, what does narrative look like in the space? How much interactivity is useful and how much of it is maybe too distracting or too much? And, you know, one of the biggest questions I had, and I can't remember the guy's name, but there was some sort of conference where someone was like, well, why hasn't there been, like, a Citizen Kane in video games? And the conclusion he came to was there's no real tragedy. You always, like, respawn. You always sort of win. Like, there haven't been really games that harness the tragic elements. I think Red Dead did it with, you know, I'm not going to spoil it, but Red Dead Redemption has some tragic elements in it. But ultimately, you know, you're on top at the end of it. I think that, you know, why? Why is that? And I think there's something really interesting about bridging. It's not about games, it's not about film, but there's some interactivity, narrative type experience that is going to come to fruition. And, you know, something that was said earlier today by Aaron, and it's very true, we've had a hundred years to deal with film, figure out what films what the language of film is, what the best format is, what the best way to portray it is, or show it, even at a festival like this. And we're just at the beginning of something like this, of VR.
[00:22:25.960] Kent Bye: And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:22:33.102] Jessica Brillhart: Oh my gosh, yeah! I mean, the holodeck would be great, right? I mean, that's a thing. I think that, I don't really know, honestly. I think that, well, first of all, there won't be lines. I see that happening. I feel like we're gonna be able to have this stuff more close, like it'll be something that we have more by our side. I think the footprint will be a lot smaller, but that's more tech-related. You know, I feel like there's something about mixing AI and VR that I'm really excited about seeing. I'm in a really good spot at Google to really experiment with a lot of that, so that's something that I'm really excited about and something that I want to explore. But I think that that's the thing. There's going to be some interesting, because the things that we're finding out about VR, that's something I actually kind of want to write about, which I'm really excited. I just need the time. But it's like what AI has been figuring out, machine learning has been seeing, and what VR has been noticing. There's a lot of overlap, a lot of interesting overlap. And I think you're going to see a lot more projects that are going to be more involved with both of those things very soon.
[00:23:32.805] Kent Bye: OK, great. Well, thank you so much.
[00:23:34.206] Jessica Brillhart: Yeah, thank you. This was fun.
[00:23:36.351] Kent Bye: And thank you for listening! If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.
