There was a moment when Philipp Maas and Dominik Stockhausen realized that their sci-fi horror film would be a lot more intense and work better if was a virtual reality experience. So they took the opportunity to explore the language of virtual reality from the perspective of filmmakers who were trying to tell a story. In the end, Sonar makes a number of different innovations to the language of cinematic VR that I think are worth unpacking. I caught up with the two creators at Sundance New Frontier where Sonar featured within the mobile VR section.
LISTEN TO THE VOICES OF VR PODCAST
In a contrarian piece arguing that VR films are still a terrible experience, Mashable author Josh Dickey has some valid criticisms about the state of cinematic VR. However, I think it’s also shortsighted to completely blow off some of the positive innovations that could be found at Sundance. For example, while his last reason of “the storytelling language hasn’t been cracked yet” is still largely true, I would have to disagree about his conclusion that “no one has any answers yet” and that we are starting to see early signs of what works and what doesn’t.
Josh asks a number of open questions that I feel like were explored by Philipp and Dominik in Sonar including: “Should the camera move? Should the action be seen from the viewer’s first-person perspective, or some fixed point? Both? How fast can you fly around without getting sick? How close can you get to an object?”
Sonar is set within a cockpit to reduce vection and motion sickness when moving within around a scene. They actually switch from first-person to third-person in order to make editing more of a comfortable experience, and found that by starting the movie out in the third-person perspective that it makes it easier to switch between the two perspectives. They also find a nice pacing that moves a story forward by using fading to black while also moving the action all around the 360-degree space. They also use objects invading your personal space at the climax of their drama, and they’re interested in breaking VR comfort rules in order to physically affect the viewer for dramatic effect.
While it’s true that the language of VR is still very early in development, I would point to a lot of the innovations of Sonar that are informed by film but also transcend what film can do. Sonar feels like an experience that starts to really use the strengths of the VR medium to tell a story, and use art to create another world that’s only possible in VR.
Sonar is being released as a Gear VR app this week, it’s available for download on Oculus Share and it will be on Milk VR until February 12th, and it’s also on YouTube for Google Cardboard.
https://www.youtube.com/watch?v=QXTHELTL_9M
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:12.135] Dominik Stockhausen: I'm Dominik Stockhausen. I'm from Germany. I'm a CG artist and a director and I work in animated VR content.
[00:00:21.358] Philipp Maas: I'm Philipp Maas. I'm the co-director of Sonar, our virtual reality short film. And we made a six-minute science fiction horror film for the Samsung Gear VR headset. It gets shown at the New Frontier at Sundance. And it's basically about a deep, dark secret of an asteroid that emerges from the darkness of space. And we start to discover this dark secret while following a drone that tries to locate the signal. So we both made this shot at the Film Academy in 2014 and improved on it since then. So it has stereoscopic 60fps and 4k images and yeah.
[00:01:04.177] Kent Bye: Cool, so talk a bit about your process of what was it like and what did you learn about telling stories in VR?
[00:01:10.303] Dominik Stockhausen: Well, we actually originally set out to make like a traditional movie for traditional screens, but we kept the story evolving and the whole setting and the themes that we wanted to cover. We just realized at one point that actually all these factors would be so much more intense if we try to do it in virtual reality. So at just one point we like stopped all the pre-production that we have been doing so far and tried to create the whole story anew for virtual reality.
[00:01:39.579] Kent Bye: One of the things that I noticed about your experience in particular is that you have a lot of stylized editing which talk a bit about like what types of things you're doing to kind of create these seamless transitions in between different scenes.
[00:01:52.564] Philipp Maas: So one of the main focus we had was having nice transitions regarding the movement of objects so that people can pick up from one shot to the next. and having like traditional black dissolves gives a sense of, you know, you compress the time in VR as you do it in traditional films with dissolves and it helps people to stay in the experience basically. So another thing is that we basically keep switching the perspective in between the shots. And even more for people to not get out of the experience between those different perspectives. You need to have slower transitions and really know where people are looking to pick up from that again.
[00:02:36.267] Kent Bye: Yeah, that was the thing that I noticed about your piece in particular is that it had probably the most film-like feel in terms of the editing and the pacing. It just felt like it had a movement and trajectory that I haven't really seen in any other VR pieces so far. So talk about that process of creating that pacing and timing.
[00:02:55.512] Dominik Stockhausen: Yeah, that was like the most prominent goal that we had from the very beginning just to really give it that cinematic feeling because when we started with the project all the content we found were like demos or little experiments and with the medium so we just wanted to like to bring our background as movie makers into it and create this whole cinematic feeling and We started out the whole thing actually way too fast. We had to just get used to what it was like to experience the story evolving in space rather than just on the screen. So we kept editing very quickly. In the very, very first stage, found out that actually we had to take all the timing and all the pacing back quite some notches to make it easier for the audience to actually dive into everything and have enough time to experience
[00:03:50.500] Kent Bye: Yeah and it's interesting the thing that you just said about like switching between like first and third person perspective and that I didn't actually notice that until you just said it but it kind of makes sense that if you want to have a little bit higher pace of editing that if you switch to the third person perspective establishing shot then you did you find that that was helping you kind of increase the pace and make it less disorienting for watching it?
[00:04:14.233] Philipp Maas: I would say, especially in the introduction sequence, it helps to not start out with a first-person view. So people don't get the feeling of themselves and their subjective view. So they should have a feeling of a cinematic introduction sequence. So they know that it can get edited and they know there is time that doesn't show up in VR, so it's not a continuous timeline. I think that helps a lot to start off with a third-person camera or a free-flying camera. And as you said, that was our goal for this piece because every other demo that were out there there were games and they didn't have the traditional structure of a movie. So that was really, really our goal to keep it that way and to lure people into the story first and then we start with the actual experience.
[00:05:02.944] Kent Bye: Yeah, I found my, there's this really interesting moment for me as I was watching your piece where I was just watching this probe kind of go up and looking at things and nothing was really happening interesting and then I look over to my left and I was like, oh wow, there's like a whole monitor here that I could like, pay attention to what's actually happening. I actually stopped the video and rewound it and then just like, okay, I'm going to watch this instead. It was kind of this interesting moment where I could kind of just imagine being in a spaceship where I could get this close up view by looking to the monitor on the left, or I could just look forward and kind of watch at a distance, but not really see anything. And you're kind of like, Have this moment where this tension between which one do you really pay attention to so to me? That was a really interesting VR type of experience that you couldn't really do in a film.
[00:05:46.784] Philipp Maas: So talk a bit about that scene We tried to set it up the way that people couldn't really see what's happening up there so it was staged and framed to force people to look around actually and I And also the monitor on the second screen was placed in a way that people could just see a little bit of it in their peripheral view. So if something is moving, they might catch it and they notice, oh, this is like the live feed of the drone filming. But with different headsets, we noticed that the field of view changes a lot and we staged it for the DK-1, which had more field of view. So in newer headsets, it's actually a little bit too far to the left. But yeah, I think the whole concept of second screens is interesting because you go down another level of immersion You see a screen and then another screen and you it draws you even more into the thing I think Was there any other scenes that you're particularly proud of that you felt like you were either pushing?
[00:06:46.728] Kent Bye: the language of VR storytelling forward or something that you felt was like super compelling and
[00:06:52.882] Dominik Stockhausen: I'd say the sequence that works best or leaves the highest impression by the audiences, by the feedback that we got, is the whole introduction, where we just try to basically open up the whole 360 space for the audience. So we really try to have them look in all the different directions and just to tell them from out of the movie that they're actually able to look anywhere they want and that every space can become important at one point. And I think another sequence that works pretty well with the medium itself is the very ending. where we have a lot of, I don't want to give away too much, but there are a lot of objects sort of intruding your personal space. And that really makes you feel very uncomfortable in that peak moment of the movie. And that is something that we only can achieve in virtual reality where we can really interact with the physical space of the audience around them and sort of deliver emotions through that. So I'd say that is like a sequence that requires this medium to be told.
[00:08:03.264] Kent Bye: I'm curious because you have filmmaking backgrounds, what type of other lessons or insights that virtual reality could learn from film?
[00:08:14.073] Philipp Maas: Virtual reality is a new medium, but I think to get people into the medium you can't just play with the first idea you have and the first, let's say, gag you can do in VR. It has to follow certain rules in the beginning because people are so used to movies. So I think it's a good way to keep this traditional structure while incorporating ideas that only work in VR. So I think like give it all a little bit more cinematic feeling and not too much interactivity because it's still a different thing for people to get active and to feel like they're in a game rather than being passive from the very beginning.
[00:08:58.142] Kent Bye: What do you mean by rules? What kind of filmmaking rules would be an example of breaking one of the rules, for example, or one of the rules that you should follow?
[00:09:07.107] Philipp Maas: I would say it's the basic structure. You need an introduction, a second act and a third act. It's a basic storytelling thing, and it has always worked in every other medium. I don't know why people don't set it up this way. I mean, it's very avant-garde what people are doing with the medium, but still, to get it to mainstream, I think you have to do that.
[00:09:28.559] Dominik Stockhausen: Yeah, what I'd like to add just to the question of what virtual reality can learn from movies is probably the thing that VR has kind of yet to find its own stories like movies have had so much time to like in perfecting telling stories with everything that movies can offer and I think in virtual reality it is necessary to try to come up with original ideas that implement the medium itself as a storytelling entity instead of just like adapting themes or stories for it. Yeah, so just to find the aspects that really make this medium unique
[00:10:09.985] Kent Bye: Yeah, I had this pretty extensive discussion with Devin Dolan from Zonetic Media, and he had this matrix of different types of VR experiences, and you kind of think of it as either you're a ghost or you're a character in the piece, or you have impact on the story or you don't. And so, you know, I think a lot of the traditional media has been you're a ghost, you're an observer, and you're just kind of watching what the director has put forth, which is kind of what sonar is in a lot of ways. You know, we're going to start to see a little bit more interactivity where maybe the character has impact, but it's not of consequence to the overall story. You just kind of choose your own experience. Or, you know, maybe you just feel like you're a ghost. You're not even a character that's addressed. You're just a voyeur. Then you have no impact. you may be an actual character, and you may actually change the total outcome of the entire film. So to me, there seems like those four different types of VR experiences that are going to be emerging. And what I wonder is if this traditional three-act structure is going to be universal amongst all of those different experiences, or if this is just going to be taken from the passive observer that has no impact, that that's just the kind of language that works really well.
[00:11:21.568] Philipp Maas: No, I think it's the very beginning to keep this traditional structure. But as you say, as the medium grows, there will be overlaps between all those different points, between presence, passive, active and interactive things. Yeah, we thought about it a lot, about self-awareness. Do you have a body or are you a ghost? I think it's good to start out with passive experiences because you can start dialing in basically the interactivity at some point. So you can make it interactive without implementing game mechanics. But if people now it's going to immerse them and they don't have to think about how things work and what they can earn or if there's a high score and you don't need a tutorial in the beginning to get what this experience is about. I think that helps a lot to draw people in and then further into the experience you can still play around with those subtle interactive things that you can only do in real time basically. but that really perfects the immersion in the end. But it has to be involved in the story.
[00:12:27.517] Kent Bye: And are there any VR experiences that you use for inspiration for seeing that this is a really great application of narrative storytelling within VR?
[00:12:37.715] Dominik Stockhausen: Well, actually for Sonar, most of our inspiration came from traditional movies, just with concerns like the whole feeling, that cinematic feeling that we wanted to deliver. So Alien and 2001 and all these sci-fi classics, those were really giving us the mood that we wanted to go for.
[00:13:00.088] Kent Bye: So what's next for you guys? What kind of pieces are you working on now?
[00:13:04.188] Philipp Maas: We are very busy with this project still so it's a small team and we try to keep it up and get it on the GVR store so that keeps us busy but we're constantly thinking about very early ideas for our next things that are probably one or two years ahead and for me at least it will be some sort of real-time engine and probably a room scale thing But still, it's a personal preference to do something more passive and it will definitely be a character-based piece. So I'm starting to put these pillars for me to know where the medium is going and then think about what's the best kind of story that I personally want to tell in that kind of space.
[00:13:48.424] Dominik Stockhausen: Yeah, pretty much the same for me. I'm working on a couple of ideas, just trying to get that story right for that medium.
[00:13:58.205] Kent Bye: So what type of experiences do you want to have in VR then?
[00:14:01.347] Dominik Stockhausen: I'd like to be emotionally engaged and yeah, maybe even irritated. I think a very, very interesting aspect is like we hear a lot of these guidelines and all these suggestions of how you should create a perfect immersion. But to me, I find it interesting to play with these aspects and to work the technological aspects into storytelling. So yeah, maybe breaking the whole concept of creating a flawless immersion and implementing that into the storytelling and really tell a story, not by showing the audience something, but by actually impacting their feelings in the moment and their emotions. And that's what I'm excited about.
[00:14:47.988] Kent Bye: Yeah, just to follow up on that there seems to be in the language of filmmaking ways of creating dramatic tension with how you're framing and cutting and just a lot of different techniques to be able to build the dramatic arc of the piece and so I've heard that like Chris Milk and the Evolution of Verse kind of like did some acceleration for example as you're going up in order to create this kind of dramatic tension and even though that's not recommended to go against these rules to be able to make people slightly uncomfortable to add to that. Is that the type of things you mean?
[00:15:19.991] Dominik Stockhausen: Yeah, absolutely, because that is like the power of that medium that you can actually make the audience physically feel something. And yeah, in order to achieve that, I think it's very interesting to just look at what is necessary from a technological point of view, and then just try to implement that into the storytelling.
[00:15:39.978] Philipp Maas: For me it's not a contradiction to move people although they're not really moving and even if that maybe introduces some kind of motion sickness which will only get better in the future and even if it does disconnect the people from their immersion if it helps the story it immerses them even more so like it's the other way around. You cannot just transport people into a space and keep them static and hope they're immersed in the story. You have to have the immersion in the story and then you can do whatever you want with the camera as long as it helps that kind of storytelling. So I think that's important to know you can do actually whatever you want and those ideas from film like generating tension and moving the camera or framing stuff We will all do all that and we are at some point. So there's no way that static a roller coaster like experiences Will be that immersive in the future even if you can walk around you have to think about movement actually and Disconnect people from the physical world outside to draw them into the virtual world Yeah, I know that it's recommended to have kind of a cockpit in order to reduce infection And so, you know, obviously the your piece you've been able to kind of reduce the motion sickness in that way.
[00:16:49.649] Kent Bye: I And so, finally, what do you see as kind of the ultimate potential of virtual reality and what that might be able to enable?
[00:16:58.102] Dominik Stockhausen: Well, to me, the most powerful thing about VR is just that very immediate connection that you have between the story and the audience, so that you're really able to tell a story in different ways than you can with movies. because you have that, not only that psychological aspect, but that physical aspect. I just think that it's a whole different experience. It becomes so physical and it becomes so real and so direct that I'm excited to see where things are going and what people come up with.
[00:17:35.109] Philipp Maas: I would really love to see some non-conscious interactivity with a VR piece like monitoring body functions and stuff like that in the future so that people can get into a virtual world that doesn't look anything like the reality. So I think animation is the perfect tool for that. having stylized worlds that are perfectly designed for VR and that have a new kind of even design language than traditional art. I think that's a space that no one has really created because it costs so much to do like real installations and production design and building sets and you can do all of that now in VR. and change it based on the user, basically, so they can interact with their environment. I think that's the greatest opportunity for VR storytelling. As long as it helps the story, you can interact with your environment.
[00:18:28.788] Kent Bye: Why do you think the stylized art is so important?
[00:18:31.390] Philipp Maas: Because it... That's a good question. I mean, art always has a certain intention to give people the mood or guide people through a painting, basically. It's with installations and art exhibitions. You guide the viewer or the audience through rooms and try to enclose them into a space that is only made for them for a certain period of time. And it has nothing to do with reality. It shows them at an exhibition something completely different and something that is connected. So I think with design and stylized environments you can really create a whole different set of feelings and invoke those feelings in people. I mean, you can go to deserts or to places on the world you've never been to with VR, but you cannot go to a piece of art from an artist that created this world and it's only available in VR.
[00:19:24.486] Dominik Stockhausen: Yeah, and I think that a lot of times an abstracted theme or message or whatever can even be way more impactful than just the mere representation of it, because it can work on so many different layers. I mean, we're surrounded by reality all the time, so why just reproduce that instead of finding an abstracted or a different way to delivering the topic or the issue.
[00:19:56.552] Kent Bye: Okay, great. Well, thank you so much.
[00:19:58.113] Philipp Maas: Yeah, thank you too. Yeah, thanks a lot.
[00:20:00.824] Kent Bye: And thank you for listening! If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.
