The Night Café is an amazingly surreal immersive VR experience of walking through a space that’s inspired by a painting by Vincent van Gogh. I speak with Mac Cauley about his inspiration and process to in creating The Night Café, which won both the Platinum and Community Choice awards in the experience category for the Oculus Mobile VR Jam 2015. It’s one of my favorite VR demos to show people in order to introduce them to VR because it really creates a sense of presence in a place that’s really only possible in VR. Mac talks about his design process for The Night Café (be sure to check out his two detailed behind-the-scenes blog posts #1 & #2), his experience of using Oculus Medium, as well as what he’s working on next.
LISTEN TO THE VOICES OF VR PODCAST
The Night Café: An Immersive Tribute to Vincent van Gogh is available for free on the Gear VR store as well on the Oculus Share website.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:12.017] Mac Cauley: My name is Mac Cauley. I actually created the Night Cafe experience, which is a tribute to Vincent van Gogh. This experience allows you to basically step into one of his paintings and explore it in a three-dimensional way. And there's several different paintings that I use as inspiration, but mainly his interior, The Night Cafe. It's a Gear VR experience. It's also on the DK2, but that was the idea.
[00:00:39.576] Kent Bye: Yeah, so maybe talk a little bit about the genesis of this idea, how you got the idea, and then what you had to do to actually execute it. Sure.
[00:00:49.051] Mac Cauley: Yeah, basically, I was just getting into the idea of developing for VR when I came up with the idea. And this is around the time that DK2 was just starting to come out. I already had my DK1 and I was really fascinated by the way it could put you into a virtual space in a very sort of tangible way. Just visually, you really felt like you're there. And the scale of objects and The ability to put you in a space with characters and see these animated characters right in front of you was really impressive to me. I have an animation background, so that was really cool too. The idea of being able to put myself in the same space as the characters I'm creating. So at the same time I was also working on a couple film ideas and I had this idea for a film about a fictional painter that was going to be heavily inspired by Van Gogh. And in the film you're going to progressively go more and more into his world and see the world as he sees it in this painterly way. And I was going to do that through special effects, CG effects. And then, you know, I started thinking about VR and how really this would work so well as an immersive experience that allows you in sort of a first-person way to experience that painted world. And so very quickly it moved from a film to a VR experience and I just saw all the possibilities with that. And so from there I just had to figure out how I wanted to translate that painted world into 3D and there was a few different challenges in that. But the main thing was kind of looking at the paintings and trying to be as faithful to those as possible. That meant creating the geometry in a way that sort of warps along with his brush strokes, making sure all the textures felt hand-painted and unique, and really giving the colors the vibrancy they needed. And I did that through several different texture painting and rendering and shader techniques.
[00:02:55.522] Kent Bye: Yeah, I think the Night Cafe for me is one of my favorite experiences to demo to people for the first time if they've never been in VR. One reason is because when you go into this experience, you're completely transported into another world, a world that's really only possible in VR. And you do get a real sense of proportion and scale and really being in another place. It can really feel transportive. Like you're going into this night cafe that is this surreal experience. And it kind of really blows people's minds. I've demoed it to hundreds of people now as I go to different meetups and groups and The other great thing about this experience to me is that I can be running something on the DK2 with the PC, something where I have to really manage it. But demoing the night cafe to me, I found is pretty self-service in the sense that I can give somebody some instructions and say, yeah, whenever you're done kind of exploring the space, why don't you just go ahead and pass it along to the next person and give them the instructions on the controls. The controls and how to use the trackpad, I think is the one thing that people just don't automatically understand or get. That you kind of have to tell them like, okay, if you want to move around, you have to do this. So the navigation scheme that you did for the night cafe is pretty innovative. Maybe you could talk a bit about, you know, how you came up with that.
[00:04:09.995] Mac Cauley: Sure. One of the goals was to create an experience that could be explored at your own leisure, and you didn't have to feel like there was a set end or where you'd have to leave the experience. So, yeah, I mean, part of that was designing a way to move around in a comfortable way, at least as comfortable as I felt like I could make it, considering the limitations of movement in VR. So, you know, initially I had thought of potentially using a gamepad. I thought I would need all the different input options that a gamepad offers. But as I developed the experience more, I realized I was really simplifying it down to a very just visual exploration of this space, and it became less about Partially due to time constraints, it became less about interacting with all different elements from it. There was going to be some character interactions and ability to interact with objects, but it really became more just moving around the environment. And with that, I kind of realized I didn't need that many input options, and that the touchpad, which comes with every Gear VR, would be a pretty good way to actually get people moving. So there's simple tap and hold to move forward, and then you can swipe down to crouch down or swipe up to stand up. That felt like enough to actually get people to see things from different viewpoints and really move them around. But there were a couple little tweaks I had to make to the speed of the movement and also to the acceleration, which is the main thing that I realized was causing some people to feel a little bit more nauseous, which is if you have acceleration when you're moving from that stopped position state to the full speed, and if there's a gradual acceleration, it actually triggers a little more stomach upset because Generally speaking, in real life, you perceive that acceleration, but when you're going at constant velocity, you're not actually aware that you're moving or not moving. And so I've seen this recommended a lot. It's basically switch from zero to 100% movement speed right away, and you just kind of get rid of most of that uncomfortable stomach feeling. So that was one of the main things that I realized I had to do. And because the movement speed, I decided to make it quite slow so that you can explore the environment in a way that allows you to just spend more time there. It wasn't a jarring thing to go from that zero to this slow speed.
[00:06:46.900] Kent Bye: Yeah. One thing that I noticed the first time that I experienced this is that I was actually sitting down and I get fairly motion sick. And I will say that when I was locomoting around the environment while sitting down, I did get a little confused because I got stuck in a corner one time and then I realized that I just have to turn my body. And sometimes if you're sitting down and kind of exploring the space, it can actually be a little bit more uncomfortable to kind of like really turn around. And I think that was part of the reason why I got a little more motion sick. I found if I have people stand up, it's a lot less inducing of motion sickness, even if they are susceptible. And even if they start to like walk in place with their knees, I found that it even reduces that even more and gives a little bit more comfortable feeling as if they're walking around.
[00:07:29.527] Mac Cauley: Yeah, that's actually pretty interesting. I've done that a little bit myself, just sort of intuitively seeing if that helps you feel like you're being more immersed in that environment. And it is interesting how, yeah, just sort of like moving your legs, bobbing your knees up and down can actually give you a little more sense of actually being there and having that physicality to it. But yeah, definitely there is the sort of issue of needing a 360 degree space that you can move around in. That was something that I did decide to go with locking it in so you have to physically turn your head to look the other direction. And I felt like that was going to give a better one-to-one connection with the space and the movement controls and hopefully eliminate motion sickness for certain people, which I think probably it does for certain people. But I know there are people that would like to just be able to sit in a chair and kind of look straight ahead and move wherever they want. And I'm even one of those. Sometimes when I'm playing a VR experience and I'm just like, well, I don't really feel like getting up and turning around. But, you know, it's a shorter experience, so I thought it would be worth experimenting with just making sure people are always rotating in the direction they want to move in. And so really the ideal situation, like you're saying, is yeah, definitely standing works well. A swivel chair I think works pretty well. But I think, yeah, the worst case is just sort of sitting forward and craning your neck around to try and turn around. That's not so comfortable.
[00:08:57.327] Kent Bye: Yeah, that's actually how I first experienced it. So I just remember kind of like doing some critiques of different experiences from the VR jam. And that was sort of my comment was that I didn't realize that that was part of the reason is that if I would stand up, it would be a lot less comfortable, but yeah, it definitely did get a little motion sick. Cause I am fairly sensitive and I noticed just that it was rated for comfortable for most on the gear VR store, the Oculus home gear VR store. So speaking of which you just released this. And so, you know, going from the VR jam, were there other things that you improved or changed in order to get it ready for kind of like a consumer release?
[00:09:33.464] Mac Cauley: Yeah, there was a lot of really under the hood small optimizations that didn't really affect the visuals very much. Like, for instance, I had to go from Unity 4 to Unity 5. Mainly, there's some optimizations there that were needed for Android 5 plus. And so there was that. There's, for instance, the Unity logo is not in stereo on Unity 4, which is kind of annoying. It's something that they really wanted for any Oculus Store releases. In addition to that, it's just, yeah, really just optimizing, making sure the textures have all the proper compression settings, which actually took a while because there's kind of some bugs with the Unity VR build process. that doesn't necessarily compress things the right way. And also making sure there's OpenGL settings that had to be optimized. But yeah, really the overall experience is very similar. It runs at a much better frame rate now. It was dipping down to the 30s in some parts of the VR Jam build. And now with the release build, it's pretty much a constant 55 to 60. So it always stays above that 55, which I think goes a long ways towards making it a more comfortable experience. Particularly when you're looking out the window and you're seeing the starry night with all this particle effects going on, that would really slow it down before. Now it's a very constant 59, 60 frames per second, which was nice.
[00:11:05.942] Kent Bye: Yeah, that's one thing that I would tell people is like, you got to go find the window and look out of it because that's, I think one of them are really compelling parts to see kind of the stereoscopic effects when you're looking out the window to be able to see the night. And maybe you could talk a little bit about your workflow, your process, like, you know, what does it take to kind of create a visual look like this in a VR experience?
[00:11:27.170] Mac Cauley: Sure. I sort of experimented with several different ways of achieving a painterly look, or what I thought would be the closest visual style to the paintings. And initially I thought I would probably go with The 3D geometry warped in the different ways of the way you see him in the paintings. It is sort of this warped perspective that you really look at his paintings. Not everything has these clean lines receding into the distance. They kind of go off in different ways. So achieving that with the geometry was basically building it and kind of using sculpting tools like ZBrush to sculpt it in ways that felt that way. But from there, it was really a decision of how I wanted to render the environment with shaders. The Gear VR obviously has more limitations than if I was on a desktop, but when I was experimenting just with not worrying about performance, I was looking at ways to use normal maps and even parallax mapping to show the brush strokes in a textured way and allow light to actually, lights in the scene that would reflect off these textured surfaces to give sort of this, it almost looked like a wet paint look, so as if everything was covered in sort of freshly painted brush strokes. And that was kind of cool, but it also looked a little tacky in the way that it almost looked like a cardboard box that was painted just recently. And it just kind of like, it didn't have the real wow factor I was looking for. So when I sort of stumbled upon this just flat shaded, there's no lights, it's unlit, it's just the textures, the pure color from the textures in the final frame with no lighting effects between shadows and highlights and whatnot. And so you're just getting the colors I painted on the textures directly in the rendered frames. And that really gives that direct colors from the palette from the paintings in the final renders, which I felt worked really well. And even though it's even a little less photorealistic in terms of the way lighting affects surfaces, it actually, I was surprised. I didn't know that it would work, but when I tried it, you still get this sense of 3D space and scale and actually seeing things in stereoscopic way still works basically just as well and really gave the results I was looking for.
[00:13:51.365] Kent Bye: Yeah, one of the more striking components of the experience for me is the lights and how the lights are kind of radiating this animated way of beaming out the light. It's just a really unique kind of thing that I've never seen anywhere else other than in that VR experience.
[00:14:07.097] Mac Cauley: Yeah, yeah, yeah. Those, I definitely spent some time with those. If you look at the actual painting that it's based off of, The Night Cafe, One of the key components is really, yeah, these radiating lights where you see these light waves sort of emanating from these lanterns that are hanging from the ceiling. And it's so impressive just in the painting alone and I really wanted to find a way to achieve that in some way. And I knew that animation was going to work well for those. It was going to be one more technique I could use to bring some life to the environment and not just have everything feel completely static. Characters was one way I did that. There's some sort of gently swaying curtains as another, but the lights are actually the key thing that kind of brings some life and movement into the scene. And so yeah, those are carefully tuned particle effects to give those brush strokes that sort of emanating look. And yeah, so that was a big thing for me.
[00:15:05.691] Kent Bye: And in terms of the shaders, did you have to do a custom shader or what was your process of kind of finding the right shader to be able to use?
[00:15:14.318] Mac Cauley: Right. Yeah, they're mostly actually stock unlit shaders in Unity. There's a couple custom tweaks for different areas of the environment. I needed some alpha cutout shaders, like the sunflowers, for instance, use alpha cutouts so that you can actually, I didn't have to actually define each petal in geometry. I could use textures to define the edges of the petals. And so that was alpha cutouts. Those are heavier performance, especially for gear. It actually really struggles with alpha cutout. So basically, I found certain shaders online that I could use that work better for that. But yeah, most of the shaders are all unlit, just sort of the stock unlit shader. And that just has no lighting influence whatsoever. It's just directly the texture. And yeah, so that was just a process of Really then it became more looking at the ways I was painting the textures, which was all done in Mudbox, just painting directly on the surfaces of the geometry digitally. And so I could then sort of use similar techniques, painting techniques, layering colors, pulling colors from the painting, and then just sort of layering them on in ways that I was trying to achieve those brushstroke patterns as best as possible.
[00:16:29.352] Kent Bye: Yeah. And it's interesting that this, this experience kind of originated from the idea of creating a story and the actual night cafe experiences that you're kind of just roaming around a space without much of a narrative or story. And I'm curious if you've thought a little bit more about how to blend that sense of agency to be able to explore a space, but yet have other different narrative components within it.
[00:16:53.192] Mac Cauley: Sure, yeah. I mean, that was sort of a big part of the sort of early development process. I was really thinking about adding more story elements and more interaction. And it became clear that I did want to focus on visuals first, just because I knew that was going to be a big part of the experience. But as far as story, actually, the DK2 version does have an additional portrait character sitting at one of the tables. And this was one of the characters, Portrait of Madame I think her name is, and she was going to be someone who you could sit next to and she might actually start talking to you and give you a little bit of narrative flavor, some story elements in there. And I wanted there to be a very dynamic way that she would move, so she would look at you depending on where you were sitting, and if you're not sitting near her, she's not going to start talking. So, you know, sort of very common techniques used in video games. But I think you can use them in ways that aren't necessarily telling a video game story. They could be telling, you know, a smaller story or a short story like a short film almost. So that was just going to be one example to just bring some more life and interaction into the environment is having one of the characters in the scene actually be talking to you. You might be able to nod yes or no to things that she's saying. And from there, I mean, I had ideas of a character walking into through the front door and there being a big dramatic introduction and sort of a scene playing out as almost if it was a sort of a really independent short film done with one scene as the entire film in one environment. I wanted to use motion capture with Kinects, which I did some experiments with. The Kinects weren't really capturing the fidelity and the precision of the motion as well as I wanted, even with the two Kinects that I had set up. And so I did end up going with basically keyframing everything by hand, and that also meant limiting the amount of animation I had time to include. Specifically, if I wanted to animate a whole scene, it would have taken a lot longer, but I do see a lot of potential in that area, whether it's motion capture or hand keyframed, is bringing characters into a scene, allowing them to carry out, play out a scene, and you can be standing there, and depending on where you're standing, especially with real-time animated environments like this, you could really change the dynamics of the scene quite a bit.
[00:19:19.364] Kent Bye: Yeah, and the Night Cafe was actually the winner of the mobile VR jam for the application and experience portion, and maybe you could talk a bit about that experience.
[00:19:30.560] Mac Cauley: Sure, yeah. I mean, that was a great surprise when that happened. I sort of was coming up with the idea early on when I heard the jam announced, and I thought it would be a perfect place that I could actually submit this experience, especially because they had the apps and experience category, which fit perfectly with what I was doing. When I submitted it, it really helped sort of nail down the deadlines, so I was meeting these milestones on time. And, you know, a lot of sleepless nights towards the end, but I was able to submit it and I was, I mean, winning was just, you know, amazing. I was super happy, super thankful for that. So actually it's allowed me to sort of really devote more of my time to my next VR project, which is a gear VR game and kind of gives me a little bit of resources to go and be a more full-time VR developer.
[00:20:25.827] Kent Bye: Yeah. What can you tell us about what you're working on next?
[00:20:28.784] Mac Cauley: Sure, it's a little less in the experience realm, more in the game realm. It's very early in development. I'm actually just doing prototypes this week as we speak. I'm doing a lot of different prototyping for seeing what's working and what's not. It's sort of an exploration of a haunted house that you are walking through and it's procedurally changing every time you play the experience and you're looking for different objects in order to basically exercise the ghost from the house and it's a game it's probably going to be played in 15-20 minute play sessions which I think is a kind of sweet spot for some of the early mobile VR stuff. And I wanted to be heavily focused on creating some really cool visual spaces to be in when I'm using some of the techniques I use in Night Cafe to sort of build out these stylized houses.
[00:21:23.213] Kent Bye: We both were at the Oculus Connect 2, which recently happened. And I saw that you had a chance to experience the Oculus Medium, which is like the 3D sculpting tool. And you did quite an extensive write-up on kind of your experience with that, given that you've had so much experience being a 3D modeler. So maybe you could kind of summarize some of the main takeaways and points that you had from experiencing being able to 3D model within a virtual reality space. Sure.
[00:21:49.070] Mac Cauley: Yeah, I mean, that's one of the things I'm most excited about with VR is sort of the content creation possibilities, both in terms of things like that medium where you're sculpting, and that's similar to Tilt Brush 2, I guess, which I haven't had a chance to try, but also with constructing scenes and pulling geometry and props and objects around, pushing walls out as you're in that space. That seems like it has a ton of potential for creators. But yeah, Medium was a really cool experience. I imagine it being similar to Tilt Brush, where you're actually able to, you have both your hands in that 3D space, they're tracked with a very high level of precision with the touch controllers, and you're able to actually make sculptures within that. I found it a little bit trickier to get the hang of the controls at first, because I am used to some of these sculpting packages on PC, like ZBrush and Mudbox, which the way that you control the brush along the surface of the sculpture is slightly different than you do in Medium. where the main difference is in ZBrush, you're generally pulling some of the mass out of the sculpture, whereas in Medium, you're actually just creating these almost tubes of geometry along the surface, and then you can kind of smooth them. It's a subtle difference, but it actually makes a big difference in terms of adjusting to it. I think both have a lot of potential, and I think even in Medium, they could probably use certain tools and tricks that ZBrush and Mudbox have to kind of enhance that artistic workflow and allow for a higher level of detail and fidelity with the sculptures. The one thing that I found that I really wanted that wasn't there with Medium was the ability to step around the sculpture I'm working on. Physically, in the real environment, take steps to, in the virtual environment, step around that sculpture as well, in a way that you might do if you were in a sculpting studio, instead of swiveling the sculpture towards you all the time, which works quite well in medium, but I think there would be more of a sense of being in a 3D space with your sculpture if you could take those steps. I know there's some limitations with tracking area with the Oculus. So that's something that I think in the future as that improves, that'll be really cool as well.
[00:24:15.774] Kent Bye: Yeah, it's interesting because I had not experienced tilt brush either when I first did the medium and then a week later I did tilt brush. And so I've had a chance to try both now. And I actually treated medium more like a tilt brush. I was painting in 360 degrees. I was painting stuff all around me. And I actually found a lot of limitations of doing 360 swivels. I don't think medium was actually designed given that the two optical cameras that are tracking you for the touch controllers were front facing. So you're really kind of meant to be facing in one direction, which kind of speaks to your point that you made just then, which is like in order to kind of walk around, that would require you to be able to be tracked with your back facing those two cameras, which is kind of not optimal because you start to lose tracking because of collusion. Because when I, as I was spinning around and drawing stuff, I was really kind of like unknowingly breaking it because I was losing tracking when I was doing that. When I did Tilt Brush, I didn't have that experience at all. I kind of did a similar kind of drawing circles around me in a sphere. And it is a lot much more of like, first of all, kind of if you think about just more of an illustrator 2D vector approach versus something that may be more 3D modeling. And so you're kind of like painting those 2D brushes. And so you can start to sculpt and construct 3D type of objects by you know, just kind of painting over that Intel brush, but it's less about kind of creating volumetric type of experiences and more of lines. And they just have a lot of really amazing brushes that are like you're painting light and you're painting rainbows and you're painting sparkles. So it's more of like, you feel awesome. Like you can just paint a little squiggle and you just feel like you're the best artist in the world. Uh, with medium, I felt like, God, I suck. Like I can't, and the thing about medium though, too, is that I could scale things up and down and I couldn't scale things in tilt brush. And so I was kind of not using it as it was meant to be. You know, I was kind of like, Oh, I want to prototype some objects in a near field and kind of do it small and then kind of blow it up into the large space to see what it would look like. But I think it's really meant to create 3d objects that you could eventually export and then use in a VR experience as a 3d asset.
[00:26:29.028] Mac Cauley: Right, right. Yeah, I think I did notice that there were some cool pressure sensitivity controls within Medium that allowed you to vary your brush strokes, the size of these tubes that you're creating in 3D space. So that was really cool. It was very sensitive, the sort of trigger that you'd use. So I had trouble sort of adjusting to that and modulating that. But yeah, the whole interface where you have sort of a palette on one side of tools on your left hand and your right hand, you can move to select those and other little techniques they use to switch between tools and different settings. It was very intuitive once you sort of understood how it worked. But yeah, I think that you're right. From what I've seen of Tilt Brush, I think I see all these really cool, almost more like drawings in 3D space rather than sculptures with these light waves and all these different ribbons of color and stuff, which which looks like a lot of fun.
[00:27:25.403] Kent Bye: Definitely. Oh, it's amazing. Yeah. I mean, I think those content creation tools are going to be a thing that's For what I see is going to really unlock people's creativity. And, you know, I'm not a 3d modeler, but I can imagine with enough time and medium, I could be able to create stuff that is a little bit more stylized than, you know, kind of the more angular symmetrical type of things that you might get out of an asset store. But I think that's part of the appeal of night cafes because it doesn't have that very symmetrical look to it. It does look very unique and looks more human in that way.
[00:27:57.157] Mac Cauley: Right. I'm really happy to hear you say that. That was one of my main goals, was to make every area of where you're looking in the environment, every small detail, feel unique and like it was handcrafted. Because in many ways it was, but I'm also using tricks to kind of change the way the perspective and the warped perspective is. But yeah, so between the textures and the geometry, that was a goal.
[00:28:23.043] Kent Bye: Great, and to kind of wrap things up here, I'm curious what you see as the ultimate potential of virtual reality and what that might be able to enable.
[00:28:32.921] Mac Cauley: Right. Yeah, I mean, you know, it's such a huge potential realm of possibilities with VR and being able to sort of simulate spaces to a high degree of realism and detail. And so I just think for me personally, I mean, one of the things I'm most drawn to is entering into worlds that I've sort of created and meeting characters that I've created as well and sort of having them you know, be able to interact in that space. And so I think from there just this idea of really the idea of the metaverse where you're in a world with everyone else having their own ability to affect the environment in these big ways and have their personal expression showcased within the environment. I think that's the true potential for me is just seeing that and a community developing around that.
[00:29:22.527] Kent Bye: Yeah, I think that a number of conversations I've had, you know, the tools that make it easy to create 3d environments and have them interconnected to each other and less of a closed walled garden way, but more of an open way where you could kind of be tool agnostic in terms of what kind of environments and virtual worlds are connected to what I think that's going to be a key thing to the metaverse in terms of being able to actually create something that that has that unique human crafted feel and really unlocking that human creativity.
[00:29:53.505] Mac Cauley: Right. Yeah. I mean, I think something like Second Life really touched on this, you know, over 10 years ago without VR. And I think, you know, that sort of reached a plateau because it wasn't as immersive as it really needed to be for a real metaverse. But like you're also saying, like, if you're tool agnostic, and you allow people to create with their tool of choice, and then you have these very immersive spaces, I think it's going to open up a lot of different possibilities.
[00:30:20.285] Kent Bye: Great. Awesome. Well, thank you so much for joining me today. Yeah, thanks, Ken. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
