MONSTRORAMA is a mixed reality narrative game from Atlas V that transforms your living room into a monster museum. There is a lot of environmental storytelling that’s used to modulate your home, and you end up using your hands to draw mixed reality portals in order to battle both inner and outer monsters. I had a chance catch up with director Clement Deneux (previously directed Missing Pictures) at Tribeca Immersive to break down the design process, mixed reality considerations, future plans, and collaborating with Andy Serkis as a voice actor and narrator for the piece.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the potentials of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash Voices of VR. So continuing on my coverage of Tribeca Immersive, today's episode is on a mixed reality narrative game called Monsterama from Atlus 5. So this is an experience that will eventually be transforming your living room into a monster museum where you're able to explore around and see how your room's transformed and have these different interactions and explorations, learning about these different monsters and having different gameplay interactions with them as well. So this is a piece that explores mixed reality and hand tracking and the fusion of storytelling with interactive game elements as well. It's a piece that we were able to see the first episode, and so we're just interacting with a werewolf character, but it's setting the scene for a much broader experience that we start to dive into and get a bit of a sneak peek into how they're exploring the metaphoric uses of both the inner and outer monsters within us all. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Clement happened on Friday, June 9th, 2023 at Tribeca Immersive in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:32.091] Clement Deneux: I'm Clement Deneu, I'm director and I work with Atlus 5 on different pieces. And I am in Tribeca to show the first episode of Monstorama, which is a mixed reality game.
[00:01:44.927] Kent Bye: Great. Maybe you could give a bit more context as to your background and your journey into this space.
[00:01:49.730] Clement Deneux: So I'm a director coming from the traditional commercial video, music video. I made some documentaries working on feature in development. And I just collaborate with Antoine Carroll, one of the founder of Atlas 5. We made together a documentary about movies. And when he created the company, he asked me if I have any idea on VR, so I pitch him Missing Pictures. It's a VR documentary, adaptation of another documentary we made together. And we collaborate together to make the project, and it was my first VR project.
[00:02:26.859] Kent Bye: Great, so maybe you could give a bit more context for how Monster Aroma came about.
[00:02:31.322] Clement Deneux: So it was quite different, it was not my original idea. It was a project created by Emilie Valentin, Arnaud Collinard, and other Atlas V founders, and Antoine Carroll. This idea of Museum of Monsters, which you can visit in VR. So they tried to develop the thing for a few months and the project changed a little bit when we had access to a mixed reality device. So they asked me and Aurélien Noir and Alice Lepetit, a game designer, to work on a new version of the project in a mixed reality device. So we rewrote everything and we changed a few things to create a museum of monsters that you cannot visit, but the museum needs to visit you in a way. So we had to find some creative way to make the museum appear inside your own apartment, flat or wherever you will try the game. So it was kind of the technical challenge.
[00:03:31.015] Kent Bye: And so, was the Mixed Reality device that you had access to, was that a MetaQuest Pro, or what is it that you had access to?
[00:03:37.379] Clement Deneux: Yeah, it was not specifically this device, it was one of the devices that we considered to use for the project. We showed the project on the MetaQuest Pro in Tribeca, but it could be used in other devices in the same way. But yeah, the fact that the pass-through in MetaQuest 4 is in color, it's supposed to be better quality than the Quest 2, with a lot of improvements, was one of the key decisions to move from VR to MR, Mixed Reality, for this project.
[00:04:08.441] Kent Bye: Yeah, I feel like some of these projects that Monsterama could have been just a purely VR experience and been for the most part a similar experience because you're mostly in a room and you have these portals into these other worlds that are entering in so like you have this cut out of a wall where you're looking into a space. And so it's extending your existing reality. But I'm curious what you think the mixed reality component or the augmented reality component gives to this particular story for why you think it's more interesting to do it in your existing context rather than in a purely virtual reality room where you might have a little bit more freedom. So I'd love to hear about some of the design intentions for why the mixed reality component.
[00:04:49.822] Clement Deneux: Mostly because we wanted to do something about monsters and the mythology behind mythological creatures and the fact that the monsters are inside your place. There is something relevant about the fact that we wanted to talk about inner monsters, the fact that all the monsters are metaphorical for fears, for societal, psychological problems. But monster, it's always an incarnation of a fear or something like that. So the fact that we would be able to put the monster inside the homes of the players was something interesting for us. There is something in it like relieve the monster inside you. So it was kind of interesting to do it in Hammer. And also because we all have the monster in the closet or the monster in your bedroom when you were kids. this kind of very archetypal, very deep fear that follows you from childhood, things like that. When we thought about why do this in mixed reality and it kind of makes sense to do this project like that.
[00:05:57.825] Kent Bye: Yeah, so at the very beginning of the piece there's a selection wheel where there's lots of different monsters that are listed there, but it seemed like the only one choice it could make was to choose the werewolf path, but is the idea that eventually you would have all these different monsters laid out and then that you'd use the process of speaking and saying I want to bring up this monster and then it'd be able to identify that and then have an interaction with each of these monsters?
[00:06:22.648] Clement Deneux: Yeah, originally this idea was really more a technological challenge that we wanted to try the voice recognition feature of this headset to be able to speak to the headset, ask for a specific place to want to see, I want to see a specific monster and the AI responds to you, a specific text relative to what you ask. but we are in the first episode we did only have one monster for this but yeah in the future we I don't know we will keep this particular part of the story because the idea is to have like a theme park where you can visit different monsters and try to maybe free them but the idea was a mix between testing a new technological feature and also a way to show the characters of the main antagonist of the story which gives you a false choice of what monster you want to visit first, but in fact there is only one he wanted to show you and because this character is present to you first as a nice anchorman or something like that and you start to figure out that he's probably worse than that. So it was a way to introduce the character and his ambivalence.
[00:07:34.026] Kent Bye: Yeah, there's a mechanic that, in order to start the experience, you have to draw a circle connecting the dots with what feels a little bit like you're painting in Tilt Brush to create a circle. And there's a mechanic where that plays out later. But you're, in some ways, teaching people how to do the circle, which has this portaling effect that I think is a compelling aspect of augmented reality, because you are able to be immersed into your existing context. But you have these windows into these other worlds, and that you're using Mixed reality is a way of extending the walls and having these other dimensions, but this is also a way of teaching the user to use this mechanic. When I first did it, the tracking might have been not solid enough that I was able to draw like three quarters of the circle and I had to keep doing it, but Eventually I was able to get the technology to work, but to go through that process, it was teaching me to do that later in the experience. So I'd love to hear about that process to get that dialed in of working with that hand tracking as well as having that as a core mechanic of the piece.
[00:08:35.063] Clement Deneux: Yeah, we thought about different core gameplay of the experience and one of the ideas behind this gameplay was to let one of the metaphorical behind the pieces that you need to use your fear as a creative process. So the floating liquid that you can use to create portals or draw in a space like floating bubbles was like, okay, this is a representation of fear that you can experience as a visitor of this Museum of Monsters. And at some point the story told you that you can in fact use this bubble of fear as a weapon or as a creative process. So once you understand that, you figure out that you can draw portals to free the monsters. And later in the game, we will try to create different ways to use this freedom of fear in different ways relative to different monsters. So that's kind of the idea behind the gameplay.
[00:09:34.851] Kent Bye: Yeah, and here at Tribeca, there are these big, long, tall walls and then a big, wide, open space, and I felt like this piece works particularly well in this big, wide, open space because you end up being in an elevator where you're going down, but you're extending from the top of the walls on up, and so you're able to then extend the existing space. And I felt like the mixed reality component of that ends up being really quite compelling because you're able to also extend it up beyond where the walls were to create this more vastness of space architecturally. But I imagine if I was doing this at home, then it may be a little bit less plausible if I'm like seeing it cut through my ceilings. And so I'm curious to hear about that challenge of if this is meant to be an exhibition space and be able to use that tallness versus if you do that in someone's home, if it's going to be a little bit less plausible because people are going to know that there's ceilings there and it's not going to be as good as an effect.
[00:10:32.107] Clement Deneux: No, I think contrary that it works better in a home space where it's in fact smaller. Because one of the main technical challenge was to design an experience that can fit almost every room. We have like a minimum of space that we can't go smaller, but it works in almost every living room from people. So the main challenge was how to give the feeling to the audience or the player that his space is actually moving in space. So yeah, we went to the elevator so we enlarged the ceiling of any space so we can move up or down. in like a huge elevator. Once we fixed this effect, we realized that, yeah, it works quite well. The effect of moving inside and we had some fear about motion sickness or we had to look for the right speed of going down or going up. But once we fixed all these questions, we believe that it really works as the feeling of your own space, your own living room is actually moving and going down. I think it works even better in a place you know, because you will really feel the magical effect of mixed reality that you have a window and suddenly it's not a window anymore, it's a frame to an upper world. For us, it's like you have a frame to the engine of the museum, a lot of gears or magical mechanical pieces.
[00:11:58.093] Kent Bye: So yeah, it was one of the challenge of the thing to find creative way to enlarge your own spaces Yeah with the windows on the side you see these moving gears and then in front of you you eventually see this big cage that has a glass containing some of these monsters, but I wanted to ask did you put a purplish or bluish filter over everything with the color pass-through or
[00:12:22.096] Clement Deneux: No, I think it's more relative to the lighting of the Truebreaker setup because none of it didn't add. Controllability, because we don't have a lot of access to the pass-through of the user. It's really something you, as a developer or creator, you don't have any info. I'm talking about the Quest Pro. It's something Meta don't want to share with any creator. So you don't have any control of the pass-through of your user. You can just control elements you can add on the user space. So you just can a little bit make it a little darker or add a few things. But also you don't have a feedback of the pass-through, we don't add nothing special on it.
[00:13:01.366] Kent Bye: Yeah, I guess there's a lot of like blue lights in the Tropica space, so that was probably that, yeah.
[00:13:06.388] Clement Deneux: And it's also because at the beginning of the exhibition we realized that the Quest is quite struggling with the low-light tracking. So that's why you struggle a little bit to draw portals and everything because the limitation of the camera, tracking camera. So now we raise the lights and it works better.
[00:13:24.883] Kent Bye: Okay, so you're able to work with Tribeca to make it a little bit brighter?
[00:13:27.785] Clement Deneux: Yeah, absolutely.
[00:13:29.687] Kent Bye: So in these mixed reality pass-throughs, there's your physical hands, but sometimes you overlay this monster watch thing that's on my hand as I'm moving around. So sometimes when I move my hand quickly, there's a disconnect. But if I move slowly enough, it is connected. So I'd love to hear that process of trying to add these other embodied components that aren't on someone physically, but you have virtual wearables.
[00:13:52.768] Clement Deneux: Yeah, it was a narrative decision in a way. We have two characters in the pieces, you have like a drone, we can follow you during the experience, so it's like the main bad guy and you have the other character which is a monster wrapped around your wrist and he will at some point talk to you telepathically and it's like a war between these two characters and you are in the middle and you will probably help the monster to escape and to free all the monsters from the museum. So this monster is the creator of the bubble of fear that I was talking about previously. And we worked a lot about his design, the fact that he needs to be disgusting but also touching and moving. And the fact that he will look at you every time, or he can speak to you without moving his mouth because he doesn't have a mouth. So we worked a little bit about the character design, how it works. But the original idea was to have a device that could have been a gun in a different type of game, or a big digital watch that has a lot of devices, or a laser or anything. But for a Museum of Monsters, we thought it could be a good idea to have a monster as a device, in fact.
[00:15:07.423] Kent Bye: Yeah, I've seen a lot of immersive stories that have this floating drone-like AI character that ends up being a docent that's guiding you through an experience. I mean, it's something that seems to be an emerging trope that I see again and again and again within immersive experiences because you need to have some guidance, and instead of having like an omniscient voice of God, it's sort of embodied in this AI robot character that's helping guide you around. So it seems to be a decision that a lot of different immersive creators are settling upon, but I'd love to hear your own process of deciding to create this AI flying drone-like character.
[00:15:41.410] Clement Deneux: Yeah, it's not an AI because we tried to make it as a real character. In fact, the voice actor is made by Andy Serkis. He really played the character as the main antagonist. But yeah, I think we realized that after we created this character that a lot of other work using ML came with the same creative decision. So yeah, we need a drone following you. So yeah, I think you're right. It's like a trap or it just makes sense. But it's starting to become a cliché. There is no pieces out yet. So we need to be careful next time. But I think for us, it's more like a first version of the main guy of the collector, the main guy, the owner of the museum. So it will have the shape of a drone for this episode, but it could take another shape. It could become another thing. And he has like a human appearance at some point. So it's more like a first glimpse of the character we will see after.
[00:16:40.343] Kent Bye: OK, so I guess to clarify, there is AI that is being driven by these human characters. And then there is a character, narratively, is an AI robot. So narratively, is this a person that's using this robot as a telepresence robot? Or is it AI within the context of the narrative?
[00:16:56.098] Clement Deneux: No, no, in the story it's like there is someone somewhere in a museum controlling the robots but we are still in the process of writing the next episode so we're still in a discussion about that but yeah the idea is it's not like a robot it's more like someone somewhere as a main bad guy of the stories controlling all the museum all the mechanical parts so he sends you this robot as an invitation to the museum In fact, we had to cut the first scene for budget reason, that the drone was in fact like a jack-in-the-box. That's why he has this look like very squarish robots. So at the beginning we are supposed to receive this strange musical box and you open it and it's like jack-in-the-box and it will be the open of the invitation for the museum.
[00:17:41.549] Kent Bye: OK, OK. I guess I projected onto it as an AI because I've seen it repeated, but it is the voice of Andy Serkis, who is well known of playing Gollum, another voice actor. So what was it like to work with Andy on this piece?
[00:17:52.768] Clement Deneux: It was really great because also we wanted to have the same actor playing the two characters. So you also make the voice of the monster you have on your wrist. And when we were thought about actors, we were able to make the two characters with very different voice. It's not something the audience needs to realize. that's the same actor playing the two roles but because we wanted to like create ambivalence between the one character who locked all the monsters in cages and one character who want to free them so it's almost like metaphorical about the relationship you have with monster of your own fear Or you can be like a control freak, making all the monsters in cages. Or you can try to release it and maybe be creative with your own monsters. So it was a little bit too French, complex storytelling, but it was our drive and process to make the story happen. So when we think about some actors with great voice and the ability to create a monster and also a monster voice and also be very like, I don't know in English, the guy in a circus showing everything with a big hat and a presenter or anyway. Because I really liked his work as an actor in All the Monkeys, Planet of the Scissors. I think he's a really great actor. But he's also known as the Gollum. So he really can do both kind of roles. Like he can pass very different voices. So yeah, and he liked the project and he was really involved into the process of making these two characters.
[00:19:36.785] Kent Bye: Yeah, and so there's another thing that I've seen more and more in these mixed reality experiences. There's a piece that was at South by Southwest that did this where you're standing in a space and then the floor opens up and then something comes out of the floor. So it's a way of bringing objects into your space in a way that feels plausible. And so there's a moment where there's this cabinet of all these exotic objects that you start to touch the objects and then you get a little bit more context in the story and so there's one object that you touch that then brings up a button that I figured if I would push that button then I would move on and so before I push the button then I wanted to actually touch all the different objects because I wanted to hear as much of the story as I could, but it's sort of like this way of doing environmental storytelling where you're using objects to have a narration over it, but to give a little bit more context. So, love to hear about the design process of this cabinet of all these exotic mysteries that are helping to give more context and backstory, and that you have this ability for people to choose how much additional context they want to have. You have to touch at least some of them to move on, but left up to the user to see how much they want to explore for some of these. So I'd love to hear about that process.
[00:20:48.318] Clement Deneux: Yeah, at the beginning it was always a project about also the backstory of each monster and how we wanted to give information, some context and not to have just like action game about one monster but wanted to give some glimpse of what is the myth behind this monster. For example in the story we have like a fresco of what is the first werewolf. We have a few element of backstory we put everywhere Almost like, yeah, if the player wants to stay a little bit more and to learn a few things. Also, it was a funny way to write a funny sentence. And this, I guess, will have pleasure to play. And it's really funny and that. And ideally, we wanted to make all the objects physical, that the player will be able to touch them. But we didn't have enough time or budget to make it, so it's just only touchable. But yeah, the idea was to create some context, backstories, and also almost like a small or easy enigma into this cabinet.
[00:21:55.599] Kent Bye: Yeah, and at the end of this piece, there's quite an exciting, dramatic turn of events that turns more into this embodied gameplay that I won't go too much into detail. But I'm curious, is it possible for the person playing the experience to die? Or have you created it so that there's no way to die? Because then it becomes more of a game than a story. So I'm wondering how you navigate this aspect.
[00:22:17.653] Clement Deneux: We're still in the process of writing the gameplay. It's not possible to die now. At some point it was possible. We just finished the piece you saw two weeks ago. It was quite a sprint at the end. We take this decision to not die on this episode. Maybe we will change later. But yeah, we wanted to do something. It's supposed to be a video game. So I think we need to become a little bit more gameplay-driven for the following pieces. It was more like an introduction, so it's really narrative-driven. But in the future, we need to develop more the gameplay. But yeah, for the first episode, you will see a werewolf into your own space. And the idea behind it was to, in a way, let the gamer experience the metaphorical that I was talking about previously. So each monster will have a different gameplay. So you will be able to experience, for example, the werewolf is all about animality. It's just something coming to eat you. So the gameplay, you just have to move to avoid, so it's really very simple, but it's a good way to experience for a player the ferocity behind the myth of the werewolf. But if we have, for example, a next monster for Minotaur, so we will have to relieve them to the maze, so it will be a totally different kind of gameplay. It's almost like an escape game inside your own place.
[00:23:45.267] Kent Bye: Yeah, I have to say that the experience of that did have a certain amount of intensity of having this thing lunge and jump at me and having to escape it, but also to try to draw a portal. If the portal was too small, it would disappear faster. And so then you have these little hints that are like, OK, if you want it to last longer, you have to draw bigger. But the bigger you draw it, it's sort of like these different trade-offs that I felt worked pretty well for at least this first episode. I feel like as you move forward, then I guess this is an opportunity in some way to experiment with what's it mean to have hand-driven gameplay and interactivity as you're embodied in these spaces that are being modulated. So you're in your, let's say your living room, but then you're adding all these contextual elements to change the context of that living room into this monster museum. And then here you are interacting with these different monsters in different ways. So it seems like you have constraints of just using hand gestures or hand movements, but also using your movement through that space in order to have this embodied gameplay with both your hands and your body.
[00:24:43.533] Clement Deneux: Yeah, absolutely. And we chose the fact that we want to have an untracking experience. So you can do it with the controller, but we decided to add the feature that you can do everything with the controller, but we really prefer that people experience the piece only with hands. And also the fact that you can draw this fluid as almost like a quill game, as you say. It's really more efficient and enjoyable with your own hand. And we wanted also to remove all the devices. Everything can create a separation between you and the space, because that works very well in Mixed Reality, the fact that you are in your own space, you see everything around you, and sometimes you can go to a VR world and go outside. We want to go this way for the next episodes, be able to navigate between your world, VR world, touches, and lots of different gameplay that we'll explore in the future. relative to each monster.
[00:25:43.285] Kent Bye: Yeah, and I'm curious to hear what's next with this project and if you have an estimated timeline for when you're shooting for either to complete the piece, to continue a festival run or to have a public launch of Monserama.
[00:25:55.879] Clement Deneux: We are in the process of writing the full game. It's supposed to be around two hours. It will depend on which kind of budget we get. So we will write and design all the gameplay in the next month. And parallelly, we will be looking for budget and funds. And ideally, we will start the production of the full game around October or November of this year.
[00:26:19.885] Kent Bye: Awesome. And finally, what do you think the ultimate potential of mixed reality and spatial computing and the intersection of gaming and storytelling might be and what it might be able to enable?
[00:26:32.852] Clement Deneux: It's really hard to have something intelligent to say in the middle of while you're actually doing things. I think you need a little bit backup and think about everything. But what I saw is that a lot of people who are not VR enthusiasts or they just try one VR headset sometimes and when you give them the opportunity to test the MR, the fact that you see your own environment, you are not lost into VR things, I think it looks very less intimidating. So I think it could be a good way to convince more casual gamers or more people who are not geeks on gaming. The fact that you can see your own space all the time is something that can feel safe. So that's why it's quite efficient when you are in a safe place and suddenly there is a hatch and something get out and try to attack you. In the same time you feel safe but you can feel very scary at the same time. So I think it's a good opportunity or good way to be even more immersive for people who don't feel confident to go in Total VR world.
[00:27:44.237] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the Immersive community?
[00:27:48.299] Clement Deneux: No, thank you for the chat.
[00:27:51.560] Kent Bye: So that was Clement Deneu, he's the director who's working with Atlas V on the first episode of Monsterama, which was premiering at Tribeca Immersive 2023. So, I have a number of different takeaways about this interview, is that first of all, well, this is a piece that's exploring the frontiers of technology from mixed reality to hand tracking and trying to come up with both the game and embodied interactive aspects to it, but also how do you start to transform your existing context into this magical realism that's trying to take you into another place by not even having to leave the comforts of your own home. Now, this sounds like it's a piece that was really intended to be able to modulate your perception of your own world around you and stuff that you're familiar with in your own living room. However, I saw it at Tribeca in a big, nondescript, essentially white square. Now Clement said that over the course of the showing this piece they made some modifications and they made it brighter and just I think there were some issues with the hand tracking whenever I saw it so it was dropping the hand tracking and made it a little frustrating but being able to use your hand to draw these circles of liquid that are representing fear. And as Clement said, a typical French story that's trying to have all sorts of metaphoric explorations of both the inner and outer monsters within all of us. And so there's much more depth to the story than I necessarily picked up on the first pass. I was more focusing on the embodiment and the gameplay and the overall story that was being told. So different elements of environmental storytelling when you have these objects like the whole cabinet of all these different mysterious objects that in you pick up and start to learn more about the world around you and also having these little portals on the side so that when you are seeing your wall that you normally see but then you see like these gears that are turning around as you go and up and down the elevator and expanding different aspects of the walls out into these different cages for the monster but also as you look up having the elevator as you're going down and trying to give you this sense of moving through space by not moving anywhere at all. So this is a much larger game and intended to be more of a game than a narrative although I think right now it's certainly leaning much more towards this narrative aspect. I think there is a core gameplay loop that they're trying to teach you, which is just drawing a circle and that creates a portal. With that portaling effect, the simple act of drawing a portal and then being able to see into another immersive world is, I think, a compelling aspect of both augmented reality, but also different aspects of virtual reality to see these windows into these other worlds. AR and the way that I've seen mixed reality is really leveraging on some of those different affordances of you're in the center of gravity of your existing context, but you're expanding out your context by adding on all these different elements. And so this is an experience that's probably leaning more heavily into cutting out existing aspects of your environment to be more of a transportive aspect of augmented reality and mixed reality by pulling in those different virtual reality components. So I think there's still probably a lot of tuning that will need to happen before this gets launched into the mainstream by the gameplay loops. If it is indeed a game, then trying to explore the different mechanics and the embodiment. So you're essentially moving your body around to dodge and then trying to draw these circles. The smaller you draw the circles and they disappear faster. So you have to take more time to draw bigger circles. So it lasts longer, but as you take more time, then you have the urgency of this thing that's like rushing at you. So. Yeah, just different gameplay mechanics like that that they're exploring. And as they go through the different monsters, then they're going to have presumably different ways of exploring both gameplay using your hands, but also using the mixed reality to interact with everything as well. And yeah, just working with actor Andy Serkis to be able to voice both the monster and the main character. And I think this is an arc that wasn't necessarily clear to me that it was the same character, but I think as time goes on exploring some of the deeper themes of their story, Because it does sound like this is a piece where they can take these creatures from mythology and start to give you an embodied interaction with them, start to bring in different artifacts that these creatures might have, and create this virtual object portal and immersive experience portal into these larger mythological stories that are ultimately trying to reflect upon us as humans and our own fears and inner monsters within our own lives. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listed support podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
