I interviewed Jim Henson’s The Storyteller: The Seven Ravens + The Orbit director Paul Raphaël at Venice Immersive 2023. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast, the podcast that looks at immersive storytelling, experiential design, and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So continuing on my series of looking at different immersive experiences from Venice Immersive 2023, this is episode number 25 of 35, as well as episode 5 out of 10 of looking at the context of ideas, adventure, and then also the second of seven within a series looking at animation and different immersive storytelling innovations that are coming from the context of animation. So this piece is called Jim Henson's The Storyteller, The Seven Ravens by Felix and Paul. And they also, in the second half of this conversation, we actually dive into the VR space explorers, Blue Marble, Orbit One, where you have this 360 video shot on the International Space Station as it orbits the Earth, watching the Earth go by. And you're watching the sunrise from the International Space Station and the sunset. So around 20 minutes, you're able to watch a half cycle of the International Space Station as it orbits the Earth. But the first part of this conversation is around Jim Henson's The Storyteller, which is a mixed reality piece where you have a physical book that has a lot of fiducial markers within it. So as you turn the page, you progress the story. So it's a bit of a augmented reality pop-up book using The Magic Leap 2, and it's telling a Brothers Grimm story, The Seven Ravens, where the father curses the sons, and the sons all turn to ravens, and this daughter has to go out on this kind of epic grand adventure in order to undo the curse. So this context of adventure and the hero's journey, as well as the context of family. And the primary center of gravity of presence is really focused on this embodied environmental presence. You are holding this book and you're using your body to turn the pages and have in different ways that as you move your hand, you get these different perspectives on the piece. And so the sense of environmental presence is a little different because you are in the center of gravity of the physical presence of the installation that they have, but also this kind of tabletop experience of the piece. And so you're kind of looking into this piece. You know, we've seen that a lot within the context of VR and a lot of augmented reality experiences, just get a better field of view with the AR devices, but lots of different innovations when it comes to actually the grammar of how they're telling the stories that we dive into as well. So there's other aspects of interactivity and actor presence as you're able to move the book around to get different perspectives. And sometimes the way that you move the book around actually changes different aspects within the scene. It doesn't change the story, but it just changes your perspective on it. And then the third aspect of his emotional presence of just the story that they're telling. And also the space explorers, blue marble, the orbit very much around environmental presence as you're going around the earth and, and just bearing witness to the earth as it passes by just a really poetic piece and very simple Zen experience with minimalist approach with the soundtrack. So also just very emotionally provocative as well, invoking different aspects of the overview effect. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Paul Warfiel happened on Thursday, August 31st, 2023 at Venice Immersive in Venice, Italy. So with that, let's go ahead and dive right in.
[00:03:16.810] Paul Raphaël: I'm Paul Raphael, co-founder and chief of innovation at Felix & Paul Studios, and we have been creating VR and now AR experiences for close to 10 years now, mostly in cinematic virtual reality, but lately also in interactive, 6DOF, large-scale installations, and here at Venice launching Jim Henson's The Storyteller, an augmented reality storybook.
[00:03:43.325] Kent Bye: Great. Maybe give a bit more context as to your background and your journey into the space.
[00:03:47.709] Paul Raphaël: Sure. Well, I was originally in film and cinema. That's what I studied. After leaving school, I met Felix, who I've been collaborating with now for close to 20 years. We were both filmmakers and interested in creating immersive films. And by that I mean films that create a sense of immersion. more so than just trying to get the story out to you in as efficient a manner as possible. Eventually that led us to explore outside of the cinematic frame and experiment with technologies such as projection mapping, interactivity, reactivity. We also did some work with Cirque du Soleil and some live shows and installations. that eventually led us to play around with stereoscopic projection, basically 3D films. We were using the technology mostly to create installations rather than films, and by that I just simply mean, yes, it was a film, but we would constrain the variables of how the film was projected and where you would watch it from, basically creating a form of anamorphosis with these 3D films. So the way we shot them and the way we projected them basically allowed us to create a proper sense of scale and distance like you'd have in VR. So essentially we were creating a limited field of view VR where you couldn't look behind you before we actually started playing around in VR which happened around the time of the DK1, the Oculus DK1 Kickstarter. When we got our first kit, we realized that everything we were trying to do was actually pointing towards this technology. Of course, when we first tried it, there were no films in VR, or at least no stereoscopic films in VR. And there were a few interactive demos and games. And so we set out to figure out how to create what at the time was the very first cinematic VR experience called Strangers with Patrick Watson and that basically kick-started our studio. Since then we've created over 30 experiences, most of them cinematic VR and we've also created a lot of technology. to create these experiences and we have a team of today about 70 people at the studio. Some of them are compositors, some of them are programmers, some of them are creatives and basically everything else that makes a company run. And yeah, how's that sound?
[00:06:06.529] Kent Bye: Yeah, it's quite a journey that you've been on. And I remember being at Sundance 2017 when you were showing Mayubi, which is this transition from just doing cinematic 360 video and starting to have more wrapped into a Unity wrapper and having a little bit more interactivity. And is there any other purely CGI projects that you worked on ahead of working on this Jim Henson's Storyteller piece?
[00:06:29.019] Paul Raphaël: So Jim Henson's was the project for which we started building our internal interactive team. So when we did MIUBI, basically, we had software developers that we had at the studio. They were mostly there to create the VR video players, basically, because when we started again, there were no video players. So we created our own. And our video player also had all sorts of custom optimizations to push as many pixels as we possibly could on limited hardware. We also had to create our own way of playing back immersive sound. There were no standards for that at the time. So that's the team that ended up creating the bits of interactivity in MIYUBI. But when we set out to create our first fully real-time rendered project with the Storyteller, we actually ramped up a real-time software development team and from there we also started developing The Infinite which is our large-scale multi-user VR exhibit where up to 160 people can simultaneously free roam around a life-size reproduction of the ISS while experiencing capsules of contents that are geolocalized in the space and that have been shot in and off the International Space Station. So this is content that is related to the Space Explorers documentary series that we've released over the past few years but is an experience in and of itself. So interestingly we built this theme for Storyteller We then started creating The Infinite about a year or two later, and we finished The Infinite before we ever released Storyteller, which is quite a few bumps in the road on that one. But yeah, now we have a pretty robust team that's actually growing. We are wrapping up, as we wrapped up Storyteller and have finished The Infinite for a little while, we're ramping up on our next large-scale LBE, a location-based experience like The Infinite, and hopefully making more experiences like Storyteller.
[00:08:24.907] Kent Bye: What's the next LBE? Has that been announced yet?
[00:08:27.003] Paul Raphaël: It has not been announced yet, but it is about a year in the making already. And we're also in talks with a few other partners to create other shows as well. So depending on how things go, our team may grow a lot in the next year. But yeah, we're the biggest we've ever been today at close to 70 in-house. That doesn't include the contractors and partners that we have on just about every project we're working on as well.
[00:08:50.129] Kent Bye: Yeah, I want to come back to the Space Explorers and the experience that you're showing here at Blue Marble. But let's go back to the Jim Henson's storyteller experience. How did this project come about?
[00:08:59.659] Paul Raphaël: So in 2017, 2018, around the launch of Magic Leap, you know, there was a big push to get creators involved with the platform and we started thinking, okay, whether it was for Magic Leap or anything, it was clear to us that AR and VR were two sides of the same coin and today we can only really optimize hardware for one or the other, but these two things are destined to converge and You can see that happening already with the Apple Vision Pro, and eventually you have thinner devices, eventually glasses that can do both these things. Regardless of what hardware you use, even you think of the two mediums, they are fundamentally different, but also fundamentally similar, and that really the only difference between one and the other is the opacity of the pixel, right? It's like windowed mode versus full screen mode, in a way of speaking. So we were very interested to see what that meant for us, you know, as immersive storytellers, what does not being fully immersed in an experience mean, right? And does being immersed in reality and bringing virtual elements into it, could that even be a higher sense of immersion? Because there is no higher immersion than being in the real world. So we started thinking about that and of course one of the many big challenges is, well if you're going to bring fantastical elements into the real world, how do you justify their presence? How do you suspend this belief? It's a lot easier I think when you control every aspect of an experience, whether it's a film or a virtual reality film, you're creating the context, you're creating the characters, it all kind of makes sense together and we have over a century of practice knowing how to suspend this belief for that kind of story. But we've had to kind of rediscover what that means for virtual reality storytelling and with AR it was kind of starting again from scratch but trying to bring everything we've learned from immersive storytelling and VR to this and see if any of the instincts we've developed could serve us here. So the first thing that came to us was this idea of a book and the book for several reasons. So the very first reason was the book itself is a ritual that people are used to taking them into a story world, whether it be fiction or nonfiction. You crack a book open, and we have this inner switch. It's almost like a proscenium mark that you hold in your hands, even if it's text, or whether it's text or images, and whether or not it's moving. You look down at this object, and you know that you're going into story mode. And so that was very important for us, because, again, to have characters on your coffee table, it could be cool and cute, but a huge challenge for the suspension of disbelief. And then the next thing was the intuitiveness of it, right? So we've been creating a new language with virtual reality storytelling, and this is the same. However, again, a book is not only something that's a shortcut to suspending your disbelief, but it's also a modality and an interface that we are very familiar with, right? You just hold the book in your hands, you turn the pages, right? And that's how the story moves along, and it's exactly how it works here too. So that's another, you know, we've always wanted to create experiences that are as accessible as possible, mainly because it's... Well, for many reasons, you know, one is it's such a small industry that we want to reduce the amount of niches within niches that we have to deal with. But also, you know, it's such a powerful medium that we want as many people as possible to experience it and to want to experience it and to be able to experience it. So anyways, this accessibility, this intuitiveness of holding a book in your hands and just knowing exactly what to do There is no interface, there are no buttons. Once the app is running, you don't even need to know much, right? And it's up to us to make the experience as intuitive as possible. There's still a few little things that are not quite there yet, but we're working very hard and I think we've gotten quite far into making it as seamlessly intuitive as possible. And then there were a few more other things, right? Like, you know, we started developing on the Magic Leap 1, which had a quite limited field of view, and the book was basically designed so that at comfortable viewing distance, it fit perfectly within that field of view without any cropping. Now, of course, it was such a tight fit that if you moved your head ever so slightly, you would start cropping the image. But thankfully, with the Magic Leap 2, we got that buffer around the edges that I don't think the field of view is even much of an issue anymore, especially now with this experience. I think it would still be if you had a hologram of a character standing again in your living room, you would still lose their feet if they were much closer than six feet or something. And that's not an issue again. that is not just a nice technical thing. You know, if you're going to suspend this belief, you don't want to have such glaring technical limitations in your face the whole time. So that was another reason. And if I could keep going, the pages themselves are very, very dark. They're almost black. And aside for the trackers, they allow very high contrast image because we have additive light with these augmented reality devices, right? You can't project black. You can only project lights and color. So having a black background allows you to have the highest contrast and most beautiful image possible. And then finally, why a book? Well, you're holding this book in your hands. We're creating a tactile relationship between you and this miniature world that you can hold and move in any direction. And honestly, to us, this is the most tactile thing we've ever done, even though we've worked in virtual reality. for so long, you know, even when you're holding controllers, even when you can track your hands, unless you're in, again, in a location-based setting where you control everything and you can align virtual objects to physical ones, most people's tactility and immersion are holding controllers that vibrate and that pretend to be objects that they are not really. But here you're holding a book physically and you're holding a book virtually and it's just like perfect alignment and there's a beauty to that and there's an immersion that comes with that. There's a warmth that comes with the leather binding, right? And so, yeah, all sorts of really good reasons to go with the book.
[00:15:00.244] Kent Bye: Yeah, I think it's a really quite compelling interface. So it's called Jim Henson's The Storyteller, the story of the Seven Ribbons. What does Jim Henson have to do with this project?
[00:15:09.941] Paul Raphaël: Sure. So the way Jim Henson's involvement came to be was we were actually, I guess about six years ago now, we were in talks with his daughter, Lisa Henson, who's running the company along with her brother, Brian Henson. We were talking to her about a different project that actually never got made that was in development at the time. And when we were sort of thinking about this AR book, We actually asked her, you know, like, hey, you know, we've been thinking about doing this thing with this new device, the Magic Leap, using a book, and we basically described the idea in a vacuum, so to speak, the idea of kind of a AR pop-up book, and asked her, you know, Jim Henson had this show in the 80s called The Storyteller, where you brought, like, folk tales to life, and you're like, we think this would be perfect, you know, first of all, the title is perfect, right, and then the format, you know, at the time was, I don't know if you saw it or remember it but it was basically a man and his dog sitting in a fireplace and they would start telling you a story and it would slowly come to life within their environment and as a form of AR and then in kind of full immersion with live action actors and Jim Henson creatures and all that. And so it was just perfect because we knew we wanted to tell a short story, we knew we wanted it to be probably a fairy tale because it's a great way to not be too literal and have this kind of Dream like haziness that fairy tales have we you know because we didn't want to fully lose what a book does Which is allow you to imagine what you're reading right now, of course By definition by replacing words by images we're taking away a lot of that but we didn't want to take it fully out because that's what reading a book is usually is you bringing yourself and the words and you coming together to create something that is unique to your experience. And so we didn't want a story that was too literal. We wanted something that was open to interpretation. Anyways, a fairy tale was perfect. The storyteller was perfect in a lot of ways, too. Anyways, and it turned out that they were actually working and developing on a reboot of Storyteller for television with Neil Gaiman. So she was super excited about the idea. She even put us in touch with Neil Gaiman. And yeah, that's kind of how it all came together.
[00:17:20.064] Kent Bye: And could you give a bit more context as to the story that's being told here? Does it exist in other media? Is it like a book or film? Or where's the story coming from?
[00:17:29.108] Paul Raphaël: Sure. So the story is the Seven Ravens. It is most famously known as a story from the Brothers Grimm, but it actually goes further back than that. And you know, like most fairy tales, there's so many versions of this story. Some of them have more or less ravens. Some of them are not ravens. They're other creatures. And honestly, the meaning of the story also, well, first of all, it's very much open to interpretation as it is, but throughout cultures, fundamental meaning seems to change as well. So we went with this iteration of it. We wanted a story that we could tell, you know, with a certain level of efficiency that had set pieces that would be optimal for the format. And we also liked the art visual style of Billy Bin, who's famous to have drawn many such folktales. In fact a lot of the 2D art that is in the book are inspired by him and even the 3D art is kind of a extrapolation of that visual style. So yeah those were all reasons I think that this was a great way to start the storyteller but you know there's literally an infinite amount of possibilities moving from here.
[00:18:33.202] Kent Bye: There's been a number of years that augmented reality headsets have been out in the world. And there's been a handful of compelling, immersive storytelling AR experiences that I've seen. I think that your experience here with Jim Henson's The Storyteller, with that book, starts to really get into all those affordances. I think of what I've seen a lot from the VR world. We have the tabletop scale type of aesthetic, where It's been a lot of animation that's been using that to start to tell a story. So I'd love to hear if there's any particular experiences that you were drawing inspiration from for how to start to utilize this type of tabletop scale of immersive storytelling.
[00:19:10.407] Paul Raphaël: Yeah, I think there have been quite a few AR and VR tabletop scale experiences that I think it was clear fairly early in the game that there's something very satisfying about these miniature worlds. You know, our perception of depth diminishes as things get further away from us, right? In real world scale, I can see you in quite a bit of depth, right? But everything beyond that, you know, it kind of becomes flat. Even though I see in 3D, most of it is actually monoscopic. And that's true of any experiences you do in VR where most of the things you're looking at are further away, right? With miniature scale you really get to relish in this pleasure of sensing three-dimensionality, right? And you also have this kind of ability to move yourself and become kind of a helicopter shot over objects and characters and, you know, maybe... The pleasure we had as children playing with toys or figurines that kind of bubbles back up. Or the way we find little creatures, cute little animals, babies, right? There's something there, I think, that's inherently pleasurable, right? And yeah, definitely, you know, going into this project, yes, I've talked a lot about the book, but really what you're holding in your hand, the images that are popping up, are these miniature worlds, right? And so there is a pleasure in just changing the perspective, looking around it, and the physics of the world kind of reacting to your movements as well. There's something very precious there.
[00:20:34.434] Kent Bye: Yeah, and with the fiducial markers in the book, that means that you could buy a physical copy of the book, but you could have literally an infinite number of different types of experiences that you could have with the book.
[00:20:44.187] Paul Raphaël: Absolutely, and of course we're presenting one and the first experience such as this here at Venice, but really this is meant to be a demonstration of what can be done. We hope to do more storyteller fairy tales, we hope to work on different IP as well. You know, we're in talks with different platforms and studios and IP holders to make this a library of content. You know, today is still a bit early for that to happen. The devices that could make this book come to life are very limited. Well, they're expensive or not out yet. There's a few things coming out and even those are going to be expensive. Although some of them less than others. So we're still a couple years away I think from this being a viable product. In the meantime, I think it is a fantastic way to demonstrate the potential of augmented and mixed reality. Again, for all the reasons I mentioned earlier. For something to be intuitive and to lean into the strengths and away from the weaknesses of a medium, I think is a great thing as a creator to do, but it's also great to show off a device, right? And that's what we're hoping device manufacturers are going to realize and we're having those conversations now to hopefully partner up with some of those.
[00:21:58.549] Kent Bye: Yeah, one of the other things that I really enjoyed was the pop-up book nature of how the animations kind of fold on top of each other in a way, or at least they pivot or sometimes they even have the person that you're seeing is standing still and when you turn the page it sort of has more of an environmental effect. transcending my expectations. It sets up an affordance that you're used to with the pop-up book of how when you open it up, then you see the scene and it kind of folds on top of each other, but then that's played with throughout the course of the story as well. So I'd love to hear about, you know, drawing inspiration from pop-up books and how you are trying to recreate the essence of that within AR.
[00:22:32.113] Paul Raphaël: Yeah, definitely. You know, when we were brainstorming about all the different ways we could exploit this format, we came up with kind of a palette of different tools and tricks and cool things we could do. We decided to create a progressive reveal of these different methods. So the very first page is actually just two-dimensional, and it's just about, oh, wow, this book that was blank now has content. That's the first trick, right? There's a loose feather in there that reacts to your physics, but it's essentially a flat image. And then you turn the page and then we reveal depth and it's like you're looking through a keyhole and there's a little scene happening in there and we've revealed a dimension moving into the book like it's a window into a world. You turn the page now we're revealing a relief, right? Like depth coming out of the book and it's a little maquette of a little house and a little well and these little boys that run out of the house. and we're also revealing movement in this scene because the past two scenes have been fairly static, right? And then we start mixing those things together and then creating more depth and also ways that the page turn affects a scene, right? So there's this one moment where turning the page doesn't actually change the scene, it sets it on fire, right? And the page turn essentially triggers a wave of fire that destroys the feel that the little girl is walking through. Yeah, so there are just all these different methods of surprising and creating these moments of pleasureful kind of reveals of what can be happening with this medium.
[00:24:00.043] Kent Bye: Yeah, I did an interview with Curtis Hickman of The Void, and he was talking about how his perspective is that virtual reality is a mimetic medium, meaning that you're looking at representations and that it's more about the show, very similar to film in a lot of ways, rather than more of the diegetic where you're using a lot of narration and relying upon other aspects of dialogue or whatnot to tell a story. And I feel like, in some ways, I see it as a combination of both of those. Show, don't tell, but you're trying to both show things but also tell things. And I think this story works particularly well because you are doing a great job of showing each of the different stuff so that you could potentially watch it without any sound and still get a sense of what was happening, but the sound is adding a whole other layer. But I'd love to hear any reflection on the use of narration and dialogue to be able to tell these stories and how you think about this kind of diegetic forms versus memetic forms of storytelling.
[00:24:53.512] Paul Raphaël: Yeah, I think we've been a lot more experiential in our storytelling than we have been, you know, like I said earlier, getting as much story beats through as possible. It's really about relishing in a scene and with characters and holding it longer than not, you know, whenever possible. And we do that here in some way in the sense that Most tableaus are like little sculptures, you know, they begin one way, there's a transformation and they end a different way. And you can hold that for as long as you want, right? And it's really up to you to turn the page when you're ready. But you can play with that as if you had this little castle in your bedroom that you were just, you know, imagining, you know, just having pleasure looking at or moving around. And so that's one of the things that this mechanic also grants is as slow or as fast a rhythm as you feel like having. And one of the things I've learned over the past 10 years of making VR films is for some people our rhythm is just right. Some people wish we would linger longer and some people find it insufferably slow, right? So here, you know, you can do your own editing this time and it happens totally naturally, right? You don't even need to tell people that that's happening, right? And then you basically have Neil Gaiman's voice telling the story, which honestly on its own could be an experience. It could have been, without any of the augmented reality, a great way to experience the story of the Seven Ravens. But the two combined, I feel it's just like, we're so grateful that he was into doing this. It allows us to show some restraint, right? We didn't want to have long scenes where a bunch of stuff happened visually. We really wanted each one to be like, almost like a GIF or like a cinemagraph. I forget what you call those short looping images, right? You know, like it's alive, but it is a frame, right? It's a volumetric frame and it's a sculpture more than it is a scene from a movie that you're holding in your hands, right?
[00:26:54.992] Kent Bye: Yeah, there is a certain rhythm that I got into in watching this piece because there were some moments where I was trying to play with, okay, I'm going to see how long this scene plays out because what happens if I just don't turn the page? And so at some point I think the music does stop and then you can turn it, but there is a bit of a beat where it's like, okay, this is, as an editor, where I would make the edit. Because there's this thing where you don't want to turn the page too early, because I was told at the beginning, don't go backwards, because you'll screw things up. But you'll find out when you play the experience. So you want to make sure you experience everything, but also you kind of know when, OK, there's a rhythm to the story that's being told, and the beat has stopped, and now there's no more dynamic change that's happening. So there's a way of kind of understanding, OK, here's the beginning, and here's the end. and knowing and being able to identify that, but you kind of get into this rhythm of being able to progress through the story. In this, like you said, you become the editor of the story and fall into your own rhythm of reading it. Or you're actually not reading it, you're watching it.
[00:27:55.454] Paul Raphaël: Exactly and you know this is obviously the first project that we've done like this and we've had so many ideas of what else we could do like for example you know when you're done with the scene and you're there we could do more without while keeping it this sculptures living sculpture right and we could also make as clear as I think it is when it's time to turn the page It isn't 100% clear for some people, and I think there are subtle hints that we could add there. There's a plot twist about halfway through that, again, I think we do a fairly good job of giving you cues into what you're supposed to do, but, again, it's simultaneously too heavy-handed for me and still not clear enough for a lot of people, so I think we could do a bit better there, too. But really, what I'm trying to get at is we've learned so much creating this first piece, And we always thought we wanted to show more restraint than trying to go too crazy with it. We really wanted to be like the MVP of our version of an AR book. But there's just so many things that we want to do now and that kind of things we didn't do here, ideas we've had while doing it, that could make this quite a bit of fun. Quite a bit more fun, yeah.
[00:29:05.345] Kent Bye: Well, they have the Magic Leap 2, which they did a bit of a pivot from entertainment, immersive stories type of stuff into purely more enterprise applications. And so it's still on the Magic Leap 2. And there was actually a piece at the Shed that was in New York City that was like more of a volumetric piece that was showing they had like 80 different Magic Leap 2s with People watching it be able to walk around which was kind of a wild experience to be at a concert but to like walk in front of other people but to not feel like you're Interrupting what they're experiencing because lights being added on top of what you're seeing and so you're not really getting in their way So that was kind of like a weird trippy experience but other than that I haven't seen a lot of other uses of Magic Leap 2 and LBE context because it's mostly like I said enterprise and and a little bit less entertainment now. So as you start to move forward, we have, like, Apple Vision Pro, but even that's, like, $3,500. And then we have the Quest 3 that's coming out that has, like, more mixed reality components. And you have the Quest Pro. So just curious to hear what your thinking is in terms of, as you start to bring this experience out, is this going to be, like, more of a location-based experience? Or are you expecting to have some sort of, like, consumer launch for the types of people that would have some of these mixed reality headsets that are, right now, very expensive?
[00:30:14.978] Paul Raphaël: So the current plan is to obviously show the piece in exhibition settings for the next short little while. We are going to announce a more precise date soon, but we will be showing this at the FI Center in Montreal early next year. Ultimately, though, it's going to be hard for that to be a business model that works, showing a handful of people the experience as it travels around the world. Really, the reason we're doing that is to get the piece seen, but also to show the potential of this format. So what we're also doing behind the scenes is showing it to a lot of people. within the industry to get them excited about it. So yes, I do see, you know, Quest 3 in our future, Vision Pro or future versions of the Vision Pro. So, you know, this is not yet a commercial product. It will hopefully be one in not too long and along with a more robust library of content. Even the Quest 3, I think, could do a very good job with this piece and it's definitely maybe one of the best candidates right now to have it released sooner than later.
[00:31:19.842] Kent Bye: Yeah, in terms of a location-based entertainment context, especially here, like, at a festival screening, the one thing that I would like to see is, like, more access to the corrective lens inserts because, you know, I have to wear glasses, and when I saw it, I tried to wear it with my glasses over it, which sort of messed up some of the head tracking, which made it a little bit more jitter. So I have the choice as to whether or not everything that I see is blurry or whether or not I discover some of these head tracking issues if I try to wear my glasses. It's a frustration I had with a lot of different Magic Leap demos where I've had to wear it without my glasses. This, I think, was actually close enough that it was better because it's more in the near field rather than the far field, but still just the same. I think it's one of those things where having those lenses, but also as we look at something going forward with the Apple Vision Pro, it's the exact same problem where you have to have corrective eye lenses to be able to see it. So in a location-based entertainment context, it's less than ideal because you have to do the additional onboarding and get people's prescriptions and like swap out all those different lenses. So anyway, I'd love to hear any thoughts on that.
[00:32:21.625] Paul Raphaël: Yeah, so I mentioned the Infinite earlier, where we basically pump 150 people an hour or 160 in Quest 2s for the time being. The Quest 2 is fairly accommodating to people keeping their glasses on. Correct me if you think I'm wrong.
[00:32:37.014] Kent Bye: It depends. A certain width, it has to be a certain width in order for it to be. Like mine's small enough, that's OK. But often the docents when I'm at these festivals will advise me taking it off. And I'm like, no, no, no, it's OK. So sometimes it's too small, and other times it's OK. It just depends on how wide it is.
[00:32:50.903] Paul Raphaël: Yeah. So I think you know in a right now we had no choice in the type of hardware we would use this Magic Leap 2 today is the best way to experience the storyteller and sadly it is very difficult to get access to those lens inserts so you know we do our best and sometimes we stack them on top of glasses if the vision accommodates without glasses and we recommend that. Certainly, you know, the primary way I think this experience will be experienced will be at home, right, even though that's not what it is today. And, you know, if you've got a device at home, then presumably you've either gotten those inserts or you're used to wearing it somehow, otherwise you wouldn't own it. So yes, in LBE settings, I'll do everything I can to get at least a few of those insert kits. Because, you know, even showing it to you, I realized it wasn't just a question of blurriness or comfort, but really some software instability or hardware instability that is introduced. So we're definitely going to be digging further into that. Which is one of the advantages of showing a piece publicly for the first time, well officially at least. And so yeah, definitely, like I said earlier on, it's all about accessibility, right? And if all the creative is going in that direction, then certainly we want the hardware and the presentation itself to be as accessible as possible as well.
[00:34:10.547] Kent Bye: Yeah, we did a little bit of debugging to see that in fact the glasses were creating like additional jitter into the experience that wasn't there for most folks. But yeah, I guess I wanted to give a few moments to talk about the Blue Marble experience that's also shown here as a part of the best of. And yeah, when I had a chance to see that, that was just the shot that I was looking for. You know, just an uninterrupted sunrise, sunset from the International Space Station. Yeah, just a really beautifully poetic piece. And so I'd love to hear a bit about what it was like to be able to get some of this footage in and be able to see it in real time and put this piece together.
[00:34:42.724] Paul Raphaël: Yeah, well, you know, in some ways, Blue Marble is the realization of one of the very first dreams we had when we had our very first VR camera up and running and captured our first demo shots. From that moment, we were on a quest to get our cameras in space somehow. It sounded absurdly ambitious at the time, but we never gave up and we eventually met with NASA and started a relationship with them. that led to the Space Explorer series, which started with, you know, as a documentary series down here on Earth, following astronauts as they go through their training. And we figured, you know what, as we build our capacity to film in space, well, we are following the astronauts that are themselves following their capacity. And we're also bringing our audience along on this journey of, you know, the first steps towards going to space. And sure enough, through that, we built some relations and gained the trust of NASA and various other international space agencies, including our own in Canada, and we were able to send cameras up to space. At that point, it was already like, if we can get one shot in space, just one, whatever it is, that will be amazing. We ended up spending three years up there and it turned out to be the largest media project in space of any kind, which is to this day mind-blowing. And not only did we get our one shots, we ended up getting over 200 hours of footage with the collaboration of over a dozen international astronauts. And, you know, all of this culminated in a couple of outings, so EVAs, extravehicular activity recordings, and most of that time was spent filming the ISS and different shots of the Earth. We filmed an entire spacewalk. But while we were out there, we got to capture three uninterrupted orbits around the Earth. So when you're on the ISS, it takes 90 minutes as opposed to 24 hours to do an orbit around the Earth. Half of that time is spent in darkness, so that leaves you with about 45-ish minutes, a little more than that, where you see the Earth. And we took those three orbits and we did a bit of a time-lapse with them, not too much, I think we're at 2-ish X acceleration. And, you know, coming from having done experiences like Strangers with Patrick Watson, where, you know, you're basically at an uninterrupted moment of presence with an artist performing, I think that back then, our ultimate version of shooting in space couldn't have imagined anything more perfect than an uninterrupted orbit around the Earth, when we had this dream of filming in space. And so, here we have it. Even though we've been releasing content from our three-year shoot in space over the last couple of years, in the form of those documentary episodes, in the form of the Infinite that I mentioned earlier, This is kind of the pearl, or these three orbits, so Blue Marble, Orbit 1, Orbit 2, and Orbit 3, this here being the first of the three, are kind of the pearls of this whole series in a way.
[00:37:51.700] Kent Bye: Yeah, so 90 minute total orbit, half of it in darkness, roughly, meaning that you have around 45, 48 minutes. And so you sped it up. And so it's basically like a 22 to 24 minute experience where you see a sun rise and a sunset. And it happened to be that on that day, there was also somewhere between a first quarter and a full moon. So somewhere in between there, you're able to see as the sun is rising, you see the moon setting. And then as you see the sun setting, then you see the moon rising. So I thought that was also really amazing. Moment to be able to see that as well and I actually watched it twice just looking forward and then I watched it Looking backwards and I just noticed how the light was changing over the course of the day and how the solar panels are moving And so yeah, there's a lot that's happening that is worth like multiple viewings but also just very simple and Zen in a way that you are not adding a any talking, maybe some music to give more of an arc emotionally. But yeah, just a really beautiful awe-inspiring piece that I've been waiting to see something like that. And I feel like it's one of the perfect use cases of VR to give me an experience like that that I will literally never have a chance to see that perspective. So yeah, just a really an amazing experience.
[00:39:01.035] Paul Raphaël: I'm glad you felt that way.
[00:39:03.236] Kent Bye: What was it like for you to be on the receiving end of seeing some of these images coming through?
[00:39:07.722] Paul Raphaël: Well, the entire time we were filming the EVA, I mean, well, the minute we started filming anything in space was a bit of a surreal experience. And, you know, just being in the presence of humans in microgravity, there's just something primarily you feel like you're seeing the first steps of humanity's evolution into something else, right? There's something quite powerful about just seeing the humans, you know, just flowing and talking to you and telling you about their experience being up there and how it's transforming them and just feeling so viscerally that energy. But then filming outside was also just a whole other type of mind-blowing. You know, the way we did it was we would see a two-dimensional image of what we were shooting.
[00:39:56.929] Kent Bye: Like an equirectangular flattened image of everything?
[00:40:00.253] Paul Raphaël: basically a rough one at that and what I would do is every few minutes we would because you have the limited bandwidth and a lot of that is just to get your live stream but we were able to download stills you know every once in a while and then I would manually stitch you know and pop in a headset and just like you know show it around we were mission control in Canada. So we did a few shoots, but in this particular one we were at the Canadian Space Agency, connected to mission control at NASA, and then they were connected to the ISS. I mean, we were all crying. We were basically all in tears as we were just looking at these stills. And then another moment, this is one of the orbits, forget if it's two or three, where there was such a insane synchronizing, because, you know, we had to calculate, you know, what you're about to do, right? You have for exposure, but also like what's the angle you want? Is the arm moving? What speed is it going to move at? What are you doing? Is anything happening on the ISS? And we had like drawn this like plan of all these things we wanted to sync up. And with, you know, again, you have to be so prepared for none of your plans to work out when you're doing anything like this. and yet all of the things kind of synced up, right? And we weren't even looking at the stitch footage as all these things, like a cosmic clock, right, that we had like tried to capture, right? Just knowing that it was all coming together was just a miraculous moment. And so, yeah, that whole experience is just, I don't know how I'm going to top that, but I guess, you know, trying to follow NASA now into the Artemis missions is probably the only way we can top that.
[00:41:37.727] Kent Bye: Have any astronauts who have actually been on the International Space Station and done a spacewalk, have they had a chance to see this VR experience of Blue Marble?
[00:41:46.590] Paul Raphaël: Yeah, so I don't know about the Blue Marble specifically, but when we launched the Infinite in Montreal, we then brought it to Houston. And almost everyone at NASA has gone to see the show. And so there are parts of the experience that are very similar to Blue Marble, not the uncut. full orbit, but definitely a lot of overview effect type shots filmed outside the ISS. And just, you know, all the content that was filmed up there. I think one of the reasons this project is even, A, existed, and B, survived for so long as a project, right, like with the support, because NASA was giving us so much, right, and they had every excuse to stop at any moment. And of course we had lots of support from a lot of people at NASA, but Some of our biggest supporters were the astronauts themselves. Whether it's because they had seen some of our earlier work and could extrapolate what the effect of actually shooting in space would be, or whether it was because they had seen some of the footage we had shot in space, because we shot for three years, some of them saw it before they even went up there. They were all very, very much supportive about the project. You know, this was the first time that they could share, right, what it was like to be up there with their family, with their friends, with their co-workers, the thousands of people down here on Earth that allow for them to go up there but will never actually go, with the larger public, which is one of the main reasons NASA and most the national space agencies exist, is to inspire. And for these institutions to even exist, they need the public support. And why do we go up there? You have hundreds of good reasons to go up there, if not more. And there's many good reasons not to. And I think the impact that these images have on people, I've seen all sorts of footage and films of people going up in space, shots of the Earth. And those have had an impact. And throughout history, they've had a huge impact. But today we're kind of used to seeing just about anything in 2D, right? And being able to get this sense of actually being there through this medium, I think is quite rare and precious and one of the reasons we've been so adamant in making this project a reality. And sure enough, that's how the astronauts felt, that's how a lot of people at NASA felt. And yeah, when they see the footage, we've been in their presence as they've seen a lot of our content as well, the content that was filmed up there. And honestly, they're like our number one fans because they've experienced it. They're the only ones who know what it's truly like to be up there. And they will themselves tell you that this is the closest it gets to being able to share that. And, you know, they continue to support us as we try to do, you know, crazy things that we still want to do, like getting up on the moon and sharing with millions of people what that is like, including ourselves. I mean, I just, I want to see, I want to experience that, right? And I know I'm not going up there anytime soon, so, yeah.
[00:44:48.644] Kent Bye: Yeah, I know that Edgar Mitchell, who was the sixth man to walk on the moon on Apollo 14, on his way back had this whole spiritual awakening, this Samadhi experience of this turning around. I actually did an interview with him at the Institute of Noetic Sciences back in like 2009. But he talked about how he was turning around and all of his job was done and to see the Earth, see the sun, see the stars, and just had this unity effect of seeing that all of what we're made of is out of these stardust. But a lot of astronauts have had that, and they've termed it the overview effect of going into space and having this third-person perspective of looking down on the Earth and seeing that we all come from this one place, and it sort of changes our perspective of trying to be in right relationship to not only the Earth, but to each other. And so, yeah, I'd love to hear if you've heard any anecdotal experiences of people watching some of these experiences and having a virtual reality induced overview effect.
[00:45:40.876] Paul Raphaël: Yeah, I mean, so many people are so profoundly affected by, I mean again, this content exists in a lot of forms and some of that is the documentaries, some of that is the Blue Marble orbits, some of that is the Infinite. So I'm just going to pull all of those together, but yeah, definitely. I know some people who go to the Infinite regularly as a thing to do, right? Just to go and wander through the space station and just live up there. And there's enough content where you can actually go three, four, five times and still not have seen everything. But it always ends with the spacewalk, you know, which is essentially a version of spacewalkers, which is available online as well. One of the main reasons we wanted to do this is not so much for the spectacle, although it certainly is a spectacle, it really is for the way it gives you a taste of that overview effect. It's hard to say to which extent it does that, but I think it does to some extent. Astronauts have confirmed it and people seem to be affected by it. It's what has drawn us to virtual reality actually to begin with, even though You know, obviously not every project is about the overview effect. I think there is something inherent to, you know, we talked about the sense of presence of virtual reality, right? Well, you talk about a sense of presence when you talk about Samadhi and meditation and these types of experiences too. You know, we live life, right, present. You can't not be present, you know, otherwise you don't exist. But you can be more or less in the moment, right? And I think that when we're being told a story, it pulls us in, it creates kind of a focus, right? And if you can have that focus and a sense of presence like you do in virtual reality, I think that's the main strength of that medium, right? Or any immersive art is when you really have someone's attention and you have them in a sense of presence and immersion, it can be a profound experience, right? And so that's kind of what we strive for in everything that we do. And then you bring, you know, space and the overview effect into it, and it's like, okay, well, what else? What else can you do, right? I mean, I think as a storyteller, who has been interested in having that kind of effect on people with their work. It's kind of the ultimate and you do learn some things from that as well that you can then extract and apply to other things, right? There's the reason you get the so-called overview effect is because it gives you a sense of perspective like you can't get in any other way. It is because it gives you a sense of the fragility of life and of just the sense of collective. Whether you like it or not, we are all connected by the fact that we are part of a single ecosystem. And you could see that in one glance in an experience like Blue Marble or like with the Overview Effect. So I feel some of the most powerful things that you can experience and they're almost like a shot of something that can wake you up and alter your course, right? The course of your life. I know it's certainly The astronauts, you know, these are the testimonies that they give. And again, while this is a facsimile of that experience, I think even a small jolt of that is probably priceless. And like I said earlier, the kind of stuff that I hope to be able to infuse with all the work that we do.
[00:49:09.865] Kent Bye: Awesome. And finally, what do you think the ultimate potential of virtual reality, augmented reality, and immersive storytelling might be and what it might be able to enable?
[00:49:21.367] Paul Raphaël: Well, I mean, we just talked about a big part of it. I think the magnitude of power that the medium has to communicate whatever it might be that you're trying to communicate. Of course, you know, we're still in a relatively early stage where we're still all figuring out how to maximally employ the tools that are at our disposal to tell stories in immersion and to have that kind of impact. But the potential is, I think, beyond any other medium before it. And by it, I mean AR and VR. Just basically volumetric storytelling, immersive storytelling. And that can be more or less interactive. It could even be a game. It doesn't have to be a film, right? It is, to me, pretty clear that it is inevitably going to be the main way we tell and experience stories at some point. It's hard to say exactly when. It's going to progressively become more and more influential. But as we see Apple enter the game with the emphasis on spatial computing, what does that essentially mean? It's that the way we've been interacting with the digital world, with most forms of entertainment, has been limited to an abstracted, two-dimensional form, a rectangle. That was not inherent to what the data actually ever was, right? Information is not limited to a two-dimensional plane. In fact, our most common interaction with information is in the real world, right? The world, physics of the world, everything we experience is information manifesting in what to us the most natural way is spatially, is volumetrically, right? And so I think we are finally graduating to the point where the digital realm can actually resemble reality and be more natural, be more human finally. So I don't see it so much as an evolution as I see it into growing naturally into what it always needed to be, right? Now once it gets there, We can then go even further because there's nothing limiting us to what we know today, right? And in a way, the abstracted two-dimensional form in which it has been for the last few decades is in itself a particular branch of what it could have been, right? And what it could remain. There's a certain efficiency to a list and to things that you can interact on the two-dimensional plane, right? We have had books for the longest time, magazines, right? but we've also had sculptures and theatre and architecture. Well, now I think the digital will no longer be as limited to this two-dimensional plane or even 3D has been a 2D representation on a two-dimensional plane, which is not quite the same. I think it's going to have a huge impact on everything that we do. Of course I'm mostly focused on telling stories and the more artistic side of things, but I do believe that even the way interfaces are going to evolve, or the way new tools are going to evolve, all those things are going to impact everything else. And I think that even when we do something like Storyteller, Of course, again, we're telling a story, as the name implies, but I think that hopefully we're inspiring people that are not necessarily just interested in telling stories, but could see other ways of using similar sort of methodology to do other useful things in AR or MR or even VR. So anyway, all this to say that I think it's an incredibly exciting time to finally see what has been a revolution, right, the digital revolution. I mean, we have an AI revolution happening in parallel, which is its own wormhole. But even the form that all these things can take is actually graduating to the point where we will more intuitively, naturally, powerfully interact with an essentially unlimited amount of potential, which is basically the digital realm. So it's a very exciting time. Yeah.
[00:53:16.675] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:53:21.872] Paul Raphaël: Wow. I mean, I have to just say that I feel incredibly grateful to be launching these two projects here. These are the two projects we've worked on the most ever. So, you know, five years on the Storyteller, even more than that on Space Explorers at large. But as I said earlier, these orbits were really a dream for close to a decade now. And to be having them both here side by side at Venice, my first time here, is pretty intense, pretty cool. And I hope as many people as possible will get to see this. I mean, it's great to launch it here, but, you know, there's only so many people who can see it coming here. And we're going to do everything we can to get the storyteller out there. And, you know, Blue Marble is already actually accessible on the MetaQuest. Blue Marble 2 and 3 will be coming out soon. So definitely seek those out. And if you're anywhere near where the infinite is showing, definitely come see that because I think the way we have combined the physical immersion of being able to free roam a large open space with the social element of being there with all the other... You mentioned that we are all made of stardust. Well, that's basically what the avatars are made of as well. And combining that with the lifelike immersion and presence of the cinematic virtual reality is creating a very powerful reaction, I think. If you take the same piece of content that people may watch at home sitting down versus in that context, we usually get a much stronger reaction, even though we get pretty strong reactions at home as well. So I highly encourage you to seek out The Infinite. We will be trying to have it in many more cities in the coming months and years. So we have one kit that's been touring for the past two years. We hope to have three or more by the end of next year. So yeah, if you're anywhere near Montreal until the end of the year, that's where it will be. And then we will be announcing other cities soon. So yeah.
[00:55:15.968] Kent Bye: Awesome. Well, congratulations on your launches of these projects here. Just seeing all the 43 different projects, both your Jim Henson's The Storyteller and the Space Explorers Blue Marble Orbit One made it to my top 17 of all the different experiences I saw. I think each of them are emotionally evocative for the Blue Marble in a way that's really just awe-inspiring and invoking this kind of, as much as possible, this overview effect. And also just all the different innovations you have in the Jim Henson's The Storyteller with all the different that you were starting to lay out. I think each page you start to unravel some of these different novel surprises, but also, yeah, just a simple fairy tale story that's very well told. And, you know, there's moments of mystery and confusion, but I think it's kind of designed to be that way as well, to have your own interpretation as to what it means. So, yeah, I just really appreciated that kind of exploration of that in this format as well. So, yeah, thanks for taking the time to help break it all down.
[00:56:08.471] Paul Raphaël: Thank you. It was, as always, a pleasure talking to you.
[00:56:11.913] Kent Bye: Thanks for listening to this interview from Fitness Immersive 2023. You can go check out the Critics Roundtable in episode 1305 to get more breakdown in each of these different experiences. And I hope to be posting more information on my Patreon at some point. There's a lot to digest here. I'm going to be giving some presentations here over the next couple of months and tune into my Patreon at patreon.com slash Voices of VR, since there's certainly a lot of digest about the structures and patterns of immersive storytelling, some of the different emerging grammar that we're starting to develop, as well as the underlying patterns of experiential design. So that's all I have for today, and thanks for listening to the Voices of VR podcast. And again, if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
