Your brain can fuse together so many different layers of reality so seamlessly that it’s quite flexible and capable of blending the digital and the real in a way that will soon be completely indistinguishable. The Dial at Sundance New Frontier blends together a physical installation with projection mapping on to a house on a table, and then audience members watch a narrative through a phone-based AR window. He’s also blending direction audio of dialogue through bone conductance headphones, and blending together a dynamic soundtrack with quadraphonic sound. It’s also a social experience in that there is one navigator and two passengers who are watching the experience, and the navigator controls the flow of the authored narrative by walking forward or backwards. There’s a lot of technological innovation that’s happening within The Dial, and director Peter Flaherty has thought deeply about the implications of storytelling with AR and produced one of the most technically-sophisticated experiences showing in all of Sundance New Frontier this year. I had a chance to have an extended conversation where we were able to unpack many of the challenges that Flaherty Pictures & Nightlight Labs had to overcome, lessons learned from immersive theater, the open questions about spatial storytelling and augmented reality, and how to seamlessly blend together many different layers of digital reality.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Photo by RYOT.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So one of the most technologically sophisticated experiences that I saw at Sundance New Frontier this year was called The Dial by Peter Flattery. So The Dial is a phone-based AR experience. And there is a table with a blank white house on top of it. And on that white house is being projected with projection mapping. You're seeing the baseline context. And then with the phone-based AR, you see other vegetation and 3D models and motion-captured actors that are unfolding in this drama. Now, it's also controlled by different people that are watching it. There's three different people with two different roles. There's two passengers and one navigator. The navigator is basically walking around counterclockwise around this table in order to control the unfolding of time, which controls the different sequences and unfolding of the actual story. And you're also wearing these bone conductance headphones that is basically sending the dialogue through the jawbone of your face, but your body kind of hears it as sound. But they also have this quadraphonic sound mix. And because you're doing this as a social experience with other people, you have the opportunity to be able to openly talk to them if you choose to. So technologically, this is adding just a lot of different things. It's like AR, projection map, interactive narrative, all these different layers of digital reality that are somehow magically being fused by your brain. And on top of that, he's telling this interactive drama through these tabletop immersive theater techniques. And I had a chance to talk to Peter Flaherty about all the different things technologically that he had to do in order to actually pull this off, as well as the compelling things that he finds about spatial storytelling and augmented reality and the future of immersive storytelling with these technologies like projection map, augmented reality, and immersive theater. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Peter happened on Friday, January 25th, 2019 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:02:15.840] Peter Flaherty: So my name is Peter Flaherty, and I'm the director of the Dial here at Sundance. And I am doing work with AR. I'm doing work with projection mapping in this piece and with interactive storytelling. And my background is in projections and interactive design for performances, large-scale performances that incorporate media elements and interactivity, and also for interactive installations in the art gallery and museum world. So I sort of come from this mixed world of performance, art, and then also VR and AR for the last five or six years.
[00:02:53.665] Kent Bye: Great. So yeah, I had a chance to do the dial this afternoon at the New Frontier section. And there's a house that has a projection mapped on top of it. And then I'm looking through an AR phone. And I'm seeing that house. But on top of that, I'm also seeing lots of digital characters that are engaging with it. But I found that it was interesting that usually when I'm looking at an AR piece, I'm just looking at the full scene within the context of that window. But because I see in my peripheral vision the larger context of this actual model with the projection map on top of it, I think it did something weird to my brain in terms of like maybe believing it even more than had I then know like how's there at all and it all would have been digitally mediated. So maybe you could just talk a bit about this blurring of the line of the digital and the real and how you think of that as an artist to try to play with that line.
[00:03:46.640] Peter Flaherty: I think that's a great point. When we first started this project, when I was conceiving it, I was thinking projection mapping and AR felt like they were both these digital media that they're kind of like a magic trick. They use a certain level of illusion in their creation. So they rely on the real world to create the platform for that illusion. So with our house and tabletop, we are using wood. It's in the shape of a house and a table. But through the projections, you create the illusion of scale and dimension and all these other details. And the same is true with holograms. Like, we rely on our real world to create the foundation for where they live. And then their dimensionality and scale and relationship to those things, like, create both, like, artifice of the real and also give them their substance and position in our world. So they seem anchored to us in real life. And so the two things felt like they were ripe for mixing because you get that sort of big-picture view and narrow-picture view simultaneously. And to just see, like, how a screen-based media like AR, where the light is emissive, would connect with projection mapping, where the light is refractive, and how those two things go together. And when we first did the proof of concept in February of 2018, there were no characters yet, it was just a really rough foam core model of the house with gaff tape and a couple of projectors on a scaffold just to see how it felt and it felt like the two things really melded perfectly because they were these two slightly opposite types of digital layers but the way that they combined in creating a total illusion felt really compelling.
[00:05:24.645] Kent Bye: Yeah, I'm wondering if you could talk a bit about your plan for this project moving forward, because we're at a festival here, you have the house, you have all the projection mapping, there's a lot of logistics and infrastructure just to even see this experience, but yet, because it's phone-based AR, then the question becomes like, well, couldn't you sort of port this in a way so that anybody could see it? So I'm just curious to hear your thoughts on where this project goes from here, if you see that it's really best to see only in an installation context, or if there's ways that you are thinking about how to translate this into just a pure phone-based AR experience.
[00:05:59.316] Peter Flaherty: Yeah, no, I think I have always been thinking about this as a downloadable at some point in the future where the hope is that it'll play some more festivals and we'll sort of maybe even build out a few more animated sort of an interactive sequences. I feel like we're in a good place right now but as with most projects I still have ideas for all kinds of other stuff that we could build into it that would be really interesting in the dramatic arc and just fun to explore, more Easter eggs, sort of more things for the passengers to experience, sort of ancillary bee stories and things like that. So I think hopefully there's a festival run or some sort of location-based experience type of run first, and then I think we take all the digital assets and drop them down into an at-home experience where you still get all of the interactive storytelling components of moving around, say, your dining room table where you place this story, you still get to move through it and determine how the story unfolds based on your body motion. And then also it's up to you to decide, OK, let's play it again. Let's do it a different way. I have it. I don't have to wait in a line or be on a wait list. So I love the idea that even people who've done it in a festival or people who are just at home can download it and play with it that way. So I see that as the next stage.
[00:07:16.975] Kent Bye: describe to me a little bit about the logistics for how people watch it. Because there's three people watching it at the same time. There's like a navigator and two, I guess, observers who don't have as much agency. So maybe you could just describe these two different roles with three different people watching this experience and what you were trying to experiment with there. What type of questions you were trying to ask and answer through this experiment in that way.
[00:07:41.232] Peter Flaherty: Yeah, I mean, I think if it's okay, I might even sort of pull back to set the frame a little wider, which is, I feel like there's three stages of the experience as you move towards being an actual participant. So the first thing you see when you walk into a room with the full installation is this glowing cube that's just changing colors and you don't quite know why. And then as you approach it, in the center of the cube there's a sort of band or belt around the middle that's clear, like a window that allows you to look in and you see other participants engaging with the story with the AR phones and moving around the table and going below the table and having fun. And then you grab the phone and become a participant and that's sort of phase three. And the idea was, how do we create a social multi participant experience where people can have the pleasure that I find most compelling about AR, which is that we can make eye contact. Ideally, we can hear each other so that we could hear each other's reactions or say, oh, wow, let's go do this. Let's try that. And so we came up with this. I had this idea that we could use these bone vibrating headphones to do our dialogue and still have it be spatialized and then do all of our music as like a surround. We'd ended up with a quadraphonic mix. And so the question became, OK, if you're doing multi-participant AR, how do you make it interactive for three people at the same time? And so we started testing it with one navigator who is able to move through the story and is the one who drives the projections, which can obviously only be one thing at a time. So they drive the projections in the story and so then we had these two passenger characters who are sort of along for the ride in that sense. And when we started doing more experimentation with it in the studio, we started to get feedback from people who were trying it who kept saying they loved being a passenger because unlike the Navigator, you don't have to stay in the scene you're watching. You can move around the table, watch that scene from other sides, you're completely free to move. Now you don't have the agency as the interactee in that experience, but there's a certain pleasure to being able to move around and sort of enjoy it from whatever angle you want. frame a reverse shot where you watch the scene unfold from the other side or go into another season and watch some of the, like there are two characters that are these young girls that live next door and they're frequently hiding behind a stone wall on the exterior of the experience. You get to sort of see the world from their perspective. So we sort of like to think that there's pleasure in both sides of the experience.
[00:10:16.735] Kent Bye: Yeah, I've only had a chance to play the navigator role, so I haven't done the passenger role yet. I hope to do it again just to see the differences. But as I was doing it, I felt like I was like a cinematographer in an independent film or something, and I was trying to capture the scene. But because it's on a table, it's a tabletop, then I found myself crouching down just to be able to get at eye level to see, because it's in the near field, so it's these little miniature characters that are playing out, and I feel like I see that a lot with an augmented reality is to shrink down the scale so that you can use the windowed phone and still get a full experience of entire character without having something be real life and then only get these little fragments of a body. And so it seems like the limits of the technology are forcing these best practices of creating these little tabletop experiences. But because every time I move from one side of the table to the next table, I was triggering the next scene, then I was also like, Minding and paying attention to make sure that all the story that was coming out had come out to my satisfaction And you know waiting for that pause to move forward and so there was a bit of responsibility I felt as a navigator to make sure that I wasn't trying to break the system and give a bad narrative experience because the problem with giving too much interactivity to a player is that it could actually make it a way worse experience if they decide to push the limits of their agency and basically break the experience. And then you're not really actually receiving what is trying to be told. So it could make it a bad experience for me and everybody else. And because some people don't have a problem having a bad experience for themselves, but when there's other people there, there's a bit of a social pressure to, in some ways, not deviate too far and experiment too far so that, as the navigator, I feel a little bit responsibility. So anyway, there's some interesting dynamics there. Once you introduce the social element, it actually makes my role as the navigator a little bit more conservative of how I express my agency.
[00:12:08.289] Peter Flaherty: Yeah. No, I think it's interesting. And we see that emerge in different social settings, too, where different groups of people will have a very different social experience of it, depending on what their relationships are. So I believe you did it with two people that were strangers to you. Is that right? Yeah, I thought so. We saw three people who worked at Sundance do it together. So colleagues, that experience was very unique because there's that level of day-to-day intimacy but not personal intimacy. And then you watch people who are all friends do it together. We've seen three family members do it together. And there's a very different dynamic. in terms of like how that level of concern about the other person's experience expresses itself. Is it something where you feel comfortable enough to take that risk to go backwards and go back in time? And we often see it with friends, it's much more comfortable. It's not like that person is judging you based on that choice alone. Whereas with a stranger, it is a very different dynamic. Some groups speak to each other during the experience, Can I go backwards? Or, oh, what if we go this way? So there's an interesting social dynamic and communication that emerges, because we don't see people having extensive conversations, but a little bit, because the open-ear headphones allow them to hear each other while also experiencing this rich, spatialized sound of the whole narrative. It's interesting to see how people experience a story in a social setting that way. But I also, to speak to your first point, I saw you at one point crouch down framing that stone wall in the foreground with a character in the foreground in front of the stone wall and then another one in the background behind. And I was really excited because it's like one of the reasons we chose the phone as the AR device in this case First of all, it feels very comfortable to all of us and I think it disappears in a lot of ways in our experience because it becomes so ubiquitous in our day-to-day lives. But also because the frame is there and that does feel like it's a nod to cinema because I do think that this is very clearly a cinematic AR experience and people say is it a film and I wouldn't say it's a film but I do think it's cinematic like I'm drawing on a lot of genre-based storytelling techniques to make people feel comfortable with the story to figure out how to create arc from scene to scene. and definitely in terms of framing, and that idea of foreground and background is expressed in the staging of the characters, and I'm always excited when I see viewers take that role on in full, so they're not just waving the phone around, they're trying to actually frame their experience in a meaningful way, and frame, settle, watch, reframe, try it from another side, I think that's really exciting.
[00:14:47.025] Kent Bye: Well, there is also the mechanism of you moving your body through space in order to progress the story forward. And when I've talked to different people about the differences between VR and AR, some people from AR say, you know, some people when they do VR, they're pretty static. And once they put the headset on, they don't actually move around a lot. And I think that's, to some extent, how the VR locomotion is not very comfortable. And so people tend to either do this teleportation of moving around. But once you start to teleport, then how much are you really moving your body around? And that in AR, because you don't have the issues of having the boundaries come up if you get too close, then you actually can see what their boundaries are because it's the wall and you know how to deal with walls as you walk around and so there's a little bit more freedom to move around but I've found that there's a bit of an emphasis of actually physically moving your body through space that is happening with augmented reality and I think I'm seeing from Immersive theater influences in order to actually have the story that takes place and you're moving your body through space But there seems to be a blending of that with an augmented reality And so I'm just curious to hear you talk about how you use bodies moving through space in order to move forward the story here
[00:15:59.995] Peter Flaherty: I completely agree with you. I mean, I think in VR experiences I've made in the past, because I feel like I'm a hybrid, I do both, recognizing the differences in the form, but also recognizing so many similarities. I think I always want to move my physical body as much as possible in any given experience. Because I feel like it connects you to the story and to the emotions. If I'm engaged in something that feels interactive and my body's moving, then I feel like that interaction is a richer experience. That I feel like if it's something where I'm just pointing or button clicking or flipping pages in a choose-your-own-adventure book, the stakes are lower. Whereas when my body's engaged, it engages all of my neuromuscular systems which are connected to my brain and my vision, my hearing, but also my emotions. That's the key for me is that I have feelings that are richer than when I'm sitting still or when I'm just turning my head. When I'm engaged in that way with my whole body, I think people engage more deeply with the story and the choice-making becomes a little bit more complex. Because you're not just doing a binary, suddenly you have a true 6DOF experience, and I think that really creates a richer experience. And I think with the capabilities of AR to map a world space untethered, even with a phone, you do have all of those technical abilities to move around. There is, in a sense, no limit. You sort of have to, as a creator, you have to frame the experience to make it seem like an open world is truly open world, but there's a lot of methodologies you can use to try to do that. And I think immersive theater is a really great example of there's so much that's going on on a stage that unlike film where you sort of dilate your eyes and go into a wide shot to take it all in, in theater you're studying details and when you add the immersive theater component you're doing that while moving your body through a space and I feel like with AR that's sort of my goal is like I think of it as adding details to a world so you have to choose the world you're foundationally working in and hopefully you can do it in a way that makes sense. And this one I've limited myself because we're building that world for the most part using the projections. But even if you were doing something that would layer over the real world, you choose an environment that you then place details in that drive a story forward. And I think that's really exciting because then you do have a three-dimensional physical reality that you can use to frame that experience while still allowing your participant to move around as much as they choose to.
[00:18:30.332] Kent Bye: Yeah, one of the things that I've noticed that's happening in virtual reality is that when you have the sun move, you actually get this sense of time passing. And so it's referred to as a zeitgeiber. And so as you have these indications within an experience that something is moving or dynamic or changing, then it tends to make it more engaging and more interesting. And so I found that when you go into a VR experience and there's nothing changing, nothing happening, then because this is an interactive narrative in the sense that you're giving the audience a decision as to when to move on to the next scene there has to be in some ways different cues that you're giving to the audience to say Okay, you could stand here for 10 minutes and there's nothing happening But most people don't do that because once dynamic things have stopped then you kind of move on to the next scene So maybe you could talk a bit about what type of cues you're using to sort of give that cue to the audience Yeah, absolutely.
[00:19:22.796] Peter Flaherty: I mean Well, one of them for us, of course, is the presence of the characters that, you know, a lot of scenes, I'll try to start with enough space that you can kind of move into it and the characters seem to emerge. And often there are groupings of characters. So you might have group A appear early and then only later when group B appears to have kind of a reaction moment where they're observing group A or vice versa. they kind of appear later, so your eye is naturally drawn backwards and out, and you sort of move from the close-up to the wide shot in your mind, and then usually your body will follow. So I think that's one strategy. I think sound is an enormous part of it, of course, so we kind of came up with this Jesse Garrison, our technical director, working in collaboration with Sage Lewis, our composer, came up with this elaborate audio control system. And Scott Hirsch is our sound designer. So those three together figured out a strategy for having scene music, scene sound design that's totally spatialized by sound effects and by character. So in Unity, all of the dialogue tracks are linked to the characters that are moving in physical space. So you have these layers of sound that are also giving you cues. And essentially at the end of every scene, the music tails. So it's almost like we're customizing the tail. It doesn't just fade out. It doesn't even go off. In fact, there's like two minutes of music that you could sit through at the end of every scene that is original. It never runs out, but it gradually becomes less and less interesting. to let you know that nothing else is going to happen. And that's nothing against the music, that was by design. You know, so first you see the characters fade, then the music starts to tail a little bit, and usually by that point, people have got the picture. And then, to bridge the scenes, they came up with this brilliant strategy for transitional music that It sort of has this complex fade structure where it's able to fade out of a scene into a transition that itself is controlled by your physical movement over a small arc of the corner of this table and then out into the next scene music. And there's certain delay times based on the characters and your body motion and timing as to when that next scene strikes precisely. So that it always kind of feels like you're just walking into the next scene exactly on cue. which took a long time to sort of figure out, but I think that's a key driver. You know, and then also just working with the time of day idea and sort of the natural ebb and flow of scenes. Every scene is like, you know, roughly between 45 seconds and a minute and 10 seconds. So it's all within a certain window. Once you get the rhythm, you sort of start to feel the piece and you can see people becoming more intuitive with the UI and also just with the rhythm of the piece itself and the story.
[00:22:14.833] Kent Bye: Yeah, it's interesting as you're talking about all the sound design because when I did the experience I didn't necessarily think about why the bone conductance type of headphones, which by the way was really super trippy to have these things like vibrating on my jaw and then to like hear sound but you know it'd be purely through the bone conductance. I haven't actually had a chance to experience that before so it was a weird experience to feel the vibrations of the haptics of that but to have my ears open meant that we could presumably could have talked but it kind of felt like I was in a movie and that it would have been rude to actually start to have much conversation in fact if they would have talked too much I probably would have been annoyed because I was trying to listen to what the story was but at the same time there was one moment when there was a scene that was happening underneath the table that I didn't even notice and then someone else noticed it and pointed it out and then I sort of discovered this whole other world that was happening. So it was like, oh, I'm glad that I had my ears open to be able to actually hear that that was going on. But maybe you could talk a bit about this strange bone conductance audio solution combined with this other quadraphonic audio surround sound that's happening at the same time.
[00:23:20.551] Peter Flaherty: Right. Yeah, I mean, so as I was thinking about how the layers of sound would work, I came upon these headphones. I heard about them somehow. I'm not sure how. But I knew that there was a problem for me if I wanted to do multi-participant because I needed to spatialize sound for everyone. Because the spatialized sound, especially for the dialogue, really gives people the cue as to where those characters are. And as you said, they're miniature so that you can really take in the full scene. So, you know, on the table they look like they're about, what would you say, six inches tall approximately? So, you know, sometimes the neighbors are kind of hiding under a tree or by a wall or people are on the left or the right. So the spatialized sound really helps clue people into where the scene is taking place. And if there's a shift in the middle of the scene to another character who's in a different location, it helps cue you into that too. So I thought, how could we do that, spatialized sound, where I want every one of the participants to have that experience. So if you're on the left side of the table and I'm on the right, I need it to be customized for each of us. And I just have this feeling, when I do VR and AR with full over-ear headphones, I feel very cut off from the world. And in VR that's sort of fine for me because I feel like that's the nature of VR as well, I'm going into a wholly manufactured, constructed reality that's not my own, or not the one I just left in my physical reality. And so to be fully immersed in a headphone is not a problem. But in AR, because there is this pass-through capability, seeing the other participants in the room and hearing them to some extent, I didn't want people to be shut off. So I found this technology. And we experimented with it and worked with my collaborators from Nightlight Labs and also with all of the sound team I mentioned before. And we started experimenting with those headphones and different speaker setups and trying to discover how it sounded. And what was interesting is that the brain really blends all of those sources together magically. You don't have, I don't think, other than the sort of tickle of the bone-vibrating headphones touching your jawbone a little bit, you have that little haptic experience. But otherwise, when you hear the dialogue and then you hear the music coming out of the quadraphonic speakers, your brain really mixes them together pretty naturally. You don't think to yourself, oh, these are two different sound sources, which is kind of an amazing thing about the plasticity of the brain in terms of how perception works, that it is capable of putting those things essentially in the same mental channel.
[00:25:55.118] Kent Bye: So when you say the spatialization of the dialogue, does that mean that depending on where I'm pointing my phone is going to somehow put a spatialized sound into the bone conductance headphones?
[00:26:06.578] Peter Flaherty: Yeah, absolutely. So we're tracking your phone as you move around the table in order to do the seasonal rotation. But even the passengers that are not actively triggering the projections are also being tracked so that we have a position for every person in the room. And essentially, that position defines where the system thinks your head is. And then so if you were, say, on a on the edge of the table to the left and there was a character to the right, you would hear them in your right headphone on the bone vibrating headphone. So you'd hear it coming from your right to clue you in. Or in some cases, some passengers really got to go crazy and go to the other side of the table and are watching the scenes over the top of the house and like whatever they want to do. And then a new scene starts and they need to be aware of either where the navigator is or when the scene starts, they can clue into where the scene is through that spatialized sound.
[00:26:58.737] Kent Bye: Now I'm getting to why the passengers might have so much more fun because they're able to really get that full experience of the spatialization. And I guess as the navigator, I was subconsciously cueing into that because I was reacting, but it wasn't like I was like, oh yeah, this is totally spatialized sound in that way. It's something that is below your level of conscious awareness, at least for when I went through it.
[00:27:19.545] Peter Flaherty: Yeah, totally, because you're experiencing it and there's a certain pleasure there, but because you are the center of the system's attention, yeah, you don't have to clue into it as much because you always know when you're moving into a new season. It's sort of like driving a car, you know? So you're kind of unconsciously aware of it. It just feels right. But the passengers, like, it's something that they're more reliant upon in some ways. But some groups we see clump together, and so they're always aware of what's happening and where things are. But we got a lot of positive feedback from early testing about how helpful that was for some of the passengers.
[00:27:55.185] Kent Bye: Well, because I've done so much virtual reality, there's a certain solidity that happens within VR experiences because the tracking is pretty solid and there's not a lot of jitter that happens within VR. But in AR, I feel like because you're having to do these SLAM algorithms, so the simultaneous localization and mapping, it has to keep track of these two things and constantly trying to approximate. And it's not like a precise process, so there's always some jitter that happens and sometimes, I rotated the phone from, you know, being in one orientation to be more of a widescreen, and then tracking started to be, like, wasn't too happy, and so then I was like, okay, this must be tracking off of something, so then I went back to the vertical orientation, but then there was other times when I was sort of moving around, and then I would see that it would, like, slightly get offset, and then sort of click back into place, so... You're doing lots of really sophisticated things with occlusion and you're tracking the house, but then you're able to see things like behind it. So there seem to be a lot of really sophisticated things that happen, but at the same time, because it's at the bleeding edge, it's not solid to the point that it's at the same level of occlusion that I might see in a virtual reality experience, for example. you have everything virtualized, and it's easy to control when you don't have to take into account what's real and what's virtual in that way. So just curious to hear about that process of trying to minimize that disruption of the breaking of the world together, but how to actually track using either ARKit or ARCore to be able to lock in to what the scene is, and then on top of that, layer that digital reality on top of that.
[00:29:23.505] Peter Flaherty: It's a massive problem and we made it far more complex on ourselves by creating an environment on the table that's constantly changing. So the shadows, as the time passes in the story, the shadows, for example, of the giant oak tree in the front yard of the house sweep across the yard and up the side of the house. you know, the lighting changes, the color changes, there's the glowing cube that you're inside of during the experience, which is changing color as well. And so essentially all the foundational knowledge that ARKit should have of the world around it, we're taking away by changing it every step you take around the table. So it was a massive challenge for Nightlight Labs and specifically for our technical director, Jesse Garrison, who's just like a mad genius to figure out how to figure out just how do you track in that scenario. And so I think he tried a number of different types of tracking paradigms and in the end it's a sort of hybridized tracking paradigm that combines a few different ways of tracking objects and elements in the world essentially I think that's as much as I would want to say without him he should speak for himself about it but it's a it's a pretty brilliant solution and he worked alongside Brian Chazalot especially in the final run up to Sundance to really make that system relatively solid and so particularly finding the way that I think the occlusion is a massive challenge And then figuring out, how do you essentially create parameters so that the phone knows when it gets off? Because that's actually one of the hardest things to do. It's about when should we do a big reset? When should we allow the system to recognize that it's off and make a radical change? Because you don't want to do that much. At the same time, if parameters are too narrow, then it will never make that radical shift. And so you might always be a little bit off. And so I feel like we've dialed it in to kind of an amazing degree, given the difficulties that we have. And so most of the time, I think the occlusion is pretty rock solid. But then there are these little moments where you see these sort of calculated drifts where essentially we're letting the system try to figure out if it's missed something and decide essentially between two possible states. Do you decide you've missed something and then fix it by moving the world subtly so that it doesn't feel like a jarring experience narratively? Or do you discover you've missed something and acknowledge that you've missed it enough that we should do like a rethink where it might snap back because it needs to actually be able to do that calculation. So there's a lot going on under the hood to figure out how all that works that's pretty sophisticated and in thinking about where we've come since the first time we did our first proof of concept was in February 2018 and where we've come with the tracking since then is like unbelievable.
[00:32:20.818] Kent Bye: Yeah, there was one time when I changed the orientation of the phone where everything started to shrink down. And then I switched it back, and then it self-corrected. And then I did see at one point where it was off, and then it self-corrected. And then I was actually really impressed. I was like, oh, wow, there's something going on there. But as you were talking about that, it makes me think of how we're entering this new computing paradigm. into this real-time computing paradigm from both the Unity or Unreal Engine, but also in something like machine learning, where it's less about you going in as a computer programmer, figuring everything out ahead of time, and just programming it. But to create a complex, adaptive system, machine learning has to have a flexible enough neural network architecture that is able to then have an experience of data. And then as it has that experience of data that's being labeled by humans, it's able to then make these different judgments. But it's much more. fails fast in the sense of it has to have a lot of different iterations, and then it sort of figures it out. But it's a different paradigm where you're being thrown into a situation. It's drifting or getting things wrong, but having those opportunities to self-correct and to have that real-time processing. And so, to me, it feels like it's a deeper philosophical shift for how these types of things are programmed because it's not like you just ship one thing and it's done and you have figured everything out. You have to create something that's dynamically adaptable, but also go in there and as a QA tester, try to break it and then see how it's able to self-correct. And as you are going through all these different seasons and everything's changing, then there may be these little edge cases where it's really difficult to do quality assurance when you have those many different changes that are happening. give it to human beings that are going to do things that you've never even thought of.
[00:33:57.521] Peter Flaherty: Absolutely. I mean, all of those things are completely true. We've sort of tried to break it so many times over because that's the only way you're going to learn what might happen. And truthfully, it really is hard to predict what people will do. And for someone who's just coming to something fresh, they might do something that you just think you just never would do in a million years, you know? And we saw that time and time and time again. And it just emphasized the importance of playtesting. And it's hard to show people work that's not finished. But we would try to do that really regularly. We sort of did playtesting on something that we built in May. We did some playtesting in July. We did playtesting in August. We did playtesting in October. We did playtesting in December. We did playtesting before we came to Sundance in January. That was where the learning happened, because then you do find those edge cases. And particularly with multi-participant, That was really where we'd start to find the edge cases because you do see those social tensions emerge as well. the responses for those things are so different than what people do when they're by themselves, which itself is so different from what you do when you're testing something you already understand. Because I think when you're testing something you already know or that you built, you recognize the limitations unconsciously. And you avoid them for the best possible outcome also unconsciously, you know, because you want the best possible outcome for the work. So I find that that's how it happens. So you have to be really diligent to try to come to it fresh every time and be kind of egoless to show something that you know is not finished to a bunch of strangers. You know, it's usually friends, but still it's, you know, people who are coming into the space and seeing something that you acknowledge at the beginning is not done, but it's worth it because what it teaches you is so critical.
[00:35:48.118] Kent Bye: Well, what it's really emphasizing to me is that there's so many really tough problems to do augmented reality spatial storytelling because there's all these technical issues that you could very easily get bogged down in just to get the thing to work. But, you know, let alone how the medium is going to allow you to discover these new storytelling affordances that you've never been able to do before. Now that you've showed this here, I'm just curious to hear from you in terms of what either some of the biggest unsolved problems that are left to be solved or where you see going forward from here in terms of the spatial storytelling of augmented reality, what's next for you?
[00:36:24.485] Peter Flaherty: The problems are myriad. I mean, the problems, I think, I'm excited to play with other types of AR devices and try to solve some of the problems that exist in sort of work within the positive and the negative that exist within headset-based AR. I think, you know, the headset-based AR has unbelievable capabilities. It also has some limitations that we're aware of. But that's exciting. I think I still want to try to refine some of the methodologies that I'm using for how the interactive aspect of the script works. I have ideas now for how what we call break points, where essentially you create points in the chapters where you allow people to leave so that it feels like it's a curated point, but it also feels like it's close enough to the moment they physically leave that chapter. So essentially, I don't feel like it feels like true storytelling if when they leave, you just sort of fade out of that one and fade up into the next. It needs to feel like the story is an organism responding to you, which as a creator who works in language essentially means I need to find a break point in the story, in the dialogue, or in the interactions, in the physicality of the characters, whichever it may be. And it's sometimes any one of those things, depending on the scene and the moment. But I need to find enough of those that it feels like when a person leaves, the given chapter they're in to go to the next one that it feels natural, but it also feels fast enough that they can feel the interactive aspect occurring. So it doesn't feel like it's too late in responding, that there's latency there. So that's a big thing I would still like to keep working on. I think I still want to work on how characters, how you derive the maximum emotion from them. And I think there's sort of two primary approaches. One is abstraction and one is photorealism, like on the two ends of the spectrum. many in between, but that's a sort of overview, and then in between those two things, there's a whole range of approaches, ranging from, you know, motion capture with a sort of painterly texture, like we use in the dial, to some of the abstractions you see when Depth Kit is used as a technology for volumetric capture, to the photorealism you might see coming out of Metastage, or what I'm excited about is taking a number of those different elements and starting to combine them together, so that character design is less about one thing in that respect and more about hybridizing some of those things. So, you know, you might have like a volumetric head and hands together with motion captured bodies that can go longer distances outside of the six to eight foot volumetric capture space. You know, so there's some interesting approaches to how you create characters that you emotionally connect with, because that's a really big thing for me with The Dial, is I wanted this to be a story that people were going to be excited about. I wanted it to be a story that they connected to the characters in an emotional way with, and where they were feeling the arc of the narrative and wondering what was going to happen. That said, it's a genre story about a car crash in the middle of the night, so right off the bat you know what's going on and why you're there. what you're trying to uncover and what you're interacting with as a viewer. But how do you create that emotional experience that feels really rich and cinematic? Because I think in both AR and VR, we all as a community have been trying to solve that problem for a lot of years. where we know that there's character-driven VR that can be really effective because the boundaries are relatively narrowly drawn. And then there can be demo experiences that feel like they're executing on one level of experience, but not necessarily delivering on a full three-act drama. And so I think trying to push those boundaries and being willing to fail as a storyteller in terms of trying to push it forward and figure out what works, I think is really important to me because I think We all know that it's incumbent upon us to find some of those answers for this form to fully mature. And in order to do so, we're going to have to break some eggs, you know, so we have to be willing to really push it and think really critically about what's effective and what's not so effective in terms of those arcs and structures.
[00:40:37.382] Kent Bye: Yeah, I've seen some augmented reality narratives where you're asked to essentially hold up a phone for a really long time, for like maybe a half hour or 40 minutes, and you're just like sitting there with the phone. For me, it feels like it's a bit of a bad experience to see a really long experience. But I found that as I was moving my body around, I wasn't minding that so much because there's something about, I don't know, maybe just the movement of the body that It didn't feel like I was holding the phone for too long. But I do think that one of the limitations of phone-based AR is that you do run into this thing where how long can you expect someone to hold up a phone in front of their face to see an experience and how to really optimize for that. So I don't know if that's something that you've tried to think about or optimize for in different ways or try to mitigate the impact of just standing there with your hands up in a way that's not comfortable.
[00:41:26.545] Peter Flaherty: I absolutely think about it. I mean, I think about that as a user as well, having done many experiences with that same sort of feeling. I mean, I think, I feel like at this moment in AR, there's no magic bullet in terms of the device that you're going to use. I think phone has its advantages. They're in people's pockets. That's a great long-term advantage. Your adoption rate is much higher potentially. I think headsets have the advantage of keeping you hands-free and not necessarily straining your body in that way. I do think moving your body, creating experience where you compel the participant to move their body is great because just kinematically your body feels much better when it's moving. You know, your muscles don't tighten up, you don't slouch into an uncomfortable position unconsciously. You know, I think that's really helpful to create an experience where that's possible and to have a technology that allows you to do so. You know, and that's the beauty of AR is most of it's untethered so you can do that. But I think about it a lot, and I mean, and I think, I do not think that phone-based AR is, at the moment, better than headset-based AR. I just think for this particular piece, I gravitated towards phone-based AR, and in this particular piece, I gravitated toward motion-captured characters, and it was for very specific reasons. So with the characters, I wanted the motion-captured characters to be able to move over long distances. That was part of the narrative, and I knew it had to be that way. Otherwise, it was never going to feel quite right. And with the phone, it felt like I wanted that field of view to be really large. And I wanted people to be able to actively flip between the phone in front of them and the projection mapped house and table behind them and be aware of those two different planes of digital existence. And I think when you feel the sense that there's a raster or a frame in a headset, you're less likely to be aware of that background, that background play in the house and the table. So for me, for this piece, it felt right. That said, I think this would still be a really cool experience. Some of the folks from Magic Leap went through it and they were like, wow, we love that. But we were also thinking like, wow, what would that be like in a Magic Leap? And I thought, you know, that is a really interesting question. And I think the experience would be a little bit different. But I think it would be just as valid and really exciting in a lot of other ways. So I'm excited to branch out and I don't think there's a clear answer yet. I would never say that one is better than the other. I think everything is still sort of evolving in the AR space when it comes to the hardware. which is part of the fun, frankly. For me, being involved in these bleeding-edge fields is always just so pleasurable. There's so many unknowns and you're trying to figure out big, big structural problems that you're sort of borrowing from cinematic screenwriting, you're borrowing a little bit from immersive theater creation, you're borrowing a little bit from stage plays and three-act structure that was invented, what, 400 years ago in Elizabethan theater? I mean, you're sort of pulling, and it's in the same way that the remediation of technologies happened over the last, whatever you want to call it, take it back to the Greek theater, you know? The way that Opera was remediated into television. The first television system was a camera at the back of an opera house, and it was a live feed. There was no remote broadcast delay. That was the remediation of opera into television. Now we see television remediating the theater when we see live musicals streamed. And it's the same thing. This is going to take learnings from other forms and incorporate them into its own form of storytelling and then create a bunch of other new rules that are wholly unique for this form. And I think that's sort of the beauty of remediation and telemedia technology is that you do see this evolution that is a little bit of a snake eating its tail and then also branching out into other snakes, you know, so that you get to see this lineup. And I think it will be the case that the next generations will look at VR and AR as among their primary mediated art forms and entertainment forms. But I don't think that it means the death of the cinema or the death of television or the death of theater. I think all of those things will live on. you know the extent to which people gravitate towards them will be personal choice and age and generation and it will be this interesting flux and I think we'll get to see that evolve over the next, you know, over the rest of our lifetimes.
[00:45:54.412] Kent Bye: Yeah, one of the things that comes to mind is that when I think of a concert and then you have people holding up their phones looking at the concert through their phone and I'm against that in the sense that I'd like to just be present to what's emerging and not mediate my experience through trying to look at it through the screen because I feel like it changes my experience of that and so I feel like there's a bit of that has been happening in our culture. And I feel like augmented reality could potentially start to shift that in a little bit. And the reason why I think that is that when I was at the Spatial Realities art show curated by Jesse Damiani in Los Angeles, there was a piece by Sutu, who's, I think, one of the most amazing augmented reality artists. But he had this piece that had like 10 different panes. And you would look at it, and then you would hold up your phone, and then you would see what would change. It took me a while to figure out how to experience this piece of art because I was like I needed to look at it and then hold it up and see what was emerging and then I found like okay I need to actually look at it and take it in first of all and then change it and then see what changes and then when I would put the phone down, then it would change the way that I had seen it in the first place. And so there was like these different shifts, and I feel like there's these different shifts where it's actually calling for me to pay attention to things that I might not have noticed had I just looked at the piece of art without it being augmented in any way. So I feel like there's something magical about augmented reality that's asking us to do this, to pay attention, to then put this layer of augmentation on top of it, And then to be able to go back and look at it without the augmentation and then look back again and to see how it's shifted. And there was one scene where there was a silhouette of somebody in the doorway. And I was like, oh, I just had bought that that was a character within this augmented reality version that I had been seeing through the screen. And then I looked away and I was surprised to see the character was still there and then none of the other characters were there. And then I looked back and all the characters came back. And so it was like this similar A-B testing of being able to see and notice what is there and then look at the screen and then look at what is real again. And I found myself Experiencing this piece. I was looking at the screen for maybe 90 to 95 percent of the time But when I go through it again now that I've seen it I may spend more time actually taking in what's happening in the projection map Seeing what I notice and then seeing how that changes as I look mediated through the augmented reality and kind of jump back and forth but I feel like there's something about Augmented reality is asking us to pay attention to our surrounding world and to change our relationship to that world. Because as you start to overlay these different layers of reality on top of the real world, it's going to change your relationship to that, and it may change the way that you see that.
[00:48:30.537] Peter Flaherty: Yeah, I think that's absolutely true. I mean, we get a lot of feedback when people leave the piece that they don't know what's projection and what's AR, which is for me great. That means that the marriage of the two is working, you know. And when you do look at the two separately, there's like some great pleasurable moments. There's a moment where the police arrive and you see their flashlights in AR, they're kind of volumetric. You see them like kind of foggy in the night, like they're cutting through a low fog, you know. But then if you look at the projections, you notice that there's also a circle of light on the ground. That's the beam on the ground, but it's there like a ghost ship. You know, there's no indication that the character is there in the projection. There's no shadow, but there's that flashlight moving around. And so we're kind of playing with those ideas of these different digital layers that are creating a holistic reality for you. So it's like these two forms that are both about illusion combined to create a third illusion. And again, like the sound, your brain just puts it all together because it essentially obeys the laws of physics, you know, as you understand them in the real world. And so you buy it as one holistic piece. But I think the idea of the way that AR asks us to reframe our reality is really true. And it's exciting because if you choose the right details, you only have to add a few to reshape your understanding of the whole of physical reality. That you don't have to, like in VR, you don't have to create a whole world that's completely believable. If I add one element to the actual physical world, IRL, around me,
[00:50:01.833] Kent Bye: It can change my whole perception of it or a perception of others in relation to me or vice versa great and and finally, what do you see as the ultimate potential of augmented reality and What am I people to enable?
[00:50:18.438] Peter Flaherty: I mean, I think we can all see that AR has so many different applications. Everything from remote surgery to heads-up displays, training, things like that. There's so many possibilities. I think my focus is still in storytelling. I still think that storytelling and visual art making and sort of creating these projects that ask people to reframe how they're taking in the story of their own life. That's really compelling to me. That's my primary lane, I think, in this way. So I think, for me, I am looking to continue creating You know experiences that mix some of these other elements like the projection mapping is an interesting one. There's a few others that I'm interested in trying to play with a little bit. Architecture is a really big one. I'm interested in increasing the scale. Interested in working in spaces where public and private are quite blurred. So like on a city street for example. Having stories that emerge from a private space into a public space and back. And I think figuring out how to create really pervasive narrative experiences that position your participants in unique ways relative to each other in the story. So they have to think about themselves in relationship. to their own world and their own reality and their own relationships. I think that's really interesting because it goes even deeper than simply watching a film and having to empathize with a character on screen, which is a powerful experience. It goes a layer deeper, though, because if it is truly pervasive, it starts to feel like you really are inhabiting self and not other. I'm not empathizing with the character on screen. I'm empathizing with the character of me that is suddenly involved in this narrative that expands my sense of what my immediate reality or surroundings are. And I think if you can do that in subtle ways, it can be really emotionally compelling because suddenly you're asking someone to believe that they are in a different world or to believe maybe that they are in a different body. And I think there's so many permutations of what that could be. and that the subtlety of AR allows you to draw on so much of the foundational reality around you and then modify certain details that create those problematic spaces and those rough edges that ultimately lead to those hard emotional questions and transformative experiences, right? Like that's what stories do for us is they position us in the lives of others and use that as a way to reassess our own lives and choices and emotional relationships. I mean, it's like when you go back to the Brechtian theater, it's all about stimulating that outrage that makes you walk out of the theater and try to foment change in your, in your world around you. And I think that's ultimately where we're all headed and what better place to do it than with a story that can exist in the world around you, but with this other layer added to it that problematizes it in these beautiful emotional ways.
[00:53:33.171] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:53:37.972] Peter Flaherty: I'm just excited to be a part of it. I have so many great relationships in this community with so many different people that are working hard to figure out these hard problems and it's so much fun that there are so many big questions left unanswered and people are making such incredibly bold choices to try to solve some of those problems. I mean I look around just in Sundance this year alone and I'm completely inspired by some of the pieces that I'm seeing and some of the stuff people are playing with and the colleagues I see working. It's just like, I love coming back to this group because everybody's thinking big and taking big risks and making really incredible work.
[00:54:15.835] Kent Bye: Awesome. Great. Well, thank you so much for joining me today. So thank you. Thanks so much for having me. So that was Peter Flaherty. He's the creator and director of The Dial. So I have a number of different takeaways about this interview is that first of all, well, to me, the big takeaway is how you can set up all these different layers of digital reality and how your mind will just automatically fuse it together without even really thinking about it. but technologically in order to actually achieve that took a lot of technological aptitude. A lot of things that I was seeing at Sundance in terms of these augmented reality storytelling experiences was that there were actually these installation pieces which were very site-specific. I mean this is a piece that had a big glowing cube And Peter was saying that it was really like these three different phases that you were experiencing this. So you'd walk in into the New Frontier Central and you'd see this big glowing cube. It was changing colors. You're hearing a sound coming from it. You walk up and there's like this band of a window and you can look in to see these three different people that are walking around this table with a house on top of it. But they're looking through these phone based ARs and they're clearly seeing some sort of narrative experience. And then you actually become a participant within this experience. And so you've, you've kind of built up this context, wondering what the story is all about and how are these things are going to actually be fused together as you are watching it as an experience. So there's a lot of moving parts here. I mean, there's a lot of just technological innovation they had to do just to even get this done. But I wanted to just comment on a number of different things. First of all, there's the actual installation aspect of these augmented reality pieces. I think in the future, the real power of AR is going to be context dependent. So in other words, it's going to be able to detect what's happening in your environment. It's going to be able to respond to it and perhaps do things that are contextually relevant within these different aspects of your life. And coming up with what those cartography of contexts are, I think is a huge open philosophical question because it actually gets into what are the domains of all of different human experience and how do you decide what context anyone is at any given time. especially because there's a relationship between the location and the context, but there's also things that are happening internally within some individuals that could put them into completely different emotional context. And so the individual's context and the environmental context are two things that I think are one of the big open problems with augmented reality. But as soon as those types of contextual information have some sort of framework to be able to understand it and detect it, then I think eventually we're going to see augmented reality start to be contextually aware and be able to be responsive. to that, but because that's still a very difficult problem, a lot of what you're seeing at Sundance experiences is that if you're able to control the context, then you're able to push the boundaries of what's possible with the storytelling medium and really start to explore some of the affordances of the medium. And I'll go into Jester's Tale, which I think had a completely different take in terms of a little bit more of an interactive, narrative-driven experience, where this was mostly a passive experience in terms of that it was a pretty on-rails narrative experience, where you were controlling the unfolding of these different sequences, and you have the opportunity to walk forwards and backwards. But that's essentially just kind of remixing what part of the story you're going to get when, rather than you actually engaging and changing the narrative. And I think You know, a lot of immersive theater also does that, where you are kind of a ghost, where you're not actually participating. And so there's this spectrum between authored narrative and generative narrative, and this was definitely completely authored. Although you were able to remix those authored pieces, if that makes sense, rather than you being able to actually change anything that is actually unfolding. So I think that that's one existing modality that we're seeing from immersive theater. And I'm not necessarily sure that that's the most compelling format. I think the real interesting things for me, at least, is when you start to be able to engage and participate. But I think there's always going to be a role for those authored narratives. I actually did have a chance to run through this experience twice. And the first time I was the navigator and I was actually able to control the flow of the narrative and I made sure to listen to every single aspect of the narrative as it was unfolding. The second time I saw it, the navigator wasn't as concerned about what the story actually was. I think they got a little bit bored as to what was happening in the story and just kind of like started to kind of skip and actually miss a lot of what the narrative was unfolding. What I got from that is that the things that I remember about the narrative were the embodied interactions of taking a gun and throwing it in the water, or a crash, or these two little neighbors that were sneaking over the walls and eavesdropping. And so there's a certain aspect of starting to get a sense of these ecosystem dynamics of how these different characters are related to each other. And it was more about that that I took away rather than the specific things that were being said and communicated through the language that was being said. This is taking a very didactic approach where there's these 40 to 50 second sequences of dialogue Which they're trying to actually convey a lot of information very quickly But I found it difficult to track all the specific details of what was happening with each of the different characters Something like sleep no more and then she fell is completely nonverbal where there's very little speaking at all and most of the communication that's happening is through like this interpretive dance where a there's a level of ambiguity as to what's actually happening, but you can start to get a sense of the relationships between these different characters based upon how they interact with their bodies with each other. So I also think that there's something interesting about the things that actually stick with me are those spatial embodied interactions that are unfolding less than the actual specifics of what each of the characters were saying to each other. especially because the construct of this narrative is that you're walking through these four different sides of a table and you're getting like a whole different season. So you're kind of like jumping through presumably three years of the story unfolding and you're checking in season by season to see how things are changing and unfolding. So I think I would have to actually watch it again as the navigator to be able to be in full control as to what was happening, to be able to maybe get some of the deeper aspects of what he was trying to say with the narrative. But I also found that in some of these immersive theater pieces, like there's a speakeasy in San Francisco, and it's very different to walk in into a middle of a scene and leave before the scene is over because there's something that once people start to speak, you almost feel a little bit of an obligation to see what is unfolding and what's actually said. If you come in late and you've already missed so much context that it's sometimes very difficult to understand or you may have missed a really critical part, in some ways having like interpretive dance makes it so much easier for you to kind of come and go as you please and it's easier to pick up and figure out what may be happening based upon someone's really visceral embodied interactions with each other. So this is one of the challenges that Peter is facing in terms of like, if people are moving in the middle of the scene, then where's the break points? And how do you organically have the piece respond as if it was an organism? Because that's not actually the intention for most of these narratives that have these different vignettes is that you're really kind of meant to see the whole thing unfold. And it actually makes it a worse experience if you move around. And I think that's interesting because there's some experiences in immersive theater where you can actually have like a way better experience if you just feel like you can have the law of two feet and just move anywhere you want throughout that space. But there's in some ways feeling a bit penalized if you try to remix as much as you can. I mean if somebody were to just watch this experience and to kind of run around the table like three times like super fast you would basically miss everything and so there's this trade-off between giving the audience control over the narrative versus you actually having something that's very authored in the sense that you just passively consume the way that you're meant to experience it and so there's something about like what is the role of that user agency and the authored narrative when you kind of have this like tension between like is it actually make it a better experience if you decide to express your agency and maybe it does because if you get bored or whatnot but in this case there's these extra layers of social dynamics where if you do it with strangers and do with people you know then there's all these other specific considerations like I as a navigator was very interested and concerned about delivering a quality experience as the author would have perhaps intended it without trying to break it or give a bad experience in some ways. Whereas the navigator that I had as a passenger wasn't so concerned about that and was more concerned about like how they felt in the moment, or they were in some ways very impatient to see how things were changing and unfolding. And sometimes they would go to a scene and not see something happen within a few seconds and then be like, Oh, okay, well, I'm just going to go to the next one. When, you know, if they just would have waited for a few more seconds, then the scene would have started, the characters would have come out, and they would have seen something that was changing. And so, yeah, there's a lot of different things that I'm taking away in terms of like, okay, what is this actually happening in terms of the narrative aspect? Technologically, I think it was absolutely one of the most advanced and brilliant things that I saw at Sundance, but I don't think it was necessarily the most amazing narrative that I saw there. But I think there's still a lot of really interesting, fascinating open questions that Peter is exploring here. And moving forward, I think one of the interesting things about a piece like this would be, it would be really interesting to see if you were allowing each of the people who were experiencing it to record what they were experiencing and then have other people watch that perspective. Because I think that a piece like this, I would expect to see that you were seeing a lot of the very similar things amongst the three different people because it was the same dialogue, but Let's say that if you were on one side, maybe you would hear dialogue from these neighbors that you'd be able to get a little bit of information that would be creating this fear of missing out FOMO, having many different characters and different perspectives. Like if you were able to move around and listen in to different characters and have that as a user agency choice to see like what are the inner thoughts or inner dialogue. And if you were able to actually record what was happening in the context of your view, then you could have many people record their own perspectives. And then you could, after you saw this experience, then you can see what other people experience. And so you get a little bit of this, like, okay, how do you actually piece together these many different pieces of something? But having actual choice of if you're actually going to give the audience the ability to move their body through space, then is there somehow an adaptive narrative that allows them to tune into the perspectives of each of the individual characters that then would give them a different flavor of the experience, especially if they did it time and time again? Because this is an experience. If you did it like 10 times, you would pretty much have the same experience, except for if you had people that were mediating and going forwards and backwards in time. But more or less, each vignette is pretty much the same. So that's some of my thoughts on the narrative. One other final thought that I have is that you're physically moving your body through space, which I think is a very interesting, compelling aspect of AR. And I think that we're starting to see what that means in terms of immersive theater pieces like Sleep No More or Then She Fell. You actually get this different experience of space. There's something about the tabletop, though, that I think that you have a little bit of a different relationship between spatial storytelling when you're looking at something that's like this small scale. I think it has some impact, but it's not nearly as impactful as if you're completely surrounded by that architecture in that space, let's say like either an immersive theater piece or in virtual reality. Because you're looking through the context of a small phone, then the affordances of the storytelling within AR makes sense to miniaturize things down into like this tiny tabletop type of experience which i think that is in some ways the limitations of the field of view of the ar headsets as well as like you're looking into this virtual world through this very small window that's about the size of your phone so your body's moving through space has a little bit of a different experience than i think eventually where you have a better field of view within the AR headsets. And as the phones go away, then it's going to be completely immersive. And we're going to be completely surrounded by architecture in a way that you could start to really connect these large scale aspects of a story in your physical spaces, which I think is going to be pretty incredible to be able to remix and change it. Because I do think that AR is asking us to reevaluate our relationship to the objects that are around us. And because this had many different layers, it was a little bit of this trippy, like, through your peripheral vision you will see additional layers of reality as you're looking through your phone and then you're listening to the bone conductance as well as quadraphonic mix and so there's also all these different other layers both on audio and visual that are being fused together. So what Peter is saying though is that as you're moving your body through space you're actually activating different aspects of your nervous system, your emotions. It's just more active and more engaging when you're physically moving your body around, then you have the capability to have a much more visceral emotional experience overall. Peter actually goes into this a little bit more into a panel discussion that I hosted later in the week, talking about the trends of AR storytelling with virtual characters. And we'll be airing that here in the episode coming soon. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast, and if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and I do rely upon your donations in order to continue to bring you this coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
