Jessica Brillhart’s Vrai Pictures & Superbright launched a spatial audio platform at SXSW called Traverse, and they premiered a spatialized audio experience that turned your body into a mixer. As you walked around a space, then your movements changed the relative volume in a spatalized mix of the different individual tracks of a multi-track recording of Elvis’ “Power of My Love.” If you walked closer to the back-up singers, then that you could hear the subtle nuances of their performances. Overall, it gave you the feeling that you were able to walk around the recording studio while this song was being recorded, and it’s opening up new ways to experience some of your favorite music. Traverse won the Special Jury Prize for The Future of Experience at this year’s SXSW Virtual Cinema.
Brillhart is a former principle VR filmmaker for Google (Voices of VR #291 & #502), but she set off on her own with Vrai Pictures. It was working on a VR project on Beethoven’s Fifth Symphony that gave her an opportunity to walk around the Philharmonia Orchestra as they were playing their music. This was such an immersive experience within it’s own right that it inspired Brillhart to expand her thinking about the potential of spatialized audio with multi-track recordings. After seeing a prototype demo of the Bose AR Frames at SXSW 2018, then she eventually started collaborating with Superbright in creating an entire mobile phone-based application called Traverse that aims to be an open platform for creators to be able to create and distribute spatial audio experiences, which has integrations with the Bose AR platform with either the Bose AR Frames of Bose Quiet Comfort 35 noise-cancelling headphones.
Brillhart gave a Convergence Keynote at SXSW where she gave a live demo of her Traverse platform, but also announced a project called “Into the Mind” where she’s collaborating with Myst, Riven, and Obduction developer Cyan Games as well as with That Dragon, Cancer developer Numinous Games. They’re going to be exploring the inner mental workings of some of the great scientific minds to explore a form of knowledge representation and the evolution of ideas in what Brillhart describes as a cross between Powers of Ten and Google Maps, but with information rather than geographic maps.
I had a chance to sit down and catch up with Brillhart on March 11th at SXSW, the day of the debut of her Elvis piece and a piece where you’re exploring Mars (The Arm of Insight) on the Traverse platform. We talked about her evolution from film to VR film to immersive audio, and how audio has always been a critical part of all of her work. She gives a sneak peak of a project that’s debuting at Tribeca that features a nine-track audio recording of Yoyo Ma playing a piece by Bach that’s going to be spatialized throughout an entire building. She covers how game design is becoming a bigger part of her experiential design repertoire. Finally, she talks about the dynamics of the birth of a new medium and how our existing systems may not be able to fully contain the complexity of spatial computing, which means that we may need some new systems and infrastructure including spatial audio distribution platforms like Traverse.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Brillhart mentioned immersive audio artist Janet Cardiff who creates “sound sculptures” and this video shows her ‘The Forty Part Motet,’ which is the type of project that can now start to be translated into a spatial audio AR experience on the Bose AR platform.
Here’s some excerpts from Brillhart’s SXSW Convergence Keynote
#VR pioneer and founder of @vrai_pictures Jessica @Brillhart stopped by #SXSW 2019 to deliver a Convergence Keynote. Check out some of the highlights. pic.twitter.com/03mlZznZze
— SXSW (@sxsw) March 16, 2019
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So on Monday, March 11th, 2019, there was the opening of the South by Southwest interactive experiences that were in competition. There's about 26 different experiences that were there at South by Southwest. And I had a chance to see all different experiences over my time at South by Southwest and One of them on the very first day that I wanted to see straight away was Jessica Breilhardt's new experience called Traverse, which is using the Bose AR QuietComfort 35 headphones. It's basically this platform that allows these spatialized audio experiences. And what they did was to be able to take these tracks from Elvis and to create this music experience that took all those individual tracks and they spaced them out in a room so that when you're in the immersive experience, you can walk around. And then as you walk closer to those different sound sources, it's almost like you're using your body as a mixer and you can hear the backup singers or the instruments or Elvis, wherever you're standing, you can have this very specific audio experience. It kind of recreates the feeling of if you were actually in the recording studio just walking around. And it's quite an immersive experience. And I think there's so much potential for where this could go. And so I had a chance to talk to Jessica Brillhart about her company, Vray Pictures, as well as her journey into doing more and more of these type of immersive audio experiences and her decision to actually go out and build this platform called Traverse. So that's what we're covering on today's episode of the Wisdom of VR podcast. So this interview with Jessica happened on Monday, March 11th, 2019 at South by Southwest in Austin, Texas. So with that, let's go ahead and dive right in.
[00:01:52.518] Jessica Brillhart: My name is Jessica Brillhardt. I've been on this podcast like five times now. No, three. It's third. I think this is number three. Yeah. Yeah. Happy third. Yeah. Welcome. Yeah. Hey, good to be back. I am the founder of Vray Pictures, which is an immersive mixed reality content studio. And we do all sorts of immersive media. So I come from Podcast Your. You would hear that I come from the 360 VR space when I worked at Google on Google Jump. And I use that rig to explore the medium that way. I still love VR, still thinking about what I can do in the future with that. We also do augmented reality as well. We are thinking about intelligent systems and how they play a role in all of these things. And most recently we launched a spatial audio platform called Traverse, which explores the potentials for the spatial audio medium that is imminent and is in fact already here and has been here for quite some time. So I just like to say it's like radical experimentation. We like to dabble in all of it and see what we can do and how to mix different technologies together to approach what this medium is all about.
[00:03:01.857] Kent Bye: Cool, yeah, I think the last time I saw you was at Sundance 2017, where we did an interview. Since that time, you were working at Google as one of their VR filmmakers that was really exploring what was possible with the medium. You've since left to start the Vray. Is that how you say it? Vray, yeah. Vray. And you've had a number of different experiences that you've created. What have you done since then, since last time I saw you and had you on the podcast?
[00:03:24.805] Jessica Brillhart: Oh my gosh. I think we had already released Beethoven's 5th, which explored audio as a perception in multiple ways. So it was using Beethoven's 5th, a performance of Beethoven's 5th by the Philharmonia Orchestra in London, conducted by Esa-Pekka Salonen, and then we explored various modes of what people may experience if they're deaf. what people might experience if they have tinnitus, both we integrated sub-pack haptics to it, we explored the way that the Voyager probe was able to hear in space. So we kind of brought a bunch of those things together and I actually had, I told someone today that I didn't really realize that I was a huge champion for audio through my work, but when I look back, audio, spatial audio has always played a really big role In what I do, I was a big proponent of it when I was working at Google, especially as Jump was being developed. I told the engineers, even with the first film I made, World Tour, or Experience, I should say, that it was really important for Spatial Audio to be supported because I feel like it's the glue that binds us to the visual that we see. So if it's just the visual, stereo it's not very convincing but if it's a visual and spatial audio you tend to believe that you're actually there and that's the component that some people just completely shun or dismiss and they shouldn't because I think that's the thing that makes something truly immersive.
[00:04:45.980] Kent Bye: So how did you do the audio with a symphony because there's lots of different instruments and so was it specialized with ambisonic audio or what kind of recording were you using to
[00:04:55.400] Jessica Brillhart: I mean, it's a lot of different, lots of mics. Lots of mics. I'm not, myself, an audio engineer. I get to work with really great ones, though, and they'll probably give you better answers than I'm about to give you, I'm sure. But the, yeah, it really involves getting a sense of where is the audio being emitted, what's the kind of vibe you want to get, where are we going to be placing the rig or the visitor as they come in, and also talking to, especially since we're working with the orchestra and Esa-Pekka, talking to Esa-Pekka about what he would like in terms of quality of sound as well. So my understanding of sound is fairly limited, very much compared to him. So really just working with the best people and getting a sense of what would make the most sense in terms of recording, it shifts a little bit each time, for sure.
[00:05:37.248] Kent Bye: I think last year at South by Southwest, there was the announcements that Bose was working on these AR frames. They had like a 3D printed version. And then this year they're actually launching the AR frames as well as putting like the different sensors of magnetometer and gyroscope and accelerometer within the headphones to be able to actually do these spatialized audio experiences. And so maybe you could talk a bit about your journey for how you got connected to Bose and started to launch this platform that is debuting here at South by Southwest.
[00:06:06.403] Jessica Brillhart: It's funny. I actually reached out to a guy named Matt Neutra over at Bose. Actually, we were in a bit of a bind with Beethoven's 5th, which was going to have its North American premiere here. And it did. But we were struggling because we were like, well, we need headphones. But I really wanted to get Bose noise cancelling headphones. I'm like, those are the best. I always use them for this piece. We need them. And we were struggling to kind of find a way to get them in the kind of quantity that we wanted. So I looked up whoever I knew at Bose, like I was on LinkedIn, just like, do I know anybody? And this guy, Matt Neutra, the year before at South by had said, Hey, I saw your talk. It'd be really good to connect. And I didn't respond because I guess there's too many social networks and I can't keep up with all of them. And so I reached out and Matt and I got connected. They were super supportive of that install. And then Matt said, well, come on by the Bose AR house. I was like, Bose AR house? What do you mean Bose AR? What does that even mean? And a lot of companies are like, we're like doing VR now and like AR and it's great. You kind of take them kind of half seriously because you're not sure what it means or what it is. And then I went and I was really blown away by their demo, which was sincerely just somebody telling me that the restaurant was behind me. Not me looking at the compass on my phone, but literally just the restaurant is over here. And I'd worked on Glass at Google just from a creative standpoint, just in terms of marketing, thinking about ways of telling that story. And it was really hard because they kept trying to make it more complicated than it was. The simplicity of it was that you could actually use audio to help direct people and connect them with the world as it always has done, but in a more direct way. And I just found that fascinating. Like, they were embracing that. And they also, they had a whole like column of devices. I remember this very well. And there was one that I was like, it's that one. It was like the, this neck thing. Cause I was like, that's the weirdest looking one. So that's the AR device. I bet you. And they pulled off a pair of Ray-Bans and they're like these. Again, it's just like, wow, okay, so you've already thought about the way that people are actually wearing things, and about the needs that they have on a very basic level, and the ways that immersive audio can help fix that. I went to Bose, I talked to all their engineers, all of them are fantastic, and they said, well, what would you do with this if you could? And I've been working with the Elvis Presley folks for a little bit now, and coming up with another project with them. And I said, hey, look, I have been researching this guy you might know named Elvis Presley. And I wonder if he's ever done a multi-track recording, because wouldn't it be great if you could actually walk up to him in audio? Like, maybe that's enough. Maybe you don't need to really see him, because we've all kind of seen him, or we've seen representations of him. Maybe just hearing him is good. And all of them were like, yeah, just ask them, which is like, OK, go ask Elvis's people to use his music. And I asked Authentic Brands Group, and I asked Sony Legacy, and I said, hey, I really think there's this project that we should do together. They said, yeah, OK, well, once you have a prototype or something, show us, and we can go from there. And what I thought was, OK, we all make these one-off bespoke things, right? A lot of us do. And they're great, and they happen, and we're like, all right, good. See you next time. And I thought, well, what if there's a lot of music out there like this, a lot of audio out there like this, that has a dimensionality that hasn't really been translated or communicated, but has always been there, and just needs sort of an activation switch or something, with very little on the part of either the creators or the owners or whatever to do, other than to deliver us the stems or the multi-track or something that they've done. Films could do that. Documentaries could do that. Field recording folks could do that. Podcasts, wink wink, can't. So I was like, well, heck, let's make a platform, which is a bit of a shock to me internally. And I was like, OK. So I talked to my close friend and colleague, Igal Nasima, who runs Superbright. And he's about as weird as I am when it comes to this stuff, except he's more in the engineering space, which is a great combination for both of us. And I said, I want to do this spatial audio platform. What do you think? He's like, yeah, absolutely. Let's do it. So we built a prototype. It worked great. We figured out a lot about what was working, what wasn't working. Specifically, you have to have, we wanted to have a visual on the phone that helped to anchor at least the audio experience. We still rely on that a little bit, right? Album covers, music videos, track listings. We love seeing some sort of representation of what we're hearing in some way. We showed the prototype to a bunch of folks, including some folks at Bose. They were like, this is great. Keep going. Showed it to the Elvis folks. They were like, yep. This is awesome.
[00:10:39.541] Kent Bye: What was the demo track that you used?
[00:10:41.102] Jessica Brillhart: We actually used a piece that was actually recorded, a multi-track experience that was recorded at Bose. Because we didn't want to go down the licensing track just yet. We just wanted to see what was possible. And so we used something there. I had been talking to Yo-Yo Ma, and he had just recorded the Bach cello suites already. And again, it was very easy for them to say, OK, well, don't release it, but try it out. And so I put Yo-Yo Ma in the space and let people walk up to him to see if that was interesting. And it very much was. And I was using just a nine-track recording. So he was mic'd nine different ways. So it was like, create a Yo-Yo Ma audio object and have that be something you could maneuver around spatially.
[00:11:16.861] Kent Bye: Are you doing the room reflections as well? Or are you just having the raw audio that's already embedded within that?
[00:11:23.647] Jessica Brillhart: It's a bit of both, yeah. We kind of just see what we get, and then we decide, like, what's the glue that binds that together? Do we need to add, like, room tone that's not there? Most times they do record some of the reverb in the room just to have it for the studio recording as well. So it really depends on what we get. I think with Elvis, we were working with Ant Food, which is the creative audio studio based in Brooklyn, and they help work on the Elvis tracks as well as the Arm of Insight experience, which is the Mars Lander experience. And I believe I saw a stem that said like tone or something, but I don't know if we use that or not. I think it was just there in case we needed it.
[00:11:57.934] Kent Bye: Yeah, so I had a chance to try both of those audio experiences on Traverse this morning at South by Southwest and to me it was like I was walking around that room and it really helped for me to see different tracks that were happening. So you take the different tracks from the Elvis track and then you space them out so that as you walk spatially into that area you have like either the the drums or the backup singers or the strings or Elvis and so I was able to walk around and almost like I was like using my body as a mixer to be able to get a special mix of the track but depending on how I was moving through that space it was giving me the sense of like oh this kind of feels like I'm in a room where they're kind of seeing this live and I'm experiencing it as if it was happening but I'm there.
[00:12:43.533] Jessica Brillhart: Yeah, exactly. I actually, when I was making Beethoven's Fifth, I had the opportunity to walk around a practice session that they had with the rigs, the jump rigs, because I wanted to make them feel comfortable with it. And I had the opportunity to just, as they were playing, just walk around. them which was bizarre because like who gets to do that so I was trying to keep that down and be like it's fine it's cool play it cool but I was walking around and feeling like it was so amazing the space just felt so alive and it was such a different way of experiencing music and sound I was like that would be awesome to give someone that. Of course, the first thing in your head is volumetric capture, and there should be 6DoF, and there should be interactivity. But sure, of course, it could be. There could be all of that, or one or two of that. But then when this hit, I was like, oh, no, you actually don't need any of that. I mean, you can give people a really simple sense of where they are in the scope of the space, but you don't need all of that to feel that good about walking through an orchestra, or through a rock band, or through music, or any kind of audio. I also really am proud of being able to give these other session musicians some credit, you know? Elvis worked with some amazing musicians and one of my favorite tracks, Power of My Love, is this guy named Ed who played harmonica and he is the best. And he's just going for it. And I actually just stand next to Ed and still listening to Elvis. You can still hear Elvis in the distance. But you just hear Ed just wailing. And it's wonderful. And we don't call him Harmonica. We call him Ed. Because he deserves it, you know.
[00:14:22.528] Kent Bye: In the music that I experienced, was it the exact same master tracks that were used or are you diving into the archives that are like using these different takes and trying to, you know, create a new version of it? Like, what's that process for you to create the best spatialized experience, but also remain true to the source material that was actually used?
[00:14:42.212] Jessica Brillhart: So we definitely, we were using master stems from the track. We've been talking with Sony about, you know, can we bring in some deep cuts? You know, there's many Elvis albums that have been released in the past few years where they've included takes as opposed to the masters, just to get people more excited about the album being re-released again. So they are very much embracing his methodology and how he wrote and how he practiced So I think there's definitely a lot of potential for working with that. I actually think that stuff resonates Well, it's a different kind of experience But it resonates very well with people when they're able to acknowledge that this person that they've heard in a pure form Isn't perfect because I think that's a very human thing, imperfection and nuances. We don't cut out any of the breathing, obviously. You hear him laugh at himself when he hears the playback. You hear them kind of chatter in the beginning of the track before anyone goes. All the breathing that Ed does in his frantic way as he's playing harmonica, you can finally hear it. So you do get this sense of breath and life that drives these people's talents. I'm really a big fan of Janet Cardiff and her speaker installations that she's done in the past, which also very much embrace this, where she basically has a set of speakers in the room, and as you walk up to them, it singles out one singer. And so you can hear it as a communal experience, like with everybody, or you can just go directly to one individual and celebrate that individual. What's great about this, I thought very hard about how do we bring powerful, emotional, immersive experiences into the home? Festivals are great, but how do we actually deliver immersive experiences to everyone? And I think that this kind of stuff is very powerful, very emotional, but the file sizes are tiny. Not like VR where it's like massive, it's like tiny and all you need really is a pair of headphones and a phone. And you can map any of these experiences to any space, to a classroom, to your living room, to your kitchen, to your backyard. You're traveling and you're by yourself and you're like, I kind of want to feel like I'm with a group right now. There's a way to do it. We've had people come in being like, I want my entire album collection to be like this. I want all my MP3s to be like this. Why can't Spotify be like this? Why can't iTunes be like this? Why can't there just be this? And I'm like, well, because we had to build it first. Like the idea of like, can I walk inside a film scene? And it's great because it takes away a lot of the tension points that VR has had. As a creator in that space, I can say it's been a struggle to have to recompress, re-edit, re-export visuals for a billion different devices. And if you're trying to sync up different devices to play the same video feed or VR feed, they crap out or the resolution's bad. But the one thing that has always been consistently great is the audio. So I was like, well, why battle that when you can embrace it and actually say, okay, you know what? Maybe this is the way that people understand the potential of immersive. If we really want everyone to realize the value of what VR can be, or should be, or immersive can be, or should be. this is the way in.
[00:17:58.403] Kent Bye: Yeah, just the fact that it's like a Bluetooth headset where you can take phone calls and it looks nice, it's easy, and I think the way that it projects audio directly into your ear is really elegant. To me it just feels like, oh yeah, I definitely want these to play around with, but I feel like Traverse and other types of experiences to be able to get away from screens in the way that there is gonna be more conversational interfaces, more AI, more ways that you can start to speak into these glasses that, because they're connected to your phone through Bluetooth, you could use it to take phone calls, you could use it to talk to AI assistants, you could get information ambiently, you could have social interactions with people. So I feel like there's so many potentials that coming here at South by Southwest this year to see a lot of the apps that were there and to talk to some of the creators and to see what you had created and When I think about immersive media, whether it's VR, AR, or immersive theater, the thing that's different is bodies moving through space. And as bodies move through space, how does that modulate your experience? And so I feel like your experience is tapping into that. You're creating a platform so you can actually move your body through space, and that's going to modulate what you're experiencing and have that interaction by using your body as a controller.
[00:19:10.623] Jessica Brillhart: For sure, yeah. I mean, it's sort of, again, it's that we all kind of want people to start thinking about that stuff anyway and start doing that stuff. So it's a really interesting way to get people excited about the potential for what that is and what that could be. You know, we're also doing a piece at the Tribeca Film Festival with Yo-Yo Ma called Into the Light, which actually maps Bach's Cello Suite No. 2 in D minor to Spring Studios, which is where they hold the virtual arcade, or I guess the immersive arcade. So you start on the sixth floor and then you work your way up to the roof. And it's essentially you moving through the movements. And we think a lot about the physical experience. You know, how does your movement release this music? How does it feel when Yo-Yo's too far away from you that you can't get to him? Or he's right next to you and you can. What's that like? How is the way in which you move something that we can consider and have audio be responsive to? And so talking to Yo-Yo about the physicality of Bach, the mathematics behind something like Bach's work, is also something we're considering as well. It's very, very mathematical. And seeing how that and the physicality that Yo-Yo has experienced just in performing Bach's music a billion times over, it's beautiful. It's going to be really fun to put that together for sure. But it's something we're thinking about. What we're doing at first with Traverse is really working with artists that we feel could really push this in interesting directions. And we're actually releasing it tomorrow, which would be March 12th, on the App Store so that anyone at home who's lucky enough to have a pair of Bose Frames or QC35s can download the Traverse app and experience the same experiences that they had here at South By, or did not have, and they're just at home somewhere and they want to try it out. So we built the app from the ground up to actually be something that can calibrate to their space. And we're able to obviously bring that to festivals as well. But the intention is always to allow anyone anywhere to do what we have the great fortune of doing here.
[00:21:10.993] Kent Bye: So imagine that you're going to be announcing this tomorrow in your South by Southwest Convergence keynote. Maybe you could tell me a bit about what you're going to be covering tomorrow in your keynote here at South by Southwest.
[00:21:21.840] Jessica Brillhart: Sure. We're going to talk about Traverse. We're going to do a live demo of that, which should be fun. Well, I should say, I'm going through an immersive 101 course. It's not just 101. It goes to, I think, 106. that follows my journey as a traditional filmmaker making sense of how to create in this space, which goes from, you know, my frames to worlds, viewers to visitors, transition from a language perspective to what does that mean to connect a visitor to a world, how do we look at engagement and attention, stuff we've talked about a billion times over. Thinking about the visitor herself, how she thinks, nostalgia, memories, and content that I made around exploring that. Deep Dream included, Weather Channel included, Beethoven's 5th as well. And then I talk a little bit about how hard it is. You know, at a certain point, I think I found myself really wanting to do better than I had been doing, and I felt that I was running really fast, but I was looking around and realizing that where I was at that stage, it felt like I was forced to run in slow motion. which is frustrating. And I also wanted to work with all sorts of technology, not just one. Because I felt that the technology is very inspiring, and they provide great tools, but they're not the goal. And I wanted to get closer to the goal, and not be beholden to the technology in order to do that. So I talk about free pictures, what that means to me, what that represents. I talk about a couple of key lessons, I would say, relate very much to the projects that we're doing, which is Traverse, We are creating a platform called the Los Angeles or the LA River AR platform with River LA Gary Partners, which is Frank Gary's team. And it allows for the community of Los Angeles to connect and interact and educate themselves about the history and the potential of the LA River. But it's the first round of an urban development tool that we're developing. So we're doing that, and I'm talking a little bit about that, and what it means to dig deeper, go beyond just, oh, we're going to think about how to build stuff. It's like, well, you've got to get people on the same page here. You've got to have them have the same understanding as the planners so that everyone has the same information to make the best decisions, the most educated decisions. So how can AR help to bridge that gap? And think very much about the community and how to bring them into the conversation the best way we can. And then I'm also talking, you're gonna be excited about this one, I think, because this one is like, it's virtual reality. It's a VR series. And it's something that I realized I had to do with game designers that were great at making emotional immersive games. One particular company did great stuff in the past and is doing great VR now. Another one did a more recent emotional game and is also doing VR now. So when I looked at these creators, I was like, oh, they're doing great games, but they also are doing great VR. They're exploring VR. So I'm working with Cyan Worlds, which is Rand Miller's company. They're a company that created the game Myst and Riven. Their latest VR venture fully realized is Abduction, which did very, very well. And it's also with Numinous Games. They're the creators of the game That Dragon Cancer. And they've made a game, I believe a VR experience called Untethered for Google Daydream. And we basically sat down together and I said, I have this idea, we all talked about it, and it was like coming home. Which makes me think that maybe I would have been a really good game designer, but I ain't there yet, and so I'm still in this immersive space. But I think it's all game design anyway, and I think that's something I'm talking about during my talk as well, is that it's all game design. So working with great game designers is actually very important, or at least trying to learn a little bit about it. So we're all working together on a series called Into the Mind, which is reimagining the biopic in VR. So it's allowing people to explore the mental workings of some very unique individuals who have existed or do exist. on this planet. In a way, it's a bit of Powers of 10 meets Google Earth a bit. Like, what if some of the greatest wonders of the world were in the minds of those who inhabit it? What if you could keep going, not just stop at street level, but just keep going through someone's head, see how they think about reality, how they dream, what the uniqueness of mind means in terms of how they experience the world. and think about their own reality and their place in existence, which is heavy stuff, but we all have touched on that, and I think the cerebral nature, the inherent cerebral nature of this series is something that really excites Ran and Ryan and myself, and we're just gonna go for it. You know, just make the best damn VR stuff we can, and then we'll see whatever technologies around us support it. and obviously constantly be aware of it, but we're just going for the medium, we're going for the goal, and allowing for tech to be the thing that supports us as we move forward.
[00:26:15.967] Kent Bye: What are the artifacts that you're using in order to show someone's thoughts? Because basically you're, in some ways, proposing to create a memory palace of someone's entire lifetime, of the evolution of their thoughts or their work. And people have artifacts of letters or pieces of work they did, time stamps that can put you in a specific place in time for where they were. So there's things you can do, but what does that look like? What does that feel like? Because that's recreating an entire consciousness of an individual. What does that look like?
[00:26:45.174] Jessica Brillhart: Yeah, I mean, so I once did this sort of exercise with Stephen Hawking. I was gonna do a VR thing about him. I was asked to. Lucy Hawking is an amazing woman. I got to know her. She wanted to create VR around her work in autism. And so we got to know each other. I helped sort of mentor her through that process. She ended up making one piece, which was really, really nice with the Guardian. And she and I had talked and she said, you know, it would be really wonderful if you had an idea for my dad, if he could maybe write something. And I said, OK. No pressure. I'll give it a shot. And the first thing I thought about was if I ever was stuck in a room with Stephen Hawking, not knowing who he was, truly, I may not want to go near him. I wouldn't be immediately like, oh, let me talk to that guy. He obviously is ready to talk to me. Like, it wouldn't be like that. And so I thought, well, but his mind is so beautiful and wonderful. And there's so much that he thinks about. and ponders and he's been so successful at getting that out and he struggled so hard to be able to still communicate that to people. Well what if you could just keep going, like why don't you just go in to see just connect with him in here instead of having to rely on this. So, that sketch was, you know, day in the life of Stephen Hawking going to buy cereal at a supermarket and he's just surrounded by the infinite nature of cereal and consumerism and just being like, okay, got it, you know, on that one. Or telling someone else to go get it. But the second is... It's a bit Freudian, which is I think kind of weak sauce in the sense of like psychoanalyzing someone, but the idea was, okay, so it's consciousness, pre-consciousness, and subconsciousness, I believe, right? So it's this idea that consciousness is like how we relate to reality, pre-consciousness is like dreams and memories, which I thought, oh great, machine learning, intelligent systems can recreate maybe his voice as it was when it was there and then have it slowly transition to this mechanical voice and there would be some sort of visualization along with that. And then thinking about the uniqueness of mind. What's in the depths of that? How does how he thought about the world revolutionize the way that we can experience the world in an entirely new way? Does it change the physics? Does it take us to a completely new place? Does it visualize something very complex that hasn't been visualized before? For him it was mostly, can we get some sort of equation that we can then just dive into and to really just express in some beautifully dramatic way? And that, could we be left in space at the end? And then we have to find our way back to earth and find, you know, the idea was to map some stuff to earth and then be able to find people on earth and so on. So like it would be cyclical. It wouldn't be like, okay, you're done. Like press that button. It would be something that could be eternal and forever. But yeah, I think that's the general idea, you know, thinking about what are we actually trying to uncover here from a truth perspective, like with say Elvis Presley. wink wink, it would be like really just a question about fame and how fame affects a person. He was the first major rock and roll artist. He had to deal with fame in a way that he had no way to offset. There was no one to tell him, this is how you deal with it. This is how you do your Instagram feed. There was none of that. He just had to roll with it. And he did, but it affected him greatly. And you think about the Wizard of Oz and how that whole idea of like, okay, there is this perception of what we believe this person to be, but if you pull it back, Who is it? Who do you find at the end of this yellow brick road? So for him, it feels more like that's the narrative, right? As you're on this journey to try to find the king. And at the end, who is the king? And along the way, you just dive deeper into something that feels a bit strange, but familiar in ways. And then when you get to the end, you learn about how we feed this fame beast. So it doesn't have to be something that's very, you know, visceral and aura based, but it could be something that's a little bit more direct and more of a commentary on what he represents to us and helps us learn about ourselves in the process as well, so. I don't know if that gives you a good, I mean it's a lot, but it's something that we're all exploring and thinking about in great detail now and we definitely have some really great ideas that we're really excited to explore and experiment with. We feel it's really important to put ourselves in a development phase and really start to hash out some of these pretty cool ideas and Ran's done his. a fair share of experimentation, as is Ryan. So it's going to be fun to be able to cobble all our stuff together and say, OK, what do you got? What can we do? And our experiences are so vast and wonderful that there's so much really awesome potential. We're all very excited about it.
[00:31:18.559] Kent Bye: Yeah, that's amazing. It reminds me of a lot of the stuff that I've been thinking a lot about lately, which is a memory palace of all space and time, but specifically of the archive of interviews that I've been aggregating in both VR, AI, mathematics, philosophy, and consciousness. all these different separate podcasts that I've been working on, like how to put them into a spatialized memory palace that can allow you to explore these ideas and concepts. And the thing that I've been coming to is that it actually represents this fundamental philosophical shift from reductive materialism, which is very linear, into something like Alfred North Whitehead's process philosophy, which is looking at things in terms of processes and pattern relationships between each other, and that it's more of like a metaphor of an ecosystem that you can imagine the graph of the internet or the sort of network graph analysis but it's that level of complexity that's non-linear and cyclical but also it's more about those relationships between those things and that as you start to take music that has been very linear you're sort of breaking it up into these ecosystems that you can start to actually spatially walk around and have a different audio experience of, but in terms of knowledge and information, the tricky thing is like, if you think about things in terms of concrete objects, then you think about, oh, they said that at this point, this is what they believe for all their life. But that's hardly ever the case. It's like you're from a mathematics community. They're really interested in how ideas evolve and grow from the community perspective. So they will do a prosperography of 800 people looking at how these conversations were happening and how did this idea first come about. In mathematics it's very easy because it's got a very specific notation where they can really start to pin it down. There's still conversations that are lost to the historical record, but you can still get a sense of the trajectory of ideas that were recorded and shared to the larger community for the first time. I see that same level of rigor when it comes to historical analysis of the evolution of thought of mathematicians are going to start to take those same insights and practices and start to do what it sounds like you want to do, which is to take these great minds in these communities. I would hesitate to say great minds because it gets into a myth of the great men of science, the great men of math, because it's actually usually a whole society and an ecosystem that with salons and people that may have been contributing through conversations that weren't necessarily archived and documented. And so even within the philosophy community, for example, they're trying to find ways to bring in and expand the corpus of different texts that they're using to be able to talk about specific times to get away from that linear way of thinking about it, but more of a ecosystem approach. For me, I see that this is like a fundamental shift that the spatial computing paradigm of VR and AR, all these immersive and experiential technologies together are shifting us away from the linear thinking into more of those complex ecosystems.
[00:34:08.231] Jessica Brillhart: Yes. Yes, I would say that sounds pretty good. You know, we're all a product of our influences and the many things that we've done before. I feel that very much now. I think I may have said this to you, but I'm eerily calm and I don't know why. And it's the feeling of somehow everything that I've learned and done so far has adequately led me to this point where I'm able to create these projects and work with these great people and have conversations with great people like yourself. And I do very much believe in that ecosystem, personal ecosystem and cultural ecosystem. But I feel like, you know, one of the things I do want to mention that I will be very, very upfront about in my keynote is that we are of a generation that has embraced game design very naturally for a long time. We are women and men who have grown up with computers and the internet, and we've seen it before it was a thing and then after it was a thing. So we've seen transitions. We've seen changes. And from an ecosystem standpoint, we're also never really the people that companies will say, you should go. I mean, I got lucky, right? I was in a company that was developing these technologies, but they don't necessarily, old systems tend to want to be old systems forever. They just want to be the system. And the immersive medium cannot be contained, not in the traditional systems. And that's why it's failing in a lot of respects, based upon the standards of old systems. And the worst thing we can do, and I actually learned this from Geoffrey Hinton, he said this, the last film I made at Google before I did the VR thing, the traditional flat A film, was about artificial intelligence and machine learning. That never got released because it was 20 minutes long and it's Google, so why would they? But it was important and I met some really great thinkers in the space and I didn't know how to end the film. And Geoffrey Hinton said something that I was like, that's it. That's the end. Where he says, he's like, we don't know what's going to happen. We don't know what the future holds for this stuff. Like we simply make this stuff and we hope that we didn't make any mistakes. But the worst thing that we can do is to assume that things are going to be the same forever. And I think That's as true to any medium, whether it's AI, whether it's filmmaking, which I think has been very steadfast in its systems, whether it's tech. It's not disruption that should be what we're scared of. It's believing that these systems should be the systems. forever. We have to evolve. Not because it's something that would be fun to do. It's because it happens. That's just how it works. And I think cultural shifts, ecosystem shifts, all of that feed very much into that. And the process and the beauty of seeing that stuff transition is something we can all appreciate.
[00:36:55.782] Kent Bye: Beautiful. And I'm curious from you personally, like what type of immersive experiences do you want to have?
[00:37:03.224] Jessica Brillhart: I would love to go to sleep. I would love a vacation immersive experience. I actually very much want to play that vacation simulator real bad, but I know it's not going to be a nice little vacation thing. It's going to be something very funny and lovely, but not what I want.
[00:37:18.163] Kent Bye: You want to actually go to a beach.
[00:37:19.384] Jessica Brillhart: I actually want to do an actual experience. I want to just go, yes, and read a book. You know what I mean? Just lie on my couch and read a book. or a beach and read a book. I'll read a book anywhere at this point. I just, yeah, I kind of, I want, you know, I love, I love this stuff. You know, I do. And I love making this stuff, but it's all inspired by the experiences we have and just having a little bit of a reset would be nice.
[00:37:44.812] Kent Bye: Well, imagine that you're about to launch this big thing, and that, you know, when I look at the evolution of the VR as a medium and an industry, there's people like Felix and Paul who, they have this vision for what type of experiences they want to create. And they have to go out and, like, build the cameras to do the stuff that doesn't exist yet. And that, you know, Open Frameworks is another tool that is trying to very specifically explore algorithmic depictions of these mathematical equations in spatialized ways. for you there was a certain moment it sounds like when you were walking around the beethoven's fifth where you had this spatialized experience of audio and then at some point you had decided that you wanted to explore this but in order to do that you actually had to like build this whole platform called traverse to go from being creator to a technologist platform creator in order to enable your own creative expression but also to enable it for other people as well so i see that as this dialectic between the technology that's there, and the software in order to mediate that technology. And then the creators have to come in and see how they can use what is there with both the hardware and software, and to push the limits of what's possible with what exists today. And if it doesn't exist, then they have to go out and create the software. But they have this sort of back and forth, and then they show it to the audience, and then the audience has to learn how to see it. And then there's a distribution platform to actually get it out into the people's hands.
[00:39:05.530] Jessica Brillhart: And so this is the birth of a system. This is the birth of a medium. That's what it's supposed to be, right?
[00:39:10.945] Kent Bye: Well yeah, but in some ways it sounds like Traverse is doing two of those, which is a potential distribution platform with software, but allowing people, enabling people to possibly imagine, upload, and create their own experiences with that.
[00:39:24.309] Jessica Brillhart: That's precisely it, yeah. It's meant to be open and agnostic. And yeah, letting people be able to do what we do. I mean, we're putting it through its paces now, just so that we understand what the potential of it could be. know where it breaks, know where other people would like it to go and sort of see how we can accommodate to an extent, right? Obviously. But yeah, we just want to, it's great to build your own platform as I'm sure it was great for Felix and Paul to have their camera because you could tinker with it. You can make it better. You can do with it what you would like, like being able to have a filmmaker come up to me and say, I really could use this platform for this audio that I have for my film because I'm trying to raise money for my film. but I only have audio and this seems like the thing that could help me. And for me to say, yeah, sure, of course, I'd love to help you. What a feeling, you know, that's big. And I think it's wonderful. I personally love that. And I love being able to say, yeah, you have this thing, you want to try it out? Let's talk. Let's do it. And it's freeing and it's liberating to be able to not just make content that makes you feel like you can tell stories or create experiences where people can feel X, Y, and Z, but then to also say, you know what, and you can do that too. And I think I've always loved that about technology and technology has helped me to develop my sensibility. Like I learned how to edit in Final Cut Pro by reading a book at Borders. You know, but I also learned how to edit and found out that I could actually craft films because my dad gave me iMovie. I don't know how he got it, but you know, we were in the middle of nowhere, Pennsylvania, but he was like, here's iMovie. I'm like, oh my gosh, who's Ken Burns? This is great. You know, like just doing all this stuff. And so I was, I, my, my, my creative thought, my career as a result of being able to use some of this stuff for a technology creator to say, here you go. You have an idea? Do it. So if I can give that experience back to people, then I can go on that vacation, I think.
[00:41:22.983] Kent Bye: Great. And finally, what do you think the ultimate potential of immersive media is and what it might be able to enable?
[00:41:33.391] Jessica Brillhart: Really? Oh, my gosh. I don't know, you know? Someone told me once that every time they see my work, they feel like I'm chasing something. Maybe now they feel like maybe I've caught up a little bit to whatever that thing is, but... That's a tough one. I mean, I think there's so much of this world to experience in all sorts of ways. And to really be able to let someone craft their own story and to be able to have the experiences they want to experience and be able to tell great stories at the end of it, to be empowered to do so, I think is very important. I mean, I don't know. I mean, I think this kind of space brings together a lot of the things that I love. Again, game design being one of those big things that I didn't know I could do, but apparently I've been doing this whole time. You know, thinking about accessibility, about being able to craft experiences that don't see accessibility as an afterthought, and you're welcome, and rather something that can be baked into an experience from the get-go, and how that can help reimagine the world in really beautiful ways. That's something I'm really passionate about too, and I think the medium can do as well. I don't know. I mean, it's ultimately, it's just, there are all these experiences and ways that we can be in this world and live life that we don't necessarily have the capacity to for whatever reason. And I think that immersive media extends our perceptual ability, allows us to connect to the world in a myriad of ways. And maybe makes us feel a bit more fulfilled. So I'd say I think that's kind of where we want to get to. But that may change depending on when we talk the next time.
[00:43:16.148] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:43:20.832] Jessica Brillhart: Hope everyone's doing great. See you at the next one.
[00:43:26.475] Kent Bye: Awesome, great. Well, I'm really excited that you're launching this platform, and I hope that people start to try it out and use it, see the power of spatialized audio. And yeah, it takes a lot of emotional labor to build platforms, and it's kind of a thankless job in a lot of ways. But hopefully, you'll be able to provide the opportunity for people to do things that could never be possible before, and that it'll just kind of unlock people's imaginations to be able to explore what's even possible with this medium.
[00:43:52.643] Jessica Brillhart: Yeah, I want to, I want to add that, you know, I couldn't have done this at all without Superbright, EGOL, Nasima, Nate Turley, Erica Newman, Max. We had a small but mighty team that have been working on this and, you know, I would be nothing without my engineer and my technical director and my, my friend. And so he's been there and has, we've struggled together along the way to make this work. And I think it's important to have both sides working together in order to make this stuff happen. Yeah.
[00:44:20.644] Kent Bye: Awesome, great. Thank you so much.
[00:44:22.145] Jessica Brillhart: Thank you, Ken. It was awesome talking to you. Hopefully not for two years. We can actually talk sooner.
[00:44:29.382] Kent Bye: So that was Jessica Brillhart of Vray Pictures, and she's doing immersive and mixed reality content. So I have a number of different takeaways about this interview is that, first of all, well, Jessica is one of those creators who I think has been deeply following her own intuition as to where this whole medium is going. And that while her heart is still in virtual reality, it seems like in the short term, there's so much potential in immersive audio. And actually, in terms of a platform, Just in my conversation with Michael Lutton of Bose AR, they're planning on having this integration into over a million devices over the next year or so. And so this is going to be one of the XR platforms that has the most penetration that's out there. And so it's going to be able to get in front of so many different people, probably people that aren't even in the XR community right now. They're just consumers that may be exploring these different spatialized audio solutions. even the QC35 headphones are just these noise canceling headphones and it just so happened has all the technology that they need in order to do the firmware update and unlock all these Bose AR headset features. And so I think people are going to be hungry for having these types of different experiences. So one of the things that I noticed about this very specific experience that they were showing at South by Southwest is that they were using the camera on the phone in order to do very specific location as you're walking around, like very small distances. And so, you know, it has a GPS that's able to generally locate you pretty close to where you're at. But like Michael Luton was telling me, there's so many different aspects of, you know, looking at Wi-Fi and looking at your velocity to try to like extrapolate where you're actually at. I think it's probably a little bit easier when you're driving in a car just because there's a little bit bounded as to where you're at. And so when you're walking around these small scale, a couple of feet in the course of a room, I think you have to turn to the camera on these mobile phone in order to actually get that tight level of positional tracking. And so they made sure that you weren't covering up the camera lens. And this actually happened to me when I was looking at Pilgrim at the IDFA doc lab, which is this experience where you were walking with these different pilgrims. I actually had put my phone in my pocket and I wasn't able to get the location information that I need to be able to get the very specific where I was at. And so with some of these experiences in order to detect like stopping or moving around a small space in the short term, they're using this stop gap of the camera on these mobile phones in order to actually locate you positionally. And so Jessica actually mentioned this artist named Janet Cardiff who does these sound sculptures where she would take a piece like the 40-part motet and put it into these 40 different speakers in a circle and you're able to be able to walk around there and to hear these different parts and be able to actually modulate your experience again by using your body as a mixer as you move through your space you're getting a different experience of this music. And so I see that there's going to be a lot more opportunities for where that's going to go with Vray Pictures' platform of Traverse, working in collaboration with Superbright. They've created this platform in order to take these multi-tracks and to be able to put them into the system to be able to give these specialized audio experiences. So I'm really looking forward to Tribeca that's coming up at the end of April be able to see this Yo-Yo Ma experience where you're actually like walking through multiple levels of the building and as you're walking through this building you're overlaying on top of that this whole audio experience and so I love the idea that you're going to be able to create a very specific memory and context of these spaces, and so I'll likely always have a memory of walking around a building and having a whole experience of this piece of music from Yo-Yo Ma, and to have that correlation between as I move my body through space, it's actually going to be modulating the experience of the music that I'm getting. And so Jessica said that in her keynote, she was going to be giving this like immersive 101 to 106 course where her journey as a filmmaker, but this transition from frames to worlds and from viewers to visitors, where it's not just a 2D frame, but you're immersed within a whole context. So you're able to move your body through space to experience that context, but also to make those decisions. As a visitor, you're expressing your agency within this world that you're visiting. And you're not just passively consuming and viewing something, but you're actively participating. And these experiences, I think, are training us to move our bodies through space. And as we move our bodies through space, we're able to modulate these different aspects of these immersive experiences. And that Jessica said that, you know, she wasn't necessarily even aware that audio was such a crucial part of her work as an artist, but looking back in hindsight, she could see that she's always been a champion of the audio because the audio is like the glue that's giving you this full immersion audio is the only sense that we have that is truly spatial we can detect where sound is coming from with our two ears where you know with our sight we only have a limited field of view our body we can sense different aspects of the temperature and environment but it's not as nuanced of being able to geolocate specific sounds with our hearing and i think Because of that, there's a whole layer of deep unconscious contextual awareness that I think audio gives that I think has maybe been a little undersold or underappreciated within the virtual reality community, just because VR, the visuals are so dominant. So I'm actually really happy to see something like the Bose AR frames and the QC35s to be able to provide a platform for people to really explore the potential of audio. And I think that's what I was seeing within the experiences that Jessica was showing both from this Elvis Presley song where you're able to kind of walk around the room where they're singing but also this Mars experience where you're walking around Mars and hearing these different commands and tasks that you're trying to do and there is this small correlation between being able to actually look at your phone to be able to almost provide a map and overview to see where you're located at and I actually did find that to be really helpful because you're walking around a space but as you walk around and you see the visual correlation is to like oh now I'm right next to these backup singers and so it allows you to make the connection between what you're hearing and what you're seeing and so kind of using the phone as this overview of a map to locate you in that position in that space but to then put the phone down and let yourself just kind of wander around and experience that soundscape of these worlds that are being built within this spatialized audio realm which is like this whole mural world of an audio layer of this augmented reality. And it sounds like that she's working with some pretty amazing collaborators with Cyan Games. They did Myst and Riven and Abduction and Numinous Games, which did the That Dragon Cancer. And so they're trying to create a way to be able to explore the mental workings of these great minds. And she said it's going to be like powers of 10 match with Google Earth and to be working with these other game designers to be able to explore the depiction of information and the full pattern relationship of context and meaning of all these different thoughts and to be able to kind of step into the mind of some of these great thinkers. And, you know, my emphasis was to not focus it just on those great individual minds, but I think the real power of VR is to show the full power of an ecosystem and how it's actually these entire communities that were coming up with these different concepts and ideas. And the evolution of thought is not being developed within the vacuum of an individual's mind, but it's actually within the context of a larger community. And I think it's the strength of the VR as a medium that is really able to emphasize that ecosystem dimension and that it's a process that's continually changing and unfolding. And that reminded Jessica of Geoffrey Hinton, who was talking about how we don't really know what the future holds and that some of the worst thing that we can do is just to assume that things are going to stay the same and they're not going to change and to evolve. I thought that was profound because it's basically saying that we have to assume that there's going to be these huge disruptions and that this transition from 2D to 3D and breaking out of the frame and into these immersive worlds and spatial computing and immersive computing, all these is this huge paradigm shift. And Jessica is saying that she's afraid that some of the prevention of what has been able to really have these immersive media really take off and thrive is that it's still within the context of these old systems. And so maybe we have to figure out some completely new systems to be able to contain the full complexity and breadth of what this immersive media can do. And I think that's right. You know, what Jessica said is that the immersive media cannot be contained and maybe it can't be contained within these existing systems. And we actually have to grow and expand those existing systems in order to live into the full potential of this immersive media. And what that looks like, I think, is a big open question. But I think a big part of it is moving away from centralized systems into decentralized systems and finding ways for independent creators to be able to be supported and to have ways that they don't have to go through the gatekeepers, but they can be able to participate and I think things like an open platform, like Traverse that's being developed, I think is going to help enable that type of dynamic where creating the tools and having the technology available is going to allow these different creators to be able to push the limits of what's even possible. And as time goes on, I think it's going to become easier and easier to create these specialized experiences within VR and AR and these audio layers of spatialized audio. And we're going to see a lot of innovators and creators who are coming into their own age in their power within the context of all these different platforms and services that are being made available right now. But we shouldn't think that these existing systems are going to be around forever and that they're going to be going through all these different changes and evolving and that there's going to be lots of cultural shifts and ecosystem shifts that are going to be happening over the next five to 10 years. So that's all that I have for today, and I just wanted to thank you for listening to the Wisses of VR podcast. And there's a couple things you can do if you want to support the podcast. First of all, just tell your friends about the podcast. This is independent journalism and media within the immersive community. I try to go to all these different events and talk to the different creators, like Jessica Pearlhart, to share with you what they're thinking about and where they're at now and where they're going in the future. And if you'd like to support that, then you can become a member and, you know, $5 a month is a great amount to become a member and to be able to sustain this type of work and to be able to provide this information for not only you, but the entire VR community. And, uh, you know, I like to think that the listeners of the Voices of VR podcast are the people who are thinking about the future. You're the future dreamers that are going to be helping create these new systems in these new worlds that we're going to be basically changing so many different aspects of our society, which need a lot of help. And so I think that's a deeper thread and a deeper thing that I think ties everybody together that's listening to podcasts like this and to be able to track the work like people like Jessica Brillhart are doing and these new platforms that are coming out like the Bose AR headsets, which I'm super excited about where this is all going to go. So if you've been thinking about it for a long time and you feel like you have this sense of guilt of not doing it, well, just alleviate your guilt by becoming a member. And you can do that by going to patreon.com slash Voices of VR. Thanks for listening.
