The Bose AR House at SXSW 2019 featured the first round of interactive audio experiences that are integrated with the Bose AR Frames and Quick Connect 35 noise-cancelling headphones. Devices on the Bose AR platform have an accelerometer, gyroscope, and magnetometer so that it can detect head movement and velocity, cardinal direction, and relative position, and Audiojack was showing an interactive audio experience where you could take different branches exploring an audio landscape by turning your head at key moments.
I had a chance to catch up with Audiojack founder David Tobin to talk about his initial inspiration for Audiojack, his vision for how interactive audio landscapes can reinvigorate our imaginations by inviting us to become the authors of our own stories, how Audiojack is being used in educational contexts to inspire creativity, and why he things exploring the imaginal real is one of the most powerful tools we have to catalyze new ways of things, cultivate empathy, and to project out into the future to imagine the consequences of our actions.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Photo from Audiojack
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So I went to the South by Southwest Interactive Festival a couple of weeks ago, and it's quite a scene. There's tens of thousands of people that are there, lots of different panels. There's a whole VR and AR section with lots of panels from Monday, Tuesday, and Wednesday, 26 different experiences that I had a chance to see all those experiences and talk to a number of their creators. But a whole other aspect of South by Southwest are all these activations that are done by these different companies. There's entire streets now that are just filled with different houses that have companies that have printed them out. They have immersive and interactive experiences to be able to talk about what they're doing in a form of experiential marketing. Overall, South by Southwest is like a giant experiential marketing expo where all these companies are trying to find out the latest ways that they can bring people out of the main South by Southwest area and into all these different immersive experiences. And one of those places was the Bose AR house, which was at 75. one-and-a-half Rainy Street. And Rainy Street was this street that was a few blocks from the Convention Center that had all these different companies with these different experiences that were marketing the types of stuff they're working on. But last year at South by Southwest 2018, there was the initial revealing of these Bose AR frames where they were showing a 3D printed prototype with maybe some brief demos, but it was kind of like the revealing of this technology, which are basically these frames that you put on they've got all these different sensors and then as you move your head around it's this spatial audio platform and so they were showing off lots of different demos of new applications that were the first round of the integrations with the Bose AR frames and the Bose AR quick connect 35 headphones So I had a chance to talk to David Tobin, who's the founder of AudioJack. He's been creating this application that is creating these interactive audio soundscapes where now that you're able to turn your head as you do these head turns, you're able to actually make these decisions within these audio landscapes that he's creating, and they're pretty sparse. I mean, the narrative is not like they're trying to spell out what you're supposed to experience, but they're kind of open-ended of creating this audio landscape and context for you to then tell your own story about what is happening within this experience. And so he's really trying to activate your imagination in this way. So I had a chance to talk to David about his talk that he gave back in 2018 at South by Southwest about the power of the imagination and what it means to unlock the imagination and all the benefits of your imagination and how he's been inspired to create this audio jack application to be able to try to explore how to use audio and interactive audio and eventually spatialized audio to be able to tap into the deepest aspects of your imagination. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with David happened on Sunday, March 10th, 2019 at South by Southwest in Austin, Texas. So with that, let's go ahead and dive right in.
[00:03:18.913] David Tobin: My name is David Tobin and I'm the founder and creator of AudioJack. We make audio-based movies. There's no words, there's no video, there's no music. Hundreds of sounds are edited together to tell a story for your imagination. So you might hear the wind and trees blowing and then feet, and they stop, and then a door opens. Your imagination and memory kick on and you start to create a narrative because we all have an association to sound. Inside the AudioJack app, There's hundreds of different, well, combinations, but there's almost 50 different audio jacks in there that range from environmental experiences to wildlife that we created with the World Wildlife Fund and Humane Society, where you feel like you're an animal in different situations. There's historic ones, where you feel like you're in the Civil War, the Gold Rush. There's action adventure. There's day in the life. And it's all relative to your imagination and how you connect.
[00:04:05.520] Kent Bye: Can you tell me the story of how AudioJack came about?
[00:04:08.711] David Tobin: Sure. I have a background in film and music. I used to manage the Roxy Theater on Sunset and Hollywood. I've been working as a TV producer for a long time, and I play a bunch of instruments. I've always wanted to find a way to connect into the education space, and my mom was a teacher for a while, and I had come up with AudioJack when I was working with a bunch of sound effects one day. And I started putting them together, putting them together and created this, and it blew my mind. I surprised myself, and I played it for some friends, and they were freaking out about it. And they're like, what is it? And I'm like, it's an audio jack. And they're like, OK, what's the story? And I wouldn't tell them what the actual story was. And they were like, OK, this is crazy. I love it. And then I played it for my mom. And she was like, we need to bring this in the classroom. Let's see how the kids react to this. And so started creating exercises around it where you listen to an audio jack once with your eyes closed or a blindfold. And then you listen to it again and brainstorm with your eyes open. and you start going through the story and creating a narrative based on what you're doing. We brought it in the classroom and kids were super engaged. Their grades started going up on the performance of what they were doing whether it was art or writing and it took off from that space and it was my way of contributing into the education world but then realized there were way bigger ramifications because we're helping people take back their imagination. And when you use your imagination, you're actually helping reduce your stress and anxiety. And we're in this always-on generation, where people always have a phone or some device in their face all the time, and it's restricting your ability to creatively think. You're being told something the whole time. And we actually just did a study showing that 67% of people daydream less. Because when you have that time to daydream, people pull out a phone, or they're dilly-dallying, they pull out a phone, they pull out a phone. Well, that's actually causing you to be more anxious and causing more problems than you realize, because that's your time to process through the day, because you get a form of REM activity during daydreaming.
[00:05:52.684] Kent Bye: Oh, wow. So it sounds like that you gave a talk at South by Southwest last year about imagination. What was it that you were trying to really communicate in that talk last year?
[00:06:00.755] David Tobin: Yeah, last year I did a talk at South by Southwest about the importance of imagination and how it reduces your stress and anxiety. And the title of the conversation was Take Back Your Imagination. And what I'm trying to get people to realize is our imagination is one of the most powerful tools we have. And the more we use it, the better we become as people, as humans, as a society. And you have to think about imagination in a really unique way. Think about it from you had to imagine what you would look like wearing the clothes you're wearing right now. You might not think imagination is just, oh, I'm creating a cartoon, or I'm making a movie, or I'm making a song. It's much smaller than that, but much more important. And when you're actually thinking, you're getting creative thought. You're actually engaged with people when you're using your imagination. You're processing differently. And the more you use your imagination, the better you can relate to other things, because you can create empathy, and you understand, because you're imagining yourself in someone's shoes. You're imagining what something might look like. You're imagining the consequences of your actions. Those are all forms of imagination. And the more we use our imagination, the better we are at existing with the rest of society.
[00:07:02.729] Kent Bye: I'd say for like the last hundred years we've had the moving image with film and then we had radio and radio plays and then television and so we've kind of been going back and forth between the image and audio and it feels like that right now with the resurgence of podcasting and storytelling and podcasting it's almost going back away from the visual medium and I think there's something about the audio that is asking the listener to participate in the co-creation of whatever you're listening to. I don't know what your thoughts are in terms of the different trends of this resurgence of the podcasting medium and audio in general right now.
[00:07:38.282] David Tobin: Well, I think why it's coming back in such a way is because for a long time it was a very walled environment. You had to be with a radio station or a broadcast medium, and it was really tough to get in. The barrier to entry was so high. But now anyone can create a podcast, and there's a lot of people that really enjoy that, and they weren't able to get what they wanted. And now we have such a higher variety of it, so people are really engaging with it. Plus, it's also nice to be able to listen. and to walk around and not have to be in front of a screen. Because whether you do VR or watch a TV show or something, you can't move around. You can't walk down the street. You can't be present in another way. And I think people are missing that. I think people miss being active in the world and want to be able to communicate with other people in a unique platform. You can share podcasts in a unique way that you can't really do with a TV show with other people.
[00:08:30.013] Kent Bye: When I went through the demos that you just had here at AudioJack, it's a part of the Bose AR installation activation here at South by Southwest, and there seemed to be detection as to as I'm turning my head. So as I'm listening to audio normally, I'm not getting head pose information at all. But it seems like now that you're able to have either headsets with accelerometers or with the Bose AR glasses, you're able to determine where people are looking, or at least accelerometer information to see if someone's turning their head to the left or right. So maybe you could talk a bit about how you started to use that information for Audiojack to create these interactive narrative experiences.
[00:09:06.487] David Tobin: Sure, when you listen to an audio jack, there's a story going on, and it's up to you to create the narrative in your imagination, but it's static. You know, you're there listening, you're being passive. I wanted to find a way for people to engage with that, and so with Bose AR, you're able to react because you have these tools inside the hardware, inside the frames, inside the quiet comfort headphones, that allow you to turn it into a choose-your-own-adventure element, if you will, or something like that, where you hear an explosion to the left and you turn, And the story goes off that way. It's giving more freedom to the listener. It's also letting the listener be a part of the experience. And the AR audio jack that's in the app right now has 20 different story combinations. So you can wander around and try different head movements at different points and investigate different areas and end up at the beach or fighting aliens or being chased by horses or dogs. And there's all these different combinations of it. And it allows the listener, I think it's really important, to give the listener these tools to be able to be more immersed. And I look at audio as the true form of VR, because it's your imagination. It's real VR. Virtual reality, yes, you can virtually be in something. But augmented reality is you're stepping inside another dimension. of your own existence. And I think that's really compelling and really neat. Now that these tools exist, it allows people to do it much more freely and it's much more accessible, especially since the cost is so low now because the technology has become so prevalent.
[00:10:33.087] Kent Bye: Well, so I've done a lot of VR experiences with spatialized audio. And when I went through the AudioJack narrative, the thing that I found difficult was that there wasn't enough spatial audio cues for me to determine whether or not I was turning my head was actually doing anything at all. So it's hard for me to know whether or not my agency, as I'm turning my head, was impacting the experience at all. And so I know that there's an existing challenge with ambisonic audio and existing standards for what's the format that is going to be used to be able to take User agency of head pose and feed it back in real time. Is it matter of not having an open standard for ambisonic audio? Is it like the real-time latency? What is it? Is there anything technologically that's holding you back for doing fully specialized audio experiences?
[00:11:15.358] David Tobin: No, there's nothing holding us back for this experience. This was kind of a later evolution of what we were doing and so we didn't have time to build out the spatial audio for the South by Southwest experience. In the next couple months it will be spatialized and you will have that because you know tools like Unity exist where you can do that and these gaming engines that allow you the spatialized audio to move around and it was just a matter of the fact that we just didn't have time to get this ready because we knew they were going to do something here we didn't know what they wanted to do with us and at the end they're like hey by the way we want to make this thing Let's go. And I'm like, OK, great. Let's go. Oh my god, we don't have a lot of time. So we built this one that actually is almost like turning pages or switching chapters in a book. And then it's unfolding. You move your head, and all of a sudden, things change because you moved at this one experience. And then things shifted. And that's what we've done here. But the next level of this, the next ones we're rolling out, are spatialized. They do allow you to move around and exist in that platform.
[00:12:04.703] Kent Bye: Maybe talk a bit about how you got connected up with the Bose AR and started to do some of these experiences with the Bose glasses and what those afford for you.
[00:12:12.514] David Tobin: Yeah, so last year I was doing the talk here at South by Southwest about the importance of imagination and how it reduces our stress and anxiety. When I was done with the talk, two gentlemen approached me and they were like, we work with Bose and we love what you're doing. We've been following you for a while and we would love to show you something. And so they brought me over to this bar called Half Step and they brought me in the back, showed me these 3D printed glasses that work with AR technology. And I was blown away. I'm like, holy cow. And they're like, yeah, we got to work together. And like, great. And so that's where we started the relationship. We started figuring out, you know, how does this need to work? What can it do? And those elements. And then here we are today. And we have an AR version of our app. There's more technology coming out. And this is just the beginning. I mean, this is literally the very start of what's about to happen. and the fact that there's other companies that have, like Nike and Under Armour, that have sensors in their shoes. So we can take that data from that, process it through Audiojack and the AR capability, and then all of a sudden you're running, and it sounds like, you're looking down at the cement, but it sounds like you're splashing in water, and actually truly augment your reality.
[00:13:15.233] Kent Bye: Oh, wow. So usually when you think of AR, I think of visual augmented reality, like the HoloLens or Magic Leap. But with the Bose AR, it seems like there's no visual augmentation, but it does have the spatialized audio that seems to be projecting audio into your ear. It sounds amazing. It sounds like that you're starting to think about what can you actually do with this combination of having a headset that's tracking your head pose, which direction you're listening, and then taking all these other devices and figuring out what kind of narrative experiences you can do with that.
[00:13:45.953] David Tobin: Exactly. I mean, that's the way I look at this. I'm like, OK, we can do this now, but we need to be pushing it further because we need to truly augment your reality. And the only way to do that is really augment your movement, your body movement with that. And so I think that's where I see this going, where I want it to go, because there's no visual construct with an audio jack. There never will be. That's the whole point is that there's no visual element to this. That it's strictly your imagination, the way to do that. Because if I have a visual element with this, then I'm telling you something. If I have a man's voice in it, or if I have a song telling you something, then I'm telling you it's one thing or the other. With ours, it's totally agnostic of race, of gender, of language. Anyone can use it. If I break a glass and take that same glass and go to France, it doesn't go, le break. It's the same sound. So it's about creating a universal platform that we can all connect with and engage with.
[00:14:30.532] Kent Bye: Well, when I was at Sundance this year, there was a piece called Sweet Dreams, and it was looking at taste. And when you have a taste, you have these memories that are coming up that are associative, but they're kind of below your conscious threshold. It sounds like sound is doing something very similar, where we have sound and we have associations for what that means, but you're trying to, in some ways, paint a story with this associative language of these sounds that are trying to weave together to larger meaning.
[00:14:55.867] David Tobin: Yeah, absolutely. Like if you listen, if you jump into the audio jack, there's different categories. There's a historical section and you go through that and there's one called Boston Tea Party and it really sounds like you're gathering with a mob and then you're heading through the town and then getting to the docks. and you're breaking open the crates and throwing in the water and the whole thing. And we're using only sounds that would have been heard in that time. So we're being very careful of that. The stuff we do in our wildlife section, those are the actual sounds of those animals. It's not like I just put out a recorder and record it. We took hundreds and hundreds of elements of those sounds and edited them together to create a sound. Like the Siberian tiger one, for example. Those sounds are all animals you'll find in the Sea of Japan. you're going to hear those elements out there. That's indicative of that environment. It's not just random stuff and birds you would hear or whatever. Same thing for like black rhino, that's all stuff you're going to hear in those elements of those parts of Africa. And like the bird you hear in the background here, these are indigenous to right now and here in this moment, you know, they're not going to be found in other parts of the world. So if I was making an audio jack about South by Southwest, I'd make sure I include those things. So it's real and it's honest. So it's true to the environment. So you really, if you're going to augment reality, if you're going to do that, it's not just, hey, here's some cool stuff. Let your mind go. I want people to know this is the real deal. You're really feeling what you would be like if you were there right now or if you were back in time in that moment.
[00:16:13.964] Kent Bye: For you, what are some of the either biggest open questions that you're trying to answer or open problems that you're trying to solve?
[00:16:21.187] David Tobin: What I'm trying to solve is I'm trying to help people get off their, not get off your phone because obviously we have an app. It's on a phone. Like I get that. But what I want people to do is start realizing the importance of their imagination and get them back to thinking for themselves and realizing that your imagination and reasoning are tools that aren't getting used as much. I mean the study we did showed that 67% of people aren't daydreaming as much as they used to. During that time, they pull out a phone. If you have time to let your mind wander, people pull out a phone. And that's crazy. It's horrible. I mean, you don't think it's a big deal now. And oh, I do this or everyone else does it. It's a way of culture. But you have to think in the long view of this, what it's actually doing to society, what it's doing to people and humanity, that we're always being told something instead of having a chance to have reason and free thought and discussion and those elements. And that's my campaign, is take back your imagination. Because it's one of the most powerful tools we have that makes us so damn unique that it's invaluable. And by not using it, it's use it or lose it. And that's a scary thought. It's a really scary thought.
[00:17:21.930] Kent Bye: And finally, what do you think the ultimate potential of immersive media is and what it might be able to enable?
[00:17:29.153] David Tobin: Well, I think with immersive media, it's going to enable a lot, but I think it comes with, as they say, you know, great power comes great responsibility. And I think we need to be responsible about what we're doing with immersive media because I've seen some stuff, I've worked with some companies and seen some crazy augmented elements, some very big VR stuff, and people get lost in that space and they forget what it's like to be human. And I think that's the thing that I see as a little scary in that element. Yes, that changes humanity and we grow with it and those elements, but I also think there's something about being immersed in audio and immersed in technology that can be really healthy and fun and engaging as long as we have people out there showing how to do this, how to take limits, and how to engage with multiple forms of it and not just one thing. Not just plug it into a VR set for 12 hours a day. and exiting the rest of the world. Sure, you're engaged in that platform, but you're missing out on the actual socialization of humans. We're social animals. We need to be connected, and we need to be able to talk and touch and feel, and the visual cues start to change with people over time. And that's what I think we're trying to solve. And with audio, you can still connect with people in that space. And as long as you're not walling yourself off, and there's something else that kind of scares me too, is like, you know, people think like, oh, I'm using this translator and you're being told something. Well, if someone hacks that translator, they could be telling you something totally different and you're not reasoning and thinking logically, you're just taking it because, oh, it tells me that. I think that could be another slippery slope to end in. So I think it's really about responsibility and people educating people about like, hey, here's this really cool tool, but let's also look how it could be used in other contexts. Let's have some time to think about why we should do this and how we should use it.
[00:19:05.013] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:19:08.995] David Tobin: I think keep creating. Keep doing more of this. Keep pushing the limits of this. Let's get immersed in this area. And then let's see how it helps people. We work with the federal program for the blind. We work with education programs around the world. And it's really neat to see them immerse themselves in technology and VR and audio because it allows those people who can't connect. which is really neat. And I think that's the thing. We need people out there that are pushing the limits of this so we can help those people who can't communicate and those people that are creating amazing immersive experiences to share with other people and show them how they're doing it. Because the only way we move ahead is if we all work together and not wall ourselves off.
[00:19:50.101] Kent Bye: Awesome. Great. Well, thank you so much.
[00:19:51.742] David Tobin: You're welcome.
[00:19:53.584] Kent Bye: So that was David Tobin. He's the founder of Audiojack. So I have a number of different takeaways about this interview is that first of all, well, I think the whole idea that you're going to be able to explore your imagination with this whole layer of audio, I think is fascinating because it's about doing this world building through setting up all these different relationships of sound and space. and allowing people to explore that space and come up with their own story. So in a lot of ways the authorship is happening within the audience members like they are the last authors of the story of these experiences. It's a lot about trying to create a larger context for you to then interact with these different experiences. So the specific experience that I saw did not have spatialized audio integrated yet and it sounds like it's going to be coming on the way. I think at this point it was actually a little bit difficult to know when my agency was changing or modulating the experience at all. It's a little mysterious and I think that that's something that I think will be interesting to see how that evolves and grows because when you have an interactive audio experience but yet you can't see the traces of your agency like you don't know when you actually make a decision then it's kind of imperceptible that you even have any agency at all. And I think this is a challenge that I see in a lot of all interactive media, basically. In virtual reality experiences, you're in a room often, and then there'll be an object that highlights, and then you're like, oh, well, now I can know that I can interact with this object. You know that is actually like a thing that happens in real life where you can basically interact with any object but within these virtual simulations there's a bit of a constraint on your agency and so in order to emphasize the things that you can interact with and you have these little conventions like that. And what that looks like within the audio realm, I think is still yet to be determined. I can imagine a time when you're able to actually walk around a physical space and it's less about you sitting in one spot and turning your head around. And if you're actually locomoting through space and being able to actually hear the impacts of your agency, then I think it's actually going to be a little bit more clear as to what decisions you're going to be able to make. because it's going to be maps to you moving your body through space. And as you move your body through space, you're going to have a little bit less of an abstraction of you just kind of having an audio that's unfolding. And then you're moving around within the context of the experience, but you're not actually physically moving. And so it's a little bit difficult for you to know, like what direction you're running in. So if you kind of imagine just creating a unity file and then creating lots of different objects with sounds all around that space, it's kind of equivalent to that, where you'd be navigating and walking around a space, but because of the spatialized nature of that audio, you get a little bit better sense of you navigating through this larger possibility space of exploring and seeing these different branches of the narrative that you can explore. The other thing that I think I was kind of blown away with when David was talking about it was just hearing about these other wearables that are being integrated into like sensors within the Under Armour and Nike shoes. So that when you actually have your foot hit the ground, that could be integrated with something like AudioJack to then trigger a sound of you running through water. And so what would it feel like for you to be actually moving your body through space and to have more and more of these different wearables and these sensors? Fused all together for you to create these different immersive experiences all in the audio realm But it has your body moving around and having different aspects of your agency And so that's what I found was really interesting about this is that I don't like to think about how like the HoloLens or Magic Leap is going to be like the singular device that's going to cross the chasm into the mainstream where people are gonna all of a sudden be walking around with AR headsets on I Actually don't think that's gonna happen. I think that most people are gonna be using that very specific context at their work that it's gonna be much more likely that there's gonna be something like the Bose AR frames that you're gonna be wearing around because it's essentially like a Bluetooth headset that has spatialized audio and has a microphone and it's just like a super elegant solution where I could just see so many different use cases of you driving on a scooter or riding a bike or in your car and you want to listen to music but also at the same time potentially move your head around and because it has Both the GPS on the phone fused with how your head's moving and what direction you're facing. Like, are you facing North or East, West, or South and your acceleration and velocity, you can do all sorts of different gestures with that. So I think there's just going to be so many different potentials for having these embodied interactions with these interactive audio scapes. And I think that is a big thing that I came out of for South by Southwest is just the potential for where this is all going to go. And I think the Bose AR frames and the Quick Connect 35s, these are an amazing first step to unlock what's gonna be possible in the future. And, you know, just the fact that we spend so much time on our screens and that as we do that, there's been this huge decrease in the amount that we let our mind wander or that we daydream. And that I think there's something that David is saying that he said, like there's 67% of the people they're daydreaming less because every time they have a spare moment, they're looking at what other people are trying to tell them through these different social medias. I think that what he's trying to get across is that if you actually just let yourself be present to the surrounding context, then you can just let your mind wander and you can be the author of your own experience rather than having other people mediating what you should be thinking about or what you should be doing. And I think there's gonna be something about training your ear to be able to hear the full context of these different situations. Something that I do on the Voices of VR podcast from the very beginning is that I'm embedded within the context of these different environments. And so you hear these different aspects of the diegetic sound of that environment in that context is seeping through. So this interview happened right outside of 75, 1 1⁄2 Rainey Street. Austin, Texas during South by Southwest. We had a walk across the street because the music was so loud. You can hear the music in the background, but you can also just hear these birds chirping and singing in the background as well. And as we're having this conversation, David actually points out that there are these birds singing in that if you wanted to try to recreate what it was like to be at South by Southwest. you want to actually integrate some of these bird calls into these different experiences just to give it that authentic feel and that's a lot of what he's doing is going in and trying to capture these authentic soundscapes so that when you hear them it takes you back to another place in time and your imagination can start to fill in all the other gaps about what is happening in any specific context. So that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then tell your friends, spread the word. I rely upon the grassroots marketing and the word of mouth to be able to continue to just get the word out about the work that I'm doing here on the Voices of VR podcast. If you'd like to support the podcast, then I could really use your help and to just become a member and to help grow and sustain what I'm doing here with the Voices of VR podcast. And $5 a month is a great amount to donate in order to continue to just sustain and support and to grow this podcast. And so you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
