OSSIC’s Jason Riggs told me at CES last year that the future of music is going to be immersive and interactive. Interactive sound design where user agency is considered a part of the experience is a key ingredient to creating fully immersive audio experiences, and the AR & VR tech platforms are creating new opportunities for musicians to explore interactive music experiences. Sigur Rós’ interactive music collaboration with Magic Leap was recently featured in Rolling Stone, and that interview reminded me of a deep dive discussion that I had with VR immersive audio evangelist Sally Kellaway talking about the fundamental principles of immersive sound design. Kellaway provides an overview for how audio designers for games have been creating demand for toolsets and plug-ins with existing Digital Audio Workstations to help break out of the normal linear, authored paradigms. At the time of my interview, Kellaway was a creative director for OSSIC, but has since moved on and is freelancing as a VR/AR audio designer, strategist, and immersive audio evangelist.
LISTEN TO THE VOICES OF VR PODCAST
Because designing an interactive experience is an iterative process, then there’s no one philosophy or paradigm for how to plan, execute, or post-produce all of the audio components of an immersive experience. The process is driven by the tools that are available, and so Kellaway described to me some of the most popular production pipelines and workflows for immersive audio. There are plug-ins to Unity and Unreal like FMOD and Wwise, but there are built-in, object-oriented audio features in Unity & Unreal, as well as a range of different audio spatialization options for VR.
Hearing is sense that can sense a full 360-degree space, and it adds a layer of emotional engagement that speaks to the more primal aspects of our brain. There are also a lot of decades old audio technologies like ambisonics and binaural audio that have found a new home with VR, and so the surge of immersive technologies of AR and VR as a technological baseline is catalyzing a lot of innovation and experimentation within the realm of music and sound design. The Magic Leap Tonandi music demo by Sigur Rós shown to the Rolling Stone journalist Brian Crecente was not “a recorded piece of music”, but more of “an interactive soundscape.” Crecente describes his experience:
Tonandi starts by creating a ring of ethereal trees around you and then waiting to see what you do next. Inside, floating all around me are these sorts of wisps dancing in the air. As I wave my hands at them, they create a sort of humming music, vanishing or shifting around me. Over time, different sorts of creations appear, and I touch them, wave at them, tap them, waiting to see what sort of music the interaction will add to the growing orchestral choir that surrounds me. Soon pods erupt from the ground on long stalks and grass springs from the carpet and coffee table. The pods open like flowering buds and I notice stingray-like creators made of colorful lights floating around me. My movements, don’t just change this pocket world unfolding around me, it allows me to co-create the music I hear, combining my actions with Sigur Rós’ sounds.
This is in line with what OSSIC’s Jason Riggs predicted as the immersive and interactive future of music that VR & AR technologies will be to enable. The OSSIC demo at CES last year was a really impressive immersive audio experience that allowed you to fully interactive with the audio environment. OSSIC’s audio hardware made it an even more compelling spatial audio experience, but it’s just the beginning of how VR and AR will change how music is composed and experienced.
Will the existing linear authoring, audio tools adapt to become more spatialized? Or will the game engines have integrate enough plug-in support to DAWs to become the defacto production pipeline of audio experiences? Either way, AR and VR technologies will enable new distribution platforms for audiences to experience the spatial dimension of sound in a way that’s much closer to how we hear the world every day, and it’ll enable musicians like Sigur Rós and Miro Shot to push the boundaries what’s possible with spatialized music.
This is a listener supported podcast, considering making a donation to the Voices of VR Podcast Patreon
Music: Fatality
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So last year at CES, I had a chance to talk to the CEO of AASIC, Jason Riggs, and what he told me is that the future of music is going to be immersive and interactive. A lot of the music that's out there right now is authored and it's completely linear, but if you look at sound design that happens within immersive environments like Unity or video games, then that is going to be what the future of music is going to be much more like, is these immersive interactive environments. Just within the last couple of weeks there was an article by the Rolling Stones, Glitch, where they had a chance to try out Magic Leap and some of the demos that they had were Sugar Rose, this band that was doing these interactive music experiences that you could actually change the music as you were interacting with these digital light fields and augmented reality experiences. Within Magic Leap you were actually able to have this immersive interactive audio experience. And reading about that reminded me of this interview that I did with Sally Calloway back at the Silicon Valley Virtual Reality Conference, where she talks about what it means to be an immersive and interactive sound designer. Sally at the time was a creative director for AASIC, the immersive audio company creating these headphones that will be able to track your position of your head and basically replicate what it sounds like to hear a 5.1 or 7.1 stereo, but also just do a lot more sophisticated binaural audio with ambisonic fields. So Sally has since moved on. She's a sound designer implementing audio strategy and an audio evangelist for virtual and augmented reality. And she talks about the process of audio design and some of the metaphors that you can use to start to understand what it means to do immersive and interactive audio. So that's what we'll be covering on today's episode of the Voices of VR podcast. So this interview with Sally happened at the Silicon Valley Virtual Reality Conference on Thursday, March 30th, 2017 in San Jose, California. So with that, let's go ahead and dive right in.
[00:02:14.535] Sally Kellaway: Hi, my name is Sally Calloway. I have just started working at a company called AASIC as their creative director. I'm actually a sound designer by trade. I've just come over from working in game audio and VR games specifically, but I actually started working with audio way back, like in high school when I was playing instruments and just being a little audio nerd that also liked computers. With VR at the moment, what I'm doing is I'm applying all my knowledge about music and sound design working in games, interactive spaces, to this new creative field and trying to make sense of it and help AUSIC, the company that's making the 3D audio smart headphones, actually make a product that is useful for content creators and actually really empowers them to make the best content possible.
[00:02:57.478] Kent Bye: Great. So yeah, when I did an interview with Jason Riggs, one of the things that he said is that the future of music is going to be immersive and interactive. And so there's this immersive quality that you get from virtual reality, but also the 3D spatialization and all the technology of AASIC. The interactive part is the thing that I think is really interesting, and it seems like this is kind of what you've been doing for a long, long time. So maybe you could start by describing to me how you think of, you know, interactive audio as contrasted to something that is more authored and in the studio.
[00:03:30.641] Sally Kellaway: It's really interesting because I came from this background of pre-authored, pre-rendered, recorded sound. My first degree was in sound recording, right? And I really quickly made this decision where I was like, is this boring or am I bad at it? I had to make a decision about whether I wanted to just become really good at knowing what different guitars sound like, what different amps sound like. trying to find my niche in recording arts or whether I wanted to find a different type of medium to engage with. So at that time I had a friend who played a lot of games and this game called Bioshock came out. Now Bioshock for me there was this one very very simple thing and it's sort of like the genesis of my sort of addiction to interactive audio stuff where you're in this like underwater city right and you get to like walk around and if you're anything like me that gets scared at everything you're walking around exploring very very slowly so nothing attacks you. in this underwater city there's like window walls that see out into the ocean. Now in Bioshock the closer you get to these window walls the more of the ocean outside that you hear and for someone who's like so used to like just playing music which is like sheet music and like canned stuff like recording bands and recording jazz bands and stuff like I immediately thought like how do they know when I'm getting closer to the window or like further away from the window that's like that seems like magic and I went down this little rabbit hole where I just wanted to learn more and more and more about how a piece of software can know, like, what a couple of button presses and, like, directional changes that a player is making, like, what has to happen for that to happen. But then also, like, is there a creative person that's saying, no, we need this exact amount of whale and this exact amount of water washing, and now that's kind of, like, my job and what I do. And there's this beautiful amount of creative practice that goes into defining what happens around the player and when. and how the whole game responds to what the player does. And eventually, like now I've arrived at this philosophy where in some ways we're just trying to create what these virtual spaces sound like and how they behave around the player. And it's sort of like interactive audio, we're just trying to make it like real life and kind of recreate the complexity of everything that happens in real life. So games is like a perfect vessel for that. Like people from other industries, they kind of sniff at games because Like you just shoot things, it's not very interesting, it's not very like classy or like cultural, but it actually prepares you very, very perfectly for this whole like new virtual reality, augmented reality and mixed reality future that we're looking at now, where we're finding that people have the most immersive experience when they're having the most amount of agency or at least able to feel that they're part of what's happening around them. So how do you do that? How do you define all those types of like sound relationships that the player has with the world around them? You actually have to go back to a point where you're thinking about what does the player need to feel? Like what do they need to think about the world that's around them? How do we even inspire them to turn their head? You need to have like some level of awareness from the game or from whatever that particular player is engaging with to what the player is doing. It can't just be about rotation because people can also reach out and move around. If you're giving them the opportunity to actually interact with the world around them then the sound should respond to that as well.
[00:06:45.665] Kent Bye: Yeah, and so when I think about the metaphors in terms of understanding kind of the mechanics of how this is actually implemented, I kind of think of the spectrum between recording on tape and it's sort of linear, and you listen to it in a linear fashion. But here we're kind of moving into an interactive game engine where you have a 3D immersive environment that is defined. And so you usually have a player character that has a camera attached to it where you're able to kind of navigate through this space. As you're coming into physical proximity to these triggers, then you have these audio events happening. And I think what you're saying here is that you're trying to mimic what real life is, but you obviously can't have all these things happening all the time just because of the CPU and all the other things that are required. So you're kind of triggering things dynamically based upon the physical location. So then I guess the question is when you think about on the other extreme of authoring you have like sheet music that you're kind of writing it down but you know how do you kind of map that out and model it and kind of understand the architecting this interactive audio? Is there a specific file format that you tend to go to or is it sort of a combination of a spatial awareness and you know you're adding the space and time dimension here and so how do you kind of map it out as somebody who's creating and architecting and designing these interactive sound environments?
[00:08:07.465] Sally Kellaway: It's like so vast in some ways and particularly like when you're just given a brief or told about an idea from like the studio that you're working in or a client that's come to you or something like that you can kind of start wherever you like which is the fun part and that's sort of like sort of how I work is because like as a creative person you sort of want to start where you feel the most like the strongest vibe and kind of work from there and But if you're working in a larger studio, you might have to work with the part of the game that's working the best and doesn't crash the most. But when it comes to defining all those relationships and what you're actually going to be doing with your sound in the game and what you can actually do, because you're going to be able to do less if you're working in mobile VR than if you're working on something that's going to be published to an Oculus or a Vive, right? is really deciding what's most important to the player and what are you trying to get the player to do, to think and to feel and really crafting not only the design of what the sound is meant to sound like but also what it's meant to do to help that player feel whatever they're meant to. One of my favourite examples is actually I was working on a zombie game last year where you have six players in one space and they all have three different guns they can choose from and then you drop in. We were trying to decide how many zombies should come at the players at any one time to make it suitably challenging enough but also kind of fun. but if you put like a lot of zombies in one space and like a lot of like six people all with guns like with their backs to each other shooting all the time it just sounds like white noise because there's just way too much sound going on so then as a sound person you kind of have to like go through all the stuff that you've like crammed in as quickly as possible because you never have enough time and actually go all right so What do I need to do for this all to sound like it kind of makes sense to players, and then also to get them to move around the level of the game in a way that makes it more fun for them? Because if you're working in large-scale free-roam VR, or even with room scale, where you can move someone a couple of meters left or right at a time, if you're really super lucky, or even if you just want them to turn around and look behind them, you have to make space for those sounds and give very clear sound cues. that the player should be doing something at that particular point in time. And that's part of the super fun thing that sound people get to do is we literally get to close our eyes and think about this whole philosophy of what we're trying to get the player to do, when and why, and how they should feel when they take off the headset as well. Sound is such a huge part of that, and it just makes me really sad when people are like, no, we wanted heaps of guns and we turned it all up to 11 and then there's explosions everywhere. It's like, no, man, if the player's meant to feel scared, then they should feel really scared. If they're meant to feel powerful, then they should feel like they could blow everything up. It's very philosophical in a way that's really creative and really fun.
[00:10:55.682] Kent Bye: Yeah, a couple of things that makes me think of is that first of all, our ears are kind of like our only 3D sense that we have. We're able to kind of sense all around us with our ears. We can't necessarily, maybe our body we can, but in terms of things that are coming from the outside, coming in. But also you're talking about feeling and emotion, you know, when I talk about the different levels of presence and I think about emotional presence, I sort of think of both the music and the sound really connected to our emotions in a way. I'm not sure if you've found that as well as, you know, kind of really invoking different emotional states in the players based upon what people are hearing.
[00:11:31.716] Sally Kellaway: You guys couldn't hear it, but I was like giving two thumbs up to Ken. He was just like asking that question then. And it's because when you said that sounders are only 360 degree sense, it's totally true. And whenever I get the chance to stand in front of a microphone, giving a talk at a conference or anything, I'm always like, I have this one slide that I give to people and put up in front of them. and it's just a field of view slide. And I lay out the different fields of views of the various headsets. You're pretty much restricted to, at best, excluding Star VR, 110 degrees or 115 degrees, depending on what headset you're looking at. It's even less for mobile VR headsets. So you still get this rectangle that you are trying to get the player to understand their visual field through. So they still have to make this rectangle pan around with their head. And most of the time, when people are presented with a rectangle, they just sit there or stand there and they stare at it like at a cinema or television all that kind of stuff. So people are like really trained to just look forwards because that's the direction our eyeballs are pointing in but sounds like you can even have your eyes closed and hear a sound and know where that place is. It is such a, like, deeply emotional and deeply memory-based thing. You could make people feel nervous immediately with just, you know, a quiet forest and a few little creaks of, like, wood and stuff like that. That would scare me. Or you can make them feel that they've come home for the first time in years. You know, all this, like, delicious range of feelings. So, really making sure that you're paying attention to that and really using it for good and using it for true immersion so that you You're giving people 360-degree sound cues. It also really helps point their eyeballs in the right direction.
[00:13:11.721] Kent Bye: And so I'm curious if you could talk a bit more about the different types of authoring tools that you've seen. I know we have Unity and Unreal. They have different options built in. But I imagine that there's more sophisticated tools for sound designers to really get in and create these interactive types of experiences. And I'm just curious what those options are and which one you really are drawn to.
[00:13:35.405] Sally Kellaway: That's a great question. Coming from games, I have a really deep experience and knowledge of interactive tools and naturally when you're making any sort of sound, you also have to use linear tools as well. So coming from sound recording and sound design, we have a term called a DAW or DAW, which is a digital audio workstation. That's kind of like a digital tape machine that lets us use plugins, EQs and reverbs and all that kind of stuff. So we'll like walk out with a microphone like you've got now and point it at something and record a sound and then we'll take it back to our studio and we'll do a bunch of manipulations to it and then we export that out and if you're like a real tech head like me what you do is you get something called a piece of audio middleware and that's like a very specific piece of software that replaces the audio engine of an interactive game engine like Unity on real so it literally just goes it carves out the audio portion of its processing and says well you know we give a better UI more features more effects might be like the CPU usage might be better or like the libraries might be really lightweight so you can like cram it on mobile devices and still get like heaps of sounds and heaps of features in there. So you'll do a bunch of adaptive audio design in that particular piece of software. I actually worked for a company called Firelight Technologies back home in Australia for nearly two years and they make a piece of audio middleware called FMOD. There's another one out on the market that's called WISE, W-W-I-S-E, and that one's by Audio Kinetic. They both have different features and people get really, it's really funny because people get really dogmatic about which software they like to use, sort of like the duality between Unreal and Unity, but it happens with audio nerds. They like to get really stuck into these arguments, it's hilarious. So you'll do all your authoring into a tool like that, and then it exports out all the files in a special format that the game engine also understands. And then the programmers go and make the game not crash in the game engine. And they also do like a whole bunch of extra audio programming on top of that. So very special behaviors that help control what the sound does when and how it responds to the player. And then you hit the build button and it hopefully all works. But like dispersed through all of that, there's also a whole bunch of other tools that you can use as well. And there's like more and more coming every day. Not every day, but like every second week it feels like. If you're working in the VR space, you'll probably also be using a plugin, like maybe there used to be 3Dceptions. now it's the Facebook 360 Spatial Workstation. There's also a company called RealSpace that has their own spatializer. Of course, ASIC is developing our own pipelines as well as trying to fit in with other people's pipelines, like Steam Audio, that's another one that's out there now as well. All these have different features that they do, and these particular spatialization plugins, as they do, I just call it the math in inverted commas, like the magic maths. to help people hear things where they actually come from in 360 degree space because you don't get 360 degree sound through just stereo or through just surround. You actually need to have a very special piece of software like these spatializer plugins.
[00:16:38.990] Kent Bye: Yeah, and it sounds like I get this image in my mind of like you're building the railroad tracks and then the train is coming down and you're kind of doing this iteration where you're trying to design the audio with a game that may not be fully developed yet. So there's this kind of back and forth of the game actually has to get to a certain point because, you know, essentially you're assigning audio to audio objects that are in the game. And so how do you go from FMOD and assign audio to specific objects that are in this kind of immersive environment?
[00:17:09.753] Sally Kellaway: Actually, it's kind of good now. Most of these tools will have integrations so that you can work to some extent within Unity or Unreal and design your own logic. I also do a little bit of scripting myself in C Sharp so I can hack together code, which then a programmer will usually come through and be like, oh my god, I can't believe you did it this way, or it makes the game crash and then I get in trouble and do it anyway. So going from having your sounds in FMOD and then actually having them work in Unity or Unreal, These plugins, it's like an intermediate piece of software that operates within Unity Unreal which exposes a bunch of the features from Fmod or Wwise in a way that the game engine can kind of do things with them. I guess that's the simplest way to put it. It's still sort of complex but it's really cool in games and like any sort of simulated space because essentially you're getting a lot of the 3D information for free because like if this was a simulation and you're standing in front of me and you're a zombie like I'll be like okay so that zombie growling noise I'm just gonna put that on you as like a 3d object in the game engine and we're gonna say okay so only growl this many seconds and you know then be quiet for this many seconds and then go back to growling and then be quiet but you know maybe if you get really close to me I'll give you a different sound file so that you sound even hungrier for my brains so all these sorts of behaviors like you take a step back from being like a super creative hand-waving person that I'm being right now and you just kind of go all right What's the system that gives my player clues about what's going to happen? Because like essentially a piece of software is like it's just a big fat system and you sort of need to feed little mini systems into it so that it knows what to do when because it's like a computer just follows instructions. So you sort of need to take your creative design where you've been waving your hands around and making like lots of cool sounds and kind of fit it into a structure that the computer will understand. That's what being a technical audio designer is all about.
[00:19:06.941] Kent Bye: Yeah, I get this picture as I'm hearing you talk. There's a number of different components here. There's kind of like the underlying game engine, which is taking care of all the 3D graphics and the virtual space that you're going to be interacting with. And then the audio middleware is kind of like a more refined authoring tool to be able to have more control and maybe modulate the sounds a little bit more as you are interacting with this game engine. And then on top of that there's sort of like the audio spatialization which in some ways is trying to characterize the physical space of that area and the reflections and all the sounds so that it actually sounds like you're in that space. And then I guess the question I have though is that most audio output is that also being kind of rendered out to like stereo binaural audio. But yet with AUSIC you potentially have the capability to take more sophisticated audio, like maybe an ambisonic output, or maybe you need more channels, or more information that is normally coming from those audio specialization plugins. And so, from your perspective at AUSIC, it sounds like you're in the process of potentially developing your own specialization. system to be able to output those and also potentially also working with these other plugins to be able to get the type of output that you need. So I guess the question is like what output beyond binaural audio can AASIC handle?
[00:20:26.175] Sally Kellaway: So to answer your question of what output beyond binaural can also handle. So we can deal with binaural if you have pre-processed audio, we'll have pipelines for that. We'll also be able to spatialize stereo output. We'll be able to spatialize surround output as well. So when I say spatializing these traditional formats, it's literally taking it out from the inside of your head and putting it in the world. instead of it just being like left to right existing on the axis of your ears actually being able to put that a bit further out in the space to give you a bit more your brain a bit more room to breathe almost. When it comes to surround content that's a little bit easier to understand because like if you have like 5.1 surround you'll have a center channel, left front, right front and then you'll have your two surrounds out back so stuff that's mixed to sound like it should be coming from behind or should be coming from out front we'll put that in those places so that you actually hear in those locations. Even better for 7.1, which is like the actual standard output for things like the PlayStation 4 and stuff like that. So that's super cool and super useful. But with games, games are even a little bit special and different because with games and something that's been happening in the last couple of years is we've seen the whole advent of 3D audio, object-based audio as well. So where we're able to take those audio objects, we can actually put them in the space as well. So literally, I love using zombies as examples because they're just so funny to me. They make squishy sounds and all sorts of stuff. literally just saying well that sound is attached to that object and as you see the zombie running around the sound is just going to follow that zombie but we'll also be giving people the capacity to lay in like a surround track as well so that if there's like a bunch of like environmental sounds that are not very important and don't need the cpu to be processed in real time to be like a bird flying over there when the zombie's like chewing on your left ear like that doesn't make any sense to be like prioritizing a bird at the same level as a zombie. So you just, like, put all your bird sounds, like, somewhere, like, just in the ambient environment all around you and actually save your CPU for the really important sounds that are going to help the player know to turn to the left and be like, hey, stop chewing on my ear, man. That's really rude. Bang.
[00:22:40.149] Kent Bye: Yeah. For you, I'm curious to hear some of either 2D games or immersive VR experiences that you think have really great sound design.
[00:22:48.823] Sally Kellaway: I used the example from Bioshock before because I just think Like, as a total package, like, the audio design and delivery of that game was just amazing. And of course, like, it helps. It was like one of those Genesis moments for all of the ideas that I now execute in my professional career. Obviously, like, just so immersive and being trapped in it and having, like, the music design on point as well. Just a really perfect, like, amazing game. I also really liked... I hate scary games, but Dead Space. Dead Space is a really super great game. Again, like, this whole idea of being trapped on a spaceship and, like, shit has hit the fan and, like, there's these, like, evil aliens that have, like, torn apart all these people and you get to see blood and guts and all this kind of stuff. It's, like, super scary and the sound design is really great, but I just loved the music so much because it's actually, um, the way that Jason Graves wrote the music is just very, very different to normal music. He used a technique called aleatoric music composition, so that's kind of like, uh, we call it chance music as well. where you can give like the orchestra or the strings like instructions about some instructions of what to do but also just be like just make really bad sounds where it might be dissonant or screeching, tapping, scraping all that sort of stuff together and doing that in a way where it's all structured so you get really like low intensity scary bits sometimes and like really high intensity scary bits that are just really overwhelming because the whole string orchestra is doing it at the same time and then actually tying that into the game so that when the player is creeping towards one of those spots where those necromorphs are going to drop from the ceiling like the music just ramps up so much and if you're anything like me you're just like dying by the time you get there And they also did this thing, so cruel, where there's also false triggers in there as well, where they'll be ramping up the music when something's not going to happen. And if you're as much a scaredy cat as me, you're literally stopped at a point where you're just like, I can't go any further. Because you've worked out what the system is, right? But other people love it. I dig it for the effect that it has on me, because it really makes me feel something. Those two games, they're some of my favorites. I haven't been playing very much lately because I've just been working too hard.
[00:25:04.577] Kent Bye: Yeah, and I'm curious to hear some of your thoughts about what you're going to be doing at AUSIC. It sounds like you're going to have the opportunity to build some demos to really show off the technology in a certain way. And so I'm curious to hear from you what was really exciting to you to come work for this, you know, hardware manufacturing company that was really developing kind of the next iteration of immersive audio and what you can do with that.
[00:25:27.667] Sally Kellaway: So throughout this interview I've spoken quite a lot about my history and where the genesis of a lot of my interests that I practice in my professional career now came from. And I've come from playing music, I've written a little bit of music, I've come from games and simulation and VR. I've touched a lot of different parts of like this pipeline in a way and this like spectrum of like interactive media and it's always been about like the audio and the sound and just the technical practice of that like I didn't mention this before but I've also like done my masters in in acoustics as well so I have a really deep interest for the science behind how people hear and how that precipitates into an experience that works in industry. So this is like the perfect time for someone like me to actually be out there just like mucking around with things. So last year I was working for a VR game company back home in Australia and like I wrapped up my contract and I just emailed Aussic because I have a couple of contacts that work there. I just said, oh, can you please not use this email anymore? And they were like, oh shit, are you free? Do you wanna have a chat to us? And I just went, whoa, whoa. a sound like such a nerdy tech sound person like me and a company that literally wants to change the world and how people hear sound and how content creators actually use sound in new media and specifically VR and AR and any R that comes in the future. It's like match made in heaven. So I'm super stoked to be working for AASIC and a lot of what I'm going to be doing is actually coming to conferences like SVVR where we are now. We were at GDC a couple of weeks ago. We'll be at VRLA and Vision Summit and every other summit it feels like. But a lot of what I'm doing is speaking to content creators and other developers and people that have cool tech, you know, the big game engines, the everything, everyone, so that we can kind of all get on the same page about what 3D audio and spatial audio should be in our new realities and really help bring the best technology possible to the rest of the world. So hopefully I get to do that. So that means a lot of biz dev, business development stuff, but you know, I love talking to people and I love talking about what I'm passionate about and just trying to show them a way that they can really augment their experiences with sound. We'll be doing a lot of that and hopefully I'm working towards making some demos as well. and just helping bring the product to market. So we're going live in the Northern Hemisphere summer, which is just in a couple of months, actually. And yeah, just trying to prep everything so that we're communicating the best way possible, communicating the best documentation, the best help, the best content, the best everything. Yeah, just trying to be the best, basically.
[00:28:08.986] Kent Bye: Well, I'd be curious to hear your thoughts on this paradigm shift that AUSIC is really advocating for in a lot of ways. I mean, you're talking about going to the traditional music industry, for example, where they're very used to having a very fixed pipeline for how they produce audio. And yet to do this paradigm shift into interactive game engines, to me, feels like it's a complete change of how they even think about what they do. And so you're adding agency to music for the first time. And so I'd be really curious to hear how you think that's going to necessarily play out, if it's kind of like thinking about it in terms of designing a physical space that is reacting to you, rather than you being located in one place and having an experience delivered to you that's completely authored.
[00:29:00.013] Sally Kellaway: It's interesting that you separate those two types of experiences and kind of like, I don't know whether you're really trying to polarise them or not, but they kind of exist on a spectrum and in some ways almost I think it's like it turns out being an axis as well, where like AUSIC as a technology applies to not just VR and not just interactive or immersive media or VR, AR, but it also belongs to traditional media and helps people keep in touch with that traditional media. and hear it in a way that, you know, is similar to what we've heard before, but is also a little bit better in some ways. Like, if you just have your stereo stuff, like, placed out spatially in the world, like, that'll be a little bit easier on your ears in some ways. Or, like, that favorite artist that you've heard, like, stereo music from might be like, whoa, I've been waiting to, like, break out this whole idea, like, of breaking out from the stage and this whole idea of, like, putting all the instruments in one spot in front of you or in your head and actually, like, putting them in the whole world. in some ways it's like this, it is a paradigm shift because you're just asking people to change the way they think about music and sound and for it to be just like real life. And I'm putting my hands way out in the air because I'm just trying to convey the fact that 3D audio and like spatial audio and all the ways we describe what we're trying to do is just, it's the way that you hear in real life. So why can't you have that normally through any sort of media that you consume? That's just what we're trying to do Yeah, I hope that it goes over and that people get it because it is a lot of mental arithmetic to do for some people, but I feel that when they hear it, they really believe it. And yeah, we just want everyone to hear it and kind of get it and get on board. And particularly for content creators, like there's like a whole world for them to explore with so much more creativity that they can have when they just open up from stereo and go to 360.
[00:30:51.273] Kent Bye: Yeah, I guess the reason why I was putting in that dichotomy in some ways is from the perspective of a musician who's used to just creating music and putting it out, they haven't necessarily thought about user agency, of how we're going to take in this user agency and really incorporate it into our workflow, how we're going to really account for that. And so I guess that's the thing that I see that's new, is really having people who are coming from more of the music production, how are they going to introduce agency? And what are some metaphors that they can use to really think about how to do that?
[00:31:20.976] Sally Kellaway: It's interesting, we deal with this in game audio as well, we deal with this when people kind of go, oh I'm sort of interested in writing music for games and you sort of have to teach them a little bit about what dynamic or adaptive music is and why it's different and why you have to think about it, like not in terms of just going from like 5 seconds, 10 seconds, 15 seconds, 20 seconds, like linear time but then also thinking about like how intensity and other effects within music, like structure and content arrangement, all those sorts of things can shift to help complement what's happening in an interactive environment. So that's something that we naturally do in games anyway, there's always new people that are trying to come in and experiment and try and find their creative expression pattern musically through that medium. But then when we think about interactive and VR as well, you're giving people another new paradigm to think about too, and then saying, oh by the way, spatial audio and 3D audio is a thing, so just think about that as well. I think it takes a while to kind of wrap your head around all of that sort of stuff. I mean this is something we're trying to work on as a whole industry and the reason that we had like the panel today and every other panel that we'll have in the future where we get like audio professionals together and we just get them talking about their creative practices and why they've chosen to do specific things in specific circumstances for specific projects and what they do, how they think, what challenges they face and how they solve them. It's because there is no one way to do anything in this field and there's just so many different types of jobs that you could get and projects that you could work on. that you need to have that smorgasbord of ideas and practices that you can pick from to experiment and try. We really want more people from diverse types of media and diverse backgrounds to come into this space and go, well, in film we do this, and in simulation we do this, and at NASA we do this. And when we're doing this weird musical thing from a country that I've never heard of before, we do this. They might have a way of dealing with this new paradigm that we've never thought of. I've never thought of coming from Australia or other people have never thought of coming from like Los Angeles or any other country in the world or city in America, you know?
[00:33:27.908] Kent Bye: Awesome. And finally, what do you think is kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:33:36.635] Sally Kellaway: The ultimate potential of virtual reality. We've really seen, and I've had conversations with people this week that really mirror my perspective on this. When it comes to audio technologies and all the different types of processing and research that we've had happening in the world in the last, literally, if we're talking about ambisonics, it's 40 years. If we're talking about binaural, it's over 100 years now. All these technologies, they've sort of just existed on their own and tried to be the scrappy people that you know, scrappy technology in the corner that nobody pays attention to but one researcher thought it was a really good idea to come up with it and now there's like two abstract artists over there somewhere that are making like binaural radio players and putting that on SoundCloud somewhere and they haven't really gotten much traction but virtual reality is given some of these like weird audio technologies a home and something that has actually really kick-started like those technologies into being something that's not only commercially viable but something that's actually developing at a much quicker rate now and becoming part of our workflows and pipelines and creative practice in a way that is like I've never experienced before which for me as a content creator it's like it's super exciting but for virtual reality it's also really exciting as well so I don't want to say that virtual reality is like super cool because it's just a vehicle for audio but there's this like beautiful symbiosis that's happening. I really feel that when it comes to the future of virtual reality Being able to really experience such a depth of sound and to see where that goes and how virtual reality can really help, particularly musicians and sound designers think about sound spatially in a way that we just can't do on a screen, on a 2D screen, or even with our imaginations, because it's just too hard. Being able to see all that information, that sound information, and work with that in a way never before, that's my big wish.
[00:35:33.115] Kent Bye: Awesome. Well, thank you so much.
[00:35:34.635] Sally Kellaway: Thank you.
[00:35:36.298] Kent Bye: So that was Sally Calloway at the time of this interview. She was a creative director for AASIC and is now a freelance sound designer and VR audio strategist. So I have a number of different takeaways about this interview is that first of all, I was really struck by how the whole realm of interactive audio is a new paradigm that isn't necessarily locked down to the point where there's one specific way to do it. It's kind of like this iterative process where I was really struck by what Sally said is that oftentimes she's coming into a video game where things aren't completely finished yet and that she has to kind of figure out where to start in this big game that's still being developed and that she'll go into an experience and then start to iterate and add different dimensions of ambience and layering and texturing. If you're familiar at all with something like Unreal Engine or Unity then you know that there's these 3D spaces and that as you come into proximity to different 3D objects and that can trigger different audio events and so to me that's kind of the best metaphor that I have for thinking about these interactive audio experiences is that that's object-oriented. and that you are attaching audio to these objects and these objects are moving through space and then there's different layers that they have in order to actually produce that sound so there's the digital audio workstation that is kind of like the most precise way to control the waveforms and add all these different effects and whatnot and then There's a middleware that interfaces with the 3D audio. And then within that 3D environment that's being artificially generated, then there's going to be different ways to simulate the spatialization of that sound. So it sounds like it's actually going out and reflecting off the walls. And you get this sense of being within a physical space when you have that spatialization that happens. And that there's these different layers that are there that she's working with. And that there's both the plugins for FMOD, but also the Ys with two Ws. In that as a sign designer you are really creating this overall vibe and ambience where you're able to really dial in a lot of the emotional tenor of what you're trying to create. One of the questions that she asked when she was starting to do the sound design is how do you want someone to feel? Also, this whole idea of chance music or allyoric music, which is that there's a certain amount of giving agency to each of the individual players. Allyoric music in a symphony would be to tell a violinist, okay, just try to do something to make your violin sound really creepy. And then if everybody's doing that, then that's a way of generating this kind of like eerie feeling collectively. and that they're able to do that in some of these different experiences where you're able to use this aleatoric music or chance music to be able to build up that sense of tension. And I think Sally also reiterated one of the points that Jason Riggs had said, which is that, you know, the future of music is going to be both immersive and interactive. And one of the things that she said is that you have some of the most immersive experiences when you have a lot of agency within that experience. And I think that's why a lot of like sound design within video games has a lot of things to teach the future of these immersive and interactive stories and narratives that come to virtual reality. And that a lot of these different weird audio technologies have been around for many many decades with both ambisonics but also binary audio but that it's virtual reality that is able to give these audio technologies a completely new home because you're able to have the associated visuals to go along with it and that The audio is adding a whole other layer of dimension to the experience in that our eyes are only going to see one small wedge of any 360 degree experience, but that it's our ears that are able to actually have this fully 360 degree sense of awareness and spatialization. And I think that as we start to move forward within virtual reality, people are going to realize the deep importance of putting much more attention into having very robust audio experiences to go along with these virtual reality experiences. And from this interview, it sounds like what Sally is saying is that this is still a fairly early art that's still evolving and developing and that there's no one way to be able to both plan and produce and execute and do the post-production of some of these immersive audio experiences. So there's many different approaches and philosophies for how to do that. So that's all that I have for today, and I just wanted to thank you for joining me on the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member to the Patreon. Just a few dollars a month makes a huge difference to allow me to continue to bring you this coverage. So you can donate today at patreon.com slash Voices of VR. Thanks for listening.
