OSSIC’s Jason Riggs told me at CES last year that the future of music is going to be immersive and interactive. Interactive sound design where user agency is considered a part of the experience is a key ingredient to creating fully immersive audio experiences, and the AR & VR tech platforms are creating new opportunities for musicians to explore interactive music experiences. Sigur Rós’ interactive music collaboration with Magic Leap was recently featured in Rolling Stone, and that interview reminded me of a deep dive discussion that I had with VR immersive audio evangelist Sally Kellaway talking about the fundamental principles of immersive sound design. Kellaway provides an overview for how audio designers for games have been creating demand for toolsets and plug-ins with existing Digital Audio Workstations to help break out of the normal linear, authored paradigms. At the time of my interview, Kellaway was a creative director for OSSIC, but has since moved on and is freelancing as a VR/AR audio designer, strategist, and immersive audio evangelist.
LISTEN TO THE VOICES OF VR PODCAST
Because designing an interactive experience is an iterative process, then there’s no one philosophy or paradigm for how to plan, execute, or post-produce all of the audio components of an immersive experience. The process is driven by the tools that are available, and so Kellaway described to me some of the most popular production pipelines and workflows for immersive audio. There are plug-ins to Unity and Unreal like FMOD and Wwise, but there are built-in, object-oriented audio features in Unity & Unreal, as well as a range of different audio spatialization options for VR.
Hearing is sense that can sense a full 360-degree space, and it adds a layer of emotional engagement that speaks to the more primal aspects of our brain. There are also a lot of decades old audio technologies like ambisonics and binaural audio that have found a new home with VR, and so the surge of immersive technologies of AR and VR as a technological baseline is catalyzing a lot of innovation and experimentation within the realm of music and sound design. The Magic Leap Tonandi music demo by Sigur Rós shown to the Rolling Stone journalist Brian Crecente was not “a recorded piece of music”, but more of “an interactive soundscape.” Crecente describes his experience:
Tonandi starts by creating a ring of ethereal trees around you and then waiting to see what you do next. Inside, floating all around me are these sorts of wisps dancing in the air. As I wave my hands at them, they create a sort of humming music, vanishing or shifting around me. Over time, different sorts of creations appear, and I touch them, wave at them, tap them, waiting to see what sort of music the interaction will add to the growing orchestral choir that surrounds me. Soon pods erupt from the ground on long stalks and grass springs from the carpet and coffee table. The pods open like flowering buds and I notice stingray-like creators made of colorful lights floating around me. My movements, don’t just change this pocket world unfolding around me, it allows me to co-create the music I hear, combining my actions with Sigur Rós’ sounds.
This is in line with what OSSIC’s Jason Riggs predicted as the immersive and interactive future of music that VR & AR technologies will be to enable. The OSSIC demo at CES last year was a really impressive immersive audio experience that allowed you to fully interactive with the audio environment. OSSIC’s audio hardware made it an even more compelling spatial audio experience, but it’s just the beginning of how VR and AR will change how music is composed and experienced.
Will the existing linear authoring, audio tools adapt to become more spatialized? Or will the game engines have integrate enough plug-in support to DAWs to become the defacto production pipeline of audio experiences? Either way, AR and VR technologies will enable new distribution platforms for audiences to experience the spatial dimension of sound in a way that’s much closer to how we hear the world every day, and it’ll enable musicians like Sigur Rós and Miro Shot to push the boundaries what’s possible with spatialized music.
This is a listener supported podcast, considering making a donation to the Voices of VR Podcast Patreon