sundance-new-frontier-2020
In my final episode of my coverage from Sundance New Frontier 2020, I speak with chief curator Shari Frilot about some of the themes of the line-up of XR experiences that were programmed this year. There were a number of themes the embodiment, social dynamics, and integrating more aspects of theater culminating in the new Biodigital Theatre this year featuring two social VR experiences (Atomu for 5 people and All Kinds of Limbo for 20 people) as well as Anti-Gone, which was a mixed reality play with a capacity of around 50 people.

We also talk about the dynamics of empathy, and how pieces like The Book of Distance and Still Here are providing embodied interactions that create connections to the pain of experiences. Frilot differentiates how a virtual reality simulation can’t replicate the full experience of trauma, but it can create embodied situations and environmental contexts that help to evoke the pains of an experience. Immersive experiences are painting with memories and embodied experiences of the audience, which means that XR as a medium can evoke associations of our own traumas, but it’s impossible to recreate the full context and character of experiences that unfold over a significant period of time (such as being incarcerated for 5, 10, or 20 years).

Another theme that came up this year was the increase of skepticism of technology. That’s seen in documentary films like Coded Bias and The Social Dilemma this year, The Great Hack and The Cleaners in previous years. This was explored more in depth in the New Frontier VR piece Persuasion Machines, and Frilot is actively looking to artists and creatives as they explore the dynamics of this emerging data economy and where they could fit into it.

Finally, we look at some of the underlying affordances of immersive storytelling and how it differs from other mediums like film. We explore the intersection of different mediums that had an influence on narrative feature films Zola and Spree, but also the combination of iPhone video composited in a cubist 360 video in Flowers and a Switchblade, the fusion of theater, Twitch.tv streaming, and open world exploration in Anti-Gone, the fusion of 360 video and literature in Go, and the combination of group ritual, embodied performance, and dance in Atomu.

Here are links to all of my Sundance 2020 coverage:

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s my live coverage of Sundance New Frontier 2020

Here’s my tweet storm of all of my podcast coverage of Sundance New Frontier 2020

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

new-frontier-central
Milo Talwani (pronouns they/them) is a curator, and creative technologist who oversees the selection process for New Frontier at the Sundance Film Festival. They’re also the technical coordinator for Tribeca Immersive, and they also help curate exhibition events such as the Robot Remix XR.

I talk with Talwani about the process and challenges of curating and selecting immersive projects for the New Frontier in collaboration with chief curator Shari Frilot. Throughput is always a challenge with location-based entertainment experiences, but especially at Sundance where there’s a lot more people than capacity to be able to screen all of the work (I approximated the throughput for each of the Sundance experiences in my real-time Twitter coverage.)

There are only a couple of viable solutions to the throughput issue. Talwani says that either Sundance curates less pieces, or they find additional funding to support expanding out to multiple locations in order to show the existing selection to more people. It’s an impossible tradeoff, and the Sundance staff has preferenced showing a wide diversity of content that they find worthy rather than attempting to expand out the screening capacity to increase the numbers who can see the work.

These capacity limitations have inspired Talwani to collaborate with Paisley Smith in curating their own show that focused on one hour of XR content, and that a ticket buyer would be able to see all of the work that was shown. Here are Smith’s Robot Remix XR field notes reporting on their intentions and lessons learned from this experiment.

Another key feature of Sundance screenings are the installations, and Talwani describes the increasing importance of onboarding and offboarding, and how this is a key consideration for submitting to Sundance. The New Frontier is attempting to create a liminal space of exploration, and so the transitions into and out of experiences is a key factor for the overall program of experiences that they’re curating.

We also explore their journey into becoming an XR curator, the challenges around distribution of Location-based entertainment, some of their favorite experiences, the dream of immersive themselves into a VR experience with interactive characters for an extended period of time, and some of the themes and trends from the 2020 slate of experiences at the Sundance New Frontier.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s a trailer of the Robot Remix XR exhibition that ran at the Sp[a]ce Gallery in Pasadena, California every weekend starting on Saturday, July 6th until September 8th, 2019

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

metamorphic
One of the most powerful explorations of embodiment at Sundance New Frontier 2020 was Metamorphic. The creators describe it as exploring the “ephemeral nature of the self by allowing participants to redefine their relationships to their own bodies, to the bodies of others, and to the environment they share.”

The visual aesthetic is quite amazing in this piece. I’ve already been a fan of Wesley Allsbrook’s Quill art work in Dear Angelica, Sun Ladies VR, and 30-Seconds of Gunfire, but there’s something different about how they created their own custom backend to be able to parse and reanimate the Quill brush strokes with more fidelity and responsive control.

The brush strokes are dynamically animated both within the world around you, but also within your own embodiment. There are actually multiple timelines of animation that are contingent upon your interactions with the world around you including other people. It’s ends up producing a cutting-edge exploration of context-dependent, embodiment that is able to cultivate entirely unique and novel social dynamics.

At Sundance 2020, I had a chance to sit down with the creative team of Sensorium co-founders John Fitzgerald & Matthew Niederhauser, artist & writer Wesley Allsbrook, and programmer Elie Zananiri. We talked about the need to create an infrastructure to facilitate the dynamic animation of art, the collaborative tensions that can come between the creative talent and the engineering implementers, the throughput challenges of location-based entertainment experiences like this, their next milestones and desired architecture for scaling up the viewing capacity by multiple orders of magnitude, new exhibition opportunities at the ONX Studio by Onassis USA and New Museum NEW INC, and the challenges associated with cultivating second-order, emergent social dynamics within an immersive experience.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s the trailer for Metamorphic

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

animalia-sum
ANIMALIA SUM is an animated short VR 360 video / application hybrid with some light, embodied, 3 degree-of-freedom interactions on the Oculus Go. It’s a mockumentary exploring the potential of whether or not humans should consider moving from animals to insects as their primary source of protein.

The visual aesthetic is very striking in that they primarily use hand-sculpted minatures that have been volumetrically-captured with photogrammetry, rigged in Cinema 4D, animated by a mix of hand animations and a Perception Neuron Motion Capture suit, edited in Adobe Premiere, and then imported into a 360 video player wrapped in a Unity application in order to add additional embodied interactions.

The workflow and production pipeline of this piece is unique and extremely impressive as the creators Bianca Kennedy and The Swan Collective were able to achieve a unique, hand-crafted aesthetic that has a lot more character than the typical, purely CGI-generated art has. They’re able to infuse additional layers of personality and charm to add to the already humorous send-up of the classic British nature documentary in the vein of David Attenborough.

It’s hard to fully describe the experience of ANIMALIA SUM, but at the heart, it’s a light-hearted comedy that continues to surprise and delight for the entire 9-minute duration.

I had a chance to catch up with Felix Kraus, founder of The Swan Collective, as well as Kennedy to talk about the evolution of this project, their experiential design process, the difficulties in trying to storyboard and plan a piece like this versus just iteratively creating it, their production pipeline process for this piece, as well as how they were able to overcome the throughput challenges of Sundance New Frontier by having five seats during the ticketed sessions at the Ray.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s a trailer for ANIMALIA SUM that gives you a sense of the visual aesthetic:

Here’s some footage of their Sundance screening installation at the New Frontier at the Ray location:

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

anti-gone
Anti-Gone is a mixed reality play that features two actors performing in front of a wall that has an open, virtual world projected on it. One of the actors spends the majority of the time in VR navigating this world and interacting with the other actress who is a motion-captured suit. They each have simultaneous embodiments in both the virtual world and the co-located, physical reality as the story unfolds through a series of choices they make on their journey through this virtual world.

The narrative structure is based upon the comic book also titled Anti-Gone by Connor Willumsen, and the mixed reality adaptation is described as “set in a post–climate change world raging with late-capitalist rebellion and excess, Anti-Gone is a quest for happiness in the sunken city, where the characters shop, score drugs, and hotbox cinema.” The narrative structure is an inverse tree in that the characters have more options at the beginning of the show, but it narrows as time goes on and always ends the same way.

This gives the characters a lot of latitude to make a lot of choices at the beginning of this experience for how they want to navigate this open world. The interactions are pretty low-key, hanging out, chilling, and largely center around taking psychotropic substances that invoke a variety of different altered states of consciousness.

For me, the implications of the technological infrastructure behind Anti-Gone is one of my biggest take-aways from this experience. They self-describe Anti-Gone as “part theatre, part Twitch.tv, this groundbreaking play is a feature-length livestream game telecast.” It is a seamless blending of the virtual and the real, and the virtual world is being driven by an open session of Unity where the director and lead artist Theo Triantafyllidis is controlling a virtual camera rig with five different perspectives. He can control NPCs in real time and trigger virtual events based upon what the actors do, all while being the actual virtual director of what cinematic, window/portal view that the audience and actors see of this open virtual world.

There’s a lot of other live elements including musician Cameron Stallones and lighting, as well as an off-stage actor to provide additional narration and other characters that appear throughout the narrative experience.

I had a chance to sit down with director Triantafyllidis and musician Stallones to talk about the evolution of this project, their inspirations from Twitch.tv livestreaming and Dungeons & Dragons, how they formed the bulk of the potential narrative arcs in rehearsals, what aspects of the experience are live, dynamic & changing, and their overall experiential design process that is blending the virtual and the real, the improved moments and planned narratives, and the open world exploration and more limited set of possibilities.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

machine-for-viewing
A Machine for Viewing was a series of three film essays that were shown to someone in virtual reality, whose first-person perspective was then projected onto the Egyptian movie theater screen at Sundance with a crowd who was watching the film essays through the eyes of the one person in virtual reality. There were embodied interactions that the person in VR would perform, and so the audience member became a performer who’s interactions would slightly modulate what the rest of the audience would get to see. So the embodied interactions of the viewer could shift the aspect ratio of the film, which was a film essay about the evolution of aspect ratios throughout the history of film.

There are many layers of Inception-like meta-analysis of using VR to watch someone watch a film essay in VR about the evolution of communications mediums while watching how the new VR medium can simultaneously modulate the individual and collective experience of said media. A Machine for Viewing was a collaborative project between VR artist and VRTOV creative director Oscar Raby, educator, filmmaker, and “virtual reality tourist” Richard Misek, and documentary filmmaker and “guest/visitor to VR” Charlie Shackleton. I had a chance to talk with the three lead artists just after their final screening at Sundance 2020 to explore what film can learn from VR, and what VR can learn from film.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

chomsky-vs-chomsky2
Chomsky vs Chomsky: First Encounter is a virtual reality experience where you get to interact with a virtual representation of Noam Chomsky, and essentially ask him any question that you want to. Trained on a corpus of 60 years worth of Chomsky interviews and data covering a wide range of his expertise as a linguist, philosopher, cognitive scientist, social critic and political activist. As with all natural language processing applications, the detection and comprehension of the input speech can be hit or miss, and then there’s the question as to whether or your not your inquiry will be matched up with a contextually relevant response that’s synthesized in real-time.

So it’s still early days aiming towards the dream of artificial general intelligence, and so constraining the bounds of possibility within an immersive narrative can help showcase what AI can do successfully. When an inconsistent or incomplete answer came back to one of my inquiries, then I found that it was a stark reminder that I was interacting with a primitive machine that had a hard time understanding my deeper meaning. But when there was a contextually-relevant, direct response to question, and even sometimes joyfully novel or interesting, then it’s a magical experience that increased my sense of social presence and gave me some early glimmers of a feeling that I was interacting with an intelligent entity.

However, this form of plausibility illusion is like a house of cards, and it doesn’t take much for my suspension of disbelief, expectation detectors to get triggered and for me to be reminded of the limitations of AI. Perhaps part of the point is to demystify the capabilities of what AI can do, but it’s still worth iterating on and incrementally increasing the capacity, accuracy, and training of their models. This was one of the unique experiences at Sundance this year where each interaction was helping to train and improve the underlying models.

I had a chance to do an interview at Sundance with lead artist Sandra Rodriguez, interactive developer Cindy Bishop, and visual designer Michael Burk to unpack the evolution of the project and their experiential design process. Combining interactivity and user agency with vignettes of immersive stories can be a challenge when you’re working with a machine learning black box that makes it hard to predict how it’ll react to a given input at any given time (and how even that will change over time). It’s a moving target, and they shared some of the milestones they were able to achieve and whats still yet to come in order to have more of a memory and a context-preserving functionality in the future.

Like I said, it’s still early days for these types of AI-driven narratives with virtual beings and conversational interfaces, but they’re continuing to learn and get better over time and so it’s important to keep iterating, experimenting, and trying to find the right constraints and narrative contexts in order to hide some of the current limitations of comprehension and reacting in a contextually-appropriate way. And I’m glad to see groups like The National Film Board of Canada, Schnellebuntebilder and EyeSteelFilm continue to experiment and push forward what’s possible with the technology that exists today.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s the trailer for Chomsky vs Chomsky: First Encounter

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

solastalgia

Solastalgia is a site-specific augmented reality experience on the HoloLens 2 where you explore the surface of a depopulated planet in ruin, and colonized by a number of human ghost holograms. “Solastalgia” is a neologism coined by philosopher Glenn Albrecht in 2005 that describes a form of emotional or existential distress caused by environmental change. At Sundance, I went through through this experience with another audience member, but there isn’t any social interactions built into the experience as we’re each experiencing different aspects of the narrative at our own pace.

I had a chance to catch up with Solastalgia co-creators Pierre-Alain Giraud and Antoine Viviani at Sundance where they talked about the evolution of this project, their experiential design process, the differences between the smaller and shorter Sundance version and the larger and longer version that’s playing at museums around the world, and how they wanted to use immersive technologies to look into the future of the Anthropocene epoch, and explore the “potentially redemptive power of technology and the scientific forecast of a compromised future.”

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

all-kinds-of-limbo

All Kinds of Limbo featured the ability to show a volumetric-captured, virtual reality performance to 20 users within the Oculus Quest. Nubiya Brandon narrates her experience of being a mixed-race woman in the United Kingdom, and takes a tour through the musical genres of reggae, grime, classical, and calypso as the theatrical staging is shifted through different worlds.

I had a chance to unpack the experience with Brandon as well as Toby Coffey, who is the head of digital development at National Theatre where he runs the immersive storytelling studio. We talk about the evolution of this project, experiential design process, how they designed for large-scale social interactions, and how the National Theatre started an entire publishing arm in order to publish projects like All Kinds of Limbo, which can be heard on Spotify.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

The National Theater collaborated with Accenture Interactive, and here’s a video they produced about All Kinds of Limbo.

Here’s a behind-the-scenes video from the National Theatre

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

book-of-distance
The Book of Distance is one of the best VR narratives or immersive stories I’ve ever seen. Director Randall Okita tells the story of his grandfather Yonezo Okita’s journey from Hiroshima, Japan to Canada in 1935 to begin a new life. It has some theatrical staging with motion-captured actors as you move from traveling to homesteading, and then eventually being forced into a Japanese internment camp during the World War II and exploring the nuances of the state-sanctioned racism that followed.

There are moments when you become an embodied participant going through the motions through each of the scenes, which allowed me to drop more fully into the story and have a much more immersive and viscerally emotional journey. It’s as if the story is in my body now in a profoundly deeply way, and even just writing about it brings back swells of emotions as I think about my experience of it at Sundance.

I had chance to unpack the experiential design process with lead artist Randall Okita as well as National Film Board of Canada (NFB/interactive) producer David Oppenheim during Sundance 2020. We explore the design aesthetic of Japanese woodblock prints, evocative character design, the choreography and motion captured scenes performed by Okita himself, and the subtleties of interaction design. Overall, The Book of Distance packs in so many immersive storytelling innovations that I really hope that general audiences will be able to experience it soon because it really showcases the power of what virtual reality narratives can achieve.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality