arte-360

kay-meseberg
ARTE is a French German television network that promotes cultural programming that was started in 1992, and they’ve been pioneers in the digital space for a long time. They’ve also funded over 60 VR projects over the past 5 years including many landmark projects including Notes on Blindness, Battlescar, Gloomy Eyes, Alteration, and I, Philip.

Kay Meseberg is the head of mission innovation at ARTE looking at the the “TV of After Tomorrow,” and so he’s been involved in looking at the immersive storytelling potential of virtual reality, augmented reality, and artificial intelligence. Meseberg authored a paper with Regina Kaplan-Rakowski about the “Immersive Media and their Future” where they look into the past to see the evolution of previous communications mediums. They saw that there was almost immediate benefit for every previous medium, and they’re seeing very similar patterns for XR. It’s in part of these insights, that ARTE has been so forward-looking and a believer in immersive technologies as a new medium for storytelling that they’ve invested heavily over the past five years in working with a number of immersive storytellers from around the world to push forward what’s possible from a storytelling perspective.

I had a chance to catch up with Meseberg at the IDFA DocLab where we talk about his journey into tracking the intersection of story and technology, his research into how VR fits within the larger trends of previous communications mediums, the work they’re doing for digital distribution as well as experimenting with location-based entertainment, as well as some of the production highlights from the roster of more than 60 immersive narrative titles that they’ve produced.

We also talk about French media theorist Bernard Miège’s definition of a communication medium as being the “distribution and the edition” and how he’s been getting some recent inspiration from Pierre Klossowski on “Liquid Currency” as well as from MIT’s György Kepes and inventor of holography Dennis Gabor on how technology could be used in balance with nature and the environment.

ARTE has been doing an amazing job of helping to support over 60 different VR projects that have been pushing forward the language of storytelling within VR, and I look forward to seeing more US-based companies follow their lead in helping to fund a lot of pioneering work. Oculus has been funding quite a lot of content in the U.S., but there’s not been nearly as much experimentation of funding cutting-edge narrative content from companies like Netflix, Amazon, HBO, Disney, or Hulu. Without additional distribution channels and experiments with producing location-based entertainment content, then many artists and immersive storytellers have to find alternative sources of funding or do international co-productions. Hopefully there will be more companies who look to see what ARTE has been doing as the European immersive storytelling community has been getting a lot more support and funding to produce narrative experiments.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

look-inside

nadja-van-der-weide
Look Inside is an immersive experience the premiered at IDFA DocLab 2019 where you get to take an audio tour of a complete stranger’s home. This was one of the hottest tickets in all of IDFA DocLab selling out quickly, and being one of the hardest pieces to get into. It was a completely novel context of an experience, one that tested my boundaries, and ended up provided a lot of really deep and fundamental experiential design insights in looking at the dialectic between the self versus the other and public versus private contexts.

Creator Nadja van der Weide did her a master’s degree on the theme of “reinventing daily life” at the Sandberg Institute, which was a part of the temporary theme that lasts for only two years. She’s been exploring how to use theatrical experiences for us to consider how we can make novel connections to other people in her Common Good performances, and so Look Inside in her second installation that orchestrates unique encounters with “the other.”

After getting a ticket to this experience, you get an email with an address and instructions for how to open a lockbox that contains a phone with an audio tour as well as a key to open the front door. You enter into the home, and then are guided through the home and invited to test your boundaries about what you feel comfortable doing. There’s a number of specific situations you’re invited to play with, and each one is difficult to predict how you’re going to react until you’re in that actual situation. Van der Weide likes how these contexts are so specific and unique, and how we end up using a lot of intuition in order to explore our boundaries character in these situations.

Overall, this was my most memorable experiences at the IDFA DocLab, and it also proved to provide some of the deepest experiential design insights. The context of entering into a complete strangers home while they’re not there is something that is extremely intimate. There’s all sorts of curatorial decisions that the owner had to make about what would be public and what would be private, and the experience ends up being a bit of a mystery in trying to piece together fragments of this owner’s life, their story, and different aspects of their character or personality based upon what you might be able to glean from the environment.

So in some sense, it’s an environmental narrative. But in another sense, it’s more of an open-ended experience and generative narrative that’s creates opportunities for the participants to test their boundaries and sense of ethical thresholds in deciding what is and is not okay to do within this situation. It was a completely unique and novel context for me to be in, and so there were surprising things that I did that I wouldn’t have been able to predict.

I had a chance to unpack my experience with van der Weide, and for her to explain her experiential design process including the logistics of trying to find people who were willing to open up their home for this type of art project (feel free to contact her if you’d like to open up your home for this anywhere around the world). It turns out that it was a pretty involved development process with a lot of people who end up dropping out once they realize the full implications of what it’d mean to have complete strangers poking around their house. I talk a lot about my own experiences in taking the tour, and what I discovered about traits of my own character. Finally, we talk about this type of performance are as an intervention into daily life designed to find points of connection and common ground of humanity in the midst of an economy and technologies that cultivate patterns of loneliness.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s the trailer for the Look Inside experience

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

through-the-wardrobe

rob-eagle
Through the Wardrobe was an augmented reality experience by Rob Eagle about gender identity that took place in a pop-up shop in the hallways of Amsterdam’s Central Station. The IDFA DocLab had three storefronts in Central Station this year during their festival to show a couple of 360 video programs, an interactive narrative experiences, and Eagle’s experience that appeared to be a clothes store on a quick glance. But each of all of the clothes in the store were associated with four different people with gender fluid identities. You pick a piece a clothing to try on, and then you go into the back of the store for a spatialized AR experience on the HoloLens 1 that dives into the gender identity journeys for each person through five different chapters that were anchored to furniture in this mock bedroom.

I had a chance to talk to Eagle about his experiential design process, and why he decided to move away from virtual reality and focus more on augmented reality experiences so that he could focus on his actual body and experiences that allow you to play with gender identity and gender expression. It definitely provided me with a safe context for me to experiment with my own gender expression, and Eagle talks about the power of mixed reality experiences to create a magic circle as well as a “hybrid space” that’s talked about by theorists like Edward W. Soja and Peter Sloterdijk in being able to blur the digital and the real in order to create a “third space” that goes beyond the affordances of each modality. So there are many parallels for how mixed reality is providing us embodied metaphors for what it means to go beyond the binaries of the virtual and the real.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

last-whispers

lena-herzogLast Whispers is a virtual reality experience that visualizes the extinction of languages around the world. Director Lena Herzog says that there are about 7,000 active languages on the planet, but that we’re loosing one to two languages a week due to climate change displacement as well as different factors of economic and cultural colonialism. She says that we’re on pace to live in a world that only has 30 different languages, and so she wanted to try to tackle the problem of “How do you tell the story of silence?”

I had a chance to talk to Herzog at the IDFA DocLab about the process of designing and developing Last Whispers, her collaboration with Emblematic Group, why she felt like this piece needed to have the full immersion of VR, and the need to move beyond binary thinking in being able to preserve cultural heritage through pride of language while also being worldly and engaged with the rest of the world. We also cover some of the more philosophical aspects with Herzog saying that language is our first creative act, that it’s extremely democratic, and how it is able to embed many aspects of culture and understanding.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s an introduction to her VR film that she gave leading up to Sundance this year:

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

artificial-room-one2

alexander-devriendtAlexander Devriendt is an immersive theater creator with the company Ontroerend Goed, and he was presenting an interactive piece about artificial intelligence called Artificial: Room One. It was an early-iteration prototype that is starting to explore the evolution and future of the CAPTCHA tests that we take in order to prove our humanity. When originally tasked with creating a project around AI, Devriendt was surprised to find that AI was simultaneously a lot more limited than he had expected, but also at the same time made some huge technological leaps to do things that he didn’t think was possible. It was this gap that he wanted to explore in his piece, but also using the experience as a mirror for us to reflect upon our own humanity, and how would we go about the process of trying to prove that to other people using the affordances of our technologically-mediated modes of communication.

I was a part of the first batch of users to go through the experience on the night that it opened, and had an opportunity to reflect upon the experiential design process with Devriendt, and then dig a bit deeper into some of the deeper philosophical reflections about artificial intelligence and what it means to be human.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

nerd_funk

ali-eslami

Ali Eslami
is an Iranian VR creator whose livelong project is an ever-expanding virtual city called False Mirror, and he was showing off the latest edition at IDFA DocLab called Nerd_Funk. Nerd_Funk is a surrealistic, guided tour through an spatialized interpretation of his social media landscape. Eslami collaborated with video installation artist Mamali Shafahi to archive the Instagram stories from over 600 of their avant-garde artist friends who are using these ephemeral stories to document their artistic experiments including the modulation of their sense of identity and embodiment through augmented reality filters.

Eslami takes a very VR architectural-inspired approach to archiving a cross section of artists who are experimenting with new modes of identity augmentation by placing these videos within a spatialized context that mirrors the different categories and topics that they assigned to the videos. He’s also playing with virtual embodiment in interesting ways as you are holding a virtual phone within this experience that sends push notification haptic buzzes that connects you to your body throughout the experience, and also provides a whole other layer of context and meaning by sending you videos and art that’s related to the world around you. It manages to evokes a very familiar Pavlovian-response of receiving these types of texts and push notifications, but it’s all completely recontextualized as you float through these spatialized art installations that explore different modes of embodiment in VR.

I had a chance to talk to Eslami at the IDFA DocLab about his journey into VR, some of his deeper design intentions with Nerd_Funk, why he believes that VR will potentially render science fiction to be dead, and his worldbuilding process that he hopes to continue throughout his artistic career in order to create virtual archives of his life as an immersive artist, but also to help create a larger spatialized context to reflect upon the ephemeral nature of avant-garde experimentation with self-destructing Instagram stories. You can follow the Nerd_Funk Instagram account to keep tabs on this project.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

rozsypne2

lisa-weedaOn July 17th, 2014, Malaysia Airlines Flight 17 was a scheduled passenger flight from Amsterdam to Kuala Lumpur that was shot down while flying over eastern Ukraine killing all 298 on board. This has been huge news in the Netherlands, and when someone asked writer Lisa Weeda with family from Ukrane if she felt guilty, then she got really angry and inspired to write a story about the crash from the Ukranian point of view. Part of the plane crashed in a sunflower field in Rozsypne, and so she collaborated with co-director Nienke Huitenga-Broeren in transforming a series of scenes and broader context of the ongoing conflict in the region into an immersive virtual reality story.

nienke-huitenga-broerenI had a chance to talk with Weeda and Huitenga-Broeren about their experiential design process of creating Rozsypne, the larger cultural context of Ukrane as a border country caught in the middle of many international power dynamics, their iterative process of translating specific contexts into a series of different scenes, and how they wanted to use VR to put you into the shoes of the Ukranian villagers from Rozsypne who discovered the downed airplane and the loss of life.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

only-expansion

duncan-speakmanDuncan Speakman’s Only Expansion was an augmented audio tour showing at the IDFA DocLab that was doing real-time modulation of the surrounding soundscape mixed with produced sounds in order to create a unique experience. The headphones had microphones attached in order to do the real-time modulation, and it also used a custom-built audio processing device to do the real-time processing that is impossible on existing phones and AR head-mounted displays. There was also a companion book for the people going through this experience that provided prompts for what they should pay attention to. The overall experience covered themes of climate change, and used a number of different real-time audio modulation techniques in order to amplify the building and releasing of narrative tension.

Only Expansion received a Special Jury Award for Creative Technology at the DocLab, and I had a chance to talk with Speakman about his music composer background, his journey into producing audio tours, Torsten Hagerstrand on time geography, Kristine Jørgensen on extradiagetic sound, Christina Kubisch’s Electrical Walks, time and our perception of time, and the different ways that he modulates sound in order create and release narrative tension.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

in-the-event-of-a-moon-disaster

In the Event of a Moon Disaster uses AI deep fake and speech synthesis technology to produce a Nixon speech that never happened. Bill Safire wrote a contingency speech on July 18, 1969 for President Richard Nixon to read in the event that something went wrong with the Apollo 11 mission and astronauts Neil Armstrong and Buzz Aldrin were to be stranded on the moon until they would ultimately die.

fran-panettaImmersive audio artists Francesca Panetta and Halsey Burgund were captivated with the speech, and wanted to use the latest in AI technologies to bring it to life. They wanted to raise awareness of how these new technologies fit within a long tradition and spectrum of misinformation and disinformation tools, and so they collaborated with Canny AI on the visuals and the Respeecher on the speech-to-speech synthesis of Nixon’s voice after training thousands of clips with a voice actor.

halsey-burgundWatching the experience at IDFA DocLab was a surreal experience as they recreated a 1960s living room with an authentic television from that era to be able to watch footage of a manufactured moon crash and then a completely fabricated and synthetically-created Nixon speech that never actually happened.

I had a chance to talk with the artists Panetta & Burgund about their design process, their deeper intention behind the work, the philosophical implications of being able to modulate truth and reality, and the ethical implications of deep fake and synthetic speech synthesis AI technologies.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s a brief excerpt of the In the Event of a Moon Disaster Nixon speech.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

the-under-presents

Tender Claws’ The Under Presents is a landmark experience in the evolution of immersive storytelling. It packs in so many narrative innovations, novel game mechanics, experimental VR interactions, and even includes live immersive theater actors in a four-month run starting on November 19th.

Tender Claws is probably most well-known for their critically-acclaimed Virtual Virtual Reality, which seamless blended open world exploration with a very well-written and humorous narrative. They’ve been working on The Under Presents for over two years in collaboration with immersive theater troupe of Piehole. They’ve cultivated an innovative fusion that’s part VR game, part narrative, part music platform, and part experimental playground for live immersive theater researching the question of what exactly the magical “live” aspects of intimate one-on-one interactions in a social VR space.

The Under Presents experiments with time loops both a gameplay mechanic, but also as a narrative conceit in order to explore the story of eight characters who set off on a ship. The experience has a masterful on-boarding process that introduces you to this world and how it stretches across space and time, the unique locomotion system that warps space and time, as well as some of the methods for how to navigate the narrative portions of the experience. It’s probably best to just play through the experience before reading too much about it, and so feel free to pause and just check it out if you already know you’re interested in discovering everything for yourself. I played through the first third of the experience during a press review period and so there’s a lot about the narrative and story structure that I’m still actively exploring now that the finalized build is available.

What’s really quite unique about The Under Presents is that for the next four months, there will be live immersive theater actors who will be roaming around the multiplayer areas of the experience. Neal Stephenson’s Diamond Age sci-fi novel dreamed of reactive actors or “ractors” who would be paid to roam around virtual worlds acting out interactive stories, and so The Under Presents may be the first consumer VR experience that’s employing a set of actors for an initial four-month run of performances that run around the clock. Tender Claws producer Tanya Leal Soto said that they have the ability to monitor the number of users who are roaming around the multiplayer areas in order to help with the capacity planning for populating the world with these live actors.

The actors will be primarily colliding with the users of The Under Presents in what are known as “one-on-ones” within the immersive theater world. These one-on-one interactions with actors are usually randomly sprinkled through immersive theater shows like Sleep No More, but receiving one is rare and kind of like winning the lottery. There’s usually hundreds of audience members in an immersive theater show like Sleep No More, and so there’s only a tiny handful of intimate interactions that happen, and they’re usually up to the actor’s discretion to chose who will receive a highly-customized experience.

Other immersive theater shows like Then She Feel architected their show in order to optimize for these intimate one-on-one interactions. However, these shows only have a throughput of 15 people at a time for this two-hour experience. Because the The Under Presents doesn’t have any space constraints, then they’re able to spin off virtual instances where they’ll have immersive theater actors roaming around who are based initially in New York City and Los Angeles. Time will tell how they deal with scaling this out and how probable it will be that you will run into a live actor, but they’re trying to recreate these intimate interactions in a virtual space.

I was able to have four different one-on-one interactions with live theater actors in my two different press review periods at Sundance and this past month, and there’s definitely a unique quality of having a live interaction. As a user in The Under Presents, you can’t speak and so you’ll be forced to use body language and gestures in order to communicate. Director Samantha Gorman hopes that this will create different elements of emergent play with users, and she commented how it’s really quite amazing how much of someone’s temperamental energy and character can still be transmitted into the virtual space. There are a number of recorded interactions in the experience, and so the live interactions with actors stand out in how there is an emergent conversation that can unfold where there’s an asymmetry of information loss much like playing a game of charades.

Because the live actor can be very specific in how they react to muted users, then it creates a very special live moment where you get to be taken to a secret place, given more context about the world, and perhaps even taught a few tricks of ritual magic. It was immediately obvious to me when I was interacting with a live human and not artificial intelligence. We’re still quite a ways away from artificial general intelligence that could pass this type of Turing test of a live interaction, and so Gorman will be researching these interactions as part of her Ph.D. thesis that she’s in the process of actively researching writing with this project. There’s a lot more insights about emergent conversations and emergent play that she’s sure to find, patterns of user temperament, as well as the component parts for what exactly makes a live theatrical moment in VR so magical.

tanya-leal-sotoI had a chance to talk with Tender Claws producer Tanya Leal Soto as well as the co-director of Tender Claws Samantha Gorman, who also conceived of, directed, wrote, and directed the live immersive theater actors. We talk about the inspiration for The Under Presents, their collaborative process, and how she architected the interactive story.

samantha-gormanGorman said that she had surreal sci-fi / horror writer Brian Evenson develop the initial treatment, but she found that she needed to add an additional spatial treatment that determined how stories unfolded in parallel, but also how the story of the space unfolded. They developed their own motion capture solution that allowed them to do on-the-fly motion capture that was extremely efficient in being able to contain somewhere between 12-16 hours of motion capture data for entire experience. There are eight characters that each have their own storylines, and Gorman said that there are different stories and endings that are unlocked as you see a certain percentage of each of the character’s stories. Gorman also said that they developed their own set of customized tools in order to visualize and architect the parallel story lines that were unfolding across the ship as the coordination and timing was both very interdependent and multi-faceted.

Piehole was also very critical in helping to fill in the gaps for the characters in this experience. They would often improvise character traits, but also help to flesh out the characters by having different actors embody the avatars as they walked through the virtual sets. Stories are usually produced in a pretty linear fashion, but The Under Presents would take a highly iterative approach where acting would happen in virtual scenes with roughly sketched and temporary virtual props, and then the final art product was produced based upon those recorded motion captured interactions. So they were blending in the traditional waterfall approaches like the story and character treatments with a much more iterative design approach with improvised acting and bottom-up exploration. This would be fed back into the script, and then iterated on through multiple passes.

Soto & Gorman also said that Piehole was instrumental in recruiting many of the musical performances from the New York City artist scene who are featured on the stage, which serves as a centralized hub and multiplayer space. They were able to capture the essence of a live musical performance while being able to leverage the affordances of virtual reality to have a much more theatrical & surreal production than would be normally possible given the budgets of a indie musician. I found myself captivated in watching a number of performances while other users were roaming around performing ritual magic transmutations on interactable objects.

The Under Presents really does sit at the cross section of an indie game, experimental narrative storytelling, VR experience, and art piece that incorporates many influences from the theater world. Gorman has a background in theater and seemed happy to get back to her theater roots to explore how VR is able to create many site-specific theatrical interactions. Ordinarily there would be a lot of down time when following around characters who aren’t doing much in this type of immersive theater piece, and they actually have the characters blink in and out of existence in order to minimize dead time and make editing the narrative a lot easier as well. There’s a Matrix-glitch mechanic which actually helps you hone in on the narrative moments where something interesting is happening either through a monologue or dialogue with the actors.

Gorman also said that she was taking inspiration from other indie games like Braid for navigating space and time, the more passive narrative VR experience Invisible Hours, emergent play mechanics from Journey, the French theorist Guy Debord’s concept of dérive where “participants drop their everyday relations and ‘let themselves be drawn by the attractions of the terrain and the encounters they find there.’” They also took narrative inspiration from the sci-fi novel The Invention of Morel, which inspired the story that involves time loops. Gorman said that she is using virtual spaces to explore narrative structures that go way beyond simple branching narratives, and that virtual reality allows her to play with time in a narrative that goes beyond what any other medium can provide. There are many levels in which they’re exploring the bounds of time and space.

There are many puzzles and mysteries contained within the underlying structure of this experience that are a joy to discover. There’s also so many innovations when it comes to VR locomotion, motion capture, live immersive theater actors, an entire behind-the-scenes control schema for live actors to seamless navigate and perform actions in virtual spaces, scrubbing through a narrative collaborative and emergent play, and pushing the boundaries of what’s possible with narrative structures once you’re able to navigate through space and time. If you’re interested in tracking the evolution of storytelling and interactions within virtual reality, then The Under Presents is a must-watch experience that I suspect will continue to grow and evolve over time as a community forms and the many hidden secrets continue to be discovered. The Under Presents is is available for the Oculus Quest for $19.99, and was released on Tuesday November 19th, 2019.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality