The White House VR documentary People’s House by Felix & Paul Studios won a Emmy for the outstanding original interactive, and I had a chance to talk with Paul Raphael about how the challenges of producing a high-profile piece. They didn’t know how many rooms they’d be able to shoot, and President Obama was such a fan of the project that he literally opened doors for the crew to record more than twice the number of originally scheduled rooms. They were limited to only two 15 minutes interviews with Barack and Michelle Obama, and so they collaborated with speech writers to capture memories and stories for this virtual guided tour.
Felix & Paul Studios create their own VR camera hardware, and they’re starting to use their fourth generation cameras while designing a next-generation, digital lightfield camera. Raphael said lightfield VR shoots are essentially visual effects shoots, which require shooting in different wedge segments that need to be composited in post-production. He also said that they’ve been consulting with most of the major HMD manufacturers including Facebook on an open standard for immersive 3D audio. Even though they’ve been creating a lot of hardware, they’re more interested in using it to stay on the bleeding edge so that they can continue to innovate and push the creative limits of what’s possible in immersive storytelling.
Do patients with anorexia nervosa suffer from body image distortion due to how they perceive their body or is it due to attitudinal beliefs? Betty Mohler has been using VR technologies to study whether body representation is more perceptual or conceptual. She captures a 3D body scan of patients, and then uses algorithms to alter the body mass index of a virtual self-avatar from a range of plus and minus 20%. Patients then estimated their existing and desired body using a virtual mirror screen which tracked movements in real-time and showed realistic weight manipulations of photo-realistic virtual avatars. Mohler’s results challenge the existing assumption that patients with anorexia nervosa have visual distortions of their body, and that it’s possible that body image distortion is more driven by attitudinal factors where patients consider underweight bodies as more desirable and attractive.
Mohler works at the Space & Body Perception Group at the Max Planck Institute for Biological Cybernetics. She’s collaborates with philosopher of neuroscience Dr. Hong Yu Wong to research foundational questions about self perception like “Who am I?” Where am I? Where is the origin of my self? Where is the frame of reference? What is the essence of me? How do we know that there’s an external world? What does it mean to have a shared self where multiple people share the same body experience? What does it mean to have a body? How big is my body? Is it possible to be at multiple locations at once while in VR?
I interviewed Mohler for the third at the IEEE VR conference in Los Angeles this past March exploring all of these provocative questions (see my previous interviews on the uncanny valley and avatar stylization).
Marshmallow Laser Feast is a collective of artists who are interested in using VR technologies to capture the aesthetic beauty of nature, and provide immersive experiences that inspire people to cultivate an even deeper with the reality that surrounds them. Their Treehugger provides an immersive experience of the lifecycle of water in trees as rain makes it’s way up from the roots of a Sequoia tree and is released as oxygen in a highly-stylized & beautiful point-cloud aesthetic. Their experience included smells and passive haptic feedback to make their simulated volumetric time-lapse even more immersive, and it actually won the Storyscapes award at the Tribeca Film Festival. I caught up with co-founder Barnaby Steel to talk about how VR could be used to inspire us to cultivate an even deeper relationship with the world around us.
The National Theatre has created an Immersive Storytelling Studio to better understand the practices, protocols, opportunities of how virtual and augmented reality technologies are creating new storytelling possibilities. They collaborated with the National Film Board of Canada on an immersive theater piece called Draw Me Close that premiered at Tribeca Film Festival. It featured a one-on-one interaction with a live actor in a mixed reality environment while the audience was unveiled within a virtual reality headset where you play the archetypal role of a son/daughter as your mother embraces you, draws with you, and tucks you into bed as she narratives a memoir of her life. I talked with Immersion Storytelling Studio producer Johanna Nicolls about the reactions, intention, and overall development of Draw Me Close, which is their first immersive theater VR piece.
LISTEN TO THE VOICES OF VR
The spatial storytelling techniques and skills that theater has been developing for hundreds of years translates really well to the even more immersive 360-degree, VR environments. But with Sleep No More and Then She Fell, there’s also a whole other “immersive theater” movement within the theater world that is bringing new levels of embodiment, choices, and agency into authored theater performances. No Proscenium podcast host Noah Nelson wrote up a great introductory primer of immersive theater that explores the nuanced differences between immersive theater, site-specific performances, and environmentally-staged theater. One differentiation that Nelson makes is that immersive theater has much more of an explicit experiential design that “feels more like an event you experienced than a performance that you witnessed.”
The version of Draw Me Close that I saw at Tribeca took a powerful first step in exploring how live actors sharing the same physical space within a mixed context provides a new dimension of emotional and embodied presence. The haptic feedback of an embodied hug from a co-present human is something that may never be able to ever be fully simulated in VR, and so this illustrates a clear threshold to me of what can and can not currently be done in VR. I also saw the Then She Fell immersive theater piece which featured a lot of one-on-one interactions with performers, and so I think that there’s a profound depth of emotional presence and intimacy that you can achieve with another person without the barriers of technology. You still can’t see the more subtle microexpressions of emotion or perceive the more nuanced body language cues when interacting with other humans while you’re in VR, but feeling the actor touch me provided a deeper phenomenological sense of embodied essence that I was interacting with an actual human in real-time. Directly interacting with another physically co-located person and feeling their touch closed some perceptual gaps and took my sense of social presence beyond the normal levels I have in distributed social VR experiences.
This was also such a new type of experience that I didn’t know the rules of engagement for how much I was expected to speak or interact. There weren’t a lot of prompts for talking or engaging, and so I mostly silently received the story as each moment’s actions were being actively being discussed, analyzed, and contextualized by a steady stream of real-time narration. There were not a lot of prompts, invitations, or space made available for dialogue, but there were a number of interactive actions I was invited to do ranging from opening a window to drawing TiltBrush style on the floor. There was a deliberate decision to be fairly vague in casting a magic circle of the rules and boundaries of what to expect, since the story, characters, and loving embrace of a motherly hug was all designed to be a surprise. This shows the challenging issues of balancing how to receive explicit consent to being touched while also maintaining the integrity of the mystery of a story that’s about to unfold.
Draw Me Close is an ambitious experiment to push the storytelling possibilities that are made available within a one-on-one interaction of an immersive theater piece while the audience is within virtual reality. It was a profound enough experience for a number of people who needed to have some level of decompression and help transitioning back from exploring some of the deeper issues that were brought within the experience. There are obviously limitations for how this type of experience could be scaled up so that it was logistically feasible to be shown on a wider scale, but it’s refreshing to see the NFB and National Theatre’s Immersive Storytelling Studio experiment, explore, and push the limit for what’s even possible. If too much effort is focused on what’s sustainable or financially viable, then it could hold back deeper discoveries about the unique affordances of combining immersive theater with immersive technologies.
Here’s some footage from Draw Me Close at Tribeca by Steve Rosenbaum
One of the most emotionally-moving VR experiences that I’ve had in VR was bearing witness to Holocaust survivor Pinchas Gutter share his experiences at Majdanek Concentration Camp in The Last Goodbye. Gutter not provides a guided tour, but he is able to achieve a level emotional catharsis through the process of sharing his story that his virtual presence within the experience amplified my own sense of emotional and social presence, which is what helped to make it such a profoundly moving VR experience. The Last Goodbye uses a unique blend of photogrammetry and billboarded stereo video that helped to transport me into a room-scale experience of multiple key locations at the Majdanek Camp as Gutter shared his memories of being there as an 11-year old child.
The Last Goodbye premiered at the Tribeca Film Festival, and it was an epic collaboration catalyzed by co-creators Gabo Arora and Ari Palitz and included HERE BE DRAGONS, MPC VR, and OTOY on the technical production side as well as the USC Shoah Foundation to oversee the process of capturing testimony about the Holocaust. Arora is the founder and creative director of LightShed, and is now an advisor and former founder of the UN’s VR initiatives where was able to gain access to the Syrian refugees featured in his famous empathy piece Clouds Over Sidra. In this interview, Arora shares his collaborative process, pushing the boundaries of volumetric storytelling by blending photogrammetry-based, room-scale VR with live-action, empathy-based storytelling, as well as how he had to guide Gutter to achieve the depth of emotional presence that makes the piece so powerful.
The design team behind Bioflight VR has worked on television shows such as CSI and ER, and they’ve been able to translate their VFX visualization skills into a virtual reality medical education venture. Their original plans were to use virtual reality to help doctors utilize the volumetric information captured in MRIs, CAT scans, and ultrasounds to improve upon medical diagnosis from 2D slices of data, but they started to gain more traction with creating a couple of different types of educational experiences. They started creating time-lapse experiences showing the long-term impacts of sodium consumption and smoking in videos meant for doctors to show patients to inspire behavior modification, and they also created a number of interactive medical training scenarios that would allow medical students to experience intense emergency room scenarios that would allow them to be evaluated based upon their competency and performance.
I had a chance to catch up with co-founder and chief creative officer Rik Shorten at the Silicon Valley Virtual Reality conference both in May 2016 as well as a follow-up and update in March of 2017. This interview tracks the evolution of Bioflight VR starting with ambitions to use VR for medical diagnosis, and then their pivot focusing more on medical training and patient behavioral modification and education the following year. There are a lot of opportunities for virtual reality to become a huge part of telemedicine and providing a platform to visualize data that you collect about your body, but virtual reality seems to be making it’s first strides into the medical field through patient and student education before the more advanced and higher-end applications of medical diagnosis and distributed telemedicine are adopted.
Dr. Skip Rizzo heads the Medical VR Research Group at USC’s Institute for Creative Technologies, which has been exploring how to use VR for psychological treatments, cognitive assessment, motor rehabilitation therapy, as well as interactions with virtual humans. He’s been on the forefront of using virtual reality to treat soldiers with post-traumatic stress disorder with virtual exposure therapy. VR is used to incrementally trigger an embodied sensory experience in PTSD patients by placing soldiers within the virtual sights, sounds, feelings, and smells of common combat scenarios in Iraq and Afghanistan. This virtual exposure therapy stimulates the original embodied experience of traumatic events for soldiers so that they can connect to specific details of their memories so that they can do a process of cognitive restructuring by telling the story of their experience. Rizzo says that a key component of healing from PTSD is if the patient is able to connect to the underlying emotions of the experience while sharing the narrative of their experience, and that this can unlock a cascade of healing effects that USC has been able to measure over the years.
LISTEN TO THE VOICES OF VR PODCAST
I had a chance to try out a demo of the VR PTSD Exposure Therapy project during a reception at USC ICT during the IEEE VR 2017 conference in Los Angeles in March, and was struck by their holistic multi-modal approach of using subwoofers, smells, and passive haptic guns. I caught up with Dr. Rizzo to talk about his work in using VR to heal from PTSD, the importance of storytelling and emotional presence, and their future work in expanding treatment scenarios for victims of Military Sexual Trauma, and moving into civilian trauma with first-responders like police and firemen. He also talked about an episodic, interactive storytelling experience that will be like an emotional obstacle course of navigating different traumatic scenarios with the help of a virtual human that is helps guide the patient through the cultivation of coping skills for stress management, mindfulness techniques, and cognitive reappraisal. This work at USC ICT shows that immersive virtual environments can stimulate a deep sense of embodied and emotional presence that has vast healing potential that goes well beyond just the gaming and entertainment applications.
Here’s a 20-minute VICE Canada Story on Virtual Reality PTSD Exposure Therapy
Dean Radin is the lead scientist of the Institute of Noetic Sciences where he’s been working on cutting-edge consciousness research of showing measurable mind and matter interactions with quantum phenomena. He’s published 17 papers showing statistical significance of meditators being able to subjectively project their consciousness and make an “observation” in a double-slit experiment that collapses the quantum wave function. You can read more in his Physics Essays paper “Psychophysical interactions with a double-slit interference pattern,” as well as this lecture on mind-matter interactions at a consciousness conference. Radin’s team is starting to look at whether or not they can use virtual to amplify the effects of this remote observation triggering signs of a quantum wave function collapse. Rather than imagining an observation in your mind, will they see similar effects by replicating this experiment within VR where they can be more embodied.
The quantum measurement problem is an open problem in quantum mechanics, and the Von Neumann–Wigner interpretation that consciousness causes collapse of the wave function is merely one of many possible theories, but it’s one that Radin is explicitly researching. He’s also researching whether or not consciousness can entangle photons beyond the theoretical strength of Tsirelson’s bound. Radin’s team is still too early in this research to report any results either way, but it’s the type of consciousness work that they’re pioneering.
I had a chance to do a deep dive with Radin at the latest Institute of Noetic Sciences conference about what his research into quantum mechanics and consciousness means for coming to a better understanding about the ultimate nature of reality. Radin says that there’s some scientists and physicists are coming to a more Pythagorean belief that perhaps the fundamental base reality is some sort of mathematical or symbolic realm. This is a theory that Max Tegmark explores in great detail in his book Our Mathematical Universe, and a high-level summary can be found in his technical preprint called The Mathematical Universe. Radin says the idea of a fundamental mathematical base reality is one that comes from the Pythagoreans, and that it forms the foundations of the more esoteric Neoplatonic and Hermetic traditions of magic. It’s led him to write a book coming out in 2018 titled Magic is Real where he came to the conclusion that he sees enough scientific evidence that would support the claims made from the lore of these magical traditions.
LISTEN TO THE VOICES OF VR PODCAST
Rain says that perhaps virtual reality will serve as a reality turing test of sorts to be able to help up discern which level of reality that we’re currently living. In his book, Supernormal he examines the claims made of Siddhi superpowers that have been detailed in a 2,000-year-old manuscript known as the Yoga Sutras. He says that if you’re able to achieve an enlightened state of samadhi, then that could unlock all sorts of latent human potentials including many of the psychic phenomena that he’s explored in his previous books The Conscious Universe and Entangled Minds. So in essence, if you reach a state of enlightenment, then you’re able to start to literally cut through the matrix of reality and start to achieve things that are only bounded by your belief and imagination.
Radin is on the frontier of science, and so a lot of his research is not yet widely accepted within the wider mainstream thinking yet.
Even though Radin has personally experienced being able to bend his spoon with his will, he’s a data-driven empirical scientist who has more faith within the wider scientific method. Some of these more extreme claims of supernormal human potential made within the Yoga Sutras presuppose that people are able to more readily reach these mythic states of enlightened consciousness, which is not an easy feat. But overall, he’s committed to doing the type of basic research into quantum mechanics that could help solve the mysteries of consciousness, and potentially lead to larger paradigm shifts towards these philosophical ideas of panpsychism.
I didn’t have the capacity to go through his entire work within the context of this podcast interview, but I’d recommend that you check out this lecture about his mind-matter double slit interactions talk at a consciousness conference, David Chalmer’s TED talk on panpsychism, as well as a more extended documentary about the philosophical metaphysics of panpsychism.
The hard problem of the mind/body split and the ultimate nature consciousness is an open question in the science community, and there are a range of philosophies that try to handle this split. Cartesian dualists explicitly acknowledge this split as different realms, but interactions between the mind and body have started to break this down. There’s a lot of scientific materialists who are holding out that consciousness will eventually be discovered to be an emergent property of the our neuroscience. Idealism is the opposite of materialism in saying the subjective experience is primary, and it’s similar to saying that consciousness is fundamental in that matter could be an emergent property of a base reality of awareness or information. Panpsychism sees consciousness as universal in that every photo is conscious or carries a certain level of information processing capability. University of Sydney professor Kai Riemer says that phenomenology tries to get rid of the whole idea of this subject/object split, and that it’s a much more holistic approach of centering everything around the interconnections of the meaning of objects and our direct human experience.
Phenomenologist Gabriella Farina has resisted a precise definition by saying, “A unique and final definition of phenomenology is dangerous and perhaps even paradoxical as it lacks a thematic focus. In fact, it is not a doctrine, nor a philosophical school, but rather a style of thought, a method, an open and ever-renewed experience having different results.” I had a chance to catch up with phenomenologist Kai Riemer at SIGGRAPH where he gave his perspective on what phenomenology is and why it’s holistic approach could provide some vital insights for people working in VR.
LISTEN TO THE VOICES OF VR PODCAST
Specifically, Riemer talks a lot about French philosopher Maurice Merleau-Ponty’s Phenomenology of Perception book from 1945 that talks about the role of the body in perception. He also cites George Lakoff’s Women, Fire and Dangerous Things of show how a lot of our primary metaphors for understanding the world come from our direct experience of the world through our bodies. The holistic approach of phenomenology shows that the stories and narratives of direct experience should be given equal weight to the objectified data that is seen as primary by reductionists and physicalist materialists. Archeologists need to be able to understand the full story behind what people thought artifacts meant within the full context of a culture before they can fully understand what they’ve discovered.
What’s clear from talking to Riemer and other philosophers is that VR provides a embodied experience of philosophical discussions that are otherwise pretty abstract and disconnected from our direct experience. Phenomenology is an elusive concept to firmly pin down, but my conversation with Riemer has helped me appreciate it’s holistic approach to the connection between reality and experience.
Killing Floor: Incursion is a horror game featuring lots of gore that is sure to evoke visceral reactions in players. I had a chance to play a demo at GDC, and I can say that it’s definitely an experience has stuck with me. The mechanics of beating enemies with their dismembered limbs had an extreme amount of blood splattering that it was a mix of being at the same time grossly disturbing and ridiculously comic. I had a chance to talk with project lead Leland Scali about the horror genre in VR, pushing the boundaries of how far to take gore within immersive VR, and their deeper game design process of creating an experience within their Killing Floor universe.
Scali admits that they’re treading a fine line of it being funny or amusing versus taking it too far, and so it’ll be interesting to see how VR gamers and the larger media react to this experience as it could be a catalyst to larger discussions about the impact embodied experiences of gory violence within virtual reality. He says that this is not an experience about rainbows and happiness, but rather one that’s gory & dark with a dash of quirky humor. Ultimately it seems to be about power — specifically giving the player the power to complete the task, and assaulting them heavily to see if they can handle it.