Starship_Commander_Arcade_KeyArt
Alexander-Mejia
Human Interact’s Alexander Mejia has a hard time describing what exactly Starship Commander: Arcade is. For movie lovers, he says it’s a choose-your-adventure narrative where your the main character of your own movie. For gamers, he calls it a cinematic narrative with some story choices driven by language and voice. In the end, the vast adventure game was scoped down to a 12 to 15-minute sci-fi, arcade experience that uses your voice to interact with a repository of nearly two hours of pre-recorded video clips of Sergeant Sarah Pearson (played by Human Interact VP of Business development Sophie Wright) answering your questions. It’s a very unique combination that explores how you can do worldbuilding for an experience by just using your voice, and it sets the foundations for what it could be like to become a character within a cinematic, immersive experience.

Starship Commander: Arcade came out on September 10, 2020 after 4 years of working on it. From original idea to 2016, to prototype demo shown publicly at GDC 2017, to re-scoping it as a location-based entertainment, arcade experience, and then to it’s eventual release on Steam. Included in the purchae of this experience includes a very well-produced documentary called Passion at All Costs where Mejia documents his whole journey as an indie VR developer. It really captures the early days of VR, and will prove to be a valuable historical document of this time period that tells a larger story through the lens of Human Interact’s struggles and successes over the years.

There’s quite a bit of innovation and experimentation that’s contained in this experience, and I hope that folks will check it out. Despite the fact that AI voice services have been available for quite a while, then there haven’t been a lot of commercial VR releases that really explore and utilize the potential of using natural language processing technology to deepen your sense of social presence and immersion within a world.

So if you’re interested in the future of interactive narrative, then I’d encourage folks to listen to the first half of our podcast if you haven’t already decided to try out Starship Commander: Arcade. Go have the actual experience, check out the Passion at All Costs documentary, and then come back and listen to the last half of this podcast to get the full story. We are moving into a world where we become protagonists in our own interactive & immersive experiences, and there’s a lot of key insights that Mejia and Wright have figured out here.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

stanford-study-motion-tracked-data
mark-roman-miller
Stanford University has just published an important research paper that hows how motion tracked data in VR can be identifiable of specific users. The paper titled Personal identifiability of user tracking data during observation of 360-degree VR video was published in Scientific Reports on October 15th with authors including Mark Roman Miller, Fernanda Herrera, Hanseul Jun, James A. Landay & Jeremy N. Bailenson.

I had a chance to catch up with Miller on October 12th to summarize their major findings that included a 95% accuracy in being able to identify one of 511 different participants from a 20-second sample size from a 10-minute session of watching a 360-video, and then rating their emotional reaction using the HTC Vive hand-tracked controllers. Even though they’re watching a 360 video, they have access to a 90Hz feed of 6DoF information from the head pose in addition to two 6Dof-tracked hands. From this basic motion-tracked data, they’re able to extrapolate a unique signature of someone’s body size, height, and nuances of how they hold and use the controllers, which ends up being enough information to reliably identify someone given the right machine learning algorithm.

I talk with Miller about the experimental process and analysis, as well as some of the implications of this study. Currently this type of motion tracked data is typically considered to be de-identified data, but research like this may start to reclassify motion tracked data as personally-identifiable and potentially even classified as biometric data. We also talk about how specific medical information can also be inferred from the recording on this motion-tracked data. There’s more ways to make this type of research more robust across multiple contexts over time, but it’s generally pointing to the possibility that there are some immutable characteristics that can be extrapolated and inferred from this data.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

vrx-brennan-spiegel
On October 6, 2020, Dr. Brennan Spiegel released his book titled “VRx: How Virtual Therapeutics Will Revolutionize Medicine,” which is an amazing survey of different applications of what could be called virtual medicine, experiential medicine, digiceuticals, or cyberdelics. The FDA is calling it Medical XR (MXR), and they’re recognizing it as an entirely new field of medicine that needs special regulatory considerations.

Not only is Spiegel’s book a comprehensive survey on what’s happening with virtual therapeutics, but also explores this fascinating intersection between neuroscience, psychology, clinical medicine, technology, the Mind/Body connection, embodied cognition, & different branches of philosophy but specifically the philosophy of mind and the mysterious nature of consciousness itself. Spiegel is able to deftly explore all of these different dimensions through George Engle’s bio-psycholsocial model of medicine created in 1977 or from how the World Health Organization in 1947 recognized that there’s a physical, emotional, and social dimension to health. VR is helping Western Medicine transcend the biomedical lens of health and healing, and start to leverage the body’s innate healing capacities that can be unlocked though digitally-mediated experiences.

Spiegal has also been deploying VR to over 3000 patients at Cedars-Sinai Hospital in Los Angeles, California as a part of patient care and research projects. He was a co-author of a paper titled “Recommendations for Methodology of Virtual Reality Clinical Trials in Health Care by an International Working Group: Iterative Study” that aims to develop a methodological framework to “guide the design, implementation, analysis, interpretation, and communication of trials that develop and test VR treatments.”

I had a chance to talk with Spiegel about this whole new field of medicine, and what it can teach us about the underlying mechanisms of perception, health & healing, and the mysterious nature of consciousness itself.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

xr-association
The XR Association is a non-profit 501(c)(6) trade association with members including Facebook/Oculus, Google, Microsoft, Sony, & HTC. They’re focused on general awareness and education, lobbying Congress on XR policy issues, and promoting research and best practices. On September 20th, 2020, they released results from an industry survey covering “A New Reality in Immersive Technology” within Healthcare, Education, Job Training, Retail, Manufacturing, and Public Safety.

elizabeth-hymanI had a chance to talk with XRA CEO Elizabeth Hyman on September 28th to talk about the results of their survey, a bit of history and context of the XR Association, the Reality Caucus and Future of Work Caucus, their recent Supercharging the Virtual Workforce Webinar, previous relationships between trade associations and consumer privacy advocates, as well some of their overall strategies for educating and lobbying the U.S. Congress on policy issues that are relevant to the XR industry.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

privacy-history-1973

joe-jerome
Virtual and Augmented Reality pose unique privacy challenges, and so it’s worth going back to look at the evolution of privacy laws in the United States to see how we got to this point today. There’s also a lot of current discussions for the need for a U.S. Federal Privacy law, which started in early 2018 with the aftermath of Cambridge Analytica, the launching of the European Union’s General Data Protection Regulation (GDPR), and California’s Consumer Privacy Act (CCPA). There seems to be pretty widespread, bipartisan consensus on the need for some type of U.S. Federal privacy law, but the issue is both sufficiently complicated with enough sticking points that there isn’t a clear path for what exactly that would look like.

After watching the U.S. Senate Committee on Commerce, Science, and Transportation Hearing on “Revisiting the Need for Federal Data Privacy Legislation” on September 23, 2020, I had a chance to talk with Joseph Jerome of Common Sense Media. He’s a privacy and cybersecurity lawyer who has been embedded within the Federal privacy law debates for the past seven years, and he’s also taken an interest in the privacy implications of VR and AR writing about AR mapping implications and privacy controls for VR.

Jerome and I were able to cover an incredible amount of ground within our 90-minute conversation, and I added over 150 show note citations based upon our discussion as he provided a really amazing historical overview of the evolution of privacy policy within the United States. It’s a great primer that I know I’ll be referring to often as I get up to speed in looking at the evolution of privacy law in the United States and beyond.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Show notes: References of the History of US Privacy Law

Continue the conversation:

Here’s a thread of some of my previous coverage of XR and Ethics with links to over 40+ Past Voices of VR podcast episodes

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

darkfield-radio-double
darkfield-radio
Darkfield Radio have released a couple of binaural audio performances that transform your home into a immersive audio stage. Their experiences are intended to be experienced with another person, each of their own mobile phones at a specific time so that it recreates the feeling of attending a performance.

I had a hard time differentiating between reality & the binaural audio recordings within Darkfield Radio’s Double as it was one of the most immersive audio experiences that I’ve had yet.

I had a chance to talk to Darkfield Radio’s creative directors David Rosenberg and Glen Neath during the Venice VR Expanded Festival, and I’d highly recommend checking out one of their performances before tuning in to hear more about their journey into immersive audio, their creative evolution from site-specific audio performances, to exploring the future of spatialized audio and blurring the lines between what’s real and what’s pre-recorded.

Be warned that their work is within the horror genre, but it’s more in the vein of cultivating some underlying tension rather than cheap jump scares. It can provide a great escape and cathartic release as Halloween in coming up, and give you the feeling of attending a performance without even leaving the comforts of your own home. You can check out the trailers for Double and Visitors down below.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s the trailer for Darkfield Radio’s Double

Here’s the trailer for Darkfield Radio’s Visitors

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

gnomes-and-goblins3

wonder-bright
My wife, Wonder Bright, has absolutely fallen in love with the VR experience Gnomes & Goblins. Her mindset and approach to the game was almost totally opposite from my own. While I focused trying to solve the scavenger hunt puzzles within their core explore, craft, and collect game mechanic, she immersed herself into skipping rocks, role playing with virtual characters, and becoming fully immersed reliving her childhood fantasy adventures in the woods of Eastern Washington. Needless to say, she was able to discover a lot more about the true nature of the game than I was in my initial 90-minute session, and it was her six-hour exalted exploration that inspired me to take another look at so many of the things that I had missed in my initial playthrough.

I mentioned in Wonder’s experience in my interview with the creators of Gnomes & Goblins and how she had a completely different mindset. This actually matched with Wevr’s playtesting where they saw a lot of newer VR users able to discover a lot of their experience if they had a more chill, meditative, and Animal Crossing mindset rather than a goal-oriented, checkbox checking, Call of Duty mindset.

Gnomes & Goblins is the first VR experience that my wife has really gotten excited about. It’s also her first video game experience that’s she really ever played. So it’s the first time she’s so fully inhabited, explored, and fell in love with any interactive game / immersive experience. She’s already logged over fifteen hours exploring, playing, and discovering all of the delightful creatures and stories embedded into this master class of worldbuilding.

We’ve been sharing a lot of conversations about Gnomes & Goblins over the past five days unpacking it all. Her feelings range the full spectrum of unabashed love and praise for the joy that this experience evokes in her, but she also has some specific feminist critiques in that she’s not able to fully project herself into a world that doesn’t have any female-identified bodies, and she’s not a huge fan of having to steal artifacts from the homes of Goblins without their explicit consent.

She found that the core game mechanic of collecting “artifacts” actually got in the way of the types of reciprocal relationships and experiences that she was yearning to cultivate. “Stealing” these objects and bringing them back to her tree house felt like a colonial narrative trope that ended up making this beautiful and sacred world a lot more like the type commodified world that she’s trying to escape. She just wanted to have more opportunities to be in deeper relationship and interact with these little virtual goblins rather than hunt for objects. Her desire for deeper connections with virtual beings point to the future of AI-driven character interactions, which are certainly on the technological road map for future sequels or future iterations of where these types of immersive experiences are headed.

As I was preparing to record the intro and outro for my interview with the Gnomes & Goblins creators from Wevr, I realized that there was a lot of Wonder’s perspectives and experiences that I wanted to include, but that I didn’t feel that I could properly digest and capture in my own words. So we decided to sit down on Wednesday morning to record a recap of the conversations that we’ve been having about this experience covering the full spectrum of why Gnomes & Goblins has so quickly become her first and most favorite VR experience of all time, and what changes she like to see to be able to more fully inhabit, cultivate reciprocal relationships, and immerse herself as a participant within the culture of the Goblins.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

gnomes-and-goblins

The full release of Gnomes & Goblins was released on September 23, 2020, and it has some of the most sophisticated worldbuilding that I’ve seen in VR so far. Hollywood director Jon Favreau started to collaborate with Wevr’s Jake Rowell and Neville Spiteri after seeing their The Blue experience over five years ago.

Neville-SpiteriGnomes & Goblins has a 45 to 60-minute linear narrative, onboarding and prologue experience, and then a 5 to 18-hour open world exploration phase called “Goblin Life” where you can explore, craft, and collect objects within this world. The narrative portion is a great introductory experience for new and first time users, but it’s also something that the most seasoned VR enthusiasts should enjoy.

I talked with Wevr’s Rowell & Spiteri on the day of their release to discuss their design intentions as well as challenges for how to best serve the brand-new VR users (much as they did with The Blue), as well as create a satisfying VR exploration game for the hardcore VR gamers. The open-world portion of the experience doesn’t give a lot of hints or clues for what to do or how to do it, and so it actually requires either a different mindset or some patience for either figuring stuff out on your own or reaching out to community resources for clues. Some of Favreau’s most favorite gaming experiences is when he’s had to go out of game and search for answers, and so the open world portion definitely embodies this philosophy of needing to talk with other people to fully figure out some things. If you need some tips on how to explore, craft, and collect, then be sure to check out their post called “Clues for your adventure” with more some tips that I certainly found helpful.

Animal Crossing was also a huge inspiration to Favreau, and so team members like former Call of Duty developer Rowell had to figure out how to approach that type of game with the right mindset beyond what he calls his “Call of Duty” mindset that’s very goal-oriented to check all of the explicit boxes. Rowell said that they found that people without any expectations of what a VR game should be could actually get farther in the game and have a better time as they often approach it with a more chill and meditative mindset, which they wanted to emphasize as we’re all in the midst of this pandemic.

I did find that my wife was able to discover things within the experience that I had missed on my first play through, and she inspired me to go back in with a different mindset to discover even more things. After talking with Rowell and Spiteri, I went in for another 10 hours exploring around the world through their explore, craft, and collect core gameplay mechanic. I personally didn’t find the collection mechanic on it’s own to be a compelling enough reason to sustain extended play times, but my motivation instead came from wanting to more fully explore this beautiful world that they’ve created, and to start to unpack more of the narrative clues that are embedded throughout this world. I’ll actually have a follow-up conversation with my wife to talk about her experience with Gnomes & Goblins as she’s completely fallen in love with the piece, and we’ll talk more about both what she loves and what more she wants from a fantasy VR adventure experience like this.

There is a lot that is new about this experience, and it’s definitely trying to push the VR medium towards a mode of “being” rather than a mode of “doing.” It’s certainly got some things exactly right, and there are some other aspects of how to more fully onboard & guide users, or to generally help to cultivate the types of experiences that a range of people would find interesting and compelling. It’s an ambitious world with a lot of deep mythology and other stories ready to be told, and so I hope Gnomes & Goblins can find it’s audience so that they can continue to keep iterating and pushing the medium forward as they’ve done here in this experience.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s the trailer for Gnomes & Goblins

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

agence

The creators of Agence call it a “dynamic film,” but it’s part immersive story with game-like interactions and puzzles where you engage with either traditional heuristic game AI bots or bots trained through reinforcement learning processes. There’s some automatic cinematic cutting that leverages the language of film for you to project all of your human dramas and stories onto these five bots who are interacting with each other. But after an average of 8-10 minutes of interaction in this simulation (or as long as 30-45 minutes depending on your actions), then the simulation comes to one of many different endings, rolls the credits, and immediately starts you up again.

Pietro-GaglianoThe National Film Board helped to produce this piece by Pietro Gagliano and his Transitional Forms studio based in Toronto, and it’s got quite of a lot of interesting experiments looking at sharing authorship between the creators, the audience, and these AI entities that have different “brains” ranging from heuristic game AI to reinforcement learning.

david-oppenheimI had a chance to talk with Gagliano and the NFB’s David Oppenheim on September 8th after I saw the world premiere at the Venice VR Expanded, and we unpacked the design process as well talked about all of the implications of training AI within the context of these story worlds. They’re also releasing a set of tools for AI researchers to be able to train their own brains, and so Agence will serve as a publishing platform for experiments in AI architectures within the context of this story world where audiences can interact with them within either virtual reality (Steam, Viveport, & Oculus), 2D PC game (Steam), as well as on mobile phones (iOS & Android).

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

in-protest
alton-glass
In the aftermath of George Floyd’s murder by three police officers on May 25th, 2020, there were Black Lives Matter protests that erupted around the world. Immersive filmmakers Alton Glass and Adam Davis-McGee felt the historic nature of this global movement for racial justice, and they wanted to help document this movement with 360° video in order capture a digital archive of the ephemeral memorials that are being created, but also footage from the frontlines of protests from around the United States. The result is a multi-part series called In Protest, with the first episode that launched on September 16th focusing on Minneapolis & St. Paul, Minnesota, which was ground zero of Floyd’s murder.

adam-davis-mcgeeIt’s a profound and powerful inaugural episode that uses the power of VR to transport you to the frontlines of the protest, to some of the support networks that have been emerging, but also some of the Say Their Names memorial spaces that have been created to honor many of the black people who have lost their lives. Glass and Davis-McGee also wanted to focus on other ways that Black people are protesting that goes beyond our typical idea of what it looks like to protest, whether that’s by making food, writing poetry, or doing community organizing behind the scenes. They collaborated with Professor Allissa V. Richardson, who wrote a book about Bearing Witness While Black, and she helped to set a broader context for what it means to bear witness to these racial injustices without adding more trauma to the already overwhelming experience of systemic racism.

I highly recommend checking out the first episode of In Protest, which can be tricky to track down. You can either look for it within the Oculus TV app under Editorial Picks, or you can search for “In Protest” on the main page and scroll down to “VR Media,” or you can try logging into Oculus website and click the “Save to VR” button and then look for the “Recently Saved” area at the bottom of Oculus Home on the Oculus Quest. It’s harder to find and access than it should be, but it’s definitely worth hunting down and watching within Oculus TV. I’m looking forward for future episodes as they continue to tell stories from Minneapolis and St. Paul, but also Washington D.C., and Atlanta.

These 360° videos are helping to document these movements, and they’re going to be really valuable historical documents that will allow people to drop into a more visceral and emotional experience of the Black Lives Matter movement. They found that all of the people that they shared their footage with before filming an interview would have a similar emotional reaction to being transported to these different memorials and pieces of art. Glass says that he’s hoping to open doors to help to make the XR industry more diverse and inclusive, and that we’re still at such an early phase of figuring out the cinematic language and grammar of VR storytelling. He expects that future generations will be able to pick up where we leave it, and that the stories they’re capturing can help to capture a spirit of pride identity, worth, authenticity, and respect for people involved in this movement.

There’s a lot of uncomfortable aspects to this story, and Glass hopes that it’s through sitting with that discomfort through an immersive experience that could help create space for equal-footing empathy, and help to hack our reality a bit to catalyze some paradigm shifts. Davis-McGee says that he’s hoping that this will help people be able to sit with some of their uncomfortable emotions, and that the medium of VR helps to access a more unfiltered experience of a wide range of rage, pain, joy, camaraderie, and solidarity. I know that’s certainly been true for me after the four times watching their initial episode, as each time brought me closer to emotionally connecting to the collective grief and trauma of the many injustices of black lives that have been lost and that are being honored within the many Say Their Names memorials. In Protest is a powerful and profound series that starts to show the power of how virtual reality storytelling can be used to tell stories that go beyond what other mediums can achieve.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality