AR-neural-input
I participated in a Facebook press event on Tuesday, March 16th that featuring some Facebook Human-Computer Interaction Research on AR Neural Inputs, Haptics, Contextually-Aware-AI, & Intelligent Clicks. It was an on-the-record event for print quotes, however I was not given permission to use any direct audio quotes and so I try to paraphrase, summarize, and analyze the announcements through a lens of XR technology, ethics, and privacy.

I’m generally a big fan of these types of neural inputs, because as CTRL-labs neuroscientist Dan Wetmore told me in 2019, these EMG sensors are able to target individual motor neurons that can be used to control virtual embodiment. They even showed videos of people training themselves being able to control individual neurons without actually moving anything in their body. There’s a lot of really exciting neural input and haptic innovations on the horizon that will be laying down the foundation for a pretty significant human-computer interaction paradigm shift from 2D to 3D.

The biggest thing that gives me pause is these neural inputs are currently being paired with Facebook’s vision of “contextually-aware AI” that is presumably an always-on, AI assistant who is constantly capturing & modeling your current context. This is so their “Intelligent Click” process can extrapolate your intentions through inferences and aims to give you the right interface, within the right context, at the right time.

I don’t think Facebook hasn’t really thought through how to opt-in or opt-out of specific contexts or how third-party bystanders who revoke their consent and opt-out or if there’s even any opt-in process. When I asked about how Facebook plans to to handle consent for bystanders to either opt-in or opt-out, then they pointed me to an external RFP to get feedback from the outside community for how to handle this. I hear a lot of rhetoric from Facebook about how the fact they are in charge of the platform is allowing them to “bake in privacy, security, and safety” from the beginning, which sort of implies that they’d be taking a privacy-first architectural approach. But yet at the same time, when asked how they plan on handling bystander consent or opt-out option for their always-on & omnipresent contextually-aware AI assistant, then they’re outsourcing these privacy architectures to the responsibility of third parties via their RFP process, which has already closed for submissions in October 2020.

They also have been mentioning their four Responsible Innovation principles announced at Facebook Connect 2020 of #1 Never surprise people, #2 Provide controls that matter, #3 Consider everyone, #4 Put people first. My interpretation is that these are stack ranked because there’s language elsewhere that indicates that the “#3 Consider Everyone” specifically refers to non-users and bystanders of their technology (as well as underrepresented minorities). Part of why I say this is because there are other passages that seem to indicate that the people in “#4 Put people first” is actually referring to Facebook’s community of hardware and software users, “#4 Put people first: We strive to do what’s right for our community, individuals, and our business. When faced with tradeoffs, we prioritize what’s best for our community.”

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s some research prototype videos that Facebook has released:

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

loveseat-venice-2019
Kiira Benzing‘s Loveseat was an ambitious fusion of live theater performance with VR technologies that premiered at the Venice Film Festival in 2019. They performed live for the audience in Venice while wearing VR headets as their performances were simultaneously broadcast within the virtual reality environment of High Fidelity (back when they still had VR components before their pivot to spatial audio).

There have been a lot of other fusions of live theater with VR technologies recently, including Benzing’s follow-up live theater in VR piece a year later called Finding Pandora X, which won the Best VR Immersive User Experience at Venice 2020. She took a lot of the lessons learned from Loveseat and applied them to Finding Pandora X, especially the fact that when you do a live performance simultaneously in VR and IRL, then you end up doubling the production needs and staff that need to attend to both performance contexts.

I had a chance to unpack some of the other lessons learned from Benzing at Venice 2019 after one of their performances, including some of the specific acting insights from all three of the actors involved in the production: Jenn Harris, Jonathan David Martin, and Sam Kebede.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

SXSW-VR-in-VRChat
Blake-Kammerdiener
SXSW Online is happening March 16-20, and I had a chance to get a sneak peak of the SXSW Online XR world in VRChat and talk with the chief curator of the SXSW Virtual Cinema program Blake Kammerdiener about the program and special events that he’s been able to put together. The $399 entry price for SXSW is a bit steep if you’re only interested in the immersive storytelling program, but this also includes all of the tech conference, music conference, film conference, film festival, and music festival events in additional to the SXSW Online XR program. I’ll be attending a number of the different aspects of the SXSW Online next week, and talking to Kammerdiener helped me get a bit more of an idea of what to expect next week and where to track all of the different events, talks, meetups, and live performances.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

WebXRAwards-Polys

The Polys WebXR Awards was an awards show on February 20th, 2021 founded by Ben Irwin in collaboration with Sophia Moshasha and Julie Smithson. Irwin wanted to feature a lot of the work that’s been happening on the immersive web in the year 2020 since the official WebXR spec was finally shipped on the Chrome Browser on December 10, 2019. The Polys WebXR Awards was a live show streamed on Twitch and with “Meta Multiverse” watch parties within Mozilla Hubs, AltSpaceVR, Engage, and Tivoli Cloud. They had an hour-long pre-awards show featuring pre-recorded, red carpet interviews, and they awarded 11 awards across a number of different categories.

I brought together Irwin, Moshasha, and Smithson three days after the show in order to unpack their journey in producing the event, as well as some of their highlights and takeaways in celebrating the experiences and developers who are helping to make the immersive web possible. There is not a video archive available for the show as they wanted to keep it ephemeral and in the moment, but you can see all of the nomminees on their WebXR Awards website or links to all of the nominees and winner can also be found in my Twitter thread coverage of the WebXR awards. They are also planning on posting more clips over the next year on their WebXR Awards YouTube Channel considering they captured a lot of historically interesting interviews and conversations.

Here’s a full list of the Polys WebXR Awards winners:

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

Here’s my Twitter thread from the event listing all of the winners and nominees:

UPDATE (March 11) Ricardo clarifies on Twitter that he as indeed won a few awards prior to his WebXR Lifetime Achievement Award.

https://twitter.com/mrdoob/status/1369958753823522820

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Traveling the Interstitium with Octavia Butler - Still 2
One of my favorite pieces at Sundance New Frontier 2021 this year was a series of four open web experiences created as a part of the piece Traveling the Interstitium with Octavia Butler. Born out of The Guild of Future Architects‘ Futurist Writer Room, lead artists Sophia Nahli Allison, idris brewster, Stephanie Dinkins, Ari Melenciano, and Terence Nance participated in a series of worldbuilding workshops featured on the themes and imagination of Octavia Butler’s body of science fiction work. Their original output were going to be live performances, but with COVID-19, they decided to use open web technologies to distribute their speculative design art pieces. You can see these four immersive web pieces on the website Interstitium.space/.

kamal-sinclair2I had a chance about how this project came about with Kamal Sinclair, Founding Executive Director of Guild of the Future Architects, as well as with Ari Melenciano, a creative technologist & founder of AfroTecTopia. We trace the lineage of these worldbuilding processes that take inspiration from Alex McDowell’s World Building Institute, Afrofuturist designers, Allied Media Projects, AfroTecTopia, NYU’s Interactive Telecommunications Program (ITP), Skawennati Fragnito’s Initiative for Indigenous Futures, Afrocentric Design, Processes Centered in Blackness, Janet Wong & Bill T. Jones of New York Live Arts, Future Imagination Summit 2019, as well as Octavia Butler’s body of work.

Ari-MelencianoSinclair and Melenciano talk about how this type of speculative worldbuilding allows Black artists to go beyond deficit-based narratives focused on trauma, and the space to step deeply into “the audacity of bold imaginations of our future” where reconciliation is possible and new potentials are released. They are cultivating a practice of creative & collaborative foresite that’s able to “liberate minds of calcified understandings” and ultimately democratize of the imagination of our future through these creative, worldbuilding processes. Sinclair has become convinced of the power of radical imagination facilitated through these worldbuilding processes, because she has witness multiple times how these imaginal Afrofuturism visions expressed through art have come to pass when given enough resources and community members with the capacity and willingness make it happen.

Each of the four pieces within the Traveling the Interstitium with Octavia Butler have their own speculative designs and take on the future, and you can experience them yourself on the Interstitium.space webite.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Beyond the Breakdown - Still 1
Worldbuilding and speculative design was a big theme at Sundance New Frontier 2021, and I had a chance to participate in an experience that facilitated a collaborative & deliberative process of worldbuilding that was called Beyond the Breakdown. Created by Tony Patrick, Lauren Lee McCarthy, and Grace Lee, it builds off some of the foundational work and processes developed by Alex McDowell and the USC Worldbuilding Institute. The core idea of worldbuilding is to design the underlying context and structures of society projected out within the context of a future time and place, and then to apply an evolutionary cultural, technological, economic, political model in order to imagine some potential futures from a variety of different perspectives and points of view. In the end, there will hopefully be some common themes and consensus that emerges.

In order to facilitate this process the Beyond the Breakdown collaborators created a simplified teleconference application that replicates the feeling of a group Zoom call. There were six participants who are on this call along with an AI-assistant named Serenity that’s puppeteered by a human off screen. The goal is to project out into the future into 2050, and then have a group discussion that’s catalyzed by a series of prompts provided by the AI assistant. The goal is to find the underlying principles and values that are consistent today and in the future, to imagine a better potential future, and then create a collaborative community dialogue to see where there are common interests and goals so that people can individuals within a community can start to think about what types of actions that can take today in order to make these imaginal futures a reality today.

I’m really excited about the power and potential of democratizing these types of community worldbuilding practices, an especially the potential of using immersive storytelling to actually build out some prototypes of these speculative futures within a virtual environmnt in order to start to prototype the large-scale designs, architecture, and emergent social dynamics of some of these imaginal futures. We’ll be taking a look at some specific examples of this in our next episode on Traveling the Interstitium with Octavia Butler.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Nightsss - Still 1
Nightss is an sensual experience that’s structured around an erotic Polish-language poem by Weronika Lewandowska that uses dance and spatial metaphors in VR to create an immersive poem. She collaborated with co-director and co-screenwriter Sandra Frydrysiak who also has a background in dance. They both are very interested in researching how the immersive experience they created impacts the neuroscience of embodiment, perception, and empathy in collaboration with the University of Social Sciences & Humanities in Warsaw.

Lewandowska and Frydrysiak are also interested in creating immersive experiences that help the audience feel embraced, immersed, safe, intimate, and sensual, and they’re working with the Visual Narratives Lab do help do some research into directing attention and other foundational research topics for immersive storytelling. They coded Lewandowska’s poem, and used it to structure multiple layers of story that included the emotions, visuals, movement, music, interaction, and overall immersion. Poetry uses a lot of powerful visual metaphor, and so it makes sense that the translation of poetry into immersive poems will help to form the underlying affordances of the spatial language of virtual reality and immersive storytelling.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Here’s a performance of Weronika Lewandowska’s poem featured in Nightsss

Music: Fatality

depthkit-scatter
Depthkit’s volumetric capture solution is democratizing access to the tools of volumetric filmmaking. I had a chance to do a deep dive with two of the co-founders of Scatter, James George (CEO) & Alexander Porter, who have been pioneers in the volumetric capture space. We talk about the talk about the history, evolution, and future of volumetric filmmaking. But also how tools like Depthkit play an active part of shaping the storytelling affordances of the medium. We cover a number of different sensemaking frameworks to makes sense of emerging technology evolution and diffusion, but also the funding that catalyzes innovation and aspects of culture that are on the bleeding edge of digesting, disseminating, and pushing the boundaries of innovation around creative expression and immersive storytelling.

In the second half of this in-depth interview, we do a technical deep dive to look at some of the pragmatic challenges and open problems when it comes to integrating volumetric capture within an independent immersive storytelling project. We talk about Scatter’s intention of democratizing access to the tools of volumetric filmmaking and the decision to move from a scrappy open source project to a fully-fledged start-up company with funding and customers.

The team behind Scatter and Depthkit are some of the most experienced immersive industry veterans since the modern resurgence of consumer VR. Be sure to check out my previous interviews with George, Porter, & Yasmin Elyiat in Zero Days VR as well as The Changing Same, which debuted at Sundance 2021. I also have a number of unpublished interviews with the Depthkit team that I hope to publish at some point in order to fill the gaps within the history and evolution of volumetric filmmaking.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

the-changing-same2
The Changing Same is one of the most ambitious pieces of volumetric filmmaking yet since it combines all sorts of volumetric capture techniques with Scatter’s new Depthkit Studio multi-depth sensor set up, photogrammetry, and a fusion of rigged avatars and volumetric capture. It also has scenes with multiple captures, and new innovative spatial storytelling techniques like fusing multiple realities together, magical realism, a rich set of symbolic and visual metaphors, and using the spatial affordances of context to draw out similar archetypal patterns of oppression, systemic racism, and white supremacy that repeat throughout the course of history.

The Changing Same is the fourth production from Scatter (Clouds, Zero Days VR, & Blackout), and they continue to use these immersive storytelling pieces to innovate on their Depthkit volumetric capture solutions as well as content pipelines and workflows. Depthkit provides a democratizing impulse into volumetric capture by using a single or multiple commercially-available depth sensor cameras paired with a DSLR camera.

I had a chance to get more of the backstory The Changing Same and the exploration of the history racial terror in America by talking to creators Joe Brewster, Michèle Stephenson, and Yasmin Elayat. We talked about the backstory and evolution of this project, the struggles of funding independent and experimental works of volumetric filmmaking, and how they were exploring questions like What does an an American Pilgrimage look like? What is Magical Realism within context of The Changing Same? And what is the definition of Afrofuturism from an American perspective?

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

True Justice: Bryan Stevenson’s Fight For Equality

Music: Fatality

XR-Ethics-Invitation

I’ve been an executive committee member of the IEEE Global Initiative of the Ethics of Extended Reality for the past six months, and we had our official kickoff meeting on Friday, February 5th. The goal of this effort is to produce a white paper that helps augmented reality and virtual reality designers and developers to understand the landscape of ethical & moral dilemmas when designing for XR, and to provide a framework with some guidance of applied XR ethics & ethically-aligned design. There will also be some technology and privacy policy and recommendations for the government policymakers and technology policy lawyers.

Timeline-for-White-Paper-IEEE-Global-Initiative-of-Extended-Reality-Ethics

I’ve volunteered to chair and head up the white paper committee, and so I presented an proposed timeline and an initial survey of the landscape of ethical and moral dilemmas of XR based upon my previous work within my XR Ethics Manifesto. Here’s a video of my presentation from that initial meeting on February 5th, 2021:

Here’s my initial survey of the landscape of ethical issues. This is merely a starting point, and how we end up breaking this up into chapters is very much open to deliberation by this community-driven effort.
Landscape-of-XR-ETHICS

There’s going to be lots of stuff that’s not included, and there will also be generalized principles independent and universal across all contexts, but also other lenses and perspectives to be included. I’m going to attempt to be as inclusive and pluralistic as I can in how we structure this white paper on XR Ethics.

To get more involved, then you can sign up here to stay informed over the whole time period, or if you want to dive in and help collaborate on one of the subcommittees, then send an email to xr-ethics-chair@ieee.org with the “Subject: Add Me to iMeet Platform.”

The full slide deck can also be found here.

We’re getting our collaboration platform set up, and I look forward to working on this project with the community. If you have any questions, then feel free to ping me @kentbye on Twitter.

LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality