Dr. Mary Whitton has been working with interactive computer graphics since 1978 and virtual reality since 1994 in collaboration with Dr. Fred Brooks at the University of North Carolina at Chapel Hill. I had a chance to talk with Mary back in 2015 at the IEEE VR in France about her fundamental research into VR locomotion, haptics, and cultivating a sense of presence.
There have been a lot of military grants over the years to research the impact of haptics, spatialized audio, latency, and field of view into cultivating presence within VR. She’s also investigated a number of different issues around walking-in-place locomotion techniques, passive haptics & redirected touch, as well the nuances of Mel Slater’s presence theory with the place illusion and plausibility illusion.
LISTEN TO THE VOICES OF VR PODCAST
In 2015, the IEEE Visualization and Graphics Technical Committee (VGTC) awarded a technical achievement award to Oculus’ Brendan Iribe, Michael Antonov, and Palmer Luckey. At the end of this interview, Mary Whitton made a call out to the role that Mark Bolas and USC ICT may have played within the creation of the Oculus Rift.
Robin Arnott had a peak experience from a LSD trip at Burning Man, and wanted to see if he could replicate this type of peak psychedelic experience within virtual reality. The result of four years of work is a transcendent experience called SoundSelf that creates a diffuse feedback loop between your toning and a variety of generative abstract art visuals. So I had a chance to catch up with Robin to talk about using VR to cultivate a synthetic peak experience, using VR as a transformative technology, and the role of contemplative spiritual practices.
LISTEN TO THE VOICES OF VR PODCAST
SoundSelf is the closest experience I’ve had where I felt like I had taken a digital drug, and it’s the type of experience that is really difficult to describe. Pendleton Ward was raving about his peak experience with SoundSelf VR back in episode #410, and I expect that this could be one of many unique lived “erlebnis” experiences that you just have to try to understand.
There’s a beta release available at SoundSelf.com, but the full release of the experience should be launching sometime in 2017.
SUPERHOT VR is one of the standout VR games to come out with the Oculus Touch launch with it’s unique blend of the mechanics of a first-person shooter but with the strategy of a puzzle game. Physics-based interactions in VR are already compelling since it helps to cultivate plausibility within our brains through the expectation loop of prediction and observation. Tying your movements to the progression of time within the game provided me with a awe-inspiring experience of the fabric of space-time that’s completely unique to VR and feels like it’s re-wired my brain.
LISTEN TO THE VOICES OF VR PODCAST
I had a chance to catch up with the game designer Piotr Iwanicki at Oculus Connect 3 where we talked about how they had to ditch the original teleportation mechanic and really focus on cultivating a sense of body presence using the Oculus Touch controllers. He also talks about the joys of a moving in a slow motion ballet environment while being the midst of a non-stop, intense action-movie sequence. Even though the bullet-dodging mechanic is primarily based upon your head positions and not your lower body movements, the low fidelity graphics offer a blank slate for you to project your full sense of body presence into the experience. As Piotr says, “most of the action is happening in your head.” So much so that one developers kicked over a monitor after getting so immersed and forgetting that their feet were not even being tracked.
There’s a distinct lack of abstracted gameplay within SUPERHOT VR that’s based upon your physical body movements, and so it’s able to cultivate a deep sense of embodied presence. The perceived danger of red crystal enemies running at you with guns pointed at your head also contributes to the “bat test” insight that presence can be increased when there’s an artificial threat presented. You’re also able to do a series of slow-motion actions that just make you feel like a complete badass. Overall, SUPERHOT VR is one of the more innovative gameplay mechanics that I’ve seen in VR so far. It’s compelling enough to give new VR users a taste of what type of gaming experiences are uniquely possible within VR, but also has a lot of deep lessons for VR designers for how cultivate and maintain a deep sense of presence.
VR escape room games are one of the emerging genres that are particularly well-suited for the affordances of VR, and Schell games’ I Expect You to Die sets the standard for creating this type of puzzle game. I had a chance to catch up with game design director Shawn Patton at GDC in March where he shared some of the design strategies for I Expect You to Die.
LISTEN TO THE VOICES OF VR PODCAST
I Expect You to Die finds the sweet spot of not making it too difficult as to be impossible, but at the same time challenging enough as to not be too easy or boring. So they do expect you to die, but to also learn something new about the puzzle with each iteration. Shawn talks about their primary design considerations which included making it a comfortable VR experience, creating it within a certain budget, making sure that available objects had clear affordances and goals to be achieved, and that solving the puzzle would make the player feel clever. They have also included a number of other achievements and being able to complete a level with style so as to encourage replayability.
There is also a lot of fun narrative elements sprinkled throughout the experience, and the game has one of the more epic introductory sequences that calls back to the early days of cinema where the credits showed before the movie begins. There’s also a really great tutorial at the beginning to teach you the button combinations that you’ll need to know in order to manipulate objects within an experience. They’ve designed the experience such that you can use some of the six degree of freedom affordances of the touch controllers, but most of the gameplay is done through analog stick controllers and buttons that are also available on a gamepad or even a mouse and keyboard.
I Expect You to Die was originally released as a short demo on Oculus Share where it was the top-ranked experience for many months, and it also earned up a number of different VR industry awards. I previously interviewed Jesse Schell after winning 3 Proto Awards in 2015 as well as after winning the Vision Inspire Award at the Unity vision summit. Jesse has a lot of experience in working with virtual reality prior to the consumer renaissance of VR, and has applied a lot of insights for how to cultivate and not break presence within an experience.
I Expect You to Die is also one of the five games that’s currently included within the Oculus Touch bundle selling for $89.99.
The Khronos Group announced on Tuesday that they have a critical mass of major VR players who are collaborating on a VR open standard. This VR open standard will have a software and hardware component that will enable VR application portability across VR platforms, but also minimize the cost for hardware integrations across different VR platforms. The Khronos Group has been able to get public support for this initiative from VR headset manufacturers including Valve, Oculus, Google, & OSVR as well as the major GPU and CPU players of AMD, NVIDIA, Intel, and ARM.
LISTEN TO THE VOICES OF VR PODCAST
Back in March 2015, Neil told me that the VR industry needed a period of innovation before trying to pin down an open standard. But now that the major VR headsets have now launched, there’s not enough significant differences between the major VR SDKs to warrant third-party hardware peripherals to have to create custom integrations. The benefits for standardization outweigh the costs of having a fractured ecosystem, because this VR open standard will enable smaller VR companies to write a single driver that allows them to interface with all of the major VR headsets.
I had a chance to catch up with Khronos Group President Neil Trevett to talk about this call for participation on this VR open standard. There’s not a preliminary specification that’s been developed yet, but they wanted to go ahead and make this announcement in order to let the industry know that there’s enough consensus for this to happen, and to encourage participation from other upstart VR companies.
Neil estimated that this standardization process usually takes around 18 months to finalize, but it may move quicker depending upon how motivated the major companies are to lock it down and remove barriers to innovation. Designing an open standard is more of an art than a science, and Neil said that they would try to limit the scope of the standard, but also allow for extensions. You can read more information about this initiative here with a number of quotes including Oculus, Valve, Google, OSVR, and Epic Games.
The Oculus Touch motion-tracked controllers are launching on December 6th after eight months of the Rift solely supporting gamepad experiences. I wanted to take a moment to dig into some of the more subtle technical nuances when comparing the Oculus Rift & Touch with the HTC Vive, but also some of the larger VR ecosystem considerations to take into account. So today’s episode of the Voices of VR podcast is an op-ed analysis comparing the Touch controller and room-scale tracking technologies, as well as the overall content and developer ecosystems.
LISTEN TO THE VOICES OF VR PODCAST
In terms of industrial design and ergonomics, I do prefer the Rift’s head-mounted display over the Vive’s HMD. There has also been a lot of praise for the ergonomics of the Touch controllers, and I do agree that they a lot more comfortable than the Vive’s lighthouse wands. However, Oculus’ camera-based tracking system was not optimized to support full room-scale VR experiences, and TESTED has confirmed that a diagonally-configured, two-camera Oculus sensor setup is not as robust as the Vive’s lighthouse beacons.
Oculus’ tracking system is really optimized for “standing 360″ VR experiences that are specifically designed for their front-facing cameras. Robert McGregor has argued that Oculus wasn’t prepared for room-scale VR when they launched the Rift CV1, and that they had made a strategic bet that most people would be playing VR games sitting down in front of their computer. This strategic bet has hampered Oculus with their decision to go with shorter cables for both the Rift headset as well as their sensor cameras. These work great for forward-facing standing 360 experiences on your desktop, but the cord management logistics and camera-based tracking volumes are highly suboptimal for achieving a fully robust, room-scale experience.
This thread on the Oculus subreddit discusses more of the technical differences in tracking capabilities between the two systems. I think it’s important to remember in comparing the Rift with the Vive that a lot of the Touch launch content has been specifically optimized for front-facing, standing 360 experiences. I’ve had multiple developers tell me that Oculus has had developers change their experiences to avoid limitations in their tracking solutions such as picking things up off of the floor because the Touch can easily loose tracking when you touch the ground. So to really push the limits of Oculus’ room-scale capabilities, then it should be compared to a variety of Vive room-scale games from Steam that allow for an equal comparison.
This is made possible because Valve decided to take an open development philosophy in having their SteamVR SDK support both the HTC Vive and Oculus Rift hardware. This means that Vive games bought on Steam will have automatic support for the Rift, but that same game bought on Oculus Home won’t support being able to play it on the Vive. The important point is that SteamVR’s support for Rift games enables people to test the same game out on the different platforms.
I unfortunately ran into a number of technical issues with my Oculus Touch set-up that prevented me from testing the limits of Oculus’ camera-based tracking, and so I’m going to have to reserve my final judgment on this matter. But I think it’s important to remember some of these VR design nuances and limitations when comparing the two systems.
The Oculus Touch controllers do have more buttons that are available for gameplay and professional applications than on the initial Vive controllers. This means that the Oculus gaming content has easier access to these buttons for abstracted expressions of your will. A lot of the VR games on the Vive have avoided this level of button abstraction, and I believe that there’s a tradeoff in different levels of presence that these buttons can cultivate. The Vive focuses you more on embodied presence while the Oculus has a bit more capabilities for an abstracted, active presence.
It’ll be interesting to see in the long-run whether this abstraction advantage enables more rich gameplay on the Oculus platform or if players prefer the types of experiences that minimize the abstractions and maximize the number of intuitive movements. Having user interfaces with complicated button manipulation combinations could limit VRs reach beyond gamers, and so it’s a risk that VR developers create experiences with too high of a learning curve. But having access to more buttons could also enable richer gameplay mechanics for some VR games.
The other big point that I wanted to make is that Valve opened up royalty-free lighthouse tracking back in August, and so I expect to see a lot of new lighthouse peripherals launching at CES. There was not a similar announcement from Oculus at Oculus Connect 3, which solidifies my impression that the Vive is actively cultivating an open ecosystem while Oculus is going down a more closed, walled garden route. They’re focusing more on vertically-integrated solutions with a highly-curated selection of exclusive games, which prioritizes the benefits to the end-consumer rather than supporting a diverse developer ecosystem. It’s enabled Oculus to launch with a robust line-up of games, but time will tell as to which ecosystem the VR developers will be investing their time in supporting in the future. For some developers, a minimum viable product is going to be to launch their game via Steam for the Vive as a first-class platform with Rift support as an automated afterthought.
At the moment owning an Oculus Rift headset with Touch arguably gives you access to more content if you include their exclusive content as well as the Vive content that’s automatically supported on Steam. However, there may be potential limitations of Oculus’ tracking technology in viably supporting some of the full room-scale experiences, which may hamper some people’s experiences of that Vive content. It’s also unclear as to whether or not Rift users will be able to fully utilize the new lighthouse peripherals that are expected to launch next year.
From a VR design perspective, I believe that Oculus’ decision to wait eight months to launch Touch controllers as well as not natively supporting room-scale experiences has fractured the VR developer ecosystem into three distinct groups: Sit-down gamepad VR, Front-facing Standing 360, and full room-scale VR. The lack of tracking parity between the Rift and the Vive has a created complicated and fractured ecosystem for VR developers to navigate since there are so many tradeoffs depending on which group is targeted.
While there is a lot of press consensus that the Oculus hardware with the Rift HMD and Touch controllers enjoy an ergonomic advantage at the moment, I personally have deeper concerns about Oculus’ inferior tracking technology, the fractured ecosystem that’s being propagated through not making room-scale a first-class citizen, and a number of private developer frustrations with Oculus’ closed mindset. These are larger issues that make it more difficult for VR developers to easily support both hardware platforms, and ensure a healthy ecosystem of content development.
In the end, mainstream VR has a higher chance of success if VR developers can financially survive, and that the quality bar for content is high enough for consumers justify the time and money invested. Time will tell how the story plays out, but right now the Touch launch is a significant day for Rift consumers who have been waiting to fully step into the game since that promise was made in the original Oculus Kickstarter.
Hassan Karaouni is one of the 11 winners of an Oculus Launch Pad scholarship for his project My: home, which allows people to share 360 videos of locations that are meaningful to them. In my Voices of VR episode about Google Earth VR, I talked about how the principle of embodied cognition explains how our memories are tied to geographic locations. But right now Google Earth’s resolution at the human scale is really uncanny, and you can’t go inside.
That’s where Hassan’s project tries to fill the gaps by enabling people to share 360 videos of places that are meaningful to them, while being able to navigate between them using a model of the Earth. This is quite an intimate and effective way to get to know someone, but it’s also the type of content that’s going to be a lot more meaningful to the creators in 10-20 years from now because it is so effective at evoking memories.
LISTEN TO THE VOICES OF VR PODCAST
Hassan Karaouni is also one of the co-founders of the Rabbit Hole VR student group at Stanford. They’ve held a number of events, and have deep philosophical discussions about how VR can impact human life and the human condition. So Hassan and I go down the rabbit hole in this episode by exploring the deeper philosophical implications of simulation theory and our relationship to fate and free will. In the wrap-up, I talk how a recent sci-fi film helped me gain some more insights into the differences between chronos and kairos time and how a VR experience is a non-linear portal that tips the balance towards creating more kairos time type of experiences.
Cultivating presence is one of the main goals for a lot of VR experiences, but our brains are like a black box of perceptual soup that makes it hard to know all of the right ingredients to achieve this. Kimberly Voll is a cognitive scientist, programmer, and VR developer who is a part of the Fantastic Contraption team, and she has a framework for cultivating presence that she refers to as the “VR Fidelity Contract”.
LISTEN TO THE VOICES OF VR PODCAST
The VR fidelity contract sets up the expectations of a virtual space so that there’s a match between how the affordances are presented to a user and the wealth of knowledge that each person brings based upon their lived experiences. So if there’s a door with a knob on it, then the user would expect to be able to open or close that door. Each time a user performs a successful action in an experience, then it slowly builds trust and the fidelity contract is maintained. If the user takes an action that wasn’t accounted for, then there could be an expectation mismatch that breaks the fidelity contract and ultimately results in a break in presence.
I had a chance to catch up with Kimberly at VRLA where she shared some of her process of cultivating plausibility through user testing, the most common things that break presence, as well as her thoughts on the future of artificial intelligence and storytelling in VR.
Fantastic Contraption recently made a significant Kaiju Update that allows users with limited space to scale the environment down and work in the near field.
I had a chance to talk about storytelling in VR with three of the co-founders of Oculus Story Studio during Oculus Connect 3. Saschka Unseld, Maxwell Planck, and Edward Saatchi were showing off a preview of their third VR experience Dear Angelica as well as their immersive storytelling tool of Quill, which enabled them to create a VR narrative experience entirely within VR.
LISTEN TO THE VOICES OF VR PODCAST
They all emphasized to me that it’s still very early days of figuring out the unique affordances of virtual reality as a storytelling medium, and that Oculus Story Studio is still doing quite a bit of experimentation. They were in agreement in believing that it’s likely going to take a long time to figure out what narrative in VR looks like, and that it could be another generation before VR finds it’s true form.
While I agree that VR storytelling is still very much within a Wild West phase of development, at the same time I do believe that there have been a lot of solid lessons learned about VR as a storytelling medium that I’ve covered on The Voices of VR podcast. At the bottom of this post is a Top 50 List of Voices of VR interviews about storytelling in VR where the list is broken up into the following seven categories: the language of VR storytelling, interactive storytelling, multiple perspectives and empathy in storytelling, social storytelling, world building & environmental storytelling, plausibility & presence in narrative, and audio.
Some of the key discoveries that Oculus Story Studio made with Dear Angelica are first of all that changing scale as an effective way to evoke different emotional reactions. They also discovered that stopping and scrubbing through time was a very compelling experience that allowed audience members to have more control over their pacing through an experience. They also developed a unique “Quillustration” aesthetic that is like a lucid dream that’s trying to mimic how memory works. Perhaps having tools to create VR stories within VR will provide new narrative devices for how stories will be told in VR.
Saschka defined the essential components of a story in VR as simply having a beginning, middle, and end, and this broadens the scope of what could be classified as a narrative within a VR experience. Edward says that it often feels like they have the “dead hand of cinema” hovering over whatever VR storytellers do within a VR experience. The target VR demographic right now is so familiar with the film and video game mediums that they are bringing a whole set of expectations that impacts how they consume and receive VR narrative experiences.
Saschka was also really cautious and skeptical about creating stories that have branching narratives with multiple endings. He interprets multiple storylines as a sign that the author may not know what he/she wants to say, and this blocks his process of cultivating a personal connection with the content creator.
We also had a wide-ranging discussion about narrative vs interactivity, and the balance between creating authored stories versus balancing the amount of control a user has within the context of their sandbox of interactivity. Oculus Story Studio is made up of a lot of filmmaking gamers and so they cited a number of 2D narrative games as inspiration including Stanley Parable, Papers Please, Tacoma, Virginia, Gone Home, LMNO, and Façade. In the end, they imagine that VR experiences will be like the Holodeck in that it’s social, it’s a game, but it’s a movie.
We’re still quite a ways away from having a widespread consensus on where VR storytelling is going, and Oculus Story Studio will continue to try to find that sweet spot between authored narrative and the sandbox of interactivity.
The language and terminology around virtual reality is still evolving, but the VR community has been settling into using the phrase “VR Experience” in order to describe the process of a going through a piece of virtual reality content. I had a friend/business partner over who got her Ph.D. in German literature, and she had a lot of really interesting insights about how the German language actually has two separate words for experience with “Erfahrung” and “Erlebnis.”
On today’s episode of the Voices of VR podcast, I talk with Dr. Jenn Zahrt about the differences between erfahrung and erlebnis, and whether or not a third term might be required in order to describe the process of going through a virtual reality experience. Jenn says that erlebnis could be thought of a unique “lived experience” that you actually go through with your embodied flesh, and that an erfahrung encompasses a much wider range of types of experience ranging from an archive of memories created from your erlebnisse but also knowledge that’s gained from indirect sources like the media, books, and external sources of authority.
LISTEN TO THE VOICES OF VR PODCAST
Merriam Webster has a number of different definitions for experience such as “the fact or state of having been affected by or gained knowledge through direct observation or participation.” This definition closely mirrors the direct sensory experience that could be translated as an “erlebnis.” And “erfahrung” could be thought of as the “practical knowledge, skill, or practice derived from direct observation of or participation in events or in a particular activity.”
Jenn and I discuss the other nuances between erfahrung and erlebnis as well as some of the open questions and insights that these two German words for experience bring up. For example, the process of other people telling a story and curating a VR experience could be thought of an erfahrung, but yet having a direct sensory of experience of it could be thought of as having an “erlebnis” type of experience. Or is it really possible to have a “lived experience” erlebnis experience within VR? Does it require a new term to describe this blend between the two? And is it possible to have an “erlebnis” type of lived experience in VR if you have high enough levels of fidelity of emotional, social, embodied, and active presence?
There are many more open questions here to be explored, and I’d be curious to hear any feedback about other lessons that the VR community could learn from looking at the differences between “erfahrung” and “erlebnis.”
UPDATE: perhaps the third type of simulated direct experience could be called “incepted.” If erlebnis is direct experience, erfahrung is learned experience, then perhaps a synthetic erlebnis could be thought of as an incepted experience:
Dr. Mary Whitton has been working with interactive computer graphics since 1978 and virtual reality since 1994 in collaboration with Dr. Fred Brooks at the University of North Carolina at Chapel Hill. I had a chance to talk with Mary back in 2015 at the IEEE VR in France about her fundamental research into VR locomotion, haptics, and cultivating a sense of presence.
VR escape room games are one of the emerging genres that are particularly well-suited for the affordances of VR, and Schell games’ 
The 
Cultivating presence is one of the main goals for a lot of VR experiences, but our brains are like a black box of perceptual soup that makes it hard to know all of the right ingredients to achieve this. 
I had a chance to talk about storytelling in VR with three of the co-founders of 
They all emphasized to me that it’s still very early days of figuring out the unique affordances of virtual reality as a storytelling medium, and that Oculus Story Studio is still doing quite a bit of experimentation. They were in agreement in believing that it’s likely going to take a long time to figure out what narrative in VR looks like, and that it could be another generation before VR finds it’s true form.
While I agree that VR storytelling is still very much within a Wild West phase of development, at the same time I do believe that there have been a lot of solid lessons learned about VR as a storytelling medium that I’ve covered on The Voices of VR podcast. At the bottom of this post is a Top 50 List of Voices of VR interviews about storytelling in VR where the list is broken up into the following seven categories: the language of VR storytelling, interactive storytelling, multiple perspectives and empathy in storytelling, social storytelling, world building & environmental storytelling, plausibility & presence in narrative, and audio.
The language and terminology around virtual reality is still evolving, but the VR community has been settling into using the phrase “VR Experience” in order to describe the process of a going through a piece of virtual reality content. I had a friend/business partner over who got her Ph.D. in German literature, and she had a lot of really interesting insights about how the German language actually has two separate words for experience with “Erfahrung” and “Erlebnis.”
