One of the most common ways to describe the experience of VR is through the lens of presence, and in this episode I’m going to do a deep dive into the existing academic literature of presence in order to more fully contextualize my approach to it. This is my Storycon 2022 keynote where I elaborate on my elemental approach to presence as well as my thoughts on experiential design and immersive storytelling. I use the four archetypal elements to break down presence into four primary components where fire represents active presence, air represents mental and social presence, water represents emotional presence, and earth represents embodied and environmental presence. And I use the lens of quality, context, character, and story to explore the fundamental components of experiential design.
I started to develop my approach to presence through a number of Voices of VR podcast interviews in the fall and winter of 2016, and I was then introduced to the work of Dustin Chertoff, whose presence work from 2008-2010 draws upon the field of experiential marketing. He similarly boils presence down into the components of Active Presence, Cognitive Presence, Relational Presence, Affective Presence, and Sensory Presence. When I interviewed Chertoff in February 2017, we agreed that our approaches to presence were functionally identical. I then went on to present my preliminary ideas on presence at a Silicon Valley Virtual Reality conference keynote in March 2017, and I’ve continued to develop these ideas over the past 5-7 years.
I gave this Storycon Keynote on May 5, 2022, which happened to be the 8th-year anniversary of the Voices of VR podcast. Now two years later, I’m celebrating my 10-year Voices of VR podcast anniversary in part by airing this talk as well as a few others. I consistently refer folks to it as one of my more fully-formed and rigorous elaborations that I’ve given of my elemental theory of presence. I not only contextualize it with Chertoff’s work, but also with the broader body of academic presence research and it’s history.
I would often find there were would be different presence theories talking about some of the same concepts, but using different terminology. In this talk, I take an archetypal approach that synthesizes these different frameworks through an archetypal lens. At times it can be a bit laborious reciting an archetypal complex of keywords, and there are definitely sections of this talk that are probably better off read than spoken. So I’d highly recommend also checking out the video version as well as the PDF of the slides, and/or the fully-annotated transcript in the shownotes that includes the embedded slides and linked citations.
That all being said, be warned that this is still probably way too dense for an hour-long talk, as it’s more like an entire semester’s worth of material. It’s also probably closer to a Ph.D. defense than a synthesized book or practical handbook that’s ready for prime time. As with many other aspects of XR, many theoretical aspects are still developing, emerging, boundaries being pushed, and rules being broken. So it’s in that spirit that this is my latest fully-formed iteration of these ideas.
This talk also leans more into the theoretical parts, and some of the more practical applications often come within the context of individual interviews where the context is a lot more bounded to a specific situation, experience, or story. Be sure to check out my 30+ hours of coverage from Venice 2023 as an example for how I’ve put these ideas into practice.
It’s been through many invited lectures and keynotes around the world where I’ve been able to develop these ideas by engaging with audiences and interviewing thousands of creators over the past decade. At some point, I’m still hoping to go through a more formal write-up of these ideas to give them even more rigor, and perhaps potentially even go through a more formalized peer-review process. But as of now, this is still independent research that has not gone through the rigors of peer review. So take it for what it’s worth. Hopefully this podcast version and write-up with a full transcript, embedded slides, and citations will provide a format that makes it accessible in a new way, and perhaps it takes me another step closer towards other written forms that allow me to further develop and formalize these ideas.
So it’s in that spirit that I hope you enjoy this Storycon keynote about my elemental theory of presence as well as my sensemaking frameworks for experiential design and immersive storytelling. This talk was given on Thursday, May 5, 2022 at the the Storycon Conference in Brussels, Belgium. So with that, let’s go ahead and dive right in!
So my name’s Kent Bye. I do the Voices of VR podcast. And today I’m going to be talking about a primer on presence, immersive storytelling, and experiential design.
So like Paul said, I’ve been doing the podcast. It’s my eight-year anniversary today, and I’ve recorded about 1,600 interviews and published around 1,080 or so. So for every two episodes I’ve published, I’ve got another one that I haven’t. So I’ve just been attending around 100 different events over the last eight years and going to lots of different experiences across many different disciplines and domains of virtual reality. So I’ll be talking today about some of the different frameworks that I use personally to understand the experiences that I’ve been seeing, but also to hopefully help you from whatever orientation you’re coming from to maybe understand what unique insights that you may be contributing to the medium.
Here’s the agenda for today. I’m going to be talking about the underlying experiential age context, and then I’ll be diving into my own elemental theory of presence, which will get into some of the connections of that to experiential marketing, some of the underlying dialectics of presence, and then looking at some of the literature on presence theory and the evolution of that and synthesizing it through my framework. And then I’ll be diving into my immersive storytelling and experiential design framework, and then applying it to a number of different breakdowns of what I see as some paradigmatic examples of immersive storytelling. So first, let’s get into some underlying context for what I think of moving into this experiential age.
And so back in 1998, Pine and Gilmore wrote this landmark piece in the Harvard Business Review. It’s called Welcome to the Experience Economy. And they were documenting how we’re moving from extracting commodities to making goods and delivering services. And then from that, we’re now staging experiences.
And so then they wrote a book called The Experience Economy, where they were talking about moving into staging experiences that are memorable and personal, and that there is someone who is delivering a series of different sensory experiences that go above and beyond just, say, buying a cup of coffee.
Starbucks is, I guess, one of the examples of a company that has really adopted this type of experiential marketing, where it’s not like you just go in and buy a cup of coffee. You’re having an entire experience. And they call this notion the third space, which is not home, it’s not work, but it’s this third area that you go in. And in that experience, then you’re able to have all the music and access to the New York Times, and you meet with your friends. And so they’ve created this whole experiential aspect for their business.
And back in 1999, there was different pieces about experiential marketing and then brand experience in 2009. And so again, focusing on the experiential aspects, and the marketing folks have been on the forefront of pioneering some of these early patterns of experiential marketing, which now I think has moved into experiential design with both AR and VR.
I think another big example, if folks are not familiar, is an immersive theater piece by Punchdrunk that premiered in New York City in 2011. They had been working on this for a number of decades before that in London, but this was the flagship experience of immersive theater, which takes away the proscenium and then it puts you as the audience into the experience. Now, you’re wearing masks and you’re anonymous, and so you don’t have any narrative agency, but you have the ability to choose where your body goes. And also every actor has their own narrative, and so you’re able to, in this looping narrative, have different fragments of the story that then you are piecing together, and it ends up being Macbeth that they’re playing through. But this is a good example of moving into more of a living story where you’re embedded with your body.
One of the ways that I start to look at this is the evolution of different communication media. And with each new media, it encompasses certain aspects of that previous media, but also has something new. So you go from oral storytelling to theater to books and film and radio and games and TV, internet and mobile phone. And then eventually, now that we have this VR and AR, it’s encompassing all the other previous media, but it’s adding something new. So that’s what I’ll be digging into, “What’s new about VR, AR, and immersive theater?”
Celine Tricart is an immersive storyteller who at South by Southwest this year did a whole keynote and she was starting to map out existing media. So at the bottom, you see the mediated through a 2D frame and then you have all the existing media. And then when you start to get over in here, you have the illusion of non-mediated presence within a spatial context, meaning that you’re embedded into the experience and that you’re not actually experiencing the medium. It feels like you’re actually in the experience. And we have LARPing live action role play, as well as immersive theater. We have virtual reality, and then 360 video.
This is a famous spectrum that’s modified a little bit from Milgram, which on one end is the physical reality and the other end is the virtual environment. So as you start to go onto one extreme, you have virtual reality and then augmented reality is you being embedded into the physical context of where you’re at, but it has virtual information that’s overlaid on top of that.
You can see this from Microsoft Mixed Reality Spectrum has adapted this. And so you could see on one side is the HoloLens, on the other side is fully immersive VR. And with AR, you’re in the existing context and you’re putting information to modify your existing context. And with VR, you’re doing a complete context switch. So you could be at home, but then all of a sudden you’re at work or you’re in a completely other world that has more of a context shift than from AR, where in AR you’re still in the center of gravity of that existing context.
Presence is something that has come up quite a lot in the early days of VR, as people start to experience. And here’s Palmer Luckey talking about the magic of virtual reality presence.
And then at the very first Oculus Connect One in September of 2014, Brendan Iribe, who was at that time the CEO of Oculus was on stage, and I just remember sitting in the crowd, hearing him talk about presence, which I always thought was more of a thing I heard from Buddhism or more of an esoteric concept. But here, it was like the future of technology is about cultivating the sense of actually being somewhere. And so this aspect of presence, I think, is the thing that comes back to again and again. And I’m going to be doing a little bit of a deep dive into how we start to think about presence.
Mel Slater is probably the one that is very well cited in terms of his place illusion and plausibility illusion. So the place illusion is the strong illusion of being in a place in spite of the sure knowledge that you are not there. And then the plausibility illusion, this illusion of what is apparently happening is really happening even though you know for sure that it’s not. For Slater, he’s really trying to figure out what is it about the VR technology that is tricking our sensor motor contingencies to create the illusion that we’re in these other places. And for him, he uses this aspect of illusion because we’re not actually there and things aren’t actually happening.
But I think as we start to move on, David Chalmers has criticized this illusionary framing because for Chalmers, there’s certain aspects of the immersive experiences that feel just as real as not. And I’ll be digging into some of those differences here later on.
So Bystrom has this mapping where at the very top you have these different aspects of the display technology and the immersion. Those are the more objective aspects of the hardware. And then you have what is the mysterious black box of our consciousness. This is something that’s an open question around what even is consciousness and how you start to define it. So the subjective experience of presence is still not fully figured out, but you can see different aspects of our census are coming in. We’re starting to allocate different attentions and resources. We have the suspension of disbelief and presence, and there’s certain aspects of you expressing your agency. And so there’s lots of different components here that I think has been a little bit confusing as to sort out.
And some of the objective hardware aspects, this is something that Oculus was talking about, where you have all the different degrees of, say, the tracking technology and the latency and the persistence and the resolution, the optics. These are all the technical aspects of VR that now with consumer VR, they’ve reached a threshold where they’re good enough for us to really have an immersive experience.
And there’s also different aspects of perception and cognition where you have the sensory experience and you have the mental models of reality.
So the predictive coding theory of neuroscience is saying that our brain is a prediction machine. And so we have a mental model for what we’re expecting to happen. And then we actually have our sensory experience. And there’s this constant comparison between what we’ve experienced before and what this virtual technology is being able to mediate. And if it’s able to match close enough, then it’s tricking us into actually having that sense of presence there. But this iterative process of our expectations and our input of our sense of reality and the predictive coding theory of neuroscience, I think, is a really good foundation to understand what the mechanics are, what’s happening with XR technologies.
Then we have embodied cognition, which is another really important concept, which is essentially saying that our mind is not just the only thing that is making sense of the world, that our mind is actually embedded within the context of our body, and our body is embedded in the context of the world around us. And that when you start to put yourself into virtually mediated environments, you could see how that is actually starting to change the way that we think. And embodied cognition is a key foundational principle of explaining that.
Frank Steinecke had a keynote at IEEE in 2021, where he starts to talk about different loops from perception to cognition to action. And so again, the input that’s coming in, we’re making sense of it. And with our motor cortex, it’s actually all about movement in the world and taking action. And so each of these different phases, VR starts to do things like perceptual filtering, selective attention, and motor adaptation. But you can see this whole loop that’s going again and starting to describe the mechanics of our consciousness.
So there’s a lot of open questions as to what the subjective experience of presence is.
And this is my launching point of looking at Skarbez’s 2017 survey of presence concepts with all these different comparisons. And then my approach today is trying to synthesize that into a meta framework to help navigate all these other ideas.
Alright, so now I’m going to dive into the elemental theory of presence.
So I first presented this in February and March of 2017. I had been developing it throughout 2016. I first started to present it as a keynote at the Silicon Valley Virtual Reality Conference, but being in dialogue with creators as I was testing these different ideas, talking to different people.
The basic idea is that there’s these four major buckets in terms of active presence, mental and social presence, emotional presence, and embodied and environmental presence.
And I guess one way to start into looking at how to start to understand some of these dimensions is to look at existing communication media, where you have video games that’s all about expressing your agency and taking actions. Then you have both the computers with the internet, World Wide Web. You have the social media with your phones, and you have literature. These are all the mental abstractions in using language to be able to both communicate but also to surf the information highway or think about the different principles of human-computer interaction that have been developing in web design for the past 20, 30 years. Then you have emotional presence, which you have film, which is all about you receiving passively an experience that’s already been authored by someone who is directing your attention with all the framing and the lighting and the pacing. It’s something that is able to control the building and releasing of tension in a way that is really, at the end of it, modifying and engaging you emotionally. And the thing that’s new is the embodied and environmental presence where you’re actually being taken into the experience and being tricked that you’re actually in another place and that’s actually there and that your body’s actually in those places. So you have different aspects of, say, theater and dance and architecture, but also your raw sensory experience that’s going above and beyond what all the other media are doing. Looking at your sight, your touch, your taste, your smells, all these things are bringing into what is being called as a bucket XR, which is both virtual reality, augmented reality, and immersive theater.
So at the heart of it, it’s making choices and taking action and being in the sense of emotional immersion and sensory experience.
As we start to look at this evolution from oral storytelling to theater, books, film, radio, TV, each of these are having something that’s new for each of these areas. So if we look at video games here, it’s all about having new degrees of agency. So you can see kind of the color coding in both film and TV or developing the language of film for the last 120 years. We have people at the very beginning sitting around a fire telling stories. And then at that point, people were participating in a way where it was happening live in the moment. It was like a living myth that was constantly evolving each time the story is being told. And then having that into theatrical productions and then the internet and mobile phones are adding new dimensions of presence on social media. So now we have the VRAR and immersive theater that’s synthesizing all these things together and adding new degrees of embodied environmental presence.
One of the basis that I started developing this were the elements of fire, air, earth, and water. And these are archetypal elements. And Jung had this quote that I’m going to read here that describes the nature of archetypes.
One of the things I found by looking at the VR presence literature was that a lot of times people would be talking about the exact same thing but using different ways of defining it. And part of the challenge is when you talk about phenomenology, the qualitative aspects of your experience, it’s difficult to draw firm boundaries. But I think Jung and his archetypal approach is avoiding that in some sense where archetypes inherently are resistant to being reduced down to a simpler formula.
So I wanted to read this from Jung’s Archetypes and the Collective Unconscious. He says that “clear-cut distinctions and strict formulations are quite impossible in this field of archetypal psychology. Seeing that a kind of fluid interpenetration belongs to the very nature of all archetypes, they can only be roughly circumscribed at best. Their living meaning comes out from more their presentation as a whole than from a single formulation. Every attempt to focus them more sharply is immediately punished by the intangible core meaning losing its luminosity. No archetype can be reduced to a simple formula. It’s a vessel which we can never empty, never fill. It has a potential and existence only. And when it takes shape and matter, it is no longer what it was. It persists throughout the ages and requires interpreting ever anew. The archetypes are imperishable elements of the unconscious, but they change their shape continually.” So you get the sense that every time you try to describe something, it’s like Gödel’s Incompleteness. There’s always something that’s outside of that that wasn’t completely integrated in that first definition.
And so there’s a metaphor of like a diamond that has many facets. If you circumscribe those facets, then you can kind of get a sense of what the meaning of that archetype is. And that’s essentially what I’ll be doing for the rest of this presentation is trying to give many different interpretations of these different elements.
And so I’m going to start with more of the Western esoteric archetypal tradition and just kind of read some of the different keywords that are underlined here to give you a flavor of some of these different elements that I’ve been using as a baseline to then expand. In the next slide, I’ll be saying how these actually get interpreted into the XR technologies. With fire element, you have energetic, creative, aspiring, spontaneous, passionate, assertive, courageous, active, spirited, and impassioned. For the air element, you have mental, intellectual, abstract, rational, logical, conceptual, theoretical, communicative, social, gregarious, inquisitive, alert, objective, impersonal, impartial, observant, clever, and witty. For the water elements in the emotional presence, you have nurturing, sustaining, providing, protecting, shielding, feeling, emotional, sensitive, empathetic, compassionate, bonding, merging, unifying, dissolving, absorptive, flowing, receptive, yielding, changing, deep, hidden, and mysterious. And then finally, the earth element, which is the embodied and environmental presence. You have practical, pragmatic, deliberate, cautious, grounded, solid, stable, physical, tangible, material, substantial, structured, corporal, and sensuous.
Now, as I start to go into the specific things, again, that archetypal nature is that they’re always ever expanding and flowing. How I see the active presence is all about agency and interactions, dynamics, action, locomotion, exploration, spontaneity, creativity, will, engagement, gameplay, intention, and live performance. Again, going into like video games. Then you have mental and social presence. You have thoughts, language, choices, expectations, and novelty, mental models, possibility, coherence, attention, suspension of disbelief, puzzles, mental friction, rules, social presence, and virtual beings. And then emotional presence, you have the degree of emotional immersion, the feelings, mood, vibe, color, lighting, consonants and dissonance cycles, music, story, time, flow, narrative presence, empathy, and compassion. And then finally, in the body and presence, you have self-presence, body, embodiment, sensory experience, biometric and physiological data, brain-computer interfaces and neural inputs, haptics, immersive sound, spatial presence, and environmental immersion and architecture.
Given all of those, and we go back to active presence, mental, social presence, and emotional presence, and environmental presence.
And then looking at the different communication media, the center gravity of each of these are reflecting that. So again, degrees of the video game that’s really expressing agency and interactivity. And then the social presence and mental presence. The internet, world wide web, and literature using language and abstractions, but also the social media element where we think about the metaverse is all about being present with other people in a social dynamic. The plausibility, the suspension of disbelief, and then the emotional presence for film, and then the embodied and environmental presence that is new within the virtual and augmented reality technology, but with that new perspective is allowing us to see how each of these other media have a center of gravity for what they’re really good at. So XR is all about blending all these things together. And so then the question becomes, well, how do you start to take each of these very different design disciplines and start to fuse them together?
All right, so now I’m going to look back into experiential marketing and the connections to this theory of presence and to what’s already been happening with experiential marketing.
Richard Skarbez is someone who tipped me off to the work of Dustin Chertoff on December 4th of 2016. I was able to follow up with Chertoff, and there’s kind of like an independent discovery that my elemental theory presence had already been talked about by experiential marketing back in 1999. And then Chertoff had said, we need to expand a theory of presence by looking at experiential marketing. And I got a brief mention in Skarbez’s survey because of that.
This is Chertoff’s first piece that he did in 2008, Proving Presence Theory Through Experiential Design. Basic theory was that, hey, there’s all this stuff that’s happening in the advertising realm that can start to provide insight into what’s happening with immersive technologies.
Then in his dissertation that he did in 2009, he started to do a whole survey of all the different theories of presence at that time.
And then he started to come up with what his categories were, were sensory, cognitive, affective, active, and relational.
And as I start to translate that back, it’s the active is kind of the active presence, the cognitive, relational is the mental and social presence, the affective is the emotional presence, and the sensory is the embodied and environmental presence. So you can see the archetypal nature where it’s the same themes repeating, but in different forms.
So if we go back to some of the original articles, back in 1999, there’s this piece on experiential marketing, and they called it the strategic experiential modules of sense, feel, think, act, and relate.
So again, these translate over into the different elemental act, think, and relate, feel, and sense.
So to just read the definitions to give you a little bit of flavor. So act is enriches customers’ lives by targeting their physical experiences, showing them alternative ways of doing things, alternative lifestyles and interactions, so a lot about the behaviors. Think is appeals to intellect with the objective of creating cognitive and problem-solving experiences that engage the customers creatively. And then relate is expands beyond the individual’s personal private feelings, appeals to the need to be perceived positively by individual others, and they relate to them in a broader social system. So starting to get to more communal social aspects with social presence. Feel appeals to customers’ inner feelings and emotions with the objective of creating effective experiences that range from highly positive moods to strong emotions and joy and pride. And then Sense is the appeals to the senses with the objective of creating sensory experiences through sight, sound, touch, taste, and smell.
So, again, if we start to look at how they start to measure brand experiences, and so then they’re starting to find ways of, like, how you think about this brand, and you can look at it at the dimensions of sensory, affective, behavioral, and intellectual.
So this was in 2009, starting to look at behavioral, intellectual, affective, and sensory.
And then, again, these are all, in my mind, all connected to active presence, mental, social presence, emotional presence, embodied and environmental presence.
All right, so I’m going to talk a little bit about the underlying dialectical nature for why I think this is an approach that starts to get to the nature of qualities.
So going back into Earth, Air, Fire, and Water goes all the way back to Empedocles…
and then even before that for each of these individual elements. And at that time, it was much more about physical descriptions of the world. But for me, I think that the elements still have a qualitative phenomenological experiential aspect of them that even though the original physics theories weren’t playing out, that they still have an experiential phenomenological validity.
So they had breaking down into hot, dry, wet, and cold.
So if we look at this in the dialectic, the fire element, as an example, is hot and dry. Air is hot and wet. Water is wet and cold. And then earth is dry and cold. For me, the hot and cold is really a key thing that I’ll dig into here in a bit.
But the origins of this goes back to the medical aspects of taking this temperament and humors of looking at the phlegm, blood, yellow bile, and black bile.
So again, you have the hot and dry is the choleric, hot and wet is the sanguine, the cold and wet is phlegmatic, and the cold and dry is melancholic. So this used to be the early theories of medicine that was ways of understanding the different temperamental nature. In the Chinese medicine, that’s continued in that sense of using these elemental aspects, but in Chinese medicine, they have the five elements instead of the four elements.
But Jung took this as a baseline for his theories on psychological types. So he had thinking, feeling, sensation, and intuition. And he was really inspired by these temperaments and the humors, and also looking at the four elements of earth, air, fire, and water.
And so his conclusion was that there’s intuition, thinking, feeling, and sensing. And this actually got fed into the Myers-Briggs personality test. So this was like the first foundations of looking at different psychological type.
But for me, I think the hot, cold, wet, and dry, the hot and cold is a really good dialectic to look at.
In terms of one is giving outward and one is receiving inward. And then the wet and the dry is you can kind of think of it as water is added. It sort of has a bonding aspect. And the dry is more individuating. I think the hot and cold are a little bit more intuitive because it has a dialectic between —
in Chinese philosophy, it’s the yang and the yin.
So the fire and air being the more yang elements, and the earth and the water being more the yin elements.
And so the yang being more outward and yin being more inward. It’s the giving versus receiving, talking versus listening. So the yang is all about energy that’s going outward.
It’s the sun that’s radiating the light. And the moon is the one that’s receiving the light and reflecting it. And so it’s much more receptive. And so you have one that’s active and one that’s passive.
And just even if you look at the parasympathetic versus sympathetic systems, There’s always these dialectics between these two aspects.
With the fire and the air being more outward and the earth and the water being more inward, in some ways, all of these are all happening all at once. But any given interaction they’re having, you’re in a center of gravity of one of these or another. And I think the process of experiential design is doing things to modulate between these gives and takes in different ways.
So, again, each of these different communication media have a center of gravity. Things become a little bit more game like or they become a little bit more puzzle like or mental friction that you’re creating or they come out a little bit more story beats that you’re showing aspects of the story that are unfolding or they become more about you being embedded into the experience. And it’s more about you receiving the environment with your senses and just taking it all in.
So one last part here, the Quadrivium…
looking at how there was astronomy, math, geometry, and music.
And then the Pythagorean approach of saying that all is number, that mathematics is just number, and then geometry is numbers in space, music is numbers in time, and then astronomy is the numbers in space and time.
And so we start to extrapolate that of how this gets applied to looking at all these different experiential design. You have the more fire elements of objects moving through space, more of the physics and dynamics aspects. You have the abstract patterns and forms and the design with math and all the coding, putting everything together with all the rules and mechanics. And then the unfolding processes are the consonants and dissonance cycles that are unfolding in time that we have a direct experience within music. And then finally, the architecture embodiment in space is the world and the body. All right. So that’s the aspects of the Quadrivium.
One last part here is the presence qualities within biometric sensors and brain-computer interfaces. As we move forward, we’re going to have new sensors like EOG, EEG, EMG, EDA, temperature. And so basically, our body’s radiating all these biometric signals. And so because of that, those signals are going to be fed into the experiences and have these feedback loops.
And so as you start to look at all the many different physiological and biometric inputs, this is what’s happening right now in medical fields. So you can imagine any number of these are gonna start to be fed into XR technologies as we start to move forward. Brain computer interfaces, all these things, what are they trying to measure?
And so if we could start to define all of these sensors are going to be able to extrapolate different aspects of our behaviors, our intentions, our actions, our movements, our creations, our engagements. Mental and social presence from mental thoughts, cognitive processes, cognitive load, social presence, and predicted expectations. So being able to literally read your mind. Emotional presence of your affective state, emotional sentiment, facial expressions, and micro expressions. And the embodied presence is the sensory input, your processing of the sensory input, stress and arousal, physiological reactions, eye gaze, attention, body language, and muscle fatigue. So as we move forward with XR technologies, these are all the basic primitives that you start to play with and have access to and then feed into your experience to be able to dial it in and to control more fear or be able to sense what their emotions are.
This also happens to have a lot of implications of our mental privacy, our right to identity, and our right to have autonomous actions. So yeah, there’s Neuro-Rights that have implications here. That is a whole other part of my work.
But I think this series of presence can help us understand a taxonomy for what all these new sensors are gathering and what is able to be done with it. So active presence, mental, social presence, emotional presence, and embodied environmental presence.
All right, so I’m going to talk briefly about some of the evolution of presence theory and some of the way that I was able to synthesize some of these other approaches, and then we’ll move on to immersive storytelling and experiential design.
So I was starting with this overview from Skarbez that was doing a survey of all these different presence-related concepts.
So going all the way back to 1965 is when Ivan Sutherland was presenting his idea for the ultimate display. And Fred Brooks was there. And he was documenting that Sutherland was saying that the future of technology, you should see it as not just a screen, but a window through one sees a virtual world. The challenge is to make that world look real, act real, sound real, and feel real. This was the vision back in 1965. And I feel like since then, we’ve been having this trajectory of technology that’s living into this vision that Sutherland had in 1965.
And then in 1968, he built the Storm of Damocles, which was like the first manifestation of the virtual and augmented reality technologies.
And then in the 80s, you have telepresence, which is a big aspect of thinking about robotics and trying to have your embodiment within that robot to make choices and take action within the context of whatever environment that was in.
So you have the first definitions of telepresence in 1983, which is that you’re receiving sufficient quantity and quality of sensory feedback to provide a feeling of actual presence at the work site. So again, it was all about having like a telepresence robot and you’re there mediated and being present through this idea of telepresence.
And there’s a Stephen Ellis essay from 1991 where he recounts a lot of the early days from the early 60s all the way up into the 90s of what was happening within the context of the military and NASA that was the consistent development of XR. It kind of went dark into the second big wave in the 90s, but NASA and the military technologies and training were all continuing to develop these ideas in virtual environments.
So 1986 was NASA’s virtual environment and display system that starts to see how they’re doing it for training for astronauts.
And then 1992 was a big turning point because that was the launch of the presence journal that started to have the larger communities starting to look at both the telepresence but also these new virtual and augmented reality technologies that people were trying to make sense of it.
Sheridan initially tried to make this differentiation between the telepresence, which is the sense of physically being present with virtual objects as a remote teleoperator. And you can see that it’s all about you having a mental model of that world. So you have an idea of a mental model of that world, but you’re not actually physically there. It’s more mediated through the technology. Versus the virtual presence, which is the sense of actually being physically present with visual auditory or force feedback that’s generated by the computer. So the difference between telepresence and virtual presence starting to get fleshed out at the very first issue of presence.
And then Mel Slater came along in 2009. Again, I was talking about how Slater was really focused on the sensory motor contingencies that were very unique to VR. And he’s not concerned about what your sense of presence is in a film or video game or any other media. He’s only looking at VR technologies and trying to define these illusions of presence and not really looking at the larger subjective phenomenological aspects of presence. So he’s defining that immersive systems can be characterized by the sensor-motor contingencies that they support as they define a set of valid actions that are meaningful in terms of perception within that environment.
So again, he’s in some ways defining immersion as the objective qualities of the technology, because from VR presence research, they already have ideas to talk about the phenomenological experience. And so there’s ways in which the VR community has adopted that immersion only means the objective aspects of the technology, which I personally don’t agree with because I think of immersion as the psychological aspects as well.
Like I said, Slater and the VR community is not looking at immersion in terms of the psychological aspects. But if you actually look at some of the game literature, they talk about immersion as engagement, engrossment, total immersion, flow, cognitive absorption, presence, participation, narration, and co-presence with others. What I see is the taxonomy of presence as well. So there’s a lot of overlap between what games folks talk about when they talk about immersion from what the VR folks talk about when they talk about presence.
And then Sheridan, he starts to talk about the control of sensors, which is all about the embodied and spatial presence and the ability to modify the environment through active presence and the extent of the sensory information through the sensory presence.
Heeter talks about subjective personal experience and the social presence and environmental presence. Again, this is, as we start to go back through the early literature, you start to see ways of starting to define social presence for some of the first time.
And then Witmer had the presence definition was all about your embodied presence of being in another place. And so it’s defined as the subjective appearance of being in one place of the environment, even when you’re physically situated in another.
Skarbez starts to pull in different aspects of social presence to kind of flesh out the place illusion and plausibility illusion. Again, there’s a lot of illusionary talk in terms of the social presence illusion and the co-presence illusion. So when you talk to someone on the phone, you think, oh, well, that was not a real interaction because I wasn’t actually co-present with them. Adopting that illusionary perspective is grounded in only thinking about things in terms of being tricked.
But Chalmers disagrees with that assertion within Reality+, saying that all of these VR experiences can be just as real and just as meaningful. And I’m more on that side, pushing away from this illusionary framing.
There’s other research into social presence that looks at the modality, visual representations, haptic feedback, depth cues, and audio quality.
At Sundance, I started to have experiences of emotional presence.
And then the dramatic presence is something that was talked about very early within the context of presence literature, which is really about becoming emotionally engaged and causing immediate personal emotions.
And then the emotional involvement via transportation. So being transported in a world is being defined as the experience of cognitive, emotional, and imagery involvement within a narrative.
So Skarbez and this table starts to look at all these concepts.
And the way that I would present them is more from this way. So looking at active presence and looking at all the different literature, there’s aspects of agency or modifying your environment, the degree that you’re engaged, the participation control, the plausibility illusion, the degree that you’re interacting with the plausibility illusion, and the action fidelity. The mental and social presence has all about the plausibility illusion, the coherence, the reality judgment, the cognitive absorption, social presence, selective attention, psychological immersion, telepresence mental model aspect of the telepresence, the co-presence with other people, anticipation of events, consistency of information, interface awareness, and communicative immersion. The last two, the emotional immersion, Involvement, engagement, flow, absorption, engrossment, transportation, transportability, total immersion, narration, dramatic presence, and meaningfulness. And then finally, the sensory presence, modality, embodiment, body ownership, avatar, self-presence, and the subjective personal presence, perceptual immersion, spatial presence, place illusion, realism, fidelity, scene realism, experiential fidelity, multimodal presentation, telepresence, and separation anxiety. That’s a lot, but just to give you a flavor of how the existing presence research, this is how I start to think about it.
So now how do you start to apply these degrees of presence as you think about overall a framework for immersive storytelling and experiential design, which is what I’ll cover here and then spend some time at the very end breaking down a few experiences.
So there’s a quote from Robert McKee from his book on the story, substance and structure, style and principles of screenwriting. He says that true character is revealed in the choices a human being makes under pressure. The greater the pressure, the deeper the revelation. The truer the choice to the character’s essential nature. I think this quote actually encompasses all the different ways that I think about story and as we think about immersive storytelling as we move forward.
Because you’re being placed within a context that’s being put under pressure. You’re making choices and taking action. And then there’s a part of your essential character that’s being revealed. And it’s all within the context of these unfolding processes that are happening over time.
So you can break this down in terms of context, quality, character, and story.
So we’ve already talked about quality ad nauseum, about the different qualities of presence.
I’ll just quickly show that within the context of, say, Half-Life Alyx, how that starts to play out is you can look at even within the first scene that you go through, you have different aspects that are focused on either you gathering resources and observing the environment. There’s different cut scenes and dialogue and soundscapes for the emotional presence. There’s puzzles and navigation that you’re doing within the game. And then there’s also combat and interaction. So you’re either engaging with the environment or you’re fighting enemies. So you can see as you go through scene by scene, there’s different combinations.
And then the whole experience is that you have this balance across all these different domains. And you’re channel switching or combining them in different ways throughout the experience. And even throughout the entirety of the experience, even if you go level by level, you can see that some levels have a lot of story aspects, and some of them are just about gathering resources, or exploring the environment, or you solve puzzles, or there’s different combats, or different ways that you’re engaging with the environment.
So in one way, you can think about it in terms of if it’s an immersive experience, it’s all about you as an individual taking action, making choices, and you being emotionally immersed in the sensory experience that you are having within experience. But there’s also, if you’re watching a movie, this is all about the characters are going through all these different degrees where they’re having to make choices and take action. And the way that they’re expressing their emotions or being immersed with their senses.
Again, if we go back to the different communication media, you could see that for each additional media, it’s a center of gravity of adding something new, but it’s also incorporating all the other previous media so that by the time you get up to XR, it’s including all the other previous media and all their different affordances. And so each of the design disciplines are uniquely contributing to that.
Going back to this chart from Celine Tricart, you could see that from the first person point of view, it’s all about you’re experiencing the qualities of your own presence. So if you’re immersed within an XR experience, it’s all about those qualities of presence for you as an interactor. Whereas if you’re watching a film, again, it’s watching someone else’s character’s presence.
So that’s the quality. If we think of story…
Alex McDowell actually has this picture where he says, if you go back to the early days of telling stories, it was a living myth. It was alive and constantly changing and evolving. It would be different every time that you would tell the story. What I think is different, when you have the advent of the printing press, there became a canonical version, that that was the authoritative version of how you tell the story. And it’s fixed in time. But as we move now into this more post-linear area, it’s all about creating these realms of potential that you can start to explore the ways that people, as they’re interacting, become a part of the story. So it’s not just about storytelling. It’s about story living, where you’re actually living into the story.
And so as you have the story structures, so there’s different degrees to which that you are allowing the participant to explore those realms of potential and to make choices and take action. You have on one end the authored story where you as a participant, you have no agency or impact on the story. On the other end, you have the generative narrative where you’re really maximizing the degree of which that you have agency and will of expression. Now, the challenge here is that stories are all about building and releasing tension and really being in control of that. So a filmmaker can have maximum aspect of authorial control of how to control that. But once you give the user the ability to make choices to take action, you disrupt that normal flow of the building and releasing of tension. A user could be looking at a chair for 10 minutes. And if you’re trying to be very timely for how you’re telling a story, that means that it’s disrupting that.
So there’s a number of different structures from Lebowitz and Klug, where you have on the far extreme, you have the fully traditional stories, interactive traditional stories, multiple ending stories, branching past stories, open-ended stories, and fully prayer-driven stories. So again, on one end is kind of more the emotional presence, and the other end is the active presence. And so to what degree are you combining those? They’re kind of a dialectic that’s the heart of the challenge of telling stories within game-like mediums.
So you have the traditional structure that has the three-act structure from Aristotle that we are all very familiar with. Again, as a viewer, you’re not making any choices, and so the author has complete control over how they tell the story.
Then you have a little bit of interactive traditional story. This is where you see a lot of in terms of interactive media because everything is coming back into more or less an essential linear story, but you’re flavoring it by little choices that you’re giving the user, and they’re modulating their experiences. But at the end of the day, everyone’s still seeing mostly the same story because it’s all converging at the end.
The branching path story is kind of like in the middle where you have the author who has to make an exponential amount of new stories for every choice that you’re giving them. But as a viewer, you have the experience of going through all that, but you may not even know that you’re making a choice. And so all that work the author is doing may be lost on you because you had no idea that there was even other options that you could take or that there was other branches. So you tend to have smaller stories that you play multiple times to explore all those realms of potential.
And then the open-ended stories is when you start to flip more into the game-like elements, where it’s more about exploring a probability space and interacting with different objects or having a little bit more open-world sandbox type of experiences. But once you’re in this world, it’s harder to tell the story because, again, you don’t have control of how you’re building and releasing that tension.
And then you start to get into the drama manager, like Facade, where you have 2,500 dialogue pairs. And based upon your actions that you take, you have a calculation of what you’ve made. And then you’re delivering the most intense story based upon the game state that you have at that moment. So having a drama manager that’s able to reveal that story over time.
And the final aspect is the fully player-driven stories, which is more about creating a probability space of environmental storytelling or an open-world sandbox, and then finding other ways of telling the story through the objects that you’re interacting with or the characters that you’re interacting with.
So on the one end, you have the passive experience with static particle-like actuals, and then the other end, it’s more about the participatory experience with dynamic waves of potential. The realms of potential is all about what is possible? What are the possibilities? The immersive storytelling is trying to find that perfect balance of that.
Now, when you are providing opportunities for people, often times you have this metaphor that comes from the skimmers, the dippers, and the divers, meaning that some people are going to want to be fully immersed with an experience, and some people just want to skim at the top. So an example of this is Alien Rescue, where you have people who are acting on stage and actually are participating as a live action role play. And those are the divers. And the dippers or skimmers are maybe the embodied robots who are able to move around and have embodied agency, but they don’t have any narrative agency.
And also, the active presence could be the deepest diver, but you can also have it so that the deepest diver is people becoming the most emotionally vulnerable. So there’s ways that you can order what you consider, for whatever your experience is, what the skimmer, dipper, and diver is going to be.
OK, so that’s the story, the context. So the context is the thing that’s also very new with immersive technologies.
This can be everything from the genre, but it could also be from the context of AR. You’re switching your existing context in VR. You’re doing a complete context switch.
There’s different models of context.
So there’s world building that starts to get into all these different degrees from Alex McDowell’s world building, where you’re building entire worlds.
You have different phases here from the data, technologies, products, experiences, systems, and implications. So it’s kind of neurologically nested as you get more and more data. It sort of builds up more and more larger contexts.
Lessig’s Pathetic Dot Theory is something that I look to a lot in terms of describing how any complicated ethical issue has a combination of cultural norms, the law, the market dynamics, and the technological architecture and the code.
And so when you start to look at it from the nested context, you have the largest context. You have the Earth. Then you have the culture. Then you have the laws. Then you have the economy. Then you have the design guidelines. And then you have the network architecture, XR hardware, operating system code, app code, and then at the heart, you have the user experience. So when you think about the metaverse, it’s incorporating all these different dimensions of having economic dynamics and the rules and the laws. But for each of those existing at the user experience, it’s always within the context of the app, of the operating system, the actual hardware. And so the contextual dimension starts to bring all those in.
The other aspect are the domains of human experience. As I ask over 1,000 people of the ultimate potential of VR, they usually say, well, it’s going to be all about education. It’s going to be about technology or medicine or entertainment or empathy and partnership, expression of identity, being able to use it in your career, higher education, travel, spirituality, being able to connect to friends and family.
And so each of these different contextual domains are able to take you into those contextual domains for a character. And as you’re immersed in that contextual domain, you understand more about that domain for that character. And something like The Book of Distance does this in a really effective way.
Now, this is also a lot of the basis of work that I’ve done with ethics, and there’s an XR Ethics Manifesto that you can look up on YouTube that looks at all the ethical and moral dilemmas and say that here’s all the different ways in which this technology could go horribly wrong if we don’t have guardrails in place to start to make sure that we’re keeping things in ethical order. So context ends up being an important framework to understand all those different moral and ethical dilemmas.
And the final aspect here is the different axes of privilege, domination, and oppression. This is all the ways in which that on the top are all the more privileged aspects of people who are white, male, young, attractive, ways in which our culture is kind of exalting that. And then there is power dynamics and structures that there’s whole other lenses that you could just look at it from a feminist perspective or a critical theory or ways in which that there may be class bias. So I consider this aspect within the context because a lot of what VR experiences are doing are trying to reveal the hidden aspects of our culture and our context and provide you an experience that allows you to reveal different aspects of hidden implicit biases that we have as a culture. So each of these domains could be their own experiential design perspective.
So that’s the context. And the last aspect is character.
And character, you can talk about how you’re placing a context, you’re making choices, taking action, and your character is revealed. And so how is it revealed? What’s it mean to reveal someone’s character?
And so at the baseline is are you a ghost or are you embodied? Meaning, are you a passive witness to something’s happening or are you actually embedded into that experience and you’re being addressed as a character? So that’s more of the embodied presence. And then you have the active presence, which is to what degree is your character able to make impact on the story or not? So you either have impact or you don’t have impact.
And so there’s also the dimension of whose character is revealed? If it’s an authored narrative, you’re watching other people. So it’s all about the character of an event, person, place, or culture that they’re making choices to take an action, and you’re seeing their character be revealed. But if you’re in an immersive experience, it could be that your character is being revealed. So what’s it mean for you to be in an immersive experience for 40 minutes and learn something about your own essential nature of your character based upon this contrived context you may be in, but it’s maybe able to be universal in the sense of revealing something about yourself that you’re walking away from that experience and understanding more about who you are as a person.
So there’s many different ways of talking about character. This is from Westworld, where it starts to look at all the different ways in which they’re designing their AI characters and all the different temperamental aspects.
You have the virtue continuum, so virtue ethics and how you have a golden mean between two extremes of different aspects. So there’s a virtue continuum that you can start to look at.
There’s a big five personality characteristics and another way of talking about different aspects of someone’s personality or their character.
And then there’s the feelings and needs that you, at the heart, are saying each character has their feelings and their needs and desires, and that’s driving them to make certain actions.
So again, there’s you being placed in the context, making choice, taking action, a central character being revealed, and there’s unfolding processes over time.
All right, so the final section here, I just have like 10 experiences that I’m going to go through quickly and try to highlight how I start to break this down. I’m not going to be able to cover every aspect, but I’ll try to give you a sense of how I start to apply this to what I see as 10 examples of experiences within XR that you may or may not have seen, but helps break down how this theory works.
All right. The first one’s Beat Saber, which doesn’t really have a story. It’s more about an AR or VR rhythm game that gamifies this experience of exercise through these embodied puzzles through music. So it’s a lot about, for me, I think Beat Saber is so successful because it actually integrates all these different qualities of presence pretty equally in terms of, like, it is a game, so you’re expressing your agency, and it’s a satisfying game mechanic. It’s easy to play. It’s difficult to master. And it needs your very precise embodied movements. And so you’re solving puzzles with your body. You’re ducking and dodging walls. It has haptic responses with your hands. It’s spatializing rhythm, developing your sense of spatial presence, and has these unconscious reactions where you’re doing things that you don’t even aware that you’re doing. And it’s a fitness application. So then in the mental presence, you have to be able to notice all the different patterns and be able to predict what your body needs to do. And it takes a long time for you to connect that prediction with your embodied movements. And then there’s a degree of emotional immersion that’s cultivating these flow states. And it’s very immersive and fun. And the music is a big part of it. The character is incrementally improving perception, timing, and body movements, development, speed, agility, precision, and cultivating flow states. So again, I think this is a very balanced experience. And I think it’s part of why it’s the biggest VR game that has ever been created. And I think in some sense, it’s very balanced in all these different areas.
VRChat is a social VR application. And it’s closest to what I imagine that the metaverse is going to become in terms of what’s happening in VRChat. It’s a social VR platform that’s emphasizing custom avatars, custom worlds, and private instances. So for me, the two things here are the embodied presence and the social presence. So it is a social VR application that’s all about you having interactions with other people. And there’s all these underground communities and cultures that are developing. It has lots of different UI interfaces for how to navigate and explore the metaverse. And it has a social graph where you have your friends that are there. But the other aspect is the embodied and environmental presence. You’re able to upload your own avatar, but also create your own worlds. And so it’s the most fully fledged. You have full body tracked avatars. It has the most advanced systems of avatar representation and identity representation. It’s basically going all in in the virtual body ownership illusion and lots of different interactive and So it’s all about expression of your identity and connecting to a group of friends. This is kind of the essential character.
The Collider is an experience that you enter an imaginal machine that’s colliding you with another audience member and you’re doing this one-on-one interaction. For me, this was a piece that started to explore character in a very unique way because it’s exploring your relationship to boundaries and power. You’re with another person and you have to make decisions as to if you’re going to transgress in their boundaries. Someone else is in VR, and you’re not in VR. So it’s all about exploring the idea probability space of where are you going to choose. There’s an asymmetry of information between you. For me, it was a really powerful way because it was reflecting my own relationship to power and boundaries.
On the Morning Wake to the End of the World, just was at South by Southwest. It won the best experience there. You lived through the false missile attack through Hawaii. People got a text message saying that you’re going to be suffering from nuclear annihilation and seek shelter. So it puts you into that experience. And it allows you to have this sense of embodied presence of what it would feel like to be in that situation. You’re not really having any action that you’re taking. But for me, I felt like it was focusing on this sense of emotional presence within that context.
And then movie Alien Rescue is a project where you are being embedded as a live immersive show. There is role play with two main branches and you can either save the enemy or go down another narrative branch. But it’s immersive theater, live action role play. So it’s like the closest thing to a living story. And so because of that, it’s exploring all these different aspects of you feeling like you’re with these other characters and you really feel a part of a community when you do that as the main protagonist.
The Book of Distance, is one that I think really explores context in a very specific way. It’s all about the Japanese internment. It’s a linear, immersive story that has some participatory parts, and it’s very theatrically staged. But it’s moving from a story of migration from a journey from Japan into Canada. You have a new home, and then you work, and then you then go into Japanese internment. In this experience, you’re watching it, but also then you’re embedded as a first-person character as if you were participating alongside this family as they’re going through this. And it really takes you and embeds you into each of those contexts as they tell the story. So I feel like the book of distance that’s on Steam and free is really exploring the potential of using context as a mode of storytelling.
Pearl is like a music video, and so this is something that was really emotionally evocative. You’re sitting in a car, and so you have this deep sense of embodied presence, because we all know what sitting in a car is like. And so it’s like a time lapse of the relationship of a father and daughter, but it also has the car as a character And the car is embodying the story of The Giving Tree. So it’s the story of how the car is participating in the context of this relationship of the father and daughter. And so you end up having both the father and daughter, but the car ends up being the main character as well as you watch this piece. And it’s really, really well done and emotionally evocative.
Virtual Virtual Reality has lots of really innovations in terms of you’re embedded as a player. So you feel like you’re making choices, but the creators have really put a lot of guardrails from the point where you feel like you’re having an open world exploration, but it’s really quite linear. So they have a lot of control of how you’re receiving the story. You’re also embedded within the context of a mech that has five other characters who are talking to you. So it’s a character-driven story. Also, it’s a game. So it’s actually blending all these things in a really, really innovative way.
The Key is a great experience that is all about a refugee crisis and losing home, but it gives you these choices and interaction that is a dreamlike metaphoric connection for the interactions that you’re doing in that story and how that connects to the story of the theme overall. So The Key is something that I highly recommend folks checking out to see that.
And Traveling While Black is the last piece here. And this is an immersive 360 video project that uses the power of 360 video where you don’t have any agency. The extent of your agency is you can decide where to look. So you’re looking around. But it actually makes it feel like, as the camera, that you’re a character in the story. And you’re sitting in a circle with other, in this case, it’s African-Americans who are talking about their experiences of racism within America. And sometimes in those conversations, if you were actually there, it may actually change the tone of the conversation. So you feel like you’re kind of, as a ghost, being to listen to the depth of intimacy of conversations that you may not be a part of. And it does a really amazing job of using the power of 360 video to start to show how this sense of emotional presence that you can have with intimacy with a group of people.
So that’s kind of an overview of all the different aspects of experiential design, my element to the theory of presence and how it’s connected to experiential marketing, the underlying dialectics of presence, presence theory, how I start to think about immersive theory and experiential design, and then starting to break that down. And again, you can look at the slides to dig into more details.
But I’m happy to take some questions with however much time that we have left here.
Paul Taylor: Brilliant. Give him a big round of applause. Grab a seat, Kent. I did warn you, right? Yeah, I did warn you. My word. I think we’ve peaked. Should we just go home now? I mean, it’s absolutely incredible. And you should see my notes. It looks like a knitting pattern, basically. We don’t have a huge amount of time for questions. I have a question. You’ve taken us onto so many different levels and you’ve revealed so much about the work that you do. And by doing so, just like in a story, you’ve revealed something about yourself as a human being. I’m curious. You talk very much about inwards and outwards and the way in which, you know, the direction of flow. You seem to be someone who captures a lot of stuff that’s going on and you use this incredible brain of yours to frame and to contextualize and to categorize and to make sense of it by using all sorts of internal reference points and external ones. What interests me is where is the feedback loop? Have you gained any sense that what you’re doing by chronicling this and being journalistic about this is having some sort of feedback loop onto the industry itself? Have you noticed that people and the crafts that they are exploring are being influenced by your work?
Kent Bye: It’s hard for me to make that assessment. I think that I’ve had certain there’s people that are in the industry that have been listening to my podcast. Actually, the latest podcast, Roman Rapic is someone who has done lots of different experimentations with music and listening to the podcast was very crucial for him for understanding music. what was happening. But it’s hard for me to judge the overall scope. What I can say is that as I talk to many different creators, having this very refined philosophical approach means that as I see experiences, I’m trying to pay attention to all these different things all at the same time. And VR can be very overwhelming. So I’ve seen 3,000 or 4,000 experiences. And so you develop your own sense of phenomenology of how you start to pay attention to all what’s happening in an experience. and to be able to notice what’s new and what’s different, and then to be able to talk to the creators to then ask them what their process was and why they made this decision or why there was some trade-off between here. And then I think it’s through that practice of, at this point, doing 1,600 interviews, which is to talk about the practice and put language to what’s happening with this, because it’s all fusing all the stuff together in new ways. And so it’s trying to find ways of describing at the baseline what I feel and what I experience. And then what I noticed as I was putting together those last slides, it was sort of like a memory dump. This is a capture of what I remember from this experience from these different domains. So it, for me at least, becomes a schema for how I make sense of the experience. But it also provides me a context under which that I can talk to creators. And what I often find is that because I am paying attention to all these things, I’m able to have conversations with creators that go really, really deep into the nature of their practice. And for them to be able to articulate their work and reflect on their work in a way that they’ve never had an opportunity to because the way that I’m either asking the questions or reflecting to them my experience allows them to articulate in a way that, you know, go beyond what they’ve been able to explain before. So I feel like that dialectical process is a big part of my own process.
Paul Taylor: You’re becoming a therapist in a way. You’re becoming a frame maker for other people’s more looser thinking.
Kent Bye: Yeah, and I’m a pluralist at heart, meaning that I don’t think there’s any one fixed way. I have my own philosophy, but there’s going to be other philosophies. And so I think that’s important to say that there’s a theory and practice to this. And so as a creator, you should always abandon all of the theory and just follow your deepest intuitions. Then I come in at the end reflecting upon having a very defined ways of thinking from an experiential design perspective to reflect upon that and then they can articulate it. But for whatever balance that people have between the theory and practice, I think it’s going to be your own intuitive process. You should trust that first and then When you either get stuck or you’re starting to have feedback from people, that’s where I think we start to have like a critical framework that starts to then articulate some of these different trade-offs. Because as experiential designers, you’re kind of like cooks in the kitchen and you’re trying to add different ingredients and you’re tuning your experiences. And so depending on either what type of emotional experience, what type of active experience, what type of mental experience, what type of embodied experience, and then there’s the story you’re trying to tell, and then there’s different aspects of the context that you have, and then there’s the character that’s being revealed. All of those are all independent variables where the context, quality, character, and story are within those, there’s equivalence classes within each of those. So there’s trade-offs that are inherent to those, but there’s also things that are completely independent from that. There’s also things that are kind of nested within it. For every story you’re trying to tell, it’s going to be some combination of all those things that you’re trying to essentially mediate and modulate someone’s consciousness to convey what you’re trying to say. And I think the thing that’s interesting for me is that it’s going beyond what we’ve been able to do with written text or linear media. And there’s something about the immersive media that’s speaking to people’s deep unconscious and their deep emotions. And maybe it’s hitting people in a way that they can’t even fully articulate, but it’s just like evoking emotions that on the morning you wake to the inner world did that to me. Like I can’t fully explain why it was so emotional, as I was facing my mortality and empathizing with these people that went through this, it’s taking something that’s a big geopolitical issue, but putting it into a personal story. And it’s that relationship of that personal story that’s reflecting of the larger story. When you’re able to do that, you’re tapping into the universal archetypal dimensions of taking something that’s very personal, very phenomenological, but you’re able to tap into these vast, larger archetypes that people, when they see it, it just resonates with them. And I think amazing stories are the ones that are able to do that. taking something that’s very specific to one individual, but it transcends that individual and everybody can relate to it in some way.
Paul Taylor: Absolutely. Well, listen, I’m going to sit here and chat all day, I’m sure. And I’m extremely grateful for you setting up the frame for today and setting out a number of lenses that I hope I can borrow and attempt to look through as we interview the various different guests that we have. Could you please give a very large round of applause to Kent Bye?
Bye, K. (2022, May 5). A Primer on Presence, Immersive Storytelling, & Experiential Design. [Presentation] Storycon: A collision of (he)art and technology. Brussels, Belgium; BOZAR – Centre for Fine Arts. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Photo by Amir Khidojatov.↩︎
Pine II, B. J., & Gilmore, J. H. (1998). Welcome to the experience economy. Harvard Business Review. Retrieved December 10, 2021, from https://hbr.org/1998/07/welcome-to-the-experience-economy.↩︎
Pine II, B. J., & Gilmore, J. H. (1999). Experience economy: Work is theatre & every business a stage. Harvard Business School Press.↩︎
Michelli, J. A. (2007). The starbucks experience: 5 principles for turning ordinary into extraordinary. (pg. 22). McGraw-Hill.↩︎
Image via Starbucks (2012, March 5). Starbucks First European Concept Store to Open in Amsterdam. Starbucks. Retrieved on April 25, 2022 from https://stories.starbucks.com/stories/2012/starbucks-first-european-concept-store-to-open-in-amsterdam/.↩︎
Schmitt, B. (1999). Experiential marketing. Journal of marketing management, 15(1-3), 53-67. https://doi.org/10.1362/026725799784870496.↩︎
Brakus, J. J., Schmitt, B. H., & Zarantonello, L. (2009). Brand experience: what is it? How is it measured? Does it affect loyalty?. Journal of marketing, 73(3), 52-68. https://doi.org/10.1509/jmkg.73.3.052.↩︎
Bye, K., & Nightingale, C. (2018, January 10). #611: “Sleep No More” Creative Producer on Blurring the Lines of Reality with Punchdrunk’s Immersive Theater. Voices of VR Podcast. Retrieved November 15, 2021 from https://voicesofvr.com/611-sleep-no-more-creative-producer-on-blurring-the-lines-of-reality-with-punchdrunks-immersive-theater/.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Tricart, C. (2022, March 14). Immersive Storytelling. SXSW Film & Interactive Film Festival. [Keynote presentation]. Austin, TX; Austin Convention Center. Retrieved on April 1, 2022 from https://voicesofvr.com/1074-celine-tricarts-sxsw-keynote-on-immersive-storytelling-live-action-role-play-emotional-bleed/.↩︎
Milgram, P. and Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems, E77-D, 1321-1329. https://search.ieice.org/bin/summary.php?id=e77-d_12_1321.↩︎
Modified with Clay Bavor (2017, May 18). Google IO 2017 Day 2 Keynote, https://twitter.com/kentbye/status/865244388057694208.↩︎
Microsoft (2020, August 26). What is mixed reality? Retrieved April 02, 2021, from https://docs.microsoft.com/en-us/windows/mixed-reality/discover/mixed-reality.↩︎
Variety. (2014, February 6). Oculus VR’s Palmer Luckey – “Virtual Reality: The Road Ahead” – D.I.C.E. 2014 Summit [Presented on February 6, 2014 at D.I.C.E. 2014 Summit at the Hard Rock Hotel, Las Vegas, NV]. YouTube. Retrieved April 21, 2022, from https://www.youtube.com/watch?v=cs58OrTG6LU.↩︎
Meta Quest. (2014, October 1). Oculus Connect Keynote: Brendan Iribe and Nate Mitchell. [Presented on September 20, 2014 at Oculus Connect, Loews Hollywood Hotel, Los Angeles, CA]. YouTube. Retrieved April 21, 2022, from https://www.youtube.com/watch?v=1xeK8zUXAvQ.↩︎
Slater, M. (2009, Dec 14). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557. https://doi.org/10.1098/rstb.2009.0138.↩︎
Images via Toast (2019, July 21). Richie’s Plank Experience Oculus Quest Trailer. YouTube Retrieved on April 25, 2022 from https://www.youtube.com/watch?v=faNsP7ExSt0.↩︎
Chalmers, D. J. (2022). Reality+: Virtual Worlds and the Problem of Philosophy. (pg xvii and 205-206). Penguin Books.↩︎
Bystrom, K.-E., Barfield, W., & Hendrix, C. (1999). A Conceptual Model of the Sense of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 8(2), 241–244. https://doi.org/10.1162/105474699566107.↩︎
Meta Quest. (2014, October 1). Oculus Connect Keynote: Brendan Iribe and Nate Mitchell. [Presented on September 20, 2014 at Oculus Connect, Loews Hollywood Hotel, Los Angeles, CA]. YouTube. Retrieved April 21, 2022, from https://www.youtube.com/watch?v=1xeK8zUXAvQ.↩︎
Stefanics, G., Kremlácek, J., & Czigler, I. (2014). Visual mismatch negativity: a predictive coding view. Frontiers in Human Neuroscience, 8. https://doi.org/10.3389/fnhum.2014.00666 Retrieved Feb 20, 2020 from: https://www.researchgate.net/figure/Simplified-scheme-of-the-hierarchical-predictive-coding-framework-Friston-2005-2008_fig1_266401430.↩︎
Millidge, B., Seth, A., & Buckley, C. L. (2022, January 22). Predictive coding: a theoretical and experimental review. arXiv preprint arXiv:2107.12979. Retrieved on April 23, 2022 from https://arxiv.org/abs/2107.12979.↩︎
Mendonça, D., Curado, M., & Gouveia, S. S. (Eds.). (2021). Philosophy and Science of Predictive Processing. Bloomsbury Academic.↩︎
Klein, T. J., & Lewis, M. A. (2012). A physical model of sensorimotor interactions during locomotion. Journal of Neural Engineering, 9(4), 046011. https://doi.org/10.1088/1741-2560/9/4/046011.↩︎
Steinicke, F. (2021, March 31). B(l)ending Realities. Keynote lecture presented at The IEEE Conference on Virtual Reality and 3D User Interfaces in Spain (virtual), Libson. Retrieved from https://www.youtube.com/watch?v=Ds-h1J4MFMI&t=13496s.↩︎
Bystrom, K.-E., Barfield, W., & Hendrix, C. (1999). A Conceptual Model of the Sense of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 8(2), 241–244. https://doi.org/10.1162/105474699566107.↩︎
Skarbez, R., Brooks, Jr., F. P., & Whitton, M. C. (2017). A Survey of Presence and Related Concepts. ACM Computing Surveys, 50(6), 1–39. https://doi.org/10.1145/3134301.↩︎
Bye, K. & Brillhart, J. (2017, February 8). #502: An Elemental Theory of Presence + Future of AI & Interactive Storytelling. Voices of VR Podcast. Retrieved on April 22, 2022 from https://voicesofvr.com/502-an-elemental-theory-of-presence-future-of-ai-interactive-storytelling/.↩︎
Bye, K. & Chertoff, D. (2017, July 19). #557: Using Experiential Design to Expand VR Presence Theory. Voices of VR Podcast. Retrieved on April 22, 2022 from https://voicesofvr.com/557-using-experiential-design-to-expand-vr-presence-theory/.↩︎
Bye, K. (Recorded 2017, March 30, Published on 2017, March 31). Historical Context of Virtual Reality. Silicon Valley Virtual Reality Conference & Expo. [Keynote presentation]. San Jose, CA; San Jose Convention Center.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Jung, C. G. (1968). The Psychology of the Child Archetype. In R. F. C. Hull (Trans.), The Collected Works of C. G Jung | Complete Digital Edition (Vol. 9: Part 4 of The Archetypes and the Collective Unconscious, 2nd Edition, pg. 179, paragraph 301). Princeton, NJ: Princeton University Press. (Original work published 1940).↩︎
Jung, C. G. (1968). The Psychology of the Child Archetype. In R. F. C. Hull (Trans.), The Collected Works of C. G Jung | Complete Digital Edition (Vol. 9: Part 4 of The Archetypes and the Collective Unconscious, 2nd Edition, pg. 179, paragraph 301). Princeton, NJ: Princeton University Press. (Original work published 1940).↩︎
Bloch, D., & George, D. (1987). Astrology for yourself: A workbook for personal transformation. (pp. 9-10). *Wingbow Press.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Skarbez, R. (2016, December 4). Thank you! From a Voices of VR alum. Personal Communication to Bye, K.↩︎
Skarbez, R., Brooks, Jr., F. P., & Whitton, M. C. (2017). A Survey of Presence and Related Concepts. ACM Computing Surveys, 50(6), 1–39. https://doi.org/10.1145/3134301.↩︎
Chertoff, D. B., Schatz, S. L., McDaniel, R., & Bowers, C. A. (2008). Improving Presence Theory Through Experiential Design. Presence: Teleoperators and Virtual Environments, 17(4), 405–413. https://doi.org/10.1162/pres.17.4.405.↩︎
Chertoff, Dustin, “Exploring Additional Factors Of Presence” (2009). Electronic Theses and Dissertations. 3910. https://stars.library.ucf.edu/etd/3910.↩︎
Chertoff, Dustin, “Exploring Additional Factors Of Presence” (2009). Electronic Theses and Dissertations. 3910. https://stars.library.ucf.edu/etd/3910.↩︎
Chertoff, D. B., Goldiez, B., & LaViola, J. J. (2010). Virtual Experience Test: A virtual environment evaluation questionnaire. 2010 IEEE Virtual Reality Conference (VR). *https://doi.org/10.1109/vr.2010.5444804.↩︎
Chertoff, D. B., Goldiez, B., & LaViola, J. J. (2010). Virtual Experience Test: A virtual environment evaluation questionnaire. 2010 IEEE Virtual Reality Conference (VR). *https://doi.org/10.1109/vr.2010.5444804.↩︎
Schmitt, B. (1999). Experiential marketing. Journal of marketing management, 15(1-3), 53-67. https://doi.org/10.1362/026725799784870496.↩︎
Schmitt, B. (1999). Experiential marketing. Journal of marketing management, 15(1-3), 53-67. https://doi.org/10.1362/026725799784870496.↩︎
Schmitt, B. (1999). Experiential marketing. Journal of marketing management, 15(1-3), 53-67. https://doi.org/10.1362/026725799784870496.↩︎
Brakus, J. J., Schmitt, B. H., & Zarantonello, L. (2009). Brand experience: what is it? How is it measured? Does it affect loyalty?. Journal of marketing, 73(3), 52-68. https://doi.org/10.1509/jmkg.73.3.052.↩︎
Brakus, J. J., Schmitt, B. H., & Zarantonello, L. (2009). Brand experience: what is it? How is it measured? Does it affect loyalty?. Journal of marketing, 73(3), 52-68. https://doi.org/10.1509/jmkg.73.3.052.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Empedocles, & Wright, M. R. (1981). Empedocles: The extant fragments. (pp. 26-27). *Yale University Press.↩︎
Printed image from Leibniz, Gottfried Wilhelm, (1690). Ars Combinatoria.↩︎
Printed image from Leibniz, Gottfried Wilhelm, (1690). Ars Combinatoria.↩︎
Empedocles, & Wright, M. R. (1981). Empedocles: The extant fragments. (pp. 26-27). *Yale University Press.↩︎
Solis, Virgil (1530-1562). Sanguine Temperament, From the Four Temperaments (Plate 1), Choleric… (Plate 2), Phlegmatic… (Plate 3), Melancholic (Plate 4). Minneapolis Institute of Art. Retrieved on April 26, 2022 from https://collections.artsmia.org/art/121403/sanguine-temperament-virgil-solis & https://collections.artsmia.org/search/virgil%20temperament.↩︎
Walkington, Thomas (1607). Optick Glasse of Humor. Retrieved on April 26, 2022 from https://becker.wustl.edu/news/humoralism-and-the-seasons/.↩︎
Jung, C. G. (1976). Psychological types. In R. F. C. Hull (Trans.), The collected works of C. G Jung | Complete Digital Edition (Vol. 6). (paragraphs [7] and [883]). Princeton, NJ: Princeton University Press. (Original work published 1921).↩︎
Jung, C. G. (1976). Psychological types. In R. F. C. Hull (Trans.), The collected works of C. G Jung | Complete Digital Edition (Vol. 6). (paragraphs [7] and [883]). Princeton, NJ: Princeton University Press. (Original work published 1921).↩︎
BYJU’S. (2020, March 20). Differences between sympathetic and parasympathetic nervous system. BYJUS. Retrieved June 25, 2022, from https://byjus.com/biology/difference-between-sympathetic-and-parasympathetic/.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Lundy, M., Sutton, D., Ashton, A., Martineau, J. (2010). Quadrivium: The Four Classical Liberal Arts of Number, Geometry, Music, & Cosmology. Walker & Co.↩︎
Bromm, V. (Fall 2006). “The History and Philosophy of Astronomy.” [Presentation]. The University of Texas at Austin. https://www.as.utexas.edu/astronomy/education/fall06/bromm/secure/lecture7.pdf.↩︎
Bernal, G. (2021, April 15). Developing Galea: An open source tool at the intersection of VR and neuroscience. MIT Media Lab. https://www.media.mit.edu/posts/galea/.↩︎
Bye, K. (2021, May 30). Twitter. https://twitter.com/kentbye/status/1399143885427728384.↩︎
Bye, K. (2021, June 2). “State of Privacy in XR & Neuro-Tech: Conceptual Frames” [Presentation] VRARA Global Summit. https://www.youtube.com/watch?v=pIpD4-gYImU.↩︎
Yuste, R.,Genser, J. & Herrmann, S. (2021). “It’s Time for Neuro-Rights.” Horizons: Journal of International Relations and Sustainable Development, no. 18, pp 154-164. JSTOR, https://www.cirsd.org/en/horizons/horizons-winter-2021-issue-no-18/its-time-for-neuro–rights. Accessed 31 Mar. 2021.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Skarbez, R., Brooks, Jr., F. P., & Whitton, M. C. (2017). A Survey of Presence and Related Concepts. ACM Computing Surveys, 50(6), 1–39. https://doi.org/10.1145/3134301.↩︎
Brooks, F. P. (1988). Grasping reality through illusion — Interactive graphics serving science. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems – CHI ’88. https://doi.org/10.1145/57167.57168.↩︎
Sutherland, I. E. (1965). The Ultimate Display. [invited lecture], IFIP Congress 65. An abstract appears in Information Processing 1965:Proceedings of IFIP [International Federation for Information Processing] 65, vol. 2, pp. 506-508. W.A. Kalenich, ed., Spartan Books, Washington, D.C., and Macmillan, New York.↩︎
Bye, K., & Brooks., F. P. (2016, May 10). #359: Fred Brooks on Sutherland’s 1965 Ultimate Display Speech. Voices of VR Podcast. Retrieved on April 23, 2022 from https://voicesofvr.com/359-fred-brooks-on-sutherlands-1965-ultimate-display-speech/.↩︎
Sutherland, I. E. (1968). A head-mounted three dimensional display. Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I on – AFIPS ’68 (Fall, Part I). *https://doi.org/10.1145/1476589.1476686.↩︎
Minsky, M. (1980). Telepresence. Omni, 45-51.Retrieved on April 23, 2022 from http://www.housevampyr.com/training/library/books/omni/OMNI_1980_06.pdf.↩︎
David L. Akin, Marvin L. Minsky, Eric D. Thiel, and Clifford R. Kurtzman. 1983. Space Applications of Automation, Robotics, and Machine Intelligence Systems (ARAMIS), Phase II, Vol. 1: Telepresence Technology Base Development. Technical Report. Massachusetts Institute of Technology, Cambridge, MA. (No. NASA-CR 3734-v.1 c.1). Retrieved on April 21, 2022 from https://ntrs.nasa.gov/api/citations/19840002515/downloads/19840002515.pdf.↩︎
Ellis, S. R. (1991). *Nature and origins of virtual environments: a bibliographical essay. Computing Systems in Engineering, 2(4), 321–347. https://doi.org/10.1016/0956-0521(91)90001-l.↩︎
Fisher, S. S., McGreevy, M., Humphries, J., & Robinett, W. (1987). Virtual environment display system. Proceedings of the ACM 1986 Workshop on 3D Interactive Graphics – SI3D ’86., Chapel Hill, North Carolina, October 23-24, 1986. https://doi.org/10.1145/319120.319127.↩︎
- [unnamed editors] (1992). Welcome to the new journal! Presence: Teleoperators and Virtual Environments, 1(1), iii. https://doi.org/10.1162/pres.1992.1.1.iii.↩︎
Sheridan, T. B. (1992). Musings on Telepresence and Virtual Presence. Presence: Teleoperators and Virtual Environments, 1(1), 120–126. https://doi.org/10.1162/pres.1992.1.1.120.↩︎
Slater, M. (2009, Dec 14). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557. https://doi.org/10.1098/rstb.2009.0138.↩︎
Slater, M., Linakis, V., Usoh, M., & Kooper, R. (1996). Immersion, presence and performance in virtual environments. Proceedings of the ACM Symposium on Virtual Reality Software and Technology – VRST ’96. https://doi.org/10.1145/3304181.3304216.↩︎
Brown, E., & Cairns, P. (2004). A grounded investigation of game immersion. Extended Abstracts of the 2004 Conference on Human Factors and Computing Systems – CHI ’04. https://doi.org/10.1145/985921.986048.↩︎
Haywood, N., Cairns, P. (2006). Engagement with an Interactive Museum Exhibit. In: McEwan, T., Gulliksen, J., Benyon, D. (eds). People and Computers XIX — The Bigger Picture. Springer, London. https://doi.org/10.1007/1-84628-249-7_8.↩︎
Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., & Walton, A. (2008). Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies, 66(9), 641–661. https://doi.org/10.1016/j.ijhcs.2008.04.004.↩︎
Sheridan, T. B. (1992). Musings on Telepresence and Virtual Presence. Presence: Teleoperators and Virtual Environments, 1(1), 120–126. https://doi.org/10.1162/pres.1992.1.1.120.↩︎
Heeter, C. (1992). Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments, 1(2), 262–271. https://doi.org/10.1162/pres.1992.1.2.262.↩︎
Witmer, B. G., & Singer, M. J. (1998). *Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), 225-240. https://doi.org/10.1162/105474698565686.↩︎
Skarbez, R., Brooks, Jr., F. P., & Whitton, M. C. (2017). A Survey of Presence and Related Concepts. ACM Computing Surveys, 50(6), 1–39. https://doi.org/10.1145/3134301.↩︎
Chalmers, D. J. (2022). Reality+: Virtual Worlds and the Problem of Philosophy. (pg xvii and 205-206). Penguin Books.↩︎
Oh, C. S., Bailenson, J. N., & Welch, G. F. (2018). A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI, 114. https://doi.org/10.3389/frobt.2018.00114.↩︎
Bye, K. & Darnell, E. (2016, February 2). #290: Storytelling in VR & the Tradeoffs of Empathy and Interactivity. Voices of VR Podcast. Retrieved on April 22, 2022 from https://voicesofvr.com/290-storytelling-in-vr-the-tradeoffs-of-empathy-and-interactivity/.↩︎
Kelso, M. T., Weyhrauch, P., & Bates, J. (1993). Dramatic Presence. Presence: Teleoperators and Virtual Environments, 2(1), 1–15. https://doi.org/10.1162/pres.1993.2.1.1.↩︎
Green, M. C., Brock, T. C., & Kaufman, G. F. (2004). Understanding Media Enjoyment: The Role of Transportation Into Narrative Worlds. Communication Theory, 14(4), 311–327. https://doi.org/10.1111/j.1468-2885.2004.tb00317.x.↩︎
Skarbez, R., Brooks, Jr., F. P., & Whitton, M. C. (2017). A Survey of Presence and Related Concepts. ACM Computing Surveys, 50(6), 1–39. https://doi.org/10.1145/3134301.↩︎
Bye, K. (2021, June 2). “State of Privacy in XR & Neuro-Tech: Conceptual Frames” [Presentation] VRARA Global Summit. https://www.youtube.com/watch?v=pIpD4-gYImU.↩︎
McKee, R. (1997). Story: Substance, structure, style, and principles of screenwriting. (pp. 101). ReganBooks.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K., Cojocaru, A. I. & Hellberg, F. V. (2020, May 2). Half Life Alyx’s Immersive Architecture & Experiential Design (3 presentations + discussion). [Presentation]. YouTube. Retrieved on May 4, 2024 from https://www.youtube.com/watch?v=wnnGjoZ6Qs.↩︎
Bye, K., Cojocaru, A. I. & Hellberg, F. V. (2020, May 2). Half Life Alyx’s Immersive Architecture & Experiential Design (3 presentations + discussion). [Presentation]. YouTube. Retrieved on May 4, 2024 from https://www.youtube.com/watch?v=wnnGjoZ6Qs.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Bye, K. (2024, May 13). Elemental Theory of Presence + Primer on Experiential Design & Immersive Storytelling. Voices of VR Podcast. https://voicesofvr.com/1389-elemental-theory-of-presence.↩︎
Tricart, C. (2022, March 14). Immersive Storytelling. SXSW Film & Interactive Film Festival. [Keynote presentation]. Austin, TX; Austin Convention Center. Retrieved on April 1, 2022 from https://voicesofvr.com/1074-celine-tricarts-sxsw-keynote-on-immersive-storytelling-live-action-role-play-emotional-bleed/.↩︎
McDowell, A. (March 2019). Storytelling Shapes the Future. Journal of Futures Studies. 23(3): 105–112. https://doi.org/10.6531/JFS.201903_23(3).0009.↩︎
Lebowitz, J., Klug, C. (2011). “Interactive Storytelling for Video Games: A Player-Centered Approach to Creating Memorable Characters and Stories.” (pp. 120). *Focal Press>.↩︎
Smale, E. D. (2009, December 23). Three-Act Adventures. The Welsh Piper. Retrieved on November 13, 2021 from https://welshpiper.com/three-act-adventures/.↩︎
Riedl, Mark. (2009). “Incorporating Authorial Intent into Generative Narrative Systems,” Association for the Advancement of Artificial Intelligence. https://cdn.aaai.org/Symposia/Spring/2009/SS-09-06/SS09-06-015.pdf.↩︎
Laskow, S. (2017, June 13). These maps reveal the hidden structures of ‘Choose your own adventure’ books. Atlas Obscura. Retrieved November 14, 2021, from https://www.atlasobscura.com/articles/cyoa-choose-your-own-adventure-maps.↩︎
Koster, R. (2013, April 24). On choice architectures. Raph’s Website. Retrieved November 13, 2021, from https://www.raphkoster.com/2013/04/24/on-choice-architectures/#more-11622.↩︎
Mateas, Michael. (December 2002). “Interactive Drama, Art, and Artificial Intelligence.” PhD Thesis. Technical Report CMU-CS-02-206. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.↩︎
Short, E. (2017, February 28). Spirit AI presented at “Narrative Innovation Showcase”, Game Developers Conference, San Francisco, CA.↩︎
Abouhaib, A. (2014). Retrieved on November 13, 2021 from http://ffden-2.phys.uaf.edu/webproj/212_spring_2014/Abdel_Abouhaib/101559128653564b3256663/uploads/2/8/6/1/28617611/1047737_orig.jpg.↩︎
Kumar, G., & Singh, K. (November – December 2013). Enhanced Learning and Retention of Course Content among Global Education using Multimedia Aids. International Journal of Emerging Trends & Technology in Computer Science. Vol 2, Issue 6, Retrieved on November 13, 2021 from https://www.ijettcs.org/Volume2Issue6/IJETTCS-2013-12-25-097.pdf.↩︎
Kumar, G., & Singh, K. (November – December 2013). Enhanced Learning and Retention of Course Content among Global Education using Multimedia Aids. International Journal of Emerging Trends & Technology in Computer Science. Vol 2, Issue 6, Retrieved on November 13, 2021 from https://www.ijettcs.org/Volume2Issue6/IJETTCS-2013-12-25-097.pdf.↩︎
Microsoft (2020, August 26). What is mixed reality? Retrieved April 02, 2021, from https://docs.microsoft.com/en-us/windows/mixed-reality/discover/mixed-reality.↩︎
McDowell, A. (March 2019). Storytelling Shapes the Future. Journal of Futures Studies. 23(3): 105–112. https://doi.org/10.6531/JFS.201903_23(3).0009.↩︎
Whearley, N. (2020, December 11). *Teach a class. Retrieved March 03, 2021, from https://web.archive.org/web/20201101172454/https://dschool.stanford.edu/teaching-learning/teach-a-class.↩︎
Lessig, Lawrence, (2006). Code 2.0, Chapter: What Things Regulate ( pp. 120–137). New York : Basic Books.↩︎
Bye, K. (2016, April 28). The Human Experience of Virtual Reality: A Model of the VR Landscape [Presentation], Silicon Valley Virtual Reality Conference, San Jose, CA; San Jose Convention Center. Retrieved on November 10, 2021 from https://voicesofvr.com/355-the-human-experience-of-virtual-reality-a-model-of-the-vr-landscape/.↩︎
Bye, K. (2024). [Forthcoming] Privacy Pitfalls of Contextually-Aware AI: Sensemaking Frameworks for Context and XR Data Qualities. [Unnamed Proceedings of the Stanford Cyber Policy Center’s Existing Law and Extended Reality Symposium 2023].↩︎
Bye, K. (2019, Oct 18). XR Ethics Manifesto. [Presentation] Greenlight’s XR Strategy Conf. San Francisco, CA; Microsoft Reactor. https://voicesofvr.com/844-xr-ethics-an-xr-ethics-manifesto/.↩︎
Diller, A., Ayim, M., Pauly Morgan, K., & Houston, B. (1996). Figure 8.1 (p. 107). The Gender Question in Education: Theory, Pedagogy, & Politics. Routledge, Taylor et Francis Group>.↩︎
Bye, K., & Dolan, D. (2016, February 4). #292: The Four Different Types of Stories in VR. Voices of VR Podcast. Retrieved November 14, 2021 from https://voicesofvr.com/292-the-four-different-types-of-stories-in-vr/.↩︎
Passi, E. (2017, January 10). *Bringing the Westworld attribute matrix to life. Medium. Retrieved November 14, 2021, from https://epassi.medium.com/recreating-the-westworld-attribute-matrix-3e72d9d419df.↩︎
Lanctot, J. D., Irving, J. A. (2010). Character and Leadership: Situating Servant Leadership in a Proposed Virtues Framework. International Journal of Leadership Studies, Vol. 6 Iss. 1, 2010. Retrieved November 14, 2021 from https://www.regent.edu/acad/global/publications/ijls/new/vol6iss1/2_Final%20Edited%20Lanctot%20and%20Irving_pp%2028-50.pdf via https://www.regent.edu/acad/global/publications/ijls/new/vol6iss1/home.htm.↩︎
Barrick, M. R., & Mount, M. K. (1991). The Big Five Personality Dimensions and Job Performance: a Meta-analysis. Personnel Psychology, 44(1), 1–26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x. Image via https://commons.wikimedia.org/wiki/File:Wiki-grafik_peats-de_big_five_ENG.png.↩︎
Sacharin, V., Schlegel, K., & Scherer, K. R. (2012). Geneva Emotion Wheel rating study (Report). Geneva, Switzerland: University of Geneva, Swiss Center for Affective Sciences. Retrieved on November 13, 2021 from https://www.unige.ch/cisa/files/4514/6720/4016/Geneva_Emotion_Wheel_Rating_Study_Report_2012_08_11_2.0.pdf.↩︎
Tschannen-Moran, B. (2012, February). Appreciative Empathy: New Frameworks for New Conversations. International Journal of Appreciative Inquiry: AI Practitioner. Volume 14 Number 1. Retrieved November 14, 2021 from http://www.schooltransformation.com/wp-content/uploads/2012/06/Appreciative_Empathy.pdf.↩︎
Photo by Amir Khidojatov.↩︎
Photo by Amir Khidojatov.↩︎
Podcast: Play in new window | Download
