Dustin Chertoff has pulled experiential design insights from the advertising world to come up with a more holistic theory of presence in virtual reality. In 2008, he was in graduate school and was dissatisfied with the major theories of VR presence. His gaming experience showed him how much of his feeling of immersion was related to the content of the game. He wrote an essay published in the journal Presence where he laid out what he saw were the two major limitations of VR presence theory at that time, “First, many models tend to focus heavily on perceptual issues while focusing less attention other facets of virtual experiences, such as cognition and emotion.” Second, “these models fail to provide an interpretable, extensible framework with which to understand and apply the theoretical principles to practical applications.”
Chertoff finished his Ph.D. thesis titled “Exploring Additional Factors of Presence” in 2009, and then published his Virtual Experience Test questionnaire in the 2010 proceedings of the IEEE Virtual Reality Conference. Presence researcher Richard Skarbez first alerted me to Chertoff’s work after I asked him if he’d seen any prior research into presence looking at the different dimensions of my elemental theory of presence, which breaks down the subjective quality of an experience of VR into different combinations of Embodied Presence, Emotional Presence, Active Presence, and Social & Mental Presence. I was encouraged to see that Chertoff had independently come to an identical framework through his survey that was designed to holistically “measure virtual environment experiences based upon the five dimensions of experiential design: sensory, affective, active, relational, and cognitive.”
I had a chance to catch up with Chertoff in San Francisco during GDC this year, and we each concluded that our experiential design frameworks are functionally equivalent. We talked about his FearlessVR company that he co-founded where they design VR exposure therapy experiences for different phobias, but we spend the bulk of our discussion exploring how he came to looking to looking at the field of experiential design to inform presence theory. We also compare and contrast how each of our experiential design frameworks create tradeoffs and amplify different qualitative dimensions of an experience.
LISTEN TO THE VOICES OF VR PODCAST
Chertoff’s Ph.D. thesis “Exploring Additional Factors of Presence” has a great survey of presence research (see his summary graphic down below), as well as inspiration he’s drawn from flow theory and video game design frameworks like GameFlow. He summarizes Forlizzi and Battarbee’s definition of “experience” by saying it’s “something that can be articulated, named, and schematized within a person’s memory. Experiences of this type have beginnings and ends, but anticipation of, and reflection on, the experience may take place before or after the event.”
The idea of experiential design is that deeply immersive experiences form stronger memories, and that it’s easier for us to store new information along when we’ve had related experiences. Chertoff says, “Experiential designs are successful when they encourage people to create meaningful emotional and social connections—personal narratives that involve episodic memories, and positive associations with the artifacts of that experience.”
Chertoff cites Joseph Pine & James Gilmore’s book The Experience Economy as his source of inspiration for the five dimensions of his experiential design framework that make for a memorable and immersive experience, which map nicely over to my elemental theory of presence.
Embodied Presence corresponds to Chertoff’s sensory dimension which he says “includes all sensory input—visual, aural, haptic, and so forth — as well as perception of those stimuli.” I’d also include different virtual body representations, as well as the virtual environmental components which help transport you into another world and trick your perceptual system into believing that you’re in another world. Most presence researchers research embodied presence as virtual reality uniquely stimulates our sensorimotor aspects of perception beyond what any other communications medium can do.
Emotional Presence corresponds to the affective dimension which “refers to a participant’s emotional state. For simulation, this dimension can be linked to the degree to which a person’s emotions in the simulated environment accurately mimic his or her emotional state in the same real-world situation.” For me, this includes the storytelling and narrative components as well as music, colors, symbols, and deeper myths that all engage the emotions. Emotional presence can also come into the experience through social engagement with other people.
Mental Presence corresponds to the cognitive dimension, which “encompasses all mental engagement with an experience, such as anticipating outcomes and solving mysteries. For simulation, much of the cognitive dimension can be interpreted as task engagement, [which] is related to the intrinsic motivation, meaningfulness, and continuity (actions yielding expected responses) of an activity.” Game designers often talk about mental friction, and if there isn’t something in an experience to stimulate your mind then you’ll risk getting bored.
Social Presence corresponds to the relational dimension, which is “composed of the social aspects of an experience. For simulation, this can be operationalized as co-experience — creating and reinforcing meaning through collaborative experiences… Experiences that are created or reinforced socially are usually stronger than individual experiences and they further enable individuals to develop personal and memorable narratives.” I combine mental & social presence into the air element, because they both deal with the abstractions of thought and communication. But it also emphasizes the fact that not every experience has to have a social dimension to it, and that solitary experiences can be just as immersive and engaging to your mind.
Finally, Active Presence corresponds to the active dimension, which Chertoff described in the interview the degree to which you can express your agency, and physical engagement through taking action within the experience. He also sees it as a form of subjective engagement by saying “Does he or she incorporate the experience into his or her personal narrative; does he or she form meaningful associations via the experience?”
Chertoff assigns a few things to the active dimension that I would categorize elsewhere. For example, I think empathy is more of a function of emotional engagement, and that connection to the environment, avatars, and identity are more related to embodied presence. I tend to think of active presence being primarily as an expression of agency and will that includes exploration, curiosity, creativity, physical or virtual locomotion, and any type of interactivity.
There are qualitative dimensions of an experience that are sometimes hard to clearly schematize into a single category, and I believe that all of these different dimensions are happening at the same time all the time. But I do see that there are tradeoffs between active presence and emotional presence that I explore in much more detail in this introductory essay about elemental theory of presence..
A Survey of Presence Research
Chertoff has a lot of other references to presences research, the evolution of experience economies, insights from the user-system-experience (USE) model of user-centered design, as well as combining game design theory with flow theory with GameFlow.
Here’s a summary graphic of Chertoff’s survey of presence research as of 2009, where he summarizes the major components of presence in the different major models:
Chertoff argues that a lot of these models don’t have a good conceptual framework that can account for role of content in cultivating a sense of immersion and presence, and he sees that Experiential Design and Flow Theory can fill in those gaps.
Flow theory shows the relationship between the objective skills of a user and the objective challenges within an experience that result in a variety of different internal subjective states in the user.
When the skill and challenge is low, then that can lead to apathy or boredom. When the challenges are high, but the skill is low, then it can create anxiety or arousal. Flow states happen in a sweet spot of high skill of the user and high challenge within the experience.
Flow theory connects the objective aspects of an experience to the internal states of a subject, and it’s also helping to evolve existing user-centered design models. Most user-centered design research has focused on the usability of an interface with a specific productive output, but for gaming the experience of playing the game being the reward itself and so it’s more about the playability of the game and whether or not it helps the player achieve a flow state. Flow theory has been applied to Game Design in the GameFlow theory, which shows the role of content in creating a sense of immersion and presence.
Anyone interested in different frameworks and models of experience will find a wealth of references in Chertoff’s three papers: Improving Presence Theory Through Experiential Design (2008), “Exploring Additional Factors of Presence (2009), and Virtual Experience Test: A virtual environment evaluation questionnaire (2010).
Moving to An Experience Economy
Finally, I think it’s important these two experiential design frameworks are independent of virtual reality, and they can be applied to creating any human experience. Pine & Gilmore wrote a prescient article “Welcome to the Experience Economy” in the Harvard Business Review in 1998, and they say, “We expect that experience design will become as much a business art as product design and process design are today. Indeed, design principles are already apparent from the practices of and results obtained by companies that have (or nearly have) advanced into the experience economy.”
I say we're moving from the Information Age to the #ExperientialAge.
But what's that mean?
Here are some metaphors & emergent properties: pic.twitter.com/AnDSXLu7wB— Kent Bye (Voices of VR) (@kentbye) June 6, 2017
I’ve talked before about how Snapchat shows how we’re moving from the Information Age to the Experiential Age, and this article from 1998 lays out some of the economic trends that led experiential design paradigms used by companies like Starbucks and Apple. Here is a graphic from Pine & Gilmore’s article that describes the fundamental characteristics of this new experience economy that is moving from delivering intangible & customized services on-demand to staging memorable & personal experiences that are revealed over a duration of time.
Virtual and augmented reality technologies are driving our culture towards having more and more immersive experiences, and learning how to holistically create immersive and memorable experiences is going to continue to become a vital part the future of our economy and our culture.
For a more in-depth discussion about my elemental theory of presence, then be sure to check out this interview with Jessica Brillhart, an early discussion with Alex Schwartz, my SVVR 2017 keynote talk, and this No Proscenium podcast interview with me talking about how it could be applied to immersive theater.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Tripg
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to The Voices of VR Podcast. So I'm continuing on the theme of presence research and coming up with different models and frameworks. And this interview with Dustin Chertoff has had a huge impact on me. just because the research that he's done in terms of going back and doing an overall survey of virtual reality presence research and applying principles of experiential design, he's independently replicated the same ideas and thoughts that I've come to, but he did it like eight years ago while he was in grad school. He was looking at presence theory and seeing that it was just really focused on the sensory motor contingencies, and he just thought, you know, there's something about the content that makes it so much more compelling He had a lot of experience playing video games and he knew that a good game goes way beyond just what it looks like in the graphics. It's a lot about the content and the immersion and gameplay and the mechanics and cultivating the sense of being able to achieve this state of flow or immersion or what within the virtual reality community we would call presence. So he wanted to apply the principles of experiential design to presence theory. And so he did this survey of all the different presence research and then came up with some sort of overarching qualitative framework to talk about the subjective experience of someone going through a virtual reality experience. So we compare and contrast our different experiential design frameworks and talk about how he's applying it within his company called Fearless, which is trying to train you to overcome your different fears. So that's what we'll be covering on today's episode of the Voices of VR podcast. So this interview with Dustin happened on Monday, February 27th, 2017 in San Francisco, California. So with that, let's go ahead and dive right in.
[00:01:56.074] Dustin Chertoff: My name is Dustin Chertoff. I'm the co-founder of FearlessVR. We're a startup based in San Francisco that is using the principles of exposure therapy to help people overcome their fears. We started with spiders since that's one of the most popular or I guess not popular creatures out there that people can be fearful of. And so we're out here to help people get better and improve their mental health and just lead better lives so they don't have to be afraid of the stuff that's out there because there's no reason to be.
[00:02:28.445] Kent Bye: Awesome. Yeah. Well, I just had a chance to try out the experience and I want to dive more into that my own experience of that and what you were doing here, but I wanted to talk a bit about the process of experiential design because I've been talking to a lot of different presence researchers and Explore the different theories that are out there I know that Mel Slater's breakdown of the place illusion and the plausibility illusion is probably one of the more popular ones and yet I As I have been going through for the past three years experiencing all the consumer VR I felt like you know that my levels of presence were a little bit more detailed than what that sort of generalized theory is really explaining and I feel like it still holds true but yet I I think there's some more nuances to it that I'm trying to flesh out. And it sounds like you've kind of been on a similar track of looking at this through the lens of experiential design, but also trying to come up with your own variations of presence theory. So maybe you could just walk me through your entry into looking at this question.
[00:03:27.495] Dustin Chertoff: Sure. So I guess like many people who end up making science fiction reality, it all started from reading books for me. I started reading Tad Williams' Otherland novels. This was back in like the late 90s, I guess. And he built this world that was all about going from one virtual world to another and how it was all connected. And what set it apart from other cyberspace novels, or even like The Matrix and other movies, was how embodied the people were inside of those worlds. So in the story, people actually got trapped inside the virtual world, inside these locomotion machines and everything. And that just started me thinking like, It's just really cool how you can be completely immersed in a world. The worlds can be fantastical to normal. And okay, I got to know how you actually go about making all these things a reality. And that basically ended up being what Presence is. It's how do you feel that you're actually there in the environment. And lots of research papers later and lots of playing video games and all that stuff, you kind of came to this conclusion that presence is a lot more than just, you know, how sensory immersive an environment can be. It's more than just the graphics. I mean, you hear people talk about this in computer games all the time. The graphics are great, but, you know, after I played it for two hours, why do I want to play this game again? There has to be something else that keeps you drawn into the world. And that ultimately is what experiential design is trying to bring into virtual reality and just the virtual environments in general as well. The sensory aspects are kind of what makes you want to take a peek, but there has to be other elements that's going to keep you there and keep you involved and keep you wanting to come back for more. Game developers have been doing this for years. Presence researchers have been studying certain aspects of it, but there was never any coherent theory to try to tie all of it together. So I kind of took the best stab at it. As a grad student, everyone thinks that whatever they're going to do, it's going to be the best thing that's going to unify everything. Years later, you realize, okay, well, it was one little step on getting to the right direction. And okay, how can I actually apply this stuff to more practical things? So experiential design, which is what I studied, it's actually this marketing concept. So if you think of Starbucks or the Apple Store, and what goes into creating the experience that surrounds the product itself, from your barista or the apple care specialist when you first walk in, that greeting that you get, the music that's playing, how everything is laid out. You're actually paying for more than just the consumable product, but it's everything that goes along with it. And this concept has five dimensions. There's the sensory dimension, which in presence is very well studied. Cognitive, which is kind of what you're thinking about, everything that's going on inside your head with regards to the tasks and things of that nature. There's this affective dimension, which is the emotional connection that you have, what you feel, how you feel to feel about everything going on around you. active, which is kind of your sense of agency in a way. If I do something in this world, is the world going to respond in a way that I would expect it to? And there's a relational component, which is, I don't do things in the real world on my own. There's other people around me. I may not always be interacting with them, but I know they're there. And when we have these shared experiences with other people, that reinforces that the experience itself is kind of more real for us as well. So I just got really interested in that and wanted to apply it towards how to design virtual reality experiences because I felt that if we have these more holistic experiences, we can really build more than just visually stunning things, but experiences that people want to come back to and will consistently feel that they are real.
[00:07:28.798] Kent Bye: Yeah, and to me, that is so exciting, because I've sort of come independently to the exact same model. And I would say instead of five, I would go down to four. And so I'm kind of coming at this from starting with Mel Slater's theory of presence, where he talks about the place illusion and the plausibility illusion, where you feel like you're in another place and the whole world feels like it makes sense. And so there's a objective and a subjective dimension of that, which is, you know, your internal state of your expectations of whether or not you feel like the world is matching your kind of predictions as to what you see is happening. And the more that the level of the world is coherent in that way, then the more immersed in that place that you can feel. That's the essence of what Richard Scarbez is talking about in the place illusion is that level of immersion, and then the coherence of that is sort of the plausibility of that. So that was sort of my starting point of looking at the presence theory. But then I started to realize that, okay, social presence, well, how do you, you know, that was the first thing that I felt like, okay, I actually had this interaction of social interactions with other people. And it felt like I was actually having these conversations and that these people were, you know, I was talking to them and it felt like it was, my memory of that was just like I had been sitting around the table and having that same conversation. So I started asking people, well, how does social presence fit into this, you know, Slater conceptualization of plausibility and place illusion? And then there was other dimensions like agency and being able to actually feel like, you know, from a job simulator, having that sense of hand presence and being able to actually control something within the experience and to do that in a natural, intuitive way. And then there is the embodiment, which is the virtual body ownership illusion, which, you know, has this process of, you know, actually feeling like you're in that experience. And then there's the emotional component that I saw. So I started with both the social and then the embodiment and then started to look to natural philosophy and looking at the four elements. So we have the earth, air, fire, and water. And so with fire, fire is all about agency and about exerting your will into an experience. And then you have the earth element, which is all about the body in natural philosophy. So it's that level of embodiment that you have in that experience, which in your conceptualization would be the sensory component. And then you have the water element, which is all about the emotions. And so that's the effective dimension in your model. And then I think the difference that I would say is that I would actually combine the social and mental presence, because the air element is all about abstract constructs and thoughts and ideas. And that can be within your mind, but it can also be through language and communication, which is how we relate to other people. And so my four elements, I have the fire element as active presence. The earth element is embodied presence. the water element is emotional presence, and then the air element is a social and mental presence that combines. So when I sent this to Richard Skarbez, he's like, you know, I had heard of this Chertoff paper, and he sent it to me. I was like, oh my god. And then I was like, okay, I'm gonna like go out with this elemental theory of presence, because I feel like it's kind of independently coming to the same conclusion.
[00:10:40.532] Dustin Chertoff: And whenever you have multiple people come to the same conclusions, that's usually a really good sign that the conclusions are sound. So that's good news for my research and good for you. I think we are onto something here as far as what really makes presence and where the potential I don't want to say boundaries, but distinctions between the elements that all contribute to some greater holistic experience. And ultimately, what I was always about in going about this research was a holistic experience, because we don't go into our daily lives just focused on, what do I see? It's, what do I see? How do I perceive it? How am I thinking about that? A lot of this happens automatically within our brains, just from our perceptual loops and everything. when we sit back and actually think about it, there's a lot going on that's not just the individual sensory component. And I mean, even in this room, there are other people around me. And so I'm trying to be more cognizant and aware of what they're doing to have an interview, but not interrupt them at the same time and make sure that what we're talking about is potentially interesting and relevant to what they might be working on. And those are things that when you consider experience as a whole, you take into account. So if you're just in a virtual world and you're looking at something that's beautiful, if there isn't anything else kind of going on at the same time there, you might get bored. You might wander off in your mind and experience essentially a break-in presence, which is another term that Mel Slater had coined. and we tried to look at as well in my graduate research as far as how can breaks in presence be mitigated through experiential design and focusing on more holistic experiences. Is there a way that, okay, when you hear some interruption, maybe that's going to break you from the task for a second, but this world looks so pretty, why would I attend to that at all? And just try to keep people at a high enough level of presence or holistic experience that they don't want to turn away.
[00:12:44.876] Kent Bye: so yeah and i think that you know i've been thinking a lot about this different dimensions of presence and you know how these concepts actually apply to all experience it doesn't necessarily get limited to vr and just thinking about how you have these different mediums that are out there and how like the film medium, for example, because you don't have a level of embodiment and you don't have a level of agency, then it tends to be a little bit more of a mental and emotional experience. And so a lot of the 360 degree films that you watch on a mobile VR, I think I would argue that mobile VR actually could be a better platform to be able to watch some of these 360 videos because you actually have a constrained active and embodied presence, which makes you focus on what is actually happening in that content. Now, that said, there's still dimensions of agency and fire presence, because you can choose where you're looking in a 360-degree experience. And I think all experiences have all four elements into it, just whether or not you're trying to minimize and optimize them. And then on the other end of the spectrum, you have the games. And so you have much more about the expression of agency and much more about character embodiment. And I think that virtual reality is kind of like in the middle between those two, trying to really combine the best parts of the storytelling from the film and the best part of the interactive agency from a game, and then you're able to create these experiences that if you're able to really optimize that embodied presence, active presence, social and mental presence, and emotional presence, then you have an experience that once you have that experience, it's almost like indistinguishable from whether or not that happened in reality or in a synthetic simulation. And so you store it as a learned experience that mimics direct experiences that you have. And so I think that's the goal. I'm trying to optimize all those different levels of those dimensions of presence in a virtualized experience. And then you're creating these virtual experiences for people that just feel really satisfying and just become a part of their, their being.
[00:14:44.612] Dustin Chertoff: And once someone creates that experience, which I think everyone is still kind of looking for at this point, there will be no going back because people will have experienced kind of the ultimate in VR and now why would you ever want to go and do something that is a lesser experience? Everyone else will have to bring up their game essentially to meet that point. One of the things about 360 videos and just having more passive experiences in a more immersive environment, there needs to be some element of you doing something in that world because there can be a lot going on around me in a 360 video. And that doesn't necessarily mean I'm going to be focusing on what the director wants me to look at. Because there could be someone skateboarding in front of me, but I see a dog barking in back of me. And okay, I like dogs more than I like skateboarders. even though the video itself could be all about skateboarders, I could be staring at the thing going on in back of me. And so now, how, as the 360 video director, are you going to get my attention back to where you want me to be? With 2D movies, you're always looking in front of you. So you can do things like, okay, well, is it in the center of the screen or is it slightly off screen and coming on? And there's different techniques for trying to capture focus. When you add the additional degree of freedom there, it becomes a way more complicated medium to direct. And I think there are a lot of people trying to figure this out, and I think there's a long way to go still before we've really solved it. But one thing that was really powerful to me, so the Oculus Quill Dear Angelica story. which was essentially a 360 novel that just expanded around you. And the way that it just kind of unfolded and how the lines themselves were drawn, the lines and the direction that everything was drawn in was constantly pulling your attention to what should now be in focus. And when lots of things came in all at once, it's like, all right, well, now look around. That was kind of the cue that this is the point where you need to be turning your head around, looking in every direction, trying to bring in as much information as possible, which was then followed by kind of a, now let's slow it down. Because we just let you take in a lot of information. There's a lot of stuff that you need to process now. We need to give your brain a second to figure out and encode all of this information, and then we'll ramp it back up again. And it was a really powerful moment. In addition to the story just being incredibly touching and tear-jerking, it was also just a great example of what storytelling in VR could be.
[00:17:25.380] Kent Bye: Yeah, yeah, I agree. And, you know, it was a very, you know, it was about grief and loss. And I think there was an emotional component there that I think that VR can start to give you that sense of embodied presence. And once you have that embodied presence that triggers your mirror neurons in a certain way, then you start to have a different emotional experience in that way. And so I think embodiment within VR gives you this deeper emotional connection to some stories that may, I think of it as, you know, when you have really hard news to tell somebody, you want to actually be in the same room with them at the same time. and that it's akin to that and the difference between film and VR is that there's some types of content that is just better suited to be talked about in VR, because it just cultivates that level of intimacy. But if we take a step back and look at experiential design, because I've been thinking about these four different elements, and the earth element I think is one of the stronger new elements that we have that people are playing with, this sense of having your body in the experience of what's it mean to have your hands and adding in haptics and different senses that you're really Feeling with your body and then I think there's a lot of agency as well and trying to balance different Storytelling elements of authored stories and allowing you to participate in the experience in a way that still maintains that possibility But I'm curious to hear more from you in this lens of experiential design to hear about you know These five different components that you see and how you kind of break that down and how that can be applied to VR
[00:18:49.690] Dustin Chertoff: Sure, so I think the simplest one, obviously, is the sensory dimension, since that's what you see. But sensory also includes haptics, and how are you using your hands in the environment? How is the environment showing other people's use of their hands? Where are their eyes looking? All those components, while they're kind of grounded in, let's say, the social or the active elements as well, how we perceive them is still sensory. So there's still a lot that goes on there. I mean our eyes are our primary intake into the world. So how things are visually presented is always going to be really important. But once you get beyond that initial sensory presentation, people still need something to do. We have lots of passive experiences in the movies and even reading. But we are still giving ourselves an activity in that we are focused on watching and processing either what we're seeing or what we're reading. So in the virtual worlds as well, we need tasks. We need engagement and involvement with what's going on. Entertainment, as far as games is concerned, that's really easy. That's kind of the whole point of why you're playing the game. So it's going to be your game mechanics and the interactions that you can have. how you go about having those interactions, that's all tied into that cognitive dimension. As far as affect, that's kind of just how a narrative unfolds. Whether that's an external narrative, someone wrote a story, you're hearing it from other characters, as you're interacting with the world, new elements of the story unfold and you are now going to have a reaction to it. Or if it's kind of a internal monologue of the actions that you took while you were playing a game. So if you think about people who play sports games, there's not much story to what's going on. You're playing the game. But the story gets created by your actions. It's emergent. So you have both of those directions that you could go in when you're trying to design a virtual experience and how you want to bring in a narrative. Is it something that I'm going to design for the person to take in? Or is it something where I'm going to let the person tell their own story through the actions that I allow them to take? I would actually say that really, really good experiences let the individual do both simultaneously. So there's some kind of guiding narrative that can be really powerful, but also there's enough choice within the world that you not only get agency, which is important for the active dimension, but that the choices that I make are going to change the story for me in some way. So how do you do that? You need to have the experience, let the person make choices. Those choices need to be meaningful in some way. It could be as simple as click forward to continue. And when they do that, the next chapter of the story unfolds. Or it could be, do I save the person from imminent disaster or not? What will happen if I don't? What will happen if I do? In either case, the choice that I made is going to affect the outcome. And the other dimensions are going to get impacted in some way from that choice. Agency is also just like, do I really believe that I am this person in this world? Hand presence is so important for this. Even for experiences that might not have you use your hands, it's still very powerful to just see them in the world. So if you think about, like, mobile VR experiences, where maybe there isn't a hand tracker yet, although we will have them pretty soon for gear, and Daydream has a simple hand tracker as well, you can still get some sense of, I have hands in the world, even if I'm not really doing anything with them. That element of looking down and saying, I do have some kind of a body, I'm not just a floating camera in space, can be really powerful. So I would always encourage that to exist within the world. Finding the right way to implement it and kind of couch that in the experience is going to be more challenging, but design always is in that regard. And the last one, the relational component, at the most simplistic way you could say, well, everything should be multiplayer. But that doesn't always make sense. Sometimes we have solitary experiences. So for example, Fearless, the application that I'm working on and co-founding the company for, we have users sitting at a desk looking at things that they could be afraid of to help them try to overcome their fear. there's a voice that they hear that's kind of guiding them through this experience. So while it's a potentially solitary experience, there's still someone else in the room with them even if they can't kind of see them. Which is important because even if you aren't with another virtual real human or some kind of artificial intelligent entity within the room, there's still a connection that there's another voice so somewhere there must be another person and I'm going to accept that I'm not doing this completely on my own.
[00:24:02.887] Kent Bye: Yeah, and that's part of the reason why, you know, when thinking about social presence, I, you know, looked at the air element and said, actually, it's both the cognitive and social. And I would argue that not every experience has to have that social dimension. And then you just think about it in terms of the air element, because it's, you know, it could be more about your mental process rather than, you know, the communication process. And a lot of the things that you're talking about are a lot of the exact same things I've been talking about on the podcast, as well as just really trying to synthesize all the different dimensions of these types of experiences. And for me, I feel like the one element that is probably the least well understood or explored is the emotional presence, the affective presence of really how to really engage the emotions. And I think that in terms of narrative and storytelling, that's probably usually the best way to do that while, you know, just the construct of telling a story, you start to really project yourself into that story. And then you start to project your own experiences into that story. And there's just something about the mechanism of storytelling that engages our emotions and empathy in a way.
[00:25:06.794] Dustin Chertoff: Storytelling is kind of how we've transferred knowledge from generation to generation. I mean, it's only within the last few hundred years that we have print really available for the masses to learn things. So before that, mythology. It was storytelling. It was encoding important lessons in something kind of fantastical that would allow people to get quickly engaged. And stories can be very memorable. So that allows us to then transfer that knowledge to other future generations. So I mean, it's, yeah, storytelling is very powerful and definitely is one of the first ways that you can introduce emotion and affect into an experience. But it doesn't have to stop there. I mean, just how we respond to each other in a virtual world can elicit a non-intended response from the designers. I mean, if you have a social experience, the act of two people communicating is going to potentially be very different than what a designer might have hoped for, which can lead to unfortunate things, which are, you know, unfortunate.
[00:26:15.750] Kent Bye: And I think another dimension of the affective and emotional presence is music and how music can really kind of speak to emotions in that way. And I've just noticed the experience Pearl, for example, is a beautiful experience that is based around a song. And essentially taking the giving tree and translating that into instead of the tree, it's your car. And you have this experience of seeing how a car has been relating to this father and daughter relationship. You know, the thing that I think really carried that experience was the music and the song and how much emotional information can be encoded within the music. And also just using colors and other different dimensions to invoke different feelings. And so you can do a lot of environmental design as well. But it feels like, you know, there's a lot of rich exploration here to just, you know, as you go into these big corporations to see how they've constructed these different experiences where they try to encompass all these different things into experiential design. And to me, the fact that it's come from more the advertising and marketing angle, but now there's a lot of, certainly a lot of components of advertising and virtual reality, but as people are trying to create just generalized experiences that you can start to break down all these different dimensions and If you look at the Oculus Rift, for example, there's a lot of abstracted expressions of agency, so there's buttons. And combining the embodied dimension of your body and being able to do natural intuitive movements with your body, but being able to have that translated into the experience with your agency, that combination, I think, is kind of the core and the essence of what can give you that extra sense of embodied presence. once you have tetherless VR and you're walking around like the experience of the void, for example, and using like redirected walking without a tether, that level of embodiment that happens and the haptics of being able to reach out and have this passive haptic feedback of walls, it just gives you that extra dimension. So yeah, for you, as you start to look at some of these insights from advertising and experiential design, what are some of the other components that you're really trying to extract and apply to VR?
[00:28:26.795] Dustin Chertoff: I really just try to come at things from kind of an open design perspective instead of me trying to force my own ideas into what the experience should be. I kind of just start with what's needed for this experience to feel right and then work backwards from that in a way. So if I want someone to feel a sense of fear because there is a spider in front of them, and it's important that they face that fear and feel a sense of it so that they can learn that they can overcome that fear, then how will I go about doing that? Okay, well, the spider itself is an important first element. how the spider moves. It needs to be correct so that it's going to trigger the, hey, that's what a spider would really do kind of emotion. Have the spider come near them. Now you're using proximity to increase the emotional stress of the person so that they can try to stay calm and learn that they can actually be in the presence of a spider next to them and they did okay. They faced it. They can overcome this. So, really, it's in many ways user-centered design and these other design principles that have really come into the forefront these days. So, take my user, what do we want them to experience and start plugging in those things and then test it constantly to make sure that the experience that we want people to have is what they're actually getting. And the components of experiential design give us kind of a, and even the four element theory that you're putting forward, which, as far as I'm concerned, is the same thing at this point. You have a way of kind of seeing which aspects of this experience are working really well, where do I need to try to make improvements? And now, of the places where I think I need to make improvements, what would actually help the user in some way get to the experience that I want them to have? And will those improvements actually benefit them in a tangible way versus, well, I can have a really realistic bird flying around in the background somewhere, but it's not part of the experience. So that didn't really bring anything additional to what we wanted that user to feel.
[00:30:45.943] Kent Bye: Yeah, and after going through your experience that you had with Fearless here with the sequence of increasing levels of spiders, you know, starting at the level 10, which is your the most scared of it and all the way up to kind of the most intense level is that, you know, there's a certain level of plausibility that you had in an experience where everything looked real. And I had a sense of deep presence of sitting at a desk where I've had a lot of experiences, and it's very familiar. and just seeing a spider and seeing how it moved and just the animations and the lighting. I think there's just a lot of focus on creating that sense of presence in your experience. And I think that focusing in on the guided tour and just very educational in terms of like unpacking the different dimensions of fear and how you overcome a fear. And even if you are completely fine with giant tarantulas crawling in your face, you know, there's still a lot to be learned from kind of going through this experience in that way. And so, yeah, I encourage people to check it out. But just to kind of wrap things up here, I'm just curious to hear what you think is kind of the ultimate potential of virtual reality and what it might be able to enable.
[00:31:51.755] Dustin Chertoff: Oh, everything. I honestly believe that VR is going to become the primary medium for both entertainment and non-entertainment moving forward. From how we go about learning new things to how we do job training, which is actually a very quickly growing segment and has been for decades. to even just like how you can bring remote health workers, let's say, to people in areas that they don't necessarily have access to things. So if you take mental health professionals and bringing them to areas through virtual reality to provide some sense of counseling or veterans who have PTSD and don't necessarily have a VA nearby them to go and get help. If we can bring them into a community of other people experiencing these things, they can see that they're not necessarily alone and that we can bring people there to actually help them. That's really, really powerful, can really help a lot of people, and would really be helping the world in many ways by just breaking down the barriers, communicating ideas, and transferring knowledge that are a lot harder to do when we have physical walls in the way. VR just lets you knock that all down and just allows people to come together and do things in a safe space that they wouldn't necessarily be able to do otherwise. So I can't wait.
[00:33:22.493] Kent Bye: Awesome. Well, thank you so much for joining me today. And thank you very much. So that was Dustin Chertoff. He's the founder of Fearless Incorporated doing exposure therapy treatments in virtual reality for different phobias. And he also created a number of different research papers when he was in grad school looking at how to apply principles of experiential design to VR presence theory. So, I have a number of different takeaways about this interview is that, first of all, I went back and read through all of Dustin's research and it took me a while to get through. I had planned on trying to get this podcast out last week, but I just had been really immersing myself into this research and finding that I was actually in very similar trajectories of Dustin in many different dimensions. I mean, he talked about this vision of creating this unified theory of presence and trying to tie all these different disparate theories together. He realized that at some point he had to let that go and focus on different frameworks that are going to have some sort of pragmatic application to actually making different experiences. And so he's looking at the principle of user-centered design. And in user-centered design framework that he was looking at, you're breaking up into three different systems. You have the user, the system, and then the experience. So in this case, the user has their own subjective experience that they're going through. And then there's the system, all the hardware, all the logistics. In my interview with Mel Slater, that's a lot of the stuff that he's looking at of how the technology is impacting the sensory motor contingencies. And that is a lot of what the focus of a lot of presence researchers have been looking at as well as the technology angle. And the thing that is probably the most overlooked is the content or the experience that you're actually looking at. I think if you look at up to this point, a lot of the process of creating a VR experience, it requires all sorts of different skills that the video game industry have been using for years and years and years. But yet within the academic research community, there's been less of a focus of creating these crazy, immersive, wildly engaging content that you're going through when they're doing these different various studies and process and research. So it makes sense that a lot of the academic research up to this point hasn't been focused on, you know, how can you create these wildly immersive experiences. But when it comes to like advertising and marketing, There's been this trajectory of creating more and more immersive experiences for many, many years. Dustin Seitz, Pine, and Gilmore wrote a book called The Experience Economy. So from that book, he's pulling the five major different dimensions of experiential design, which is physical, cognitive, effective, active, and relational. Now, Chertoff changed the physical to sensory, because when you talk about a virtual reality experience, you're mostly talking about tricking your sensory perceptions. For me, I call it just embodied presence. Now, in terms of emotions, there's the affective, and then I call it the emotional presence. In terms of expressing agency within experience, he calls it active, and I also call it active. And then the split between cognitive and relational, I call it mental presence and social presence. So there's pretty much a one-to-one translation between these theories and Chertoff's pulling from Pine and Gilmore from the experience economy, as well as looking at the broader field of experiential design. And it's really about the internal subjective experience that it's the quality of experience that you're having. Defining an experience, it's things that you go through that you remember, you call later, you can reflect on it. And it's ingrained within your memory as something that you went through, that you were there and you have vivid memories of it and that you can refer to them and reflect upon them. And that's something that experiential marketing has been trying to cultivate in terms of branding for many, many years. And that's also just essentially what virtual reality is trying to do with cultivating this sense of presence. So I think it's a huge insight that Dustin was able to make these different connections between these two different realms and to be able to describe it in this way. And, you know, I've sort of independently came to a lot of those same conclusions. And for me, instead of talking about a theory of presence, it could be just a framework for experiential design, such that when you're designing VR experiences, you're considering all these different trade-offs between all these different dimensions of presence. So Chertoff also went through and looked at all the different presence theories that are out there. And I think a lot of them are focusing on the sensory information and the tricking of the sensory motor contingencies that are happening. But there's also some theories that start to pull in other dimensions of the content as well as the social construction. So in the write-up here, I have a graphic that he had in his PhD exploring additional factors of presence. But he's got this great summary of the eight major different presence theories and how they break down. The other thing that he talks about that we didn't really dive into into this interview was looking at flow theory and how flow theory is also a part of thinking about presence. Because when you're in the state of flow, you're also in this complete level of immersion within a virtual reality experience. Mihaly Csikszentmihalyi has this whole theory of flow and looked at why some of these artists were continuing to do work even though they weren't necessarily selling it. It was like the work was good enough for them because they were achieving these flow states while they were doing it. And so we started studying that and found that there is like this relationship between the task that you're doing how challenging it is and your abilities and your skill. And you reach these flow states when you have this balance of high skills that you have, but also the challenge that you're doing is also high enough. And that's what a lot of games are doing with the game progression curve is that they're increasing the challenges and you're getting better as a player. And then you see this progression and then you get into these different flow states as you play these games. So looking at the insights from game developers and game design and seeing how the content of the experience is really cultivating these different levels of immersion and what VR people would call presence, but looking at these different models to describe how engaging any given experience is. And so in this model, there's this conceptualization on two different axes. It's combining different dimensions. The challenge level is the objective experience that you're going through. So the different rules and things that you're actually going through within the experience. And then there's the objective skills of the individual and the subject such that your skills are either matching the challenge level or they're not. And there's an eight channel model of this, which at the low level of challenge and range of skill, you're either experience apathy, boredom, or relaxation. And at the moderate level of challenge, you have this range of worry to control. At the highest levels of challenge, you're either anxious because you don't have the skills to be able to do the task, you're aroused. And in those flow states are achieved when you have this harmony and balance between the challenge that's being presented to you, as well as the skills that you have to be able to execute that. And when you achieve that, you get this magical feeling of flow. And when you're in those flow states, it's very similar to these sense of presence within virtual reality. So in this flow model you're kind of combining different dimensions of the user system and experience. You have the user skill level at one axis and then the experience's challenge level on another axis and then from that you're getting the subjective internal state of the user from those two different combinations. So to me, the big takeaway is kind of echoing this last interview that I did with Mel Slater, which is that there is a separation between the subject object and the content of the specific experience, such that in the user centered design, the subject in this case would be the user. The object would be the system of the hardware and all the associated virtual reality technology. And then the content is kind of like the experience that you're going through. And so you kind of have this mapping between these different dimensions in the process of experiential design, as well as the elemental theory of presence or the elemental theory of experiential design are both looking at the different trade-offs that you have within the different qualities of the experience that you're providing to somebody. such that they're able to have this very rich, immersed, engaged experience. And that the more that you're hitting all those different dimensions, the more memorable it'll be and the more impactful it will be as a virtual reality experience or any type of human experience that you're designing for. So that's all that I have for today. Just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, there's a couple of things you can do. You can just spread the word, tell your friends, share this podcast with somebody who is interested in designing experiences for virtual reality or helping understand their own experiences. And become a member to the Patreon. There's going to be more and more membership benefits that are going to be fleshing out. I know one of the things that a lot of people are really interested in is doing gatherings within virtual reality. I'll be doing some live streams and Q&A. And just generally, you could just support the podcast by helping ensure that I continue to do this coverage and bring you these types of discussions and analysis. So become a member today at patreon.com slash Voices of VR. Thanks for listening.
