Oculus employees Yelena Rachitsky and Isabel Tewes gave a great talk at Oculus Connect 5 titled “The Hierarchy of Being.” It’s a great melding of theory and practice of the impacts of embodiment on the phenomenological experience of a VR experience. They take a human-centered design approach of centering on the self, avatar representation, and the phenomenological human experience as being the bottom as the core foundation, and then looking at how the world, environment, and object interactions creates a certain set of behaviors, interactions, and overall context of an experience, and then finally on the top is the social interactions.
I had a chance to sit down with Yelena at Oculus Connect 5 to break down the key findings of her research process, and some of her key experiential design insights that she presented as a part of “The Hierarchy of Being.”
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Here’s the video of The Hierarchy of Being talk at Oculus Connect 5, which I highly recommend watching:
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to The Voices of VR Podcast. So one of the presentations that I was really looking forward to seeing at Oculus Connect 5 was a presentation called The Hierarchy of Being. It was done by Yelena Wojcicki and Isabel Tues, and they were looking at, from an experiential perspective, what are the different dimensions of embodiment, and how do you do this translation of giving yourself an avatar representation, then you go up to the environment and you set the context and have different affordances of those objects, and then finally you get connected to other people. but they're really trying to match this theory and practice by talking to a whole bunch of different researchers as well as pragmatic experiential design creators who were in the process of implementing these different ideas and iterate on what's possible and trying to synthesize this dialectic between the theory and the practice. And before the presentation, Jelena reached out and I passed along a number of different interviews from the podcast and I think it was a part of their research process, both to listen to the interviews of different people that I've talked to, looking at things like embodiment with Mel Slater and Jeremy Bellenson and Jaron Lanier, but also just generally what the other practitioners are doing as well. And they had this big, vast research process where they talked to a lot of people and synthesized it into their presentation. I highly recommend people check out the presentation because it was jam-packed with a lot of great insights, lots of videos. And this interview was to kind of unpack it a little bit more, to get a little bit more of the context and the backstory, but also to engage in some of the different concepts and ideas. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Yelena happened on Thursday, September 27, 2018, at the Oculus Connect 5 conference in San Jose, California. So, with that, let's go ahead and dive right in.
[00:01:59.312] Yelena Richitsky: Hi, I'm Jelena Rochitski. I'm an executive producer at Oculus on the Experiences team. I help oversee our slate of non-gaming, more narrative-based content. Right now, we're really getting into the place where interactivity and narrative meet together, and I'm excited to be back on. Thank you, Kent.
[00:02:20.043] Kent Bye: Great, yeah. Well, you were putting together quite a presentation here at Oculus Connect 5 about the hierarchy of being. So maybe you could tell me a little bit about what was the backstory for how this presentation came about, your research process, and sort of everything that was leading up to what you were able to present yesterday.
[00:02:37.873] Yelena Richitsky: Sure. Yeah. So I gave a talk, it was called the hierarchy of being embodying our virtual selves. I think for me, I've been deep in interactive storytelling for the past six, seven, eight years or so across different technologies. And then for the past few years, specifically just VR, investigating what that is. A lot of the talks I've given in the past are purely around storytelling. what's been apparent to me and I've been highly curious about for the past couple years and actually more so. You know, I used to be at Future of Storytelling. It was all about storytelling through the senses. Everything was related to the body. But what I found was oftentimes in content creation, the process was creating something, prototyping, iterating it. Does it feel good? If not, let's do more. But I wasn't necessarily getting to the root of why something would feel good, what it meant to basically have this body in space. If you think of VR, it's just another recreation of another type of reality, but our bodies are actually in there. So I really wanted to understand and investigate what it meant to have my body in there. How does my body respond to things? How does perception work? What feels good and why and what doesn't? And find that point between what academia and research is doing, and I'm very lucky to work at Facebook and have access to Facebook Reality Labs, and we just have so much incredible Research but also outside of this and a lot of it through your podcast. So thank you Kent Of that type of investigation and then also talking to a lot of content creators and figuring out how did you approach this and why and starting to find out where that middle point is and can we start creating a conversation where what researchers and scientists are finding and figuring out and to help content creation be more powerful and be more meaningful and that we started a stronger foundation. So in a sense it was a little bit of a stepping back because the actual process of storytelling or creating experience is the ultimate outcome. where is it all coming from and so this talk was really high-level because we only had 45 minutes to basically talk about the body and it was very ambitious and we initially had 165 slides and had to cut it down to 85 so there's a lot more within that investigation and I would say that this is probably just the beginning of that journey of deeper investigation about what it all means but it was it was a pretty great process it was really starting to understand the physiology of things, the studies that have happened in the past and learning from people in the past, like what Jeremy Billinson's been doing, what Jaron Lanier has been doing, what past experiments we've had in the past, both in VR and out of VR, and how does that affect what we do. So the idea for the hierarchy of being and the way we put it together within the structure is this concept that first you have to understand who you are and what your body is in space how much body do you need and your connection to whatever that body is. And sometimes your body can be pretty much invisible, but you're still physically in that space. So even if your body is invisible in a space and something comes at you, your physical body actually feels it. And you have to understand that and then understand what space you're in, because it kind of gives you the context of why you're here. And then through that, you can start really connecting with others in more meaningful ways. So we kind of set it up in that structure.
[00:06:09.164] Kent Bye: Yeah, and one of the slides that you had showed the academia versus the content creators and I kind of see this as this dialectic between the theory and the practice where people who are in the academia are trying to come up with the underlying patterns of reality and come up with some theories of how human perception works, how the human body works, do these experiments like with Mel Slater and the virtual body ownership illusion where What does it take to get the sense of embodied presence in the context of VR? And then you have the people who are pragmatically just making experiences where they're taking all of this theory, but they have to collapse it down into different decision points and trade-offs that they have to actually create something that's real. They have to have an experience that has a context and environment with objects that people are interacting with. But at the core of it, the thing that I found really fascinating was that you have everything centered through the lens of human experience. and that you have this, I guess, analogically thinking of like the Maslow's hierarchy of needs, but with the hierarchy of being, having the self at the ground level, and then the world, and then others of the social interaction says, metaphysically and philosophically, that the experience of yourself is primary. It's fundamental. It's foundational. Like everything within an experiential technology starts with your direct experience. which I think is in some ways like a philosophical shift for how most of reductive science looks at the world, which is like anything that can be objectively verified as concrete reality, consensus reality that we can all agree upon. That is the world that is the foundation of physics and chemistry and biology and psychology and basically everything is coming from this naturalistic perspective of concrete things. But now going from an experiential perspective, you're saying that the self is the center and that is the foundation and everything kind of from there provides a context for how you relate and how your experience is mediated. Which I was resistant to that at first, but I found that that was quite a really interesting and sophisticated shift, a philosophical shift. So I'm just curious to hear what your thoughts on that are.
[00:08:05.427] Yelena Richitsky: Now, VR is human-centered design, right? So it comes with a human first, and that's the way that we're thinking about it. I think when you're starting to design for it, it's trying to understand the intention of who do you want your audience to be? And with that, I mean, how do you want them to feel? And that determines how they want to behave. And I think that's one of the most important questions to ask when you're creating for that. And you're basically creating for the body. And yes, there's theories that are out in the natural world, but part of the idea is And VR kind of brand new and we're learning things in a little bit of a different way. And so when you go into it, it's you in it. And what does that mean?
[00:08:47.703] Kent Bye: So what is the self?
[00:08:49.684] Yelena Richitsky: Whoa, Kent. The incredible thing in VR is that the self is whatever the creator can help create for that audience and that person. And the big point of that talk is understanding that that's a powerful thing that you can do. And the studies that were talked about through things like what Jaron Lanier did, and a lot of other people did, and Jeremy Bailenson and so, was that a strong effect of whatever you do end up embodying actually really shifts the way that you behave.
[00:09:28.213] Kent Bye: And you showed different experiences in terms of Life of Us, for example. Maybe you could talk about some of the experiences that you referenced in terms of using metaphors, of seeing how these different avatar embodiments may be able to change how you behave within a virtual experience, and maybe some of your experiences of that.
[00:09:44.233] Yelena Richitsky: Yeah, definitely. So I do really love Life of Us a lot. It's created by the company within with Chris Melk and Aaron Koblen. And in it, you start going through the process of evolution, starting off as a little amoeba and going off to a little fish and monkey to a gorilla and pterodactyl and business person and then a future human and you do it with another person. So ideas within embodiment in VR show that you recognize your body either through looking at a mirror or something mirroring you. In this case it's a social experience and the other person looks exactly like you. So you see who you are through the lens of how they are. which is an interesting way to do it, and you learn the actions and interactivity you can do through feeding off of each other and seeing what's possible, because in VR, it's first person. Within this experience, you can only really see your hands when you have them, especially in this experience. But you start feeling very different. It's compounded by the shape of your body that you see them alongside the voice that you're given in there. So when you're a smaller figure, it's just very high and you talk like this and you just feel funny and you feel fun and you have a different weight on you than when you turn into the gorilla and you sound like Barry White and you're like hey and with that it's just you feel heavier you start feeling like this gorilla and you start feeling like everything's just very heavy and What's interesting about this one is that you change forms very, very quickly. And that showed to me how quickly you can actually shift your behavior based on the form that you have. And there's other really great examples. I'm working on another project with Within called a lamb child superstar, and your avatar is basically a lamb child superstar. So you look like a lamb Elvis, which is really fun. And part of that is that you also change, and it's also social. And so when you see the other person, because you're both so funny looking and silly looking, you end up having a lot more fun. You end up not taking things as seriously because of the colors that they use, because of the shape of the body that they use, A similar thing, I showed some samples from Max Weisel's project Normal Chat and he's doing really, really interesting things where I talked about this idea of realism versus abstraction, where it's really hard to actually recreate realism because we as humans are just so wired to recognize facial expressions and the nuances of all of that that you know so quickly when something's not real because it's so built into our evolutions and I'm sure you know. So instead a lot of these creators, and I talk a lot about the role of artists generally, to take these concepts about things we know and create something that's meaningful and fun for us. So they took it in a direction of highly abstracted And that's kind of like Journey, which a lot of people have tried, which is not VR, but also abstracted characters that a lot of people took meanings from, but that you can derive a lot from just minimal expression. And the way they approached it was very cute. so you look super cute and I showed some video footage of it and you right away love that character and when you're in that character it's really hard to be angry at other people or to act in an aggressive way because you've embodied this cuteness and then you start feeling a lot sweeter and this also helps with connection with other people where helps with friendlier communication and helps with what I think is more positive behaviors. Of course these types of avatars can't be used for everything. I would love to go to a work meeting dressed as Max's avatar because she's so cute and maybe people would really be on their best behavior, but it's great for that purpose. So in some senses, we're gonna definitely need realism, but In these cases of abstraction, it works really, really well and creates this, what I think is positive behavior. And I think that's something that's really, really important to think about.
[00:13:57.929] Kent Bye: You also had a screen that was showing a lot of Snapchat or Instagram filters where people were able to have an augmented embodiment of a character which was through the mirroring effect as you are able to see this visual synchrony of you being able to move your face and see the corresponding movements and you start to feel embodied in that character but then I have this theory that there's a certain amount of these augmented reality filters are kind of a portal into different dimensions of our personality that may already be there or maybe sort of like latent and we don't know that they're there until we have this portal interface to the augmented reality facial filter that allows us to tap into the deeper aspects of our personality that we're able to express different dimensions of ourselves that were maybe dormant before but were able to really tap into as we have this embodiment with the augmented reality filters. I think people probably have the phenomenological experience of that within augmented reality and messing around with Snapchat or Instagram filters, but having something like that within a virtual embodiment where you're actually embodying a full character, I think there's a similar type of as you embody a character, you start to take on the different personalities that you're kind of projecting onto the different symbols of what that means, which allows you to express yourself in new and different ways.
[00:15:10.629] Yelena Richitsky: Yeah, yeah, definitely. Well, it's still you, right? You're still you in there. And maybe it's just a much more childlike, open minded version of you. I think in our everyday lives, when we're as ourselves in the world, we kind of might be also wearing masks, right? We don't always act as our most authentic selves because you have to protect yourself in some situations. So you have to act differently in some different contexts and situations. So in real life, there's definitely some version of masks as well, but simple things like Instagram filters, where you see your face as a lion or as little puppy, it actually changes your physical face. And that definitely brings it out from you. And it just shows how powerful vision is in being able to shift the way you recognize and you see yourself. I think I gave this example around the Proteus effect during the talk, how you see yourself actually alters how you behave. And in it, there was people who self-chose as what they considered attractive avatars. And they felt more confident in connecting with what they thought were other attractive avatars. And the opposite was true when they felt like what they self-chose was unattractive. Obviously, it's very subjective. It is what they considered attractive or unattractive. And that's pretty interesting. And it is probably some levels of confidence that opens up. I think we all walk around with a little bit of filters and things we feel comfortable with and things we don't feel comfortable with and when we're around different people you probably open up a little bit more than when you're with other people and you feel a little more guarded because we are who we are in relation to everything that's around us kind of, right?
[00:16:59.645] Kent Bye: Yeah, well what I see this going is that people being able to maybe pick an emotion that they want to have and have an avatar embodiment that maybe reflects that and so maybe if they're feeling really confident they can if they're a cat and they can choose that line but maybe if they really want to amplify feelings of shame or self-doubt or whatever else you could pick an avatar that would take the worst aspects of your own self-consciousness and Blow it out of proportion so that then you could walk around and really wearing your shame out there So I think it could go both ways that you could start to have people Choose the emotions that they want to have but have some way of dialing that in and to say if you want to feel happy if you want to feel sad if you feel a grief if you feel like you are experiencing a lot of shame and you want to remediate it in some way. Maybe this is a way for you to amplify it, but then at that point have this polarity therapy where you embody the extreme polarity point so that you can embrace the opposite that somehow emerges out of the process of doing that. So, I don't know if you've experienced that, noticing this connection between how these avatars are mediating some sort of emotional effect as you're embodying these different types of characters.
[00:18:05.401] Yelena Richitsky: Yeah, I'd love to see someone create an experience like that where you can actually embody the different emotions that you want to go through. I would love to see an artist take that on, to be honest, because that's also what I do as I enable artists to take some of these ideas and concepts and see what we can make that actually feels emotionally connective and meaningful. entertaining and interesting for people but it could also be really interesting as a therapeutic thing. I mean if you like for instance VRChat is basically a place where avatars are currency and there's not that much activity like Rec Room for instance has a lot of activity and it's about games that you play together and many games that you play together and games that you create but VRChat is specifically around various identities and who you are. And when I talked to friends about it, they said most of the conversations start from who you chose as yourself and your avatar and where you got your avatar from. And I found that to be pretty fascinating. It also reminds me a little bit about Burning Man, for instance, and how big it's gotten. And for people, it's so incredibly freeing to be able to go in there and basically embody what they find their most truest, comfortable selves. without the judgment that they feel in everyday lives. And that feels somewhat similar to how people talk about VRChat a little bit. So part of it is freedom to be yourself, and part of it is, like you said, possibly to embody these different emotions that they can go through in VR, helps them get through within VR that they can't necessarily do in real life. I think the thing I'm really interested in, because I focus on creating experiences for people, is how these allow for them to behave, and especially positive behaviors, and creating spaces in VR where you enjoy being in it. And taking it back to Burning Man a little bit, how did they create a space where everything was run by others and it's a gifting economy, and they created a judgment-free zone. for the most part. And the peer pressure there is, if you are judgmental, then you don't really fit in. Then that's something that's not really accepted. And there's a peer pressure around being non-judgmental of everything that goes on. And so how do you create the foundations of those behaviors, especially when it comes to VR? How do we create the positive behaviors in the spaces that we want? You can do that through different design principles, thinking about aspects and things you allow them for communication. But I do think it all starts with how you view yourself. And that, I think, is why, in this hierarchy, it all kind of starts with the self. Isn't there some Aristotle quote about knowing the self is the source of all knowledge?
[00:20:48.504] Kent Bye: Yeah, I mean I think that this idea of cultivating a culture is something that I've been exploring a lot in terms of talking to Jessica Outlaw and Lola MacIsaac because I feel like that the cultivation of a culture comes from the community but yet that's based upon individuals actions that then aggregate and so what is the sociological aspects of how that culture and those values are transmitted and it turns out sociologically and from an anthropological lens you can look at the stories and the myths and the jokes, what people find funny, but there's also different codes of conduct and rules and social taboos and social norms. So a lot of it ends up being the implicit things that you can see when it aggregates into the collective, but it can come through stories or who the heroes are, who's really exalted as really embodying these principles that you want to show. that helps communicate all that. Making jokes about different things in some ways is a way of talking about the invisible taboos that is unspoken, but yet is part of the shared culture. So I feel like the consistency of something like Burning Man, there's also the rituals, so that people who haven't been a part of it can be a part of, this is the course of a day, this is the course of the week, we have this big thing where we burn the man, and there's rituals that happen. like the Super Bowl, that's a ritual that we have in a society that has a certain culture around it. And so, looking at it through the anthropological lens, you can start to see what are the other dimensions of culture that start to get fostered. But for me, it's one of the biggest open questions, because it's that very problem with, okay, if you have something like Burning Man, where you have this group of people who have collectively made the decision to create this context under which they're having a gifting economy, where they have all these sets of rules and behaviors, then how do you encourage those behaviors as you're starting up a virtual reality environment? How do you do that from experiential design? What's the context? What's the environment? But also, is there some sort of an initiation process where you have to train people for what this even is? Like, as you're entering into college, you may get a tour where someone shares with you all the different aspects of that story. And so there's like this initiation into that. Once you're initiated, then there's a responsibility to uphold that culture. And that works at small scales. But when you talk about something like Burning Man, it's been growing over years, and maybe as it grows past a certain point, maybe there's a certain amount of that integrity of that culture that starts to get a little bit more diffuse as people who, you know, how do you deal with the bad actors who are not really in line with all the values of that culture that's being created? Like, how do you deal with bad actors is another issue. How do you deal with People are doing harassment or trolling. So in talking about online spaces, these are all very interesting issues that come to those cultural aspects. But from my question, it's like, well, what are the experiential dimensions that you have control over as a VR creator that you could help to sort of cultivate all of that?
[00:23:25.239] Yelena Richitsky: Yeah, and that's part of what I've been thinking a lot about, especially when it comes to how you design experiences in the way that it affects people. And so with starting with a self and creating your identity and how you see yourself and then being given context around what your space is and what the rules are, like Ida Benedetto, who's a good friend of mine, did her thesis on transformational experiences around sex parties, funerals, and wilderness trips, which I love. But she talks about the concept of the magic circle, where you have to create the rules of what's inside of the magic circle, and there's different ways of creating that context for it. But look at the design principles of general human condition stuff, right? So perception of taking ideas from what we already know. I mean, we all come into the world, even though you and I are living in the same world right now, we still have different experiences that have shaped us in different ways. So when you look at something, you take different meaning from it than when I take from it, which is very complex. And then you try to take us into a space like VR where everything is kind of brand new again, and you're rebuilding, and you're trying to rebuild it for all the people, even though each person has different life experiences. So how do you cut things down to the basics of things we might already know? We kind of start knowing how our physical bodies work, we're starting to learn more about the brain, and we have slight understandings of communication, even though most of the world's issues are around miscommunication still, even in this world, even though we've lived all of our lives and have evolution around communication. That's very complex stuff to try to take that into VR and rethink it and see how it forms. But you can take simple things of understanding. When you look at yourself and everything's bright colors and everything's cute and there's just sweetness in the world, it shifts the way that you behave. When you're in an environment that is friendly and brighter and taking cues from architecture, because they think about moving people through spaces and shifting how people behave, whether it's a space that they stay for a little longer, or they want them to move around more, or they want to affect mood in some way, or they want it where there's congregation, or they want where you're by yourself. and also thinking about ways of communication that we know. You know, for instance, it was interesting, I was talking to Danielle, and I'm forgetting her last name, she was on your show, she's a neuroscientist, and she was talking about communication generally, where there's these kids in Nicaragua, and they were all deaf, and they didn't know sign language, but they were all brought to a space together, and they actually ended up creating their own language with each other, and I think she might have mentioned this to you, And that was just a sign of how much we need to communicate, and I find VR is a new space. I mean, we're not all deaf, but it's a new space where we're trying to understand what communication means within VR, given that we don't necessarily have all the tools that we've had in the past, like understanding and reading body language, the nuances of facial expression, all of the full body tracking gestures, because that's not here yet within consumer VR. But what's been emotive ways that people have been able to communicate, even with the things that we infer big meanings from, are even simple, like GIFs and emojis, hand waves and hand movements, and small facial expressions, tones of our voice. So it's taking all of those things into consideration when you're starting to design for VR.
[00:27:01.922] Kent Bye: Yeah, and I just wanted to take a step back and go back to the hierarchy of being, because I think we've covered all the dimensions, especially in the last answer there, you just talked about the world and then the others and the social dimensions, but the foundation that we talked about in the beginning, which was the self, which was your virtual body ownership illusion, your sense of your avatar, but also your direct phenomenological experience of whatever you're experiencing inside of any VR environment. And then you move up to the next level of the world, which starts to set that context, but also you have objects in that world that you can interact with and you have certain behaviors that are encouraged by those affordances of those objects, but also the architecture and the space and the color, everything else about the world and context that has projected meanings and symbols that have certain cultural associations that you can start to then symbolically communicate through having different posters that may communicate a certain value set. Or you get a sense of a vibe of an environment that you can pick up on all the unconscious cues of what the culture is by based upon how you design that environment and what you want people to do. And then we jumped into the social aspect, which we'll get to in a moment, with how you interrelate with other people and all the body language issues and all that other stuff. But let's focus on this world part. Thinking in terms of a design framework, there's different trade-offs. And so how do you think of the different trade-offs when it comes to designing a world and a context and the objects and the behavior and the architecture and the color? If it all still goes down to the human experience that you're trying to evoke, how do you connect human experience with spatial design, color, and everything else in a world to be able to evoke that emotion?
[00:28:34.659] Yelena Richitsky: Yeah, what's interesting about VR is that you don't have the same rules that you do within what we're limited in space, so it's this constant back and forth between familiarity and new expectations that you're forming. A lot of what we talk about is that everything is based on our expectation, everything is meaningless until the particles of light go into your eyes and brain makes meaning based on the experiences that you've had before. So coming with that, something has to be familiar for you to grasp onto. And then you can start introducing unfamiliar stuff, but you have to do so in a way that's consistent. So we start believing it. I think there has been interesting approaches when it comes to space and VR that have worked in different ways all the way from something like Dear Angelica, which is not interactive. And a lot of what I've been talking about in the talk I gave was specifically our embodiment within interactive and six degrees of freedom and being able to move your body around. But I think Dear Angelica is a good example of space that is based on an idea of memory versus something that's meant to represent just physical environment that we're used to. And what's great about it is it's all hand-drawn. And if anyone hasn't seen it, please see it because it's a beautiful piece of art that I think really elevated the craft of VR. And what the space does is it uses fragments of moments to make it feel like memories that you're in. Because when you have memories, your memories are still in some sort of space. The ideas have a visual interpretation of space, but they don't all fit in chronologically together. So when the space draws in and out from each other, it's kind of how our memories work a little bit. And it's kind of in these fragments that evoke kind of a dreamlike state. So the space that they created was specifically meant to evoke this dreamlike state. And in addition, they use really great things for scale to evoke a specific feeling of smallness or bigness, right? So sometimes when scale gets super big and you feel really small, there's like a frailty to the experience and everything in that is meant to serve the story. The same thing happens in Wolves in the Walls by Fable Studios, which I really love, and the space there, even though it's more representative of a physical house, the way that they approached it with colors and with shapes was all meant in service of the characters that you're with. and all intended to cause a specific feeling and have you emotionally connected. So just like immersive theater or theater, traditional theater and set design, they have these gradual reveals of the space that keeps you from looking everywhere at once, but has you focus in something and you connect the emotion of what you're looking at that moment of what they connect you with so you're not fully distracted. And they have this slow reveal of the world but only so you can start getting accustomed to both the character and what the character is introducing you to, which is incredible things you can do in VR that you can't necessarily do in the real world, obviously. So a lot of the same rules apply, you just don't have the limitations of physics and gravity like you've had before, so you can really do incredible things. Like, my favorite things in VR are sometimes the simplest things. It's playing with scale to evoke a specific feeling. And then I talk a lot about objects, too, and objects as our connection to space and a little bit of our extension of selves, and that pushes us into acting in a certain way, too, and this concept of object believability that Facebook Reality Labs is really working on. which, you know, is obvious when you think about it. It's connected to how your brain actually picks things up and holds things is how much you'll believe it in VR, but it's really hard because we're holding touch controllers and some people have done some really interesting things, but... Everyone's been loving a lot of simple experiences like Beat Saber. People love that. And Job Simulator. People love that. And it's just so rewarding to do things that actually feel right in VR versus make our brains feel such like cognitive overload of that might not be real that they become pretty successful and sometimes people overthink it a little bit. I mean, you have the opposite extreme where I talked about this project called Lone Echo that was done with Ready at Dawn and Oculus where they spent so much effort on the specifics around the hands to fit around objects to make it look real so your eyes believe it. But it's really hard to do. But like the job simulator or the Beat Saber types and you do it and you're just like you want to do it over and over again because It feels so good for your body. You mentioned there about the perception.
[00:33:24.692] Kent Bye: At the beginning of your talk, talking about the self, you were talking about the perceptual cycle, where there's a perception and then meaning. Maybe you could go through the different phases of the perceptual cycle as you're going into a VR experience and how that relates to this whole talk.
[00:33:39.283] Yelena Richitsky: Yeah, we really simplified it when we talked about it and also I gave this talk with Isabel Tuas who we co-wrote it together and that was a lot of fun. It's a similar concept I was talking about where everything is about the expectations that we have and that's been framed by the experiences we've had in this life, but also just how our body's been raised by evolution of when to, we should be scared of, or something's friendly, our bodies have this lizard response to it. But there's things that we've learned in this life as well. But basically it just works where you have an experience and you have an expectation. experience happens and you go through this whole process of sensory input and your body's have this response is recognizing or unrecognizing it, reacting to it based on what it is, whether it's something you should go away from, whether you shouldn't come towards or something in that gray area in between. And so you have this reaction to it and then it kind of goes around through there and ultimately if it's something that's new that you haven't had before, it creates this new memory and changes the expectations that you have in the future. And it's a similar thing kind of in VR. I think Jaron Lanier talks about this a lot, but we're constantly going through this sensory motor loop and sensory motor loop. And you can create these new behaviors in VR, which is super, super fun and exciting and great to play with. But it's just having an awareness of how we work in order to make it really effective.
[00:35:00.958] Kent Bye: Yeah, and certainly James Blaha being able to create that sensory loop cycle to be able to essentially train his lazy eye to be able to see in 3D for the first time. And you're able to do a lot of stuff with the principles of neuroplasticity with that. So one of the things that is interesting about the world is that it's setting a context. And usually there's a context under which we do things. So if you're in the bathroom, you may be brushing your teeth. And so you may be taking a virtual toothbrush and brushing your teeth. You don't have the haptic feedback of it, so it's not as convincing. But there's certain things that you do in certain contexts. Like if you're at a bank, you're more likely to tell someone your social security number than if you're on the street in New York City. So there's certain contexts that changes our behavior. And one of the things that talking to Jesse Schell about I Expect You to Die is that they really wanted to create a context under which people were really familiar. So we spent a lot of time driving. So that's one of the reasons why, you know, some of the first escape room type of experiences they created in VR, where you're in this truck, which feels familiar, but yet you're in an airplane, you know, trying to escape this death trap. But yet, you have a steering wheel and you have all these affordances that we know. Like, when you have a steering wheel, you know what behaviors can evoke, because you know that when you drive, you can do certain things. And so it's like this skeuomorphic translation of these real-life contexts that we have and all the meanings that we have associated with that. And being familiar with all those cartography of contexts of the human experience so that you can start to know how the same behaviors may be evoked by different contexts and different objects within that environment that is trying to really guide that behavior. But really starting to think about it in terms of not just the world, in terms of the physical aspect, but what is the meaning of that world in terms of the context, and what does that context mean for that human experience?
[00:36:49.091] Yelena Richitsky: Yeah, like I said, it's this constant push and pull between the familiar and the unfamiliar, and that's a really, really fun thing to play with. in VR, and I'm excited. There's a lot of projects I'm working on right now that are kind of pushing and pulling within those directions. But a couple examples are, you know, I think we're talking about more social experiences right now because there's a lot more freedom to do stuff, but especially worlds that are created. So VRChat, a lot of it is user-generated avatars and also user-generated spaces and worlds, right? And the context is you have these different identities that you can create yourself, and you basically go on all these different adventures that if people have created. And then within Rec Room, it's based on also a lot of user-generated stuff, but it's based on a lot of more games. And they created it in a recreational center, and that has a familiarity to it. You know, when you go to a recreational center, what you're probably going to start doing, even though some of the games they have in there aren't things that you can actually do in real life some of them definitely are but they created a Initial space of familiarity and walls and structures so that you can guide yourself Another great example of that is the Museum of other realities. I don't know if you've tried that but Colin Northway Yeah, so that that's actually a really great example where it's curated VR museum within VR and they have really beautiful works by a lot of different artists but they chose to create a very traditional space of white walls and you know the entryway that you feel familiar with in an outside area that feels calming and And that was all for the purpose of having the sense of familiarity because when you go to these art projects, it's not like just the static sculpture. It's actually another portal and world that you can usually jump into. It's completely wacky. I mean, you do things that you'll never be able to do in real life. It really is on the extreme of the unfamiliar. But you have a comfort in going back to the thing that's knowable. And so the balance of the two of having the unfamiliar and familiar really helps in this situation because if you come out of the art piece and you're again in somewhere like completely out there, it can be a little bit of overload. And maybe as we progress in VR and have more experiences, this shifts our understanding of familiar and unfamiliar. I personally like to like like to see everything going to the edge. I love really artsy. experiences and experimental stuff. But there is a sense of calming and comfort when I know what's going on, at least for a little bit. Another example is Orbis VR, where they created the context of a village area. And that's thinking about the congregation of people and people helping other people. They physically created environments and spaces that helped inform that. So they thought about that with their quest lines, knowing that serendipitously people would run into each other. and that creates spaces for interactions. And also a space that's village-like is something that's familiar to all of us too. It's a place where we all live together. So those are some interesting examples. There's ones that are a little bit less interactive, like Tyler Heard's Old Friend, where it's Super wacky colors, although the world, you know, there's mountains and that seems familiar, but the mountains have faces and they have hands and your hands are wobbly and things are looking at you in crazy ways. But it works for what he wanted you to do. You don't have that many options of what behaviors you can have apart from just dancing, which I love. But that space sets up that context for how you want to feel. But I find that concept of the familiar and unfamiliar, because it's all about our expectations, to be really interesting. So there's a good question just around what is pushing people to a space that feels like too much or too out there and I think that's hard to define for everyone because everyone's limits are a little bit different. Some people want be more rooted in reality and understanding but it's all really based on the intention that you have for the behavior you want in your space.
[00:40:56.045] Kent Bye: I have a little semantic quibble around the hierarchy of being because you have the self and then you call it the world and Then the other but I would instead of world I would actually call it context Because to me world implies that there's something out there outside of myself that it can be sort of a consensus reality But really what you're talking about in VR is a context that could in a moment switch and you can switch context so quickly But the context also is very different dependent on that individual for what association they have for what that means. And so you're trying to create context for people. But also what I see with augmented reality and virtual reality is that we're going to be able to use these augmented technologies to switch contexts in a way that we're fully immersed into a new context instantaneously. So if I'm here and I put on a headset, I'm now in a new context and now immediately into that context. And that context is really dependent on what is in the experience, but that context is in relation to me, the self. Whereas the world is, I guess it's more of a, from a third person perspective of what the consensus reality is versus what my phenomenological experience of that is. And so I would just say that I would name it context rather than world.
[00:42:05.668] Yelena Richitsky: All right. Well, just like VR, the hierarchy could shift and change a little bit. But the world slightly dictates the rules. And maybe instead of context and world, we can just say environment. Does that work?
[00:42:18.873] Kent Bye: I like context, personally. But this is the problem with any two systematizers. There's like these battles of trying to subsume each other's framework into each other's semantic world view.
[00:42:33.653] Yelena Richitsky: Yeah, I think context can fit in all three sections of the hierarchy, because you can create your context through how you see yourself, you can create your context through the space that's created around you, and you can create your context through what everyone else is doing, because like I said, we are who we are in relation to everything that's around us.
[00:42:54.160] Kent Bye: OK, you win. I'll go with world. But I feel like there's certain domains of human experience that are very distinct, whether you're home, whether you're at work, whether you're friends. I think there's certain dimensions of those domains of human experience that sometimes are dependent on the environment, but sometimes they're dependent on who's around. So you could be at home but have somebody at work, and so the context switches. So yeah, you're right. Yeah, it is very fluid, isn't it? Yeah.
[00:43:21.508] Yelena Richitsky: Yeah. Yeah. Context. Context is everything, right? Yeah. No, these are all interesting questions. And I think part of what I'm interested in investigating right now is getting back to those foundations of the reasons behind what we're doing, how that connects to what we're used to and how that changes a little bit when we're in VR. It's talking a little bit about the building blocks that we're creating now that we're making VR, like what are the new expectations, the new memories we're creating for people. So it becomes more comfortable and we have more movement to be able to do stuff. I mean, there's spaces that are completely unfamiliar, like Thumper, right? There's no world like Thumper. It's very abstract with colors. And with that, you just have to understand how do colors affect people, how do sound affect people. You can go to like full abstraction and if you create simplicity of knowing what your interactions are, you can have an incredibly powerful and hypnotic and meditative experience and get in that flow that we all love to feel.
[00:44:21.920] Kent Bye: Yeah, what comes to mind is Blortasia. I'm not sure if you've seen that by Kevin Mac, where it's independent of any context that I've ever seen. It's like the surrealist art that creates a new context. They actually use it for brain surgery because it allows people's brains to be stimulated because it's so novel and unique and fills people with curiosity that allows them to highlight their right brain visual processing and to be able to do brain surgery on their left side because it sort of calms it down a little bit. But yeah, different experiences where it's so beyond any context or environment we've ever seen that it creates a new context of whatever is emerging in ourselves.
[00:44:57.381] Yelena Richitsky: Yeah, no, I love that stuff. I love when things definitely go into the edge. I think the thing that I am really interested in is understanding how space and context can make us be our best selves. Because that's just a funner place to be in when you feel like everyone is just being their best selves. And that also gets us back to social behaviors of creating these connective tissues that feel good and feel comfortable for everyone. And so what does that mean? That's also a harder question to ask. It's like, you can be grumpy one day, but there's quick things that can happen. I think people say that if you fake smile, then you'll actually become a little bit happier. So when you see yourself as something and you behave differently, it could actually probably shift your mood and shift your behavior slightly. But that's a question of how do you create these behaviors through what we know so far of identity and of environment, world context, we'll have to continue to debate the middle one, to get us to that place of connective behaviors that feel really good because those type of behaviors are so incredibly meaningful and that's the place that I'm really interested in is powerful, meaningful, memorable spaces where you are in and you're able to experience and it actually makes you also be better in the real life, right?
[00:46:21.291] Kent Bye: Yeah, one way to think about it maybe is that through the external avatar, it creates a sense of being, of a sense of self, and a sense of identity. And maybe the external world that you're creating, you're trying to create an internal context for each individual, but yet that's kind of out of your hands at that point. You can design an environment all you want, but it's really up to each individual to see how they're going to create their own context within that environment.
[00:46:41.707] Yelena Richitsky: Yeah, I guess this kind of goes back to the thing that you were saying at the beginning where you can be different versions and play out different parts of yourself and different behaviors of yourself in VR and potentially are able to live out these parts of yourself that are that have been hidden or that you've been uncomfortable with or that show different kinds of emotions that you've been taught to just hide the whole time that must have some impact for you after you've come in that you've felt more relief or you felt like oh that actually felt really good and makes you question have I been hiding this from myself this whole time or I didn't realize that like I've really just want to be a blue dinosaur my whole life and that's my new life's goal because I feel really comfortable in it. So obviously that's a silly example but I think if you think about it on a more serious note it could actually have some incredible benefits.
[00:47:37.991] Kent Bye: Well, if you think about the metaphor for people coming out of the closet, there's certain aspects of their identity that they've had to keep secret and private and hidden in exile because there's been certain sociological norms that feel like it's not safe for them to be their full self. And so that metaphor of coming out of the closet means that you're taking those exiled parts of yourself and you're reclaiming them. And I think that in some ways, virtual and augmented reality is allowing us to all come out of our own closets that are dimensions of ourselves that we've either hidden or ignored or not paying attention to but we're able to fully integrate but also have a context like that Burning Man non-judgmental be able to experiment a little bit and to be able to play and to be able to discover different dimensions of ourself that maybe have always been there but we've never had access to it because of the technology is tapping us into these different dimensions of ourself that are able to be expressed. And my sense is that as people do that, they're able to feel like more complete, whole, aligned beings with who they are and their expressed identity so that then they could go out in the real world and be more of themselves and those different aspects that they've been exiling and hiding because of the taboos of the culture that they can start to fully integrate that in terms of how they express themselves.
[00:48:51.216] Yelena Richitsky: Yeah. No, I think that's incredibly valid. I think one thing I, I really love about working in this space is that the more I work in it, the more I actually understand the human condition that we've never had to understand it in this way before. And that was the Genesis for the talk is that realization that really this is just understanding human condition. It's understanding human condition, but it's also studying understanding social structures and Why do certain people act in certain ways in the real world? What makes people act a little bit differently? So it's a combination of anthropologists and sociologists and everyone that's studied that before and how does that all connect and relate to VR and what aspects can we take from that and bring into VR to make it work and to be that space that we're all interested in being.
[00:49:42.222] Kent Bye: Yeah, and just to finish out the hierarchy of being, we've talked about the self, we've talked about the world. What about social? What are the dimensions of social that you think are important? Because, you know, you have your sense of identity, your sense of self, you start to have a context when you have an environment. Then when you start to relate to people, then you have new aspects of yourself that come out in terms of how you relate to other people. So what's the framework that you have in terms of thinking about the social dimension of virtual reality?
[00:50:08.094] Yelena Richitsky: Yeah, I mean that's a big question and that's everything that we're beginning to really investigate now and there's been a lot of learning so far and there's a lot more learnings to go of creating a space that is welcoming for everyone and spaces that people want to be in. One thing I've been thinking a lot about is that aspect of communication. I mean think about real-life communication as well. Language and actual speaking is not always my favorite way of connecting, communicating, because that's just one way. It's language. You know, I like to connect with people in other ways, like having experiences with people that are not necessarily verbal, going on adventures with people, having a shared goal. If you think about real life, you rarely just end up in a space with no context and other people and you don't really just say hey. It's rare for people just to walk to a random person on the street and just start chatting with them for no reason. You usually always have a context and space for creating comfort of conversation and comfort of connection, knowing that there's some element of shared goals. But what are the different ways that we communicate? I think it was about a couple months ago I went to this immersive theater thing in LA. It was called The Silent Play. And from the first moment you step in the door, you're not allowed to talk. And it was maybe like 30 people. And they made it fun and silly. And there's a lot of things to play with. And there's a very light narrative they kind of take you through. But it was two hours of silence and everyone was around and you can only communicate through eye contact or through gestures and through expressions. And there was this element where I felt like I connected with people's real selves a little bit more than if we went around and tried to have conversation because you're in conversation you're trying to say something smart or ask a lot of questions because you feel like you should be asking questions about people. There's usually a lot of buffer around conversations that don't actually feel like real. connected communication, but when you're interacting with people and there was no actual speaking, there was this different effort of finding different ways to communicate. Similar to what I was talking about with the kids in Nicaragua, I was finding these different ways to communicate that feel meaningful. I think, in my talk, I talk about Toy Box, which is a VR experience that Oculus created a long time ago, where someone knew exactly who the other person was just by the way they moved their hands, which is pretty fascinating, and there's little things you can do that create these moments of connection that feel pretty powerful without all the pressure of having to talk to people all the time, because that could be draining, it could be a little comfortable. But in addition to that, you know, some people like to be alone, some people like to only spend time with friends. There's a challenge in VR right now of having people in the same headset at the same time concurrently that is also your friend. that you're going to go into an experience with together. You have to find a time together. It's so rare that we do things synchronously these days because everyone's on different schedules. So in order for VR to succeed, we need to figure out how to create a space where you can also be with strangers. And that's uncomfortable for a lot of people, especially when you don't know who that person is. So how do you create context for places with strangers that you enjoy being with and you feel connected with? And part of that, I think, is being able to have your own safe space that you feel comfortable with, because I'm I'm kind of in between introvert, extrovert, and I really need to just go to my own space or my hotel room or my home and just have my own time to chill out. And then I'm able to go engage and have adventures and spend time with people. And I think understanding different kinds of personalities and creating settings and spaces for that can really help in creating those types of experiences. So it's taking a lot of what we already feel comfortable in doing and knowing and thinking about applying that to VR. When it comes to social, there's so many elements we can pinpoint to, and I think we're going to be seeing a lot of interesting stuff coming out soon, because people are beginning to learn what works and what doesn't work. I mean, there's Altspace, there's Orbis, and Rec Room, and VRChat, and Spaces, and Venues, and this has been a year of a lot of new types of social experiences. Coming out and I think we're gonna definitely get a lot of learnings about that but not every social space is gonna be for everyone just like all of us have Separate interests when you think about Facebook as a whole. There's so many different kinds of groups for everything Whatever you're interested in you're gonna find it however eccentric or esoteric whatever your interest is there's going to be a community for that in it so it's understanding that context and what familiarity do you have for that connection point and what is your shared purpose in the space and all the experiences we're seeing right now have different ones where VR Chaz a lot about this adventure you go on with people after you dress up and rec room about these games and it's slightly competitive and it's fun that people enjoy and Orbis has its own one and there's different types of communities that address in different kinds. So there's rarely going to be everything for anyone but there can be if you take that variability in mind and you give people the options for being able to be in their comfort zones. and create the abilities for the types of communications, or minimizing the types of communications, but allowing for that control.
[00:55:32.093] Kent Bye: Yeah, it reminds me of both Sleep No More and Then She Fell, because in Sleep No More, you wear a mask, and so you're taking any communication that you would normally have and broadcast to your face, you're putting a mask on, which makes you more anonymous, which makes you more actually centered in your own experience, or also focusing on the actors that are actually expressing themselves. It allows you to both be more inward, but at the same time, you're silent, you're not speaking, you're not communicating, but you're able to watch the actors that are happening. And that's one approach, which I think in some ways, putting on a VR headset without having eye tracking and being able to communicate micro-expressions, all these things that we don't have with the fidelity of facial communication, like we've lost a whole layer of ways of non-verbally communicating with each other when we're in VR. So in some ways, it's kind of like putting a mask on, where we've sort of blocked all that stuff off. But then there's like Then She Fell which is much more about creating a sense of emotional intimacy one-on-one with another immersive actor that is also very nonverbal and just about that depth of emotional intimacy that you can get from what I would say in some ways is like this extreme amount of what you can do with a human face in such close quarters that I think is going to take a long time for virtual reality to sort of match that same level of emotional authenticity that you can get when you have that direct one-on-one interaction in an experience like Then She Fell. So you have these kind of like different dimensions of communication that are there, but in both of them they're using dance in a form of embodied movement to see how you can start to use your body to communicate rather than using the abstractions of language.
[00:57:08.895] Yelena Richitsky: Yeah, it's using the body to communicate, but I think the difference there is that you have these live actors that are completely reactive to you, and that's very powerful when you feel like you're actually seen in the space, and that's in the context of actors that slightly serve you. In Sleep No More, it's slightly more anonymous. Sometimes you get these special one-on-one experiences where they can react to you. but mostly it's you in a slightly passive role, but being able to move through it, and in Then She Fell, you have more intimate moments, and that eye contact that you give is incredibly powerful, and I spoke to Marissa, I forgot what her last name is, and she was the one, she works for Third Rail actually, who made Then She Fell, and she talked a lot to me about the nervous system, the perceptual response, and how that connects to what they do at Third Rail, and how they make people feel comfortable in the space, how they meet people where they are, how they know when to back off a little bit or know when to go forward a little bit. But they are able to actually have live actors to react to that. In VR, if you bring characters in, the AI is not great enough to have that reactive space with you. And we're actually working on something that I can't talk about yet that does incorporate live actors that we'll talk about soon, and I'm excited. But in social experiences, you can think about how to create spaces for avatar or people generally to have connections with each other that feel pretty powerful and For instance in Coco VR. I always found it really fascinating because you have eye contact and you don't really blink So I took a lot of people through it and a lot of press through it and people you know I met for the first time so I took a lot of people through the land of the dead on this adventure in Coco and The eye contact is so intense because there's the skeleton avatar that just looks at you So when you look at each other you eyes follow each other And I kept on wondering after I got out of the experience. I'm just like, why was it so intense with the person that I, you know, I feel like I know this, this press person so well now after 15 minutes of through the land of the dead. And after I tried it again, I realized it was just the intensity of eye contact that we had to each other. Cause in real life, you look at each other, but then we kind of look away and that creates a little bit of a comfort. And when we like stare at each other for a really long time, it creates a slight discomfort, but it's interesting. It affected, Me and the connection I had with that person, it felt much more intimate because of that eye contact. And we're not there yet with eye tracking and consumer VR to be exactly normal. The eye stuff is, I think, both Third Rail and Punchdrunk who made Sleep No More, they use eye contact as a huge part of their experience. and actors are trained to understand how powerful that can be with people because we don't have that on a day-to-day basis unless you have a significant other where you sometimes stare at each other's eyes. It's rare to have that. Eye contact for people, especially in an intense way, can be slightly uncomfortable but also incredibly powerful and incredibly intimate and just gets to your emotional core. It'll be interesting once we get to a place where you have that eye tracking that is perfect to your eye, how much more connective our experiences can be with each other. But one thing I researched a lot within this talk is just how easily our minds fill in the gaps with things we know. You know, smaller emotive things can actually connect us really deeply and you imagine that you're looking at each other. you imagine that there's eye contact there. It's powerful how much we put context on things from just simple lines and simple gestures because we're so equipped to do that.
[01:00:45.021] Kent Bye: Yeah, I actually did an interview with one of the actors who played Lewis Carroll in the Then She Fell, one of the Third Rail productions. And the amount of ability that he has to be able to read people in a way and be able to interact with them and be able to judge. And it's just sort of like this embodied experience of really knowing how to sort of interact with people in that environment. And one of the things he told me was really fascinating. He said, it was like every show I ever did, there was something that I had never seen before that happened. And so it was sort of like this unending diversity of human experiences that almost become, like when you think about trying to translate what they do into AI, it's like, oh, wow, there's something very unique about the human capability to be able to respond to these varying different types of behaviors. And that will likely always have the dimension of human interaction. So I'm really curious to see what Oculus and Facebook is working on in terms of some of these experiences with Immersive Live Actor. So I'll be ready to see it when it's ready to go. But just to kind of wrap up things here, I just wanted to ask you a couple more questions. One is, what do you see as some of the biggest either open problems you're trying to solve or open questions that you're trying to answer?
[01:01:51.472] Yelena Richitsky: Yeah, I think it's, oh, there's so many. My mind is always full of questions, which is why I continue to investigate so many different kinds of things. But I think more recently, because I've been doing so much of this investigation, is how do the experiences we create shift the behaviors that we have and how do we design for places that create the positive behaviors that we want to see. I always love being completely surprised by what ideas creators have. I'm a huge fan of like Tender Claws. I think they're doing really interesting things and think about VR in a very interesting, specific way, especially with these concepts around the metanarrative and spatial storytelling. I think in our conversation today we didn't talk about storytelling, which is strange because I usually talk about that all the time, but I think all of this fits into storytelling. It's just figuring out what, again, is storytelling within this new context of spatial creation, spatial storytelling, and the Story you're creating yourself as you're being active within the experience and this can be another conversation we have but Samantha Gorman talks a lot about story for spatial design and I'm in constant pursuit of that perfect place between interactivity and agency and that slight narrative because that's where you can really Have some level of authorship around a person's experience that can be pretty powerful coming from an artist's eye So that's really a big one And yeah, just keeping on understanding what feels good to us and why. So.
[01:03:35.959] Kent Bye: Great. And finally, what do you think is kind of the ultimate potential of virtual reality and what it might be able to enable?
[01:03:46.167] Yelena Richitsky: Always end in with the easy questions. It's everything. There's not a limitation to it. The limitation is what we put on to it. And I think I like what Michael Abrash, when I was in conversation with him for the research of the talk, also, who's our chief scientists at Facebook Reality Labs as a company with Oculus. We're basically building the tools to make things possible, and it's up to the creators to show what can be done with that. And there is not really a limit to what that potential can be.
[01:04:20.002] Kent Bye: Anything else that's left unsaid that you'd like to say to the VR community?
[01:04:26.688] Yelena Richitsky: Thank you for everyone who is making work and continuing to push things and making us understand things in different ways and surprising us. I love to see the really boundary-pushing stuff, so keep on pushing the boundaries. And thank you, Kent, for always being awesome and for giving me a space to do a lot of this investigation with all of the podcasts I've done.
[01:04:50.352] Kent Bye: Awesome. Well, thank you so much.
[01:04:52.893] Yelena Richitsky: Thank you.
[01:04:54.167] Kent Bye: So that was Jelena Richesky. She works for Oculus and she gave a presentation with Isabel Tuas called the Hierarchy of Being at Oculus Connect 5. So I have a number of different takeaways about this interview is that first of all, well, I tend to think about concrete reality as being grounded with you're on the earth and everything is centered on the earth, and I suppose that that is the way that we typically think about the world, having physics and then chemistry and biology and then psychology, and that's sort of the reductive materialistic perspective, and that really, and what they're proposing with this hierarchy of being is really centering everything with the human experience, with the emotions, with the phenomenology, but also your avatar embodiment and your expression of the self as being the primary fundamental aspect of a virtual reality experience. you being embodied within an experience, and what is your embodiment, and how does that embodiment change your relationship to everything else that follows from that? And I think the research that Mel Slater has done, as well as different people like Jeremy Balanson, and Jaron Lanier, and all these different researchers looking at embodiment, that there is a bit of a phenomenological centering of everything being centered into the human experience. It's the human-centered design. And so looking at things through this phenomenological lens, you start with the self, and then you move up into the environment and the context. And each of the environment, you have options to be able to put different objects. And a big thing that they said during the presentation is that each objects have a certain final causation. So what you can do with that object, and you tend to embody the actions that that object was supposed to invoke. So you have this idea of what this object is and what it means in the real world. And so you're going to then invoke these different behaviors based upon that, but that all of that context and behaviors is really aimed towards driving some sort of phenomenological experience within the user, whether it's an emotion or feeling or some sort of embodiment of a deeper story. And then finally, once you get past the context, then you get into the social interactions and what it means to be able to actually connect to other people. So I think that this hierarchy of being model is something that really stuck with me after I saw this presentation and it really reoriented the way that I was thinking about things. I had already been thinking about different things like whether or not consciousness is fundamental or primary and it's kind of on the argument of saying that consciousness is a fundamental aspect and that when you are designing a virtual reality experience, that you start with the self, you think about different ways that you're going to provide different embodiments and how that embodiment is going to be kind of at the center of an experience of how you're going to interface with the rest of the context. And people who have been able to use the Snapchat filters or Instagram filters will have this experience of seeing how having something mirrored one-to-one with whatever you're looking at, is going to be able to change your behaviors in the moment. And there's this live real-time interaction of that. And the first time I really experienced that was with The Wife of Us when they were really modulating your voice and you're changing their different embodiments. And I just was able to find different aspects of myself within a virtual reality environment. But also Snapchat filters was probably the other time that I felt this experience of seeing how looking at myself in a reflection, but having these different embodiments would bring up different aspects of myself. And I think that's the power of what both virtual and augmented reality is going to be able to do, is that through these different avatar representations, whether it's a filter or a full-on embodiment within VR, it's going to be able to allow us to tap into different dimensions of ourself and that we're going to be able to put on these different clothes to be able to really tune into different emotions that we're feeling or that we want to project and have other people relate to us in different ways. And that was one of the things that Yelena said is that, you know, she's in charge at Oculus of trying to do these really innovative experiments and working on these different projects where she told me at Sundance and again here in this interview that they've been working on a live immersive theater type of experience using live actors, which I'm super curious to see what that ends up being, but that she's very interested in seeing like what an artist depiction would be if you want to have a whole range of different emotions than what type of either avatar embodiments in conjunction with different environments to be able to really tune into these different emotional states. And I did a bit of a deep dive into different emotional maps that are out there, different cartographies of emotion, emotional wheels, and I'd recommend people to go check into some of those in my previous episode to this one that really dives into different experience maps, customer journeys, but also emotional mappings. And just finally, the thing that I find kind of interesting that's happening in the course of this conversation is this trying to bridge the academic research and the theory into the practice. And that's something that I've been trying to do with the Voices of VR podcast of going to the IEEE VR and talking to a lot of academics and it feels like we're getting to the point where the people who are creators of these virtuality experiences are looking for deeper critical theories of social presence, of embodied presence, what kind of insights can we get from some of the research that's been done so that can start to inform the creation of these different experiences. And I think that is what both Yelena and Isabel did in the process of putting together this presentation is trying to synthesize a lot of these different concepts of what we know about embodiment, how that impacts our relationship to space. I think it's a big open question that architects deal with all the time, which is how do you translate a space into a feeling? And I'll be diving into a lot more with talking to Robin Honecke about how she's been doing that. And what essentially seems to be the answer is that you do it through iteration and practice, and you just both create a space and put a lot of different people in it and see what they feel, but also in the process of game design, it's this process of iteration, of creating these different interactions and behaviors of different objects, and you're solving these different puzzles and playing these different games, but at the end of the day, it's all going back to what the experience is of the individual and what emotions are being evoked from that, and what type of sense of embodied presence you can get from everything else. But that translation between, uh, these interactions, the context, the environment, our architecture and space down to emotion is not like an exact science. It has a lot to do with someone's direct experiences, what their own associations and meanings are, but also if there's these universal archetypal forms that are at these different levels, can those get translated down into emotion and feeling? And I think the closest thing that I've found at least is music and the different musical chord structures and how the different musical chords evoke different feelings. there's different colors, there's everything that has maybe a transcendent quality that is able to do that translation. But there's also things that are just completely unique to an individual about whatever context and meaning based upon their experience that they've had, and how you can take something that is a symbolic representation of that in a VR experience and be able to communicate some deeper feelings. And one of the experiences I saw at the Vancouver International Film Festival that did a brilliant job of this was Paisley Smith's Homestay. And that's actually going to be featured at Indicade coming up in LA for anybody that's going to be there. I'm actually on my way to LeapCon, Magic Leap's augmented reality conference that's happening in Los Angeles. As I publish these next two episodes, I'm going to be hopping on a plane and going down to LA for that. And then going to a couple of LA art shows that are also happening. And check out some different location-based entertainment experiences as well as I'm in LA. So I'll be going to LA for the next week or so. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast, and if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listeners-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
