Play episode

#100: Sébastien Kuntz on Virtual Reality Presence & Lessons from 13 years in VR

March 6, 2015by kentbye

Sebastian-Kuntz

This is the 100th episode of the Voices of VR podcast marking 33 hours of interviews with the leaders of the consumer VR revolution over the past 10 months.

I wanted to celebrate the 100th episode with an interview with Sébastien Kuntz because he helped inspire me to start this podcast. Sébastien has been working in virtual reality for 13 years doing everything ranging form training simulators, 3D engine development, and producing a middleware VR solution with MiddleVR.

I first discovered Sébastien’s work during the IEEE VR conference last year because he was tweeting about different presentations talking about the academic community’s response to the Facebook acquisition. Here’s a couple of examples of his tweets that captivated my attention:

I wanted to hear more from Sébastien and attendees at IEEE VR, but there weren’t any consumer VR publications covering what was happening in academia or with VR researchers. In fact, there was hardly any coverage from any publication of last year’s IEEE VR conference beyond tweets from attendees, with the most prolific being the ones from Sébastien.

Because of this lack of coverage, I decided to start my own podcast. I reached out to interview a couple of other attendees of the IEEE VR conference including Eric Hodgson and Jason Jerald. I also really wanted to hear more from Oliver “Doc_Ok” Kreylos who was a respected commenter on the /r/oculus subreddit, and also happened to be working in VR within an academic context.

I also wanted to hear more from D of eVRydayVR, who is a computer science graduate student and made a number of amazing tutorial videos on barrel distortion, low persistence, and time warp.

The Voices of VR podcast was born and seeded with these more academic insights and perspectives into VR, and so I’m really looking forward to being able to travel to France at the end of March to cover the 2015 IEEE VR conference.

Sébastien has a great blog on VR with summaries of a lot of interesting VR research. He pointed me to Mel Slater’s research into virtual bodies, and this video summary of Slater’s research into the Positive Illusions of Self is one of the most fascinating VR videos that I’ve seen:

I also came across some quotes about presence in VR from Mel Slater on Sébastien’s blog that inspired me to write an extended essay about it in VR that I’ve referenced in a few tweets:

A number of people have asked me where that excerpt was from, and it’s from an unpublished essay that I’ll share here so that people can more easily link and reference to it.

THE FUTURE OF IMMERSIVE VIRTUAL REALITY
Just as the Internet and mobile computing have changed every aspect of society over the course of the last 20 years, Virtual Reality is poised to have a similarly pervasive effect on our lives.

The types of visceral experiences that VR can provide are truly unique, and on the whole constitute a new communications medium that will amount to the Gutenberg Press of the 21st Century. Felt experiences will be able to be captured and shared just as books were able to capture and share information and knowledge. Just as new insights into perspective catalyzed breakthroughs in Renaissance art, then adding a new immersive dimension to computing will likely spur a similar revolutionary change in what types of experiences that can be shared through virtual reality.

In order to investigate what types of new doors and interactions that VR will be able to provide to society, then it’d be helpful to first dive into the key components of what constitutes “virtual reality.”

WHAT IS IMMERSIVE VIRTUAL REALITY?

There’s a pretty big difference between 3D virtual worlds that are experienced through a 2D screen, and a completely immersive, virtual reality experience. There have been 3D environments in computer games since the early 90s, and interactive virtual worlds like Second Life since 2003. While immersive virtual reality includes some components of virtual environments, it also provides other elements that are completely new and different that transcendent what is possible through a 2D medium.

Sébastien Kuntz defines Immersive Virtual Reality as “the science and technology required for a user to feel present, via perceptive, cognitive and functional immersion and interaction in a computer-generated environment.”

There is a sense that you’re transported into another world where your unconscious parts of yourself are fooled into believing that the computer-generated reality is an actually real. Within the VR community, this is widely referred to as the experience of “presence.”

VR researcher Mel Slater defines two key components that are necessary for people to have a realistic response to a virtual reality environment. He says, “The first is ‘being there’, often called ‘presence’, the qualia of having a sensation of being in a real place. We call this Place Illusion (PI). Second, Plausibility Illusion (Psi) refers to the illusion that the scenario being depicted is actually occurring… when both PI and Psi occur, participants will respond realistically to the virtual reality.”

Some of the elements that trick your perception into believing that you’re in another place are real-time interactions like head tracking where your physical movements are mirrored within a virtual 3D environment to the point that they match your expectations. Even though your rational mind may realize that you’re not in another world, your unconscious perceptions and primitive limbic mind will react as if the environment was real.

The second component of plausibility is when your cognition is fooled. As Kuntz says, “Everything that happens is coherent. You actually believe you’re there, your actions have a credible impact on the virtual environment and your sensations are affected by it.”

Attaining a sense of “presence” is the ultimate goal of a VR experience, but there is no fixed set of ingredients that reliably produce it. What is known is that there are a combination of components that together constitute “virtual reality,” but taken individually are merely a part of a virtual world.

If I were to boil down presence to an equation, then I’d say it’d be this:
Presence = Place Illusion + Plausibility Illusion

Sébastien and I talk about how the Crescent Bay demo really only had half of this equation with the Place Illusion. Without input controls, then there’s no ability to feel like your actions have an impact on the world and makes it more difficult to achieve the illusion of plausibility. So for the both of us it wasn’t able to achieve that full sense of immersion and presence.

This interview happened at Oculus Connect last September, and so neither one of us had seen the latest Valve Vive demo yet. Based upon the reactions to Vive, it’s clear that having an accurate tracking and input control system takes VR presence to the next level. We discuss our understanding his presence as well insights from his extended experience in VR, as well as his reaction to the Crescent Bay demos.

  • Middleware abstraction of the input trackers to use a wide range of input controllers. Also be able to determine things about the user
  • Input controllers and input devices and how should developers should approach implementing input controls, and natural hand interactions
  • What is presence has two levels of being immersed cognitively, and then there’s a lower-level of immersion at the subconscious perceptual level.
  • Virtual body being transferred to a virtual world for virtual therapy for phobias. You can also have marketing studies within VR because people act naturally. Engineers can test the ergonomics of physical designs & architectural spaces
  • Connecting the space to your interactions. You can simulate products and spaces at scale.
  • Don’t want to use VR to escape reality, but use it to improve reality. Use VR to test an assembly line with a virtual body. You can training simulations where you can actually practice the gestures that you need to do. You can have therapy towards phobias. You can use it to build empathy for others.
  • Crescent Bay VR HMD was the best that he saw up to the point of the Oculus Connect. The input device was missing from the experience.
  • Lightsaber demo with Sixense built even more presence. VR presence is about both Immersion into the Virtual world as well as Interaction with the virtual world where your actions make a difference.
  • Reflection of the VR community and how much that it’s changed over the past couple of years. It’s easier to see your own body and collaborate with other people in a CAVE environment. Having your avatar in VR and play around with your identity, which will make it easier to collaborate.
  • Telepresence within VR and social experiences in VR with haptics. Experiment within a virtual bar where two avatars are arguing, and they look at you and ask you to weigh in. You can tune into emotions with their body language in these virtual environments.
  • Most compelling VR experience where you are put into the body of wounded soldier, and your legs have disappeared and you’re waiting to die. Then an avatar comes to help you, and people smile are really grateful. Presence wasn’t broken because interactions were limited. If a virtual hand goes through a wall, then that also breaks cognitive presence, which is harder to maintain because it’s like a house a cards. It takes a long time to rebuild this sense of presence at a subconscious level once it’s broken. Have to make sure that your brain accepts the rules of these virtual worlds, which is more difficult.
  • No bodies were in the Crescent Bay demos, and you were standing. Want to be able to track your entire body. Sitted experience is a limit due to the hardware and what you can do with creating a sense of presence. Trying to recreate a whole reality. Can’t simulate everything yet because we don’t understand everything yet. Realistic rendering makes the mind demand that everything else is realistic including the sound and haptics, otherwise everything will fall apart.
  • Showdown demo where you’re moving through the space through slow motion. Felt like a ghost, and so anything could happen. That helped him accept it. If he could interact and see interactions within VR, then it’d be even more believable.
  • Really liked to interact with the alien in the Crescent Bay demo. Also really like the T-Rex coming at him because it was beautiful and scary.
  • New Consumer VR vs. Old VR communities. Misconception with VR is that everything VR died 20 years ago, but it’s still being used in professional situations as well a lot of VR research. Need to do new research with the new hardware, and great to see Facebook and Oculus are investing in new research. Working with the human mind and perception and not just hardware. We don’t understand how we perceive the environment.
  • Controlling perceptions to take us into another world, is that just escapist of going into fantasy worlds. VR can be used to improve reality. Stanford research where you see you virtual avatar and how nicer avatars can improve your sense of self-image and confidence. You can transform your self-image. It can also be used to escape reality as well. It’s up to society to decide. TV is probably worse than VR because it’s more passive.
  • Crescent Bay demos were pretty passive and not very interactive. Adding a tracked joystick is easier compared to achieving a low-level sense of presence. Then we can start a new frontier of how to interact with a 3D interface
  • Mel Slater’s research into the impact of virtual self, and how your brain accepts a virtual body. Your brain accepts it in seconds shen you hands move. Experiments of putting your virtual body into different ages and races, and it reduces racism because it builds empathy.
  • 3DUI book by Doug Bowman. Go to IEEE VR conference. There are a lot papers online. They’re friendly and they want to help the VR community.
  • Last IEEE VR conference happened right after the Facbook / Oculus acquisition. How could Palmer create a VR HMD, and the academic community couldn’t do it? Too reductionistic in the academic community and they have to be cautious and make incremental progress.
  • The Metaverse will change the world because it’ll provide a new way to communicate with your friends. People will recreate reality in VR because that’s what happens with a new communications medium. Unlock our brains to embrace the new possibilities, and looking forward to being a part of the VR development community.

I’m looking forward to many more episodes of the Voices of VR, and I hope that you’ve been enjoying them.

Theme music: “Fatality” by Tigoolio

Subscribe to the Voices of VR podcast.

Share this
FacebookXRedditEmail

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast.

[00:00:11.998] Sebastian Kuntz: My name is Sébastien Kuntz and I've been working in VR for 13 years now. I started my career on the French railways working on training simulators. Back then we were not working with head-mounted displays but rather with big screens. We also had treadmills and gloves to interact with. Then I worked for a 3D engine called Virtuals. It was very popular back in the time. It was like Unity now, really. And it was both, and then I left. And then I started my own company called Middle VR. And our goal is to simplify the creation of virtual reality applications, really.

[00:00:43.763] Kent Bye: Yeah, and so middleware is something that I think is really important in terms of having a unified interface to be able to talk to all these different input devices. So maybe talk a bit about what you've done to solve that issue.

[00:00:56.946] Sebastian Kuntz: So we've done a lot. We have actually several layers of abstractions. So first we abstract the trackers layer, for example. So if your application is using one tracker, you can use any other tracker out there. So you can use the Razer Hardware, Leap Motion, Kinect and high-end 3D trackers also. And then we have a second layer, which is the user layer. So instead of your application asking for, OK, please tell me this tracker what's doing, you can just ask, OK, where is the user's hand? Where is the user's head? And you don't even have to care about trackers anymore. You're just asking things about the user.

[00:01:31.399] Kent Bye: Yeah, and so it seems like VR input is one of the biggest open questions and, you know, I think a lot of people were kind of hoping for some direction of a big announcement here at Oculus Connect, but yet Oculus is clearly working on something but no big announcements yet. And so we're kind of left in this sort of limbo state of not really knowing how to approach doing something once and doing it efficiently. And so I guess, you know, when you look at all the different VR input devices, what would you suggest would be a viable approach for developers to say, hey, I don't want to have to use each individual SDK to implement all of these. I just don't have time for that. But is there a way to kind of do it in a way that is abstracted that you could implement it once and then have it work with multiple devices?

[00:02:17.815] Sebastian Kuntz: It depends on the capacities that you want your interactions to have. So I see maybe two different ways. You can either have a generic joystick that is tracked in space, so you have a joystick in hand and you can track it in space and you have buttons and axes on top of that. So that's what we're relying on currently. All our interactions are based on what we call a wand, like a magic wand. And so it's all based on this typical track joystick. So that's what we get in the industry. In all the caves you have a wand, you have buttons, you have sliders. So we get navigations with that, you get manipulation, selection of objects. And I think the second one would be just hand interactions. So that's the kind of two directions I see, like you have natural hand interactions and maybe a joystick with buttons. That would be the second thing.

[00:03:05.042] Kent Bye: I see. So when you think about virtual reality, I know that you've done some writing about the sense of presence and looking at Mel Slater's definitions that he used. How do you typically describe to people the two big components of presence?

[00:03:21.163] Sebastian Kuntz: So we could say that presence has two levels. So the first level would be the same level as what you get when you're playing games, reading a book, watching a movie. And this would be cognitive presence, like you're really fooling the mind, fooling the rational brain into thinking he's somewhere else. But what really VR is adding is this low-level perceptive presence. You're really fooling the body, you're fooling the senses, so you're replacing vision, touch, etc. And VR is really the only medium to add this perceptive layer, and that's something very unique.

[00:03:55.688] Kent Bye: Yeah, and so I guess to me it feels like it's a completely new communications medium. And so what do you see as the strengths of being able to trick our subconscious minds into believing that we're in another world?

[00:04:07.934] Sebastian Kuntz: Well, you have a lot of applications that rely on this sense of presence. So the first one would be you have your virtual body transferred in this virtual world, And then you start to fear for your virtual body. So you can have therapy, phobia, helping people with that. Then you can have training in dangerous situations but you're in fact in a safe environment. And then you can have marketing studies because you're really reacting in a natural way when you're in VR. Then you can have engineering also, testing visibility in a car, validating the design, testing architecture design. How do you feel when you're inside this virtual architecture? Do you feel the volumes? Do you like the textures? How do you feel in that really?

[00:04:50.251] Kent Bye: Yeah, so it seems like the spaces and our reactions to environments is something that I haven't really paid as much attention to until looking at virtual reality that I've started to pay a lot more attention to architecture and spaces. And so how in the virtual reality community do they start to make that connection between how does the space impact your reaction?

[00:05:11.894] Sebastian Kuntz: Yes, so that's the thing with VR. When you're interacting with a 2D screen, you have objects in front of you, but they're not necessarily at the right scale, not the right volume. And with VR, you can simulate any kind of product or environment. So it can be a very small object, it can be a car, but it can be a huge environment. It can be a complete factory, it can be a huge boat, it can be a planet, it can be the whole universe. And you can really play with this volume, change the volume, and things that you could not do before.

[00:05:38.503] Kent Bye: So here at Oculus Connect you were giving a presentation and so what were some of the big points that you were trying to make in your presentation?

[00:05:45.270] Sebastian Kuntz: The biggest point is that I would like VR to be used not to escape reality, but to improve reality in return. So we see that with engineering, you can use VR, for example, to test an assembly line before it exists, and you can test that with a virtual body. And the virtual body will tell you, OK, maybe if you do this kind of movement, they are not so kind for the body. Maybe we'll start to have back problems. Maybe we'll start to have articulation problems if you do this gesture too much. So then you can redesign the assembly line so that you can really help people that are working in this environment. Then you can have training. So instead of training in the classroom with CD-ROM or videos, you can actually practice the gestures that you're going to do. Or you can have therapy that really helps you overcome your fear of heights, fear of spiders, fear of planes. Or you can also raise empathy towards some disabilities, for example, you can live the life of somebody who is in a wheelchair, for example, and you can see, okay, yes, that's really a tough life. And maybe I can do something to help those people, and that's really useful for that.

[00:06:48.496] Kent Bye: And here at the Oculus Connect, we saw for the first time the Crescent Bay, which is sort of the next iteration on the trajectory towards the CV1, the first consumer version of the Oculus Rift. And so I'm curious, since you've seen just about every head-mounted display out there, how this sort of weighs against other things that you've seen.

[00:07:09.123] Sebastian Kuntz: In March I've been able to test Sony Morpheus, Oculus DK2, and Valve's prototype, and now that I've seen the new one, Question Bay, that's really the best experience I've had in my whole life, really. So you have great resolution, you have great tracking for a really large tracking area, And also the content that I was showing, the demos, were just beautiful and was made by really great 3D artists and it really blows my mind. And the only thing that's missing is the input device. How are we going to interact with that? So it was a bit frustrating because you were just looking around and waiting for things to happen but you could not interact with it.

[00:07:46.986] Kent Bye: Yeah, I had the exact same reaction in terms of, you know, when you look at your hands and your hands aren't there. It's like the classic virtual reality thing that whenever you put someone in the Oculus Rift, they sort of like lift up their hands, you know, hoping that they're there. And so, you know, it's like some part of me that was hoping for my magical like hand tracking to be there, but it wasn't. And so I did feel like it was a little bit disengaging and I just did the lightsaber demo here with the stem and for me that was much more engaging and I felt right there because you know there's like a little subtle haptics when the droid ball was like shooting lasers at me and I was blocking them with my lightsaber and just like shaking my hand and yeah I think that being able to interact and engage in environments in a way is going to be pretty key to feeling that extra sense of presence.

[00:08:33.187] Sebastian Kuntz: Yeah, but exactly for me, virtual reality is about first immersion, so the input that you get from the virtual world and interaction, the input that you put on the virtual world. And so we're currently missing half the fun.

[00:08:46.825] Kent Bye: Yeah, interesting. And sort of looking at this field, since you've been involved with virtual reality for so long, you know, it seems like the last two years, relative to the last 13 years that you've been in virtual reality, maybe you could just sort of like talk about how quickly things have just changed relative to all the time that you've been in virtual reality.

[00:09:06.131] Sebastian Kuntz: Yeah, so before those two years, the hardware was progressing really slowly. So there was clearly an acceleration even before the Oculus Rift and trackers were getting really cheaper and projectors were getting cheaper. But hand-mounted display really was stagnating and we were just waiting for that. So instead of using hand-mounted display that really were not working, we were using what we call a cave. So a cave is really a big room, typically a 3 by 3 meters room, and all the walls of this room are projected with a 3D projector. And you have nice tracking in that, and this really gives you a great immersive experience. But this costs a lot, you have to have multiple projectors, you have to have multiple computers, but really the immersion is really great. You can get a lot out of the applications that you were doing in caves, now you can do that with head-mounted displays. But I think what we will get is really the two going on together, because with caves what's really interesting is that you can see your own body. So you just have glasses like in the 3D theater, 3D movies. So you can really see your own body. So that's the first point. And the second point is that you can have people around you and you can see them and you can talk to them and you can interact with them. So that's really currently much more easier to collaborate with them and look at them in the eyes and interact with them more naturally. But I think we will get there with head-mounted displays also as soon as we can get avatars of ourselves and avatars of the other guys. And this will have two profound impacts, is that first you can have a different body that you currently have. So just for example in engineering, if you want to test the visibility in a car, you can test with your own body, with your own height, or you can test by being somebody smaller, or you can test by being somebody taller. And you can immediately see ergonomic issues, visibility issues, by being in the shoes of somebody else. And the second one is that you will have distant collaboration. So the people don't have to be in the same space. They can be anywhere in the world and you will be able to collaborate and work with them as if they were here.

[00:11:04.232] Kent Bye: Yeah, that seems to be one of the biggest themes that I've heard over and over is the social aspects of virtual reality and you know what was probably more broadly termed as telepresence in terms of projecting your presence into a virtual space and being able to feel like you're really actually present with other people. And so I'm curious, since you've been sort of jumping around from both the corporate and into this new consumer VR movement, what you've started to see that's really compelling when it comes to these social experiences?

[00:11:34.095] Sebastian Kuntz: So I've seen multiple really industrial applications in which you could be with somebody else and even with haptic devices and you could interact with multiple people and actually touching them and feeling the feedback of the other people being remotely there and that's really something incredibly strong. And this works also with non-human avatars, so really some AIs or something like that. And I like to tell the story of how I went into an experiment in which you're in a virtual bar with two virtual guys that are arguing over which football team is the best. And they were really getting angry at each other because Oh no, mine is better. No, mine is better. And you really feel tense to yourself because we are social creatures and we get to feel the emotions of others. And at one point they just look at you and they say, okay, which one is your favorite team? It's like, no, no, no, I'm not answering that. You're going to shout at me. I don't want that. And so it will do the same with real people because you can feel their emotions just by reading their body language. And that's something that's currently missing. So currently, Maybe you can just see people moving around the head. But even just that, that's really great. You can see maybe just a head floating, but you can see where he's looking, if he's looking at you, if he's nodding. And just that is really great.

[00:12:48.403] Kent Bye: And so what's been one of the most compelling virtual reality experiences that you've had?

[00:12:54.245] Sebastian Kuntz: Well, apart from tonight, before that, for a long time, my favorite experiment was one in which you were transported in World War I. and you were put in the body of a wounded soldier. So you were on the battlefield, so you opened your eyes and in real life you were sitting in a dentist chair. You can really smell the powder and you can hear the sounds and it's also vibrating. And so you're just sitting here and the excuse that they have so that you cannot navigate is that your legs have disappeared because of a bomb. So you're just lying down here and waiting to die because you have. bombs falling all over, people running, shouting. You really have no idea what's going on. You also hear your own heartbeat. And at one point, you have a rat coming on your body. And you can feel it because they've done something to your body. And you really feel like, oh my god, I'm going to die. That's sure. There's nothing I can do. And at one point, you have an avatar coming to help you. And you feel that you're being dragged by this guy. And I've seen a lot of people looking at this virtual avatar and smiling at him. Like, naturally, it's like, oh! Thank you for saving me. They were really in this world and just in the end to die. And it really worked because for the whole experiment presence was not broken. And it was not broken because the interactions were limited. I think what we will see soon is As soon as we get 3D input devices, we'll have lots of experiences in which we will break presence because your virtual hand is going through a wall or something, so your brain will say, oh no, that's okay, that's a virtual reality in real life, my hand cannot go through a wall, so this will break presence. So that's the kind of details that you have to really make sure really work. Otherwise, this will break cognitive presence, and cognitive presence is really the hardest to maintain. If you think about perceptive presence, that's really a matter of how your perceptive sense works. So if at one point you're near a high cliff and there is a glitch, but then the high cliff comes back, you still have fear of heights, even if there was a glitch, even if there was a breaking presence. But with cognitive presence, it's much more like a house of cards, as Brendan said this morning. If you break something, everything falls apart, and it takes a lot of time to rebuild this trust, this subconscious acceptance of the rules of this world. So, this is going to be the hardest part, I think, in VR. Not so many people are aware of that currently, but it's very low level. Okay, you have to have very low latency, very fast FPS, so that's right, but then we will get to the point of, okay, you have to make sure that your brain accepts the rules and the consistency of this world, and this will be even more complicated.

[00:15:36.482] Kent Bye: Yeah, I think that's the thing that comes to mind is that in this Crescent Bay demo There is no body you're completely disembodied and that there's no skeleton either and for the longest time oculus was saying that you know, this is designed to be a sit-down experience and you're not meant to turn around a full 360 and yet and In this demo that we just saw, you're standing up, you can turn all around 360, and it's not like you're actually standing up, you're not sitting down at all. So that seems to be a shift, but yet at the same time, not having a body, I think it presents a new issue of wanting to have a full skeleton, of tracking the full body, so that when I'm crouching down or leaning down, at least it's creating some sort of inverse kinematic representation of that, so that I can at least see my limbs or my legs moving around a little bit and the absence of that just made it feel like I was kind of floating and it did break the presence for me.

[00:16:28.789] Sebastian Kuntz: Yeah, but really at first they wanted to have a seated experience because they knew the limits of the hardware and you really have to know the limits of the hardware when you're designing a VR experience. Because if you're going off the limits then presence breaks. So they say okay, these are the limits you are seated and then you have a much better chance of having something good. But what we're really trying to do is in fact recreate the whole reality and this will take time so we have to go step by step and we cannot simulate everything right now. We don't know how to do that. We have no idea how the brain perceives that. we have no idea how to computationally do that. And so that's how Brendan puts it this morning, like, we will do incremental steps. And that's the issue. Every time you have something new, you will see the boundaries of this new thing. And so when we have virtual body, there will be something else missing. Maybe it's not realistic enough. Maybe the rendering is not realistic enough. Maybe the movements are not realistic enough. maybe you're going to bump into the walls, and so you will need some kind of feedback of the real world instead of just hitting the walls. The other issue is that if you have very realistic rendering, your brain will expect that everything is realistic, like physics, touch, sound. And so if any of this part is not coherent with the rest, then everything will fall apart. So it's easier to start with maybe a cartoon environment because When you're watching a cartoon, you really accept the rules of the cartoon and that simple drawing and the physics environment is also much simple and you just accept that everything is consistent. But once you start to have a realistic environment, you start to expect everything is real. So that's complicated.

[00:18:11.145] Kent Bye: Yeah, one of the last segments of the Crescent Bay demo was the showdown from the Epic Games. And the thing that's happening is that you're actually moving through this space in slow motion. Physics is sort of turned down by 10 or 20x. But at the same time, you still get a sense of being there and dodging from the rocks and the bullets. at a subconscious level, but I'm curious when you mentioned about this whole reality of this photorealistic versus cartoon, that was an example of something that was very hyper-realistic, but at the same time the physics were all tweaked. So what was your experience of that?

[00:18:47.942] Sebastian Kuntz: I felt like a ghost. I had no body and I could not interact. So I'm a ghost, so anything really can happen. So my brain just accepted that this way. Maybe if I were to interact with it, if my body touches, for example, another guy, I would expect it to react. Or if I touch a rock, I would expect it to move. And I'm not sure we're able to do this kind of physics interaction so perfect. So we will run into that kind of problems also.

[00:19:14.595] Kent Bye: I see. And were there any specific segments of the Crescent Bay demo sequences that stuck out for you as being particularly salient?

[00:19:23.019] Sebastian Kuntz: My favorite was with the alien because first it's beautiful and then when you move around his gaze follows your gaze. So he's really looking at you and he's trying to communicate with you and that's really interesting because You want to interact with him also. We are social creatures and somebody talks to you, then you have to answer that. You cannot ignore people, that's socially unacceptable, so you have to talk to this guy. But you cannot communicate with him, so you're trying to be like, maybe he will react to something, and no, it's not reacting, so that's very frustrating. But at the same time, you're kind of meeting an alien and you think, oh, maybe that could happen really this way, not being able to interact with him. And then the second one was the T-Rex. It's really impressive to have him come here in a very cinematic way, like Spielberg way of coming slowly at you and you have the sound and it's so beautifully modeled in 3D and realistic and very scary. And it reminds me of the Sony Morpheus demo, also in which you have a dragon coming, so he's huge, he's just in front of you, and in the Sony Morpheus demo, the dragon eats you. And even though you know it's not real, you're really protecting yourself, and that's so powerful.

[00:20:34.162] Kent Bye: Yeah, and I guess one of the other things I've noticed within the virtual reality community is that it's a bit splintered in the sense of like this new consumer VR community had popped up, but yet almost kind of ignoring what was happening at the IEEE VR and the academic community and the corporate world for many, many years. And there was this kind of split where there's this new thing going on and not really integrating a lot of the speakers or lessons. Since you've sort of been walking in both of these worlds, how did you see that kind of playing out?

[00:21:02.767] Sebastian Kuntz: Yeah, in fact, I think the new community doesn't know that VR has been existing before. Everybody thinks that VR died 20 years ago, and nobody knows that VR has been used professionally in the industry and that lots of research has been done. So I think it should be us coming to this new community and helping them. So that's kind of the point of my talk yesterday was to say, OK, we've been doing stuff for the last 20 years. Here's what we've learned, the value of VR for these kind of applications. I also think that we need to do new research with the new hardware because the previous hardware was kind of limited so I think a lot of research has to be redone with this new hardware and that's really great to see that Facebook and Oculus are investing heavily in research and that they are leading the way because they will have lots of resources and they will be collaborating with research labs all around the world and I think that's really crucial because as we said before we're not really working with hardware or software we're really working with the human brain and perception And really, we don't really understand how we perceive the environment, and that's the key to recreating reality. And our brain has evolved for millions of years to perceive this reality. So if you're trying to present a virtual reality that is not as realistic as the brain expects, then everything will break. But we don't know exactly what the brain is expecting. you don't have to recreate the whole reality completely and our senses are flowed so we just have to learn the flows of each sense and take advantage of that to create better applications also.

[00:22:35.978] Kent Bye: Yeah, and the thing that comes to mind in listening to that is, you know, we're putting all this technology and controlling our perceptions and really taking us into these other worlds. And the question that comes up is, I'm sure a lot of people think of like, well, isn't that just escapist that you're just, you know, escaping reality and it's not actually helping you become a better person. And there is a sense of going and seeing a lot of the Crescent Bay demos. And on one hand, it's beautiful and great and amazing, but yet I can see the allure of just going into these fantasy worlds and being a little bit disconnected from reality in some ways. So I'm curious about your take, because being in virtual reality for a while, I'm sure you get this question a lot in terms of like, well, isn't that just escapist?

[00:23:22.974] Sebastian Kuntz: Well, there is mixed feelings about that. The first, I want to believe that VR can be used to improve reality. That was also the point of my talk yesterday. All the professional applications are really used to improve reality in return. So for example, there is research done by Stanford. in which you're getting in a virtual world, you go to a mirror and then you see your virtual self, you see your virtual avatar. And so in half of the groups you had a nice avatar and in the other half of the group you had an ugly avatar. And so it has been proved that people with a nice avatar had to have more friendliness, more extroversion, more confidence. And what's very interesting is that this feeling of confidence really transferred into the real life. Like one hour after the exposure in VR, they were told, OK, this is a completely different experiment. Go to these dating sites and date girls. And they've proved that the people with a nice avatar tended to date better-looking dates than they would before. So VR can be used to transform your self-image and transform the image that you have of other people, like for the wheelchair, as I said, but it works also for other disabilities. But yes, you could also escape reality with VR and then that's a philosophical question and that will be up to society to decide that. Is it acceptable for an individual that is not happy in real life to be happy in virtual life? How do you deal with that? I don't know that and I would say that maybe leaving people in front of the TV for hours, that's Worse, really worse than people in VR because in front of TV you're just completely passive, you're vegetable, you're just no-brainer. At least in VR you're active, you're doing some stuff. That's what I hope.

[00:25:05.564] Kent Bye: Yeah, I guess that's the thing with the Crescent Bay demos is that it was very passive. There wasn't a lot of interactivity and so I guess that maybe that's part of the reason why I'm like a little bit resistant to say, oh, that was like completely amazing because I could see that, you know, there's lacking that sort of interactive component that you said was a pretty crucial part of being in a virtual environment was seeing that your agency and your will is actually changing the environment in some way.

[00:25:29.610] Sebastian Kuntz: Yes, but I think we are really close to that. In fact, adding just a simple tracked joystick is very easy. So I think they did the hardest part, which was really the low-level presence, like getting perceptive presence. And then interactions, you can add that easily. I'm not saying that good interactions will be easy, but adding a simple input device should be quite easy. And then we will be able to interact quite easily with that. And then we can start this, again, new frontier of how to interact with the virtual world. And so there's already been done a lot of research about 3D user interaction. So we call that 3D UI. Look up for that on the internet. There is books. There is conferences just about that. And there is already a lot of things done about that.

[00:26:13.179] Kent Bye: Yeah, and when you're talking about looking in the mirror and your perception, it made me think of a lot of the work that Mel Slater has been doing and the research that he's been doing in virtual reality. And so maybe you could sort of summarize like the relevance and importance of stuff that he's been looking at.

[00:26:27.489] Sebastian Kuntz: So I think his goal is to study the impact of a virtual self. First, how does it work that your brain accepts a new body? So that's really strange when you think about that. We've lived for 10 decades with our own body, and when you go to VR, your brain really accepts this new body in seconds. And you would think that it would take more... much more time but no all it really takes is that if you feel your hand moving and you see a virtual hand moving then bam your body thinks okay so it must be my hand that's the only plausible explanation so and he's been studying what happens if you're in the body of a little girl if you're in the body of a black guy and he has shown that being in the body of black guy reduces, to summarize very quickly, reduces racism because you then relate to black guys and then you're much more friendly and much more positive. And so that really proves the power of virtual bodies and the weakness of your own mind that can be really easily manipulated if you know the rules.

[00:27:30.030] Kent Bye: And do you personally have a favorite virtual reality researcher or, you know, stuff that's out there that you really enjoy following or things that people should know about?

[00:27:41.358] Sebastian Kuntz: If you're just working with virtual bodies, you have to read the work of Millslater. So just look up for body transfers and you have multiple researchers working on that. If you want to look up for 3D user interactions, you can look up for Doug Bowman. He's written a book along with other researchers which is called 3D UI. So you have the whole basic research on that. I really think all the researchers in VR are doing a great job and you should go to conferences like you would have iTrapelliVR, it's always in March, it's usually in the US but this year it will be in France, so that's good for me. But there is lots of papers online, you can look up on Google, lots of things already existing and you can just contact them. They're really friendly and they really want to help the community. They know a lot of stuff that can be really useful and they're really smart guys and I really admire them. I always enjoy talking to them because you learn so much.

[00:28:37.629] Kent Bye: Yeah, I know at the IEEE VR that happened this March, it was right after Facebook acquired Oculus, and so it was something that was very fresh in the news, and I heard that it sort of really changed the tone of the conference. Maybe you could sort of comment in terms of what you saw come out of that conference.

[00:28:54.961] Sebastian Kuntz: It was really funny because I think nobody expected that and then everybody was asking, okay, so how come this 19 years old boy managed to do a head-mounted display and our community could not do it? So that was really a shock and then they realized, okay now this will go much quicker than we anticipated and One of the thing is, researchers are only so much. Maybe there is 100, 200 people working, researching about VR. And then you have these hundreds of thousands of hackers that will discover new rules, but each and everyone will discover just one tiny part. And we will mix everything together and after maybe a decade everything will have trickled down and we'll have clear rules about what VR is. So you have really the two ways of discovering this thing. You have the academic ways that's kind of coming slowly because they have to publish papers and of course they have to be very cautious about what they do, so they have to, it takes a long time to build a really good research experiment. But hackers, they don't have these constraints. They can just test ideas, but of course maybe it's not Proven scientifically, but it's something that you feel that you you think it's going to work and then you apply that and after five years Okay, everybody accepts that it's not proven, but it works So yeah, and and finally, what do you see as the ultimate potential of virtual reality and what it can provide? The first thing that I believe will really change the world is the metaverse. So that will be the foundations of a new way of communicating. So you will be able to be with your friends. For example, I'm here in Los Angeles and all my friends and family is in France and I cannot... I only phone them. And that's not the same kind of interaction I would like to have with them. And then, where will it go from here? That's the big question. So, first we'll have some people recreating reality, because that's always how it's done with new medium. When radio arrived, people were just listening to opera. And then after a while, they started to create specific content for the radio. Same for TV, at first they were showing theater plays. And then they started to think, okay, maybe we can have cut from cameras, we can have slow motion, we can have really new grammar. And that's the same with VR currently, people are just adapting existing games or mimicking reality. But as soon as we unlock our brain and unlock our creativity to embrace all these new possibilities, then I don't know what will happen, but I can't wait to see that.

[00:31:30.027] Kent Bye: Is there anything else that's left unsaid that you'd like to say?

[00:31:33.368] Sebastian Kuntz: It's just very exciting to be here and I can't wait to buy the new prototype and I can't wait to be able to add more interactions to that and to see how this will change the world and I want to be a part of that.

[00:31:46.391] Kent Bye: Great, well thank you so much.

[00:31:48.092] Sebastian Kuntz: Thank you very much.

More from this show

Menu

#100: Sébastien Kuntz on Virtual Reality Presence & Lessons from 13 years in VR