Anthony Steed is a Professor in the Virtual Environments and Computer Graphics group at the University College London. He started his Ph.D. in Virtual Reality back in 1992 during the first wave of VR. Some of his research interests include distributed virtual reality systems and collaborative environments, 3D interaction, haptics, networked virtual reality protocols, massive models, and telepresence.
Here’s some of the topics that we discussed at the IEEE VR conference:
- Latency in VR depends on the the context and it can range from a target of 1ms for visual stability to 10ms to 20ms.
- Collaborative virtual environments & asymmetric interactions in VR that result in a difference in social power. How the UI in VR can either get in the way or support interactions
- Some of the areas of research include 3D user interfaces, haptics, sensory motor integration, & remote telepresence. Starting to build their own VR hardware
- Fidelity of avatars in telepresence applications. High-quality avatars must also behave with a high fidelity. Tend to use lower fidelity avatars. Full body tracking without full facial expressions result in zombie-like experience. Telepresence is often task-based where the avatar’s representation of identity is less important. Working with sociologists who look how eye gaze gives cues for turn taking in conversations
- Most VR don’t utilize our own bodies for haptic feedback. Creating external haptics is a huge problem because they’re very limited. Potential for body-worn haptic devices.
- On the intersection of neuroscience and VR, looking at our visual system has a left-hand side bias for visual attention, and it’s an open question as to whether they can recreate this neuroscience effect in VR. The impacts on body image when you are tracking your body within VR. Looking at frequency bands of head movement & whether the VR display matches what our proprioceptive senses are telling us about our body’s orientation. Using VR as a platform for neuroscience research into looking at discrepancies of sense queues and looking at persistent illusions
- There’s a lot of potential for education and training, and a lot of progress being made in this realm.
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:12.085] Anthony Steed: I'm Professor Anthony Steed. I come from University College London. I've started my PhD in the field in 1992, so I've been working on virtual reality for a while. When we got started, it was during the first wave of virtual reality. We were using equipment that came from a UK company called Division, which sold turnkey virtual reality systems. You'd ordered one, it would turn up at the lab a couple of months later. and I was a mathematician and got into computer graphics because of a summer job and because it was one of the interesting modules I did. But then we very rapidly learned that there was much more interesting things about virtual reality than just making immersive systems and we subsequently, I've worked on all sorts of areas from distributed virtual reality, 3D user interaction, system latency is a big area I'm interested in. And we've worked with collaborators from neuroscience through to theater. So it's been a very varied journey.
[00:01:09.190] Kent Bye: And what do you think is the threshold of latency that people should be striving for, from motions to photons, in order to create the best immersive experience?
[00:01:19.294] Anthony Steed: It's a very good question. It totally depends on what phenomenon you're looking for. We've got a recent study where the end-to-end latency is under 10 milliseconds, but the limiting factor is the human motor system, not the graphics. But that's not to say that under a millisecond is necessary for some things, such as eye stability during rapid movement or something like that. So it's a very tricky question to answer for all situations. So going lower and having people build experiment systems that are lower than the current consumer hardware is going to be productive. Either we'll find that it should be much lower for certain things, such as visual stability, or we'll discover that what's currently around 20 milliseconds is actually sufficient for most people. That's why it's a research question. Nobody knows if there's a flaw on these effects, or if different parts of the cognitive system will need different latencies. But my gut feeling is that there isn't a flaw, and there will be some effects which you can determine for latencies of the order of a millisecond.
[00:02:25.841] Kent Bye: And what type of research questions have you been looking at in terms of the collaborative virtual environments?
[00:02:32.084] Anthony Steed: Okay, so from a historical point of view, we started off building sort of collaborative virtual reality systems in the late 90s, trying to do large scale, dealing with just the networking issues, trying to get voice to work, spatialized audio to work, and dealing with large models. All of that was sort of very systems-oriented work. But again, the same sort of story happened. You show it to people and then you discover all sorts of interesting effects that happen. So, I'm talking later in the conference about studies of asymmetric interaction. To give you an example, we've had a study a few years ago now where somebody using a desktop user interface collaborated with somebody who's wearing a head-mounted display in a tracking system. and the person with the head-mounted display in the tracking system became the leader of the collaboration. And part of the reason for that is that their avatar, their representation in the world is always alive, it's always moving, it's easier for them to gesture and so on. So we started calling this sort of asymmetric interaction, so the fact that you have different interfaces to the world may give you different social power which doesn't respect actual power structure or personality. Now that's a good and a bad thing. We showed it to the military and they hated it because the person with the most stars or stripes on their uniform is the leader and you follow them and if you put them in a user interface or a distributed system where they negate some of that influence or status that comes from their uniform then they just dislike it. But then for other people who are shy or in a culturally situation that I'm familiar with, it can be immensely empowering. It's just trying to get it right. So the main thing is trying to understand what the interface, if it's getting in the way of the interaction, the social interaction, or if it's actually supporting it. And given that it's transforming the social interaction, what is the limits of this and how can you utilise it?
[00:04:26.574] Kent Bye: And what are some of the other areas that you've been researching then, or different studies or results that you've come across over the years?
[00:04:34.543] Anthony Steed: Well, we're quite a large group, so over the years we've worked in areas such as 3D user interfaces, so how to do selection and manipulation, how to use brain-computer interfaces inside virtual reality. We're getting more into sort of haptics and touch. We've done a little bit on sensor-motory integration, so a very wide range of things that we've worked on. The interesting things for me at the moment are really haptics and also how virtual reality is used to control systems. So I'm particularly interested in remote, so telepresence, so remote control of things, so using virtual reality to visit other places in real time. I think that's a great challenge. It's quite different than some other things people are doing and it's got a lot of interesting constraints such as fidelity of reconstruction. and really understanding how the person is going to interact with the world because that can steer your computational analysis and how you deal with massive amounts of data you might be getting from the right environment. We're also very interested in hardware. Like a lot of groups, a lot of computer scientists over the years, we sort of believed that eventually the likes of NVIDIA would save us. to make our virtual reality systems easier to build, but they never really did. So we've actually gone back to building hardware now. I think that's something which is not just with the rise of the consumer head-mounted displays and all the hobbyists in that area, all the open source plans you can get. but actually just building whole systems I think is something that the whole field can now engage with whereas previously it was a sort of very specialist area. When I come from a field which you know used to spend hundreds of thousands of pounds on SGI equipment and you know you didn't touch that. you locked it away and then used the input and output as was. But now we can build our own systems completely from scratch. It's not completely infeasible to do that. And for certain application areas and for certain types of user interface that's become quite possible because of open source displays but also rapid increase in power of open source system development boards as well.
[00:06:47.092] Kent Bye: And in terms of telepresence, there seems to be this issue of the uncanny valley of the higher fidelity that you have something, the more that you want it to be realistic. And so I'm curious if you're doing sort of a low fidelity rendering of avatars of people in a telepresence environment, or if you're trying to go to fully rendering someone with a Kinect and getting the full sense of their presence.
[00:07:08.973] Anthony Steed: That's a good question. We had a PhD student a few years ago who showed that if you're going to render high-quality avatars, they've got to behave in high quality as well. So there's not a lot of point in having a high-quality avatar that acts like a robot. Vice versa, if you're going to have something that's cartoony, then the expectations of its behavior are quite low. So in our work, we tend to use very simple avatars for the reason that they don't lie, really. And some of ours are very, very simple, not just because it's easy to build. So I think that area of trying to capture avatars is extremely hard. There's a lot of work that's gone on on capturing full body motion, but our experience of that is you get full body motion, that's great, you can do certain things with it. But if it doesn't come with full facial expression, you've still got the zombie walking around. It just happens to be doing more actions, but you can't interact with it. So I think that's a very big problem for the next few years, especially in the telepresence world is you wear a headset, you know, it's going to be hard to reconstruct the face to show it to somebody else. And those are great challenges for developers to get their teeth into. I would also say that telepresence is not, you don't have to study it in isolation and that should be for an application area and I think in some application areas there's a lot you can get away with in the sense that the telepresence is about a task and it's representing the task which is important, not so much the avatars. Again our experience with avatars is that we did a study a few years ago where we had people collaborating in two cave-like systems and they were there for the whole day. And they spend the first 15 minutes talking about their avatars and how they don't think it looks like them and so on. But then they get to work and the avatar is there not really to represent what they look like but to indicate what their actions are. And I think that's a subtle shift of focus which means that in some situations if you focus on the task, say it's constructing something, then you shouldn't look at what the avatar looks like but can it faithfully convey things such as what I'm interested in, am I looking the right way, are my hands doing something or not. Because if you look at collaboration that happens in the real world a lot of it is not eyes on, it's just being aware that the person is looking in a certain direction and that's that type of thing that you look at. Now, how you get into that is very hard. So, we've worked with sociologists that just study eye gaze and they just study it for how to hand over conversation turn-taking. So, a very simple effect is that if you look at a multi-party conversation in the real world, the person speaking tends to look at the person they expect to speak next at the end. It's not a very reliable cue. It's obvious to treat that with statistics. But it's quite common that they look at people they expect to be able to respond. And just getting that across is more important than making it look exactly like me.
[00:10:04.289] Kent Bye: And in terms of haptics, what are the importance of haptics and what types of things have you been looking at or measuring or investigating in terms of adding haptics to virtual reality experiences?
[00:10:14.173] Anthony Steed: OK, so the first thing to say about haptics is that most virtual reality systems are missing a trick, which is that you have a whole bunch of haptics anyway, which is yourself. You have a permanent, persistent experience of contact with your own body. And one of the very first things we learned early on in virtual reality is that if you draw a virtual body, that people can use that haptically in some senses. And that because they see their body and they see things that they could touch with their body, you're sort of using the haptic system. You're not actually causing virtual touch, but you sort of cause them to think about it possibly at some sort of subconscious level. So that's one thing, you've got all of this body already, why not use it? So using things that use two hands, you touch your hands together to do something or you use the front and the back of your hands, you're using what we've got already. Then the field of creating external haptics where it's just a huge big problem. We've got large sets of haptic devices ourselves, all sorts from Moog and force dimension and so on and the key thing is that they're very limited at the moment they're very high quality people love them when they touch them you reach out and you you feel something but it is point like poking something with a stick so i think that's a huge challenge for the future is how to recreate those touch sensations and you know it's fundamentally very hard it's a very large display surface it's got a very large frequency response and it's very individual so enormous challenges But I think there'll be some subtle things you can do in the interim, maybe with hand-worn or sort of body-worn haptic devices which use other cues to sort of simulate haptic cues.
[00:11:56.880] Kent Bye: And so in terms of neuroscience, what type of insights are you getting from neuroscience and feeding those into virtual reality then?
[00:12:05.304] Anthony Steed: Okay, so they're two big fields, so it's reasonably hard to integrate them. What we've been learning is that it's some effects we should look for, for example, so certain things which are known about, say, the visual system, which you can try and see if you can replicate in virtual reality. Simple things such as biasing visual attention. A non-recent one is something called the Posner effect, which is about visual bias, so it's a well-known effect where in the visual psychology literature, which is if you flash something on the left-hand side, then you flash either on the left or the right something they're supposed to be looking for, they'll be faster because their visual system is attracted to the left. So this is something which is now being explained in some sort of neuroanatomy, and you can look for it in virtual reality. Can you reconstruct these things? Are there results, say, on stereo perception that you can bring in? So that's one area in neuroscience. And the second one is still emerging in that, which is about body image. And I'm sure you'll try and catch Professor Slater on Wednesday. But I think we're learning a lot from that about what is important about virtual reality. Right now, we're getting extremely good visual displays. But the really thing that's important is the match between what you see and what you're doing, because you're very good at picking up discrepancies. Something we intuitively knew quite a long time ago but now because the systems have got a lot better we can sort of now look at frequency bands for example of head movement and whether or not our virtual reality display is matching what our proprioceptive sense tells us is going on or what our body image is telling us about where our arms are. So, again, it comes back to this issue I've mentioned already, which is about making sure that when you put on a head-mounted display or stand in front of a display, that you are represented in the display, so you can interact in ways that you know you should be able to. So it's exploiting all that sort of motor knowledge that you already have. And then of course we're doing a little bit where virtual reality is being used as a sort of platform for them to experiment as well. So that's a very promising area. For example, looking at sensory integration. So if you hear and hear something and see it at the same time and they're discrepant. how is that resolved in the brain, because that can tell us something about how the system could work, and it can tell the neuroscientists about, for example, why certain types of illusion are quite persistent. And I think that's very, again, a ripe area for mutual collaboration, building virtual reality systems that are good enough that they can run experiments on, for example, how top-down visual processing might work.
[00:14:43.363] Kent Bye: And finally, what do you see as the ultimate potential for virtual reality, and what it might be able to enable?
[00:14:50.996] Anthony Steed: That's such a broad question. What I'm seeing here at the moment at the conference is that a lot of people are tackling applications which are quite broad. People are making more headway on some of the traditional uses of virtual reality and training. education and so on. So I think those are interesting, that those have long been posited as being possible, but there have been relatively small instances of it happening. So I think that's what's happening, you know, the future is sort of to go back to all those things we said could be possible. So entertainment is obviously a huge area, very productive, very useful, very important for humanity. But then the training, I think, is really exciting at the moment. So people learning skills, I think, is a great use of the technology. I hope that'll come out from some of the other talks that we'll see during the week.
[00:15:42.736] Kent Bye: Great, well thank you. Okay, thank you.
