Christian Vogler leads the Technical Access Program at Gallaudet University, and has been looking into captions for VR and 3D spatial environments for folks who are deaf and hard of hearing. I had a chance to catch up with him at the XR Access Symposium where he was providing warnings about translating of 2D accessibility paradigms into the new realities and potentials of 3D volumes and spatial computing. He was talking about some of the concerns for folks who are deaf and hard of hearing including the challenges of splitting visual attention, the challenges of information overload, and the potential to communicate new channels of information through the modality of haptics. (Also don’t miss my deep dive into captioning for VR discussion with W3C’s Michael Cooper in episode #1230 where I do an audit of different captioning experiments across the VR landscape in the write-up.)
Vogler also talked about how all of the existing virtual reality headsets interfere with his cochlear implants that means that he often has to either take off his cochlear implants and not be able to hear any sound or deal with the uncomfortable strain that VR headsets have for folks with cochlear implants.
A rough transcript of our conversation is down below (as well as rough transcripts for the all 1200+ episodes of the Voices of VR podcast), and I was also able to record a video with the sign language of both my questions and Vogler’s answers:
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So this is the fourth of 15 episodes of my XR accessibility series. And today I'm going to be talking to Christian Vogler, who is at Gallaudet University and runs the technical access program there. At the very first day of the XR axis symposium, Christian was on a panel and there's a lot of just really amazing things that Christian had to say about, as you start to think about moving into designing for these different impairments from his perspective of deaf and hard of hearing, just a lot of cautions in terms of, you know, Hey, let's not standardize too quickly. We need to really explore the full potential of these different immersive technologies. How can you start to use haptics and. There's also for him the virtual reality headset actually interferes with his cochlear implants because the strap goes right where the cochlear implant is located and so he has to make a choice as to whether or not to wear his cochlear implant and be super uncomfortable or to take it off and to not be able to hear. So there's things that Christian had said in that session that I tried to capture here in this conversation that I brought up again and again throughout the course of my conversations at XR Access, just because there's a lot of really insightful things that he was saying, and also just bringing up how to move beyond the 2D paradigm as we start to think about how to make virtual reality even more accessible when you start to think about all the different new modalities of communication and sensation, when it comes to haptics and spatial audio and the visual affordances of virtual reality and how to start to blend all these things together to preserve the immersive nature of this new medium but also to not make it so that's completely overwhelming. And this is my first interview that I've conducted with someone who is deaf and there's actually a video that I have of the sign language interpreter that I'm working with who was a part of the interpreters that were there. And so you'll be hearing Christian's answer to my questions through the interpreter, but also highly recommend actually watching the sign language version because it's getting the full context of Christian speaking in sign language. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Christian happened on Friday, June 16th, 2023 at the XR Access Symposium in New York City at Cornell Tech. So with that, let's go ahead and dive right in.
[00:02:35.376] Christian Vogler: My name is Christian Vogler, and I work for Gallaudet University. I lead a research group actually called the Technical Access Program, and in this group we focus on access and communication, specifically for the deaf and hard of hearing. It's a very broad kind of research. We actually investigate many different topics, but one of the topics relates to VR and access. So right now our goal, we're interested in how captioning works in VR. We want to be able to have live communication with others in the metaverse. I know right now it's kind of a little bit on hold, but I am confident that it will come back and we will be ready to investigate that further. So it's a very complicated topic. how to show where to focus when you're looking at a person and still capture the captions of what they're saying. So last summer, we were working with an undergrad student who was involved in the research program, and we were working, shadowing, mentoring them, and we were working with captions in the VR program and what that would look like.
[00:03:44.687] Kent Bye: OK, and maybe you could give a little bit more of a context for your background and your journey into working with virtual reality through gaming.
[00:03:54.789] Christian Vogler: Sure. First, I will say that I'm a geek. And I love technology for everything, because technology offers me a whole new world. And my background is in computer science. But I have narrowed my focus into access for the last 12 years. I started becoming interested in virtual reality because actually I read a book called Snow Crash and I just was fascinated with it. And I thought, wow, you know what? I'm probably going to read this book over and over and over again as much as I have time for. And then VR headsets actually became available and affordable. So I got one and I started playing around with it. And I was playing games first. I want to add that I think gaming can actually become a serious business. And the reason why I believe that is my observation so far is that gaming can be very experimental. It could be used as research to find out and create new things. So my goal with Access is I want to see people to be as creative as they possibly can through gaming. You know, it's amazing. It's an amazing new world that you can learn through gaming. And I think that can apply more broadly.
[00:05:07.197] Kent Bye: Yeah, and I'd love to hear a little bit more context for the warnings that you have from moving captions from 2D to 3D and not just to replicate all of the bad patterns that you had relative to what you said as low-resolution television captions and how the captions are being translated into 3D and the challenges of directing attention and information overload and potentially also how haptics might be able to add another modality of interaction when it comes to captioning.
[00:05:43.512] Christian Vogler: So as a deaf and hard of hearing person, we are very visual people. So you get a lot of information through the visual environment. But it's also a risk of information overload, as you said. You see something, and I think a lot of research needs to be done on how auditory information can translate into visual information. But I think a lot of people don't really realize that, that your visual attention is actually limited. As deaf and hard of hearing people, we have to be able to divide our attention. And we struggle with that, of like that split attention that I speak of. So, for example, if you're looking at a person and you're communicating, you're having a conversation with them, and you're looking at the interpreter to actually get the translation of what the person is saying, for example, you're telling me, look at that thing over there. I have to make the decision whether to look over there, to look at the interpreter, or to look at you. It's impossible to split my attention in three separate ways. So I can't get all the information at one time. So often that means that a deaf or hard of hearing person can miss some information. It's hard to control that and it can become very overwhelming. So I would apply that to the immersive environment as well. If you're trying to represent sound or visual cues in a visual way, you're asking for us to take a lot in at once. And we only have a certain amount of visual attention that we are able to invest. So I think it's a serious issue. We need to seriously figure out how to add another way, another modality. Haptics is one way. Haptic feedback. If you look at haptic feedback, it's actually very clear you have many options. It's an obvious choice. Part of the reason would be because haptic feedback would also work for a blind person, too, or somebody who has low vision. Low vision blind people can relate to that. That's another way of getting information, and that can also apply to the deaf and hard of hearing community. So I'm really actually surprised that Apple, for example, for their new headsets, they're hands-free. I'm like, OK, all right, that's very cool. But what about haptic feedback?
[00:08:00.631] Kent Bye: Yeah, and one of the points that you made yesterday was around the form factor of the VR headsets and how they actually interfere with your cochlear implants. And I'd love to hear you maybe elaborate on that point.
[00:08:13.358] Christian Vogler: Feel free to do a close-up on my cochlear if you'd like to check that out. Yeah, okay, so yeah. So there's the processor that transmits the sound there, and there's also a magnet as you can see back there. And many VR headsets hit about right there in that spot that I was just pointing out, and it blocks it, it puts pressure on it. So once you put it on, and you like adjust it so it's tight enough and it fits your head, it puts pressure on that magnet right there in the back and it will often cause it to just fall right off. So the problem is unfortunately many VR headsets have two options for me. I can just take them off and then if I take my cochlear implants off I hear nothing. And the other option would be, I just have to deal with it. Being uncomfortable, but it really doesn't work anyway. So that's the situation we're dealing with right now. Haptic feedback would be my best solution. It would make the difference. It would work best for the... I actually have a PSVR2. I have that right now. And many of the games on that system are fabulous. They have fabulous haptic feedback. So that's the situation. You know, I'm fine with taking my cochlear implants off as long as I have that feedback from haptics, because then I can really get fully immersed into the world. But unfortunately, you know, I wouldn't be able to hear any of the sounds because I'd have to take my cochlear implants off. I wish I could have both. But the point is, we really have to look at the issues and figure out a solution for them.
[00:09:49.799] Kent Bye: Great. And finally, what do you think the ultimate potential of virtual reality with accessibility in mind might be, and what it might be able to enable?
[00:10:02.308] Christian Vogler: So I'm going to try to answer that briefly. But sign language, there it is, right there. Sign language is a 3D language. So if you're using sign language on a computer screen or a phone, that's a 2D world. It's not the same. To be fully immersed into the environment and to have sign language fully supported where everything is in space, it's in a 3D world, that would change everything. The second part of your question, do you mind? Oh.
[00:10:30.679] Kent Bye: What it might be able to enable.
[00:10:33.141] Christian Vogler: Hmm. That's a really good question. I'm going to have to think about that for a moment. I don't want to answer a one-size-fits-all answer. So, for me, the awesome and most exciting thing and potential would be collaboration. So, for example, a deaf person. I mean, we're actually very spread out. Often, we're very isolated. And the numbers are decreasing. So it's sometimes hard to find your community and to actually have face-to-face interactions. And I also mentioned that a 2D interaction is not always the perfect ideal situation. So to actually be able to interact with people, with anyone, and learn from each other in an immersive environment would be incredible. So I think that changes it. And for industries, how could that enable something in industries? I think, for example, We find a lot of pressure to commit to work and training. Sometimes we have to travel very far, but training, practice, in practice, sometimes we have to go very far for work, and that's a struggle. So I think that's the missing part. I think VR, virtual world, is a missing part. I could see a lot of potential for being creative for everyone, and I think it could be a very powerful tool to connect to people and to be able to work together.
[00:11:59.516] Kent Bye: Awesome. Well, thank you very much.
[00:12:01.878] Christian Vogler: Thank you.
[00:12:03.940] Kent Bye: So that was Christian Vogler. He's at Gallaudet University and runs the technical access program and has been looking at virtual reality technologies and how to bring more accessibility features to VR through the context of captioning. So I have a number of different takeaways about this interview is that first of all, Well, like I said, a lot of what Christian is saying came up again and again, because I think that he's just got a lot of these pithy insights in terms of how can we start to use haptics within virtual reality to communicate different levels of intensity and managing these different qualities of information overload and this dilemma of split attention of as you're looking around and trying to be immersed within the context, but also to deal with all the different metadata and information that's coming in as well. having no other conversations of the course of this conference, then having lots of different options, both of how the captions are working, where they're located, all sorts of things of tweaking it, but also just thinking about maybe even a priority level of trying to filter how much information that you're getting. If you want this firehose of information through all this text cues from all the different audio, or if you start to lean into different things like haptics or other dimensions of haptic vest or haptic controllers with the apple vision pro one of the big concerns is that there's going to be hands-free controllers and so there is no way that you can actually have that type of haptic feedback until apple launches with some type of haptic device and so I think it's really missing out on some of the different potentials for how to actually use all the affordances of spatial computing and virtual augmented reality XR medium, so that's something yet to be seen so there's a cautionary tale that he was saying during the session Which is let's not standardize too quickly because they've been dealing with captions in the context of what was created for television and this is like a 30 year old standard that is still replicated in lots of different design patterns today and So with virtual reality, there's lots of opportunities to expand beyond that. And I think if you look at both the Alchemy Labs experiences with how they're doing captionings located over people's heads, or even how Altspace was doing it for real-time communication, that's the thing that Christians were looking forward to solving some of these real-time communication issues so that you can be in these different immersive environments in a social VR context and be able to communicate the ultimate potential of spatial computing and virtual reality being sign language of being a 3D spatial language I thought was really quite interesting because I think we are actually going to be moving into a realm where we have a lot more different types of gestures that we're going to be doing with computing and it kind of makes sense that the types of American Sign Language might start to be integrated into these immersive technologies and I think you know the Apple Vision Pro has like a ton of different cameras all over I think like 15 total cameras looking at all sorts of different stuff. And so it has enough coverage to be able to potentially start to fully recognize American Sign Language or other different types of sign languages. That's still yet to be seen. I mean, there's still early days, and I'm sure that folks will be interested to see if that's even possible to start to do that type of translation of sign language within the context of a spatial computing device. I can imagine that that would be a huge use case for the deaf and hard of hearing community if that's something that would be fully integrated. Into some version of the Apple vision Pro, but you know, the lack of haptics is also a concern that Christian was bringing up So again, I'd recommend going check out the sign language version. I'll have a video within the description of this podcast that you can go watch the video version of this conversation. And like I said, in a couple of the previous episodes, I've also have rough transcripts that will be available for these podcasts and all the previous 1200 plus episodes of The Voices of VR that you can start to look at some of these transcripts that will be from my whole backlog and moving forward. Working with the open-source tools of whisper X to start to integrate this into the pipeline of adding a rough transcript so that folks who are deaf or hard-of-hearing will be able to have access to The work of the voices of VR podcast because up to this point really hasn't been all that accessible for this deaf and hard-of-hearing community So I wanted to make that commitment and say that here on this specific episode So you can look back and look at some of the previous interviews that I've done and also different categories I'm in the process of adding those as well So you can start to look at some of these other conversations that I've had over time So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a supported podcast, and I do rely upon donations from people like yourself in order to continue to bring you this coverage. You can become a member and donate today at patreon.com. Thanks for listening.
