University of Cambridge’s Dr. David Menon is a clinical neurointensivist who is specializing in traumatic brain injury. He’s an advisor for the Canadian Institute for Advanced Research Azrieli Brain, Mind & Consciousness program, and he was an attendee to CIFAR’s Future of Neuroscience and VR workshop in May. Menon is new to the realm of virtual reality, and he was impressed with learning about how far the immersive technologies have progressed over the last number of years.
Menon sees that there is a need for better cognitive and motor skill assessments after a traumatic brain injury, and that virtual reality could be the perfect medium to create a more engaging assessment tool. VR could also collect a lot more quantifiable data as compared to other methods for assessing the extent of a traumatic brain injury, which usually requires long periods of sustained concentration. I had a chance to catch up with Menon after the CIFAR workshop to talk about some of the open research problems he’s looking at related to the assessment of traumatic brain injuries, the limits of sensory addition or sensory substitution, the future of big data and tracking people over long periods of time as a form of assessment, and what the intersection of VR and neuroscience could learn from genomics.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So continuing on in my series of looking at the future of neuroscience in VR. So today's episode is with David Minnan. He's a clinical neurointensivist, also a professor at the University of Cambridge. So he's really looking at traumatic brain injuries and trying to do assessments of people who have had traumatic brain injuries. So he was really coming to this Canadian Institute for Advanced Research workshop on the future of neuroscience and VR really as a scouting to see what the state of technology was, what other neuroscientists are doing, and he was really quite blown away and super impressed to see how far the technology had advanced. So for him, he wants to see if people have experienced some level of traumatic brain injury, how can you start to use VR to give these patients a battery of tests and really get a clear assessment as to some of their cognitive deficiencies and perhaps even insight into how to treat them. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with David happened on Thursday, May 23rd, 2019 at the Academia Institute for Advanced Research, Future of Neuroscience and VR workshop that was happening in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:25.412] David Menon: Hi, my name is David Menon. I'm a clinical neurointensivist and my research interest is in brain injury and broader neuroscience, particularly in outcomes from brain injury. My research focuses primarily on traumatic brain injury, but going all the way from mild to severe traumatic brain injury, so people who come to the ER and go back home. and very severely injured patients who wind up with bolts in their heads and holes in their heads from where the surgeons have made taking cloths out. The reason I'm interested in VR is because we've increasingly been frustrated by the quality of the tools we have to assess deficits that patients have after brain injury and then use these assessments to look at the efficacy of treatments that we have. Now it may be that the treatments are not good enough and clearly we need better treatments, but it may also be because some of the assessments that we're making are insensitive to changes that our therapies may be delivering. So what we would like to do is to have more real-world assessments of cognitive deficits that make a difference to how people cope with everyday life. And we'd like to use that to pass out what are the normal structural changes, the injuries that patients have had, the disconnections that happen after traumatic brain injury, and infer what are the neurochemical imbalances that underlie those. The advantage then is that we might be able to treat it with cognitive enhancers targeting specific neurochemical systems. So that's the dream, but at a very basic level, being able to assess people in more real-world ways would be really helpful, I think.
[00:03:01.433] Kent Bye: I see. Yeah. So that's interesting because it feels like in neuroscience, you're in some ways looking at these edge cases, whether it's these altered states of consciousness through psychedelics or virtual reality experiences or meditation, where you're able to control someone's perceptual input or have a different state or mode of consciousness. But also with these injuries, you're able to see if there's different parts of the brain that are injured than how that's connected to the underlying neuroscience of how the entire brain works, not only for them, but for everybody. You can start to maybe do a little bit of controlled studies of looking at that. And so what are you able to determine in terms of this connection between different aspects of the brain and how that gets impaired in different ways and how that fleshes out into what you can observe in their behavior?
[00:03:48.420] David Menon: So lesion studies have a long and rich history in clinical neurology where patients would die and at autopsy they'd see where the lesion was and compare it to their clinical presentations before they died. The great advantage of imaging, modern imaging, particularly MR imaging, is you don't have to wait for the patient to die. You know where the lesions are. The difficulty in traumatic brain injury is very rarely do these lesions actually map to where the deficits are. Because traumatic brain injury, because of tearing of white matter fibres, nerve fibres, is much more a disease of systems rather than sites. But we can get a handle on those connections both structurally by using diffusion tensor imaging and tractography, which maps out these bundles of white matter of nerve fibres in the brain. Or you can use functional connectivity from fMRI both in the resting state and in relation to tasks to understand where the disconnections become functionally important. What's interesting is that you may be able to find that tasks are maintained even when there are structural and functional disconnections because of plasticity and other compensatory mechanisms. At the most extreme end, we've had an interest for some time in looking at patients with vegetative state, and about 20 years ago now, we showed for the first time that patients in a vegetative state might have preserved cognitive function. And other collaborators went on to show that they may have volition. So Adrian Owen, now in London, Ontario, when he was in Cambridge, showed that. And being able to detect, as you say, those edge cases where there is a little bit of residual cognitive function happening and making it possible for patients to express that would be hugely valuable, not in terms of numbers, but for those individuals and in terms of society and humanity. So some of the things we've heard about today where relatively small motor outputs from EMGs might be used to translate to movement of a virtual, if you like, mouse on a screen, might allow patients who are very severely disabled neurologically to interact in ways that I did not know would be possible, but may be quite available within a relatively short time frame. Those are interesting because they may be available soon, but the more esoteric and advanced approaches to really understanding and deciphering and decoding mental imagery might hold promise maybe in 10 or 20 years' time, and we'd be interested to look at those as well.
[00:06:14.761] Kent Bye: Well, I guess in some ways, when you're talking to these patients, are they able to speak or articulate things? It seems like you're trying to use maybe both directly communicating with them, but also giving them a stimulus in a virtual reality environment and seeing how their body reacts. I'm just curious to hear what all the input that you're taking in to be able to make these assessments. Maybe talk a bit about how you're doing it now and how you see that might change with VR.
[00:06:39.107] David Menon: So if it's difficult, then it may make things easier, but there's a class of patients, a small number of patients, who simply cannot communicate. When we think about how we communicate with the outside world, it's always through motor phenomena. If we didn't have motor activity, I wouldn't be able to speak, because my diaphragm wouldn't go up and down, my vocal cords wouldn't work, my mouth wouldn't be able to phonate. I wouldn't be able to gesture. So if I had no motor outputs, I could be having a rich cognitive experience and you'd never know about it. So to try and understand what's happening in the brain, we generally had to rely on signatures of neural activity, either directly from EEG or from the changes in blood supply that are associated with neural activation in terms of fMRI or near-infrared spectroscopy. But what's emerging now is that there may be some motor outputs, but not enough to actually make themselves known as movements of hands or legs or whatever. And under those circumstances, being able to detect those with very sensitive approaches, which look at EMG and allow you to translate into gestures, which would be very natural for people, would be very helpful, I think. So there may be a class of patients in whom there are no motor outputs possible and there I think you have to look for neural signatures of cognition. But there may be people in whom there are some motor outputs or there are some outputs, perhaps heart rate variability or something, that allow you to manipulate some kind of an interaction with the computer so that you can communicate using that approach.
[00:08:07.833] Kent Bye: So it sounds like if somebody's had a traumatic brain injury, they may have lost the capacity to be able to speak. And it seems like that it should be theoretically possible with VR to potentially do neurorehabilitation or to subtly train them to be able to cultivate a way of being able to express certain ways that they can express their agency or volition. but to be able to detect those neurological signals and then, you know, if they're in the VR environment, maybe it's amplified, but eventually be able to have some sort of device that'd be able to translate what's happening out of their body and for them to be able to communicate. That seems like that should be possible.
[00:08:41.040] David Menon: So yes, but what we're talking about throughout that entire conversation we had is a small proportion of patients, probably 5-10% at the very most, probably less than 5% if you're looking at vegetative state. There is a much larger proportion of patients where we would use VR for assessment. and understanding where the deficits lay because it would be more real-world assessment. So if patients are having trouble holding down jobs, instead of using paper and pencil tests or computer tests that ask you to move boxes around, you could ask them to do the tasks they were doing and try and infer what are the cognitive deficits that are preventing them from doing those. and really prescribe therapies either in terms of physiotherapy, occupational therapy or drugs to enhance some of these or in combination to make those things happen better and with greater ease. So I think using VR as the only solution to get people to express what they're thinking is definitely a possibility but it's at one extreme end of the outcome spectrum. There are many more people at the other end. For example, in patients with mild traumatic brain injury, we know a significant proportion of them, somewhere between 10 and 30 percent, have post-traumatic stress disorder. And being able to expose people to the stimuli that result in post-traumatic stress disorder, which has been used again and again in the military situation, may allow that really sometimes crippling psychopathology to be treated or at least mitigated to an extent that it becomes easier for people to live their lives. So it's a whole spectrum and we're just coming to terms with how we would apply it best in different severities of outcomes.
[00:10:15.270] Kent Bye: Yeah, there was a discussion about getting information and data into the brain that was happening earlier, and I was thinking about David Eagleman's Neosensory Vest, which is able to do a translation of listening to sounds, but able to translate that into an array of 32 different vibrating sensors on this vest that is able to produce what ultimately becomes a data structure that is sent to the brain, where if it's in the same data structure as what the cochlea sends, then as long as it has a visual synchrony for people to correlate what they're seeing with what they're feeling, then you can turn your torso into an ear, which was something that was originally discovered in 1968 in this concept of sensory substitution to be able to replace a sense. And it seems like that is a thing that is possible and is happening currently, but there's also some other perspectives that you had on that. I'm just curious to hear what the hesitation of that might be.
[00:11:02.627] David Menon: So I'm sure it would be possible but it seems like quite a complicated process and the question is, so it will inevitably be helpful for a small proportion of patients. But many patients who have hearing difficulties may just require it augmenting with conventional hearing aids. So it's just what proportion of patients it would be really a game changer in is the question I was asking. It's going to be a small proportion of people. The second thing to say is if there's a lot of training involved, many of these patients with traumatic brain injury who have more severe disabilities are going to find it quite difficult to train themselves up because their cognitive abilities are going to be more broadly challenged. So we mustn't provide solutions that are difficult to implement. They need to be easy for the patients to understand and be able to get used to. But the proof of that particular pudding is in the here eating, so we'd have to see what happens and take things from there.
[00:11:54.998] Kent Bye: Yeah, my suspicion is that with VR, that something that may intuitively seem difficult may become quite easy if you have visual synchrony on top of haptics, on top of sound and sonification, you know, like if you have a whole rich multimodal input, then, you know, there's a concept of replacing senses, but also expanding senses. So people training themselves to be able to detect where true north is just by having this device that has this passively sending in a signal every five minutes where eventually you just kind of integrate it into your body, but somehow the body's able to pick up the shadows in the time of day, and it's able to have this sort of sixth sense of being able to cultivate an awareness of where True North is. And so, for me, it feels like there's this rich possibility for adding VR on top of all this multimodal sensory input to be able to either expand or extend our senses.
[00:12:41.501] David Menon: I agree, and it may be that some of the technologies used in VR, even if you don't use VR, might be quite helpful. So one of the things that many patients with traumatic brain injury, and for that matter dementia, have are problems with short-term memory, and they don't lay down memories easily. so they may go through a whole day and forget everything that they've heard. So having some way of recording things has made a big difference and often the suggestion has been made so the Oliver Zangable Centre near Cambridge has pioneered patients carrying a camera around which takes snaps regularly. Now that isn't as complete an experience as you would like and it doesn't have the richness that everyday experience has and we remember So being able to use the technology that we use for VR to record some of that and then allow them to play back when they are actually interacting with those scenes might be hugely valuable for them. To be honest, I don't know what's possible and what's not possible, but it sounds to me as a lot of the technologies around here may be not just important for us to use VR, but also to augment both outputs and inputs for patients who have these challenges because of a variety of neurological diseases.
[00:13:47.790] Kent Bye: Now, are you looking at neurodegenerative diseases such as dementia and Alzheimer's in your research as well, or just with traumatic brain injuries?
[00:13:54.532] David Menon: Not directly. So there's other groups looking at it, but we are particularly interested in that because there's a suggestion that traumatic brain injury increases your risk of late-onset Alzheimer's disease, and it scales with the severity of injury. So the most severely injured patients may have up to a four-fold risk of Alzheimer's disease. Now, we don't know if that's true or not. There's a strong suspicion that it is. We don't understand why it happens, but we're particularly interested because that identifies patients who've had a significant traumatic brain injury as a high-risk population for late dementia and perhaps makes them a population for preventative therapies. So, in that sense, we're interested in late dementia, but it's not a direct focus of what I do.
[00:14:37.369] Kent Bye: Yeah, one of the other things that was mentioned today that I thought was quite interesting was this concept that you could not be so concerned about trying to isolate individual variables, but to rather be able to embrace the big data and just have lots of different information and trials and to kind of take a completely different approach of of doing machine learning or being able to sift through the data or do statistical analysis and taking insights from the genomics research. But maybe you could talk a bit about what you see as some of the shifts in how even scientific inquiry is being done with some of these immersive technologies and the ability to be able to gather this enormous amounts of data.
[00:15:13.767] David Menon: So I think that's a very important point because if there are large stores of data, we can use them to set hypotheses by just looking at how the data are connected. We had a discussion about longitudinal data on a group of gamers which might be collected over decades, say, because now gaming's been around for long enough. and just looking at whether that's a unimodal distribution or a bimodal distribution when you look at the trajectories over time would be really interesting. We could try and see what are the rates at which cognitive decline happens and the rates at which motor decline happens because fine motor skills may be a marker of cognitive decline or it may be that one or the other is more prominent or at the very least you've got to compensate for one when you're looking at the other. We can try and see whether there are regional variations in it, because we have epidemiological data on a very broad scale. For example, looking at folate intake across the United States, there's a lot of dietary information. We can look at exposure to the socio-economic stresses. In the UK it would be easier because we have an identification of socio-economic risk by postcode, what you'd call zip code here, I don't know if that's possible, and try and understand whether these are different for disadvantaged versus advantaged populations. It's almost asking the question, what are the questions we should be asking by looking at the data sets. And that I don't think we've done quite enough because we haven't known what's available and I still don't know what details are available because just the data on their own may not be that valuable. We need something to triangulate to, if it's demographics, where the person lives, what is the time frame over which these data were collected. Those are the bits of information, the metadata that come with the actual data of the gaming that will allow us to know what sort of questions can be asked.
[00:17:03.092] Kent Bye: Well, I think there's another dimension of that too, which is that these are complex ecosystems with lots of things that are related to each other in ways that are maybe hard to isolate into individual variables. Maybe you could talk a bit about what you think the VR community could learn from what the genomics community has had in terms of the way that they maybe have to deal with sort of these biological organisms that have a lot less clear causal paths.
[00:17:25.377] David Menon: Yeah, so I think having said we should look at the genomics, I think there the mechanics of how genetics relates to phenotype is unidirectional and we have a slightly better biological understanding. We may need to look at the VR data because when we look at behavioral data it's very linear so far and what we're looking at is a whole mess of data, but there is a lot of data so we should be able to organize analyses that account for multiple confounding covariates. I'm still not clear in my own mind how we would do that. We'd have to look at the data, run a few analysis, see how it looked, and then take things from there.
[00:18:00.351] Kent Bye: Yeah, and just the approach that Craig Chapman was taking, which is that they would take kind of a two approach, where you would have the movement data, and then you would do an AI training algorithm that then you could do sort of automatic labeling as the different tasks, and then from that automatic task, do another layer of looking at the EEG data. It sounds like that was a very interesting approach. If you take all of this different data from different sources, but that if you're able to extrapolate the different behaviors based upon from one signal, you're able to then use that to be able to assign and find potential other correlates into other signals. We're going to be living in a world where we're going to have all this data available. I think a broader context and discussion point that was happening here at the Canadian Institute for Advanced Research, the future of neuroscience and VR, was looking at what is the possible interface between academia and the possibility to share some of this data. Because the ability to be able to make some of this data available then you could start to look at these different scientific questions Especially once you get to the point with EEG data and I think there's a lot of privacy issues in the way to actually Functionally do that with whether it's people owning their data with differential privacy or homeomorphic encryption that decentralized architecture still yet to be designed and have a good scalable solution, but once that's figured out then you're going to have the opportunity to have people that are playing these games be able to potentially participate in contributing their experiences that they're having within these virtual environments, but to be able to do this different type of research from people like yourself who are able to look at these various different questions.
[00:19:24.610] David Menon: It's citizen research, I think, so it's making people feel that they're stakeholders in this research and that they're contributing, because there's a phrase that the NIHR, the National Institute for Health Research in the UK use, There's no research on me without me, and I think that's something that we really have to espouse very strongly. And looking at that initial data will not necessarily answer the questions, but it will tell us what kinds of questions we can answer once we can put on EEGs, once we can get on behavior, once we can look at longitudinal trajectories of what's happening to people in terms of their psychological health and so on.
[00:20:00.309] Kent Bye: Great. And so for you, what are some of the either biggest open questions you're trying to answer or biggest problems you're trying to solve?
[00:20:07.960] David Menon: So better cognitive assessments, I think, is a relatively low-hanging fruit. And being able to put that into a more realistic and authentic environment would make a huge difference for us. At a very simple level, patients with traumatic brain injury can't sustain concentration. And the computer-based tasks that we have that last a couple of hours are quite difficult to maintain concentration with. So if we had something that was more engaging, that in itself, even if it was no better in terms of its veracity and fidelity, would be hugely helpful.
[00:20:37.765] Kent Bye: Great. And finally, what do you think the ultimate potential of immersive technologies are and what they might be able to enable?
[00:20:46.614] David Menon: I'm not sure as yet. I came here expecting less than I found and what I didn't realize is how far the technology has gone and how many things are available in the near horizon. So in the next three to five years, so we've already had discussions with some people about research grants that we might be putting in together, looking at the technologies and the approaches that are being used for the VR industry at present. And it'll be interesting to know how those first few ventures go and how effective they are and how easy it is to implement some of the things that we want to. If they are easy to implement, if the translation to the clinical environment is, well, not easy, it's not too hard, then I think the opportunities are huge.
[00:21:31.927] Kent Bye: Awesome. Great. Well, thank you so much. Thanks. Thank you. So that was David Menon. He's a clinical neurointensivist as well as a professor at the University of Cambridge. And his research interest is looking at the outcomes from brain injury and looking at a wide range of traumatic brain injuries from mild to severe. So I have a number of different takeaways about this interview is that first of all, Well, it seems as though that when someone goes through a traumatic brain injury, there's some sort of like impairment. They can see it in the brain, but it's very difficult to look at the region and try to predict what type of impact that's going to have. For one thing, there's rerouting and from neuroplasticity and different other compensatory ways of adapting that it's very much difficult to see how any given brain injury that's happening in any part of the brain is going to impact. any type of range of cognitive impairments. And there's a range of different tests that you can do, but it sounds like that requires quite a lot of concentration and that something like an immersive virtual reality environment would just be so much better to be able to assess some of these things, especially if you're starting to tie in all sorts of like other biometric data, whether it's EEG, eye tracking data, motion track data. So just finding new ways of doing assessments for after people have gone through some type of traumatic brain injury, like what is the degree to which their functioning has been impaired? And then to find ways to mitigate it, whether it's what he says, finding if there may be some sort of neurochemical imbalance and to see if there's ways to treat it with cognitive enhancements that are targeting specific neurochemical systems. Now, I also think that there's probably a lot of opportunity for using VR itself as a technology to do neuro rehabilitation. Seemed like David was pretty skeptical to see like there might be a very small subset of people who go through a traumatic brain injury who'd be able to be receptive to that type of neuro rehabilitation. I guess I'm a little bit more optimistic and I want to really see what the limits are with neuroplasticity and what you can do with being able to do this real-time feedback between modulating your consciousness and your experiences through these immersive experiences, doing real-time biometric data feedback, and then I think there has to be a lot of mapping and understanding, but I feel like that's pretty open-ended as to what the possibilities might be with all these combinations of technologies coming together. Now, at that same time, I think there's going to be real limits, as he's saying, if these traumatic brain injuries, once you damage things beyond a certain point, there may be just impossible to bypass it or find other ways of getting information to the brain. thinking a lot of the work from David Eagleman looking at the neosensory vest and these concepts of sensory placement and sensory addition, where as long as you have some level of visual synchrony and you're able to correlate the input that's coming into your brain and seeing how that's related to what is happening in the world, that the brain kind of figures it out. That's kind of what David Eagleman was saying. But I also heard from other neuroscientists since I've talked to Eagleman Talking to them in that if you're rewiring things through your body It may not get to like the same level to like your visual cortex to be able to actually see in the same way So we'll be diving into some of those other conversations kind of unpacking that a little bit more but that's an interesting point is that there's specific parts your brain where information is processed and that if you're Rewiring that information and it's not getting to those same areas of the brain Then can you really have this same type of sensory experience? I? But David seemed to be very optimistic for how VR could be used to be able to do cognitive assessments. That seemed to be a pretty low hanging fruit baseline to see how you could start to get people into these immersive experiences and then start to get some assessment as to what's actually happening. There was also a couple of other points. Uh, one is that there was some talk about some more down the line, like advances of brain control interfaces where actually having like ECOG, which is like opening up the brain and putting all sorts of like sensors on hundreds of different neurons to be able to actually like react and measure the brain to the point where you could start to really get high resolution and start to literally read your thoughts. That was an invasive technology. And eventually I think we're going to get to the point where. You are going to have some ability to put something on your head and it's just going to be able to read what you're thinking, which is super scary. But at the same time, for some people who have gone through these traumatic brain injuries, imagine if you're able to then advance beyond just like words, but getting into like images and other ways of imagining and doing more structured thinking. What if it's possible to be able to actually get into those imagining into your mind and then be able to decode that in some ways? So David's talking about a lot of these patients that have gone through these traumatic brain injuries. He was also really excited to see the potentials for the EMG sensors. And a lot of what Dan Whitmore was talking about with control labs, where you can focus onto like one neuron that if you have the ability to control. movements or muscles, what if you're able to put a sensor then and then amplify that within a virtual reality environment, and then start to give people a sense of agency for being able to have these different levels of virtual embodiment. And who knows what that would be able to lead to, to be able to expand out to other parts of the body as well. It's all a whole new realm of exploration to see how using some of these immersive technologies could be used to maybe discover that people have the fine control to be able to, you know, still fire things into their motor cortex, into their spinal cord, and be able to, you know, have it on their muscles and then have that some way be amplified and then create that as an interface, either within virtual environments for exploration or to just interface with computing technologies in general. And the other thing was that David was really advocating for kind of flipping a lot of the existing philosophy of science on its head. That is just to say that rather than trying to come up with these baselines of ideas that are generic to individuals, that VR could start to have this longitudinal data where you're just gathering all this big data on people. to be able to really specify specific treatments that would be very well suited for an individual. He was talking about looking to see like the rates at which the motor system is declining or the cognitive system is declining and to see how those are related to each other in some ways. And so if people do have like these declining conditions that if you're able to put them within the VR headset and maybe get all sorts of different measurements and data, then you might be able to get a lot more insight into some of these traumatic brain injuries and then find new novel treatments that have never existed before. But it's just the idea that we're kind of moving into like the personalization of medicine, especially when you have access to all this big data. And so, you know, he's looking at from the perspective of creating a consistent experience and giving that consistent experience to people to do different levels of cognitive assessment. And I think that's an interesting paradigm shift to think about is to not think about like, what is the principles that are going to be universal to everybody? But what are the treatments that if you have enough data and information about a person, are you able to have other models to be able to give treatment to that individual person? And I think it's this turn towards the personalized medicine, which I think is super fascinating to see where this kind of combination of virtual reality technologies, big data, machine learning could all kind of open up this whole new world of personalized medicine. So that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon listeners like yourself to donate. Just $5 a month is a great amount to give. It just helps keep the lights on here. It allows me to continue to travel around the world, talk to a wide range of different experts, trying to capture this real-time oral history of the evolution of spatial computing. And if you want that, not only for you, but for the wider community, but also for the future generations of people looking back onto this time period, then please do consider becoming a member of the Patreon. You can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.