Nicolas Férey was presenting a number of papers at the first-ever workshop on Virtual and Augmented Reality for Molecular Science at the IEEE VR. Nicolas was presenting different navigational paradigms for molecular biology tasks that were designed with the involvement of experts in structural biology.
Here’s the abstract of his presentation:
Visualization and exploration of molecular experimental results play a crucial role in structural biology. By visualizing and analyzing the 3 dimensional structure of a molecule possibly over time thanks to simulation tools scientists try to understand its functional role in the cell. Stereoscopic rendering features are historically used in structural biology and experts are thus quite familiar with virtual environment and 3D interaction. Since the dawn of virtual reality, immersive environments have been used to bring scientists into the heart of complex molecular scenes while adding an interactive dimension.
However, in immersive environments as well as in desktop contexts, many issues concerning navigation need to be addressed when exploring molecular content. Among these issues, lack of spatial awareness and the cybersickness phenomena encountered in navigation tasks are major obstacles to overcome to avoid any degradation of the experience and efficiency. It is interesting to highlight the fact that even if navigation is very frequently studied in virtual environments, most of the studies produce generic paradigms that are only applicable to realistic virtual scenes. These approaches do not explicitly take into account the content of the 3D scene and the task of the end user.
In this study, we present some new implementation of navigation paradigms based on tasks and contents, dedicated to molecular biology and designed with the involvement of experts in structural biology. These paradigms are independent of the interaction context and can be indifferently used in a daily desktop context or in immersive environments ranging from CAVEs to HMDs.
When asked about some of his favorite VR experiences, Nicolas talked about a really interesting CAVE experience of a fully-immersive molecular biology docking simulation that including haptic-feedback and sonification. Nicolas also had a really interesting answer to the ultimate potential of where this technology could go in talking about some of the implications for medical breakthroughs and scientific investigations for being able to do real-time simulations within an immersive environment.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:12.033] Nicolas Ferey: My name is Michael Trellet and I'm a PhD student in my third year. I'm working between two laboratories, one in Orsay and one in Paris. One laboratory deals with molecular sciences, the other one is about computer science on virtual reality, augmented reality. What we're doing is that we're using the techniques of virtual reality to visualize basically molecular complexes. Since it's quite a new field, we're trying to see what kind of obstacles we have to overtake to visualize complexes and to visualize 3D structures thanks to virtual reality. My focus was more about the navigation techniques we're using to navigate in virtual environments and how you deal with such abstract data that are molecular structures. So we set up some navigation paradigms that are based on the content you're observing, so here molecular structures, by using symmetrical repetition of the monomers you can find in a molecule. You can set up specific paths of navigation that allow the user to keep a fixed orientation during his navigation, reducing the cyber-thickness phenomenon that could occur in some navigation process when you have an experiment. So we have many paradigms that are based first on the content as I just said but also on the task that the user would have to perform. So we set up some algorithm like the best point of view algorithm, we call it like that, that permits to observe a specific target in 3D. and to have the best point of view over this target. So it goes through different algorithms, computational algorithms that take the target, analyze the neighborhood of the target, and try to see where you have the least occlusion between your camera and your target. So we basically use the content and the task of structural biology to get and set up our navigation path in virtual environments.
[00:02:04.275] Kent Bye: And so what is the problem that these scientists are trying to solve in terms of being able to visualize these molecules? Like what type of insights or judgments are they trying to make?
[00:02:13.398] Nicolas Ferey: So basically what we truly think is that by bringing this 3D perception and 3D depth of molecule you are able to really visualize in a better way how your molecule is shaped and basically how interaction between proteins can occur because When you add this 3D component, you have extra information that you don't have on desktop facilities and we think it's really important because it allows the user to really feel its protein and to have some insight about how his molecule is going to behave on the structural point of view.
[00:02:48.325] Kent Bye: And so, molecular scientists that are using this, what is it that they're trying to do at the end, after they visualize it, then what does that enable them to do outside of VR then?
[00:02:57.872] Nicolas Ferey: Basically, we are at the first step of a pipeline that would consist first of visualization and in the second time of analyzing what you're observing. When you have this analysis on top of the visualization, you can really set up, for instance, molecular simulation. You could set it up in real time and it's something that clearly in virtual reality we are looking for. This real time capacity allows you to visualize and to analyze your molecule in the time that you're running a simulation behind and you're trying to observe a specific phenomenon, like a link between two proteins that one would try to inhibit the other one, which is, I don't know, harmful cell or whatever. So that's the first step of a pipeline that is quite huge, but we think really interesting and scientists are, yeah, they feel this need and we try to provide the tools to fill this gap.
[00:03:48.771] Kent Bye: And so is there a time dimension to this at all in terms of how things are changing over time or how they're interacting with other things? And, you know, are there specific scientific algorithms or models that you're using in order to kind of create these simulations of interactions? And just in general, whether or not these are static in one moment in time or if they're changing over time?
[00:04:06.565] Nicolas Ferey: No, what we are trying to do and that's quite complicated but that's something we want to achieve and it's already achieved at some point, I would say at some basic point, it's to visualize in real time a simulation running on a cluster of calculations behind the world in the room of visualization UI. You basically see the simulation and you can even drive this simulation, steering this simulation. Because, I don't know, you want to observe a specific event occurring at a specific time and location and you will just play with haptic devices and kind of feedback like that. You will play on your simulation and you will drive your simulation toward the point of interest and that's something really interesting and that's something now you can do with the real-time capability of virtual reality.
[00:04:50.696] Kent Bye: And when you talk about navigation, you know, having a 3D object, is the navigation sort of moving around the 3D object in some way, or what are the input controls? Is it a controller, or gesture controls, or the leap motion, or how are you actually able to kind of navigate around this 3D object in VR?
[00:05:10.322] Nicolas Ferey: Until now our work was quite independent on the way you navigate in your scene, but to navigate in your scene you have several options. I'd say that I'm mainly working in a cave system where basically mouse and keyboard are just forbidden because it just kills your immersion and you're even not able to use them in such environment. What we try to do is to use joypads or natural movements of the user. Our team is working on some algorithms that just by head tracking the user and by just providing some specific movement to the user thing, if you go one step to the left, okay, you will navigate on the left. If you do one step on the right, same thing, and you can adjust the speed by going a bit further on the left or on the right. And by using all this algorithm based on head tracking, and by combining them with navigation paradigms we set up around the protein, you are able, by just a step forward, backward, or on the side, to fully navigate in your molecular theme.
[00:06:09.513] Kent Bye: I see. So you're able to kind of move in a kind of a restricted space of just a step around, left or right, forward, backward. And by doing that, you're manipulating either rotating or translating in some way the 3D object, is what it sounds like.
[00:06:21.261] Nicolas Ferey: Yeah, exactly. You just consider yourself as a joypad, if we can say that.
[00:06:26.188] Kent Bye: I see. And what were some of the other talks that you heard throughout the day of this first annual VARMS, which is the Virtual Reality and Augmented Reality Molecular Science workshop within the context of the IEEE VR conference?
[00:06:40.143] Nicolas Ferey: So yeah, you often talk about sonification for instance, which is I think another step in the immersion world, and I think something really important. We didn't work on it for the moment, but you can see that a lot of efforts have been made on that. That can be a nice add to any immersive situation you would have to visualize molecules. It can be used, as we talked about during the discussion after the talk, to just situate yourself in your thing to always know where you are and where you want to go but you can also add some extra information thanks to this sound and it's important because you don't want to have this overload on the user visual kernel I'd say and you want also to use other way to communicate information to provide information to the user so this part of the workshop was quite interesting And the second part about serious game was also something really nice because what you want to do, okay, sometimes you work for scientists, you want to provide to the experts some tools to work with, but it's really important also to provide to the students some insight about what we are doing in molecular science. It's not always obvious to try to teach something. By using serious games and by using these educational tools, it's something you can easily do and that's something you do, like I'd say, with pleasure. You do it with some fun and it's always very nice to see that some efforts are made for that and are put in this direction. So, it was quite interesting to see what has been done and how we would have to aim our future developments to have something working.
[00:08:08.662] Kent Bye: And it sounded like you have a background in both molecular science and in virtual reality, is that correct?
[00:08:13.566] Nicolas Ferey: My main background was about molecular biology and I went to this virtual reality world a bit later because I started finally to work with virtual reality only three years ago when I started my PhD. I was coming more with a bioinformatics background but It's something when you enter in this world. I think it's yeah, you just feel there is something to do and you have another There's this other dimension the third one is like very nice to play with on Yeah, you see that there is a lot of effort that can be made to improve this visualization on everything. So But at the beginning I'm more biologist I'd say
[00:08:50.515] Kent Bye: That's interesting. And so it seems like you're using the tool of VR to do better or more sophisticated biology. And so I'm curious in terms of for your own personal research, what kind of the big open problems that you hope to solve within your domain of molecular biology using VR?
[00:09:05.782] Nicolas Ferey: Yeah, basically what we wanted to do, as I said, it's like using capabilities of virtual reality to bring the user in the center of its simulation, in the center of his simulation, to be able to drive what he has to do on a desktop, on a personal computer. You are quite limited in the way you interact with things on the way the information are provided to the user. You are always limited by the size of the screen, the amount of information you can provide. In virtual reality, you have so much new kind of information you can use. It's basically something we find really interesting on that. Yeah, that basically helps scientists to work daily, on a daily basis.
[00:09:47.890] Kent Bye: And so what kind of experiences within VR do you want to personally have then?
[00:09:52.975] Nicolas Ferey: One of the experiments I really loved to do, it was like a docking of two proteins that was made entirely in a cave system where you basically add two optic devices to manipulate your structure. So everything was in 3D, so you have this stereoscopic rendering. And the goal was just to dock these two proteins. And you had some sonification also to provide you some insight about if you are docking them well, basically. or not and all these haptic feedbacks that you had also to bring you information and you had all this information the beauty of just the rendering of the molecule and you had to dock them. I had some docking experience because I worked a lot with some docking softwares and this new experience was just like wow okay that's really cool and that's a new way to dock and to work in our field so I found it quite amazing.
[00:10:43.241] Kent Bye: Great. And finally, what do you see as the ultimate potential for virtual reality and what it might be able to enable?
[00:10:51.410] Nicolas Ferey: That's a big question. I think everybody is trying to answer it. I don't really know what would be the ultimate goal, but I think when everything will be possible to be done in real time and we will have all the tools connected in a way that everything from visual to audio will be connected in a good way, you'll be able to do, I'd say, whatever you want. I don't know in molecular biology what will be the next step. What we're trying to do on our side is to bring analysis, visualization on the whole pipeline, like put together to be able to shorten the pipeline that exists until now, which is observing a molecule, taking decision, running a simulation, getting back the results, and so on. And if you try to don't see it in a cycle, but on the lines where everything would be able to be done together, Okay, you shorten everything and you basically can provide answers to the experimentalists basically to test what you did and what you saw. Instead of taking, I don't know, a month of simulation, you could drive your simulation to make it in one week, I don't know, provide answers that would be as accurate as if you were running that on a distant cluster for months.
[00:12:02.196] Kent Bye: And if that was possible, then what would be possible within molecular biology if you were able to do that then?
[00:12:08.822] Nicolas Ferey: I don't know, you'd be able, I think, to see, I don't know, interaction between a virus and an antivirus, to molecules, to inhibitors, to whatever you would want, to see an entire ribosome system working in front of you, since you have these capabilities. So you could investigate, I'd say, so many things that I can't even think about one particular thing, but it will be, of course, very important for all the medical-related fields that really have an interest in this interaction between protein. And when you will be able to do that in virtual reality, to explore this interaction between protein, yeah, I guess you could investigate whatever you want. Okay, great.
[00:12:49.432] Kent Bye: Well, thank you. You're welcome. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash Voices of VR.
