Joel Zylberberg has a background in physics and cosmology, but then he got interested in theoretical neuroscience and he’s been researching the nature of perception in humans and in machines through the lens of computational neuroscience. He’s the Canada Research Chair and an assistant professor at York University in Toronto as well as Canadian Institute for Advanced Research (CIFAR) Associate Fellow of Learning in Machines and Brains.
Zylberberg was an attendee to the CIFAR workshop on the Future of Neuroscience and VR, and I had a chance to debrief him after the two-day workshop. He was really impressed with how much virtual reality technology has progressed over, and he’s shared that he’s a part of the EEG in the Wild research grant from CIFAR where he’ll be collaborating with Craig Chapman and Alona Fyshe on their project to automatically label EEG data with motion & eye tracking data from VR.
I had a chance to talk to Zylberberg about the neuroscience mechanics of perception, his work in computational neuroscience in trying to understand the neuroscience of perception, back-propagation, top-down feedback connections, & the credit assignment problem, deep learning machine learning models of perception, feed forward network architectures of convolution neural networks and recurrent neural networks, optogenetics,the ambitious goal allowing people to regain sight by trying to find a way to write information directly into the brain, the challenges of the data processing inequality from information theory, and the importance for VR designers to understand the fundamental mechanics of perception.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So continuing on in my series of looking at the future of neuroscience and VR, today we do a deep dive into computational neuroscience with Joel Zylberberg. He's an assistant professor of physics at York University, and he's the Canadian research chair in computational neuroscience, looking at all sorts of really interesting stuff. Joel's looking at computational neuroscience, trying to look at the mechanics of our perception, and then trying to recreate that within these artificial neural networks, and then also try to understand the fundamentals of deep learning and how that applies to how we as humans learn. So, we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Joel Zilberberg happened on Thursday, May 23rd, 2019 at the Canadian Institute for Advanced Research workshop on the future of neuroscience and VR in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:10.710] Joel Zylberberg: So I'm Joel Zilberberg. I am an assistant professor of physics at York University in Toronto and Canada research chair in computational neuroscience. Mostly we use machine learning algorithms to make computational models of how the brain perceives things in the visual environment and how the brain learns to do that perception.
[00:01:31.531] Kent Bye: Great. And so we're here at the future of neuroscience and virtual reality. And so for you, where do you land in this sort of intersection between immersive technologies and neuroscience?
[00:01:42.376] Joel Zylberberg: Very much on the neuroscience side. I think that a good understanding of perception could help make better virtual reality systems. It's also the case that the kinds of machine learning models we use in my lab to model the visual system could also help to design better stimuli, better visual environments for VR.
[00:01:59.560] Kent Bye: Well, maybe you could give me a bit of a primer of perception. Like, what is happening with all the systems to come together for us to understand what is happening in our perception of reality?
[00:02:09.584] Joel Zylberberg: Sure. Yeah. So my lab works at a fairly low level mechanistically, by which I mean we look at how individual neurons and then collections of those neurons respond to stimuli in the environment. And so we are less working at the scale of cognitive variables. That being said, light from the environment is lensed at the front of your eye, impinges on photoreceptors at the back of the eye on the retina. That leads to spiking activity in retinal ganglion cells that form the optic nerve, literally little pulses of electrical activity that flow into the brain. That then triggers additional pulses of electrical activity in neurons in the visual part of the thalamus, primary visual cortex. secondary visual cortex and so on as information travels along the ventral and dorsal pathways. And so at a really like straightforward kind of mechanistic neuroscience level, what's happening when you perceive something is that some group of nerve cells in your brain are being caused to emit little pulses of electrical activity. the link between those pulses of electrical activity and psychological or cognitive variables, like you getting a sense of it being a square or being red or being a happy scene or a sad one, that link is still a very active area of research.
[00:03:23.040] Kent Bye: Well, Anil gave a talk here remotely where he said the usual perception loop cycle would be like that there's a concrete reality, so that's the world, and then we have a sensation that we have in our body, and then we have our sense of self, and then from there we have the perception that is a linear cycle of things coming into the body, but he kind of took out this whole sense of the sensation itself and kind of consolidated those and said, it's actually like your beliefs and your category schemas that you have for understanding reality is some ways influencing what we can perceive. And so in some ways we've got prior experience, we've experienced something in that from that prior experience, we have a model and we may be looking at a cat, but we're not just looking at one cat. We're looking at all the cats we've ever seen in our life. And then it's from that that we're able to then match our perception of that cat based upon our prior experiences. So it seems like there's a drawing into memory component of our perception, but also a real-time filtering and distilling of what those sensations are.
[00:04:21.944] Joel Zylberberg: Yeah, so information in the brain, and I'll talk about the visual system in particular, really does flow in two directions. The first, which I sort of described a moment ago, is this bottom-up flow, or sort of from the sensors, like the eyes, into the brain and then through the visual hierarchy. There are also substantial connections that carry information in the opposite direction, what are known as top-down feedback connections, and they implement that kind of prior predictive component that Anil was alluding to. IT, inferior temporal cortex, which is involved in object recognition, sends information back to preceding brain areas like V4, V2, and V1, kind of priming them to look for specific shapes that might be indicative of objects that IT, in some sense, thinks might be there.
[00:05:08.095] Kent Bye: Well, I know there's been quite a lot of talk about convolutional neural networks in terms of, like, computer vision. And so maybe you could sort of recount a bit of a brief history in terms of, like, the different neural network architectures of, as we're evolving, there's been a huge amount of evolution over the last number of years. But there seems to be specific ones that are focused on vision. But I'm just curious if you could give me a bit of a context in terms of the evolution of the different architectures and the ones that you end up using for your tasks.
[00:05:35.715] Joel Zylberberg: Yeah, so to talk about different architectures, it's interesting. The diversity is actually less than it sounds like. The field of artificial neural networks started by trying to make simple computer models of the mammalian visual system. So the idea was that you would get some input into some early stage, and then a whole bunch of sequential stages of processing would happen, and then some output, similar to that bottom-up flow of information I described a moment ago, for information going from retina to V1, V2, V4, and so on. Those didn't work that well, in part because we didn't have the computing power to train really big neural networks, and in part because we didn't have good datasets to do the training. And so in 2012, there was just kind of a sea change in the field where sufficient compute power was assembled to really showcase the utility of these algorithms. It should be noted that there's a really critical subclass of these feedforward neural networks called convolutional neural networks. And there the idea is that instead of each input to the artificial neural network looking for its own feature in the world, And so you could have one input that looks for a feature at the upper left corner, or the bottom right corner, or the center of the visual scene, and they would each individually learn what their feature should be. Those are the old artificial neural nets where each unit just had its own little position in space and looked for its own thing, and they were all independently learned. The big change in going to convolutional neural nets is you trained a unit that would learn to look for some little visual feature, like let's say an edge at some orientation, but it would look for it at every location in the image and then basically make a map of the image by the extent to which that feature was present. So now for each of the input features, if you've learned a good feature, you don't just learn it at one position, you learn it at all the positions simultaneously, which gives an enormous amount more power to process images. So that advance is really one of the key things that Yann LeCun did in his early work that led him to be such a fixture in the field of machine learning. And then the one other sort of really important category of neural networks are recurrent neural networks. These are ones that activate sequentially on time series. But it has some memory as well, doesn't it, or not? So they end up having memory by virtue of the dynamics within the neural network. So imagine a system that responds to its inputs and changes very slowly. It'll end up remembering things for a long period. And that's because after something goes away in the environment, That change, the thing going away, will only be reflected slowly in the activity of the units. So by being able to learn dynamics, they can learn essentially different timescales over which to remember things.
[00:08:07.976] Kent Bye: Well, I think the complexity comes in into being able to layer these into different orders with either deep learning or deep reinforcement learning. And so are you also using deep learning techniques in order to train these things as well?
[00:08:18.843] Joel Zylberberg: Yeah, so deep learning, the name mostly applies to the idea of these feed-forward neural networks where just having many layers of them, so many sequential stages of processing, makes the thing deep. Colloquially, anything with more than three or four layers is referred to as a deep neural network. And the concept of deep learning refers to training those networks end-to-end. So you could imagine having a deep neural network and just training the top layer. That would be an example of shallow learning, but in a deep network. And when you instead sort of train every stage of that network, you get something that is deep learning.
[00:08:51.283] Kent Bye: Well, usually when you're training machine learning algorithms, you have to have data and preferably labeled data if you're doing supervised learning. And so where is the data coming from that you're using to train and how are you sort of labeling it?
[00:09:03.733] Joel Zylberberg: Yeah, so a lot of the data from my lab comes from brains. We published a recent paper where experimenters showed a bunch of pictures to monkeys, recorded activity patterns of neurons in primary visual cortex, and then we trained deep convolutional neural nets to take in the pictures and predict the activities of those neurons in V1. In other words, to make a camera-to-brain translator that would duplicate the processing of the peripheral visual system.
[00:09:27.564] Kent Bye: Wow. And so one of the things that also was shown during the course of this workshop was the idea that you can train up a neural network, but then have it in reverse to have like a visualization, like kind of the deep dream. It's a little bit of like a psychedelic, like, especially if you have a neural network trained on dogs, then everything kind of gets turned into dog faces everywhere, which is, you know, anybody that's seen the deep dream from Google can, see that kind of psychedelic imagery, but in some ways it's showing you a visual depiction of what it's looking for in terms of trying to parse out all these different scenes, and then you're able to, in some ways, peer into the mind of that neural network.
[00:10:04.179] Joel Zylberberg: Yeah, so we actually did precisely that in this recent paper. One long-standing question is to be able to look at individual neurons in, for example, V1, primary visual cortex, and know what visual thing in the world causes those neurons to be active. For a subset of neurons known as simple or complex cells, the answer to that question has been known for a long time. They respond to edges, so boundaries between light and dark things in the world. But there remained, for decades, many neurons for which we didn't have a good answer to that question of what makes them active. So in this recent paper, we took our trained camera-to-brain translator, so it takes in pictures, predicts the firing rates of neurons in V1, inverted it to synthesize images that were predicted to lead to high firing rates in those neurons, and we could then use that process to look at the visual features that cause neurons to spike for neurons that were previously not well understood. And what we found is a lot of neurons that were sensitive to textural features, so not these sort of sharply localized edges that had been known since the 1950s and 60s, but rather sort of more diffuse patterns in the image. And so those sorts of deep dream methods have really given us a new window into the visual system.
[00:11:18.101] Kent Bye: Yeah, and another big topic that's come up in the course of this workshop on the future of neuroscience and VR was brain control interfaces and BCI. And it feels like that a lot of this computational neuroscience that you're working on is in some ways creating this mathematical interface to be able to either detect what's happening inside of the brain, to be able to then detect different aspects of what is happening throughout the course of the entire body, whether that's what we're focusing on or attention or hearing what we're seeing or Recalling memories, but also being able to have like what we're thinking so like the sub vocalizations of what we're thinking But eventually into our memories and dreams and basically able to look at different ways of either invasive or non-invasive Preferably non-invasive I think for a lot of people but to be able to have like real-time interaction by just using our brain So I'm just curious to hear like where your work fits into that overall trajectory towards the brain-computer interfaces
[00:12:11.437] Joel Zylberberg: Yeah, so in the vision space, my lab is actually working on the inverse problem, essentially how to write visual information into the brains of blind individuals to restore their sense of sight. And they're having a high fidelity camera to brain translator that basically tells you the patterns of brain activity corresponding to any picture the person could see. is a key component. The other key component is a high-quality stimulator that lets you write those activity patterns into the brain. This is a long-term project that involves a collaboration with some device makers that are working on some of those stimulation methods.
[00:12:45.498] Kent Bye: Well, I've seen in some of the stimulation methods of using your tongue as an input device to be able to actually send a lot of that information into your body. There's so many sensors within the tongue that you could put some device on your tongue which could allow you to see. I don't know if that's sort of the approach or if there's other ways of how do you get the data into the body.
[00:13:02.987] Joel Zylberberg: Yeah, so writing tactile patterns onto the tongue or onto the skin is, I think, a reasonable approach to give people a sense of what's around them. It can help people with things like navigating and finding things. My goal is more ambitious than that, in that I want to actually restore the sense of sight. If I lost my sense of sight, I know that I would be very sad, and the opportunity to regain the ability to watch my kid playing with his toys would be transformational. And that's the kind of experience that we're aiming towards in my lab. That, of course, requires that we not just stimulate, say, the skin or the tongue in order to give people spatial information about the world, but actually that we manage to write into their brain visual information in a format that leads to the native sense of seeing things, of perception. The kinds of methods to do that now are highly invasive. The best known method, I think, is optogenetics. The idea is that one inserts into all of the nerve cells a light-sensitive protein called channelrhodopsin. And then if you shine a little spot of light on a neuron, you can cause it to activate. And then by writing patterned light across the visual cortex, so little bright spots on the neurons you want to activate, dark spots elsewhere, one can write patterns of activity into the visual cortex and generate the sense of perception. My hope, however, is that by the time we have finished the development of these camera to brain translator algorithms The device side will have caught up so that there will be less invasive methods to still stimulate with single neuron resolution, the visual cortex. Though that might be a little bit of an optimistic hope.
[00:14:40.775] Kent Bye: thing that comes to mind is obviously like the Matrix, where you're able to kind of jack into the Matrix. In that movie, you kind of stick something back in the back of your head. But I guess that's in some ways a dream of virtual reality, to be able to have a direct injection of these experiences and imagery. On the other hand, I've heard other people in the VR industry be like, whoa, whoa, whoa, we can use our existing senses. and especially use something like sensory addition or sensory substitution approaches where maybe you're rewiring that information but through your body, so using something like the neosensory vest, that as long as the data input gets to your brain to the point where what David Eagleman says and what he claims is that as long as that input is able to be correlated to some sort of visual feedback or haptic feedback or sonification, But you're able to do this real-time correlation that you can kind of train yourself over time so that you're able to expand your senses by using other parts of your body as, in some sense, a parallel processor to just kind of get the data into the brain. The brain doesn't care so much in terms of where it came from. It's just more of that it can get into the brain. And I guess in some ways, it's figuring out what the data input structure might be, but seeing if you could do that kind of real-time input and sensor fusion of everything.
[00:15:50.325] Joel Zylberberg: There's a big distinction between being able to use some input stream and actually restoring the native sense that would be lost. So for example, applying tactile patterns of pressure onto the skin or the tongue. can give a blind individual a sense of where objects are in the world or of the environment around them. But that is not the same experience as actually seeing things the way that we kind of take for granted. That being said, in the virtual reality space where most users are people without sensory deficits, it seems unnecessary to look for fancy ways to get information into the brain because we already have exquisite sensors on our eyes and our skin and our ears. And so there's probably no need for anything that's directly brain interfacing for VR.
[00:16:41.069] Kent Bye: Well, something like Vivid Vision from James Blaha, he was able to essentially train himself to be able to see in 3D. He's born with diplopia. He's not able to see in 3D in real life, but he was able to create a game which was able to allow him to start to slowly train himself to see in 3D. And so, just the idea that we have plastic brains enough that we can use the VR to train the muscles in a way that you could be trained. I guess that's a question as to like, what is the limit of when people come legally blind? their sensory organs become to the point where you can no longer use them at all versus like you can still use what maybe small window they have if they're legally blind to be able to maybe focus and send a high dense amount of information through their eyes and still be able to get the information to them. But I guess where's the limit? I mean, if someone is legally blind, like would you still be able to do that? Sounds like you want to be able to use their eyes, but if they're blind, then how do you do that?
[00:17:37.321] Joel Zylberberg: That's right. I mean, the data processing inequality is one of these fundamental results from information theory that says that you cannot have more input downstream of a sensor than was present at the sensor itself. So if there's really loss of incoming information from the eyes, there's not much learning downstream that can restore that missing input. Basically, the camera is not capturing the information, and so it's not there. And so in that case, to sort of fully restore sight, you need some mechanism through which to get the information in that should have been recorded by the eyes.
[00:18:12.100] Kent Bye: Great. And I'm just curious after being here at this future of neuroscience and VR workshop, like what are you taking away from the discussions that were had here over the last couple of days?
[00:18:21.081] Joel Zylberberg: Yeah, so being not myself a VR researcher, one thing that I found quite amazing is just the amount of progress that's taken place within the VR field. In fact, I should say myself not being a VR researcher or VR user, I've been quite impressed by seeing the advances in the field. Especially some of the work that Craig Chapman and Sid Kouidje and others presented on sort of ability to use VR to manipulate sensory environments in ways that could not be done with real physical objects, I think has a lot of possibility for sensory neuroscience going forward.
[00:19:00.004] Kent Bye: Yeah, it was interesting for me to see this intentional binding that happens where you're able to spoof or trick the body and thinking that you actually lifted up your arm and pushed a button when in reality you're not moving at all but you get the visual feedback of being in a virtual environment and seeing like a virtual hand move up and push a button but then giving haptic feedback. There seems to be really interesting ways of starting to do experiments within VR that you would never be able to do in reality. And maybe you start to have these virtual experiences where you're able to do this A-B stress test, where you're able to simulate the visual experience, but to try to isolate whether or not it's more of the visual information that you're getting, or whether it's a combination of the haptics. But it just seems like there's a lot of potential neuroscience applications to be able to do these types of things.
[00:19:46.329] Joel Zylberberg: Yeah, it's important to note though that neuroscientists have been studying that problem of multimodal sensory integration for decades and have done it by things like having a sound played into the ear while you're looking at something and the sound either does or doesn't match the video that's being played. or touching the person's hand while they look at things or not, and looking at how those different information streams are combined in the brain when they're either in conflict with each other or not. And so VR, I think, presents a new tool to add some richness to those experiments, but it's important to acknowledge that, you know, the neuroscientists are clever and have come up with ways to answer most of the questions that they've been asking, even with the old methods that they had.
[00:20:30.980] Kent Bye: And I've gone to the International Joint Conference of AI three times now. And so one of the things that I found was that there's sort of an open question in terms of the mathematical foundations of machine learning to really understand how you do this translation and connect the dots between these models that are being created and reality. I'm not sure, because you were in the sort of physics and computational neuroscience, is that something that is an open question or doesn't really matter? I'm just curious, hearing a little bit more of the foundations of machine learning in terms of how it's able to provide maybe some sort of mathematical formalism into something that is an organic process that's unfolding.
[00:21:07.610] Joel Zylberberg: So artificial neural networks are mathematically what's known as universal function approximators. There's some beautiful theoretical results going back to the 1980s that essentially say that a sufficiently large artificial neural network can learn any possible mathematical mapping a function, mapping inputs onto outputs. So in that sense, sufficiently large neural nets should be able to approximate reality with sufficient training data and sufficiently high quality training procedures. They should be able to approximate reality to arbitrarily high precision. That's not to say that they actually recreate it. So that's perhaps an important distinction to be made.
[00:21:46.216] Kent Bye: And so for you, what are some of the either biggest open questions you're trying to answer or open problems you're trying to solve?
[00:21:52.848] Joel Zylberberg: Yeah, so one of the biggest problems we're working on in my lab is to understand how the brain implements deep learning, if at all. And in both the brain and in the machine learning algorithms, there's the same credit assignment problem that's faced. And the idea is if you have a multi-layered neural network, or like a sequential information processing system, if there are some errors at the outputs, you then want to somehow update the whole network so that those errors become less over time. That's been the magic to deep learning, is updating all of the stages in the system. The question is then, if you're one of the early stages in the system, how should you update yourself so that the output, many stages downstream, gets better? That requires that you, as an early stage unit in the system, somehow know information about all the stuff that happens between you and the output. In other words, downstream of you. And it's still unclear exactly how neurons in the brain receive all of that downstream information about how they contribute to system-wide errors. That's called the credit assignment problem because it, in a sense, comes down to telling each neuron what their credit or blame is for system-wide errors. In artificial neural networks, we've engineered solutions to that problem known as backpropagation. It's how we train artificial neural nets. Basically, one takes the error information from the output, sends it back down through the hierarchy to each preceding stage in the artificial neural network, and as a result, broadcast to each unit sort of what their contributions are to system-wide errors. And one joint theory experiment problem we're working on now in the lab is to understand how brain areas can actually broadcast that error information back down to earlier brain areas in order to coordinate learning. This is a collaborative project with Yoshua Bengio and Tim Lillicrap, Blake Richards and myself, and then the Allen Institute for Brain Science in Seattle is actually doing all the experiments for us.
[00:23:45.456] Kent Bye: So you had mentioned that you're working with CIFAR in some capacity. Maybe you could elaborate on what your relationship and role is with CIFAR.
[00:23:52.644] Joel Zylberberg: Yeah, so I am a CIFAR Azrieli Global Scholar. This is a program they launched about three years ago that is a young investigator grant. So scientists who have recently started their own independent research lab, typically professors who have just started their first professor position, are eligible for these awards. They come with a small pile of money that's unrestricted for research, $100,000. And in addition to that, these appointments put those global scholars in CIFAR research programs. So myself, I'm a part of the Learning in Machines and Brain program, and so those programs we meet roughly every six months and talk about the science that's ongoing in our labs. And through those meetings, collaborations naturally form. And so I think, to me, for the CIFAR Awards, these Global Scholar Awards, the money was nice, but the interactions with some of the really leading figures in my field have proved to be much more transformative.
[00:24:51.231] Kent Bye: Is that a part of the reason why you're here, to kind of cross-pollinate and learn what's happening in the VR space?
[00:24:57.120] Joel Zylberberg: Yeah, so CIFAR is an unusual agency in that they have this set of experts in lots of different areas, and they're enthusiastic about bringing them together on essentially anything in which there might be some overlap. So, for example, I'm not a VR user or researcher, but I know something about sensory neuroscience and about machine learning, and so they asked me to come to this VR meeting to contribute to the dialogue. And my takeaway from this, some of the observations about how far VR has advanced are such that I could imagine in the near future doing some VR work in my own lab, possibly in collaboration with some of the people here.
[00:25:36.023] Kent Bye: What would you want to look at?
[00:25:37.671] Joel Zylberberg: Yeah, so one example is actually something that's already ongoing, so I maybe undersold it by saying it could happen in the near future. This is a collaboration with Craig Chapman and Alana Fish, who couldn't be here. We have a small catalyst grant from CFAR to do this. But these are experiments that Craig actually showed at this meeting where they are recording EEG signals while people do a task like moving boxes of pasta around between tables. And the idea is can we somehow decode in advance of their doing the movement from the EEG signals, decode what the person is going to do. And so what my role is in that project is to train artificial neural networks to take in these patterns of EEG and predict upcoming movements from the person. We had intended to start doing this with the data from the person behaving in the real world, Although, with the advances in VR that Craig has brought into his lab, I think we could do the corresponding analysis on the VR virtual motions and, well, one, compare those to the real-world motions. It'd be interesting to see if the same EEG patterns apply in the two cases, but also might lead to things like brain-based controllers for VR systems.
[00:26:45.619] Kent Bye: Yeah, what was interesting to me in talking to Craig was to hear how he's kind of taken a layered approach of labeling the data from the behavior, being able to look at it, do some machine learning to be able to then extract automatically the behavior tags and labels, and then from there, be able to use those labels to inform what's happening on the EEG. I see this future of having these many different sensors that are coming in, and then maybe you're able to have this chain reaction where you're able to label one thing, but then draw out different connections for other things in terms of trying to extract the fundamental features based upon those movement labels that you're adding onto that.
[00:27:20.883] Joel Zylberberg: Good tools build on themselves, so progress is exponential.
[00:27:24.805] Kent Bye: Cool. And finally, what do you think the ultimate potential of all these immersive technologies are and what they might be able to enable?
[00:27:34.823] Joel Zylberberg: I'm not sure I have much to say about the ultimate potential. There's a lot. And I'm looking forward to seeing where it goes.
[00:27:43.091] Kent Bye: Awesome. Great. Is there anything else that's left unsaid that you'd like to say to the wider immersive community?
[00:27:48.768] Joel Zylberberg: Knowledge of how brains perceive things will probably help to make not only more immersive experiences, but also ones that are more efficient. An easy example of this is if you look at color television screens. They have three channels, RGB. That's not because somehow the world only has three different colors of things. It's because we have three different types of cone photoreceptors sensitive to long, medium, and short wavelengths. And so insights like that from the biology can help to make efficient and yet rich experiences for users.
[00:28:18.597] Kent Bye: Awesome. Great. Well, thank you so much.
[00:28:20.738] Joel Zylberberg: Thank you.
[00:28:21.778] Kent Bye: So that was Joel Zylberberg. He's the Assistant Professor of Physics at York University in Toronto, as well as the Canada Research Chair in Computational Neuroscience. So I have a number of different takeaways about this interview is that first of all, Well, this was a deep dive into a lot of different topics, especially around the neuroscience mechanics of perception, looking at how light goes into the optic nerve and all that's translated into electrical signals and sent into the brain and, you know, stimulating different activity. And it's causing you to perceive things. But, you know, the thing that Joel is saying is that there's kind of like these bi-directional processes, they're going forwards and backwards and. The information that is flowing in the opposite direction is called the top-down feedback connections, and those are implementing those prior predictive components, priming them into different shapes that might be indicative of objects that might be there, and where what he says the IT might be perceiving, that's the inferotemporal cortex. Part of the predictive coding hypothesis is that we are coming up with existing concepts of what information should be there. And so there's this priming that comes from a more cognitive and psychological aspect. And that's a bit of a mystery is to try to decode it to that level, to see what is all of our memory and our psychological components and how that has that active feedback mechanism with the mechanics of the neuroscience of perception. It sounds like they're able to trace down pretty closely all the pathways for how the light gets transmitted into these electrical signals and gets propagated through the brain through all these different areas of the brain. But you know how that actually gets translated into getting a sense of it, the quality of it being square, being red or being happy or sad, all the different qualitative aspects. That's still a pretty grand mystery as to the nature of consciousness and how that gets translated into those qualia. And that's an open question that is still being actively researched. So, you know, like I said, there's this bi-directional aspects that are going, and then within the neural network architectures, they call it backpropagation, but it's this challenge of like, when you have an error, then how do you sort of distribute all those error codes amongst all the previous layers? Sounds like within artificial intelligence, they use this technique of backpropagation, but that's what he calls the credit assignment problem. So when you have a multi-layered network, then how do you assign the errors to all the individual neurons? and try to break it up into the whole gestalt and see how it kind of propagates through all the many different layers of a deep learning network. So yeah, very fascinating for me to hear Joel just talk about this process of trying to look at the insights of perception, being able to actually stimulate things in terms of looking and capturing all this data from how mammals perceive, and then have that data and create all these different machine learning models. And then, you know, he wants to get to the point where he can actually like understand all the mechanics of the neuroscience of vision and to be able to actually like stimulate all the different aspects of the parts of the brain that sees if you're blind and you can't see, then are there other ways to get that information into your body? He's talking about this fundamental result from information theory called the data processing inequality, which essentially says that if you can't get the data into your brain, then you're not going to be able to make up for it later. Like if you're not taking that as an input, then it's hard for you to synthesize it and make sense of it and reduce it down if you don't have access to that data in the first place. And so if you're blind and you can't see, then how do you get that information into your visual cortex? So there's other aspects of looking at sensory addition, sensory substitution. I'm talking to David Eagleman, you know, he seemed to be pretty confident that you just needed to get the data into your brain. But from what I hear, what Joel is saying is that, you know, it's kind of like putting information through your body. That's going to be interpreted through more haptic ways of interpreting information that may be able to get translated into giving you a sense of space and where objects are, but that's going to be a little bit different than actually recreating and stimulating your brain, be able to actually see and perceive things. He was talking about something called optogenetics, where you would actually like have ways of stimulating different aspects of your visual cortex. And he seemed to be talking about different technological options to be able to do that. I don't know if he was referring to something like Neuralink or other more invasive technologies. People who have huge impairments or who are blind may be more willing to do these more invasive techniques. And I think he's kind of hoping that once they kind of figure out the mechanics of it, that eventually it gets to the point where you don't need to do the more extreme invasive techniques that maybe you can start to use other existing methods. I've definitely seen different assistive technologies that, you know, if people are maybe legally blind, but they still have some parts of their vision that they could still use, that there are these assistive technologies that can actually kind of focus a highly dense amount of information into that area of the parts of the eye they can see. And you can have people who are wearing these assistive devices that are otherwise legally blind that could actually just start to function in society by using these different technologies. So I've definitely seen that. And I have an interview with somebody who has one of those assistive devices that I did back at VRTO in Toronto back in 2018 and hope to get to that interview at some point here in the near future. So this was a really fun interview for me to dig into a lot of the mechanics of perception and talk a little bit more about the mathematical foundations of machine learning. Looking at this concept of the universal function approximator, just saying that with a sufficiently complicated neural network, you can start to mathematically model just about any different types of math functions, which I think is super fascinating to hear that you can start to do this function approximation through these neural networks. So he seemed to say that these neural networks could at some point start to approximate reality to an arbitrary level, which is a pretty amazing thing to think about. And that, you know, this whole Azalea Global Scholar Program from the Kennedy Institute for Advanced Research, CIFAR, he got a Young Investigator Grant. He was able to get some money for research, but it's really, you know, bringing them together and be able to collaborate and interact with these other Scholars, he's going to be working with Craig Chapman and Ilana Fish, uh, working on this project that I talked about at the top of the series with Craig Chapman, being able to integrate the trifecta of EEG data with eye tracking and motion track data, and be able to use the behavior in the motion as well as eye tracking data to be able to automatically label information that's coming from the EEG. And so it sounds like Joel is going to be working on some of the more machine learning aspects of that and. trying to come up with this automated process and try to fuse all this information together and perhaps even start to come up with these neural predictions to be able to see what's happening in the brain to be able to predict what's going to happen next, which I think is pretty exciting to see how all these different technologies are starting to come together. And his final thought was just that if you have a knowledge of how the brain perceives things, then you're going to make it just more efficient and rich experiences that you're going to be able to create if you know more about the fundamentals and basics of perception. So for me, uh, going to this Canadian Institute for Advanced Research meeting happened over the course of two days to see a variety of different presentations, uh, hearing some information about the cutting edge of neuroscience to hear about, you know, what's happening from other people, what they're doing from the VR field. If you're listening to the voices of VR podcasts and, you know, I've been covering a lot of what's been happening in the VR side of things, but it was just nice to have those two worlds come together and to get a lot of these different information exchanged. There's a lot of excitement, I think, from a lot of the neuroscientists who weren't necessarily up to speed as to everything that was happening in the realm of VR. And I think that just kind of opened up their mind as to what type of new research possibilities could start to be opened up by starting to use some of this immersive technologies. And I think there is still this aspect and component of having the game designers, the storytellers, the experiential design creators to have really mastered the aspects of agency and interactivity and what makes something fun. And I think there's just gonna be a lot of ways in which those skills are gonna be fused together and perhaps be in direct dialogue of working with these cutting edge neuroscience researchers to figure out how to collaborate and start to figure out how to maybe create these embodied interactions and experiences that could then start to be used for neuroscience research and to understand the fundamental components of our perception even more than we already do and perhaps get more insight. to learn a lot more about the nature of consciousness as well. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, there's a couple things you can do. First, just spread the word, tell your friends. This podcast relies upon the word of mouth to be able to continue to grow and to reach all sorts of people from around the world. And if you enjoy this podcast, then please consider becoming a contributing member to the Patreon. This podcast is freely made available to you and the rest of the VR community, and I'm only able to do that with the support of Patreon and the people who want to continue to sustain this work and to allow me to continue to travel around the world and to meet with all these different people and to help document the evolution of spatial computing. So if you want to see this real-time oral history of VR continue and to grow and to continue to reach new audiences around the world, a lot of people will listen to the voices of VR and get bootstrapped into the immersive industry. And so you can start to pay it forward to have other people who have an opportunity to be able to dive in and start to really become a contributing member to this larger movement of spatial computing. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
