Dr. Jacquelyn Ford Morie was a co-founder of USC’s Institute for Creative Technologies (ICT) where she spent 13 years as a Sr. Research Scientist. She is also the founder and chief scientist of All These Worlds LLC.
Jacquelyn has been working in Virtual reality for over 25 years since 1989. She comes from an artist background where she found her medium was to create emotionally evocative virtual reality environments. Interestingly, she found that a majority of the VR experiences created before 2007 were created by women.
She covers a wide range of the history of virtual reality starting with Ivan Sutherland in the late 60s, the military simulations up into the 80s and 90s, and then up to today. She has done a number of VR experiments, and helped create a language for emotionally mapping a VR experience based upon biometric feedback. She also invented a scent collar in order to bring the emotional power of smell into immersive experiences.
Jacquelyn talks about a number of the different military training simulators that she’s worked on as well as some of the recent research into mindfulness training in virtual worlds and using social experiences with AI within VR to minimize the isolation of NASA astronauts on extended missions like going to Mars.
Overall, Jacquelyn has a wealth of information about virtual reality and has a very unique perspective about the history of VR considering that she’s been a part of it for the past 25 years. There’s some interesting connections to inspirations from sci-fi literature, and the reason why the military wanted USC’s ICT to be located in Hollywood and it’s connection to Star Trek and the creation of the Holodeck.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.916] Jacquelyn Ford Morie: My name is Jackie Maury, and I've been involved in virtual reality since about 1989, when I started working at the Visual Systems Lab, which is part of the Institute for Simulation and Training in Orlando, Florida, at the University of Central Florida.
[00:00:27.086] Kent Bye: Great. And so what got you interested in getting involved with virtual reality?
[00:00:32.326] Jacquelyn Ford Morie: It was a very exciting medium, and where I was coming from was from being a fine artist who was making assemblages and 3D environments that I wanted to put people into. When I was making them physically, they kept getting bigger and bigger and bigger, until it was ridiculous to try and build a full room, so then I started making them in 3D digitally. And then virtual reality came along, and I went this is exactly what I've been looking for. Before that I'd been showing the 3D digital environments as stereo pairs with glasses that you would look at them against a bright light. So this way they became interactive. I could give the people agency to walk around in them. The problem was that virtual reality back then was so boring. It was exciting in its potential and boring in its execution. Because I think in the original heyday of virtual reality, when it was first coming on, with the exception of people like Jaron Lanier, people did not have a lot of imagination. So you got these very boring walkthroughs. I wanted to go beyond that.
[00:01:43.619] Kent Bye: I see. And so when you go back to the history of virtual reality, where do you start in terms of where it begins and where the first seeds of virtual reality planted?
[00:01:53.657] Jacquelyn Ford Morie: I think like many new areas for humankind, it starts in literature. So it starts with the imagination of the writer who can think about things that are not able to be experienced yet. So I look back to science fiction. One of the first stories I think had to do with virtual reality in this kind of immersive environment was Ray Bradbury's The Veldt. And the VELT was about a family, and the parents wanted to give their children the latest multimedia room. The multimedia room had one wall of their nursery, as they called it, that was an entrance to a virtual world. And the world that they happened to have in the story was the African VELT. So the kids keep going into the VELT and experiencing stuff, and then they don't come back, and the parents go in, and I believe there are some dangerous lions in there.
[00:02:43.748] Kent Bye: I see. And so how did it go from there into, you know, I've heard a lot about Ivan Sutherland and the work that he was working with the sketchpad and then the first virtual reality HMD that was the Sword of Damocles in 1968. So where do you kind of see where the actual technology started to come together then?
[00:03:01.070] Jacquelyn Ford Morie: Well, like so many things that we take for granted today, our government, our DoD, funded them. So, not only Ivan Sutherland, who did his dissertation on interactive graphics, which was Sketchpad, they funded the Sword of Damocles, which was a way to put these displays on someone's head so that they were fully immersed. But if you even go to computer animation, Ed Catmull's early work was funded by DARPA, or ARPA, it might have been then. And we would not have computer animation today if they had not funded him to go off and pursue wild and crazy ideas in this new field of computer graphics. So I think the technology coming together, it's been a long, long road. And we've had the technology, basically, in place for 25 years. But it's been so expensive. We still have some of the same issues. But we have them with a $300 head mount instead of a $30,000 head mount. So these things are just coming down to a level where so many people can now work with them. They're not locked behind these research lab pearly gates. People can work with them. And as that happens, we're going to find more ways that we can really experience this kind of immersive technology.
[00:04:17.332] Kent Bye: And what happened with virtual reality from like the end of the 1960s up to when it had its resurgence in the 90s I mean imagine it didn't just disappear when you look at the history up to that point what sort of happened in that period
[00:04:29.042] Jacquelyn Ford Morie: Between the 60s and the 90s, I wouldn't say virtual reality was out there. I mean, it really wasn't out there until we started getting the likes of Jaron Lanier's VPL or the Abrams Genteel Power Glove that was kind of usurped into the virtual reality realm. When you think about it, a lot of that technology was out there, but people hadn't put it to use for something like virtual reality. So Jaron Lanier got Tom Zimmerman's air glove thing that he was using to do like air guitar and put it to use in a virtual reality situation. So the technologies were there. They were either being used for something else, by something else, or they were really expensive technologies like the sort of Damocles behind research laboratory doors that were funded by the government. So it took quite a bit to get it to come out and that happened around the mid to late 80s.
[00:05:23.967] Kent Bye: And when you said to get it to come out, I guess is that implying that it was continuing through the DOD through that whole period, but it was just underground?
[00:05:31.148] Jacquelyn Ford Morie: I wouldn't say it was underground, it was continuing, but I don't know that it was a concerted effort to put all of the different kinds of technologies together. Certainly people talked about what it might be, and the DOD's interest was to use it for training. Rather than putting soldiers in harm's way for training, they could put them in a virtual space that would train them for what they needed to know physically and kinesthetically, but they could do it in a safe way. So it wasn't that it was underground, it was just that we think of it now in history, and we say, oh yeah, it was this and this and this and this, but those were very small and isolated funding projects. There was no sort of, this is virtual reality. until it kind of hit the mass population in the mid-80s. And then Jaron became this poster boy for it. It reached a lot more people. And the science fiction novels started being made into movies of sorts. And then it kind of captured a popular imagination. Unfortunately, I think some of the movies did it more harm than good. But at least they sort of raised the zeitgeist of how we would actually live and work and think about being in an immersive world.
[00:06:45.102] Kent Bye: Yeah, the thing that I find interesting is that Ivan Sutherland was a part of ARPA, which was also funding different projects like the Internet, which the Internet started to have its first transmission like in 1969, and then jumped to the 90s. The Internet is coming into full bloom with the World Wide Web at the same time that virtual reality, the same thing that he created in the 60s, was also having this sort of revolution. Also literature like Snow Crash and Lawnmower Man and In terms of picking out like key moments that were happening within the evolution of virtual reality in the 90s, what were some of those milestone things that you would say are kind of like a turning point in terms of the history of virtual reality?
[00:07:24.235] Jacquelyn Ford Morie: I want to go back to the 80s, because I think the one thing that gets overlooked in the history of virtual reality is what was going on with multiplayer, social, virtual worlds. So, in the mid-80s, we had Habitat. So, this was Randall Farmer and Chip Morningstar working for LucasArts Film. and they made a complete virtual world. You were only 2D, but you had all of your avatar stuff. You could customize your avatar. You had virtual shopping. You had everything that we think of as a modern virtual world in something like Second Life was in Habitat in 1985, 86. And we don't really put that in, but I think those kinds of more game or social virtual worlds They developed along the same kind of tracks as virtual reality, but they're converging now. So what we're finding is that we're having social ways to experience virtual reality that we might not have had before. We did have network stuff in the early 90s. In the early 90s there were great movements to try and put it into location-based entertainment venues. So you had Jonathan Waldron doing some of his multiplayer games. You had a number of people looking at it in terms of what it could do for location-based entertainment or the theme parks. It was never able to circumvent the problem of throughput. So you couldn't get enough people through to earn your money for what it cost you to put together these things, and the people that had to be there to suit everybody up, and that type of thing. So, Disney probably made the best go of it with the Aladdin's ride, but they didn't sustain it either. So, in the mid-90s, it went off to try and be different things. It tried to be training for the military. That, again, didn't scale, because they didn't have the multiplayer thing, except for something called SimNet. So SimNet was a networked Abrams tank simulator. And it was a two-part thing that you had the driver in one part and the commander and the gunner in another part. And there were 250 of them networked together at Fort Knox. And this was created by Jack Thorpe. That was a really cool thing. So they could actually train a whole bunch of people at one time as a group. But the individual training didn't work out as much for the army because they didn't train individual soldiers. It was how to perform in your squad, in your troop there. So it had failings there. The location-based entertainment, we talked about having failings. It was just hard to make commercial products out of this stuff. There was huge technical overhead to get all these things working together. So I remember when we were showing things, I think we showed the very first VR piece at a film festival. And we were crawling around on our hands and knees and bellies trying to get cables together, running to Radio Shack for parts. And up until the moment we went on, we were still trying to get these things to talk to one another. So that technology overhead, not only in the coding, but in the hardware. We had one reality engine where they came and put in these cables that enabled us to use these very high-res head-mounted displays. They couldn't get the cables folded up back into the box and put the top on.
[00:10:42.351] Kent Bye: Wow. And so you were mentioning to me last night that you had done a PhD dissertation on a number of different pieces on virtual reality. Maybe you could describe to me what you were looking at and what some of your findings were.
[00:10:55.182] Jacquelyn Ford Morie: So my dissertation had two parts to it. One was looking at all of the virtual reality, more art pieces that had been done since the beginning until about 2007, which is when I turned that in and got the PhD. The other part of it was on the works that I had made. So there were about a hundred of these very artistic VR pieces, and I did a big matrix on what kind of equipment they used, did they use anything else like smell, or how big was the team, what kind of computer did they use, what kind of software to build the environment did they use, where it was shown, where some of the information about how it was accepted was. So it was a huge matrix. Now the interesting thing to me was in looking at that, 70% of those hundred pieces were either headed up by or primarily done by women. And that was just this huge shock because we all thought it was a very male-dominated industry or technology. So there were very few women working in it when I was in it, but it turned out that a lot of women gravitated to this as the best way to do their art that they had found.
[00:12:08.591] Kent Bye: And what do you think that is in terms of what are the things that you think were drawing women to the medium of virtual reality to express their art?
[00:12:16.453] Jacquelyn Ford Morie: That's a really good question and I think there are probably a lot more dissertations out there that could talk to this particular topic. I proposed one possible explanation and that is that Women may be more predisposed to creating an environment that isn't an authored environment. So a film is an authored environment because the director, especially an auteur director, is going to make sure that you're looking. Every camera thing is the way they want you to look. They're putting your attention here. They take you through the story arc exactly how they want. It's all very carefully contrived. When you build a virtual environment, it is an open, freewill space. So you're giving somebody something that they have to complete to find the story, to find the experience, to find the meaning in it. And I think it might be easier for women to do that, just like you're raising a child. You don't know how that child's going to turn out, but you're giving that child a space of possibilities that help it form. And I'm wondering, and I really don't have a good answer for that, but I'm wondering if that might be why I got that 70% of these things being made by women, because it's easier for them to let go of some of that authorship, perhaps.
[00:13:34.911] Kent Bye: Could be. Yeah, the thing that I think of is sort of the principles of yang and yin, where yang is more masculine, out running through video games and killing people, and the yin is more receptive, and so you're kind of just taking it all in. It's more associated with the feminine archetype, and so I see virtual reality experiences and it was really interesting to see some videos of people like Tested who are these gamers and they're like jumping into this first person shooter and their first reaction is, you know, I don't really want to run around, I just kind of want to look around and enjoy the scenery and enjoy being in this. And more of a receptive energy rather than sort of running around and killing people because first of all it's nauseating but also when you're immersed into virtual reality, you know, it's the first thing you really want to do is go kill somebody. So that's sort of how I've been thinking about it.
[00:14:21.413] Jacquelyn Ford Morie: I have a couple good stories on that. So I did this very multi-sensory, high-end, immersive work called DarkCon. It was part of a project called the Sensory Environments Evaluation Project at the USC's Institute for Creative Technologies. Started about 2000, finished about 2005. And it was a quasi-military thing. You were a forward scout. You were dropped into this area of Bosnia. You had to decide whether there were paramilitary forces or refugees living in this abandoned building complex. So I didn't want anybody shooting. Nobody got a gun. You were a forward scout. And we had two very interesting reactions. First of all, most people who played games dashed through the tunnel that you were dropped in. They just dashed through. All the clues about whether it was rebels or refugees were really part of the detritus and the environment within this culvert. and most people missed them. The second interesting story was a woman who went into it and said, why can't I be a refugee? Why do I have to be the forward scout? So I thought those were both very interesting observations.
[00:15:33.395] Kent Bye: Talking about narrative, and you mentioned that you'd done some sort of emotional mapping in terms of trying to qualify a virtual ad experience. What have you done in terms of that? Maybe you could explain what that means and what that is.
[00:15:46.543] Jacquelyn Ford Morie: So my entire interest in VR was motivated by making them not so boring. I wanted to evoke emotional responses from the people that experience these things. How do you do that? How do you do that in an open, free-will environment? So we came up with some interesting things. And way back in 1990, I partnered with someone at the Visual Systems Lab. And we started building virtual reality environments that evoked emotional responses. We did it by doing each environment to evoke a specific emotional response. So, for example, one of them was a 30-foot high spider. You're dropped into this environment with a 30-foot high spider looming over you, and you're on just kind of a flat plane that looks like a web. And, gee, a blood-red background, but, you know, we did want you to be kind of afraid. So what we did was we gave the spider eight states. It could ignore you. It could be interested in you. It could come up to you. It could attack you. It could kill you. So there were eight of these. And what we did was associate each state with a different heartbeat rhythm. So if it was ignoring you, the heartbeat was pretty calm and slow. And if it was interested in you, it got much faster. And if it came to attack you, it got much, much faster. So you'd be hearing this, dum, dum, dum, dum, dum. And what happens is what they call entrainment. So your heartbeat, this happens to you with music, syncs up with the rhythm of what you're hearing. So people got extremely scared in this, and we had some people rip the head mount off and run from the room. We did more subtle things like looking at nostalgia. We had an endless forest with wraiths that sang this haunting chorale. They would go off in little groups and float through the trees. You would try and get close to them, but you could never get close to them. There was this haunting sound and the forest went on forever. That was a really interesting one. I took that idea of evoking emotional responses much further when I did this sea project. And we looked at things that really do evoke emotional responses. So some of them are things that you might call cliches. So I know if you're in a dark tunnel and you hear strange skittering sounds, it's probably going to unnerve you a little bit. Or if you're walking through the tunnel and you step on a twig and it crackles and it startles a bunch of bats hanging from the ceiling and they go, flying off, you're going to be startled. So we know how to do that, and good filmmakers know how to do this. So we use those kinds of techniques to get these emotional hot spots throughout this environment. We also used a number of other things, but I'll talk about two other things we did. to get the emotional rise out of people, and one was we used scent. So I wanted to use scent, and at the time there was only these giant cartridges that you could put in these tower-like machines, and it would scent up the entire room, and it would be stinky, because we didn't always use good scents, mainly like diesel exhaust or something like that. And the other researchers really hated that smell. They said, you're not going to play this. So I said, all right, I'm going to invent something that keeps the scent close to your face. So I invented a scent collar that is Bluetooth triggered, releases the scent in very small amounts. It doesn't spray it at you. It's a molecular drift system. Puts the scent by your nose. And you smell this at the places in the environment where you should smell it. Smell is very powerful. go straight to our limbic system before we cognitively process it. So that was one thing. The second thing was an infrasound floor. So I invented this infrasound floor. And this was a raised floor that the person with the head mount would stand on. And it had 10 subsonic transducers that could be addressed and play from 4 to 20 hertz. So below the level of hearing. So you never heard it, but you felt it. And you actually were more uneasy when this was playing than you were when it wasn't. So we could say, okay, now you're uneasy. Cut the infrasound. Okay, they're calm. So when you were going through the tunnel and everything was scary, we played it. And then when you got to the river, all the smells were beautiful and fresh instead of the rancid smell you had in the culvert. And then the infrasound stopped. until things heated up again. So we actually scored this like an emotional score. So those were two things we used to bring really strong emotions into a free will experience.
[00:20:18.160] Kent Bye: Yeah, and I know from the consumer virtual reality perspective that a lot of the senses that they talk about are the vision for the head mounted displays, you know, haptics with the feel and touch, and then the sound with the binaural sound. But they don't get a lot of input in terms of the smell, but smell seems like it does have a potential to go straight to flashbulb memories. And, you know, maybe you could talk about, you know, how can you use smell to create an additional level of immersion in presence?
[00:20:45.798] Jacquelyn Ford Morie: Well, exactly what you said. If you are going to be in a virtual environment and you experience a smell, it's going to do something to you, whether you want it to or not. It's not like you can be inured to shooting people in a first-person shooter game. You are always going to have a reaction to a smell. Now, it may be stronger or weaker depending on your history with that smell. So you may hate peppermint, but most people like peppermint. But you may also, like I love the smell of diesel exhaust because I traveled on a bus for a long time as a teenager. So I find that smell not nauseous and a lot of people do find it that way. So scents are very personal. It's really hard to score them because you can't totally predict how they're going to affect somebody. But you can make your experience more what you want it to be. So I did one experience where you were a baby in a crib. And you don't have a lot of agency in that because you're just laying in the crib and all these giant pictures of grandmothers and mothers and stuff come fading in and out above you. But when the mothers come in, you smell the mother's perfume. And the rest of the time, you're kind of smelling baby powder or some other things. But the mother's perfume, it was like the theme song for that person. So it was a scent theme song, if you will.
[00:22:08.907] Kent Bye: Yeah, and talking about the score, it reminds me of something like Joseph Campbell's The Hero's Journey, where there's sort of this building up of tension with the climax and the resolution, and that, you know, it's kind of a standard structure that a lot of filmmakers and storytellers use, but that there's also a visual language that filmmakers can start to use to build more of a dynamic tension as things are increasing towards that climax. And so, Is that kind of what you're doing with the emotional scores, trying to also do things to kind of build the sense of story and tension to some sort of climax and resolution?
[00:22:43.162] Jacquelyn Ford Morie: That's part of it, certainly. There are a lot of other techniques that you can use. Now, we don't have the D.W. Griffith of virtual reality yet. I think we're getting there. We know some things we can do. So another technique I used was what we call coercive narrative. And coercive narrative is actually just a way to steer people where you want them to go. So, they get to the end of this culvert, and they're supposed to go look at this abandoned building complex, and they might look to the right and see just the river going down, and really they want to be over to the left. Well, gee, there happens to be an abandoned car there. So they can actually run to the abandoned car and get there before the scout on the roof sees them. So I have coerced them to go one way. There's no other way they could really go. Although if they went the other ways, we had landmines, and we had villagers who could shoot them, and things like that. But very few people went any way but what we expected them to go. So, the course of narrative, there's a lot of things you can do for that, to have people go where you want them to go, so they experience the story arc that you have intended. You can't be sure, but you can pretty much do it. The other thing is corroborative detail, which is a way to enrich the story, to make sure that there are lots of details for what you want them to be experiencing, and maybe less details in a place that isn't as important to the story arc.
[00:24:08.149] Kent Bye: So yeah, I think one of the tricky things in designing virtual reality experiences is how do you direct attention into what people are paying attention to and where they go. And so it sounds with this techniques that you have, you have some tricks of the trade that you've used. What are some things that you find really effective to be able to direct attention and to guide the experience where you want it to go?
[00:24:26.991] Jacquelyn Ford Morie: Sound is a great one. So if you're going to hear a sound, you're going to look, no matter what. You're going to look in the direction of that sound. Now, we need better spatialized sound. I've always been fortunate to be able to use 3D fully spatialized sound, not just binaural sound. But you can tell when it's over your head, or behind you, or whatever. So sound is a good one for directing your attention. Movement is another. So there are all these vectors that are used within the film industry that you can reassign to something like virtual reality. So color is another one. We have blinking lights. You can't help but look at a blinking light. So there are all of these tools and techniques that I learned over years of film and being at Disney and some of these other things in the game industry. You can use them for virtual reality too.
[00:25:17.873] Kent Bye: And you have a venture now called All These Worlds. And maybe you could describe to me what type of projects and things that you're doing within virtuality with that venture.
[00:25:27.459] Jacquelyn Ford Morie: So our primary projects with All These Worlds, which we just kind of started last year, are things using virtual worlds for, I would say, stress relief and healing and psychological problems. And I'll tell you two of the projects that we have there. One is we're in the second year of a longitudinal study to see if mindfulness classes given in the virtual world are as effective as giving them in the physical world. So we worked with experts from the San Diego Center for Mindfulness, Steve Hickman, Alan Goldstein, And we developed ways that they could teach the class in a virtual world like Second Life, and have the tools they need to know when the people are paying attention, or how to get their attention, or when they want to speak, how they can do homework, how we can record when they've done the things that they need to do. So that's still got another year to go, so we can get enough subjects in, but we're doing that study to see if that's as effective. And we're pretty sure it is. The other thing that we're doing is for NASA. So as they are focusing on these long-duration spaceflight missions where people are going to go to asteroids or Mars, these are very long trips. This is not going to the International Space Station and being able to have a real-time video conference with your family every night. You're not going to have that. There's up to a 40-minute comm delay, and there's no Internet out in space yet. So they're not going to be able to do that. So we're going to have to figure out ways to give them onboard tools that counter the psychological and social isolation they are bound to feel when they're on these trips. So the easy thing is virtual vacations. You give them a room where you say, OK, you can take this vacation in the coral reef, or you can actually just go sit on a small town front porch at sunset. And we've given them a number of other things where they can play asynchronous games with their family to keep those family ties strong. We're looking at ways to have family avatars actually still be walking around and acting like themselves in the virtual world, even if they're not attached to the real person in real time. So, there's a number of things we're going to be doing for them in the next two years that will test this in what they call a ground-based analog, so they're going to lock six people away for a year. We're going to give them this virtual world, and we're going to see if they come out healthier than people who haven't had something like that to mitigate the social and psychological problems.
[00:27:54.538] Kent Bye: Are these prisoners, or are these people who are volunteering to be isolated for a year?
[00:27:58.405] Jacquelyn Ford Morie: They are volunteering. The wait list is long. So they try to find people who are as similar to astronaut candidates or astronauts as possible. So they go through a screening process. There's analogs all over the world, from Antarctica to Greenland to this one in Hawaii that we're hoping to use. Antarctica has too many people for us to do social isolation, so we're not going to go there. But the Hawaii one and a number of others give you only room for six people, which would be the biggest crew you would send to someplace like Mars.
[00:28:29.630] Kent Bye: And so would the family members be like recording with motion capture and their actual voices and, you know, the real family, and then they just get to kind of experience it interactively, or maybe it's on rails and they don't get to interact, but they're just seeing the presence of the avatars. Is that kind of what you're doing there and creating these experiences?
[00:28:48.997] Jacquelyn Ford Morie: We're very early on in what we're going to be doing. We just know we want to do something like this. It's only one of maybe 10 different things that they'll be able to do. So some of the astronauts may want to start building, some of the family members may want to build special things for their astronaut family member who's away. So there's a number of different things that they could be doing in the virtual worlds. We envision a weather room that has crowdsourced material. So even though we only sync up the ship-based and the ground-based servers maybe once or twice a day, they'll be able to see what the weather in Houston's like. And maybe there'll be a video of the first autumn leaves falling in Boston or whatever it might be. So we see a lot of this as being crowdsourced. NASA vets it, puts it on the next upload that's going to the astronauts. So I'm sure there are lots more ideas out there. We could have more asynchronous games. We want things that elicit positive memories. We may have avatar agents who have some kind of AI that can actually be part of the interaction with the astronauts and maybe even be used as a tool to tell whether or not they're doing okay psychologically. We have a place called The Club. The Club is kind of interesting because They're not going to see real-time comedians or anything like that. But we don't want them to go sit in the club and just watch a video. So we've populated it with a whole bunch of AI avatars that you can kind of go and chat with, or you can just wave to them. And they're there, and they make like an embodied laugh track. And they can also be other dancers if we change that club to a disco at night.
[00:30:27.705] Kent Bye: Wow. And on your website, you had a section about telehealing. What is telehealing?
[00:30:34.929] Jacquelyn Ford Morie: So I think it's more telehealth and what I see happening, we have telehealth happening with video conferencing with your doctor, you know, staying in touch by email, these type of things. I think there's great potential for telehealth to expand into the virtual area. Now you already have things like Skip Rizzo and others doing PTSD graduated exposure therapy using sort of standalone VR that's in the clinician's office. They can control what you get to see when, so that they can really monitor your progress in this exposure therapy. But beyond that, there is a great potential for these things to be places where people go after they've done some kind of in-house thing. So maybe they've been in a particular part of the hospital for months, but then they want to stay in touch with their doctors, or they need to really be reminded of educational material or things, why they're doing things, why this is part of their rehabilitation routine. Rehabilitation games could all be centralized in something like a virtual world or virtual reality space. We worked with the National Intrepid Center for Excellence in Bethesda, Maryland, and this was a fairly new facility. It's funded by private donations. but active duty military personnel who have some sort of psychological or TBI traumatic brain injury problem will go there if other forms of treatment have failed for them. And they try all kinds of interesting things from art therapy to narrative therapy to they have comprehensive soldier fitness which is a way to get your mental sort of attitude in order, and then give yourself checklists and goals so that you can maintain a positive advancement during your recovery or your healing process. And they stay there for two or three weeks. And then they go back to work. And then there's no way they follow up. So we're thinking, we built the whole center. And what if they check in once a week at the center, maybe they even have office hours from the therapist who worked with them. They have a way to stay in touch and the people who cared for them have a way to kind of monitor them from a distance without being too invasive. One of the interesting things they do is the canine warrior unit. So if you go there, you can have a dog assigned to you. And these are all specially trained dogs. We had a camera in the auditorium space that looked in on the puppy farm where they would raise these dogs. But you don't get to take the dog with you. So we thought, what if we can build a really high quality, very interactive dog as an AI agent that you can still have your dog when you've gone. So just fun things like that. I think there's just so much potential. We have barely scratched the surface.
[00:33:27.342] Kent Bye: And it sounds like all these worlds are doing the implementation phase of a lot of these techniques that have presumably been tested out through studies and research. And so are you also involved in that research? Or are you taking in studies from all over and just putting it into practice with this venture?
[00:33:44.690] Jacquelyn Ford Morie: We base much of what we do for all these worlds on research from a number of areas. So we look at things like actual health research. So how does nature affect you? We have six years of research that shows what kinds of things in nature give you a positive psychological boost and what kind give you a negative one. So we implement that, so we have beautiful environments, we have lots of little chipmunk sounds and birds and things flying around, so it's very rich, it's that corroborative detail I mentioned before. We also look at research that went on in virtual reality, and usually those are isolated, so you don't get the social component, but you do get a lot of good information there. We use our own studies, so like the mindfulness thing, we're doing this full formal study for the two-year period to see if the people that go through that have the same kind of outcomes as people who have had it physically, and how that compares to people who have never had mindfulness training. So, we use everything. I'm always looking at neuroscience, I'm looking at studies from NIH, the National Institute for Health, I'm looking at studies that go in many, many areas to support what we're doing in the virtual world. Because if it hasn't been done in the virtual world, and you haven't got the study to say that it's effective there, and sometimes it takes many of these studies, and it takes a large N of number of subjects, If you don't have that, then you better be basing it on some evidence-based research or you're just maybe whistling in the dark.
[00:35:16.205] Kent Bye: With the neuroscience component, there seems to be the whole neurogaming component where there's actual interactive biofeedback. Is that something that you've looked into or is it more neuroscience in terms of the foundation of neuroplasticity or other aspects of brain science that you're using as a foundation for these studies that then you're implementing?
[00:35:35.005] Jacquelyn Ford Morie: So we're very much at the infancy of what we can measure with our brain, what we can have our brains do. So the current things that are out there, the consumer-grade level, neuro-gaming, so-called brain-computer interfaces, are very, very simple. They're really not giving us a good look into the brain. So we can use them as control mechanisms. I'm not sure I want to say walk, walk, walk, walk, walk with my brain to have my avatar walk, but I think there are other great things we can do with our brain vis-a-vis an avatar in a virtual environment, or even in that environment itself. So, for example, we can have that environment know how the person's feeling by having the environment have a camera on their face so they can see their expressions, they can see their body movement. The sensors in the virtual world could be listening to the paucity of the person's voice to know if they're upset. You would know if I'm upset with you, but they could also figure that out. You could have an AI agent come and be a calming influence. You could have a tutor come and say, you're not getting this lesson. You seem frustrated. Could we do this? So I think the sensor thing is something I've been very interested in from a long time back. And when we did that dark con experience, we had physiological monitors on the people, heart rate and skin conductance. We were using them mainly to see if we were getting the emotional response where we wanted. And what we did for the next project was to take that input and realize when the person was having issues and have something happen. So have the person's physiological state actually trigger things in the virtual world to change the outcome of what was happening. So that can be done with neural signals, too. It's just that people think neural signals just come out of your head full-blown, like Athena. And they don't. There's a lot of analysis that has to go on these to really make sense of them. The ones that we can get with these limited devices today, they're not going into your deep brain. They're not doing a lot of things that we would really like to see happen. But they're getting there. So they're probably years behind where we are in, like, head-mounted display technology. They might be about where we are in haptic technology, because haptic has probably the lowest resolution of any of the senses that we use in virtual reality. But I think we will see more of that connection to your brain, to your physiology, to whatever sensors we can have you wear or swallow or implant.
[00:38:09.444] Kent Bye: Can you talk about your involvement with USC's Institute for Creative Technologies?
[00:38:15.271] Jacquelyn Ford Morie: That goes back a long way. I was working at Walt Disney Feature Animation in 1996, and had been there a couple years, and there was a workshop put on by the National Research Council in Irvine, California, at the Beckman Center there, and it was called, Modeling and Simulation Linking Entertainment and Defense. And the military said, dang, those entertainment people are doing something right because they got all these people playing games and watching movies. So what are we doing wrong that we can't get training to be as compelling as what they're doing, as engaging? And so they put together this workshop, about 55, 60 people, half from entertainment, half from military. I was actually from entertainment at the time. And Ed Catmull was there, president of Pixar. Jack Thorpe, who did the SimNet network system, as I said earlier. It was just a real who's who of people. It was wonderful. It was led by a guy named Mike Zida, who was at the Naval Postgraduate School at the time. And so he put it together. There was a book that came out of that with position papers and a narrative of what happened through the whole thing. And out of that book, the Army Science Office decided to put together what they call a UARC, or University Affiliated Research Center. And that was the Institute for Creative Technologies. So it was given the remit to make training better. It was actually given the remit to build the holodeck. And we all believed we were going to do that. And they have actually come a long, long way. So I was there for 13 years. My task at the beginning was to take generals around to all the entertainment companies in the L.A. area so they could meet people and see how these facilities ran. I was at Rhythm & Hues for a while and I took this one general there. Rhythm & Hues, we used to bring our dogs. They had a kitchen. They had lunches. It was a wonderful kind of family environment. I want the ICT to be just like this. So it was great fun. And then they started up, they hired me. They said, what do you want to do? And I said, I want to really investigate emotion in virtual reality environments. And that's how I started that project.
[00:40:25.583] Kent Bye: Yeah, speaking about the holodeck, I know that Ivan Sutherland back in 1965 wrote a paper called The Ultimate Display and in the last paragraph he said, well actually the ultimate display would be if you're in a room and you can actually create a chair and it'd be just as good as sitting in a chair and if there was a bullet it could kill you. And so, you know, in 1987 I think was the first year that the holodeck made its appearance in Star Trek and The military were they thinking that the vision of the holodeck and you saying that we've come a long way in that way Is it you mean with like nanotechnologies or what do you foresee with it?
[00:40:59.020] Jacquelyn Ford Morie: First of all when we were given that task to go create the holodeck two interesting things three interesting things happen the person who was made the head of the ICT was dick lindheim who was at paramount and in charge of TV, and one of the big projects he had was Star Trek. And then we had another advisor who was a director on Star Trek. His name was Alex Singer, and he was very involved in the beginning at the ICT. Not only that, but the entire interior of our building was designed by Herman Zimmerman, who did the sets for Star Trek. And they were fabricated on the Paramount lot. So we had a lot of, and at our grand opening, we got the Star Trek suits out of the Paramount wardrobe, and people wore them as docents for our grand opening. When I say we've come a long way, I mean, part of the Holodeck research is getting really believable virtual humans who know what you're experiencing, what you're feeling, and I think the ICT has come a long way in that. They are probably at the top of the game for these interactive virtual humans that know how you're feeling. Display technology has gotten better, so much better. We have Dr. Paul Debevec over there who scans most actors in every movie you've ever seen in the light stage. And that's amazing work there, so getting the characters to be more photorealistic. In terms of interacting with a virtual environment, Mark Bolas at the Mixed Reality Lab is doing a lot of things that allow you to appear to be walking further than you could possibly walk in that space by using a technique called redirected walking. exploiting something called change blindness. These are all great things. You can go to their website and look at it. Actually, Palmer Luckey spent a few weeks with Mark Bolas in the lab there before he went off and started his Kickstarter campaign. And Mark Bolas built a lot of the early VR equipment like the Boom and head-mounted displays and a number of other things for NASA when they were doing a lot of work there. So, that's what I mean when I say we've come a long way. When we started that, we faked a first demo. you know, nothing was really working. And now it all works even better than we imagined. We have not tackled the nanomolecular display, which is what I call the holodeck display system. That would be where the molecules quickly reassemble themselves into something you can sit in, like a chair. hopefully not a bullet that kills, but actually Hiroshi Ishii at the MIT Tangible Media Lab, I think it is, is doing this. And he's got something that's fairly low res, but it is with actuators and stuff, making the shapes that are physical and tangible now. So, I think we've got a couple decades before that one's coming. We're much further along on a lot of the other stuff.
[00:43:37.004] Kent Bye: Yeah, what's really striking to me about that story was just the fusion of the sci-fi into the manifestation within the military. So, like, how did all these people from the science fiction kind of get into this position of directly interacting with the military?
[00:43:53.756] Jacquelyn Ford Morie: That's a very good question. It's interesting that the military is very eager to see people's ideas and then find the people that can make those reality. So if you're inspired by science fiction, as we all were by Star Trek, then finding the people to make those things a reality just seems like the natural next step. For us working at the Institute, having that kind of Star Trek connection made it very sexy. What the military was looking for when they put this thing in Hollywood, there were other places they could have put it, they put it in Hollywood for a reason. Not only was there magic that gives all of us so much in our entertainment and in sort of the richness of life that we can live, but the imagination of the people doing entertainment, whether it's games or movies or writing a script, That is all powerful stuff, and it can be put to bear on things like creating a training environment that actually embeds someone in a story so that they're living that training environment, instead of just going through a list of, we've got to do this, we've got to do that, we've got to do this. And that was one of the things the Army said early on. We don't want a list. We want you to embed them in the training so that they live it.
[00:45:16.549] Kent Bye: And finally, what do you see as the ultimate potential for virtual reality and what it can provide to society?
[00:45:24.133] Jacquelyn Ford Morie: I think as a medium, it can provide many of the same things that we have in movies, we have in games, we have in reading, literature, those types of things, music. It can be an entertainment medium, it can be a training medium. It is a blank slate. It's going to be what people make it. So what we're doing right now in this, what I call the second wave of VR, is we're teasing out what do we know about humans, human physiology, human perceptual systems that we can use to deliver something that's meaningful to the person. whether that is entertainment, whether that is how you go to school, whether that is how you relax and have stress relief or get your health care. I don't care what it is. It is one of those vehicles that is going to be very malleable and very powerful. And it's all depending on how we put it together and how well we connect it to the human who has to use it. Great.
[00:46:23.746] Kent Bye: Thank you so much for talking today.
[00:46:25.967] Jacquelyn Ford Morie: That was my pleasure.
