#359: Fred Brooks on Sutherland’s 1965 Ultimate Display Speech

fred-brooksFred Brooks was in the process of looking for his next research agenda when he heard Ivan Sutherland give a speech in 1965 at a computer conference that laid out his vision for future of virtual reality. Sutherland said that you shouldn’t think of a computer screen as a way to display information, but rather as a window into a virtual world that could eventually look real, sound real, move real, interact real, and feel real.

Brooks won the Turing Award in 1999 for some of his pioneering work in the computer architecture, operating systems, and software engineering when he was at IBM. He also wrote the famous book, The Mythical Man-Month based upon some of his experiences. In 1964, Brooks left IBM to start the computer science department at the University of North Carolina at Chapel Hill.

Sutherland’s vision was articulated fully in his 1965 “The Ultimate Display” paper that predicts the trajectory of technology towards virtual reality, head-mounted displays, eye tracking, haptics, speech recognition, and even the holodeck.

I had a chance to catch up with Brooks at the IEEE VR conference in South Carolina in March where he shared some of the highlights from the very beginning of VR, and then on through the big milestones that he’s seen develop over the last fifty years.


Fred Brooks recalled seeing Sutherland speak at the Fall Joint Computer Conference in 1965, but I could not find any mention of Sutherland’s speech from the publication of the conference proceedings listed online. I did find that Sutherland’s “The Ultimate Display” paper was published in 1965 in the Proceedings of the IFIP Congress, pp. 506-508.

Brooks had a paper titled “The Future of Computer Architecture” that was also published in that same Proceedings of the IFIP Congress in 1965, and so it’s possible that Sutherland’s speech happened at the International Federation for Information Processing conference rather than the Fall Joint Computer Conference. Whenever this speech happened may well be the first time that Sutherland shared his thoughts about what would become virtual reality to a public gathering of his academic peers.

By 1968, Sutherland had built a prototype of the Sword of Damocles, and published a paper titled “A Head-Mounted, Three-Dimensional Display” within the AFIPS Proceedings of the Fall Joint Computer Conference, Part I, pp. 757-764. Last year at IEEE VR, I interviewed Henry Fuchs who was one of Sutherland’s students and fellow VR professor at UNC Chapel Hill.

The virtual reality research that has happened at the University of North Carolina at Chapel Hill has laid the foundations for the current VR revolution, and Brooks wants the current VR enthusiasts to remember that Sutherland had laid it all out from the very beginning in 1965 with that speech of his vision of making computer graphics that look real, sound real, move real, interact real, and feel real.

Subscribe on iTunes

Donate to the Voices of VR Podcast Patreon

Music: Fatality & Summer Trip

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to The Voices of VR Podcast. On today's episode, I talked to Fred Brooks, who is the founder of the Computer Science Department at the University of North Carolina at Chapel Hill. He found that in 1964, and he had been coming from IBM, where he did a lot of really pioneering work in computer architecture, software engineering, and operating systems. for which he eventually won the Turing Award in 1999. But Fred tells a story of hearing Ivan Sutherland speak at a computer conference in 1965, where he was really articulating his vision of virtual reality for the first time. And Fred was there at that speech to hear it, and it really inspired him on his research agenda for the next 50 years. So we'll be able to hear Fred tell that story, as well as a lot of the different innovations and milestones in the evolution of VR since that time. But first, a quick word from our sponsor. Today's episode is brought to you by the Virtual World Society. The Virtual World Society was started by Tom Furness, and their goal is to become the Peace Corps of VR. They want to transform living rooms into classrooms, and so they're in the process of trying to recruit potential subscribers, as well as content creators who are interested in creating educational experiences that help solve the world's problems and help make the world a better place. So if you're interested, go to virtualworldsociety.org to sign up and get more information. So this interview took place at the IEEE VR, which is the academic conference for virtual reality. And it took place in South Carolina in March. And I had a lot of people come up to me at the conference and say, hey, you really should try to talk to Fred because he's got a big part of VR history that would be great to capture. And he doesn't usually come to these events anymore because at this point he's 85 years old. And so when I went up to Fred to ask him for an interview, he said, wait a minute, I got to check my schedule. And he pulled out these index cards, which were hand typed on them, his schedule for the entire day. He had pretty much a booked day of going from session to session, but I was able to catch him right at the end of a session and be able to sit down with him and capture some of his stories. So with that, let's go ahead and dive right in.

[00:02:28.110] Fred Brooks: I'm Fred Brooks, and I'm professor of computer science emeritus at the University of North Carolina, Chapel Hill. And I left IBM to go to Chapel Hill and start the department in 1964. And my big issue was in what area did I want to do research, because My fields had been computer architecture and software engineering, and neither of those were appropriate for the Chapel Hill environment. I happened to go to the Fall Joint Computer Conference in 1965, in which Ivan Sutherland gave this stunning, absolutely stunning speech that envisioned all of VR from the very beginning. So what Ivan said was, don't think of that thing as a screen. Think of that thing as a window. And through the window, one looks into a virtual world. The challenge to computer graphics for the future is make the picture in the window look real, sound real, move real, interact real, and feel real. All right? And I knew then that that was the interactive computer graphics was what I wanted to do towards that goal. So another participant here, Turner Whitted, made the biggest contributions towards making the picture look real, but it took forever computationally to do ray tracing. Our team and many others worked on making the image in the world, the picture of the world, move real. So we worked with interactive molecular graphics because we were doing vector graphics and you have to move those in real time to see what the molecule looks like. And so we did all kinds of things make things move real. And so in the 1970s Henry Fuchs who had been one of Ivan's students at Utah joined our faculty and he had shared this same vision and so ever since then we've been pursuing Ivan's vision. And so that's it.

[00:04:37.425] Kent Bye: So did you ever get a chance to try out the sort of Damocles that

[00:04:40.847] Fred Brooks: No, I never saw his system. This was his speech before he built his system. This was his statement of the vision. No, I've never tried the Sword of Damocles or even seen his VW.

[00:04:55.118] Kent Bye: So yeah, it sounds like he wrote this paper back in 1965, The Ultimate Display, where he kind of projects out this vision of what you could today call the holodeck.

[00:05:03.685] Fred Brooks: Exactly. Well, he said it all in the first speech.

[00:05:09.157] Kent Bye: And so how did you then start to get into the actual virtual reality then from there at UNC?

[00:05:14.479] Fred Brooks: Well, the first thing we moved to was towards interactive, helping protein chemists fit structures to electron density by making a virtual electron density in three dimensions and giving them virtual stick figure models that they could move all the pieces of interactively. And the problem is doing all that in real time. with 3D perception. We built some early head mounts, but that was after we had really worked hard on getting the real-time motion and the human interaction to the point where the user was not even conscious of the world outside the molecule. All right, they were so engrossed. They would work for eight-hour shifts at a stretch, and they were so engrossed in what they were doing that it was as if the molecule were real. Now, they couldn't see it, all right, and what does a molecule look like anyway? But they were engaged in solving this chemistry problem of what is the real structure of this thing based on the evidence I have from the crystallographic data that they were completely involved in it. So, in a sense, we were already making a virtual world of one that you couldn't see in real life, okay, but in which the models became real to the people who were using them. And we did that for 30 years. And meanwhile, we built our own head-mounted displays. Gary Bishops built an early head-tracker system, low-latency head-tracking system. And we went along step by step.

[00:06:49.065] Kent Bye: It sounds like Ivan Southern was at the University of Utah, and then Henry Fuchs went out.

[00:06:54.807] Fred Brooks: I don't think Ivan was at Utah yet when he gave that speech, but he may have been.

[00:07:02.008] Kent Bye: When I talked to Henry Fuchs last year, he said that he went to Utah in order to study. Yeah. And then with Dave Evans. Yeah. And then they went off and started Ivins and Sutherland. What were they doing? Were they doing like flight simulations or what was it they were doing?

[00:07:15.175] Fred Brooks: They were doing military simulation systems. Yeah. And display systems. They built one of the highest performance display systems early on. A colored graphic vector system that was incredible. So they were building essentially military simulation systems for a living. And Dave was CEO of that while Ivan continued to be essentially a teacher at Utah. And on the board, of course, for Evans & Sutherland.

[00:07:44.858] Kent Bye: So was UNC one of the first universities to actually build a virtual reality head-mounted display then?

[00:07:50.422] Fred Brooks: Oh yes, I think so. Yeah, we've got several hanging in our museum. Little TV cameras with half-silvered mirrors and the whole bit, yeah.

[00:08:00.468] Kent Bye: So what's been the thrust of your research since then?

[00:08:04.189] Fred Brooks: Let me drop back only about 15 years, okay. We decided early on that there was going to be plenty of money for VR for entertainment if it worked and so we were going to spend our effort on application driven VR but for training applications especially and work on the technology of VR So early on, we said, how do we measure people's presence in a virtual environment? We picked up the pit idea from Mel Slater at UCL London, and we built a pit system in which the participant begins in an ordinary room, walks around, drops beanbags on the targets on the floor, and then they're told to open the door to the next room and go in and drop beanbags on the floor there. Well, when they go in, they're 20 feet above the floor on a diving board, and the target is on the floor. below and dropping that beanbag is scary. And so Jason Jarrell worked out a, we tested several different ways of measuring response to stress. I mean, this is the low-hanging fruit, is create a really high stress, high differential stress situation, and then measure some differential physiological property. Well, it turned out that skin temperature rises too slowly for us, and breathing is too slow, and skin temperature, you have diode contact problems. So we went on heart rate, differential heart rate between the low stress and high stress situation. And that gave us a measure that was objective. Objective from the participant couldn't affect it. The experimenter couldn't affect it. And we showed its validity against presence questionnaires. But the other advantages it had is it was contemporaneous. You get signal as it happens. rather than recollection of presence by questionnaire. We get a measure of what's happening now rather than a recollection of how I felt now. It's worthwhile supplementing that with a recollection of how I felt to make sure it correlates. But once we had a quantitative measure that we could do statistically significant experiments on, then we varied all kinds of things. One of the most interesting, we started studying haptics for VR long early, and our 1990 paper is, I think, still a milestone of summarizing our research in making the world feel real. And one of the experiments we did that's very interesting is originally our pit environment had a virtual, visual diving board you were standing on. And we said, oh, but look, you get out to the end of that and your toes are still flat if you stick them off the edge. So we put in a three-quarter-inch plywood reel board, and that didn't work either because people's toes touch the floor when they stuck them off. And so we raised it to two and a quarter inch, three layers of plywood. And now, when you get to the end and you lean over to find the target, which you have to lean over to find, your toes are hanging off into space. And that's worth two heartbeats a minute, a difference from eight to ten, as a matter of fact. And we showed it was statistically significant. That purely the haptic cue supplementing the visual cues made this much measurable difference. So, there have not been a lot of quantitative experiments on haptics. Uh-huh.

[00:11:45.763] Kent Bye: For you, do you have a memory or a story of actually going through this pit demo?

[00:11:51.365] Fred Brooks: I've been through it, I reckon, 30 or 40 times, and it's, it doesn't, you don't get immune. Now, some of our measurements have shown that people's response to it declines somewhat over multiple repeats, but it doesn't go away.

[00:12:08.258] Kent Bye: What was your experience like the first time you saw it then?

[00:12:11.612] Fred Brooks: Same as anybody else's, I reckon.

[00:12:15.187] Kent Bye: So I think that, you know, one of the things that Henry Fuchs said was that VR, since like the late 60s, 70s, has been moving first towards flight simulations.

[00:12:24.030] Fred Brooks: And I'm curious... They were already there. That's an independent motion. And it was already underway. It started with the Link trainers in World War II. So it's another 25 years old. So essentially, we've been trying to enlarge the scope And there's a whole literature of flight simulators that the two communities hardly communicate.

[00:12:49.388] Kent Bye: Well, he was saying that they had like actual cameras and models that then, you know, eventually they realized that it would be more efficient to move to more computer based simulations rather than actual physical simulations.

[00:12:59.633] Fred Brooks: Yeah. But I'm not a scholar in that literature. I can't. I once had a two-hour flight in one of British Air's $13 million 747 simulators. I had a chance to drive it for two hours. And that is an incredibly bonding experience. You are absolutely convinced you're flying that airplane.

[00:13:24.095] Kent Bye: One of the things I'm really curious about is the applications and the industries that, you know, from the time periods from the late 60s to the 90s, there seems to be a lot of applications for where VR was being applied, but you seem to be perhaps on a direct interface to some of the applications for VR during that time period. I'm just curious what type of applications you were seeing.

[00:13:44.943] Fred Brooks: Well, we work principally with military training, and one of the problems we have today is the whole VR technology is not well set up to do team training. It's well set up to do individual training. And we once did a proposal in which we set forth a complete plan for addressing the team planning problem, but it didn't get funded. So I don't know of anybody who's really addressed how to train, for example, emergency response teams. And you have to train them in an augmented environment because they have to work with a real patient dummy and they have to work with real defibrillators and instruments and whatnot At the same time, they've got a virtual crowd around them shouting and hollering and carrying on and ambulances coming and all that, in which virtual is the only sensible way to do it. So... For you, are there any milestones over the last time that you've been working in VR that you think... Well, as I said, I went through some of the technological milestones that have made the cost drop, like a rock on component after component. And then indeed one of the big gains is software libraries. One of the gains is camera acquisition of models. That works great from virtual models of real world objects. That doesn't work very well for microscopic ones or nanoscale ones or galaxy sized ones. And model building is still a very costly exercise, yeah.

[00:15:13.713] Kent Bye: And finally, what do you see the ultimate potential of virtual reality might be and what it might be able to enable?

[00:15:21.175] Fred Brooks: Above my pay grade. Every time I've tried to guess the future, I've been wrong. So I think we're in a stage of hype right now. I think the same thing's going to happen that happened all over computer graphics. And that is, bit by bit, the commercial market for some piece of the technology is going to make the cost go down to where it becomes a consumer item. That first happened with displays, with television. Then that happened with frame buffers, that hasn't happened in trackers yet, but it may happen in trackers. We may be getting there with selling them for entertainment, pays the bills for developing the technology for more serious purposes. And we're already seeing that solve the processor problem. The graphics processing units Oh, they were fiercely expensive. I think the first one we bought was $300,000. And Mary Whitten and Nick Englund built it at NC State in their startup company. And, you know, you buy a machine for $1,000 a day called a game card that just solves the processor problem. Yeah. So I think we're going to see that same thing happen to tracking. And tracking is the missing link at this point. It's not economical yet for consumers.

[00:16:41.222] Kent Bye: Cool. Is there anything else that's left unsaid that you'd like to say?

[00:16:44.124] Fred Brooks: No, thank you. Except I think this generation of VR people don't even realize that I haven't said it all to start with. Yeah.

[00:16:55.411] Kent Bye: OK. Well, great. Thank you so much. You're welcome. So that was Fred Brooks, who founded the Computer Science Department at the University of North Carolina, Chapel Hill, 1964. Also the author of The Mythical Man-Month, a pretty famous book, as well as the Turing Award winner in 1999. So a couple takeaways about this is that first of all, it's pretty remarkable that Fred was there at the speech in 1965 where Ivan Sutherland really articulated his vision of trying to get virtual environments to be able to look, sound, move, interact, and feel real. This interview really inspired me to go back and try to look at all the different papers and history leading up to that moment and you know Ivan Sutherland had produced the sketchpad in 1963 and so he was really innovating on a lot of different computer graphics and so Ivan had also seen the bell helicopter helmets that were able to be tracked with your head to be able to do head tracking essentially and so if you read the ultimate display he's basically predicting all sorts of different technologies from like eye tracking, to head-mounted displays, to being able to basically create a display that was connected to a digital computer so that we can gain familiarity with concepts that are not realizable in the physical world. And so he was really wanting to create this looking glass into a mathematical wonderland is one of the phrases that he says in the article. And you can kind of hear that a little bit in terms of some of the things that Fred Brooks was working on in terms of like molecular visualizations at the time. working with these different protein chemists who were trying to fit the structures based upon electron density. You can just imagine being a chemist in the 1970s working with some of the very earliest VR HMDs doing these 3D visualizations. I mean, it was probably pretty mind-blowing for them back then. And so if you go back and just read through the ultimate display, it's really quite a fascinating bit of technological futurism, of predicting where this is all going. And I think the last paragraph of the ultimate display really starts to give Ivan's vision of the holodeck, where he says, the ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining. And a bullet displayed in such a room would be fatal. With appropriate programming, such a display could literally be the wonderland into which Alice walked. And so he's essentially like going way beyond just VR technology. But the step beyond that would be the holodeck, essentially, where you're able to just create matter. And I think that, you know, this is something that feels so sci-fi, but, you know, this was back in 1965 and who knows where nanobot technologies and all this is going in another 50 or 60 years, perhaps we'll be able to fully realize Ivan's concept of the ultimate display, or perhaps it'll be something that'll be more from a neural standpoint where we're actually hijacking our nervous system to be able to give the sensation as if these things were real and not actually be real. And so just a quick shout out to my Patreon. If you enjoy these different podcasts and want to help contribute to continue them, then please consider becoming a contributor to my Patreon at patreon.com slash Voices of VR.

More from this show