The IEEE VR held a new pre-conference workshop this year on Virtual Humans and Crowds for Immersive Environments. Movies like Lord of the Rings and video games like Assassin’s Creed use this research in order to create convincing group behaviors with NPCs, and architects want to be able to test their building designs to ensure that they are comfortable for different types of flows of people and evacuation scenarios.
There are a number of non-VR researchers who are studying how groups of move through through different situations and contexts, and virtual reality is providing new opportunities to test out some of their theories within immersive environments. At the IEEE VR, I had a chance to Rutger’s professor Mubbasir Kapadia who studies crowd simulation and his latest book is called “Simulating Heterogeneous Crowds with Interactive Behaviors.” We talk about his research and theories into how to describe and simulate crowd behaviors, some of the entertainment and architectural applications, how VR can be used to test and verify some of these theories, and some of the big open questions that are driving his research.
LISTEN TO THE VOICES OF VR PODCAST
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR Podcast. So at the IEEE VR Conference this year, there was a workshop that was held for the first time about virtual humans and crowds for immersive environments. So they had a number of outside experts of simulating crowds of virtual humans come together to talk about the different use cases and applications for virtual reality. Some of those applications range from doing simulations of crowds to see if architectural designs will be able to handle different evacuation scenarios, as well as to just create realistic immersive environments that have crowds in it. You can remember a scene from The Matrix when Neo is talking to Morpheus and they're walking through a city landscape and there's people that are walking around. And in order to create that realistic behavior, then there has to be some models in order to actually simulate those simple rules that dictate this complex crowd behavior. So I'll be talking to Mubasir Kapadia, who's an expert on simulating virtual crowds, on today's episode of the Voices of VR podcast. But first, a quick word from our sponsors. This is a paid, sponsored ad by the Intel Core i7 processor. VR really forced me to buy my first high-end gaming PC And so Intel asked me to come talk about my process. So my philosophy was to get the absolute best parts on everything, because I really don't want to have to worry about replacing components once the second gen headsets come out and the VR min specs will inevitably go up at some point. So I did rigorous research online, looked at all the benchmarks, online reviews. And what I found was that the best CPU was the Intel Core i7 processor. But don't take my word for it. Go do your own research. And I think what you'll find is that the i7 really is the best option that's out there. Today's episode is also brought to you by VR on the Lot. VR on the Lot is an education summit from the VR Society happening at Paramount Studios October 13th and 14th. More than 1,000 creators from Hollywood studios and over 40 VR companies will be sharing immersive storytelling best practices and industry analytics, as well as a VR expo with the latest world premiere VR demos. This is going to be the can't miss networking event of the year with exclusive access to the thought leaders of immersive entertainment. So purchase your tickets today while early bird pricing is still in effect at VROnTheLot.com. So this interview with Mubasir happened at the IEEE VR conference happening in Greenville, South Carolina on March 20th. So with that, let's go ahead and dive right in.
[00:02:48.079] Mubbasir Kapadia: Hi, so my name is Mubasher Kapadia. I am an assistant professor in the computer science department at Rutgers University. I would consider myself an outsider in the virtual reality community. This is actually my first IEEE VR conference and I'm super excited to be here. I'm actually a graphics scientist, so most of my research deals with developing computational models of being able to simulate autonomous virtual humans and how these virtual human characters act and interact in complex meaningful ways in interactive virtual worlds. And that sort of spans the entire spectrum of modeling the underlying biomechanics of how we as humans intrinsically move all the way up to sort of the cognitive ability of how we think, act, and behave.
[00:03:36.218] Kent Bye: So it sounds like in order to simulate large groups of people in crowds, you have to do the research and modeling to be able to replicate it so that when humans are in these virtual environments that it feels natural and real.
[00:03:49.150] Mubbasir Kapadia: Absolutely, right. So as you said, so in order to scale all the way up to being able to simulate high density crowds of hundreds and thousands of characters, we start with an agent based philosophy. So our understanding, and also those of many in the room, is that if we can model an individual's ability to locomote, to navigate, and to behave, we can sort of have those individual decisions percolate at the aggregate level. And there's a lot of researchers who've demonstrated that these individual agent-based microscopic models produce aggregate crowd-level phenomena.
[00:04:25.060] Kent Bye: Yeah, I guess it's, you know, from the outside when we watch a flock of birds fly around or a school of fish swimming around all together, we get this sense of like this magical, like, oh my gosh, how do they even do that? But yet as humans, we kind of do that same as we walk through crowds, but we don't really even think about it. And so I'm curious, like, how can you describe the algorithm of like, what are we actually doing when we're walking through a crowd?
[00:04:51.354] Mubbasir Kapadia: Right, that is a very complex question and if you would ask that and that question would give you a hundred different answers depending on which researcher you ask the question to, right? Because each and every one of us have our own favorite models, right? Yeah, so this question has been studied extensively all the way since the 1980s and the general consensus seems to have been that the simplest possible rules are often sufficient to govern these seemingly extremely complex phenomena. So it would be hard to articulate, but the idea being that we can sort of devise a simple set of heuristics that sort of govern how I will choose my next steering decision based on the presence and movement of those around me. They're essentially reactive policies and they often seem to get you almost 90% of the way there. And then there's some more recent work that sort of takes it one step further where you not only react but you also predict. the movement of those around you in order to make your next decision. And then finally, the extreme of that level of decision making would be where not only do we react and predict, but we also plan. So we actually sort of have different levels of planning where I'm thinking about what I'm going to be doing five steps ahead, as well as when I exit the building, as well as if I want to go all the way to the airport, right? So there's sort of these different levels of abstractions and planning that I have to do as I'm interacting and moving about in these crowded situations.
[00:06:16.924] Kent Bye: Yeah, the thing that I found really fascinating is the moment where they started to talk about we can observe nature and see what the fish or birds do, but we can't really simulate them in a way that we can control in any way, but yet with humans we can control like crowd behaviors and then put them in a situation with other virtual humans and then observe empirically what they do and then feed that back into these models and so that with virtual humans and crowds that we're able to get closer and closer to models that actually reflect what people are really doing in reality.
[00:06:49.444] Mubbasir Kapadia: Absolutely. So yes, I think in response to your comment, so the community has recently begun to sort of understand the importance of empirical data. And this was not possible before because we just didn't have the right hardware and the right sensing techniques to be able to do this. But now with motion capture and with the Kinect and so on and so forth, we are now able to actually track and capture the movement and interactions of a relatively large number of people. And once we have that data, we can use that to train and inform our models in an effort to better replicate reality. So yes, as you said, people have now begun to do this and there's a lot of promising evidence to suggest that data-driven models are the way to go. where we don't make any prior assumptions on our cognitive or decision-making capabilities and we start from scratch. So we essentially say, here's the data, let's train a model that fits that data. And there's some really promising evidence to suggest that's the way to go. And that's also true in animal swarms. So people have surprisingly found inventive ways of collecting data of how animals, fishes, or insects even, move and behave and interact. and there's an analog to that kind of research in both the swarm community as well as in the crowds community.
[00:08:02.966] Kent Bye: So maybe you could tell me the name of your book and kind of how it's broken up into different sections.
[00:08:07.113] Mubbasir Kapadia: Okay, so the book that we've been working on is Simulating Heterogeneous Crowds with Interactive Behaviors. We've been working on this book for more than a year now. It's in its copy editing phase, so we hope to have it published very soon. This book has been edited by Nuria Pelicano, who is a professor at the University of Polytechnique in Catalonia, which is in Spain. Jan Albeck, who is a professor at George Mason University, myself, Mubasher Kapadia from Rutgers University, and Norman Badler from University of Pennsylvania. And what this book is, is an anthology of chapters that are written by academics and industrial practitioners who are sort of presenting their own unique perspectives on being able to simulate and model heterogeneous crowds. So the book is divided into four volumes. The first volume sort of describes the nitty-gritties of how we can actually begin to model the underlying low-level decision-making of how individuals steer, avoid collisions, and navigate in crowded situations. The second part of the book describes different tools and techniques to be able to author complex crowd animations for visual effects applications, for example, and also techniques to be able to render and generate photorealistic crowd animations. The third part of the book sort of delves into how we can actually empirically, both qualitatively and quantitatively, evaluate both real and simulated crowd behavior and how we can begin to compare real and simulated crowds. And the final part of the book finally sort of describes a variety of applications where these tools and technologies may be useful, both in visual effects, for entertainment, for movies and games, as well as for real world applications such as security and disaster management and so on and so forth.
[00:09:59.087] Kent Bye: I'm curious, what are some of the real applications for this? You have the whole last section of your book talking about the pragmatic application. So how are people going to be using this information?
[00:10:08.496] Mubbasir Kapadia: Oh yeah, there are a host of applications. So perhaps the most obvious ones being that, OK, if we can now be able to sort of automatically generate these complex crowd movements, they're automatically useful for your next movie. Movies like Lord of the Rings and games like Assassin's Creed, they sort of extensively leverage a lot of the research that you've heard today. And as these models get more and more complex and are able to better mimic reality, then we can start asking and answering serious questions. or more real world questions, right? Where we can start using these models as predictors of how real crowds may actually behave in unforeseen situations, such as an evacuation or a panic. And we can use these models then to predictively analyze and evaluate and perhaps even inform the decision-making of perhaps designing buildings for urban planning and architectural design, or perhaps doing crowd management at large-scale crowd events, such as a concert, right? So that's where I think these models are intrinsically or inherently useful because they allow you to sort of extrapolate what a crowd might do without actually seeing that event take place in reality.
[00:11:22.719] Kent Bye: What were some of the big highlights that you heard today or big insights or things that you found interesting?
[00:11:28.581] Mubbasir Kapadia: Right, so we're still halfway through the workshop and I think this is an extremely interesting workshop. It's the first of its kind and essentially what the organizers are trying to do is they're trying to bring together researchers from virtual reality and crowd simulation. bring them in a single room and have them sort of bash heads to see what comes out, right? And what is, I think, emerging from this conversation is that it is clear with the maturity of VR techniques, we can now start using virtual reality as a feasible and viable substitute for conducting crowd experiments. which was previously limited to either passively observing real crowds or conducting extremely controlled laboratory experiments, which, as you might have heard, are extremely burdensome and often prohibitive for many cases, right? So virtual reality sort of offers an extremely promising substitute to conduct these crowd experiments in a low-cost and controlled fashion. And I think that's perhaps the strongest connection that I believe opened a whole host of new research directions.
[00:12:36.104] Kent Bye: So is that some of the things that you've done in your work is to actually do these crowd simulations and then like what does that look like?
[00:12:43.549] Mubbasir Kapadia: Yeah, absolutely. So yeah, most of my research all the way starting from my PhD at UCLA aims to ask and answer that question is that what is the right model of human behavior? And how can that model sort of extend to close-knit interactions between many of these people interacting in high-density situations? So, yeah, so as this research area has matured, where we were previously limited to purely synthetic approaches, where essentially we would devise these models as experts with our own expert intuition, we can now begin to use real data, use virtual reality to collect actual empirical evidence to better inform these strategies and models that we are using.
[00:13:26.292] Kent Bye: So what do you think are some of the biggest open questions and problems facing this field?
[00:13:30.979] Mubbasir Kapadia: There are many. So probably the greatest hurdle that is, I would say, holding us back is essentially the means to be able to quantitatively evaluate how close a synthetic model approaches reality? So that's probably the biggest question. So if we had some way of assessing the ability of a crowd simulator to emulate how a real crowd would behave, that would sort of make these approaches immediately more viable for some of these real world applications that I described previously. And as part of that, so a dual of that problem is before we can begin to match our simulators to reality, we need to collect data, right? And that up until now, or even now, is still an extremely challenging problem. Because collecting data of tens of individuals interacting may be possible, but if we scale that number up to a hundred or a thousand, that's still probably an impossible task in most cases. So that is perhaps the greatest challenge that needs to be answered and addressed in some feasible way. And that can, I believe, open up a whole host of new applications. And as I said, virtual reality offers one viable way to do that.
[00:14:43.194] Kent Bye: Is there like a taxonomy of different contexts that kind of points to different distinct categories of how crowds behave? For example, walking into a state fair versus like evacuating a building or Black Friday rushing into Walmart or, you know, I'm just curious about like what different types of contexts or behaviors that you see kind of different crowd behaviors.
[00:15:06.684] Mubbasir Kapadia: Yeah, that's a good question. So the way the crowds community has approached it is that so there exist these taxonomies and they're mostly geared at more fine-grained decision-making. So we have a family of different approaches which we sort of cluster as microscopic approaches. which essentially sort of model the individual interactions between one or two or a few individuals, and then we sort of classify them based on the type of interaction. So we may have oncoming interactions, crossing interactions, or a combination of the above. And again, depending on the variation of the orientation, the variations of the speed at which these individuals interact with each other, we can sort of create a broader taxonomy. A portion of the taxonomy then sort of deals with macroscopic approaches, where we essentially treat crowds as a continuum. You can sort of model crowds as a fluid and sort of observe the dynamics of how the continuum behaves. And depending on how turbulent or how guided or uniform the flow of the crowd may be, we can sort of have different categories of these macroscopic behaviors. Recently people have been trying to connect these two families of approaches and there's some work in hybrid solutions that combine both microscopic and macroscopic phenomena, but that I think still bears for the investigation.
[00:16:22.005] Kent Bye: Great. And finally, what do you see as kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:16:29.658] Mubbasir Kapadia: I think the ultimate potential for virtual reality is for me to go to Disney World while sitting at home, right? And for me to actively choose how crowded or how empty I would like Disney World to be, right? I think virtual reality offers limitless possibilities and it offers the potential for limitless amounts of control to every individual. I believe, I think that's probably the greatest strength and perhaps even the greatest weakness of virtual reality. where we have all of these things that we want to do, we just don't know where to begin, right?
[00:17:01.791] Kent Bye: Great, yeah, thank you so much. Thanks a lot. So that was Mabaser Kapadia. He's a professor at Rutgers University and has published a book called Simulating Heterogeneous Crowds with Interactive Behaviors. So, I have a number of different takeaways from this interview, is that first of all, it's interesting that in order to get some of this complex emergent behavior of crowds, you just have to have a series of simple rules. And from those simple rules, then you can get some of this emergent behavior. So that's the agent-based approach where each sort of individual actor has kind of got a specific intention and trajectory as to where they're going and what they're doing. There's also the macro approach, which is to just kind of model an entire crowd movement as if it was like a fluid. So you're looking at different fluid dynamics equations in order to describe how these crowds are moving around. I think in the context of virtual reality, though, I think it's interesting to think about how to start to prove out some of these different models of how crowds are acting in different situations and contexts. So with different immersive technology tools like virtual reality, you can start to put people into virtual simulations where they're walking through crowds with other virtual humans. And if you have kind of like a big beyond room scale space, they're able to kind of just walk through naturally and then see how they're interacting to these virtual humans. You can also have these different crowds of people in a gymnasium, and you start to just put different OptiTrack trackers on their heads. And as they are walking around, then you could start to also prove out a lot of the kind of emergent behavior that happens in different scenarios. I think that the thing that I find interesting is that a lot of these crowd movements may be context dependent. So for example, if you're just walking down the sidewalk, there's going to be a certain amount of behavior. But if you're at a conference, then there's going to be a lot more people stopping and talking to each other because there's more likelihood that people are going to know each other and stop in the middle of a walkway and start talking to each other. Or if you look at, like the example I said, like the Black Friday scenario where people are waiting to rush into a store to be able to get something to buy, that's going to have a whole other different type of behavior of people having very high agency in terms of the specific motivation and intention of what they're trying to do. So it was interesting to hear that some of this crowd movement and behavior and simulation has been used in some movies like Lord of the Rings or in some video games like Assassin's Creed. And I think in the future, it sounds like what they want to get to is to be able to have perhaps an artificially intelligent NPC or even potentially a robot in a real life crowd environment. And it's going to have to be able to do a number of things. First of all, it's going to be able to have to react to the crowd that's there around them. then it's going to have to predict what the behavior is going to be so that then they can start to plan the different course of action as they're moving through a specific space. And I think there's probably a ways to go in order to make the interactions within crowd environments something that's really believable and plausible. I think right now it's probably going to be pretty uncanny. I say that just because I haven't seen a lot of people actually do it. There's not a lot of experiences out there that I've experienced where it really just is giving me that sense of social presence and having me immersed within a crowd simulation with people kind of convincingly moving around. But after having this discussion, I think that this is something where these two communities are going to have a lot to contribute to each other. So for example, The people who are already doing this research are going to be able to use the virtual reality technologies to be able to further test and prove out their theories. And I expect eventually within VR experiences to see a lot more crowds and simulations of lots of people moving around within a space to be able to simulate this type of social presence. So, that's all that I have for today. If you'd like to support the podcast, then tell a friend, spread the word, and become a donor at patreon.com slash voicesofvr.
