Nonny de la Peña was one of the 16 Legacy Peabody Award recipients honoring historical work as a part of their new Digital & Interactive Storytelling Award. Specifically, de la Peña is receiving a Field Building honor for her body of work that has helped to pioneer immersive journalism and new forms of immersive storytelling. I had a chance to talk with de la Peña to reflect upon her body of work, and her journey into virtual worlds and virtual reality that started back in 2007.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Here’s a list of some of the Immersive Journalism experiences that de la Peña has created with Emblematic Group since 2007 as listed on her CV
- Trashed, a virtual reality documentary game about great Rock ‘n’ Roll trashings
- GCDS – Milan Fashion Week 2020, a combined virtual production rendered video and WebXR backstage experience
- New Realities, directed and produced three of a ten part 360 video series about young female activists from around the globe
- Untitled Lyme, a virtual reality experience about Lyme disease, that follows a nurse’s battle with the virus and uses mini-games to teach the science
- Stanley Hayami, Interactive Quill animation and Unity experience on Japanese American incarceration for the Japanese American National Museum
- ATT 5G Origins, A series of three interviews using videogrammetry and photogrammetry made for ATT Shape Conference featuring Trombone Shorty, former Olympic fencer Doris Wilette, and the first volumetric kiss professionally recorded
- Greenland Melting, a PBS Frontline and Nova collaboration, premiered Venice Film Festival
- After Solitary, a partnership with Frontline on solitary confinement
- Passage: Life of A Road, Adapts the work of artist Lin Yilin into a game in which the player must walk a wall full of bricks across a busy virtual Hong Kong street in an experience that mirrors Lin Yilin’s risky original performance, a statement on uncontrolled development
- Across the Line, a virtual reality experience that uses real audio to put the audience on scene to experience the vitriol shouted at young women entering health clinics where abortion is provided
- WSJ Fintech VR/AR app, an Android, iPhone and Google Daydream goggles tool providing an interactive Dow Jones stock market data visualization that was updated every two minutes
- Out of Exile: a collaboration with the True Colors Fund on LGBTQ homelessness
- Cartier: A Classic Landmark for the Modern Age, an interactive tour and 360 time-lapse celebrating Cartier’s new 5th Avenue store
- Kiya, a piece on domestic violence against women shown first at Sundance Film Festival and then the NYTimes
- We Who Remain, a piece on the war in Sudan purchases by Arte, the NYTimes and AJ+
- World Economic Forum: Project Syria, an exhibit on the crisis commissioned by the World Economic Forum for the 2014 gathering and remade for Sundance 2015
- Use of Force, which details a homicide committed by Border Patrol and winner of the Impact award at the largest independent game’s festival Indiecade for “changing the landscape of games”
- Hunger in Los Angeles, first ever Sundance Film Festival VR experience, which puts the viewer at an actual event when a diabetic waiting at a long food line collapses into a coma.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. The Peabody Awards are actually giving a whole series of 16 Legacy Awards for a brand new category called the Digital and Interactive Storytelling. It includes a number of different types of categories, and one of them is XR. There's a number of different creators that have explicitly created either XR individual experiences or being featured from their body of work. I'm going to have a series of different interviews focusing on more of the virtual and augmented reality creators, Nani de la Peña, Notes on Blindness, and then the forensic architecture. I hope to speak later today to some of the jurors to be able to get a little bit more context as to this new category and some of the other winners that didn't have time to be able to interview all 16 winners of this new category. Today's episode, I'm going to start with Nani de la Peña, who's been a pioneer within the field of immersive storytelling and virtual reality storytelling. and immersive journalism. She's the founder of Unmnemonic Group, and she's actually being awarded a Legacy Peabody Award for the digital and interactive storytelling, but for this Field Builder Award, so really honoring her body of work. So I figured it would be worth to take a moment to be able to just talk about her journey into virtual reality, but also to talk about some of the different pieces that I've personally had a chance to see over the number of years at different film festivals, from Tribeca to the Sundance Film Festival and Yeah, she's just been producing like over 20 different pieces with emblematic groups since 2007. So just really impressive body of work. And a lot of them aren't actually available for folks to see. So just to be able to talk about some of these different experiences, and yeah, how they really contributed to the overall field of immersive storytelling. So that's what we're covering on today's episode of the voices of VR podcast. So this interview with Nani happened on Tuesday, March 22 2022. So with that, let's go ahead and dive right in.
[00:02:03.255] Nonny de la Pena: I'm Nani Della Pena, and I have been working in virtual reality since 2007, believe it or not, if you count Second Life as VR. I think a lot of us did at the time. And interestingly, maybe again. But I was founder of a company called Emblematic Group. where we did a lot of virtual augmented reality and extended reality projects over the years. And more recently joined Arizona State University to found a new center on narrative and emerging media with the goal of changing the demographics of who gets to create and share their stories using these new technologies.
[00:02:42.652] Kent Bye: Maybe you can give a bit more context as to your background and your journey into VR.
[00:02:47.252] Nonny de la Pena: Sure. So not knowing how far to go back, but I'm going to go back far enough that I grew up on the edge of Venice, California. And we used to talk about my senior year at Venice High School as being a great year because we only had one shootout on campus. But I got into Harvard when I was just 17 from Venice High, not a particularly common trajectory from that high school. And I'd never been to the East Coast in my life when I showed up at school. And they had this requirement for basic programming. And I actually had a real knack for it. And I was teaching people how to pass their exam. And it was something that I was quite comfortable with. But when people started talking about the advanced computer programming courses, I got very nervous because I was seeing how hard it was, et cetera, et cetera. And you can imagine I wasn't particularly academically prepared for the rigor of Harvard coming out of Venice High School. And so I didn't continue in the programming. And I regretted it forever or for a long time until, you know, I was always drawn to those kind of technologies. I had one of the earliest CompuServe email accounts, which will date me, Nani at CompuServe, and I won their award for the best use of their technology one year because I use CompuServe databases to track down people for a documentary I was working on about Chappaquiddick, the story of Ted Kennedy driving off a bridge near Kipcod with a young woman who was drowned in the incident. And it had happened the day that actually the moon landing happened. So this was a couple of decades later. And it was my understanding that these technologies were gonna be useful for the kind of journalism practice that also I became interested in. And during that period of the 90s, I taught myself HTML. I read Howard Rheingold's book about virtual reality. And I was like, oh, this is what I really wanna do. But I just didn't know how to find the point of entry. During this period of time, I had become a practicing journalist. After college, I ended up being a correspondent for Newsweek magazine, and that didn't scratch the itch for my visual side. And then I went off to make documentary films, like the one I just discussed on Chappaquiddick, and somehow ended up writing on TV shows in Hollywood for a little while, working on some Hollywood projects, back to doc film, all the while, like I said, trying to figure out what my space was in utilizing new technologies for the kind of practices that engaged and interested me. And when I thought about virtual reality from the beginning, I was really thinking about it as a way to connect people to stories more deeply. So in 2005, I had already done a documentary that included a big segment on Guantanamo Bay prison. And I read about a grant application that was due that MacArthur Foundation was funding to help translate serious documentary content into a digital form. And along with digital artist Peggy Weil, I applied to turn our Guantanamo Bay section into a virtual Guantanamo Bay using Second Life. And that project had kind of a surprising, created a surprising reaction for us. We started seeing all these tours of universities coming through and teaching about Gitmo and invitations to show it at places and bring it as a way to raise attention to the prison and including Seton Hall Law School. we did a joint event in which they were looking at the evidence of the detainees and the loss of habeas corpus. And we were invited to Second Life and we were broadcast on the screens at Satan Hall Law School. And so we're basically, we're audience through Second Life in this live, real life conference in this hybrid form. And people who came to our Second Life conference could ask questions to the speakers as much as anybody else in the real life audience. So we were doing these kind of interesting things. And it was after that project was finished that I started thinking about it. And I really can remember the aha moment sitting in my backyard here. I was actually going to go meet Peggy, my co-creator on that project for lunch. And I remember thinking, oh man, this could be used for all kinds of journalism. This is like immersive journalism. And nobody had taken the URL yet and I grabbed it. And immersive journalism got its start.
[00:07:09.040] Kent Bye: So yeah, you're working with Second Life with Gon Gitmo. And then the first that I heard, at least from the lore of Sundance and New Frontier was the Hunger in LA project that you showed in January of 2012, which is well before the Oculus Kickstarter, but you were able to take an early prototype of the Oculus to take it to Park City, Utah and show Hunger in LA. That was back in 2012. So maybe you could just talk a little bit about how you went from Second Life into actually starting to work with immersive VR technologies at the USC ICT.
[00:07:41.690] Nonny de la Pena: So because of the Gitmo project, we were invited to speak at the University of Nevada, Las Vegas, and there was a really interesting researcher there named Jerome Friedman, an Israeli researcher now mostly in brain-computer interface, but he was a former grad student and still works closely with Mel Slater and Maria Sanchez-Vives, who are a couple based in Barcelona. Mel is one of the And Mavi, I mean, their work on research around virtual reality is just their legacies, you know, and such smart and wonderful people. So Daron at that conference introduces to Mel and I had all this Freedom of Information Act material based on my film. on this idea of like detainees being put in stress positions and was it torture or was it not torture and this term enhanced interrogation technique which I really rejected and I said to Mel what I'd like to do is put people in the body of a detainee in a stress position and they can find out for themselves so that they think it's torture or not. And Mel was fantastic. And they invited us over to Barcelona to his lab and we built this piece and the results were kind of mind blowing. People were just sitting up in a chair with their hands behind their back, but they saw this virtual mirror and people were wearing a breathing strap. And when the person in that virtual mirror who was hunched over in a stressed position, you know, as their breaths were matched, when they turned their head, they were matched to other parts of the participant and this virtual character in a stress position were matched, you know, in an orange jumpsuit. And we asked people afterwards, like, what was your body like? And even though they were sitting up in a chair, they reported that they were hunched over in a stress position. And that was pretty mind-blowing that this could have that kind of power on the mind and the body, essentially, the sensation. And that led to the publication from the MIT Journal, Presence on Immersive Journalism and Using Virtual Reality for the First-Person Experience of the News, which when I last checked was still the second most downloaded paper in the journal's history. And during the time that was being published and being worked on, I really wanted to do something about the fact that people were going hungry, they were invisible. How do we make the invisible visible? And that was a young researcher who was like my intern, Michaela Kopsamark. We started recording audio at food banks. And my original idea was, you know, these food banks would run out of food all the time and there are a lot of families. And what does a parent do when their child is being told there's no food left? But in that moment, extraordinarily, what we were finding was this dead silence. It's like head drop, this painful inability to speak. And we had no 360 video cameras, so we knew we had to produce this all digitally. And there was just no audio there to produce, right? It was just silence. Then Michaela came back to my office one day, just bawling her eyes out because she'd just been on scene at a real event where a man with diabetes waiting in this long line for food, didn't get food in time and he dropped into a diabetic coma. And the woman running this food line that was so long, she was just really overwhelmed. Even before this incident happened, she's yelling, there's too many people, there's too many people. When somebody tries to steal food in the middle of the chaos, when the man goes into the coma and the EMTs take forever to come, and they're incredibly rude to all the people standing in the line. I mean, just the audio was this unbelievable, rich trove of a reflection of what people were suffering trying to get food. And using that, I once again turned to try to learn how to be a better programmer. to learn unity, and I begged and borrowed favors, and I got avatars donated, and I ended up spending $700 of my own money that I didn't have to make the thing, and there was this peace hunger in LA. And I just didn't know how people were going to react. Sorry, I'll go back to that once again. I just want to go back to the headset. So at the time they're making this, there was this kid kicking around the lab who was like the lab intern. I know he doesn't like being called an intern. He wants to be called the lab tech, but he was just a baby boy. And I shouldn't say he was a baby boy. He was an adult, but he was like a sophomore in college. So still quite young, you know, he was in college and he didn't even know what he wanted to do. He wanted to become a journalist. I was trying to help him get into the journalism school where I was now working as a research fellow at USC. He was an intern, but he was a very talented intern. He was making headsets left, right, and center, and was a very, very capable, talented, and smart young guy. When we got into Sundance with this Hunger in LA piece, The head of that lab, Mark Bolas, was like, you're not taking our $50,000 headset anywhere. And luckily Palmer had been working on these other prototypes. The lenses we were using came out of a bankruptcy sale from a guy who was making lenses that NASA was buying. So there weren't lenses left to even make these headsets. So Palmer made a headset that, a couple of them, that was kid's name, but Palmer Lucky. made some headsets that we took with us and the rest was just extraordinary. We didn't know how people would react, but very quickly, I guess word of mouth, we had three to five hour lines. We were staying open 16 hours a day. It was nuts. And people were coming out of the piece unbelievably moved. And it was clear then that this was a medium that could grow.
[00:13:08.313] Kent Bye: Yeah, I know that I first met you at the Silicon Valley virtual reality conference in 2014 and started to get some of the history of some of those stories. And then I think actually a few months after that, I went to an education conference where I met Michael. He had demos of both Hunger in LA and Use of Force, which I had a chance to see in 2014. And so it seems like Hunger in LA was the first VR piece that you did, which premiered at Sundance 2012. And then you did Use of Force, which was to try to take a scene that was unfolding and recreate the experience of trying to record some border patrol situations that were happening on your phone. So maybe you can give a bit more context of the Use of Force and how that project came about.
[00:13:50.187] Nonny de la Pena: So, yeah, so use of force was, again, an attempt to take a story that had little visibility and put it more front and center. And what happened was on a night there was a man who would be now considered a DACA recipient, brought to the U.S. as a young boy, but had no papers, you know, worked whatever job he could, had five kids, including new twins. And on Mother's Day, he stole a bottle of tequila and a steak for his wife, got caught. He got deported. And he just tried to get back in. And he certainly felt he lived in America long enough that he felt he had some of the rights as an American. But that's not the way he was treated by Border Patrol. He was after he was deported. He tried to get back in the country. He got caught. He got roughed up when he complained to a supervisor. The supervisor let the guys who roughed him up take him to a dark pen right on the edge of the border. And some dozens of agents ended up beating and tasering him to death. And how do we know this? Because two people managed to get their footage recordings from the scene because the border patrol was actually compensating cameras, phones, et cetera, of what happened that night. And one person was a man on the ground who was holding a cell phone with limited video capture memory, a flip phone. And he kept having to decide whether to delete and keep recording or stop recording, right? Because he had no more memory left. That's a very intense moment. And we built controllers to let people try to record the scene. And also as a way to really engage them with the material, right? It really sharpens your focus. You're like, oh my God, what am I capturing? Am I capturing the right thing? I'm gonna be able to tell the truth about what happened that night. The other part of it was a young woman who was up on a bridge overlooking the scene. When the border patrol came, she snuck her camera off into her purse and walked off. And she literally got everything from above. And what I did was I brought her back to USC, put her in a motion capture suit, and I had her recreate her movements that she recalled doing that night. Of course, she had an audio track to base it on so that she could tell her story with her body as much as with her words. And that was the making of Use of Force, which is a pretty tough piece, right? Then a couple surprising outcomes from that piece. First, BuzzFeed had their employees try it out and then produced a video about their experience. And that had almost a million views and ended up with pages and pages of discussion about race in America in 2015. And more recently, I was told by somebody who worked in Homeland Security, in the Department of Homeland Security, that they've actually have used that project for training purposes. Also, this is a quick footnote on Use of Force that in much the way that Hunger in Los Angeles became the first virtual reality piece ever shown at Sundance, Use of Force was the first virtual reality piece ever showed at the Tribeca Film Festival. So it's kind of extraordinary that something you'd make because as a journalist, you feel a story is important and should be out there having these impacts in ways you didn't expect.
[00:16:56.824] Kent Bye: Yeah. And one of the other projects that you did after that, which was the Project Syria, but it was shown at Sundance in 2015. And so this experience of Project Syria of, at the same time, I guess, is the Galba-Aurora and Chris Moak did the Clouds of Residue, which is similarly trying to show different aspects of the refugee camps in Syria. But in your piece, you also did a digital reconstruction of the refugee camps. So I'd love to hear a little bit more context for how that came about with the World Economic Forum, and then what you were trying to do that was maybe different than what you could do in, say, 360 video and Clouds of Procedure, and what you thought the spatial recreation of those refugee camps were able to add with the way that you were telling that story.
[00:17:41.661] Nonny de la Pena: Project Syria, I think when we look about what's happening in the Ukraine now, I did a bunch of research about how children were specifically being targeted by the Assad regime, obviously with a lot of aid from Putin and Russia. And the way that children are being targeted again in Ukraine really underscores what they were doing in Syria. So two things that were different about wanting to do it spatially, there were a couple of reasons why I did it the way that I did it. Initially, I had reached out to the UN about maybe helping recreating some stuff in camps there. And they really wanted to put some controls on me as a journalist and what I could report. And I felt very uncomfortable with that. And so I actually ended up turning down going into a situation where the UN dictated what I could report on. Because what I really felt was happening with children It was somewhat documented, but not to the degree that it should have been. And I think that I was sort of pushing the envelope a little bit with my reporting. And I also felt that What do these refugees go through? What do these kids go through? And they can describe to you what it's like being in a camp live with 360 video, but they can't put you on scene to what they experienced previously showing up at that camp. And I really wanted people to understand why would they go to a refugee camp, right? What is that motivates them to give up their entire world experience, families, and bring their kids to these spaces? So we took video that had been captured of a young girl singing in a street in Aleppo when a bomb hits right by her. Fortunately, she survived, but others didn't. And we took as many pieces of video and audio that we could find both before and after the street, because that was not easy to do. You had to build an entire space that existed prior to being turned into rubble. And by putting you on scene with this moment, and we often show that piece where you wore a haptic vest, so you really felt the explosion as well. You kind of get this, I think the intent was to clearly embody a moment that describes the kind of terror and horror that people have to go through in these kinds of situations. I'm just going to wrap up your question also just on the Syria piece. Basically, the head of the World Economic Forum went through our lab at USC, saw Hunger in L.A., and then commissioned the piece about Syria. And I have to say, showing it to world leaders was an incredible experience. But to date, I really feel a lot of regret that it didn't matter that Syrians went through it and said this was exactly like our experience. It didn't matter that people had been in a situation where they'd been bombed. This was so realistic. For me, I re-experienced this thing. It didn't matter that as true as we could be, nobody intervened with Syria. Really, totally heartbreaking.
[00:20:39.797] Kent Bye: Yeah. And as I'm going through your filmography or your videography, your VR pieces that you've produced with Emblematic Group, which is, you know, 20 plus pieces that you've been able to do over the years, a number of them, I got to see the world premieres at different film festivals. The next one that jumps out at me is Kia, which was a piece about domestic violence, where you took some audio recordings from 911, and then were able to recreate different aspects of those scenes, which I remember just being really visceral to take something that was again, starting with the audio, you know, similar to the hunger in LA, where you're starting with a little bit more documentary aspects of audio, but this was more from 911 calls. If I recall, maybe you could talk a little bit about trying to recreate different aspects of this piece of Kia.
[00:21:26.757] Nonny de la Pena: In some ways, I still feel like Kia is one of my best pieces, or it's the piece that, you know, so often when I finish these pieces, I always feel like, oh, they're not good enough. Oh, I need to make them better. Oh, I hate that piece. Why did this part have to look like that? But Kia, with all of its rawness and all of its problems, I still feel about that story. I mean, because I feel that way about all my stories. I don't know. I don't know what is that key that I feel like worked so well. Basically what happens is two sisters are trying unsuccessfully to rescue a third sister from a fatal attack by an ex-partner. and the mother of her baby. And both the two sisters had called 9-1-1 and both of them, their calls were recorded live. And it gave me a way to intercut their audio tracks so that you could be with one sister who would sneak out to look for the police. And then inside the home where the ex-partner had Akia at gunpoint was holding her hostage and the sisters were trying to talk him down. And you're able to be in the room with them at this unbelievably tense moment, right? People sit there. I've never been in a room with somebody holding a gun before. And I had a police officer go through it who came out of it, again, saying to me, you know, that was so evocative of all the real situations I've ever been in. And what was really tough about that piece is that the sisters have almost got him calmed down. And when the police arrive, and they take a long time to get there, they don't go into hostage negotiations. They yell for the sisters to come out, and then they basically scream at him, and he just, the partner, the boyfriend, he immediately shoots her and kills himself. And you're not there in the room when this happens. I had a lot of Freedom of Information Act material. I could have depicted something extremely graphic. But this is the point where you, as a journalist, creator, artist, act as a filter to convey what really happens. And instead, I had the audience outside with the sisters as they are just in agony about the death of Kia. you know, I will forever live with Kia's last sounds in my entire body and feel her last sounds. You know, and then, but what happened in that moment is, you know, I was so kind of angry at the police officers for not having tried to negotiate with him. And I reached out to one of the officers to kind of let him know the piece was gonna be premiering. You know, I thought that was an ethical and appropriate thing to do. and he was grieving so deeply about this. And I just suddenly was, I'm sure he wasn't trained. They weren't trained to deal with this. And so, you know, the comment on this is, you know, both in terms of the easy access to guns for people who are abusive. He had prior arrests and still could get a gun quite easily without any background checks, but also how we don't spend our resources equipping police officers with the tools to help deescalate these situations.
[00:24:42.507] Kent Bye: Yeah. Yeah. As you're recounting the experience, I remember how visceral of an embodied experience within VR this was to be taking into those scenes that, you know, you're recreating. And I think another piece that had a similar vibe, I don't know if you were also, I don't recall if it was out of exile, but I don't know if that was the piece where a gay teenager was coming out to his father and it was a scene that was in a family living room. Is that that piece?
[00:25:09.097] Nonny de la Pena: Yeah. Yeah. Daniel's story out of exile. So Daniel had already told his family that he was gay and they decided to do a religious intervention with him. And he walked in the house and was like, Oh, something's up here. and actually turned on his phone so he was able to record the audio. And then Daniel helped us reconstruct what happened. And we hired actors to do the mocap for it and reenact his recollection of what happened, which is corroborated quite closely with the audio and some video that in the phone was not really, was down mostly at his side. And that situation, they end up, a stepmother ends up attacking him physically. And the way his dad talks to him, you know, you goddamn queer. It's just the hate, hate from the people who are supposed to be people who love you. You know, Daniel just wanted to be an accountant. And that's what he ended up doing. He ended up with that boyfriend. He got married and they became very happily situated. They just wanted to be a father or an accountant and just wanted a really normal life. And his family just couldn't accept that in any kind of loving way. And they threw him out of the house because of it. And so he had no place to live. And the piece reflected this huge amount of, if you go out to the streets for LGBTQ homelessness, it's massive because most of the time these kids have been thrown out from their homes. They're not accepted by their families. And so a lot of the homeless youth in the streets are because they've been thrown out due to their sexual orientation. And the other thing about that piece is I wanted to give some messages of hope. And I'd already done a little bit of work, I think probably the first piece of putting photogrammetry and videogrammetry together for a piece I worked on for the Women's History Museum. And in this case, I found folks who had been thrown out by their families, sometimes in the most violent ways. who'd found their feet, you know, had been resilient and found happiness on the other side. And I wanted to give messages of hope and resilience. And so we did an early videogrammetry of individuals telling their stories about hope.
[00:27:23.038] Kent Bye: Yeah, it's interesting just to think of the medium and the memories that I have from these pieces that I saw at these film festivals and all these moments and scenes that it really sticks with me. And those different interactions from this audio, that's like a documentary audio, then it's recreated in the virtual reality. And I guess in terms of the technique and form, when I think about other blending and blurring of different techniques across the line was another piece that stuck out to me in the sense that It's for a woman who's trying to exercise her reproductive rights and to potentially get an abortion, but being faced with all these protesters outside. And this documentary footage of 360 video of protesters who are almost pretending to be helping guide people, but to really try to be on the front lines of trying to change people's minds as they're trying to find their place into these abortion clinics. And so I'd love to hear a little bit about across the line, because you're not only covering another hot button issue within the culture, but doing it in a way that is trying to recreate the feeling of actually walking into one of these places while you're facing a lot of these protesters as you're making this walk.
[00:28:30.775] Nonny de la Pena: So that project was a very successful partnership with Brad Lipstein. You know, we kind of what we did is, you know, we found a clinic that he could shoot 360 that was having a lot of protests and brought somebody who'd experienced this already previously, who agreed to be filmed to kind of go through driving first up to the clinic. Right. So you can really, you know, see out the windows of the car lots of folks. And then we pick up into the full volumetric spatial embodied piece. where the car door opens and now you have to walk the gauntlet to get inside the health clinic. And everything that's being screamed at you, where we did mocap and facial capture on these characters, and everything that's being screamed at you is real audio that was captured at these various events across the country. so that you understand what young girls experience, all women who go to these clinics, because these clinics often are the only place around that often any kind of help for women's health issues, right? And when you're being screamed at, you're a whore, why don't you start closing your legs, you whore. I mean, just the worst type of stuff. And after that piece was done, we took it to some very conservative communities. They didn't know who made it. In fact, they actually thought it was made by protesters, ironically, a lot of them. But then what they said was, you know, in the most conservative who might not believe that abortion should be a woman's right, they all agreed that nobody should be treated this way. Protesters should be not allowed anywhere near these clinics. And it's always amazing when you find that kind of common ground, that this is something we all think of as inappropriate. and traumatizing. I got asked by Gartner to speak. I can't. They said, show us the technologies. How do you make these things? We did facial capture with a guy who did unbelievable lip syncing using that real audio. The audio is really strong. I thought they would have seen my talk before because I've shown that video previously, but I showed that video and the next day they were like, we had to send out 100 letters of apology. They were so angry at me for making people hear the truth. It was a very interesting experience to be getting something like, I had a lot of people come up to me after the talk and give me a hug. And then there, I guess there were a lot of people who felt that they shouldn't have to be exposed to that.
[00:31:04.438] Kent Bye: Yeah, well, I think that's a theme in your piece of trying to put people embedded into these different contexts that are not always comfortable to be in, but important to be able to tell these larger stories. And, you know, as with our limited time, the other pieces that stand out for me, at least are the WebXR experiences that you've done with reach.love, as well as the Stanley Hayami, which I had a chance to talk to you last year about at Sundance. And just a culmination of continuing to push forward the different volumetric techniques and different aspects of volumetric capture that you've been using over the years. And I guess being on the cutting edge of fusing both photogrammetry and CGI and Unity and trying to tie it all together in a story. And maybe a good way to reflect on your body of work that's being recognized here by the Peabody's with this legacy award with your collective work that you've been able to produce over the years is this PhD that you got in 2019 talking about inventing immersive journalism embodiment realism and presence in nonfiction, I'd love to hear just maybe a little bit of reflections on this dissertation that you did for your PhD that's reflecting on this fusion of these immersive techniques within the context of journalism.
[00:32:16.900] Nonny de la Pena: Yeah, it's a really interesting question, what I would call, you know, duality of presence, the feeling that you're here and you're there too at the same time. And I think that as journalists, that's one of our key goal of when we're trying to create our content is to give people a sense of being there, of understanding a story more viscerally, whether it's audio or film, or in my case, using these new, what we call extended reality techniques. to put people on scene as events transpire. And I think that's been the through line in always pushing the technology closer to some sort of realism. And as I say in my dissertation, it isn't always necessarily graphic realism as much as behavioral realism. That's a key point I found that if characters behaved realistically enough, your digital character doesn't walk through walls and penetrate things that we know they can't do. The mind actually accepts a lot of the flaws of this stuff. We talked about Kia being so powerful with its Sometimes some of these characters are very rudimentary in these early digital forms. That said, with the new smartphones, typically they're equipped with a LIDAR camera, which means you can capture your world with depth. So this is a place we're going to start seeing all kinds of interesting material captured that people are going to be getting to experience their world as it really is with dimension and not flat. And I'm hoping that the work that I've done over the years offers some guidance to think about how do we do spatial narrative? And that's something I reflect about in the dissertation, terms like the embodied edit. What does it mean to give people an edit in a story that takes their body from literally location to location, which is different than just watching it on a screen. With Reach.love, I also trying to find a way to put the tools in more people's hands so that we can shift the demographics of whose stories are being told and who gets to tell those stories. And who gets to share those stories? The idea with that is to kind of create a no-code solution that means that, you know, you aren't going to have to struggle with the C-sharp coding like I had to. So these techniques, like there's so many ways that people can start to create in digital spaces without having to learn how to code now. But how do we teach the ethics of making these kinds of stories? How do we tell the truth without traumatizing? What does it mean to say to people, like we did with Kia, put something up that says, warning, which in my experience is quite graphic. And also other interesting ethical things, like the two pieces we did with Frontline that I'm really proud of, after Solitary about solitary confinement, and Greenland Melting, you know, climate change piece. And really interesting questions still arise. For example, in Greenland Melting, we did photogrammetry of a NASA plane that was a research plane. And when you're in the piece, you're standing with the videogrammetry capture of one of the NASA researchers as he drops this tube containing a thermometer down the back of the plane next to a toilet. And you definitely feel like, hey, I could drop a tube down there measuring water temperature. Oh, they're not just stuck in some cubbyhole making stuff up, right? This is a couple of years ago now. I think a lot more people accept climate change as being real now, I hope. In any case, we filmed in the videogrammetry in a stage in Southern California. And the question was, what should he be wearing? Because we're putting people in the photogrammetry. But then if we put him in his flight suit, we're telling them we actually captured him on that airplane, which isn't really the truth. So what do you do? And then he was saying, I don't care what you want to film me in my shorts, my flip flops. I want to be in my airplane, in my flight suit. So we ended up filming in his flight suit, but it was a really interesting question about what is ethically appropriate to convey a fact to our audience. So I think we still have some nuts to crack journalistically, but there's so many new folks who've entered this space. who are moving from 360 video into more spatialized content creation. And, you know, don't get me wrong, I think there's wonderful stuff that comes out of 360, but the spatial thing is always something that attracted me. The idea that our world isn't flat and we should be able to experience our world as it really is, you know, that seems like closer to truth, right? Even if it's maybe not quite as graphically accurate as shooting a 360. So I think there's going to be a lot of innovations about to happen and not just in terms of technology and technique, but also in terms of storytelling approaches.
[00:37:00.062] Kent Bye: So one of the things that I also noticed is quite striking within the larger XR industry in these festivals is a lot of times these pieces will show at the film festival and then the distribution of these is not always so clear. And I know that you've struggled with taking some of these different pieces and trying to distribute like the Project Syria through Steam. And the gaming audience wasn't necessarily as receptive to some of these different immersive stories. In fact, there was quite a lot of backlash that you received with trying to put these types of stories that cover issues that can be politically polarizing in the context of these larger gaming distribution platforms, then get review bombed because people don't like the political message that may be coming through. So I feel like as I'm looking back through all these experiences, I'm personally grateful that I had a chance to see them within the context of these festivals, but also this larger frustration that it's not so clear as the best way to get these out into the world and into the headsets for people to actually experience a lot of these. And I know that the reach.love and other projects that you're doing are trying to address some of those different distribution issues, but I'm just Wondering if you had any comments or reflections on your body of work and the difficulties of trying to get it out into a larger audience to be able to see the work that you've been able to produce over the years.
[00:38:13.043] Nonny de la Pena: Well, I have to say after I put Project Syria on Steam and beyond even the pushback about that piece, because you were Syrian and a lot of them were Muslim, the pushback was really vitriolic and pretty horrible on Steam. But I also, even off Steam, I was personally attacked for having protected this story and tried to share it more widely. And, you know, it caused me to just stop putting anything personal on Facebook ever, right? So personally, I had to just say, okay, well, I have to protect my family and I stopped ever posting any pictures of my kids or anything, right? Which is sort of an interesting, you know, it was all around the edge of Gamergate and I just had so much terrible stuff hurled at me that I just, you know, I kept on making my stories, but I just sort of quietly kept my personal life out of public places. Sometimes people wonder why at Facebook, I might be, I just, you know, put my pieces or my accolades and why don't I, they think it's maybe being egocentric or vain, but the truth is I've just stopped putting anything personal for real reason. So I would agree with you that the distribution issue was hard. But at the same time, we're used to art exhibitions not necessarily being everywhere. And they can still have reach. At the dawn of early film, you had to go to these different traveling spaces. I am so looking forward to when the distribution problems are not severe like they are now. We just have a piece that we're finishing up on Lyme disease. and we were making it in the middle of the pandemic. And we had to shift everything so that we could get it on a Quest so that the distribution potential would be much wider. And that was really difficult. There's so many things I had to give up and change because, you know, basically you're taking something from a huge gaming computer onto an Android phone. and all that compression, et cetera, et cetera. But on the other hand, I do think that the Quest 2 has been a real step forward in the evolution of these technologies and the way that people can see them and the price point, et cetera, et cetera. I am a fan of the Quest 2. I really am. And I'm really glad for all the headset makers who continue to stay in the game and offer me ways that I can share content more readily. And I'm also really grateful for all the creators, because this ecosystem, you know, wouldn't be something at all if it wasn't for so many creators starving themselves literally in order to make the stories that they want to make. There's so many incredible artists who have embraced this medium, and I'm really grateful for all their hard work.
[00:40:42.016] Kent Bye: Great. And, uh, and finally, what do you think the ultimate potential of virtual reality, immersive journalism, and immersive storytelling might be and what it might be able to enable?
[00:40:54.098] Nonny de la Pena: So when I talk about all new smartphones, typically having a LiDAR camera on them, and that's going to allow people to capture their world as it really is with volume, you know, then the moment's going to be, when are we going to start seeing that married with actually being able to capture movement with volume? So we have a step or two to go, but the fact that already these are cameras in your pocket, give you an idea that people are going to be sharing their world with volume all the time. It will mean, journalistically, we have got to be thinking about our ethics ahead of time. Are we going to just let people step over the bodies when we capture a bomb explosion? I have to say, I'm really kind of astonished and astounded that we aren't seeing volumetric capture of the rebel in Ukraine being shared already. Because we could, we could right now let people really be standing in the middle of that rubble. But as I said earlier, the ethics of what we offer people as an experience is still going to require good journalistic practice, telling the truth, not necessarily traumatizing an audience. It's going to be something that we face when we put the videos of Vietnam War into people's 6 p.m. dinnertime televisions, right? That was a big moment when people realized that you can't just broadcast war scenes into homes with everybody sitting watching TV. And that was a moment where we started going, OK, when is it appropriate to show graphic material and when is it inappropriate? So those are the kind of questions ethically, journalistically that are going to become more prominent and important. But I am quite confident that we're not going to be creating all of our stories as flat media in the future. We know with radio and podcasts, these things have very long legs. I think film is not going anywhere, but I think there's this new medium. It's just getting going.
[00:42:45.243] Kent Bye: Great. And is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:42:51.285] Nonny de la Pena: Really, the only thing I want to say is just to thank all of the people who have worked with me to make these pieces. Making virtual reality content, making augmented reality content, it really requires a team effort. If you think about a feature making of a feature film and you think about all the different people involved, VR is really no different. There are dozens of people who often come together to make these projects. And I'm just really grateful for all the people who've worked with me at Emblematic over the years and put in heart, soul and many, many long hours so that our content could hopefully reach people in a way that maybe at some point might create a little change. Hmm.
[00:43:28.450] Kent Bye: Well, Nani, congratulations on this Peabody award, as well as the being inducted into the South by Southwest hall of fame, which just happened this past week. And yeah, just thanks for coming onto the podcast and being able to reflect a little bit on the different pieces that you've been working on over the years and really helping to pioneer and create this whole new form of immersive and embodied storytelling and journalism. So thanks again for joining me today on the podcast.
[00:43:51.930] Nonny de la Pena: I know you have to go, but I'm going to add one last thing, which is I'm really grateful to the Peabody's for giving me this award, because when I started, nobody in journalism thought the idea was valid. And this moment is, for my career, one of the most astonishing and powerful. And I can't tell you how hard I cried when I learned I got a Peabody. It was a validation of all my efforts. Thanks so much.
[00:44:17.519] Kent Bye: So that was Nani Lopena. She's the founder of the MMI Group, and she's been working on narrative and emerging storytelling, been working in virtual worlds and virtual reality since 2007, and is a part of creating a new program at the Arizona State University on narrative and emerging media. And there's a satellite program that's happening there in Los Angeles. I have a number of takeaways about this interview. First of all, kudos and congratulations to Nani for winning this Peabody Award. Like she said, she was really moved that this is a validation for her work, that a lot of journalists didn't really understand what she was doing when she was first doing it, and now being inducted to the South by Southwest Hall of Fame this past week, and then to receive one of these Peabodies for her body of work. There's something about the different types of stories that she's picking, and to be able try to translate it to take you into these different scenes. And there is something that's qualitatively different than this, having this sense of embodied presence when you go into these documentary scenes. And this is something that William Yurkyu has also talked about in terms of how the development of film is really focusing on these documentary forms. And it's interesting that, you know, Nani was taking A Hunger in L.A. to Sundance in 2012 before the big resurgence of virtual reality. In fact, was collaborating with Palmer Luckey in the sense that Mark Bullis didn't want to have his $50,000 VR headset to be taken to Park City, Utah. Palmer Luckey had been tinkering and creating his own different virtual reality headsets. There's something about her own personal journey that was a catalyst to the larger revolution when it comes to virtual reality. A lot of the documentary work that she's doing is really pushing the forms of how to tell stories within these different immersive media So, yeah, she's gone on to get her PhD, now she's teaching, and one of the things that we didn't get a chance to dig into much in this conversation, but Nani has been doing a lot of mentoring of other up-and-coming immersive storytellers as well, and I think that she's really been focusing on this diversity and inclusion and really embodying that within her own practice within the emblematic group. And she mentioned that's also a little bit of a focus of the ASU program that's on narrative and emerging media. So that's all I have for today and I just wanted to thank you for listening to the Voices of VR podcast and if you enjoy the podcast then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a supported podcast and I do rely upon donations from people like yourself in order to continue bringing this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
