Robert Hernandez is a Digital Journalism Professor at USC, and he serves a sort of innovation lab to see how journalists can use emerging technologies to tell stories. I talk with him about some of his projects including “The Deported: Life Beyond the Border” and a series of “Homeless Realities” 360 videos and Snapchat filters of immersive stories of the homeless population featuring audio guided tours of photogrammetry scans of their homes. So we cover what he’s been finding to be the unique storytelling affordances of VR and AR, and I share some of my deeper thoughts on the philosophy of journalism at the end of the episode.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. So Robert Hernandez is a digital journalism professor at USC Annenberg, and he's kind of like a innovation lab for journalism. He takes all these emerging technologies, and with his students, they kind of figure it out in terms of what can the technology do, and what kind of new storytelling capabilities could potentially be unlocked with this new technology. So he's been working with his students with augmented reality technologies and 360 video and actually has been creating these photogrammetry Snapchat filters where you have these photogrammetry scans of where homeless people are living and you actually have some interactive audio where you kind of get a guided tour of that space. So we'll be talking to Robert about some of these new affordances and what he's been doing with his journalism students at USC. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Robert happened on Wednesday, May 2nd, 2018 at the F8 conference in San Jose, California. So with that, let's go ahead and dive right in.
[00:01:14.659] Robert Hernandez: Hi, I'm Robert Hernandez, I'm a digital journalism professor at USC Annenberg, and my specialty is hijacking emerging technologies for journalism, for nonfiction storytelling. I've been at USC for almost 10 years, did an augmented reality journalism class in 2013, did a wearables with Google Glass class in 2014. And since 2015, every semester, I've been teaching a 360 VR immersive journalism course. My students and I produce under the name Javernalism. Journalism, but instead of a U, it's a V, VR. See what we did in there? It's not trademarked. Yeah, it was a log story where I reluctantly showed that. I was designing the website and the VR journalism didn't fit as a logo. So I was like, oh my God, this is horrible. Showed it to my students and they loved it. And so we produced from that. We produced immersive stories with ProPublica, The New York Times, NPR, NBC. We do a variety of different stories, some just in the last couple months. I was able to take my students to South Korea to produce stories about the Winter Olympics. We've done stuff about the Salton Sea with the desert sun in USA Today, about how the Salton Sea is dying and causing an environmental disaster. Right now, this semester, my students and I are producing immersive stories about those who've been deported. So during spring break, they went with the professor to Tijuana, Mexico and interviewed folks of all different backgrounds and stories, really heartbreaking stories. to tell those stories of those who've been deported and those that they've left behind. In fact, we're doing breaking news as well, right? So we covered the inauguration and the women's march, the presidential inauguration and women's march with the New York Times and NPR. And over the weekend, my students and I decided, even though class was over, to go down to Tijuana again and cover the migrant caravan that has been walking to the U.S. border from under us walking taking buses taking trains in hopes of seeking asylum they had reached the point that were doing that big walk to the border to seek asylum and we just went down with some three sixty cameras and try to capture the moments which got published by npr and that one scene videos got a quarter million views on face book so it's working it's it's we're getting our names out there were being recognized by news organizations and winning awards And all this stuff is because of my little classes like the Train That Could, being early adopters, experimenting, and pushing forward.
[00:03:38.009] Kent Bye: Great. That's amazing. And I'm curious to hear from you, because you're a journalism professor, you're looking at both traditional methods of journalism, print journalism, and yet we have all these new emerging technologies. What do you see are the unique affordances of, let's say, a story and how VR as a medium could cover dimensions of that story that are lost either through print or video or a podcast?
[00:04:02.420] Robert Hernandez: All that stuff is being defined now, right? So I was part of the Google News Lab study about the shift from storytelling to story living, the putting yourself in other people's shoes. That to me is really powerful with this medium, being immersed and holding presence with someone to get a better understanding of their perspective. Maybe you don't understand that life one to one, you'll never know what it's like to be A young girl in a refugee camp, but this technology brings you closer to that, more so than a photograph or video or audio, but it really brings you there. And that unlocks a lot of ethical challenges as well, which makes for a lively class for us to talk about how we can tell a story, especially regardless of the technology or platform. I am a journalist with a code of ethics. I need to be accurate and ethical. And for me, it's by any means necessary. So if I have to use a Snapchat filter or a 360 video or a piece of paper and text to communicate a story, we use that. And in doing so, we have to have these conversations. Conversations that are like as simple as, how high should the tripod be? Because that could communicate something subtly when you're looking at someone the cameras taller than the people you're looking down on, right? They're shorter. Or we had a conversation on our first project was with ProPublica where they approached us about working with them to tell the story about how the Houston Ship Channel was a sitting duck for a huge hurricane. And we talked about different angles of the story. They let us read the story before it got published. The students analyzed different angles of the stories, and one thing that resonated was the storm surge. A lot of people didn't understand or appreciate the impact of a storm surge. Superstorm Sandy had happened somewhat recently, and someone had made a comment from the weather service that said, you know, if we called it a tsunami, people would have paid attention. So we thought maybe we can use VR to explain what a storm surge is and to also use it to give scale. The scientists were predicting what was called the super storm surge that would generate a 30-foot wall of water. And so as we talked about how do we show that experience in VR, students started to think about what if we put you inside of a house and then there's, you know, you see the rain and wind and you hear all of a sudden the roof gets ripped off the house And I stopped them there and I'm like, that's great for Hollywood. We're doing journalism. So how do we boil this down to fact? And we just boil it down to scale. We used the meme Banana for Scale as a reference, where we went out, again, spring break, went to Houston, found a house that had an empty lot along the water, captured that in 360, brought that into Unity, and first set the table with reality and say, here we are at this neighborhood and here's this house in front of you. We drop a CG house next to that. And then we transition from reality, 360, into complete CG, and then the walls of water come. The first one is a 10-foot wall of water generated to represent the storm surge from Hurricane Ike. And you see the wall of water compared to this house next to you, again, the house as the banana for the banana for scale metaphor. And you're like, well, this could work. The house could survive. But then you see the 30-foot wall of water from the mighty Ike storm prediction, and you immediately see game over, right? We didn't have to destroy this house. We didn't have to figure out the architecture integrity. We just got the point across when you're looking up and seeing the crest of that wave. And it didn't have to be a photorealistic wave. It's just, that's a wall of water coming at you. These are things that we talk about. We talk about, you know, with the story of that we're working on now about the deportees. How do we talk about really heart-wrenching opportunities when we're not in the room because we're outside of the room conducting the interview? Tell me about what it was when you were deported, right? Tell us about really heartbreaking stories and experiences that they dealt with once they got deported, whether they turned to drugs or depression or prostitution to cope. These are journalism stories that we're telling, but there's a new nuance to it, a new set of challenges that we have to figure out as we go in telling them.
[00:08:10.962] Kent Bye: Have you seen Fran Pichetta's 6x9? Yeah, of course. So I think that, you know, this is from The Guardian, and one of the things that she said in that experience is that she was actually trying to figure out taking in first-person narratives and interviews with people. What she found was that when you're immersed into a VR experience of solitary confinement and you're in that room, that if you're listening to other people going through that experience, it actually takes you out of your own experience of being in solitary confinement. she switched from the third person narrative into a second person so that if she would do interviews, she would have people tell you as the experiencer what you are about to experience. And everything that was connected would be connected you to the environment. So it's almost like this shift of perspective of really focusing it on how do you translate an experience so that the person who's going through it will have their own direct experience and how does the audio help you get connected to that environment more.
[00:09:06.113] Robert Hernandez: Yeah, and we're finding that, like, it's a different type of storytelling, and when the students who have broadcast training are writing scripts, I have to tweak their wording a little bit to say, here we are at the U.S.-Mexico border, opposed to at the U.S.-Mexico border, right? The people on the left that you see are seeking asylum. It's almost like a virtual tour into a story. That's one of the things that we've been playing with, and actually Facebook Spaces is something I've been thinking about, and how to, give you a virtual tour to a story. So bringing 360 photos or videos into that space and I, as your journalism guide, can tell you right over here is El Chaparral. The turning stiles that you see to your right, that's the last thing you see when you exit the U.S. border and in front of you is the unknown, the city, this country that you may or may not have any relationship with, right? And we tell those stories, but then there's an active voice and a guide that is different from traditional journalism where we report, we witness, we don't direct, you consume. So these are all kind of new types of approaches.
[00:10:10.312] Kent Bye: Yeah, I just got chills hearing you say that because what I imagine with augmented reality and virtual reality, there's a certain amount of the story is layering on top of like the emotional architecture of people's experiences. This is an emotionally salient point where as people are in this point, this is usually the type of experiences that they're having and you and just had a beautiful metaphor of the unknown is in front of you, where that may just look like a land to people, but there's this layer of meaning of experience, this experiential meaning and story that people are being connected to with these different architectures of space. So I imagine that both with Google Earth VR, as well as with augmented reality, that we'll start to have a layer of these deeper stories of both the history, but also the collective experiential aggregate of the stories of those places.
[00:10:58.922] Robert Hernandez: Yeah, I like that you brought in AR because, as I mentioned, I started with AR. We actually use the 360 video as our canvas to augment. So often we can't do those difficult interviews in 360. It just isn't right. So we do traditional DSLR interviews and maintain eye contact from behind the camera and talk to the subject. And then we bring those traditional DSLR experiences into the space. So the world goes dark or we transition to an illustration. That's what we're testing out this semester. Or just their voice appears with the photo. We're trying to augment these 360 spaces to really get you to focus on different experiences or a certain point or their voice or a location. And the challenge with my class is while it's doing really amazing work, The thing to think about is every semester I'm hitting the reset button and bringing in 15 new students that have never done any of this stuff before. And I'm like, here's a camera. I don't know myself how to edit video, but we're going to figure this out. And it's just a really collaborative approach. That, to me, is the most exciting part about my class, that it brings students from cinema, and engineering, and video game development, and journalism, and public relations, and com, and business, and psychology, all this stuff coming together to figure out how do we use this technology to tell meaningful stories. As quickly as the technology evolves, that unlocks a new opportunity and a new way to tell a story. I'm already excited what we're working on in the fall is to use this AR immersive photogrammetry, videogrammetry experiences for telling stories about homelessness in Los Angeles. It's a new project that we're trying to do. I'm trying to teach myself photogrammetry. I was hoping Facebook would have an easier solution, but it's just, how do I hijack this technology, not as a gimmick, not to sell Ready Player One, which I appreciate and I love, but how do I get you to hold presence? with a topic that you might want to ignore or you lock your door and don't make eye contact with as you're driving by. That's the role of journalism to offer that human context to these types of stories. And if I can use technology and the creative minds of these diverse students to move that forward, we're really lucky in doing so.
[00:13:10.081] Kent Bye: Yeah, I know I've done an interview with Realities.io, and I know, as Balbandians, you've been doing a lot of, like, photogrammetry. I know that there's tools that are out there that you just have to take tons of photos and then do the post-processing on the back end and then clean it up, but in talking to Realities.io, I saw the potential of being able to do this archival of spaces and that really there's some ephemeral either art or just things that are going to be either destroyed that are being captured so that there is that archival of those physical spaces that can be referred to later and I think that level of archival nature I think is a big part of where I see the role of the journalism Coming in and I think that you know looking at the last goodbye which is Gabo Aurora's piece on the Holocaust where they did a lot of combination of photogrammetry of a Holocaust location and then on top of that doing sort of like a 2d billboarded 3d video on top of that so it's in essence like a guided tour of one of the Holocaust survivors who had been through this place and he's telling the story of what it was like for him to be there and I I He's there with an avatar representation or a video representation in this case. But I see that as a potential of these places being archived and then having the people that experience something there be able to try to capture what is in essence kind of the paradigmatic story of that place.
[00:14:29.292] Robert Hernandez: Yeah, you know what's interesting? The USC Shoah Foundation has been developing natural language processing and videogrammetry to capture the oral histories of Holocaust survivors because what's going to happen when they all die and the deniers say, where's the proof? And so they've been recording these pieces. You can Google new dimensions and testimony and here see a little video of Pincus, a Holocaust survivor that I think participated.
[00:14:56.414] Kent Bye: Yeah, Pincus was the same one that was in... I think Gabo had seen the piece that USC had done which I think I saw there's a little bit of an interactive oral history that he'd done where you could just ask him questions about the Holocaust and then Pincus would give an answer and I think Gabo saw that experience in that and then he sort of captured the full on-site story that he was telling.
[00:15:15.510] Robert Hernandez: Yeah, that's using natural language processing. You ask Pincus a question and he responds and I've been fascinated with that. That is outside of my brain intelligence level of how to pair natural language processing and a hologram capture to videogrammetry. But there are things along that way that the bottom line is still storytelling and that human connection, right? The thing that I tell people Coincidentally, I got hired to USC at the same time Noni de la Peña was hired to USC. And the story I always tell is that she and I were advising a magazine class on how to be more digital. It was a magazine class that was reporting on food deserts and how the recession, the economy had brought more people to food banks. And I remember advising on how to use WordPress or use data visualizations or audio slideshows. And I remember Nani saying, you know, it would be really powerful if you could feel like what it's like to wait in line in a food bank. And I remember going like, Nani, we only got 15 weeks, what are you talking about? Took her a year, but that was the piece that she proposed to the MXR Lab that was then shown to the Sundance Film Festival that Palmer Luckey was involved with that led to the Oculus. And the thing that I point out to people, two things, This technology was around for a while. I mean, it's not a new VR. There's been a series of disappointing promises for a long time, but it was when Oculus and then this story about the human condition, about waiting in line at a food bank, that it really captivated people, right? So it was storytelling that got that, not the technology. And the other thing that I point out is Nani was a journalist. and so was Palmer Lucky when he went to Long Beach before he dropped out to come to USC. Storytelling and what we do is so important and it goes hand-in-hand with the technology that they support each other and move forward. You understand we have to make money off of it. Man, in journalism we're trying to figure out that still in our past platforms, but how we can all come together to work and problem-solve and how do I make these cool, nifty gadgets and technology do something with more meaning to have impact. That's the fun part for me as a journalist and as a professor to bring all these different students together and just scratch our heads and say, I don't know, but let's figure it out this semester.
[00:17:34.810] Kent Bye: Yeah, it's interesting the focus on storytelling because I totally agree that, you know, that's the way that a lot of the journalism has been being done now. What I would say though is that I feel like the type of journalism that I am doing is more based upon like a Hegelian dialectic of conversation and debate and I'm in the process of being like an oral historian at the same time of sharing this information back to the community and that community is listening to it and then that we're continuing this sort of virtualized communication over the last four years. and that it's kind of like this unplanned, unscheduled, like I don't still know what the full story is. There's one instance where I covered a topic of the Upload VR case that happened to also be covered by a New York Times reporter, so it was a little bit of like, here's how the New York Times print covers a story, here's how I decided to cover that same story, and I feel like in some ways the New York Times editorial process was they have to figure out what the story is before they assign the reporter to go out and do it. And there's actually very little agency for the reporter to have their own free will to decide what stories they're going to even be covering. There's somebody back in New York City deciding what the story is. But for me, I've been immersed on the front lines of this story and really seeing it as it emerges. So I'm hesitant to the dangers of storytelling because I feel like there's a certain dimension of like wanting to know what the story is before you even start the story.
[00:18:59.317] Robert Hernandez: Well, there's a lot of different types of stories and storytelling. That's the beauty of it, right? And I think you straddle the line between real-time historian and journalist and documentarian that benefits the community. On behalf of the community, thank you for doing what you're doing. Keep it up. But it's very different from deadline reporting that I have to do something for the paper. An act has happened and I need to go cover it, right? What's interesting is before we, pre-internet, when newspapers were printing money, that's what the thing used to be, you would have so many reporters that they weren't on deadline, and what they would do is roam the city. Roam the city at free will, never go to the newsroom, often end up at a bar, but just gossip and shoot the shit to kind of find out what's happening, get the pulse of the town, and then a story would get triggered, right? As budget cuts came and the bottom line was more important then, really holistically covering the community that they're aiming to serve. It just became, this happened, go do it. Right? I remember a couple years ago, prior to coming to USC, my whole career has been a digital journalist. I worked at the Seattle Times Director of Development, where I ran a team of engineers and designers, and I went to a meetup. where it was journalists 30 and under having a drink and talking about our experiences and one business reporter talked about how an editor came in at 10 a.m. in the morning and saw all the cubicles were filled with reporters and the comment from the editor was, well I'm glad you all showed up for work today. A good newsroom is an empty newsroom because your reporters are out in the community. You can't cover the community by Googling them, by tweeting at them alone, by emailing them. You have to go out there and engage and be vested in the community so the community is vested in you. And I think you represent that true form of engaged journalism. It's hard to do it at scale at somewhere at the New York Times for a variety of different reasons. But thank God we have both. It's not either or.
[00:20:59.132] Kent Bye: What it reminds me of is that the Greeks had two words for time. There's the Kronos time which was all scheduled pre-planned Everything was sort of like something happens and you kind of react to it and the Kairos time is more subjective qualitative You're kind of roaming around trying to sense the quality of the moment It's more of a listening a deep deep listening and I feel like when I come to conferences I try not to plan much like we just ran into each other and we didn't plan to talk but here we are talking and having this conversation and I think that as you listen to the conversation and as you follow the conversation and as you sort of rely upon seeing who you collide with and what information comes to you, that there's a different quality of the story that emerges when you do that versus pre-planning everything out in emails before you go cover something. And so I sense that we've kind of culturally made this shift in all of journalism that is so focused on the Kronos aspect of reacting rather than that sort of more exploratory aspect. And I feel like the podcasting as a medium really is suited towards that. But I also suspect that virtual reality is also going to potentially be that type of, oh, I happen to be there in this moment. Here's a moment that I captured before the story has even emerged around it.
[00:22:10.880] Robert Hernandez: That depends on the technology, right? In order for us to go down to capture the migrant caravan, I had to charge up all my batteries, load up all my cameras, bring all the tripods. We met the day before to divvy up which backpack are you carrying, who's got the snacks, bring water, right? So it wasn't as impromptu as, let me take out my phone and capture this and tweet out this historic moment of a plane landing in the Hudson River or a protest happening in Puerto Rico right now, right? But we will get there, where the front-facing camera and the back-end camera is going to be fisheye lenses and enough that you do a 360, and therefore that 360 can turn into photogrammetry, and we can harness the different 360s in the room to make a true higher-end photogrammetry, crowdsourced photogrammetry model, and therefore I can walk into It's technology and storytelling and the appetite for it and the distribution for it and the creative storytelling and then the funding business side, all those things align, but there are many of us, yourself and creative storytellers, that have that in their hearts to continue to do that type of storytelling, right? And I want my students to be passionate, curious storytellers that have to live in both of those times, on deadline and when you're not on deadline, right? So when I was a journalist in the newsrooms, I always had the story I was working on now, the story I'm collecting string for in a couple of weeks, and then the one that maybe is that project, if I get enough time, I just chip away at it. And that's how most journalists are. Breaking news, though, is what gets clicks. It's now. Did you hear what just happened, right? And so that is dominating the real time. A lot of people blame social media and the real-timeness of that, but it's just our constant always on, always what just happened, right? And that comes at the cost of being reflective and thinking about it, right? And to use the journalism landscape, there is value in the tweet of this just happened, as well as value in the New York Times Magazine piece that took months to produce of that same event, but just to do not only more reporting, but a better narrative style. You're investing your Sunday afternoon to kick up your shoes and open up that Magazine or maybe you're reading it on your tablet and you're just giving it your attention and going on that journey And there's a lot of stuff in between and again, we're really lucky to have all those options It's not either or although sometimes it feels like that We're really lucky to have all those different types of storytelling platforms and techniques and now immersive journalism is among them, right? how the hell When someone's stressed out about tweeting and producing a long-form story, how do I get them to think about immersive or to capture something in 360 or think about AR? That's where my class and I come in. We're trying to do the R&D for the industry and inspire like, hell, I have no money. Even though I'm in a private institution, it doesn't trickle down to me. So I find ways to do it on the cheap and then I again, not a good businessman, give away my knowledge to journalists so they can do these types of journalism stories on the cheap and make it more common.
[00:25:18.807] Kent Bye: Yeah, as we're talking here, I'm like sort of comparing the different affordances of each of these different communication mediums and I have like an experiential design framework that I use to try to break down different dimensions of presence and I feel like we can sort of look at the podcast, you have the ability to have extended conversations, you can go really deep and You don't have to edit it. You can have the full context. You can see the whole holistic. And that's really kind of like this mental presence of being able to really dive deep into a conversation. And people have been able to kind of bootstrap themselves into the VR industry by listening to the Voices of VR podcast. So it can be educational. And then there's a bit of how a conversation can contain a certain level of knowledge that can't be reduced with just a soundbite. And I feel like that there's a lot of the journalism that is trying to sort of wrap the story that is trying find that narrative hook that's going to immerse people into being able to really paint a story of a picture both by setting a picture of the context of the spaces but also the deeper stories that are emerging that is really covering the human experience that is unfolding. And I feel I feel like that in VR journalism, stuff like Nany de la Pena has been doing is really about perhaps giving you more of a direct embodied experience of maybe walking to an abortion clinic and what's it like to have people yelling at you. So you're getting a direct embodied presence and an embodied experience and an environmental presence of that. And I feel like that maybe the frontier that hasn't been explored as much, but something like the Pincus conversation where you're able to ask questions and so you're expressing your agency and your active presence. And I feel like that there's likely going to be other dimensions of like video game technology where you can make it as an interactive experience in some way so that people can like both make choices and take action and be able to get this information. I feel like that, you know, if we center the qualities of human experience, we'll maybe get some insight into the different dimensions of how the communications mediums are going to be working out, but yet, you know, what kind of forms of journalism are going to be focusing on that active presence, the embodied presence, the emotional presence, as well as the mental and social presence.
[00:27:18.958] Robert Hernandez: You know, one thing I want to acknowledge is how Nani was treated when she was at the school, and how she was treated by the industry when she was pushing this new type of immersive storytelling. She wasn't treated well. They accused her of manipulation and fabrication, and she never really fabricated. You know, it's an illustration at times, or a representation, but I remember being on panels with her, at conferences with her, when someone would yell, almost like, shame on you, you're not a journalist, you're manipulating. And good storytelling is manipulation to a certain extent. I gotta get you to care. If we look at it in newspaper terms, that headline's gotta hook you. That's where the photos are too, that'll get your attention. And I gotta get you with the lead. And I gotta get you to go to a cliffhanger before you do a jump. No one remembers what a jump is, but that is when, because of the front page only had X amount of space, you would have to find A12 to continue the story. How do I keep you interested in my story to get you to go inside of the page before you drop out and go to another article, right? These are historical influences in our medium and journalism that have nothing to do with the storytelling technique but because of the way the newspaper worked, or how a telegraph worked, right? That's the inverted pyramid, which is essentially to our type of storytelling, comes from early days of the telegraph, where you gotta say the important stuff first, because you don't know when the line's gonna get cut, right? And that's where our inverted pyramid, so you cut from the bottom. All these things have influence to how we tell stories, but I always look at Nani and others who are wide-eyed and ambitious and creative, They know what they're doing, right? Nani has the ethics of a journalist and is doing the best she can to balance those two worlds and push both sides to get uncomfortable, right? Because I can make up anything in VR, and people will. But how do I not make up everything? When do I stick to the facts and nonfiction to still have an effective story? I'm still scratching my head on how to gamify it, because I don't know what story would be an appropriate way of doing that. You know, the abortion clinic piece, I haven't sat down with Nani to ask her, were those actors?
[00:29:29.756] Kent Bye: in the car. Those are those are actual well so that was live footage going into the place that was like a 360 video capture and then they were recording field recordings of people from different locations from around the country yelling at people as they were walking into the abortion clinic and so that's a composite so as you're walking into the abortion clinic you kind of get this It was almost like a contrived one-by-one, someone's yelling at you. So you're activating these individuals who are screaming something at you. But those are field recordings that were then recreated in the experience.
[00:30:02.405] Robert Hernandez: There's one scene where they roll down the window and someone says, what are you doing here? And to me, I was like, did that really happen?
[00:30:09.391] Kent Bye: As far as I understand that, yeah, that actually happened.
[00:30:11.413] Robert Hernandez: So to me, that's extremely powerful. For others? I would say, what if it didn't really happen? It's representative. This is a conversation that we have in my classroom, right? Well, we're representing this story versus journalistically purely capturing the story, right?
[00:30:26.479] Kent Bye: We talked about possibly— Just to fill people in, they were driving up to the abortion clinic and there was someone who was there from the protest and they were actually trying to convince them to not go and they were basically like, go away, don't do this. Like right as they're trying to find a place to park to go get an abortion, there's people that are intervening to try to convince them to stop. And it was a kind of a candid moment that they captured on 360 video. And yeah.
[00:30:51.872] Robert Hernandez: And so as a storyteller, specifically a journalist storyteller, you grapple with, well, how do I capture that moment? Do I illustrate that and be representative? Or am I lucky as hell to capture that exchange? But the goal is still the same. I'm trying to get you to feel like what women are dealing with when they're going to this tough decision. And talking to friends at Emblematic, they talked about how they showed it to people. One person who was a protester is no longer protesting in that way. They understood that other side. And that to me is super powerful. to really, when we say it's an empathy machine, storytelling, whether, regardless of the technology, if I can get you to understand the other, hell, if I can get you to not demonize the other, and I know that's ambitious and naive, but to just embrace that that other person you're really pissed at because they're a different religion, or a different political point of view, or they like a different baseball team, if you look at them as a human being and get you to care, That's an incredible win, and that's what I like about journalism and applying this type of technology to that journalism. And I want all types of storytelling there, but I do know, like I was trying to say at the beginning, was Nani's gone through some stuff and earned that title and earned that respect, and now here comes everybody, and every time I see Nani, I'm like, Godmother of VR, it took you longer than 15 weeks, but you're right. And we're all trying to play catch up and also define this world and really balance that. I think it's a really great way to view journalism in those two times, conflicting with each other and how much do you invest in one or the other. And the truth is for journalism, you need both.
[00:32:34.767] Kent Bye: Great, and so for you, what are some of the biggest open questions that you're asking or problems that you're trying to solve?
[00:32:40.289] Robert Hernandez: For me, the current problem we're trying to solve is, so 360 video, I think we got that. We figured it out, we know how to do it on deadline, trying to train more newsrooms to do that. The technology is enabled, you know, the Insta360 camera line is really fantastic from low end to high end. So that problem, I can get it. Distribution is figured out, for the most part. I'm trying to figure out photogrammetry and videogrammetry for room-scale experiences. How does someone who doesn't know how to code, who doesn't know how to 3D model and has no money, aka me and every other newsroom, how do we crack that nut? And luckily with partnerships with tech companies from Structure Sensor to DevKit, they've gave us some in-kind donations to us figure out. How do we capture this and tell this story? Talking to friends at the New York Times, they talked about how they did the augmented reality piece on David Bowie's costumes. And they used a variety of different technologies. And the takeaway was photography and using reality capture or the other one that I'm forgetting. I forgot. Two software options. It's like Pepsi or Coke to create photogrammetry 3D models through photography. And I'm trying to figure out how to do that. on the cheap, on deadline. So that's what we're working on in the fall. How to use photogrammetry to tell the stories of the homeless with this stuff.
[00:34:05.072] Kent Bye: And for you personally, what do you want to experience in VR?
[00:34:09.775] Robert Hernandez: When I was an active reporter and a journalist, the privilege I get to interview someone and hear their story, it's just so, I don't know, you just learn so much as a human being and you get a different perspective and I love getting that Insider information, right? I love podcast because the podcast and I'm a fan of or yours WTF nerdist where they're just having a conversation and from there come nuggets of wisdom that just add to my perspective and I like VR experiences that can offer that that just give you like wow, I never thought about that way or I just I held eye contact with a Holocaust survivor or someone's been deported or someone who is different from me that is not in my life that I don't have access to. Not because I'm against them or anything like that. It's just I don't know anyone like that in my life so I don't have knowledge of their life experience. Those things given to me are just make me a better human being.
[00:35:11.600] Kent Bye: Great. And finally, what do you think is the ultimate potential of virtual reality and what it might be able to enable?
[00:35:19.263] Robert Hernandez: That's that question. Well, I think for me, I'm a journalist, so I view it through that experience. And everything I've talked about is what I'm aiming for. That experience to unlock something that I couldn't otherwise. That the nonfiction experience helps me not only have empathy, but the whole wide range of emotions to better understand my community. Whether my community is in my neighborhood or my community is on the other side of the world, in a country I'll never visit and in a language I will never understand. But that connection through humanity, through this immersive technology is just so powerful.
[00:35:54.474] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say?
[00:35:57.694] Robert Hernandez: No. Thank you for doing what you do and I always love seeing you at conferences and honored to be interviewed by you. Awesome.
[00:36:04.336] Kent Bye: Thank you so much. Thank you. So that was Robert Hernandez. He's a digital journalism professor at USC Annenberg. So I have a number of different takeaways about this interview is that first of all, well, I really think of what Robert is doing is this innovation lab, not only just for journalism, but also for storytelling in general. Because I think that from a journalism perspective, then you're trying to tell stories that are going to be in some ways trying to reflect what is happening in your community. And so there's this strong connection as to building empathy with the world around you and being able to tell stories in a way that allows you to connect with what's actually happening with people. And there was a moment when Robert was saying that they were doing this 360 video and they're kind of walking into the United States for the first time and just kind of having this context of that environment and how for some people, that's like a beginning of a whole new adventure for them. And the work that they've been doing with the homeless population within Los Angeles is they've been taking these photogrammetry scans. Previously, I did an interview in episode 711 with Russ Limited's Adam Solzhenitsyn and he was talking about how they actually had been doing some of these photogrammetry scans of a homeless encampment. I think it was in Anaheim. And they decided not to do anything with it because they didn't think there was like enough trust built to come in and capture what is ultimately someone's home. Imagine if you wanted somebody to just come into your home and start doing photogrammetry scans without your permission. I think you, you might be a little resistant because there's a certain dimension of that. That's your private place. Even though it's in the public, it's your home. And so Russ Limited ended up kind of scrapping that project. But Robert Hernandez and his students have actually been having the people whose home it is give them a bit of a guided tour. And I think that's a really interesting point that kind of gets into the ethics of privacy, where in one context, you're kind of walking into someone where it feels kind of like unethically right to be able to just start scanning people's homes. But if someone is giving a guided tour of their home and starting to describe what all these different things are, it starts to allow you to get a sense of what the culture is, like what's it like to either take a shower or oh, here's somebody who's actually watching somebody else's home because they have this self-contained breaking down of your home and putting it into a movable cart and then have other people watch your home while you go off and do something. And so just these little moments of culture that you can get a sense of by getting the context of that space. There's something about the relationship between space and some sort of deeper intuition about a story. We go through different lived experiences in our lives, and from those lived experiences, we are somehow mysteriously able to extrapolate the deeper story of what those experiences are and what they mean. And I think one of the things that virtual reality is doing is that it's allowing us more sophisticated ways to be able to intake much more diversity of different types of experiences that we'd never have access to otherwise. So being able to give a spatial relationship of something, I think, allows you to tap into those different dimensions of your perceptual system that are, in the same ways, feeding into your experience of that process of that lived experience and intuitions that are extrapolated into the stories. And I think in that, it's gonna, first of all, allow you to help empathize with someone's context and what it's like to be homeless. As you start to learn about all these things, you start to get a spatial understanding of their homes and what is their relationship to everything else within their context. And then I just actually went to the American Philosophical Association Eastern meeting where I was talking to philosophers. And I've gone through about 10 different communities as a journalist over the last 15 years. I've covered journalism with the Echo Chamber Project. I've covered open source software and consciousness communities and virtual reality and augmented reality and artificial intelligence, mathematics and philosophy. So I've gone into all these different communities. And one of the things that I've been taking away is that we're kind of moving away from almost like a binary, he said, she said, inverted pyramid type of structure of journalism that was in some ways an artifact of the technology. The technology meant that if you wanted to tell the story in the first couple of graphs, you had to put the most important information because you didn't know if the telegraph was going to lose its connection. And so we've kind of been left with that artifact of how stories are told, which is you kind of linearize it into an individual linear story that you read and you consume, and that's the story. Well, the problem with that is it actually takes a lot of time to actually digest and process what the story is. And so if there's something like a new emerging technology, like virtual reality or artificial intelligence, if you wait until you figure out what the story is, then you may be lagging behind for years and years, which is often the case when it comes to like the mainstream media, when it comes to covering things like either virtual reality, augmented reality or artificial intelligence. And so as I embedded myself into these communities, what I'm finding is that it's more of an ecosystem. It's more of a nonlinear complex dynamic where I'm trying to define the different power structures and dynamics with each interview at a time, being able to tell you that individual's perspective and how they're related to the larger whole. And I'm finding also this in the cryptocurrency community where the cryptocurrency startups, it's not about them having a story about what they're going to do, it's about how can they actually prove that they're connected to many different parts of a larger ecosystem because a lot of this protocol development is, okay, you can develop a protocol, but are people actually going to use it? Is it have some sort of momentum in a larger ecosystem around it? Because of that, they've had to look to these social network graph analysis types of tools in order to show a more complex ecosystem that goes beyond a linear story. And I think it's because of that, that we still have journalism that's very focused on that linearization of telling the story, that you kind of get away from this ecosystem mindset. Philosophically, it's kind of the moving away from reductive materialism and moving into something like process philosophy, or moving towards something like Bayesian statistics as a foundation to philosophy of science. And it's also like machine learning, which is, it's actually kind of defining the relationship to all these individual component parts. So let's just look at computer vision, for example. When you see something, you can't point to any one individual aspect of that object. You're kind of holistically taking it in as a gestalt. And what you're doing is you're defining all the different pattern relationships between the things. And based upon those pattern relations, you're able to say, ah, that's a table. And that's not a table. That's a small table. And that's a chair. you know, archetypally these things are very similar, but given the context in which it's in, as well as the overall relationships of all the individual edges and everything else in relation to each other, it creates this matrix of relationships that are described mathematically through linear algebra, but also this higher dimensional topological math that is in some ways defining a million different higher order features and how they're related to each other. It's those higher order features that are able to describe the gestalt within things like machine learning and computer vision. So just the same, that same type of concept of moving away from a linearization and moving more to a non-linear relational aspect of trying to see how things are related to each other, that is what virtual and augmented reality are going to be amazing at. Because it's allowing you to paint a picture of an ecosystem and to see how things are related to each other. And I think that what some of the things that Robert is trying to do with his journalism students is start to look at how you can give a guided tour of someone's home. And you get a spatial relationship and you get a sense of the larger gestalt, but it's also starting to paint out like, oh, in order to take a shower, this homeless person has to take this bucket that's right here and take it into a disabled bathroom and then basically wash themselves. And so you get a sense of, oh, well, this thing that I have to do, I take a shower in the shower. Well, this is what that same aspect of what they have to do. This is what it looks like for them. So I think that my sense is that as we start to move into virtual and augmented reality, it's going to start with these types of photogrammetry, but eventually we're going to start to have like these memory palaces and network graph analyses that are able to be interactive so that we're able to actually see and visualize the relational components of how things are related to each other. So again, we're moving away from linearization He said, she said inverted pyramid and moving more towards a more relational ecosystem approach. And I think a lot of the work that Robert's starting to do is starting to tap into the very early parts of that. But I think there's going to be a whole other level of abstraction that it's going to go beyond just a singular way of looking at things. I think Banner Snatch actually from Netflix, if you look at some of the 3D network graphs that have been created in terms of trying to map out all these small choices and how they're related to each other, it's trying to do that. It's trying to give you this sense of these small choices, but it's in the context of this larger nonlinear ecosystem of dynamics where, yeah, there may be like five different major outcomes and conclusions to the story, but there's like trillions of different pathways that you could go through and you don't know how these different combinations and choices that you're doing, how that's kind of splitting you off into these other branches. And so that is giving you this embodied experience of a metaphor that is something that is nonlinear. And that's what Sam Roberts told me, is that when you go into a video game, it is building up that complex relational dynamics as you're able to play and express your agency. And as you do that, you get this sense of a metaphor of what that complex system is. And I think as we move forward, we're going to find more ways of doing that and expressing our agency to be able to see how things are related to each other, but also that fire element of locomotion where you're able to actually locomote and move through a space. And so as you move through that space, you're able to also start to see how things are related to each other, kind of triggering that within your brain. So that's just a little bit of, I guess, the, the foundational aspects of the philosophy of journalism. And I think it's going to continue to change as to moving into more of these ecosystem types of approaches. And this is something that I've personally struggled with in terms of, as I go from virtual reality and I start to branch out into AI and math and philosophy, it's been challenging to figure out like, what is my philosophy to figure out how, how can I like do this? And I think another lesson that I've been learning in terms of my own philosophy of journalism is to fail fast and to make inferences and to have some sense of a topic domain, but to be interacting with experts and to be willing to be wrong in the short term. But in the long term, if you have enough iterations and move towards more of an agile approach towards journalism, which is you're getting out there, you're failing fast, you're just doing it over and over again, and you're talking to a lot of people, then you can actually get closer to what the actual story is. And that was what was really striking to me. and hearing some of the history of journalism, where when journalism was flushed with cash, they had journalists who were just openly, freely roaming around the city and allowed them to be able to talk to people and see what was striking to them. As they start to talk to enough people, then they start to see these different themes. And then there's this intuitive process of your own gut feeling in terms of, ah, this is a story. I'm going to take this lead and start to track it down. And I think it's that being in that open-ended Kairos time where you don't have any fixed deadline or plan, and you're just able to freely roam around and see what this story is emerging. I think that's the approach that I'm personally trying to do on The Voices of VR, where I'm able to embed myself within these different communities and really sit and digest and see what the deeper stories are. And I'm starting to see, I guess, a roadmap, philosophically, what the common themes are, because My fundamental premise is that virtual and augmented realities represent this huge transformative potential. And like trying to actually define what that is and what that means, I think it's moving from linearization to nonlinear ecosystems. And there's so many different implications for what that means from the philosophy of journalism, the philosophy of science, to how we're going to break out of these, he said, she said, false binaries and really start to do a network graph analysis to see the modularity classes and how things are connected to each other in ways that you may not able to actually point a finger at but in mathematics similar kind of things happening in category theory which is trying to come up with some of these primary fundamental archetypal dimensions of a math object and if you sort of reduce it down in these different ways then you could start to see how those different branches of math could be related to each other and you can kind of jump back and forth between them and maybe prove and solve a problem in one domain and then jump back and then once you know that it's possible to then solve it in that other domain. And so why and how that works I think is a huge open in the philosophy of mathematics, but it's that same principle that you could start to see how all these different things are connected. The other thing that's huge is this siloization within academia where each academic branch has been able to isolate each other from its other branches and because of that they've cultivated their own linguistics and their own language. And I think part of what virtual and augmented reality and as well as artificial intelligence are going to be doing is it's going to be like this centralized melting pot for fusion and interdisciplinary and cross-disciplinary, cross-cultural and cross-generational in terms of like going back into ancient philosophies and medieval philosophies and contemporary modern philosophies and feminist philosophies and non-Western philosophies and seeing how there's different dimensions of the human experience that are pretty consistent throughout. So why are we creating these arbitrary taxonomic boundaries between these different ideas just because they happen to happen in the past or we discount them in different ways? They still have valid perspectives and valid lenses. And so how can we actually fuse all these things together through some sort of framework of the human experience? A couple of other things I took away from the philosophy conference is that there's no comprehensive framework for privacy. We can't define it. We don't know what the definition is. Because we can't define it, then it's basically left up to the entire technology community to decide from their own moral intuition what those boundaries are. And clearly, they've not been doing a great job of that because they've been transgressing all sorts of different boundaries and having this impact of gathering all this data that in some ways should be required to be private, but The problem is that there's no comprehensive framework of privacy that's coming from the philosophers or the legal scholars that are implementing it in some sort of guidance. And so part of that is because it's a moral intuition in ethics. And whenever you come with morals and ethics, there's no clear boundaries as to what those are. You have to have a lived embodied experience of that situation and you kind of feel what your gut is. And then based upon that, you develop your own sense of moral intuition. you know, this is a debate in philosophy for how that actually happens. Is it a logic system? Or there's these meta-ethics to try to, like, see if that moral object is related to other process of rationality? Or is it just a completely instinctual, sub-symbolic process by which you just have a moral intuition and a gut feeling about things, but it can't be firmly defined? And I think that so many of the issues that we're facing today is because we have this ambiguity in our moral intuitions and how to draw the boundaries of what's right and what's not right, both from how we act as an individual, but also how we interact with each other as a collective. So there's no comprehensive model of the human experience. There's no comprehensive model of the consciousness. There's no comprehensive model of morals and ethics, and there's no comprehensive model of privacy. And because of that, we're kind of left with, well, maybe if we're not going to be able to mathematically prove any of things, maybe we can come up with good enough models and frameworks through the lens of virtual and augmented reality and artificial intelligence. And through that, we're going to have an experiential dimension of some of these things. And then we'll be able to maybe backport some of these frameworks of experiential design and moral frameworks and ethical frameworks and maybe generalize them as we have all these different contexts. So that's some of the deeper things that I've been thinking about. And as I go out and cover the math conferences next week and then go to Sundance, I'm gonna be exploring all these other issues of what is the deeper philosophical implications of all these things, but also like these frameworks and how to kind of jump in between these different contexts. And category theory within mathematics is an interesting thing to see, like why is it explaining things so well? And what's the deeper philosophical foundations of that? And can we take that principles of whatever's happening at this archetypal translation, kind of lingua franca, going from one area to another area, and kind of bust out of these silos, because each of these academic areas have been cultivating their own set of linguistics, and we need to bust out of those specializations of those words, because oftentimes you'll have two different communities that are using the same language to mean completely different things. And so how can we make it less abstracted and a little bit more embodied and intuitive? And I think that the perspective in the lens of VR and AR, as well as AI, I think is going to help do that. And as we do that, it's going to facilitate more of those cross-disciplinary and interdisciplinary and cross-cultural and cross-generational conversations that we're able to really bust out of these kind of false binaries. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. I do rely upon my relationship to you, my listening audience, in order to actually sustain this type of work. And I've got a lot of big plans in this coming year. And yeah, I'd love to just get your support to help support this flagship podcast and eventually kind of spread out into these other podcasts. And as I do that, I hope you could see that it's going to better inform me as to what the deeper story and the deeper transformative potential of these immersive augmented and virtual reality technologies might be. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
