I interviewed Nonny de la Peña, Emblematic Group founder & Founding Director of Arizona State University’s Narrative and Emerging Media Program as well as Mary Matheson, Professor of Practice at ASU’s Narrative and Emerging Media Program, at Meta Connect 2024. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: the Voices of VR podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continue my coverage of MediConnect 2024. Today's episode is with Nani Del Pena and Mary Matheson, who are running the Arizona State University's Narrative and Emerging Media Program. So Nani is a pioneer within the context of immersive storytelling and immersive journalist who had the very first VR piece of Hunger in LA that was showing at Sundance 2012. Before even there was like an Oculus Kickstarter and actually had Palmer Luckey there with a prototype of one of the early Oculus headsets. And she's continued to go on and be the founder of Emblematic Group, which has been doing lots of different cutting edge immersive storytelling programs over the last decade plus. And she's also the founding director of the Arizona State University's Narrative and Emerging Media Program, which we talk about here and some of the intentions around this program. It's actually based in Los Angeles, even though it's part of Arizona State University. And Mary Matheson is also a professor of practice and a documentary filmmaker who I had a chance to have on the podcast before as well. And she talks a little bit more around some of what they're doing there at the ASU's Narrative and Emerging Media Program that they're now going into their third year. And also their 4D Gaussian Spot Studio that they're getting set up there. So that's why we're coming on today's episode of the Voices of VR podcast. So this interview with Nani and Mary happened on Wednesday, September 25th, 2024. So with that, let's go ahead and dive right in.
[00:01:40.037] Nonny de la Pena: I'm Nani Della Pena. I am best known for having founded Emblematic Group and have been doing virtual augmented mixed reality for over a decade. But more recently, I was asked by Arizona State University to start a new center on narrative and emerging media. And the space includes a master's degree, a graduate degree program, and a number of really interesting initiatives. that are deploying XR across the city of Los Angeles, something we call the city of awe. And we're building some really interesting labs there, including a 4D Gaussian splat experimental capture stage.
[00:02:15.983] Mary Matheson: I'm Mary Matheson. I'm a professor of practice at Arizona State University's Narrative and Emerging Media Program, working with Nonny de la Pena. And I'm also a virtual reality director of documentaries.
[00:02:29.346] Kent Bye: Maybe you could each give a bit more context as to your background and your journey into the space.
[00:02:39.073] Nonny de la Pena: So I guess I built my first VR piece in 2007. ended up later on 3D printing my own headsets in order to let you keep walking around. I had the first VR piece at Sundance, Tribeca, World Economic Forum, and for the Sundance piece, Palmer Luckey was crashing on my hotel room couch and driving the truck around, and I think most people would agree he was the lab intern for USC's Oh my God, why did I forget the name of the lab?
[00:03:10.971] Kent Bye: Mixed Reality Lab?
[00:03:11.932] Nonny de la Pena: Yeah, that's right, MXR, Mixed Reality Lab, thank you. I seem to have blotted that from my memory. It was a long time ago, I did my PhD at USC. So yeah, I hope that gives me some context. So I've just been constantly trying to innovate, push, but tell impactful stories, fun things, fashion, games. We've done a lot of different kinds of work, but always with the concept of how far can we push this new medium.
[00:03:36.678] Mary Matheson: My background is in documentary filmmaking. I was a journalist originally and a lot of my work is in post-conflict zones or reporting on pretty much untold stories. So I was always looking for ways to communicate with audiences about kind of difficult subjects and came across VR in 2016 through the UN work essentially. and couldn't quite believe how I could, instead of sort of bringing my characters into the living rooms of my audiences, I wanted to bring my audience into the space of the characters. So that's how I got into it.
[00:04:12.923] Kent Bye: Great. And so I guess with the ASU, with the narrative and emerging media program that you have there, maybe just give it a bit of context for how that came about.
[00:04:22.546] Nonny de la Pena: So, as CEO of Emblematic Group and as a woman in technology, in fact, I think you and I, Kent, met when I was like one of three women at a 300 person event at the very first Silicon Valley virtual reality meetup many, many years ago.
[00:04:35.797] Kent Bye: That conference, yeah.
[00:04:36.838] Nonny de la Pena: That conference, right? Yeah, that was a long time ago now. But I would try to bring in interns or try to diversify in the best way that I could. But, you know, as a VR studio, obviously, you're always trying to make things work financially. So it limited... the amount of people I could bring in and the difference I could make. So when Arizona State University came to me, and here was a place that, it's a state university, so fees are incredibly low. It has the largest first-generation population. Its goal was to create, as I put it, I want to shift the monoculture of who's working in this space, who's distributing this space, who's creating, whose stories are told. So when they asked me would I come in and start this new program, and I knew that I could make this kind of a difference, I said yes. and i've been lucky enough that i've been able to raise a couple million dollars for scholarships for lab space for student productions and now the city of awe initiative which is a three-pronged initiative one we call haptics for inclusion which is a big haptics push the second one is lab pacific affection how could we use the co-creation process to bridge socioeconomic divides between individuals and the final one, Reclamation, working with the California Transportation Authority and Professor Marcelo Oliva from LA Trade Tech and Kathleen Cohen, a technologist, were teaching community members how to make digital twins of discarded spaces owned by Caltrans And then we're teaching them how to rethink re-greening them as parks, as food sources. Then we're implementing those designs, we're putting in sensors, we're bringing that back to the digital twin, and then we're releasing all that data to public interest. So this is a really impactful effort that I'm allowed to make through the support of the university and by being able to helm this program. So in many ways, it's a dream come true that I can actually maybe make a difference in all the different ways this technology is utilized, how it's perceived, who's doing the making, who's doing the thinking, and hopefully leave folks with skills that are also employable from my student level to the community members who are learning how to work in Unreal to do digital twins.
[00:06:48.655] Kent Bye: Any thoughts as a professor of practice, you're thinking about the embodied experiences of actually making things. And so but I'm just wondering if there's other initiatives or like how you start to think about experiential design. And we're still telling you something that's still a little bit of like the Wild West. We've seen a lot of stuff that's been produced over the last decade or so of really, you know, looking at what's. at the festival circuits that have really started to pick it up since 2012. So a dozen years that we've been having more of a consumer space that has these different types of stories. But yeah, I guess as you are working with the students and getting people ready to both experience what's out there, but also to produce their own work.
[00:07:26.414] Mary Matheson: I think things have changed a lot, even in the last four years. So I started teaching. Actually, I started working with YouTube Creator Labs back in 2018, doing training around 180 filmmaking. And then later on, I designed and produced and ran a master's in immersive storytelling for Royal Holloway University of London. And that's then how I met Nonny.
[00:07:49.134] Nonny de la Pena: Mary and I had this wonderful lunch where I was like, okay, you know, ready to sit Mary down and see what kind of negotiations I could do. And I was like, so Mary, I have this open position that I really think might be great to have. And she went, okay. I was like, hang on, we haven't gotten our appetizers yet. So it was kind of a match made in heaven. I've been really incredibly lucky to have Mary help me take this thing and build out this program.
[00:08:15.987] Mary Matheson: I mean, look, Nonny is a mentor for all of us, you know, and she was somebody as a woman coming into this industry that I really looked up to. So I always say when Nonny comes knocking on your door, you don't say no. Right. But I think there have been some really unexpected Gosh, we're just very close in our outlook on the passion that we have about spreading the word of XR and the immersive industry. But essentially, who's actually getting to create and distribute that? So, you know, with the students now, it's really interesting just going back to like the difference between even 2020 and now. students that are coming in they know what VR is on the whole they know what AR is when I was first teaching I had to literally explain what each of these different techniques that we might use now people know what that is some of them might have some game engine experience as well some of them probably most of them will have actually watched some VR that was not the case only four years ago so that is incredibly exciting I think the unexpected nature of and the unity that Nonny and I have about this passion for changing the industry or making sure that it's an industry that grows in a very different way is super, super important. But also, frankly, I find working with Nonny incredibly inspirational in terms of her technological experimentation and research. So again, you know, answering your question about, you know, the technology itself and how that's changed my own practice. You know, I came from film, so 360 filmmaking was a really natural place for me to experiment, and I loved it, I really did. I worked on the Female Planet Google series, making five different films for them using the Yi Halo, beautiful camera, really fantastic cinematography. But I was probably a little bit more frightened to experiment with game engines and other technologies. So my latest film, a documentary about Russian war crimes in Ukraine, I have used Gaussian splats. And I would not have done that had Nonny not said, oh, come on, let's just go and play with this and let's set this up and play with it. And that's super exciting to be around as a practicing director, but also for the students as well. Like when you have somebody who's actually setting that precedence and sort of opening up the space and saying, let's just play an experiment, suddenly they all have the confidence to sort of be playing with that. So, you know, this is something incredibly new and our students are coming out knowing how to create Gaussian splats basically.
[00:10:47.290] Kent Bye: I wanted to ask around distribution, just because this is an area where we've had film festival circuits that were showing a lot of these different pieces, but then we had a lot of the issues of where's the path after that? There weren't a lot of options. I know, Nani, you had released one of your experiences on Steam, but then that's a gaming platform and not necessarily always the right context for these different types of immersive stories. For a long time, Meta had very strict curation, where they weren't letting a lot of those stories onto the app. But now we're perhaps moving into a new phase, where we have much more open with the App Lab now even getting integrated into everything, where it should be more democratized, more of these experiences that maybe have shown at the Film Festival Circuit, are even available to watch for students, but also to have the distribution outlet. And also the Apple Vision Pro with all these new channels for that ecosystem and that higher fidelity and then YouTube VR. So I'm just curious to hear where you see we're at right now based upon many years of struggle of trying to figure out Not necessarily that we've even figured it out yet, but at least it feels like now we're in a much better place perhaps than we have in the past and how that impacts the type of work that you do as both creators, but also as teaching students and even having content available for them to watch and see.
[00:11:59.212] Nonny de la Pena: I think anybody who's going to listen to you understands that we can see where this is going. We can see that this is so clearly the future of the way content is going to be experienced. And our vision unfortunately has had to wait for the technological capability to catch up. And it's clear that Multiple industry players recognize what we saw quite a few years ago. And it's just a matter of time now. You see an acceleration. You see people going, oh, Apple's trying to get into this. Oh, there really is some room. And we don't even know who's out there and who's making some cool stuff. I mean, you probably know better than I do, even some of the glasses that are being made around. But it's pretty clear that, you know, anybody growing up is used to having a spatial experience with games, to have a digital representation of themselves. All these things are kind of normal. And I think for their general sense of how do they deal with content experience. So it's just really happening faster and faster. You know, we've seen some success stories with subscription in this space. They're not that many of them yet, but clearly it's a path now. And I think it's only going to get bigger and better.
[00:13:22.293] Mary Matheson: Absolutely. And also, I think I've really noticed something recently, which is that creators are really starting to make money, you know, not just with games or with apps that are very specific around meditation or fitness. So if you think about Goliath and Impulse that have come out of Anagram in the UK, they are now making money from their experiences. East City Films, who have produced In Pursuit of Repetitive Beats. That's doing an incredible tour in the UK now. I don't think they're making a lot of money from that particular piece, but the learnings that they're getting from the distribution is honestly fantastic. And I think it's kind of interesting for us living in LA as well, because there feels like there's a downturn, obviously, in the film industry itself, whereas I feel like our industry is really... growing and developing. And so when people come to L.A. and they're part of the traditional film industry and they're talking about how depressed it is or how difficult it is to get jobs, I'm thinking, well, that's just not the experience of what we're seeing.
[00:14:25.757] Nonny de la Pena: Our students are landing really amazing jobs and obviously having huge success in their own right. like Cameron Christopoulos, right? That was great to be able to support somebody like that, like, in every which way. I had to support Cameron, his project, his tuition, his da-da-da, and to see the success, this is super exciting. But I also want to just point out, we're all at Connect. My entire team, one team member was sick, but I was like, we all need to go this time. and i think that that shows the sense that okay everybody's going are you coming are you coming i'm going i'm going i'm going it just feels like people are really back in and um i really see that we're again you know the energy here is just not going away it is not going away and at some point at some point it's just gonna break through and accelerate
[00:15:14.943] Kent Bye: Yeah, I was just in LA for the Snap Partner Summit and Snap Balloon Fest, and had a chance to drop by Cosm, which is kind of like a mini sphere. And then there's a sphere in Las Vegas. Also at Venice Immersive this year, they had a piece that was around the impressionist 1874, but they had a missive where they have people going, like 100 people per hour being able to see these experiences. you know, Horizon of Kofu, being able to show it to 80 to 170,000 people in some of these different places. So there's an incredible success of the location-based entertainment with these different types of immersive stories. So it just feels like it's hitting a bit of an inflection point of finding the theater of VR, which is like how to have enough throughput, you know, how to get like these experiences that have been created, but something like Cosm being able to show like what's a shared reality experience, but essentially like a 360 video, but with 1800 people being able to see it all at the same time.
[00:16:05.913] Nonny de la Pena: and sitting next to each other in a cozy way. I mean, yeah, Cosm's great.
[00:16:10.295] Mary Matheson: I think also, even though, I mean, it's kind of interesting, we're still quite a small group, like we all know each other, even though we might be creators in all kinds of different ways. So obviously Nonny and I come from journalism and documentary filmmaking and, you know, are using VR mostly, although Nonny's just made something in AR. for our practice, right? But actually, we still know the people at Cosm and we're rooting for them and going to support them. They come and support us at ASU.
[00:16:33.825] Nonny de la Pena: And it's the same basic technologies. It's all the same basic technologies. Everything you've said and you've talked about, it is the basic same technologies that we're trying to teach now to create this new generation of creators. And if I can say one other thing, you know that I've been, my God, I started pitching in 2016, Reach, my WebXR button-based platform, you know, and it's still out there, it's still out there. But, like, tomorrow there's a whole session on WebXR development. Like, I've been saying WebXR for eight years. And yet tomorrow here at Connect, they're devoting two and a half hours to WebXR development. Like, that is a frictionless distribution path. And, you know, you don't even have to get big ticket, big rooms, da-da-da-da. So I'm seeing that, too. Like, that's also happening right now. And so, you know, for eight years I was yelling that to the mountaintops. To see that happening now is also super exciting for me.
[00:17:25.979] Kent Bye: And with Vision OS 2.0, Safari is now shipping with WebXR by default. So we have a big green light for everybody to actually start shipping WebXR experiences, which is really exciting. So hoping to do a bit of a deep dive into different WebXR stuff that I've been holding off until we finally have all the different major operating systems and browsers, at least one for the Vision OS shipping with it.
[00:17:45.503] Nonny de la Pena: If you want to talk reach, I mean, talk about bootstrapped. It's super exciting to know that I was trying to make my bootstrapped WebXR platform, reach.love, work. And you know, it's free and you can still go play with it. So if you're just trying to understand what does WebXR mean, if you want to go and just poke some buttons, I'm still paying your storage. So you can still play in it. I don't know for how long, but in the meantime, we still use it with our students because it's a really easy tool to understand, oh, that's what spatial content is. Now I understand if I put something here, put something there, And it just gives people an idea, you know, a little sound there. It's a way to, like baby steps, to start to understand how do you create spatial content.
[00:18:25.211] Kent Bye: As we start to wrap up, I just want to get a few closing thoughts, but also just like, you know, it's a new semester that's usually starting now. Do you have like the next cohort of students coming through and how's that been going so far?
[00:18:35.661] Mary Matheson: Yeah, we've got a great cohort. We have 18 students. So we've grown from having six in our first year, 12, our second, and now we have 18. They are coming from all kinds of backgrounds, super exciting, various ages, some coming from film, TV, theatre, music. game engine development and they're really keen and they're already getting started making their work so the future is bright really for the industry as well as our masters.
[00:19:00.419] Nonny de la Pena: The cool part about our masters is if you want to you can wrap it up in a year but if you want to come back in the fall to get your capstone done you can so all 12 of our previous students came back so actually we have 30 students right now and some of them are like oh can I please have an incomplete because I want to keep working So that is extraordinary. Nobody wants to leave us. And I think that that is unique, probably. So we've got 30 students. We grew from six to 30 in just really two years. So that also is an indicator of the desire people have to not only learn this stuff, but how excited they are that they don't want to leave the incredible resources we have. As I said, I've been able to, I mean, you know, I really believe In what I'm doing, I really believe in trying to help this new generation. And I think because of that, perhaps, I've been very fortunate to be able to raise the kind of money I've been able to raise. So I really can provide scholarships to anybody from any economic background to come and participate in this growing field so that we don't have the kind of monoculture that we've had in early tech days. So it's been extraordinary. You know, you'd think that they would hate us in some way or another, but they keep coming.
[00:20:14.662] Kent Bye: Great. And finally, what do you each think is the ultimate potential for spatial computing and immersive storytelling and what it might be able to enable?
[00:20:23.106] Mary Matheson: I mean, it's here. We always talk about spatial storytelling as being the future, but it's here. It's happening. And it's just incomparable. When I now, I was talking to one of my students, actually, one of the new students, they've literally been with us for four weeks and they already said, actually, I feel like I can't go back to traditional film or TV even to watch now because the experience is just so different. It's transformational.
[00:20:45.336] Nonny de la Pena: They were making them watch a lot of different stuff. So they get a deep feeling of what it means to embrace spatial computing as a medium, to think through narrative storytelling. And we've had this conversation. We do not just experience the world through sight or sound. We experience the world literally through our entire bodies, right? We feel where we are. We feel stories. And I think that that's one of the things about spatial computing and we're trying to build is we're much closer to that sensation. And that means that you have to think differently. But all you really, really have to do is close your eyes, feel your body in space and go, oh, I understand that the way that story happens all around me in the physical world, that's what I'm trying to produce using spatial computing and these new technologies. And, you know, we've narrowed that gap a little bit through the medium that we're all embracing.
[00:21:42.639] Kent Bye: Is there anything else that's left unsaid? Any final thoughts that you'd like to share to the immersive community?
[00:21:48.314] Mary Matheson: Yeah, sure. I mean, gosh. Well, there's two things. Firstly, if there's anybody they know who would love to come on the Masters, then they should talk to us, myself or Nonny, through Arizona State University. You can find us on all the social media. But equally, people in industry. We're really interested in collaborations. We have great labs and great studios. We'd love you to come down and experiment and share your experiments with our students and research. We have a captury stage. We've got a Gaussian splat stage that we're just setting up right now. VP stage so we've got an amazing location basically in facilities so we've got great ties with other industry professionals but also artists that we've been working with so if anyone is interested in working alongside us or our students then they should give us a shout.
[00:22:33.834] Nonny de la Pena: So the program at Arizona State University is a narrative and emerging media program. And the final note on that is the work that we're trying to do there, we want to, like Reach.love, we want to open source it so that anybody who's working in Gaussian Splat or interested in Gaussian Splat 4D, meaning time, meaning video as dimensional content, we're hoping we're going to be able to open source whatever we build so that we also grow the industry that way. And I think that was always something I tried to do at Emblematic Group was also help lead the way, but also provide whatever we had to share and grow this whole industry together.
[00:23:09.723] Kent Bye: Awesome. Well, Nani and Mary, thanks so much for joining me today. I know you've both been helping to move forward what's happening in the whole industry and really helped to build the future of where this is going for the next generation of talent that's coming up. And yeah, it's just really great to have a chance to sit down and hear a little bit more about the story of the narrative and emerging media program there at Arizona State University. So thanks again for joining me here on the podcast. Thank
[00:23:31.889] Nonny de la Pena: Thank you. Thanks, Kent. Thank you so much, Kent, for your interest always. Thank you.
[00:23:35.790] Kent Bye: Thanks again for listening to the Voices of VR podcast. And I would like to invite you to join me on my Patreon. I've been doing the Voices of VR for over 10 years. And it's always been a little bit more of like a weird art project. I think of myself as like a knowledge artist. So I'm much more of an artist than a business person. But at the end of the day, I need to make this more of a sustainable venture. Just $5 or $10 a month would make a really big difference. I'm trying to reach $2,000 a month or $3,000 a month right now. I'm at $1,000 a month, which means that's my primary income. And I just need to get it to a sustainable level just to even continue this oral history art project that I've been doing for the last decade. And if you find value in it, then please do consider joining me on the Patreon at patreon.com slash voices of VR. Thanks for listening.
