I interviewed Chief co-director Catherine Eng remotely after the SXSW XR Experience 2024. See more context in the rough transcript below.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. So continuing on my series of looking at different immersive stories at South by Southwest 2024, Today's episode is with a piece called Chief. This is a piece by a company called ourworld.io, which I had a chance to actually previously speak to the co-founders back in episode 1060 with the piece that was shown at South by Southwest 2022 called Choctaw Code Talkers, 1918. And so they had another piece this year that was actually being shown on the Apple Vision Pro. This is a piece that they put together very quickly and a little bit more of a proof of concept demo to give a little bit of a teaser for some of these larger stories that they're going to be telling. And so from their synopsis, they say there are over 575 sovereign tribal nations in the United States today, each governed by a chief chairman or chairwoman. Meet dynamic tribal leaders in this XR 360-degree experience developed for the Apple Vision Pro, from interviews with powerful chairwomen to one of the youngest chairwomen ever elected to an elder reflecting upon his time in tribal leadership. Experience unique perspectives on the history of this country and the meaning of culture, family, and community. community. So they're starting to use the affordances of 360 video, volumetric capture, piecing it all together with archival photos and videos and trying to create this multimodal experience to be able to tell these Native American stories. And they've got an application that they're in the process of developing and releasing sometime in the spring or summer. And they're hoping to create more of a scalable approach to tell some of these different stories. And so they're trying to not only explore the affordances of the technology and the spatial medium, but also to figure out how do they scale it out to be able to tell more and more of these stories as well. So my previous interview, I had a chance to talk to both Kilma Lattin and Catherine Ng, but in this conversation, Kilma was not available. And so I had a chance to talk to the CTO, Catherine Ng, about some of their updates from what they've been working on from the last time we spoke, but also some of their plans of where they hope to take this project here in the future. So that's what we're covering on today's episode of the West is Vera podcast. So this interview with Catherine happened on Monday, March 25th, 2024. So with that, let's go ahead and dive right in.
[00:02:22.992] Catherine Eng: I'm Catherine Ng and I am the CTO and co-founder of Our Worlds. And we got started with this interactive XR journey after Kilman and I met at our STEM camp. I had a STEM camp in 2017 and we got a Kickstarter and we funded this group of 220 kids to come have free scholarships to learn how to code games. And Kilmer brought his son and we met there and he had created this app called Payone and it was based on a tribal hand game. And he had this idea that he wanted to extend it and do better. He knew that I'd made apps for Disney and for Nickelodeon. And so I took a look at the app and I was like, oh, this is great. It was very playable. Like you had this repeat play. sort of feeling. Is this a track loop? And you just kind of wanted to keep going. But I wanted to see what the real game was like. And I asked him if it was possible to see one. And he invited me to the Santa Isabel Reservation. I was going to have a live Paean game the next night. And so, I went with my son, drove out to the reservation, took about an hour in the dark. And when we got out there, it was just this like magical experience. First of all, we were very disoriented. But when we got out of the car, there were just kind of like these little dots of fire on the horizon and like, you know, the sound of people, you know, laughing and singing. And as we got closer, we saw how the game was played. And you know, it's a complicated game. It's a game of dexterity. You are basically singing these traditional songs, and you're also doing something very complicated with the peyons. It's bones tied with leather. By far, the people who were delivering the songs and the stories with the most aplomb were the elders. It was amazing to watch. And it was just this delight. And I asked him, you know, really, we really should be talking about immersive reality. Like, how do you preserve the experience of sitting with these elders and hearing these songs the way they sing them? You could record the words, you know, on paper. but it wouldn't even be the same thing at all. So, we just began on this journey. I began to learn a little bit more about XR. I'd made apps before, but not really immersive or VR or anything like that. I went to the Reality Hacks Hackathon in Boston that fall, in fall 2017. The team that I was working with won the Ricoh Theta Award. It was really interesting. They were the geniuses who came up with this idea of, well, the Ricoh Theta is of course a 360 camera. And it has 360, it's monoscopic still though, right? So they had come up with a way to, you know, really kind of extrapolate distance depth maps out of that. And this is back in 2017. And so they did this like in the course of, you know, 24 hours. And that really was so inspiring that, you know, I basically set out to learn everything I could, you know, about how to produce art and graphics. I mean, I'm a fine art major from Cooper Union. And so everything that I know about development and like really code and the video, I learned really on my own. I don't have any formal training in development. I've given some advice. I've mentored some young artists who want to develop and make things in XR. The technology can feel daunting at first, but if you have a vision of what you want to make, you'll figure it out. You just have to have opinions about what you want it to look like. And then instinctively which technologies to seek out. So that's really my journey. Very long way of answering your question.
[00:05:50.779] Kent Bye: Nice. Yeah, that helps provide a lot of context. And I also like to give people an opportunity to give a bit more context as to your formal training, your background and your journey into VR. You already gave a little bit of that journey, but just wondering if what kind of disciplines that you're really bringing in into your practice.
[00:06:06.515] Catherine Eng: Yeah, so for some reason I have this title of CTO at RWorlds, but it's just so much more than that. I just feel like you really couldn't do any of that unless you had some solid visual idea about what you wanted to do. So I went to art school thinking I was going to be a painter. I mean, I still have this weird ability to draw, and it's actually kind of interesting now with all this AI and mid-journey and all of this. It's like, And, you know, I have friends who are still, you know, very much in the arts school, like, you know, they're deans of some of the best schools in the country. And, you know, they ask themselves, it's like, what is this ability to draw? Like, where does that fit in anymore? Like, you know, I got into Cooper Union based on my ability to draw something very realistically or to use color. And I would argue that those kinds of expressions, those personal expressions, are very important. It's frankly very weird to me that people would want to hand off that kind of expertise to a machine. If you're good at anything at all, like if you have a talent, or if you are good at ice skating, or if you're good at cooking food, why would you give that up just so a machine could do it better? I mean, there's a level of satisfaction, and there's a level of expertise, and there's something that really feeds into your quest as a person, as a human, to kind of keep evolving and keep growing. I mean, that's really, I think, very much the human condition, right? It's just like, how do we, you know, what's the point of being here? Like, how do we learn? Like, how do we learn more? Like, how do we do things better? So, and I know that the tools really aren't meant to be the end of the conversation as far as creativity goes. I mean, that's, I think that there are a lot of really cool uses that you could come up with. But, you know, certainly, you know, you talk to like a younger person, you know, like, Some high school students are like, oh, just get AI to do it. And you know, like, oh, self-driving cars, we love self-driving cars. It's like, yeah, then we could just not have to look at the road and we could just read or be on our phones. And it's just like, wow. And so that's a very different mentality than I think maybe people from a generation where those things weren't a given are arriving at. So it'll be interesting to see. And I'm talking about high school students who are just like, yeah, let's let AI do everything. But I do see that kids that I'm talking to in college level and young adults, they're going back to the film cameras. They're going back to dumb phones. They don't want the distractions because they actually now are feeling the pressure of the real world encroach and they know that they have you know, some responsibilities that they're not being forced to do it by any means. They can do anything they want, but I mean, they want to meet these responsibilities. So they are looking for ways to remove distractions from their environment. And it'll be interesting to see how it all shakes out. So in terms of my journey or where this all comes from, it's really very fine art and visual based. And so, what do you do when you go to a fine arts school? I mean, you go and you sort of have this grand vision of things. That's what you go for. I mean, you do practice your craft a little bit and there's drawing classes and there's like color theory and there's sculpting classes and like Cooper Union was great in that the first year, it's like you have a quarter of like painting and drawing, another quarter of like actual sculpting and just like forms and you learn woodworking and welding and metalwork and all these things. So, you get kind of like a a level playing field of, let's just like try all the materials out and see which one sticks kind of a thing. And the whole time you're being asked to like, let's look at, it's kind of daunting. I mean, if you don't have any context for what the work is that you're going to make, it's hard. you know, coming from like a small town on Long Island and, you know, high school, you kind of like discover the artists you like. It was very tame stuff like, you know, Monet or like, you know, just like the Impressionists and so on. It's like, wow, it's so beautiful. I want to be a painter. It's very innocent and cute. But then, you know, you're thrown into art school and you're being asked to make conceptual art. And what's the concept? You're reacting to bad politics. Or at least that's what the call was back in my day. It was like, well, surely you don't want to make beautiful art objects that are going to be bought by wealthy collectors whose wealth the provenance of no one really knows, right? And the only appropriate answer to that, I mean, you're just 18. You don't know anything. You're like, yes, that seems wrong. There go your dreams of spending your days mastering the perfect drawing or painting. It's like, no, no, no, don't do that. That's a waste of time. You have this great graphic design ability. Make posters about the tobacco industry. Make posters about big oil. So like everyone came out being graphic designers like from art school, you know, I mean, of course, that's where the money is. I mean, that's not a huge surprise or like, you know, doing web design or whatever. But, um, I don't know, like, I just think it's fantastic. All my friends who stayed, you know, sort of like making paintings and like thinking that through because like, yes, there's the gallery world and the art world. And that's like really kind of weird and transactional and not really what art should be about. I mean, art should be you know, just pure poetics. I mean, everyone's like searching for authenticity, right? And so like the authenticity that, you know, one might have thought was there in art really isn't once you get the economics of it involved, right? So then you begin to think about how you can express just like this will to design and make nice things, or just like to design things. Like, how do you do that for good? Like, what are the venues? And so I feel like this has been an incredibly special project in that sense. And I feel really fortunate to have met Kilma. And it was weird because it was kind of like destiny because we both, you know, like ended up having the same like rare, it's called Games of the North American Indian, which is a really like weirdly anthropological title, you know, but it's a very obscure book, and we both had it, you know, this copy of it. And so anyway, you know, my background isn't you know, of course, I'm not Native American, but you know, I speak, you know, Yue Chinese or Yi Chinese, which is Cantonese Chinese. And it's really different the way it's spoken than the way it's written, you know, and there's an element of the spoken, like everyday Cantonese that's never going to be written down on paper or like printed. So there's like an oral tradition there, in a sense, one could say. that doesn't really exist the way... You could write out exactly what you're going to say, any slang, whatever, in English. But if we're going to write anything that we would say or communicate intelligently to anybody on paper, all the words would be different. The sentence structure would be different. their words that if you actually use them in regular everyday speech, they'd be like, why are you so formal? So it's really a hybrid. It's a different situation than what's happening with a lot of the language. But then it was something that I intrinsically understood. Like, oh yeah, I do also have this existential something, I feel, about languages just in general. And what is happening with Indigenous language revitalization really interests me, and I can relate to that. So this idea that we were going to recreate the experience of sitting with the elders and having their knowledge transmitted in these environments, like over a fire at night or over just a game, That's definitely part of it. But how do you use these technologies to deliver the experiences in the place where the languages were born? What does the environment contribute to the evolution of that language? That's one component of what we're doing. We have the geolocation piece, we're working on a language revitalization piece. Then of course, you know, wherever we can, it's really nice to be able to record. And, you know, Kilma has this incredible network of people just from the community and also from his time in tribal leadership. You know, one of the things we've always wanted to do, like, how do you create a state of the nation of leadership in Native American communities like right now? It would be so cool. Like, you have the State of the Union address with the president, but like, what if you had a State of the Union with all 575 chiefs? And then, you know, if you're going to do that, what's the medium? And if you do think about, you know, we have iPhones. I mean, these are, you know, at any time you, there's that phrase, there's that expression, like the best camera for what you're shooting is the camera you have with you, right? So there's all these different ways. So fast forward to, you know, what Chiefs ended up becoming. I mean, like we have some stories about Chiefs on the iPhone app right now. You know, I don't know if you had a chance to see them,
[00:14:41.605] Kent Bye: I saw the Apple vision pro app, but I didn't download the iOS app.
[00:14:45.866] Catherine Eng: Oh, okay. Yeah. The iOS app is, I mean, Apple has been very nice to us. So like they featured as this app of the day this past November and we got a lot of downloads there. And then when we first came out exactly a year before in November, cause it was in beta for a long time for like a year and a half. And then it launched November, 2022 and it was featured as best new app. apps and updates on a list of 20 for like a month, and we got tons of downloads. And then app of the day just this past November, which is really nice. But yeah, the platform is really solid. We just need more content. And so that's why when we did have this opportunity to record some of the woman chiefs who are in Kilmer Circle, we just jumped at the chance. But then it's like, We have big plans for this. What is the way that you would record a conversation with someone who is really special to the community? It could be just like a giant movie. But the thing that we wanted to do always was like, wow, it'd be so amazing to kind of... We're not doing anything weird, like turning them into some sort of game object or anything, but if you just capture them volumetrically and you just sort of have this slightly larger than life presence of them sitting with you, and you have that for posterity, just generations down and just keep evolving it to the next and the next platform so that it would work. If it's one day going to be just like glasses or something. you know, there's a whole ecosystem that one could imagine, like it's not even science fiction, but you could just say, yeah, I mean, why would not these types of works be preserved, for example, the way, you know, museums, for example, get grants every year to preserve their pottery collections or their art collections in like air conditioned rooms and so on. And these types of artifacts are very precious, of course. But then I think that these primary source recordings It's much different hearing someone and the sort of energy with which they deliver their narrative and how they deliver their hand gestures and the way they looked. It just stays with you, even more than a movie. What we made is somewhere... We're trying to reinvent how you would do a documentary. How would you do a documentary for Immersive to innovate on that platform? There's some really good documentaries out there. But what's special about the Vision Pro, we thought, is that it's a really nice place to be. When the iPhone first came out, my consultancy that I ran with my husband, we were like, oh, the iPhone, perfect. we'll put Mattel's LED football. Remember that little plastic game where your quarterback was a little red dot and you just went like this? So we made that app and it did really well. We got like 10,000 downloads a day. People were writing to us because we let the cat out of the bag before we ended up publicizing, before it was even in the store. We were publicizing it and people were like, oh, I really love that game. just someone cryogenically freeze me, please, until this game comes out. I can't wait. I can't wait. Because, you know, the idea is like, oh, that makes sense to people. This is about the size of, you know, one of those games, you know, from back in the day, the people who are going to be able to afford to buy these phones now would have a special connection to that. So people will love the app. People will download the app. Perfect. form factor and just place and time. Then when the iPad came out, we had small children who were learning music, and there's little air harps or lap harps that people use with the music that you slide under, and there's little wires going across, and you can pluck the strings. That was about the same size as an iPad. It's like, oh, perfect! That's the form factor. And that did really well. I think it debuted at like number five or something. And then with the Vision Pro, it was like, wait, what's this? It's like, what's the form factor here? We just kind of I mean, speaking just strictly for myself, me and the rest of the team had, of course, very strong ideas also. But I was really stumped actually, because of course, we know about the Quest and so on, but so much of what we'd done for our worlds was always going to be like, how do you take immersive experiences, AR, XR, all of that stuff, Kent, you can get on our iPhone, because we didn't want people to be tethered to these devices. It's very exclusive, the expense of it and the infrastructure required, et cetera, et cetera. So like you can do all that, like all the movies, like you can even like turn sideways and there's stereo and you can put them in Google Cardboard and get like a cheap effective VR that way. And that's always something we wanted to do. But I would love to hear your thoughts about the Vision Pro because I thought it was just so evanescently beautiful. And especially, you know, like the mindfulness app environments, there's something really unique about the mood and the sense of place that they create. So we wanted to do something like that. And we had, of course, some nice conversations with some engineers about how to achieve those things. But largely we were on our own. So we went out and got a Titan camera and we recorded many more than what you see in Chiefs now. But these 360 stereo recordings of places that were really special to the community. Not sacred places. We have to be really careful about what we record. We only want to record if we've been given permission. We want to be super respectful of those kinds of things. But there's so many beautiful places. We've recorded a ton already, but then we were using 6K. And we're like, darn, like, can we upscale all this footage that we have from the different places that we've already recorded, like, you know, at the ocean and, you know, like inland, you know, Apollo Mountain, for example, like, you know, drone flights. No, it just does not work. Does not work. You know, so you really, and I think by now, like we all know how, I mean, it was on X, you know, people kind of like, kind of deconstructed or reverse engineered how Environments was made. Did you see that post on X?
[00:20:37.727] Kent Bye: No, I didn't.
[00:20:39.388] Catherine Eng: It's so cool. So, you know, environments, it's like, wow, how did they do this? It's like such big sky, right? The sky is really high up. And if you do your Titan, or even if you have like, you know, your own home brewed, like 360 camera array, it doesn't feel that big. It's a model. Because of that, you can really make things look essentially wide angle. You're able to feel like the sky is way, way up there. The horizon is way, way out there. You can do these things that the camera just doesn't do. Of course, Apple has a lot of money. and a lot of resources. So they're able to do these things, but still we're trying a few workflows, you know, with like, how do you make this? But like, nonetheless, I mean, you know, we know we can deliver some really powerful experiences that don't have to have that kind of resolution. I mean, who doesn't appreciate like grain on a black and white photo, right? There's some, there must be some equivalent of that in this device, which is otherwise just so resolution hungry. you know, like you really have to, like 8K is just like, ideally you'd be working in like 16K and then you would compress it down in such a way that it looked really exceptional in 8K. I mean, we've, we've had some success there, but yeah, I mean, everything takes time. Everything takes some doing, but all that to say, we're really excited about being able to, you know, sort of crack the code there. And we're just like this weird ambition about it. We're getting there. I'd love to hear your thoughts about what you took away from Chief.
[00:22:09.835] Kent Bye: Yeah, so I had a chance to test out the test flight. It's actually a theme that I saw this year in a number of different pieces, which is more and more of the documentary filmmakers starting to take this multimodal approach of trying to integrate photos and videos. You have some volumetric interviews that you're including as well. It feels like it's the very beginnings of trying to capture these stories and these oral histories. And, you know, the resolution looks amazing. Really great introductory scene with a kind of holographic point cloud-esque CGI eagle that's flying around with particle effects that are following it. And yeah, you kind of go in and watch these like three little vignettes of indigenous women chiefs and also just indigenous members that are talking about some of their experiences of what it was like to grow up on the reservation. And it feels like the very beginning of much broader stories that felt like almost like a sneak preview rather than something that's really diving deep into one specific story, but giving a little bit of a sampling of what you can do with mixing the modalities of having like a real high resolution drone shot that you're flying around and then taking me into these different stereoscopic 360 video scenes and locations of the earth and then have people talk about these oral history captures where you have a volumetric capture of them, but then they are then referencing different photos that are then coming up. So you're able to really mash together, adding additional context with these different media elements and a really amazing moment where you turn to the night sky and see the stars in a way that is really truly epic on the Apple Vision Pro to see That level of resolution of seeing either a photo or CGI depiction of the night sky was really quite effective as well. Breaking out of what is normally a 2D frame, or at least you're still feeling like you're talking to someone, but to really transport people into these other spaces. Yeah, it feels like an opportunity to really play with how can you use the practice of oral history and doing these interviews and start to capture them in a way that with the Apple Vision Pro, there's new opportunities for distribution. Because I think one of the biggest challenges has been that these types of projects were, you know, unless it was at a certain level of stories, like, you know, of all the different stories that we're showing at, like, Sunday in South by Southwest Tribeca or you know, Venice immersive or doc lab, you know There's dozens of different projects and hundreds of projects created each year But only a small very very small percentage of that were allowed on the official meta store And so some of them would end up on that blab, but with Apple Vision Pro it seems like hey they can just you know, you can publish this app and potentially just make it available for folks that maybe there's a iPad and iOS version, but then also have these more immersive experiences with the Apple vision pro since it's still very early and niche in terms of the audience that are going to be actually have access to see it. I was actually quite surprised that you were, you know, just a kind of six weeks out from the launch of the Apple vision pro. And then you're doing some of the first public exhibitions at South by Southwest. I was expecting it to take a lot longer, but, uh,
[00:25:17.686] Catherine Eng: See that you were even able to put together this experience in that short amount of time and be able to show it there at South by Southwest I mean yeah, it was like why do creators do the things that they do I mean you know like people who make this stuff are a little bit crazy and And I'm like, I mean, you have to have that, you know, sort of ferocity to want to do this thing, like hell or high water. So we're definitely like affected by that bug a little bit. And, you know, and we're still at it. I mean, you know, we still have like new cuts that we're doing all the time and other interviews that we're going to be doing to kind of like round out. Yeah. I mean, just like the amount of, I guess it felt like part filmmaking, a little bit like filmmaking, but more like a hackathon. So it's like this 24-hour charrette thing over however many weeks. But that's how you have to do it with all of the technology stuff. I mean, it's definitely hard to do in this sort of like what people consider a work-life balance kind of way. You do it for eight hours and you're just like, hey, it's quitting time. You can, I think, at different phases of the project. I mean, that would be really nice because obviously sleeping and eating are important. But I feel like we just had so much to learn and play and just do stuff that was you know, really an important part of the process. It's like, you know, we had three different types of interview footage, you know, we recorded it with Depthkit, of course, like our friends at Scatter, you know, have that great, very affordable, but still yields amazing results, like the volumetric recording. So we recorded some of the chiefs that way. And then, you know, when you do that, you have both the high resolution, you know, cinema footage that's in 6K, and you also have like the depth map, which is like, you know, lower resolution. And, you know, we're really exactly sure how to make that look really, really fantastic on the Vision Pro. We're still working on it. We have some ideas. And then, of course, the killer app part of... I mean, there's so many killer apps surrounding the Vision Pro in general, but this idea of spatial videos. We're like, oh, let's record some of the interviews in spatial video and see how that shakes out. And so we did that. I don't know if you can tell, but watching it, it was probably really obvious, actually. Also, there's a new technology that's come out and we've been talking with them. They're very friendly and nice. It's called HaloDepth. Have you heard about these guys? Zachary Hancho and I think his name is Michael Butterfield. They're these two young entrepreneurs, I think they're just like a year or two out of like University of Texas, Austin. And they've created this online service where you could just take any video and just make it stereo, you know, which is fantastic. I mean, they're working with like Universal and Zeiss and so on. So very smart guys. But yeah, I mean, if you can do that, then why do this? Why do the volumetric? Because the workflow is just so insane. But then if you really think about doing volumetric 360, so you can actually walk around the person perhaps one day, that is probably the way to do it. Because it's like the security of knowing that you have this asset and it's not a trick. It's truly a three-dimensional portrait. You know what I mean? Like the stereo is a little bit of a trick, right? So it's just kind of like an optical illusion that you can just sort of like a view master, like you can see like left and right. Oh yeah, it feels like it's really there. And cognitively it does something to, I mean, there's this sense of delight when you see it. You know, even though it's just spatial, even though it's just stereo video, seeing it on that scale, seeing it on the Vision Pro with its beautiful optics, it's like, you get a charge from it. It makes you happy. So anyway, we recorded also one of the interviews. Talon, who's our AD, was wearing the Vision Pro the entire time. We also recorded the interview that way. When we first had our hands on the Vision Pro, we were like, this is how we have to interview each other. What we'll see of each other inside is the avatar. So we'll look normal, but then we'll interview and record on the outside. But then what ends up happening is, of course, unless you're in a brace, you know, like you're going to move your head a little bit. He stayed so still. I was like, I can't believe how still he's staying. But then even so, like when you look at that footage, it's just like, if you just do this, then all of a sudden it's just, oh, I can't look at that, you know? So there's definitely something, maybe, you know, that's not how you would do the recording. Maybe you do just use the iPhone 15. And then, you know, like the fact that you can record some stuff is great, but probably not for a long-form interview, you know?
[00:29:39.168] Kent Bye: Yeah, I think when I was watching the piece, one of the things I was thinking about was that, well, it is nice to see these archival photos being in the Apple Vision Pro and having that such a high resolution. I was like, oh, I wonder if some of these 2D shots, even though they're archival, could they be, have the processing to have the stereoscopic effects? You know, some of the stuff that Targo Stories has been doing with JFK Memento and 911 documentary that they did, you know, different techniques to add the spatialization even if the original photos don't have it because there is something about being in the apple vision pro where you almost want to have as much of the spatial effects as you possibly can but of course there's the question of the integrity of the original archival photo and how much are you digitally altering or adding stuff that may or may not have been there with the occlusion and so yeah there's the ethics of how much do you start to digitally recreate some of these different stuff but that was some of my thoughts also as I was watching through it is that it was remarkable for one how well just even the 2d footage look just because it was super high resolution and you're able to really give this, like I said, multimodal approach where you're showing both photos and videos and volumetric shots, stereoscopic 380 shots, and some depth kit or other biometric capture techniques for the interviews. And so, yeah, I think it's kind of like a very experimental of trying to mix and mash all these things together and see how it coheres in a way that's able to still tell this larger story that you're trying to bring in all these different relational dynamics as you possibly can, as you're helping to elaborate their stories.
[00:31:09.510] Catherine Eng: Yeah, it's really important to us to have sort of manageable and sustainable workflows because ultimately we don't really want to be the storytellers. We're always going to tell some of the stories, but essentially the platform is created so that people in the community would be able to extend their storytelling traditions. And we've done some workshops at ACES, the American Indian Society of Engineers and Scientists. They meet every year and then also Dream the Impossible conference. These are technology conferences for indigenous higher education and career path youth who want to use technology to extend culture. How do we make it not impossible? you know, impossibly difficult and impossibly expensive to make. And so that's the idea is like, well, there's so much other media around, like there has to be a way to use it. Like we can't go out and re-record everything. And then what do you do with like all of the beautiful archival photos? We're just at the tip of the iceberg with what our vision is for the Vision Pro in terms of putting up all of these photographs and also maps and three-dimensional graphs and charts of the economics of the tribe and what it means to have... For example, Chairwoman Santos was talking about wind farms on her reservation. That's later on in the interviews that we'll update it with. But yeah, the economics of one of those towers, it's It's a little bit like The Little Prince, where it's basically this little planet and there's this one flower on it. You have this reservation, there's just a couple of wind towers, but it's enough. It does bring money to the tribe, but it also has some dangerous components to it. It's kind of interesting. All of those things really should be conveyed with data science. How do you make data science come to life inside the Vision Pro? That's very much on our radar of what we want to do with this. I mean, the sky's the limit. And then also, there's so many other forms. I think the spatial audio provides an opportunity. How cool would it be to sit... And we've been experimenting with different kind of very subtle, but wave fields that just buzz around you as you're watching or listening. you know, to like a video or like listening to music or something like this, you know, just like, kind of like that thing of like when you were little people, you close your eyes and then people would go like this over your head. Like, do you feel that? Do you feel like my hands, like the energy field? So a little bit, that type of thing, like, how do we make that happen? There's, there's so much you could do. So that's all to say, you know, really kind of like rethinking how we can make assets for this device make a space where people just really, really want to stay. Because when we first had our hands on the Vision Pro, I was like, wow, this device... I mean, I know people are like, oh, it gives me headaches or whatever. But to those people for whom there are no headaches, I mean, I could wear the device for hours. I just have no problem. I really like it. It's really exciting to think that, I mean, the price point has to come down, but if there was the ability, if you just back away from all of the sort of things of like, oh, it's heavy, it's expensive, it's this, it's that, what they've done is really remarkable. Just like this very sharp and it meets or exceeds your expectations of what colliding realities would feel like. They've done it. There's a persistence and an anchoring of objects and all of that is just so beautifully done.
[00:34:22.362] Kent Bye: Yeah, I just went back and listened to our interview that we did at South by Southwest 2022. And Kimmel was there when your co-founders of our worlds to share a lot of the origin story of how our worlds.io has come about. So when you were both showing the Choctaw Code Talkers 1918 at South by Southwest 2022, but also you participated in the South by Southwest EDU pitch session where you were named the winner of that competition. And then you had in our interview a couple of years ago, you had mentioned how. part of the things that you were really thinking about was how could you use these immersive technologies to start to capture some of this indigenous knowledge or wisdom from elders. And it seems like that's where you continue down that path of starting with chiefs, starting to capture some of these oral history interviews, looking at some of the governance issues or some of the larger political ecology dynamics of what's happening with Native Americans on these reservations. And so Yeah, maybe you could give a bit of an update from where you were at back when you were first pitching some of these ideas back in 2022 and what you were able to build out and continue to put out there and launch with the app and what you're bringing out this year with Chief at Southwest Southwest 2024.
[00:35:30.861] Catherine Eng: Yeah, thanks so much. You know, and I think that South by Southwest has just been this incredible platform for us and, you know, really just for so many creators as well. And, you know, what has come out of the film festival, yes, we won EDU, which is great. And that's certainly led to a number of really incredible conversations with schools and educators around the country. And even though Choctaw Code Talkers, you know, we were in the competition, you know, we didn't win, but so many good conversations came out of that. And actually now in, you know, sort of like some sort of subsets of Those museum exhibits are now at the World War I, the National World War I Museum in Missouri, in Kansas City. They ordered one. And, you know, Choctaw Cultural Center also ordered one. It's just kind of like a, sort of like a bridged version of the experience that we had at South by Southwest, but we ship them the headsets and they, you know, are able to play it. And, you know, there's some of the props and things, which is really cool to us. And then since then, we've actually, because it's three and a half minute long experience, we all know that in immersive realities, things just come to life in a way that A three and a half minutes in immersive reality is like, I don't know how many minutes of reading, right? Someone could just argue it one way or another, but more. You need more time to read in order to have things come to life as vividly. Perhaps it's like three and a half minutes on a headset. So fast forward to as of last year, we started talking with the state of New Mexico education department, public education department reached out to us and we've been working with them and we're basically going to start selling curriculum based on Choctaw Code Talkers 1918 this spring, which we're super excited about. We're just about to start some pilots with high schools here in California and some high schools in Missouri. And just really excited about the idea of like, okay, so you have this story that tells a narrative of 19 Choctaw Code Talkers from Oklahoma transplanted to Northeastern France. The conditions are terrible. We go into that a little bit, you kind of see the battlefield, you see people getting gassed and just running around and it's just like, oh my gosh, this is terrible. But if you just took that and unpacked everything, it's like, well, what was that mustard gas? What was the language that they spoke? What were the trenches? Where is this place? Why are there so many memorials in Northeastern France? I mean, if you have ever traveled to that part of the country, the world, or if you live there, there are cemeteries and memorials everywhere. It was a big, big, big part of the landscape. And then we have 360 recordings of those landscapes from our colleague. from Little Planet in Belgium, they recorded some 360 landscapes for us. And it's kind of like a virtual field trip. As we were creating Chalk Talk Code Talkers back in 2021, we didn't know how we were going to use it and we ended up using only a little bit of it. But now we have it and we're going to use it as the curriculum. So it's very exciting. High school students, when they see Chalk Talk Code Talkers, They like it. I think it helps snap things together really quickly. It's like, wow, the economics of having this many soldiers fight for this long. What does the map look like? How many people died? You can see a confluence of augmented reality objects. People can, with our curriculum, play with the field phone. It's very vivid and lifelike in augmented reality, which of course looks great on the iPhone or iPad, or we put on the desktop. And then also the immersive XR 360 field trips to these different places, to Oklahoma, to Northeastern France, to the battlefields back in the day, to what those fields look like now. It's just like farmland. So just really excited about where this could go. And we're selling the curriculum to New Mexico. We're in the depository. We've signed a contract with the state. And so that's huge. And what we always want to be able to do is really be very mindful of how we gather these stories. We had a really nice conversation with the great-grandson of one of the Code Talkers. Kevin Wilson's great-grandson is a friend of ours, D.G. Smalling, the artist D.G. Smalling. He gave us this really nice narrative. We always want to just give back to the community. We're going to be bringing this curriculum to life. We want to make sure that it's done in partnership with the community. One really interesting thing, once we won South by Southwest EDU, and then it's like, oh, well, the only logical next step is to try to get funded and be a proper ed tech company. And so we went out to make a case for that. And you find out all these crazy things about how 87% of the information that's taught about Native Americans in schools and high schools is pre-1900s context. 26 out of the 50 states don't name a single Native American in their history standards, which is crazy. And so, all of these things are, of course, not in line with the thriving communities that we know today. I mean, there's the National Congress of the American Indian, and there are 575 sovereign tribal nations in this country. It's hard to wrap your head around that. what that actually means. And some of the tribes do really well, and they have money from different resources, such as gaming and so on. And some don't have those resources. Unfortunately, a luck of the draw, if you will, of where their reservations were placed. Some of them are just sort of opportunity deserts. beautiful landscapes, but then it's hard to... So anyway, we see Chief as being a really, and we have about 12 stories on our production slate for other curriculum that we would want to write. But yeah, we just want to make history really come alive. I mean, you know, there are all these stats, you know, once you begin to unearth, it's also like, okay, other things like 69% of students would not be able to find Iraq for you on a map. which is terrible. And then some horrible greater than 50% of students wouldn't be able to place the Civil War in the proper 10 year period in the 1800s. And all this emphasis on STEM in the past 20 years has been great. My kids are a product of that, but it means that classroom time and funding have been squeezed out of civics and history education. I mean, look at the state of our civic discourse today. It's just dreadful. Nobody can agree on anything. Then it's so, so, so important for us to have a big picture story that we can all agree on about what we are as a nation. That has to be inclusive. That's what I think all the skirmishes are now on the state boards of education. It's just like, are we teaching our kids? We don't want to teach them that. No, we want to teach them this." Everyone has a different idea about what our country is, and that can't continue. If it does continue, we're probably going to end up not being one single country anymore, eventually. That is where this is all headed. you know, unless we can just sort of make a difference. And like, I always credit Kilman's approach to education about Native American history and just, you know, sort of like economics in general is, you know, really kind of being a great forward thinking, common sense approach, because this idea that, you know, you kind of can't dwell too much on the negative. I mean, there's a lot of it, you know. Certainly. There's a lot that you could say about, you know, give us our land back and so on. And certainly a lot of those conversations are being had and there's like mutual agreement. And I think things are going in a really positive direction. But at the same time, like just this idea that, you know, what is our shared future together? Like, how can we, you know, sort of move forward and not like be at odds, you know? So just the idea that like, let's tell the stories of our resilience. Let's not tell the stories about our pain all the time.
[00:42:48.471] Kent Bye: Hmm. Yeah. Generally, it sounds like that a lot of the stuff that you're doing there at our worlds.io is trying to tell a lot of these stories, specifically a lot of the indigenous stories, having Kilman being a former tribal leader. And yeah, it sounds like some of the stuff that you're trying to do is just trying to also capture some of these stories that may have not been widely told yet.
[00:43:08.980] Catherine Eng: Yeah, like stories that are empowering. Yeah, I mean, there's such amazing stories that are yet to be told. Like, you know, right now we're working on a curriculum about an indigenous scholar from San Diego from like the 1840s named Pablo Tak. Have you ever heard this story? This is amazing. So he was 18 or 19, he came up in the mission system and his parents were part of the church. You know, so of course, like, people may have thoughts about that too, but that's just what it was. And he had this gift for languages. And from a very young age, he knew, you know, Spanish and English, and of course, his own Paeomkuchom, Lucenio language, and then he was also learning Latin. So he had this gift for languages. And I think the church had very high hopes for him and another student, you know, and they sent them, the church sent them in 1840 to the Vatican. to study. And for a year and a half, he made this dictionary and he wrote down the ways of his people and he made this book. But then unfortunately, he got sick and he died very young. But in an alternate history, what if he had come back and you did have from that time on just this Native American leader in the church system here? That would have made such a huge difference. And I mean, it didn't come to fruition, but I mean, just like stories like that. And then also just this idea, there's another really interesting story about how, I don't know if you'd heard, but like when soldiers went to battle in World War II, the Iroquois nation had to declare war against Nazi Germany in order to be able to allow their braves to go and fight. you know, like they had to declare war under their own authority and not just like that of the United States, which is so fascinating. And, you know, just like what that conversation is and just this idea that like in part of Chairwoman Santus' interview, she said that, you know, some tribes would consult with the mothers and grandmothers. Like, is this war important enough to risk the lives of our sons and grandsons? If we don't think it is, then they're not going. Which would be so great if we all could have that kind of authority over what's going on. It's just another way of looking at things. Very interesting. I think that there are all kinds of stories that need to be told.
[00:45:18.932] Kent Bye: Great. And finally, what do you think the ultimate potential of virtual reality and this type of immersive storytelling and ways of using it for education? What do you think the ultimate potential of this medium might be and what it might be able to enable?
[00:45:34.699] Catherine Eng: I think it's just like fast track to really understanding what pieces of history that might take you longer to really kind of comprehend it, but it might come together a little bit faster. I mean, I would just hope that it's a way in which all the dimensions of what goes into history can be presented to you in a way that you can connect the dots. You're not just reading a history, you're also looking at economics, you're also looking at maps, you're looking at geographic inevitability. They didn't just lose the war because of X, Y, and Z, they lost the war because they were downhill, that sort of thing. I think it's a game changer. And what we're trying to do now, and I think what so many people are trying to do now is find the forms that really honor the capabilities of this medium. You know? That's like the holy grail. And there's so much you can do. I mean, it's really just like the sky's the limit. And the number one thing is to do in such a way that you don't make people nauseous. That's kind of like, that's like goes without saying. But, you know, aside from that, I can see becoming extremely ambitious about like how much information you're presenting to people. Because I feel like in that context, arguably, I mean, it's just burned into your mind. I mean, you see, you know, one thing next to another that like you would not be able to see on a printed page. You see it at life scale, larger than life scale, like monumental scale. you know, those kinds of aesthetic considerations and sort of like mnemonic considerations are really, really powerful. So you can do a lot.
[00:47:04.645] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:47:09.219] Catherine Eng: No, it's just so inspiring what everyone's doing. What I would like to see is more stuff that is accessible to everyone. Because as beautiful as all of those experiences are, I just want everyone to be able to see them. That's always the thing, right? Certainly some of the experiences that I saw at this year's South by Southwest, it's like, oh, I wish more people could see them. Yeah, but you know, it's very much, you know, something and it's, it's, we're right at the beginning of it. I think there's going to be opportunity to find new ways to innovate how to do that. So that's very exciting.
[00:47:41.102] Kent Bye: And then at the end of your piece of chiefs that was showing at South by Southwest, you said stay tuned for spring. We might have updates. So what's, what's next for when people might be able to check it out.
[00:47:51.258] Catherine Eng: Yeah, no, no, for sure. You know, so we want to extend it to actually, I mean, in order to do the stories justice, it really needs to be like an hour. I mean, you know, I watched a couple of movies in the Vision Pro. It's actually really nice. So that's, that's what we're doing. We're aiming for like a long form piece. And yeah, so really excited about it. We're working like night and day on it, basically, right now.
[00:48:12.551] Kent Bye: So very cool. Awesome. We'll stay tuned for our worlds.io to see more information for this app as it starts to come out on the vision pro and saw a little bit of a sneak peek of what you're showing there at a supply Southwest. And yeah, I think, you know, like I said, the mashing together of all the different affordances of the medium and not as much of the interactive aspects, more of just the. kind of multimodal mashing together of all the different media types and yeah, just all the ways that you can start to tell that story. So yeah, really curious to see where you continue to take this in the future. So thanks again for joining me here to help break it all down.
[00:48:45.046] Catherine Eng: Thanks so much for the conversation. I really appreciate it. And I love talking to you anytime.
[00:48:49.639] Kent Bye: So thanks again for listening to this interview. This is usually where I would share some additional takeaways, but I've started to do a little bit more real-time takeaways at the end of my conversations with folks to give some of my impressions. And I think as time goes on, I'm going to figure out how to use XR technologies within the context of the VoicesofVR.com website itself to do these type of spatial visualizations. So I'm putting a lot of my energy on thinking about that a lot more right now. But if you do want a little bit more in-depth conversations around some of these different ideas around immersive storytelling, I highly recommend a talk that I gave on YouTube. You can search for StoryCon Keynote, Kent Bye. I did a whole primer on presence, immersive storytelling, and experiential design. So, that's all that I have for today, and I just want to thank you all for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
