Note: This is a sponsored content post from the University University of North Carolina School of the Arts
January 31st, 2021 is the deadline for a six-month Immersive Storytelling Residency Program that runs from May 1st to November 1st, 2021 as a part of the University of North Carolina School of the Arts’ Media + Emerging Technology Lab (METL). There are three different residency roles that will be collaborating over the six months including a game developer/programmer, 3D modeler/technical artist, as well as screenwriter/producer. There’s a monthly $3500 stipend, and a requirement to relocate to Winston-Salem, North Carolina for the duration of the 6-month residency.
I talk with METL Director Ryan Schmaltz to get more context on the emerging media and XR programs at METL, but also do a bit of a recap of the first artist residency program that ran from March to September of 2020. We talk about some of the lessons learned from the first cohort with the biggest shift of moving the program to span May to November (rather than March to September) in order to make it more accessible to graduating seniors.
We also talk about some of the open problems and challenges from design to implementation of an immersive storytelling project, and the opportunity to get feedback from a number of different mentors. I served as a mentor for the first cohort, and I’m planning on returning again for the next cohort to help provide feedback and guidance along the way. METL’s Immersive Storytelling Residency Program is a unique opportunity to get paid a living wage to work on immersive project with a team for six months, and you can get more information on their website as well as more context from Schmaltz in this episode.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of Fear podcast. So today's episode, it's actually a sponsored episode for a immersive storytelling residency program. So this is going to be featuring Ryan Schmaltz. He's the director of the Media and Emerging Technology Lab, aka METL, at the University of North Carolina's School of the Arts. And actually they had this inaugural program of this immersive storytelling residency program last year. And I was a mentor, helped out for the six months doing check-ins with the three different participants of the Immersive Storytelling Program, giving feedback and yeah, just kind of helping along the way. But it's a pretty unique program where they're going to be asking for applicants up until January 31st, 2021. And it's going to be running from May 1st to November 1st, 2021 in Winston-Salem, North Carolina. There's a stipend of like $3,500 a month. And yeah, it's a pretty unique program. And also just what is happening there at the Media and Emerging Technology Lab at UNC School of the Arts is also pretty unique in terms of trying to innovate a curriculum for immersive storytelling when everything's still being figured out. So Ryan comes from a background of startups and he's doing a lot of these really interesting collaborations with hospitals and other different applications of immersive storytelling that they're creating. lots of really interesting stuff that's happening here at metal and you know Brian brought me out to metal to help host the Inaugural future of reality summit back in 2018. And so I had a chance to go visit there Yeah, I just think it's a cool program and they're doing lots of interesting stuff and this immersive storytelling residency programs pretty unique as well So we'll be diving into the details of all that in the second half and then just describing what's been happening at metal in the first half So that's what we're covering on today's episode of the Voices of the Hour podcast. So this interview with Ryan happened on Tuesday, December 1st, 2020. So with that, let's go ahead and dive right in.
[00:01:59.785] Ryan Schmaltz: Hi, I'm Ryan Schmaltz. I'm the director of the Media and Emerging Technology Lab, or as we call it METL. It's located at the University of North Carolina School of the Arts and the School of Filmmaking. which is in Winston-Salem, North Carolina. At METL, what we do is we look at emerging technologies and leverage emerging technologies for the purposes of storytelling and content creation. And so that ranges from virtual reality to motion capture, including game engines, green screen. We're also flirting with the concept of LED walls, as you might've seen in the Mandalorian. And we're really looking at how we can use these emerging technologies to innovate how stories are told, both from an entertainment perspective and also from a corporate innovation standpoint.
[00:02:44.785] Kent Bye: Yeah, and I know I had a chance to actually visit the campus a few years ago, and we talked about this time last year. And what I was really struck by was there's a number of different programs there at the UNC School of the Arts. And some people are doing production. They're going off to work with Cirque du Soleil. They have film students that are there. And this program seems to be within this larger context of a lot of other people who want to get their foot into this industry, this larger Hollywood entertainment production industry, and there's a bit of more traditional entertainment with film and theater and theater production, but there's also like a soundstage and everything else. And so maybe you just help set that context in terms of this program of metal fit into this UNC school of the arts.
[00:03:28.192] Ryan Schmaltz: Yeah. So the school of the arts has several major programs. It's really an arts conservatory. So from dance to drama to design and production and filmmaking and music are the main disciplines within the program. And so. The School of Filmmaking is really kind of a special place as you alluded to, we have a full production backlot, and we're ranked in the top 10 film schools in the country. I would wager that we're ranked in the top one or two, when you look at film schools as a representation of public film schools so it's part of the University of North Carolina system. We're standing toe-to-toe with the AFIs, the NYUs, the UCLAs and delivering a really meaningful program. And what's really special about our program is that there's a lot of intricate details that make our program a little bit different than the other, but one of the major ones is the price point. And so for people who are interested in going into film, they have an opportunity to access that type of an education at a much lower price, which means we get an amazing pool of diverse candidates from all economic levels, which creates this exciting melting pot of content creation. And so, about three years ago, we just celebrated our three-year anniversary with Metal. We really were looking for a way to sort of take inspiration from the concept of the MIT Media Lab and the concept of the NYC Media Lab, whereby students can advance their education in a meaningful way through creating partnerships with industry and that industry may relate to entertainment industry, but it might also relate to medicine or finance or education. And so if we can use our powers of storytelling for the purposes of content creation across a variety of sectors, we're not only giving students and as well as faculty really interesting projects that they can participate on, but we're helping their skills stay relevant with what the industry really needs. And so that was really one of the sort of impetus of metal, but also the other one was the former Dean of Filmmaking, who is now the Dean of AFI, Susan Ruskin, was attending many of the film festivals and noticing this contingency of immersive storytelling starting to emerge. And she was looking for ways to innovate and think about where film was going and how to best position UNCSA for that future. And so looking through the lens of partnering with industry, how emerging technology is shifting the way stories are told. And the first one sort of leading the pack being virtual reality is the sort of elements in which formed metal and allowed us to create the institution that we built now. And we've now been around for three years. And I think that some of that initial work has changed and morphed over that time. And we've learned a lot, but we're still sticking with our core ethos, which is really leveraging emerging technologies for the purposes of storytelling.
[00:06:27.667] Kent Bye: Yeah. And I know that when I went to the future of reality summit, you were able to bring in lots of different people from all over the entertainment and immersive entertainment industry to be able to have that one day summit that I was a part of helping to be the MC of and moderate a lot of those panels. But for you personally, you're also coming more from that, I'd say industry side, rather than like the. film and media entertainment side. Maybe you could just give it a bit more context as for you and your journey and how you kind of fit into coming from this industry perspective, but then helping develop this immersive storytelling angle and also just integrating all these emerging technologies into this immersive storytelling program there at Metal. So maybe just get a little bit more context as your background and your journey into what you're doing now.
[00:07:09.918] Ryan Schmaltz: Yeah, sure. So I started my career in consulting. I think I mentioned this a little bit the last time we spoke. But most recently, before I helped start the lab, I was an executive vice president at Blipper, at that time, the most well-funded augmented reality platform. And we were looking for novel ways to engage with audiences through their smartphone and deliver AR experiences that could be related to education, information, entertainment, socializing. And we kept running into some common problems. One of those is The smartphone isn't necessarily always the best medium for augmented reality. In one of our studies, we realized as we were starting to create almost a social platform for augmented reality, that there are three different zones of consuming augmented reality content. One of those is the safe zone, where you're holding your phone down and you're looking at it and you're looking down at your phone. Then there's the vulnerable zone, where you have your phone pointed at the sky and think about scanning a billboard to deliver an AR experience, for example. And then the third zone is what we coined as the creepy zone, where you're basically pointing your phone at another person and it looks like you're taking a picture of them. Now there's a lot of interesting and beneficial behaviors that could be done in the creepy zone, but the sheer fact that it's creepy alone dissuades people from one picking up their phone and doing that and also makes the person on the receiving end feel a little bit insecure. And so that observation, in addition to the fact that we had really taken this sort of agency model of augmented reality, where we were doing bespoke campaigns for everyone from General Mills to Pepsi, to creating this sort of democratized platform for building AR content. But what we were seeing is that people were still being moved by the things that we had created three years prior to where we were at that moment, and getting very excited about the most simple implementations and simple features in augmented reality. that almost felt like we were building our latest features and products in vain and so what i was on the quest of even at blipper was finding this. Usability, finding a way to create utility for the end user. It extends beyond marketing to the end user. And we were largely focused on marketing content, but what it was really is trying to create value for the user, not just another way to be advertised to. And so I was on this quest and during that time we ran into some fundraising issues and I eventually had to wind down the San Francisco office, let the team go, let myself go. And I was looking forward. sort of the next opportunity. And as I interviewed around, I was looking at all these companies, both large and small, and none of them had quite solved this problem of how do we go from this novelty, as I called it, to necessity and creating meaningful ways to actually use AR that wasn't a gimmick. And I think Pokemon Go is the first inclination that there is a there there, because they managed to crack that. And they found something that was really interesting and exciting and addictive. And it brought people together and people were doing silly things and having a great time. And that sort of was a major uplift for the industry as a whole. But extending beyond the gamification and looking at sort of everyday life, how can we bring the internet to the physical world? in a meaningful way and do so with a smartphone. And when I saw that after leaving Blippr that a lot of companies were facing this same problem and didn't really have any answers and were really trying to solve the same problems that I had just been trying to solve with Blippr, an opportunity arose for this lab and it was almost frankly a breath of fresh air because now we got to focus on the consumers of the technology rather than increasing the breadth of the technology with no users. to use it. And so that's what initially got me very excited to join. And it was something I really missed because prior to Blipper, I was at Uber. And the way I got to know my customers was every day on the way to work and every day on the way home. I could find out what was working with our product, what wasn't. I could have relationships with our customers. The product was ubiquitous. even at that time. And so in particular, I was sort of craving that tangibility and access to my customers that I wasn't getting. And so I would say one of the main things that took me from this technology world of product development into the content side in academia, which is an unlikely place to end up. But one of the things that took me there was this obsession over being able to create an impact for our end users and for audiences. whether the users or audiences trying to create this environment of something that keeps them engaged and excited and deriving value from our products. And so storytelling and content creation and the affordances that virtual reality and augmented reality make to storytelling with agency features, different levels of interactivity, branching narratives, being fully immersed, all of those things were really enticing to me and what helped me make the shift. And so in addition to running the lab, I also am very active still with startups. I advise a small portfolio of startups and also occasionally invest in startups. So my heart is still very much this sort of startup entrepreneurial, I call it like the startup religion, really, because we have our own doctrines. We have our own vernacular. There's so much passion around it when you're in a startup, when you're starting up. overcoming problems and challenges and your own shortcomings. And once you have a taste of that, it's hard to turn it off because it's a really exciting and sometimes chaotic and crazy environment to be in. But I found a really interesting place at the lab, both being able to stand up the lab from scratch, everything from giving it its name to deciding its mission, but also from the standpoint of being able to leverage technology to create value for the audience.
[00:13:18.546] Kent Bye: Yeah. And here in a moment, I want to dive into the artist residency program that you're going to be having again this year. But before we do that, I want to maybe get a bit more sense of the existing program that you have there in terms of people coming in and learning about immersive storytelling using these emerging technologies. So we're still in this area where from my perspective, we're kind of fusing together all these design disciplines from cinematic storytelling to web design, user interaction design, user experience, human computer interaction. You have elements of game design, you have architecture and theater and all these other elements of embodiment. And how do you have these avatar representations? I mean, you're fusing together all these variety of different design disciplines and trying to like put it all together into immersive technologies with immersive storytelling. And so I'm just wondering how you're approaching that in the realm where the theory of it all is still being developed. And I guess it's more projects and people making stuff, but how do you start to break down this area and start to teach students, you know, how to get their foot into the door into this whole new area?
[00:14:25.536] Ryan Schmaltz: Yeah, it's, that's a great question. And it's definitely an iterative process. You know, I think it starts first with awareness, at least from a student perspective of this is something you could do. And is this something that you're interested in doing? and just the general awareness of this medium as an art form. And then it extends beyond there of leveraging best practices. And sometimes those best practices come in the form of, let's create a long form piece of content from beginning to end is fully immersive. And in another scenario, it might be that one small component of this traditional 2D short film requires game engines or motion capture or an LED screen. So how can we deploy those technologies in a more, I guess, traditional format or the traditional format of filmmaking? And so it's a combination of jumping in the deep end of immersive and focusing solely on immersive. and what the content could be and what you're creating and building that from end to end and thinking about all those things that you mentioned and sort of parsing those out and looking at them individually. Sometimes it's messy and we're sort of figuring it out on the fly and sometimes we take a very thoughtful approach about how we go about doing it. But other times it's about pulling one aspect of the technology out and enhancing some traditional storytelling. And so, we look at it in both of those ways. And that's really looking at it from an educational standpoint, but then there's really the partnerships with the industry. And so, as we think about the partnerships with the industry, then it's really about them coming to us with projects. And in some cases, those projects really align with their business need, but they might not align with our skill set. And what we look for is trying to find that sweet spot of collaboration where it's something that we can deliver something really quality on and we can be additive in the process, but also satisfies the need of the partner that we're working with. And then the other part of it is, is looking towards our community for projects. So both in Winston-Salem, where we're located, across North Carolina, looking at the Southeast and looking across the country, There are a lot of things that are really passionate with North Carolina legislature, for example, such as literacy and access to medicine, access to technology. And so we partner with everybody from North Carolina Public Broadcasting to local hospitals to different not-for-profits and try to find ways that we can be additive and helpful within our own community as well. And so those projects were very selective about what makes sense and where we can add the most value.
[00:16:59.287] Kent Bye: I think that's what I find so intriguing with the metal program there is that you are integrating and collaborating with what I would say going beyond just the arts and entertainment sector, but looking at the medical sector and seeing how you could start to use some of these technologies in a medical context and potentially even start to weave in elements of storytelling that you have the distribution channels for immersive storytelling are still so nascent that it's hard to like go in and get a degree in this and then come out of it and have a full-time job of just doing this all the time. I mean, that's sort of a rare thing within the VR industry at this point. You kind of have to like strike the lottery. And there are folks that strike the lottery, but it is a bit of like needing to have a good backup plan to have a way to have a paycheck. And I think some of the different collaborations you have with some of these medical institutions and hospitals, I think is also really fascinating because it to me, a really good model to see how you could not just set people up to be thrown into this industry and spend all this money in education and then come at the end of it, not being able to get any job. And so like, how do you help guide the students there? Because they may coming into this realm, wanting to really do just purely immersive storytelling, but yet at the same time, if they want to work in the industry, there's, I guess, not as many jobs. There are jobs that are out there. Certainly they could go get a job, but if they want to strike out on their own, they need to have like a day job and But how do you sort of like make sense of that larger ecosystem and the vibrancy of the career opportunities after someone would get through? It seems like you're in some ways with some of these projects, creating a situation where people have some different off ramps or different opportunities with not just putting all their eggs in one basket.
[00:18:35.925] Ryan Schmaltz: Yeah, you hit the nail on the head there. I mean, Right now, I think the value that it brings the students is in the form of a resume booster, frankly. And so if a graduating film school students can have on a resume, I've worked in the Unreal Engine. I've worked with Unity. I'm familiar with the pipeline of working with motion capture. That is something that separates them from the pool of applicants that they're competing against. So I think that's the way we're really adding value right now, more so for the students. There are those students that are very ambitious that set out on their own and want to work in this space, but I would say more so it's the former, not the latter. And so the way through the lab in which we sort of help support the students on one hand is we give them the tools and the information available that they need to know what's available to them. in this space. And then we show them the watering trough, but we sort of leave it up to them to take a drink. And there are students who become very interested in what's there and how it can enhance what their vision is for the films they want to create as part of their school projects or even independently. The second is through these corporate projects. And we recently launched our corporate membership program now. And so through that corporate membership program, we're doing individualized projects for companies. And so we engage with both students and faculty and also alumni to be service providers, collaborators, contributors on those projects. And that gives them opportunities to generate income in this space, get exposure to projects, build out their portfolio, and get engaged in things that we think they'll find interesting. And then I think the third way is really just through connections. And so in my role, I come across a lot of opportunities with boutiques and independent studios to somebody who wants to shoot a 360 video in a surgical room while they're doing a complex procedure. And so what I'm able to do is really sort of serve as a connector for our current students, graduate students, and alumni. and also our faculty to be able to learn about these projects and bid for them or take on the projects that are of short-term in length or potentially long-term. It doesn't really matter as long as they can find that sweet spot of aligning the skillset with the needs of that project. And so in that way, a lot of corporations have found some really interesting talent that they would not have and got some really great quality work out of our alumni population.
[00:21:05.100] Kent Bye: Yeah, well, let's dive into this artist residency program that you're doing again here. Uh, maybe just give a bit more context as to this program, uh, maybe how it came about and the deeper intention for what you want to achieve with it.
[00:21:19.004] Ryan Schmaltz: Sure. Yeah. So we call it the immersive storytelling residency program, and this is our second year now going into our second iteration of the program. And so we've just completed our first residency program and we're in the application process for the second. And so the beginning was some things were starting to percolate in the immersive industry. One of those is every time anyone, regardless of your background or your skillset or your expertise, anytime anybody puts on a virtual reality headset and they have an experience that moves them, or even if they don't have an experience that moves them, maybe they get nauseous on the rollercoaster ride in VR. They walk away starting to think around ideation around, well, if I were, creator for VR content or a VR game or an art piece, this is what I would do. And there is no shortage of amazing, actually amazing, untapped ideas that are out there. but yet there are very few sources of funding. And so funding is a major problem. And one of the things we try to address with the residency. The second part is if I'm a, let's say I'm a visual artist and I have an idea for an immersive experience, but in my social networks, in my professional background, I don't know a single software developer. I don't know a single game engineer. I don't know who's going to help me take this concept and bring it to life. And so the second thing we look to address is creating an environment where you're put together with a team of complementary skill sets that can ultimately deliver a project of that team or cohort's choosing. And so we take three residents. One is a 3D technical artist familiar with working in the Z-axis. The second one is a game engineer or developer, somebody that has moderate to high familiarity with the Unreal Engine or Unity. or an interest in working in game engines. We find that that skill set works really, really well and eases into the residency very well. And the third is a screenwriter and the person that's familiar with character development and engineering the story from top to bottom of how to build tension and conflict and how that story needs to evolve over the specified period of time that you have. We've also, in this year, expanded the screenwriter's role to also where the producers have, because there's a lot of other things that come into play beyond simply creating this piece. Assets to apply to film festivals, for example, imagery, marketing, standing up a website, thinking about partnerships. If they're building an experience that's about the deforestation of the rainforest, for example, then there are multiple not-for-profits out there who would benefit from being a part of the concept that they build, and also the project would benefit from being affiliated with those organizations. And so the screenwriter slash producer helps satisfy all of that. So these three people come together and are sort of like a tripod, right? You lose one leg, the whole thing topples over. And they're tasked with ideating and developing a concept from scratch as a team, and then bringing that project to fruition over six months. The three residents that participated last time didn't know each other. They were meeting for the very first time. And they came together. just before COVID really set in and were able to establish an amazing rapport, a culture amongst themselves of being able to listen to each other, but at the same time, provide feedback. And they iterated on a concept and then brought that through to fruition over the course of six months. And so when we replicate this program, it will again be six months, starting in May through November. And we pushed the date out just a few months to make it easier for graduating seniors from universities to have the opportunity to apply. We found that to be a challenge last time, so we're looking forward to helping additional applicants apply, especially from university. And so what we were really fortunate with is in the middle of this last residency, Epic Games, through their Epic Mega Grant, realized the benefit of this program. And they got very excited about what we were working on and helped subsidize the last program that we had, the first version of the residency program. And that's enabled us to do this again. And we hope to continue this into the future. But that was really instrumental in terms of validating what we've done and continuing to keep us on the path for what's coming forward.
[00:25:56.348] Kent Bye: Yeah. So maybe you could just talk briefly, like the application process, what the deadlines are, what do people get if they did get in? And because I think that's a unique part of this residency program is that you are offering a stipend for people to actually sustain themselves for that six month period.
[00:26:11.837] Ryan Schmaltz: That's right. Yeah, so the program runs from May 1st to November 1st, and applications are due January 31st. The applications can be found on our website, metl.org, metal.org. And if selected for the program, we look for one 3D technical artist, one game developer, and one screenwriter slash producer. And we welcome all applicants. If somebody looks at it and doesn't think that they might not have the chops necessary to do a particular role, we still encourage them to apply because this is a residency driven by learning. But we're also looking at applicants in the form of graduating seniors and people still attending university, but also working professionals. We don't want to cater to one particular group versus another. What we're looking for is people who are really excited about this medium and have a vision for and a desire to create in this medium. And so if selected, they would receive a $3,500 stipend per month for the duration of the residency. which isn't going to make anyone rich, but it's a living wage. And we're pleased to be able to offer that. So that way, during the course of the residency, the three residents can focus on their craft and have protected time where they're not worried about paying bills, dedicating nights and weekends only to working on this in their spare time. They have a dedicated period of time to define and build out an idea. And then also the support and mentorship through our mentoring community to help them get from point A to point B, and also help them on the back end in terms of how they look at various ways to distribute their experience once it's done.
[00:27:51.608] Kent Bye: Now, I know last year, one of the requirements was to actually go to North Carolina. And right when it began there in March, then a few weeks later, within a month, you have a global pandemic and people have all these social distancing requirements. And so this year, as you think about this residency program, are you still going to have people common in North Carolina, or is there going to be like remote options? Yeah. I'm just curious because the pandemic sort of was in the middle of this and if that's still going to be in a requirement for people to actually relocate to Winston-Salem, North Carolina, or at least be in that area to be able to potentially get face-to-face and do some of working face-to-face.
[00:28:28.890] Ryan Schmaltz: Yeah, it's a great question. I'm glad you asked it. I mean, I spent years working remotely and also the majority of my career working face-to-face with my colleagues. There is no substitute, especially when you're doing creative processes, besides sitting together and ideating, because it removes the formalities of communication. Whether that's a quick whiteboarding session or yelling across the desk at your fellow resident, I think that there is inherent value to being all together. COVID landed right in the middle of our first residency and so that obviously created some huge health risks but also operational challenges and especially as information was coming out in waves about COVID. that was in many cases contradictory than the prior information that came out. We're fortunate now in a place where we know a lot more about the virus. And so I think that for this coming residency, we'll continue to want to get everyone together in person. Now that said, if there are health considerations, we'll definitely take that into account. Obviously, because we're a part of a public academic institution, we also have to follow the University of North Carolina's lead on convening in person versus not. But what we found to be exceptional about the first group of residents that they did on their own, and something that we would hope would be able to be achievable with the second cohort of residents, is that they chose to socially distance together. So they formed really their own pod of collaboration. And regardless of whether they met at our facilities in Winston-Salem, or they met at one of each other's homes, or at a place where they felt safe to do so, We want to encourage people to get together. We want them to be in person with each other to the extent that it's possible.
[00:30:14.265] Kent Bye: Yeah. I guess sometimes referred to as the core and team. So teaming together with your quarantine buddies.
[00:30:20.372] Ryan Schmaltz: So I like it and I'm going to steal it.
[00:30:25.221] Kent Bye: Yeah, I guess I'm also curious to hear a little bit about how the pandemic has impacted the rest of the education there at metal, because you know, there are a number of things in which that it's really hands on in terms of if you're working with an led wall, it's not something that you could do remotely. You have to kind of be there. And so is there similar kind of like social distancing ways in that you're still able to gather folks together for all these other programs that you're doing at metal?
[00:30:50.860] Ryan Schmaltz: Yeah, most of it is done virtually. So our normal sort of day-to-day operations are all virtual. The School of the Arts in particular, as an arts conservatory, was in a unique position because of its various programs, as I mentioned earlier, from music to drama to dance. You can't become prima ballerina by practicing ballet over a Zoom call, nor can you become the first chair violinist by practicing with the orchestra over video conference. And so The university has embarked in, they brought in consultants, they embarked in various techniques and looking school by school at the ways to sort of enable the university to continue and do so in a safe way. And we've been really fortunate because the students of UNCSA, faculty, the staff, the administration is all sort of banded together. and taking the charge of protecting the community. And so the cases of COVID, and I don't know what they are offhand, but they're considerably lower on a per capita basis than other universities that are out there. And so I think everyone at UNCSA is really proud about that. But we have had to make concessions in terms of getting people together and look thoughtfully at all of our processes. And it's definitely impeded a lot of our work at MEDL, especially in our motion capture stage. But those things are starting to percolate again. And frankly, because it's immersive technology, in the middle of COVID, in the middle of social distancing, focus started to turn towards immersive. Because as everyone is cooped up at home, the way to make those four walls disappear was to put on a VR headset. COVID in particular has been a very busy time for people working in the immersive industry and including with our lab. And so while some things we've had to stop doing or pause, other things have picked up, including our collaborations with our local hospital here at Wake Forest. We're exploring a virtual visitation pilot within the pediatrics department. Right now, if a loved one gets hospitalized, you may or may not be able to go see them. We're working with them to deploy technology that enables loved ones to be with their family members. projects like that would have never popped up or would have taken a long time to pop up if it weren't for COVID. And so it's more of this do-si-do that's happening. And thankfully, the sort of the lemonade out of the lemons of COVID is that it's been a particularly busy time for the virtual reality industry and it's exposed such a broader audience of people who had thought it was mildly interesting before, now are focusing attention on it. and are looking at it as a potential solution for a variety of fronts, including how do you get people to function as they do when they're in person with each other in an office environment, but sitting in the comfort of their own home. So we've been very busy, but busy in a different way, I guess.
[00:33:43.887] Kent Bye: Yeah. Yeah, well, now that you've been on the other side of doing this first cohort of the immersive storytelling residency program, you know, we're going to be talking here in a few moments with the entire cohort and unpacking their whole journey. But from your perspective, what were some of the big takeaways or lessons learned that you got from this first iteration of this immersive storytelling residency program?
[00:34:06.996] Ryan Schmaltz: Yeah, that's a great question. The program is largely the same as it was the prior year. I think the changes that we're making are really. more kind of nuances. You know, I think the first one is that, you know, we've expanded the screenwriter's role to include producing responsibilities. We shifted the dates to be able to enable people who would otherwise be unable to apply. And in terms of the experience development, I think we learned a lot in terms of the nuances of seeing this first cohort go through. And I think that's something you probably saw as well as, as one of the program's advisors. But what we did was we protected time in the beginning. for defining the concept and creating the script. And I think this year and with this program, we realized in the prior year that it's not this sort of hate to use the term, but waterfall like approach. And it's different than filmmaking also. So what we're looking at doing is shortening the timeframe for defining what that concept is. because we understand that it's much more fluid and integrated with the development process. And I would say, aside from the sort of the TDM and operational slight changes, the single biggest change we're going to make is that, where we want the storytelling and narrative development to be weaved into the technology development and the art development. Because once the assets are created, that's the heaviest lift, honestly. And so making plot changes at the 11th hour is not ideal, but it's doable. But part of the reason why you make those changes is because you tried something you thought was going to work, and it didn't work. And so what we want to do is make sure that there's enough time to allow those learnings to happen for the residents.
[00:35:54.087] Kent Bye: Yeah, this was a very interesting experience for me as well, because usually when I'm watching experiences, they're finished and they're already at the film festival. And so this was an opportunity for me to kind of get involved with the creative process at a much earlier phase. And yeah, to see that tension between the iterative nature of how you have to really test things out and experience it, to see how it modulates your experience in your consciousness combined with having a very clear vision of what the overall story is and the intention, the arc of what that is mashed up with that more loose, open-ended iterative space to be able to kind of like prototype and experiment out. And how do you really balance those two? I think is a great challenge of having a clear vision of what you're trying to do, but at the same time, you can't have that clear vision be fully formed until you actually make it and experience it. And so it's like this dilemma of, At what point at the spectrum do you fall? Because a lot of times when you're pitching these projects, a lot of the existing processes of pitches are using the existing conceits of film and film pitches. And film is a medium that has pretty well evolved over the last hundred plus years so that you have less variance between that pitch phase and then development into the actual film. And so there's less of a gap between the slide deck that you're presenting the idea and the final product. Whereas I feel like that gap between immersive stories is so huge right now. And it's so many unknown variables that, you know, sometimes you don't know if something's going to be really compelling until you actually experience it. And even if you did have it in the slide deck to try to convince people of someone that it will be compelling can be difficult until they actually have the experience. So I tend to see that a lot of what happens with these pitches is that Yeah, there's that slide deck, but there's also like that initial prototype and that demo at that very early phase, at the point of raising money and starting to get a project off the ground, you've already developed the concept and the pitch to the point where people can already get a sense of that. And so that to kind of work backwards here, it's like at what phase and that trade-off to kind of balance that, getting that minimum viable product of the experience ready versus like to really fully kind of push forward with the vision that you have.
[00:38:07.076] Ryan Schmaltz: Right. Yeah. And we want to move that up to be as early as possible. And it's funny because it sounds almost counterintuitive if you think about the production process. Right. And if you watch any sort of behind the scenes for whichever Pixar movie, you'll hear them talk about how organized they were going into production and how the whole thing scene by scene was completely storyboarded out and it was ready to go. And it's really easy to apply that kind of methodology and think that that type of methodology is necessary because some of the processes are the same. It requires software development, it requires animation. And so the way of shooting is different than a live action film, for example. But what we came to realize that, especially with moments of agency and things like that, there was just some things that just weren't working and some things that were working really well. And the sooner we can tease those out, initially with the concept, but then eventually with each scene and saying, okay, well, what's working here? Are we deviating from the message? How does the audience member feel? Do they feel connected to the story? Do they know what's even happening? Are they so fixated on the tasks that they have to do that they've lost sight of the story and this is more of a game to them. And so it's trying to find and sort of that balancing act between all of these competing things that are going on. And one of those things for many audience members is just putting on the headset to begin with, because we're still encountering a lot of people who are new to the medium. And so orienting them to their environment, making them feel a part of the story and that any interactivity is additive along the way. And also at a macro level saying, OK, we've got six months here. We're not going to do a VR adaptation of War and Peace. we can get across a very simple message. It can be rich and engaging and fulfilling, but we have to keep in mind of the scope and duration that we have to work within. All of those are competing factors that all play in here, some of which we knew in advance and some of which we learned along the way with the first residency, but all things that we're going to take into account with the second.
[00:40:07.932] Kent Bye: Yeah, it reminds me of the work that IOMLX Lab has been doing with their experiences, because I think they've done a good job of creating these innovative pieces, but also trying to onboard people in all these different ways. And that's really hard when you're talking about these pieces that are for the film festival circuit, usually there's a certain amount of fluency that most people have. And so to try to design for people who have never put on a headset versus people who are quite familiar with the technology, I think can also complicate the design process, let's say. And I tend to see that most of the film festival pieces. I mean, there's a challenge and dilemma here that I've run into where to really be innovative and push the medium forward means that it's more experimental and there's more risk for whatever mechanic that you're trying to innovate on not quite working out. So it's this context of immersive storytelling where we're still in that phase where there's still a lot of room to innovate and push the medium forward. but also like to look at like well-established tropes and experiences and to look at the affordances that work and then put a story around that. And so there's like, you have to decide what end of the spectrum you're going to be on. Is it the story that you really want to adapt to what you already know is going to work within the affordances of the medium or do something that's completely new where you're in sort of like this area where you don't quite know how it's going to play out. And if you want to make some innovations for the medium at large, Then there's higher risks there that come in with trying to integrate all these new technologies or new mechanics and to see how that's going to necessarily play out and work or not within the story.
[00:41:43.080] Ryan Schmaltz: Yes, absolutely. I would say that from our residency standpoint, we think about this very open-minded. So if the residents come and have a vision and that, that is more adventurous. We're open to that, but I would say we're less Jodorowsky and more Hitchcock where You're leveraging some foundational concepts, but you're innovating on top of those foundational concepts rather than delivering something insane and avant-garde, which is amazing, but also more challenging to engage and connect with your audience on.
[00:42:20.285] Kent Bye: Great. And before we start to wrap up here, are there any other logistical details that people should know as they're applying here for this residency program?
[00:42:28.594] Ryan Schmaltz: No, I think we've covered it. So just to repeat the residency is from May 1 November 1 in Winston Salem, North Carolina will help out in terms of from people out of the area to settle in and find their place in Winston Salem and make it easy and applications are due January 31. And during the course of the residency, in addition to the stipend, we'll also be pulling in a variety of industry advisors. And some of those are said advisors that we worked with in the past were phenomenal in the first go of our program, including you, Ken. But others we'll look at really based on the needs of the project. And so we'll kind of be fluid in terms of connecting the residents to people who we think can really help them take their project and their vision to where they would like to see it go. If anyone's interested in applying, then we encourage you to go to www.metal.org, M-E-T-L dot org, or they can also email me directly if they have any questions about the application process, which is my last name, S-C-H-M-A-L-T-Z, first initial, R at U-N-C-S-A dot E-D-U.
[00:43:39.842] Kent Bye: Great. And, uh, for you, what do you think the ultimate potential of virtual reality might be and what it might be able to enable?
[00:43:48.466] Ryan Schmaltz: Yeah, thanks. I love this question and it's something I wait for with every one of your podcasts. I think the last time we spoke, if I remember correctly, I talked about computing being like electricity and I still very much feel that way. I think that is the ultimate potential and sort of the seamlessness between the physical world and the, and the virtual world. But, you know, one thing that sort of shifted that, I mean, I still very much feel that way. But now I think I keep thinking about one of your other podcasts, the one with Brennan Spiegel for VRX. And in that podcast, he talked about the psychological effects of virtual reality and the potential it brings on schizophrenia, trauma, and sort of hacking the brain to overcome potential problems, mental disabilities. or even just help with general wellness. And that's something I'm really fascinated with and something I think is also part of the ultimate potential of virtual reality, which is the ability to almost hack one's brain for the sake of wellness. I think that there could be some, you know, with great power comes great responsibility. But I think that the good here and the potential here, especially after listening to that podcast and picking up the book, is immense. And I'm excited to see where that goes.
[00:45:06.915] Kent Bye: Yeah, I'm really love to see the medical XR folks and to hear what they have to say. And I think there's actually quite a lot of insights that immersive storytellers can get by listening to the medical applications because they're really rigorous in terms of trying to suss out the affordances of the technology itself and to hear some of the stories that Brandon Spiegel was talking about in terms of trying to create immersive experiences that change our relationship to our default mode network in terms of our sense of identity and other people, and to be able to have the therapeutic effects there. And I do think generally, when you look at things like brain control interfaces and this whole movement towards starting with invasive technologies into the brain for people who have neurodegenerative diseases, but then eventually to see if there's going to be more general applications that are non-invasive for the brain control interfaces, and to see that whole range of what I call the consciousness hacking movement, which is to see ways that you can kind of recreationally rewire your brain, just like folks are doing already with psychedelics, but this sort of digiceuticals or cyberdelics or ways in which that you could use the immersive technologies to rewire your brain and to potentially unlock latent human potentials that we have no idea exist yet, or just to get into these deeper flow states or change our sense of wellbeing in the world and to really come into greater alignment with who you are and what you want to be in the world. I also get really excited about all of those different applications and And I think there's a role there within immersive storytelling. And today there was aired this talk that I gave at CreativeXR, talking about how, as we go into these virtual reality experiences, Robert McKee has that quote of, when you're watching a story, you can say character is revealed when these characters are put under pressure and they have to make these choices. And depending on what their choices are, it reveals an essential part of their character. as you start to get into the interactive media, then you become the protagonist in these immersive stories. You're putting into these contrived situations where you have to make choices and there could be aspects of your essential character that are revealed. Now, I think the challenge is you always have to kind of suss out whether or not you're making those choices within that contrived context or whether or not those choices you're making transcend that individual context and revealing something that's a part of your essential character. But I love that conceit that this whole venture is to explore who we are and to be putting ourselves into these contrived situations through this lens of immersive storytelling. And at the end of it, we get to learn more about who we are as a part of our essential character.
[00:47:37.533] Ryan Schmaltz: Yeah, I absolutely love it. And if anybody is in doubt, I mean, we're all going through the same thing in this period of pandemic where, you know, myself included, we've had some hard days where the walls felt Like they were closing in and as an avid traveler, both my wife and I were just frustrated of being in the house and doing our part. And it's amazing on a small scale, putting on your VR headset and doesn't matter what experience you're doing, whether it's a narrative experience or a game to put on the headset and be transported for a little while. to a different place and the activities that you're doing in that place, whether it's Tai Chi or creating pottery or watching a 360 video. When we took off the headset, there was a meaningful and material improvement in our overall sense of well-being. it felt like we went somewhere and it felt like we did something. And so as somebody who's fortunate enough not to suffer from schizophrenia, I have no way to relate to what somebody who has that infliction is going through. But that small inkling of mental wellness during a challenging time of social distancing really demonstrates the potential of what you can achieve in virtual reality, both like you were describing in terms of your own character development and When the chips are down, who are you really to overcoming trauma or phobias or whatever it might be, or just general medical conditions that you're working on. So I'm really passionate about that space and I find it really, really interesting. And I think storytelling can play a really powerful role there.
[00:49:12.007] Kent Bye: Hmm. Great. And is there, is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:49:17.369] Ryan Schmaltz: Nope. I think we've covered it, Ken. Thank you so much. And thank you for all the great work that you do.
[00:49:24.348] Kent Bye: Awesome. Yeah, thank you. So that was Ryan Schmaltz. He's the director of the Media and Emerging Technology Lab at the University of North Carolina School of the Arts. So I have a number of different takeaways about this interview is that first of all, well, this was, as I said at the top, a sponsored post, because last year, Ryan wanted to just get the word out for what they're doing there with this immersive storytelling residency program. And found that there was just a lot of really good applicants that were coming through my podcast and wanted to do another sponsored episode. And also, it just helps me to participate in that as well, just to give me an additional reason to be able to dedicate the time. For me, personally, I think it's just really interesting as a problem and thinking about a lot of the theory and practice of experiential design. I see a lot of content that's out there, but this is just an opportunity for me to be involved with a project sort of from the beginning and throughout the whole creative process and to see what type of things and insights might be helpful, what type of red flags I could see to help steer folks in the right direction. And yeah, just to be able to use my own insights and pointers to be able to help provide some of that feedback. And for me, that's just fun to be able to do that type of mentorship. And I'm hoping that as I move forward, I'll be able to find more opportunities to be able to find that because I think it just helps take the critical theory work and to really just put it into practice and see how it works. So I think it's a natural part of that. And eventually, I think I'll start to create my own content as well. But for now, I like being able to just see all the different stuff and to participate in helping to mentor other programs like this here at Metal. And, you know, like I said, it is a very unique program, which they're paying $3,500 a month for six months for three different people. One 3D technical artist, a game engineer and developer with some experience either Unreal, Unity or other systems that are out there, as well as finally as someone who's a screenwriter, storyteller, the story engineer, but also be doing a lot of the producing and marketing and partnerships and stuff like that as well. So kind of a dual role there. So I actually had a chance to do a debriefing with the previous immersive storytelling residency program cohort, the first cohort that went through. And I'll be airing that probably in the new year. But going back to metal in general, and I think what Brian's doing is actually quite interesting and unique in the terms of You know, he's trying to cultivate this curriculum and do these different projects and train up all the students and all the different latest technology, whether it's motion capture or using these LED screen walls, and just to get used to what it's like to do production with this new pipeline, but also to find different collaborations with hospitals or other Entities that are out there to be able to have that startup mentality as we're in this new emerging media There's lots of different industry verticals and not all of the jobs are going to be available for entertainment And so just to be able to have access to what's happening locally there within the local community and start to see how VR can start to be used but to take the talent of the Storytelling and start to see how that could be applied in these other industry verticals as well So I think it's actually quite an interesting approach that he's taking there and trying to you know create a little bit more resilience, which when you talk to folks in the XR industry, they start to already be doing that with having a diversified portfolio of all the different ways that you can use immersive technologies in these different contexts. So again, just the details of this immersive storytelling residency program. It's going to be running from May 1st to November 1st. It will be required to be relocating to Winston-Salem, North Carolina. That's where the UNC School of the Arts is located. And the applications are due on January 31st. So, you know, don't delay. Go check it out and get more information. Yeah, I think it's a cool program and I look forward to being involved with it again this year and to see this whole ideation process and creation process as you're going through and trying to focus what the story is and then do these different iterations. And one of the points that I brought up here was that I do think that there is this distinct process of, you know, screen writing and storyboarding and everything else that we do this pitches for film. And I do think that because the immersive media is a little bit more interactive, that it's a little bit more of like a game design element where there's a little bit more iteration. And so it's a little bit more difficult to have everything fully formed at the very early phases. And so, yeah, just seeing how this next iteration handles that in terms of having some clear idea of what the story and message that you want to go across but at the same time leaving it open and flexible enough to be able to iterate and do rapid iterations and game testing and just to test out different mechanics and see how the character of those interactions and those mechanics fit into the overall character of the story that's trying to be told. So Yeah, that's just a challenging problem. And I think that for anybody that's in this industry, there's ways of trying to figure out how you best go from the beginnings and how you have this interactive experience. I think that it was worth pointing out that the Tempest was one of the first immersive theater programs that were out there by Tender Claws and Samantha Gorman just within the last week or so. release the full interactive script for The Tempest. So you can actually read through this interactive script and the way that she annotates it through these different phases and what's open for each of the different immersive theater actors for them to maybe improv and innovate, but also the sets of stuff that she really wants to make sure are the consistent story beats, the main thrust and the backbone of the story still told in a consistent way. But, uh, yeah, if you want to apply to this immersive residency program, you can go to the media and emerging technology lab at the UNC school of the arts. And there's a whole tab there for the immersive residency program and you can apply and yeah, give a shout out to the voices of VR that you heard about it from here. And, uh, that certainly will help indicate that you have been listening to what's happening in the wider industry and, uh, yeah, just paying attention to some of the discussions that are happening here. So that's all I have for today. And I just wanted to thank you for listening to the voices of VR podcast and If you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a list-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com. Thanks for listening.
