Scarecrow is an one-on-one immersive theater piece where you engage with an actor in virtual reality. Every person gets the same structure of a story, but the interactions are customized based upon your actions within the piece. There are a number of novel technological integrations including thermal haptics on your of your hands, facial capture of the actor that’s puppeteered by another actor using an iPhone, and integrations of the audience member’s facial expressions captured before the experience begins. I had a chance to talk with director Jihyun Jung, producer Sngmoo Lee (director of Eyes in the Red Wind Sundance New Frontier 2018), and technical director Cooper Yoo about their experiential design process, their narrative design process, and the unique moments that they were able to cultivate within Scarecrow.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So I'm starting a new series of my coverage from Sundance 2020, where I'm going to be focusing on not only the storytelling innovations, insights into experiential design, but also the creation process. You know, what is the process that creators use in order to actually generate their virtual reality projects? So I'm going to be starting with a piece called Scarecrow, which was a one-on-one immersive theater piece where you have this interaction with somebody in virtual reality. And there's actually like six people behind the scenes doing a variety of different things. And so it's this dynamic story that changes depending on what you as a user are doing, but they have things like facial capture, like very high fidelity facial reactions from this character. They have. a thermal haptic device that's on your hand that gives this sense of hot and cold. And they're trying to create this emergent story of this interaction between you and this character. And they talk about their process of doing this. And so I had a chance to talk to the producer, the writer, and the technical director. That's Sungmoon Lee, Jihyun Zhao, as well as Cooper Yu on today's episode of the Voices of VR podcast. So this interview with Seungmoo, Chiyun, and Cooper happened on Friday, January 24, 2020 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:01:31.357] Sngmoo Lee: All right. My name is Seungmoo Lee, and I'm the producer, writer of the project called Scarecrow.
[00:01:38.875] Jihyun Jung: My name is Jiyeon Jung, director of Scarecrow. It's my first chance to participate in here.
[00:01:51.586] Cooper Yoo: Hi, I'm Cooper and I'm the technical director of Scarecrow.
[00:01:55.308] Kent Bye: Great. So maybe if each of you could give me a little bit more context as to your background and your journey into VR.
[00:02:02.451] Sngmoo Lee: I've been a traditional filmmaker, writer, producer for more than 20 years. And I'm also a film professor at Korean National University of Art. And I made my first VR project three years ago, which is called Eyes in the Red Wind, which was invited to Sundance and Venice and Cannes. From that on, I'm kind of still doing my writing job as a traditional filmmaker, but VR is at the heart of my heart right now.
[00:02:31.027] Jihyun Jung: Actually, I'm a student. He's a student. I'm a graduate film school student. Actually, I'm making a film, a traditional film, but it's my first experience VR project. And I'm excited to participate in here.
[00:02:47.670] Cooper Yoo: I majored in interdisciplinary field so I'm very interested in bringing new technology to traditional like movies or media.
[00:02:57.822] Kent Bye: So I'm wondering if one of you could kind of describe Scarecrow the story and a little bit you know how this project came about.
[00:03:05.277] Sngmoo Lee: I started thinking about this project a couple of years ago when I was in Sundance. And my first project was a 360 video, not the engine-based storytelling. And I really got to know the new trend is going. I was not an engine-oriented filmmaker. And I kind of really like the possibility and the future of this new storytelling based on the interactivity, which can only happen in the game engine. So as a storyteller, I've been a storyteller for more than 25 years, I just want to explore the new possibility of this new medium. So just I start digging in on what could work only in this medium, which is not possible in the traditional storytelling medium. and suddenly this scale grow guy also I had to know the limit of this medium which is still is in infancy that is not as fluid and flexible as the film so I kind of limit my imagination and find The topic and the character could work very good in the VR world. I think that's the first time this character came into my mind. From that on, it was ongoing revision all the time. I had to adjust my idea to the new technologies and I also got some input from the other fellows like Ji-Hyun and Cooper. So the story kept evolving for more than two years and here I am now and it's still evolving. I just changed a couple of things today by the reaction of the audiences. That's a good part of this story.
[00:04:39.336] Kent Bye: Yeah, and so in this experience, I have hands and experience and I'm interacting with another person who's embodying the scarecrow. And there's a little bit of a dance or a story that's unfolding, but you're interacting with each other. And so maybe you could talk a bit about how you try to tell a story by having these one-on-one interactions with the scarecrow.
[00:05:02.001] Sngmoo Lee: How did you develop the story in 1 on 1? We did a lot of rehearsals
[00:05:16.940] Jihyun Jung: So many rehearsals, through so many rehearsals and with Cooper and we... We did many discussions and rehearsals and trials and errors to make that alive in this form.
[00:05:36.145] Sngmoo Lee: But there were so many trials and errors and many, many conferences and debates. So it's kind of a collective idea and collective storytelling in a way.
[00:05:47.150] Cooper Yoo: What's interesting about this one-on-one show is that every show is different. It depends on who is acting and who is performing and who is our user. So I think that's one of the most interesting part that even today every show was so different. Like your show was different. We had a dancer who was one of our users. So I think like it's part of like building a story together. And I think that's like part of this whole Scarecrow project.
[00:06:14.552] Kent Bye: But there's also points where you're drawing lines in the experience. You have a snowman. You have these birds that are flying at you. So it feels like there's these events that probably happen in all the experiences, but you have a dance that is unique. But you're still telling a story. You're still having different moments that you're trying to hit. So maybe describe what those big points of the story were that you see that you were trying to convey with this experience.
[00:06:41.190] Sngmoo Lee: No matter what kind of technologies you are using is very important at this stage of the VR storytelling, but no matter what kind of medium you're using, storytelling is storytelling, and you have to know what you want to give the audience and tell the audience. So for me, to start from the Scarecrow, at the heart of this story is about The user saving the creator in a way. They kind of hanging in the cross and their heart was eaten by the fire where it could be a metaphor of the capitalism, dictatorship, censorship or your self-criticism. They kind of being zombie-like artist. The user, as a player, just coming in and saving you by interacting to your story, to your creation. I think that's the core of this storytelling, the core of the Scarecrow. And all these other elements, like games like Killing the Firebird and the Snowman, we had to be very creative to use the technology like Haptic. All those technologies should not drive the story. We have to take advantage of this technology and take that into storytelling form. And to your question, those kind of thing was actually invented out of the technology. What can we do with the heat technique? We have a heat, but this also can do the cold. So what can we do with the cold? Yes, snowman. But that's just technology for the technology's sake. But no, he was frozen there for 200 years, 300 years. And when you first come close to him, your hand's supposed to be cold. And you kind of de-ice him with the warmth of your hand. Then the technology becomes an important part of the storytelling. So it started as a limit of the technology. We tried to make that and blend that into a technology. Firebird and all the other things apply to the same principle.
[00:08:35.739] Cooper Yoo: So and what is interesting about interactive storytelling is so we have to tell a story while we have to get inputs from the users. So that's why we have those three parts of catching birds and drawing together. But I think it's always like important to have the balance between telling the story and having like playing together. So I think that's one of the like biggest part of our project of trying to see how interactive storytelling goes with having a balance between interactions and storytelling. How did you like it, personally?
[00:09:08.615] Kent Bye: Oh, I felt there were certain moments that I really responded to as I go into these different experiences. One, I didn't quite know if the Scarecrow, I heard it was a one-on-one experience, but it was kind of hard to tell. if the person was actually there because I was moving around and he wasn't mirroring me or giving me any indication as that if it was actually a person there. I also enjoyed blocking the firebirds because it felt like I had a role to play to help protect the scarecrow so I enjoyed that moment. I also enjoyed a moment where I was shrunk down so there was like a scale shift that was happening. I was big and then I got small. But the other thing I really noticed in this piece in particular was the facial expressions of the scarecrow. There seemed to be a way that they were capturing the facial expressions of that scarecrow. And I felt that actually added a lot in terms of being able to interact with a virtual character when I can actually see their face and their expression. So yeah, maybe you could talk a bit about that.
[00:10:04.151] Sngmoo Lee: One thing I'm very proud of for our team and Cooper is that yeah the facial recognition is very very well advanced and motion tracking and performance tracking is high up there and we did that last year and it's very expensive and needs a lot of manpower. What we try to do, because we are film professors and students too, we want to make this LBE or LBA, we call like location-based art, more accessible to the students and creators in the world. So we intentionally make it in a low profile, low budget, technically limited, but still have a lot of possibility kind of technology. Initially, so the facial recognition system is nothing to compare to what you're seeing in the Hollywood movie. It's based on iPhone and it only gives you like 52 faces. But still, we find a way to trigger that emotion, that 52 facial emotions which iPhone provide, right, Cooper? And take advantage and integrate that into our storyline. That was kind of the toughest part of this whole process and the actor did a great job. It's not like we are capturing the real performance of the actor's face. It's the actor's face trigger one out of those 52 sets. And it is much more difficult than you think. It's not natural acting. It's like you're acting and while you are controlling like marionette, you're controlling the puppet with your face. So it requires special skill and talents. And those guys pulled off very well and Cooper designed it beautifully.
[00:11:37.670] Kent Bye: Yeah, I'm wondering if you could talk a bit about that moment when I'm protecting the scarecrow from the birds and, you know, what you were trying to bring about and the viewer actually feeling the hand of the scarecrow on my back. I feel like that was a really powerful moment. Just wondering if you could talk a bit about that, of protecting the scarecrow from the firebirds.
[00:11:56.129] Jihyun Jung: In Korean, sorry. In Immersive Theater, I had to be very careful with my hands touching the user's body. But I couldn't do that because I couldn't communicate with the user. I couldn't see my hands when I was standing behind the user. So I had to be very careful with my hands.
[00:12:38.581] Sngmoo Lee: I'm going to add a little bit, not direct translation, but like 10% addition of my extra explanation. The problem of this low tech technology is they are not accurate. So it doesn't match the real world. And when you try to touch the scale cross hand, It's not going to be there. It's like 20 centimeters off. So it's going to be very awkward when you try to touch something, but it's not there. It totally breaks the immersion. But that's the path we were trying to go. And we still try to interact, physically interact with the characters. So how can we do it? Hiding behind your back, right? That kind of thing, that kind of shoestring, low-budget solution to make the storytelling moving is the part we are most proud of. If we had much more money, we could do it in a better way, but we kind of came up with the idea, works better. in a way because with the high-end technology we probably didn't think about the hiding our actor behind your back maybe we want to make them play together but since we cannot afford that we try to find a solution actually we came up with the solution yesterday The day before yesterday, there was no hiding behind the show. That's the beauty of this immersive theater. We kind of look at the people performing, and we know something's lacking, and we try to solve the problem. And how about we just hide? They really want to touch the scarecrow. The scarecrow wants to communicate physically with the audiences. So, okay, let's hide behind it. Everybody loved it. And so that becomes the most powerful moment. And that's like creative. collective storytelling between creator and user and their data. And I think that's the beauty of storytelling in this way.
[00:14:24.308] Kent Bye: Yeah, another powerful moment I had in the experience was when I actually reached out and touched the snowman, and I felt cold, and I was like, wait a minute, what's happening? Because something was put on my hand. I didn't necessarily know what it was. It felt like a brace, and I was like, oh, they're just using a brace for me to put this other sensor on. But maybe you could talk about that process of designing an experience that could have actual hot, because you have the heat as I'm killing the firebirds and protecting the scarecrow. I have the heat as well, but there's also the cold. So maybe you could talk a bit about the technical challenges of adding the cold and the hot into an immersive experience.
[00:14:59.777] Cooper Yoo: So one of the reasons why we used these kind of like thermal haptic feedback is because in immersive theater, in VR, mostly it's about like sound and visuals. But we also wanted to add some like touches for the feedbacks. And one of the biggest challenges for these heat pads was that human skin doesn't actually feel a certain amount of heat. It's always like a combination between time and also how long it takes to touch the area and how big area you're touching. So I think we had some challenges trying to find the magic numbers of when people feel it's hot or warm or cold. And of course, in storytelling, We had to add some things to like the situations that we want to show that we can do cold and hot. So I think that was like a different challenge, but I think it was still very powerful.
[00:15:56.212] Kent Bye: And there's this thing before I go into the experience, having my face captured with the different expressions. And then later in the experience, there's like a point cloud representation of my face in the experience. And so maybe you could talk a bit about what you were trying to do with including my face in the experience and what you were trying to do there with having different emotional expressions there.
[00:16:20.732] Sngmoo Lee: Yeah, emotional expression is one of the reason for that is for the technical reason. Some face in our $100... $400. Yeah, we are using like $400 low-end capturing devices and some face we get it very well representing your face. Some capture doesn't really do that. And everybody has different facial emotions. So there is certain amount of face really works in this kind of scheme. So that's the reason for multi-full scan. And the reason to put your face in the story is, again, like you said, the heart of this project is how can we tell the story? How can we make the story together with user and the player and the creator, writer, director? And we really want the user to be a part of this story. And it's not just interacting together. The story, I don't want to spoil it too much, but it's about capturing the best moment of this short friendship and keep the memory out of it and represent it in the sky. only can happen in the virtual reality world. So I really want the audience or user to be inside the story, not just metaphorically, but practically and visually. So we try to figure out a way to represent the user's face, which is the most important recognition part of a character or a human being. and blend that into the storytelling. So that was kind of most difficult part and most crucial part of the storytelling to me. And it was really difficult to pull up, especially with this kind of shoestring budget. And I think Cooper and the director pulled it off in a way. And many people seem to like it. So I'm very happy.
[00:18:07.945] Cooper Yoo: I think it's part of our biggest goal of this project that we want to have immersive media and then we also want the user to be part of the story. So I think that's one of the reasons why we wanted the user's name and face being inside the story. That was the whole story of trying to do these kind of technical challenges.
[00:18:29.237] Sngmoo Lee: That's one of the big advantages of this immersive theater, which is kind of live broadcasting. If you send your photo somewhere and if you see your face in the broadcasting one month later, it's nothing to be amazed by, right? But this immersive theater only happens during 10 minutes or 15 minutes, and it's very difficult to represent yourself inside the storytelling. That was the toughest part and most effective part, in my opinion.
[00:18:56.572] Kent Bye: And so, because you have a background in 2D traditional film, I'm wondering if you could compare your experiences of making 2D films with making immersive VR pieces.
[00:19:06.955] Jihyun Jung: In the case of 2D movies or animations, you can edit them in the latter half of the film. But in this case, you have to edit them in real-time, so it's very unique. It's a difficult point, but it's a fun point, and I think it's a different point.
[00:19:25.657] Sngmoo Lee: Yeah, in traditional filmmaking, you have a term called post-production, which many things can happen there, and you can fix it later on. But in this medium, especially in this kind of immersive theater, you really have to fix it on the way, and everything happens on the way, simultaneously. That is the toughest part, also the most interesting part for her. And for her background, she didn't know anything about VR six months ago. And now she's here in the New Frontier as a director. So she's such a brilliant director, I must say.
[00:20:00.255] Kent Bye: Yeah, and the other thing I think you're able to do in the VR is to build these worlds. And so you have a volcano, you have all these other scarecrows that are spread out all over the place, and you're kind of on this floating island. And so I'm just curious if you could talk a bit about the world that you created and what you were trying to invoke in this world that you created.
[00:20:22.075] Sngmoo Lee: I better answer the question because initially it was much darker than this and first when I thought about this I imagined more artistic in a way and actually Shari Frillo who is a programmer of this new Frontier after she experienced that she was like it's much lighter than I thought, right? So surreal was the first word to me. Because virtual reality, we are not there yet. There is no reality in the virtual reality. Either it becomes very cartoonish or it has to be abstract. So for me, I went to the other way. Surreal world which still works with the limitation of the virtual reality engine. So it was kind of given to me when I first thought about this project. Surreal world with the surreal character. So that kind of creating world was very important to me and also this creating world has to be believable. You are in a fantasy world so that all those hundreds of scarecrows and the firebirds and the very surreal volcano, that was very crucial part of the feeling that you are in a different world yet still believable and good enough for you to be engaged in. So that design was very important, and she played a very important part in designing that. Because she's a 2D filmmaker, but also her background is design. She's coming from a multimedia design background, so she brought that graphic aspect a lot into this project, and I really appreciate that.
[00:21:50.642] Kent Bye: You have more to say about the world that you created?
[00:21:53.778] Jihyun Jung: I think so. At first, it wasn't that scary, but it got darker for a more dramatic effect. The sounds are a bit dramatic, so it starts with a scary and horrible feeling, but in the end, it leads to a love story, so I think it's a unique love story.
[00:22:25.512] Sngmoo Lee: she's saying exactly opposite thing which I just said I said darker to lighter and she said this is much darker than she thought so and she said what her intention was to create some darker and kind of gothic world kind of scary world and then make them into a lighter love story So for her, that balance between the lightness and the darkness is important. And that's how we design and she designed to make that little bit look, changes, look shift from the gothic to a sort of like light, warm-hearted love story.
[00:23:00.987] Kent Bye: Yeah, I think one of the things that, as I was experiencing the piece, I have hands, and one of the things I would look for when I have the ability to interact is be able to play with particle effects and other things. And so I'd say if there's one thing that I'd want to see a little bit more of is, as a user, being able to interact with the world, because, you know, we were talking briefly, and you'd said that, you know, there's a challenge of how do you know for the user when to interact and when to passively receive and so you have the immersive actor and then they're gesturing but finding ways to give cues that these are things that you can interact with or not interact with.
[00:23:35.240] Sngmoo Lee: First of all, sorry about the incomplete version of this one. So the heating pad is supposed to be that kind of trigger to know that when to use your hand and when to just be passive. And it didn't really work 100%. And also the graphic, your hand is supposed to be changing from something to something. Just let's say 70% of it is there. So it's a little bit disorienting. And biggest decision making for me was whether to use the controller, Vive controller or not. If you use the controller, you can trigger the vibration. So that, okay, this is your turn. Play it, not. I mean, it's so easy to do it. And we have a heating pad actually can be combined to the controller, Vive controller. But for me, the Vive controller is such an abnormal device to be naturally interactive. so we just gave it away we know it is going to be better trigger for that kind of interaction just be quiet or be active this is your turn but we rather keep it a little bit more ambiguous and have better moment with your hand killing the birds rather than controller catching the bird because when you hit the bird i'm sorry this is a spoiler but not much of you are going to experience this so i'm just going to tell it when you hit the bird with your palm you sense the heatness on your palm, and that kind of experience can never happen in the Vive controller. It's very different. We tried that, and we had to choose either that kind of vibration or just palm, and then we chose the latter that gives less trigger for the audience member when to interact. But again, it's incomplete. I hope that in the next version, less users feel that awkwardness. I hope.
[00:25:09.602] Kent Bye: Well, there's a lot of things going on with a lot of sensors, two people in the same experience. What were some of the biggest challenges you had technologically to be able to make it all come together?
[00:25:20.772] Cooper Yoo: Well, so yeah, in the end, we're happy. Most of the parts are working OK. But I think the biggest challenge was like, so we were still talking about one-on-one interaction, right? And continuing with your question about how to give more freedom to the user. So it's good to give the user and the actor to have more freedom and to do more interaction each other. But then we can't continue with the story and sometimes because of those technical issues such as they cannot touch each other because like the graphics are not perfect. So it'll like break the mood. So I think technically we just wanted to make everything work okay. But we try to also work on making different situations that people don't feel those technical failures. And that's, I think, it's actually a continuing with your first question of you were talking about that we couldn't really tell when or how the user interact with the world, right? The users cannot interact with the particles. So were you actually expecting more, like to do more inside the world?
[00:26:25.590] Kent Bye: Well, I think that when I have hands and I have like particles I'm always testing like what can I interact with and so it's more of a Aesthetic decision, but also like I feel like when I can interact with the world It gives me more of a deeper sense that I'm in the world and when I can't interact with the world Then it makes me feel like a little bit more disembodied or and that it's sort of an abstraction of I mean, I did actually try to reach out and touch the scarecrow a few times, and I could tell that that was an actual person that was there. But the tracking was off, and that's why there was a bit of, for me, as a person in the experience, there were some moments that was actually difficult for me to tell whether or not there was an actual person there in real time. Because if there's some sort of direct mirror of my actions, then that, for me, is a cue that that person is there. and because there was sort of like disconnections there then it was sort of I couldn't actually tell when that everything was kind of just a pre-recording and it was just sort of like I was reacting to something that was there so I feel like those little cues to let me know when I can engage and interact but also as I'm going through as I'm going back there was like these different moments where I wasn't sure and I'm kind of interrogating and testing the experience to see What can I do? What can I interact with? Because that's the challenge of interactive storytelling is at what point can you engage and interact? And what do your interactions do? And even if it's like moving the particles around, at least that's like some impact that it's a level of embodiment that makes me feel more connected to the experience, even if it's not doing anything to the story or anything like that. So just finding ways to make those connections.
[00:28:00.480] Sngmoo Lee: So technical limitation and also this is still the main storylines going on and I don't want to distract the audience from the main storyline but you're 100% right when to be observer or when to be a player is very difficult part of the virtual reality storytelling and then I agree that many people and we intentionally introduced by letting your hand do something by touching his heart or just touching the snowman and then we suddenly deprive that privilege from you and you feel a little bit awkward so that's a very good point and we probably if we ever make the next version of it that's something we need to look into and make that interaction more natural as in the real world And speaking of the hard work, I was going to packing and go back to Korea three days ago, because it was just not there yet. Three days ago. We tested Korea and different environment and different magnetics and different wireless. So we're almost like ready to go. And somebody was not sleeping. You could tell for two days. And that's this technical team's hard work. Right. And we pulled it off. I feel very good.
[00:29:11.479] Kent Bye: Awesome. Great. Well, finally, what do you each think is the ultimate potential of virtual reality and what it might be able to enable? So where's this all going? Where do you see VR is going to be able to enable?
[00:29:26.803] Sngmoo Lee: Right for me, when I first introduced to the VR about like six years ago, five years ago, and as a traditional filmmaker, and a little bit technically oriented filmmaker, I kind of knew that this was going to be a dream for the storyteller. What they wanted to do for like 2,000 years or 3,000 or even like 100,000 years ago when you first paint that little animal on the cave, you want to create some world and invite the audiences or your readers, watchers. to live in that world that was a storyteller's ultimate goal and that was my goal as a filmmaker and now we have that too it's a little bit limited it's still in its infancy but i think will not end everybody compare this to 3d cinema or something maybe this pace is done maybe we're not going to see the virtual reality production in the next 10 years but it's going to happen because that's the human race's dream and the goal to create the world and invite the audience to it so i think this is it this is it if this works maybe like theater is there even after film is invented but as a major storytelling form It's not like 19th century. It's just very good, very important art form, but it doesn't just function as the mainstream storytelling format, visual storytelling format. It gave away their privilege to the film. And it's been like that for 100 years. and now the period is over I told my students that if I were your age I probably wouldn't touch the film to this cinema you know it's just my my generation and new generation is opening why are you still hanging in there you know that's still is much more powerful than VR but not many longer I think a new new age already opened and it will be that way in 10 years I think And that's why I kind of introduced talented students like Jung to get in touch with this new technology and let the imagination alive in this new medium so that 10 years after she's gonna be the major storyteller in the new format, new medium.
[00:31:36.719] Jihyun Jung: What do you think about VR? VR is really interesting. I will continue to make movies in a traditional way, but I don't think I'll miss it. There are so many attractive ways to do things that can't be done in conventional ways. On the other hand, if I can convey more sensitive emotions that I could only do in the original movie, that's what I want to do and what I want to create.
[00:32:17.340] Sngmoo Lee: Yes, VR is very interesting and this is not going to be her last VR piece for sure but still there are certain parts which traditional filmmaking is much more effective and such as delicate emotion which VR lacks. Her goal would be bring those kind of delicacy and the emotional impact to this VR and that's what she's aiming for.
[00:32:41.645] Cooper Yoo: So for me, when I was a kid and even now, I always wanted to talk with TV. So I always wanted to be part of the story, and I always wanted to change the story together. So I think VR was a good chance for me to also try to do those interactive storytelling. And this project, what I really liked about it is that we're using live acting, which means it can react to the user. while traditional film or even traditional VR, it was always pre-recorded, so they always had to have ready. So in the future, what I'm really expecting in the future is to have artificial intelligence so that we have a really more smart character. Even though it's a pre-recorded character, a more smart character that can also interact with the user so that user can feel and they can create the story together. That's what I'm really expecting from VR and also with using artificial intelligence in storytelling.
[00:33:42.701] Sngmoo Lee: That's one of the reasons we are sticking with this special immersive theater. This is my second immersive theater project. I'm not a theater person, but what lacks in the VR nowadays is that real interaction. Algorithm is not there yet and it will be replaced by AI. you know, five years or ten years. And then it will become very interesting and different medium. And I think that's where the future goes. So this immersive theater is very, very, very money and time consuming. Ten, twelve steps, serving for one audiences. So it's not going to make money. But the core of the immersive theater will be applied to the AI storytelling. And I think that's a real potential of this project, this kind of project.
[00:34:28.978] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the larger immersive community?
[00:34:35.002] Sngmoo Lee: Well, I always envied those filmmakers who were born in 1895. They didn't know what they were doing, but they were making the new medium the most powerful storytelling tool to the world without knowing what they were doing. And they just did it and pushing the envelope without knowing what they're doing and there is such a great sense of community that they are doing something great together and after 100 years yeah we are much more refined and much better grasp with the medium but I really envy those guys who first got in touch with this new medium and build the grammar and build that into an art form and the storytelling form I think we are in the same stage and I feel really strong unionship and strong partnership with all these immersive creators around the world who's not making a lot of money but they know that they're doing something meaningful and important. So we are together. Let's make it together.
[00:35:34.870] Cooper Yoo: So yeah, I'm glad. We just started our new adventure of using these kind of new technologies. So yeah, I hope, like everyone else, to join us. And let's have more fun.
[00:35:48.059] Kent Bye: Awesome. Great. Well, thank you so much for joining me today on the podcast. So thank you.
[00:35:51.161] Sngmoo Lee: Thank you very much. Thank you.
[00:35:52.382] Cooper Yoo: Thank you very much. Thank you. Bye.
[00:35:55.178] Kent Bye: So that was Sungmu Lee. He's the producer and writer, Chiyun Zhao. She's the director of Scarecrow and Cooper Yoo, the technical director of Scarecrow. So I have a number of different takeaways about this interview is that first of all, well, I know that this is a type of piece that changes for everybody that sees it. And I know that I saw it on the first day and. over the course of the week, I think they were changing it and adapting it and integrating different things. You know, one of the things that I really wanted to see was that I was actually interacting with somebody. And I think after I experienced it, just hearing other people, I think they started to do that a lot more in terms of giving the person who is experiencing this a lot more cues that they're actually someone they're live, giving you the sense that there's somebody there and it's not like this pre recording. So probably the most powerful moment of this piece for me was when the actor went behind my back and was touching me on the back and I was defending the scarecrow from these firebirds and you know having the heat on my hand and it was like a very intense moment but also felt like I was stepping into a role where I was protecting this character from being killed and that was actually a very surprising moment just to feel like I had a role to play and that I was embodying this protector role within this scene. Thermal haptics I think were interesting although you know I think that it was very contrived on one hand you know having like a snowman clearly the snowman there is solely there just to allow me to feel that coldness and there may have had other elements where they're using the cold but like he said it takes a long time for you to actually tell whether or not things are changing. So if they were using it in other moments, I didn't necessarily recognize it. Then, you know, this is just something that I think is a new technology. It'll be interesting to see how this continues to be used. Having this sense of hot and cold is something that can be powerful if it's integrated well. I think the firebirds did a great job and the snowman was sort of like, uh, you know, I don't know if that necessarily worked. It just seemed kind of arbitrary. Now, you know, how do you actually tell a story when you have this actor as a scarecrow that has to dynamically interact and exchange? And you know, I would need to see how that played out for a lot of different people just to get a better sense of the different strategies that they were using in order to do that. There seemed to be a number of different interactions where, you know, you're drawing different things together, you're kind of playing this limbo game, you change scales, and there's the different interactions. this seems like overall a project where they were being driven first by the engineering and the technology and the creative possibilities, and then from there putting on different narrative elements on top of it, even up to the last minute. Simu was saying, you know, they didn't have the whole mechanic of the scarecrow going behind your back and being protected until, you know, like just a few days beforehand. And so this seems like a type of project where they're starting with the technological innovations and then saying, okay, now what's possible with this? And this is, I think a good example of people who go see this and then they say, okay, I have this experience and you know, this worked and this didn't work. And then from there going on and being able to see how to integrate this technology in the future. The other aspect about this experience that I thought was interesting was that, you know, there was a mismatch between the tracking technologies. And so, you know, you want to be able to have this, you know, one-on-one engagement with the character and because the tracking wasn't one-to-one aligned, there was sort of offset that limited them in terms of what they could do with interactions and that's part of the reason why I have the scarecrow go behind your back because you know the tracking doesn't matter as much and you can feel the presence of the scarecrow's hands on your back and this whole technological limitations that come up in a project like this and seeing how you can work around them I think was also an interesting aspect. And, you know, I think the facial capture was also interesting just to see how they were using an iPhone to be able to capture a whole other person who wasn't the actor. It was somebody who was a dedicated facial expression who was trying to track the action that was happening, but being able to have a higher fidelity, emotional engagement with what was happening in the scene. You know, this is obviously something that it takes six people to be able to run this experience in. Actually, this is one of the experiences at Sundance that was the hardest to see just because it had so much logistical input from these six different people. But I think it took 20 to 30 minutes to get through it. So there wasn't a lot of people that were able to get through it each day, maybe seven or 10 people, somewhere on that order. So maybe upwards of 100 people or so, then we got to see it over the course of the 10 days. So the scalability of this type of experience is not very feasible, but I think one of the things that Sugmu was saying is that, you know, they really want to be on this path towards creating these artificial intelligence, virtual beings, and that in a lot of ways, having these one-on-one immersive theater actors, you're able to test a lot of the different techniques and strategies for dealing with different people, different temperaments, different actions. And can that be abstracted in some way? And can there be a way of doing this mirroring to make you give the sense that there's somebody actually there when it's actually completely generated by artificial intelligence. And so can they start to move in a system like this in order to get the sense of what are the deeper archetypal elements of these different types of interactions? And then can that be abstracted out into computer programs and to allow the storytellers a lot more latitude to be able to tell these stories that have these types of intimate interactions that happen between two people? So that's, I think, an interesting path of this type of project because it's a first iteration, but very early Genesis. But as you move forward, then this interactions with different people in the VR, in this case, you actually co-present with them and potentially have these one-on-one embodiment interactions and being able to hug a character within virtual reality. You know, having that sense of someone's hand actually being on your back, it's very powerful because you have this sense of like, this is a virtual being, but it just helps ground you into like, you know, this is actual character and you're, you're embodied and you're there. And it, I think it just changes something in your embodied experience when you have that personal touch from another human being within a virtual reality experience. But as we move forward, obviously this is going to be scaling out into different aspects of we're going to have different location-based entertainment with these different types of intimate one-on-one interactions. But if you were to try to deliver this to someone's home, could you then abstract the different interactions? of this actor and then be able to actually record the embodiment and to give you the sense that you're actually interacting with what feels like a person when it's actually like artificial intelligence driven. So I think, you know, when we look at the wolves in the walls, they're already starting to do a lot of that, you know, working with immersive theater actors and be able to integrate that into the different interactions and come up with this type of embodied interactions. And I think this type of narrative is still very early and very new in terms of like how do you tell a story through these different type of gestures and interactions and something where there's not a lot of dialogue but you're just having these one-on-one embodied interactions with another person. We see this a lot within different immersive theater pieces like Sleep No More and especially Then She Fell which is really trying to focus a lot more on this one-on-one intimate interactions. So this project has a lot of really interesting technical innovations and really pushing the edge of the tracking technologies and having another immersive theater actor in there, having the facial tracking as well as thermal haptics. And at the very end, they put your own face within the experience, which again, I'm not necessarily sure if that, you know, really cohered within the world that they were building to have like a hologram of your face in the sky and showing you some emotions that you showed earlier. I think it's an interesting idea, but it also just feels like being driven first by the engineering and the technological capabilities. And then, you know, trying to backport the different story elements on top of it. I'd like to see a little bit more of starting first for what type of feeling you want to try to cultivate and to see what kind of feelings that these technologies create. And then from there, start to create a very specific experience because some of these things, it feels like it's more of a tech demo than, you know, something that was being driven by the story itself. But overall, I think just really super impressive fusion of a lot of these different new technologies and a real experiment of trying to do this type of intimate one-on-one immersive theater type of interaction from folks that are coming from primarily a filmmaking background rather than a theater background. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a list of supportive podcasts. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. And I'm also going to be starting to have different lectures and gatherings within my Patreon as well. And so if you want to learn a little bit more of the principles of experiential design or immersive storytelling, then sign up to the Patreon to get more information on some video tutorial series and workshops coming up here soon. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
