There was an amazing projection mapped, immersive theater piece at Sundance this year by Heartcorps called “Riders of the Storyboard.” Trained street performers interacted with a virtual projection-mapped 2D objects, and through the slight of hand of magic broke these flat objects into the third as glowing 3D props. There were 15 people packed into a small room with about half a dozen performers for a 13-minute show about a these 2D characters who interact with the performers who are playing Alchemy of Light gods in the third dimension. It was an awe-inspiring performance, and the projection mapping technology provided a shared augmented reality experience. Heartcorps is proving out some of the techniques with projection mapping technology that should also work really well in the future of live performance and immersive theater designed for augmented reality glasses.
I had a chance to catch up with the Heartcorps member and performer dandypunk, who talks about their process, ritual inspiration, and mixture of immersive theater and cutting-edge projection mapping. Be sure to check out the trailer and clips from their show at Sundance down below.
LISTEN TO THE VOICES OF VR PODCAST
Here’s the Trailer for the Heartcorps “Riders of the Storyboard” that showed at Sundance New Frontier:
Final three minutes of the “Riders of the Storyboard” show at Sundance
If you’re interested in immersive theater, then be sure to check out Noah Nelson’s No Proscenium podcast. I talked with him in episode #95 about my Elemental Theory of Presence as well as the cross-over between VR and immersive theater.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR Podcast. So at Sundance this year, there was an amazing projection mapped experience that was happening within the Sundance New Frontier section, and it was by hardcore and it was just absolute magic. You could just imagine street performers who were interacting with these project and mapped light show that was being broadcast onto the wall and they were interacting with it as if they were engaging with it but it was all kind of pre-scripted and they were just kind of being at the right time at the right place but just some of the ways that they were able to interact with the light as well as manifest objects out of their hands as if they were kind of pulling it out of this 2D light world into the 3D reality It just was an amazing performance and an all-around amazing show. And in a lot of ways, what they were doing was blending this concept of immersive theater with what will eventually be done with augmented reality technology, but at this point, they're using projection map technology. So I'll have a video in the links in the description of this podcast, or you could look up the Hardcore at Sundance 2017 video on YouTube and just watch some clips from it. It'll give you a sense of what it was like, but it's nothing like being completely immersed in the environment they were doing. So I had a chance to talk to Dandy Punk about their process and what they're doing at Sundance. That was happening in Park City, Utah from January 19th to 29th. So with that, let's go ahead and dive right in.
[00:01:50.512] dandypunk: I'm Dandy Punk. I am a multimedia artist that specializes in projection mapping and live performance. And what we're doing at Sundance is an immersive projection mapped experience using projection onto hand-drawn illustrations. to tell a story kind of as if you're walking through the pages of a giant graphic novel. Characters on the walls interact with the illustrations and occasionally they will come to life in the third dimension as real-life performers using street dance, acrobatics, breakdance, freerunning and circus arts.
[00:02:29.500] Kent Bye: Yeah, it's quite an eclectic mix of a lot of different things, and the thing that was most striking to me was seeing these projection map lights, animations go across the walls, but there's a sense of timing and coordination that you have to be able to interact with it in kind of an illusionary way, because you're not actually interacting with it. It's sort of pre-scripted, but you're at the right place at the right time, and that, I think, is what gives it its magical kind of feel.
[00:02:55.155] dandypunk: Exactly. We're not always exactly at the right place at the right time, as you've probably seen. Obviously, we would eventually love to make it all actually reactive and have the audience be able to interact with it themselves, so they could help the story along by helping the characters, perhaps drawing lines or ladders that enable characters to cross chasms, that kind of thing. So really, it's just down to my low-tech abilities of not knowing how to program things to be reactive that we went with the choreography route. But we're all sort of professional performers since childhood, so we know how to be in the right place at the right time.
[00:03:32.559] Kent Bye: So this is, I guess, the medium of live theater, I guess you might call it. Or how do you describe what you do?
[00:03:39.604] dandypunk: It's just a mix of all my interests, really. I grew up breakdancing and doing parkour and street art and graffiti and just sort of mixed them all together, tried to find a way to be able to do them both at the same time, which the projector allowed me to do. But yeah, I mean, like most people, we're just trying to make a fun experience that puts people out of their everyday life into another dimension.
[00:04:06.025] Kent Bye: Yeah, and it reminds me a lot of the immersive nature of Sleep No More, which is a theater production where you're not sitting down and watching it, but you're actually in a room walking around and, I guess, deciding where you're going to look at from the perspective. And there's lots of things happening. And so maybe you could talk a bit about how you're designing this type of experience.
[00:04:24.422] dandypunk: So Sleep No More is obviously a huge influence on anyone that's involved in immersive theatre. I was super inspired by them and I saw Felix talk at the Future of Storytelling in 2014 and that was quite a life-changing moment. But the difference with the way we would produce this on a larger scale is that we wouldn't have people roaming wherever they wanted and exploring as freely as in Sleep No More. mainly because we have these live performers doing very, very high-skill, high-energy acts, which we wouldn't want people to miss. So it would be more of a guided experience, perhaps only over six to eight rooms, rather than the hundreds of rooms that are in the Punch Drunk shows. So that way, people would see what we wanted them to see, while they can still look around the room and find interesting things that other people might not find.
[00:05:16.508] Kent Bye: Yeah, and talking to the director of Lawnmower Man, Brett Leonard, he was talking about how he's looking towards more indigenous cultures for ritual in terms of him saying that virtual reality is actually a medium where you could start to induce more ritual. And you have a lot of ritualistic elements into your piece. I just wanted to hear kind of your thoughts on what you're doing there.
[00:05:38.975] dandypunk: Yeah, well, I think all the key players in this company have all had their own sort of spiritual awakening at one point or another, whether it was through meditation or I'll just go and say it, dimethyltryptamine, which has been a huge eye-opener for a lot of people. I think they're involved in this new kind of technology. I'm seeing it more and more. And we've seen that these parallel universes do actually exist, whether they're real places or they're just in our subconscious. We're just trying to see how we can, as Terence McKenna, the great Terence McKenna would say, bring these images back to show people. So I think that's really what we're trying to do.
[00:06:23.625] Kent Bye: There seems to be a lot of sacred geometry elements that you're including here. Is there a narrative theme that you're trying to also express? Because it seems like you're trying to, almost like a shaman, go into these other realms and then bring back the lessons of that and give people access to that in some way.
[00:06:40.799] dandypunk: Yeah, exactly. I mean, a lot of people haven't sort of had an awakening or done these psychedelic experiences. I mean, it's happening a lot throughout the whole culture. We're seeing more of these symbols and references to magic in music videos and pop culture. I think just the idea of magic with projection mapping being as close to magic as we can sort of get visually in this reality is what we're going for.
[00:07:11.062] Kent Bye: Yeah, it's just a lot of wonder and awe. You know, I think that's the thing that, you know, whenever you have this cube that's on a 2D plane and you suddenly pull out a cube in 3D space, it's that like, whoa, that moment of magic that I think that is unique in a live performance. Yeah, I think in virtual reality, I think there's going to be a lot of insights that I think are going to be coming from live theater in this type of performance. And I think it's going to be very difficult to do the level of acrobatics and spinning on your head and all sorts of craziness that you have. But the blending of realities, I think, is what you're doing that's really interesting, where you're having physical props, but you're working with a virtualized space. But you have this dance that you have where you know that it's a virtualized experience, but yet you are participating in it. It tricks your mind into believing that you're in this fantasy world.
[00:07:55.555] dandypunk: Yeah, it's all about the suspension of disbelief, I suppose. The show has a lot of themes around dimensions, actually. The two-dimensional characters that live in the walls. It's similar to the way that we can't really comprehend the fourth dimension or know what people that live in the fourth dimension would look like. The characters that live in the walls as flat, light beings wouldn't be able to comprehend us as three-dimensional beings. So they'd see us kind of as god-like creatures. If we drew a line on the wall, that would feel like a kind of divine act of creation. If we blew on them, they would feel it as wind. So there's a whole kind of dimensional aspect to what we're doing as well.
[00:08:39.940] Kent Bye: Wow, that's really interesting. We're in a 3D space, you're interacting with 2D characters, and you are the god in their sense, because they can't see you. You're interacting with them, and they're experiencing magic. And then, I think in VR, we can actually start to take that to the next level, where there's four-dimensional creatures, in a way, coming down. So, yeah, I'm just curious to hear your thoughts on the ultimate potential of this type of medium that you're doing here, with this type of performance.
[00:09:04.178] dandypunk: For me, I'm not too sure how to translate this into VR. AR, it's completely obvious, and that seems to be the way to go, because we're already looking at little characters which aren't there in real life. So AR seems to be a better fit for this, because then we can still keep the tangible human element of real performers and paper and ink on the walls. But I'm really interested. I've talked to a lot of VR people, and they all seem quite excited and potential for it. I just don't know what it is yet. So if anyone has any ideas, I'd love to know.
[00:09:40.486] Kent Bye: Awesome. Well, thank you so much.
[00:09:41.527] dandypunk: Thank you. Thank you.
[00:09:43.448] Kent Bye: So that was Dandy Punk of Hardcore. And they were doing an amazing projection mapped experience that was showing at the Sundance New Frontier. So I have a number of different takeaways about this interview is that first of all, I think the way that they were blending their immersive theater performance with the projection map I think it's going to be a little bit of like what augmented reality is going to be at some point. And there's something about the embodied performance and being in the same place as somebody, but also having these virtual objects being engaged with and interacted with, you know, at some point they're going to be able to have these dynamic interactions with these virtual worlds that everybody's going to have a shared consensus reality if everybody has their AR glasses. But having a projection mapped experience was something that was probably the closest that you can get to right now as a collective augmented reality experience. But there's something about that immersive theater component where you put everybody in the same room and you have performers that are embodied. You could see their faces, their emotions, and they're doing a street performance. And I think there's something about having a real human there doing a live performance. That's just qualitatively different than if something's recorded and completely virtual. This interaction between the human and the live versus the virtual, I think is a very interesting combination. And I had a number of different experiences that I've had since then that were at Tribeca this year with Draw Me Close, which was starting to do like this blending of virtual reality with a live immersive theater. It was a one-on-one experience. And this experience, you got about 15 or 20 people in the same small room, and you have about maybe five or six different performers. And you're kind of shifting around the different room in order to get a good view of wherever the action is happening. And they're kind of moving around and focusing at different things within the experience. But because there was so much intricate story that was unfolding by looking at the walls, it was something that you could actually go into when there wasn't a live performance. You could kind of watch some of these similar animations play out. such a rich experience to be completely immersed in this world with all these different various scenes happening in parallel that it was really way too much for you to take in into just one experience so it's something that you can kind of come back over and over again and now that you kind of see that these two-dimensional beings that are being featured or interacting with these like 3D gods I kind of see that as this metaphor of thinking about what it would be like if we were interacting with these four-dimensional gods, that we would have as much of a conceptualization of what that was like for those 2D entities to understand what our 3D dimensionality is like. And so they call themselves the Alchemists of Light. At the very beginning of the experience, they opened it up as a ritual, and at the end, they were closing it as a ritual. So I think there's also something that's very interesting when you look at some of the culture that is coming from either these shamanic practices or indigenous cultures or psychedelic culture with ayahuasca or DMT, where there's a little bit of a sacredness and opening and closing these portals into another world and that was something that was very deliberately built into the script of the experience and it was just interesting to see that culture be blended into like the high-end technology as well as into like this more mainstream Sundance type of culture and environment. But I think that talking to Danny Punk what he was saying is that there's a lot of the key members of Hardcore who had had a number of different psychedelic experiences with DMT. And there's a documentary on Netflix called The Spirit Molecule which documents these people who went through these medical tests who would smoke DMT and basically got shot out into these other realms. It sounds like a lot of the people involved in this production had quite a number of different experiences with DMT. and were having that lived visceral experience of these alternate realities and they were taking those insights they were getting from those experiences and then putting it into this hardcore experience and so for me it was just amazing to kind of see that different sacred geometry and symbolism and images play out in this mixed reality environment. And it was definitely the best projection mapped augmented reality immersive theater experience that I've seen. So I'm excited to see where Hardcore and Danny Punk take this in the future. I think they're really kind of pushing the edge of what's possible right now with the technology and that they're kind of laying the groundwork with what I think a lot of the entertainment experiences with augmented reality are going to look like in the future. So that's all that I have for today. I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends. If you have a friend that's into immersive theater or thinking about what's possible with augmented reality or virtual reality, send them this episode to listen to, to check out some of the videos of what's happening with Danny Punk. And if people haven't heard of the podcast of No Proscenium, definitely go check out that. Noah Nelson's doing an amazing job of just documenting and recording what's happening within the immersive theater scenes, both in New York, as well as in Los Angeles. So check out the newsletters that he's got in New York and Los Angeles, as well as in San Francisco, and I think maybe even Chicago and some other places. More and more immersive theater is coming up in many different cities and so check out the No Proscenium podcast as well as newsletter to kind of keep up to date as to what's happening with that scene. And finally, if you want to help support the Voices of VR podcast and to help support me and chasing my passions of traveling to all these places and having all the experiences and talking to all the people so that I could share these insights and wisdom back with you and the larger VR community, then please do support the work that I'm doing here. You can go to patreon.com slash Voices of VR and just a dollar or five dollars a month makes an absolutely huge difference. So donate today patreon.com slash voices of VR and thanks for listening
