Alex Coulombe’s Agile Lens takes an interdisciplinary approach to immersive technologies at the intersection between theater and architecture. He was showing off a demo piece called That Kind of Guy at XR for Change, which explored a variety of different motion capture solutions including Depth Kit & Face Cap Motion Capture. We also talked about his recent trip to the Theatre Communications Conference where he was evangelizing different ways in which theatre companies can start to integrate immersive tech at different stages of production.
Coulombe also talks about some of the general approaches to interdisciplinary collaborations that’s brought together interior designers, industrial designers, landscape designers, technical theater designers, & architects. Some of the hot topics are how spatial design is merging with game design and interaction design, which is the cross section of immersive theater. He’s a participant in the Alive in Plasticland immersive improv theatre troupe as well as a collaborating artist in Kiira Benzing’s immersive theater piece Loveseat, which is premiering at the Venice Film Festival next week.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So this episode is the last in a series that I've been doing on looking at the future of immersive architecture. The conference that I went to at the Architectural Association and just talking to different architects who have been trained in architecture, working in the virtual reality space, there's just something about the way that architects are looking at these different design problems from both an interdisciplinary perspective, but also just a broader spatial design perspectives that they bring into the space. And so I've found that over the last couple of years of talking to these different architects, I've been just really drawn into both their perspectives and to see the deeper design principles about spatial reality. So Alex Coulomb kind of embraces that interdisciplinary fusion where he's coming from these different backgrounds of both theater as well as architectural design and sort of blending those two in terms of how there's different aspects of set design but also spatial design and storytelling within virtual reality and these different spatial mediums. He's kind of at the forefront of going into these different theater communities with his architectural background and this kind of hacker mentality of trying to bring together all these different immersive technologies and constantly experimenting with what's new and latest and what's the potential for storytelling and be able to solve specific problems. So I had a chance to catch up with Alex. This is actually the second interview that I did. The first interview that I did was with him at Magic Leap, LeapCon. It's a series of interviews that I'll probably dive into as a series, just diving into all the different augmented reality discussions and the creators and independent developers that I had a chance to talk to at the Magic Leap LeapCon, which happened about a year ago now back in 2018. And so I'll be diving into that at some point, but I feel like this conversation is a good capstone to this whole exploration of architecture because Alex is again at this intersection between theater and architecture and applying his interdisciplinary approach to solving all sorts of different problems with the company that he's a creative director at called Agile Lens. So we'll be covering all that and more on today's episode of the Voices of VR podcast. So this interview with Alex happened on Monday, June 17th, 2019 at XR for Change, which was a part of the Gains for Change Summit in New York City, New York. So with that, let's go ahead and dive right in.
[00:02:40.870] Alex Coulombe: Hi Kent, my name's Alex Coulombe, I'm the creative director of Agile Lens. We're here in New York City and my office is but a few short blocks away. And in immersive media, I'm involved in a spectrum, I guess you could say, across architecture and theater. And it's kind of every combination of architecture and theater. and VR, you could imagine where that involves the design of theater architecture. We're here at the New School, and the Tishman Space was an early theater that I helped design as part of Fisher-Dax Associates. But then since then, we've mixed and matched things like doing production design, using programs like Tilt Brush to figure out set design for live theater shows, as well as putting on live events inside virtual reality. And then I just had the opportunity to show you kind of a standalone theater piece, which doesn't have a live component, but still is theatrical.
[00:03:27.080] Kent Bye: I think that the last time we talked was at the Magic Leap LeapCon where you were in the hallway giving another demo of different stuff that you were working on. And so you're giving another Magic Leap demo here at the XR for Change Summit. It's part of the Games for Change Summit. But maybe you could just describe to me a little bit about what you're showing here.
[00:03:44.600] Alex Coulombe: Sure, this is a piece called That Kind of Guy by Shawn Area Lever and directed by Kevin Labeson. And it came about because we wanted to showcase at the Theater Communications Group Conference, which was last week in Miami, to theater makers all across the country some of the differences between putting on standalone theater pieces that could be either volumetrically based or motion capture based, because I have found that there's a lot of really creative, incredible theater makers across the country who don't have a great understanding of the ways technology can be a tool for them, especially immersive technology like VR and AR.
[00:04:19.002] Kent Bye: Yeah, so I went through and saw the entire experience with just the depth kit, but then at the end you said, oh, put out your hand and you're able to actually switch into different modalities so you could show depth kit and then a motion capture and maybe get to walk through all the different iterations that you wanted to show, which is essentially like a short minute and a half experience, but you're really trying to show the different types of volumetric capture that you could use for people and what it looks like on the Magic Leads.
[00:04:44.382] Alex Coulombe: That's right, yeah, so Magic Leap, of course, with a fairly limited field of view, isn't going to be great for things that are full-scale, and I've encountered that in architecture, where a small-scale model is better than being inside a full space, and then for something like a performance, this kind of coffee-table-scale theater, such as what Magic Leap has shown from the RSC and the Imaginarium tends to be very well suited where the performer is only about four inches high. And so what we had going on here was the ability to cycle just using an open hand gesture between the depth kit volumetric capture as well as several different avatars. And the idea here was to showcase some of the pros and cons of each one of those capture methods. And so of course with volumetric capture the actor is going to look a fair amount like the actor, and you get a little bit more nuance in what they're doing with their face, and it looks like them, but then it's kind of a baked experience. You can't change much of how it looks or do much post-processing with it, versus the motion capture data. similar to what you've seen people like Andy Serkis do with Gollum and characters like that. You can map that motion capture data onto digital characters, and while they might look a little bit more cartoony, you can do a lot more with that. You can change little things and fix certain pickups where the data looks kind of strange. It's easier to edit together multiple takes. And then you can even have things that are live and interactive, such as making it so the character will always look at you and follow your gaze as you move around, which I think helps with a little bit more of a connection with the performer.
[00:06:08.715] Kent Bye: And I think we were both in South by Southwest and were able to see, maybe for the first time that was shown publicly from Imaginarium, they were showing a theater piece that was using a lot of motion capture. Maybe you could give a bit more context of that piece so we can unpack that a little bit more.
[00:06:22.583] Alex Coulombe: Yeah, so the Imaginarium, I hung out with them a bit at LeapCon in October, and they were showing an experience there called Super Zeros, but Andy Serkis, during the keynote, mentioned that he had done The Grinning Man, and it's this musical, and basically he said that they, I think they paid for a full performance for all the actors to come into their motion capture studio, and using some of the higher-end technology that's out there right now, they were able to do this very high-fidelity capture of the entire performance of the show, and because that wasn't volumetric, which is, you know, very high bandwidth, those 3D models are enormously sized, the mocap data is fairly light, so they had photorealistic scans of all the performers, and then, yes, what we saw at South by Southwest was this little tabletop theater experience on a proscenium stage with a curtain, and you got to watch, I think it was about five minutes, like, one of the musical numbers where the actors come out and they perform, and it was a nice little showcase of what could be done with basically what you'd otherwise be seen on a full stage but in this kind of God mode scenario where you can walk around the whole stage and watch all the characters do their performance with singing and dancing and acting.
[00:07:27.664] Kent Bye: Yeah, that was the first time that I really saw a full ensemble motion captured all in the same experience. And talking to Asad J. Malik, he had this piece that was at Sundance called The Jester's Tale, and it was using volumetric capture, but he said it was pushing it to the edge in terms of using every last available amount of space just to do what he was able to do, which was essentially maybe a five to ten minute volumetric capture type of experience on top of all the other aspects on top of that as well. So you have maybe a 10 to 15 minute cap of the volumetric data that you could have with an experience like that, but yet with what the Grinning Man was shown from Imaginarium, it felt like it was going to be a lot more optimized and there was probably at least six or seven different characters and the voice and it just felt like, to me, really impressive to see like, oh wow, really see the difference between the technology and what you can do when it comes to having a full ensemble cast.
[00:08:24.782] Alex Coulombe: Absolutely. Yeah, I don't have too much more to add to that other than it was a really great showcase of what's possible there. And so, you know, one of the goals with that kind of guy was also to show people that you don't need to have an enormous budget to do something like this. A lot of the companies that attend TCG are, you know, small, local, regional theaters. And especially if they want to dip their toe into immersive media, they might want to start off with something that isn't going to take too big of a bite out of their budget for the year. And so what we put together, requires a Vive, a few trackers, Depthkit, which is basically an Xbox Kinect, and some software, which goes for about $10 a month. So you could say all in for under $1,000, we're able to put together a piece that was very simple to do and also can showcase some of the potential of these different technologies.
[00:09:13.335] Kent Bye: Yeah, and it's very rough in the sense of, you know, you're able to kind of prototype it and see how it works. Then you have the motion capture version, which it sounded like that you were using some iPhone technology to be able to actually do the facial capture as well, and that, you know, you weren't able to do a single pass of all of these just because of the interference between the different signals. But maybe you could talk a bit about the other iterations that you had to go through and the different technologies you used to be able to show the different motion capture options that are possible.
[00:09:39.923] Alex Coulombe: Yeah, so we used a program called Orion from a company called Ikinema, which allows you to basically take however many Vive trackers you might have and put them on an actor. And then you basically just show on their little diagram what the configuration looks like. And it does a really excellent job of doing full motion capture data saved to an FBX file. It can go right into Unity 3D if that's useful to someone. Then you have that ready to go. And then for the facial capture, This was kind of a last-minute addition because we just assumed this would be very expensive and hard to do. But we found a $40 iPhone app, which I fail to remember at this moment. Maybe if we release this, we'll put it in the show notes. But it is $40 and it actually does an excellent job. All you need is an iPhone X, something with a depth camera, and then it basically creates what are called blend shapes. And so those blend shapes, there's a default character that it maps onto, which we use. It's kind of a creepy baby face. But you can map those blend shapes onto any other rig that also has blend shapes, and then have a really robust facial capture experience, again, for the low, low price of $40.
[00:10:41.373] Kent Bye: It's probably from Babylon. They were working on it. That was Akrima Bilhasan. And they were working on a game that I think they got a grant from Epic Games to be able to build it out into a full app. It's likely that, because it's probably why they had a baby face.
[00:10:56.367] Alex Coulombe: Yeah, it's funny you say that, because I was looking at what Kite & Lightning was using for Babylon, and for some reason we decided we couldn't use it, so this was actually something totally different. And yeah, it didn't have that many reviews, so not many people have seemed to have used it yet, but very robust, yeah.
[00:11:11.304] Kent Bye: It looked great. I mean, because the problem with being able to capture something with DepKit is that it has artifacts and that you're able to get a volumetric performance and you're able to see the full emotional expression. But yet, I think going into the more symbolic representation of doing something like motion capture and then being able to make it into a specific avatar representation, you're able to amplify and create a little bit more of a Pixar version that can actually amplify the emotions in some ways. And I was surprised to see how expressive that was because I was like, wow, what are you doing to be able to capture this? Because this is actually like real professional grade motion capture that is actually better quality than the other versions.
[00:11:51.272] Alex Coulombe: Yeah, I agree completely, and it was a surprise to us that we found we could get, in some cases, a very high-fidelity performance capture, which, again, we were going on as little money as we could, so it is exciting to think about how the technology exists now and that it's just gonna continue to get better. Just the affordances from the iPhone X and having that depth camera on it now, there's a million possibilities that come from that, so I think we're excited to continue to explore this, and also we're exploring some funding options right now to do Things that are a little bit more, I want to say, built out from what we have here. This is kind of a nice proof of concept, but we're certainly excited to do more work in this vein.
[00:12:29.035] Kent Bye: Well, a big thing about theater is that it's live in the moment, and you see it there. And that there's some sort of alchemical interaction between the audience and the performers. But yet, at the same time, you have the capability to go in there like Andy Serkis did, to see something like A Grinning Man. And it was a theater show that was very ephemeral. It has a show run, and then it's done. And then people don't see it anymore. But they're able to actually start to archive it and capture it. and show it. And now, I guess from a theater perspective, I'm curious to hear your perspective on this because there is this quality of being there live and seeing a live performance, but yet the scalability of being able to capture it and redistribute it out through these immersive technologies like augmented reality or even virtual reality, you're able to have it made available so you have a lot more people that have access to it. So, just curious to hear from you as a theater person and the immersive technologist some of those trade-offs that you find.
[00:13:25.049] Alex Coulombe: Yeah, that's a great question. And I'll start by saying, actually, what I kind of love about the ecosystem we're in now with programs like Unity 3D and Unreal Engine is it's becoming easier and easier to create these cross-platform versions of the experience where, yes, you can have a Magic Leap version. There's also an Oculus Quest version of this. But then also, you know, there's an iPad version of this. And we could even go so far as to actually build out like a WebXR version. We have some idea of what that would be. And it's really nice when you can get to that point where you don't even need to tell people to install an app or have a particular piece of hardware. You just say, you know, go to this web link and you're going to have some version of the experience. So that level of cross-platform accessibility is really exciting. And then, I guess just to segue a little bit, we're also very excited then about what this means for live events and starting to have things happen during live performances that start to take advantage of these technologies.
[00:14:13.878] Kent Bye: That's where I also have a little bit of a hesitation because if I go to see a theater performance, kind of like the last thing that I want to do is pull out my phone. And so what types of interactions are you going to have in order to make it worth people doing that and to not risk having them be more disconnected or dissociated than start to check their email or whatnot?
[00:14:33.585] Alex Coulombe: Absolutely. It's a great question. And what I find is it has to be a value add. And so I would never put forward a live performance event that requires technology, except, you know, I should mention that, you know, my little side group called Alive in Plastic Land, which has David Gotchfeld, who did the Hamlet VR and Kira Benzing, who won Best Interactive Experience at South by Southwest for running our little performance event trio thing has been live events inside of virtual reality. So that does require a VR headset. but flipping back over to sitting in a real theater watching a live event, I think it should never require the technology, but there should be something extra there. Here's kind of my pitch for what I imagine is a great use case for this. Are you familiar with Calvin and Hobbes? Yeah. Excellent. So, you know, you have a kid, he's got a wild imagination, and a friend who's a stuffed tiger, and when Calvin's with him, tiger comes to life, and when he's not there, or parents or adults are watching, just a stuffed tiger. And so I say, imagine a theater show where it's a totally valid show to watch live without any technology, and you are basically seeing the real world. So to use the Calvin and Hobbes example, you're watching Calvin playing with a stuffed tiger, and there's something kind of fun and fanciful about that. put on a HoloLens, look through your iPad, something like that, and using the powers of augmented reality, you get to see inside a child's mind. You get to see the world that they see, which is more vivid and fanciful, and there's extra characters and things are coming to life. And so you can simultaneously allow for what might at first glance seem to be like a very minimalistic live performance, but then have this opportunity for a much more built-out, complete experience that is only enabled by the technology. So the challenge, though, is making both of those experiences equally meaningful. And maybe that's an impossible task, but that's the kind of thing that I'm interested in exploring.
[00:16:17.445] Kent Bye: Well, I think it's a really interesting time right now for theater because you have this confluence of immersive theater, immersive storytelling, virtual and augmented reality technologies where, in some ways, the type of storytelling innovations that are happening within the immersive theater space is, in a lot of ways, on the bleeding edge of where all of the other immersive technologies are going. And so, you've had the situation where the theater folks have been a little bit on the starving artists and under-appreciated, under-supported, under-funded, still largely that case, but There's opening up all these new economic possibilities with these immersive theater with experiential marketing and experiential entertainment, the immersive design summit, kind of gathering all these folks together and pairing them up to create these specific experiences, but also like where all the immersive virtual and augmented reality technologies are going have embedded within it a lot of the fabric of immersive theater and theater within itself. And so just curious to hear your perspective of these shifts that you see coming, obviously being at the forefront of that intersection, but also interfacing with people who are coming from that more brick and mortar traditions of theater and maybe a little bit skeptical of the technology.
[00:17:29.109] Alex Coulombe: Yeah, well, a lot of it, and I'll just mention that I had the same challenge coming from the world of architecture. And in both architecture and theater, the answer is kind of the same, which is you take people who have a certain craft and a certain expertise, and you show them that that craft and that expertise is still relevant using these new mediums. So in architecture, if you know how to do a really beautiful hand sketch, there's a good chance that with a little bit of training in Tilt Brush or Gravity Sketch or a program like that, you're going to be able to create a really beautiful sketch that people can move through. moving into theater, same thing with set design or costume design or anything like that. And then for the live performances, all the elements of stagecraft, the way light is used, the way a set is used, the way that you direct an audience's attention, all of that is incredibly relevant in theater, especially because I think erroneously when VR and AR first started to take off, people were looking to filmmakers to say, what should we be doing in here? And I think a lot of us are like, well, theater is actually a little bit more relevant because of that immersive quality and the fact that you don't have as much control over what the audience is going to focus on. So how do you create a world and a narrative that allows people to insert their own narrative into it a little bit. And of course then you go into this world of typical proscenium style experience where you have the audience and the performers very separated as opposed to something that's more of an immersive theater experience where maybe you are either a ghost inside the experience or more active Participant Sleep No More, of course, being more of the ghost and something like Gatsby, the immersive VR experience, and London being one where you are very actively involved in telling the story. And so I think when you start to tell people who come from the traditional world of brick-and-mortar theater that their skills are more relevant than ever to this new medium, they start to see ways they can insert themselves. And Brandon Powers and I, who are both at TCG speaking on a panel, were having these great dialogues about how So, I opens up to technologists and says, hey, here's a vision I have. Is there anything you can do to help me make it happen? Especially with that baseline understanding of the capabilities of technologists, some really incredible things can happen.
[00:19:47.773] Kent Bye: Well, where do you see some of the first big wins are going to be at for the theater community using these immersive technologies? Is it going to be in the set design? Is it going to be through augmented reality applications? Is it going to be through creating these motion capture things? I mean, where do you see the kind of the first next steps here?
[00:20:05.775] Alex Coulombe: It's a great question. My immediate thought goes to some of the work that the National Theatre has already done, because they do have people there already. I think Ray Smith is someone who uses Tilt Brush in her set designs for the National Theatre. You have people like Toby Coffey, who has his own immersive studio, and they worked on things like Draw Me Close, and they're continuing to innovate on that front. But I think just looking at the theatre community at large, I actually hope that it's more at the level of prototyping, and again, this is something that I see as a huge win for architecture as well, I'll keep drawing that comparison, where the more you have a sense of what a final experience is going to be like before you're spending all the money to be in the space and build a set and have all the actors there on the day and going through all the tech cues, If you have a better sense of what that final experience is going to be like, using the tools of, say, virtual reality. And so again, you know, in architecture, look at a building before the building is even a few months into the design. In theater, start blocking the actors and playing with different set design ideas and different light cues in a virtual reality experience long before you're actually in the space. I think that's going to create better experiences in the end, even if the final experience has nothing to do with VR or AR. It's there as a tool that is fantastic for prototyping.
[00:21:16.003] Kent Bye: And the last time we've talked on the record, we've talked all about theater and immersive theater, but architecture has been coming up a lot. And since the last time we recorded an interview, I had a chance to go to the Architectural Association in London where I was participating in the immersive architecture of the Internet. participating in a lot of different discussions there. And what was fascinating was to see how much the architects are starting to embrace game design as a further evolution of what architecture is, which is like this interdisciplinary process that's very similar to game design in that sense. But you almost have like those two architectural domains that are very interdisciplinary of game design as well as architecture starting to kind of fuse and merge together in different ways. And so just curious to hear a little bit more of your own background and your journey into this immersive space?
[00:22:05.000] Alex Coulombe: Yeah, oh fantastic. Well, so I've always been interested in emerging technology and ways the technology could help enhance or otherwise showcase different kinds of more traditional experiences. You could say I really got my start with this when I wrote my first play in high school. This was 2005 and you know we had a limited audience for my show but we had like a hundred light cues and this was Overdoing it like thank God for the wonderful tech people at my high school for all the cues and I had projection mapping going on and was really pushing what my little like 16 year old mind could think of but then also I Captured video footage from all the nights of the show and I thought geez I want to reach a wider audience with this so then taught myself Premiere Pro and started to figure out how I could Release this as a DVD experience and then make it more interactive with crazy things in the menus and from there going into Syracuse University where I studied architecture and theater I was always interested in the ways that creative disciplines could start to influence each other. And I co-founded a group at Syracuse called the Warehouse Architecture Theater, still going strong 13 years later, which is really cool. And the original impetus of that was, you know, when you're working on an architecture project, it's a really long process before it's actually going to be realized, especially in the real world, but also just in that two semester long length you sometimes have for a project in architecture school. So there were a bunch of us who missed the expediency and the joy of getting a theater production on its feet really fast, so the group started as just people in the architecture school who at 1 a.m. in the morning would be like, I am so stressed out right now, I need some kind of outlet, and we would go into a classroom and just rehearse a David Ives scene for like 10 minutes, which was such a nice release, And then also really cool to then build the sets and start to bring in other disciplines to all work together. So then, at one point in my school, there was a little thing called the School of Interdisciplinary Design, where they started to look for opportunities of bringing together different design disciplines, interior design, industrial design, architecture, landscape design. technical theater design, and we had these little competitions where I was like, okay, like, let's have you all put your heads together and approach these design problems from your unique perspectives and see if you can start to create something really interesting. And the first project we did was something called the Syracuse Ribbon. which was supposed to be a way to kind of link how the whole city was connected. But again, that got me super excited about how these different disciplines can connect to each other. And so when I left school in 2010, I started working for a number of different architecture firms, Raphael Vignoli's, and Morgenstern Frasca Architects, a few others, and was always looking for opportunities where we could start to either bring in other disciplines for further expertise, David Rockwell, for example, I know has a history of hiring choreographers when he's designing his spaces to help think about how people are going to move through it. And I thought that sounded fantastic, and I was always looking for that kind of opportunity. But then as new technology started to come out, and I go back to things like Revit, which is just, you know, 3D modeling of buildings. all the way into things like virtual reality and augmented reality, which I really got into as soon as the DK1 came out. I just wanted to find ways that this could be a really useful way for us to better communicate design intent. And so again, on the architecture side, if you've got a sketch or a really simple idea for a project, how do you convey that to someone who doesn't know how to read a plan or a section or an elevation? And then on the theater side, the opportunities are so endless there in terms of showcasing what a final regular production is gonna be like, or actually starting to think about what a production might look like that uses immersive technology. One fun thing that happened in the middle of all that was in 2014, there is a group called 72 Hour Actions, I think they're based out of Germany, and they put on a competition called the World Championship of Gameful Architecture. And this was really exciting because they were basically looking for people who had a background in game design and interaction design, but then also architects. And so they brought us all together in Germany, a place called Witten, for one week. And then ultimately there was a little bit of training. And then we had 72 hours to build an architectural intervention from scratch that needed to have an interactive component to it. So going back to the beginning of this question, there is something really exciting when you start to consider the interactive side of a built experience. And so in our case, our prompt was to create a space for intercultural communication. And so we thought, OK, well, we were given this little plaza, and we had these German girls on our team who could translate for us as we walked around and interviewed the people who went through the plaza every day. And we thought, OK, well, there needs to be some kind of intercultural communication here. And we found out that in this particular area, there were the people who were from Germany who really got along and they thought they were all great but there was a bit of a divide with the immigrants and the immigrants all got along and whether they are from Iraq or the Philippines or America they felt separated from the natural-born German citizens so we thought okay here's our challenge using some kind of interactive architecture we have to bring these two sides together and ultimately the thing we settled on was this is 2014 Germany had just won the World Cup and we started asking everyone how do you feel about Germany winning the World Cup? And everyone, regardless of whether they were a natural-born citizen or an immigrant, was like, it's the best thing in the world, it's amazing. And they all felt such a sense of national pride over that. So, we created an epic, like, six-tiered foosball table, they call it Kickerman, which was really just a way to play with the space. We had it, like, weaving around trees. We had parts of it where you had to get under it. We had little hills where the balls would jump over the top. and we just kind of left it there and there were some balls in local shops and some balls we left there and we had like a full day to watch passerby just kind of stop and look at this thing and then maybe start to play it a little bit and then you know someone kind of like give them a look like can I play too and then they'd come over and all these people started to interact with each other because they had a reason to now that didn't feel too awkward or foreign. And they were laughing and we had people of all ages. And that was a really exciting moment where we realized like, oh yeah, you know, interactivity thought of in concert with architecture can really open up some possibilities for how people engage with a space. And now, of course, with virtual reality and augmented reality, there's even more opportunities for that to happen.
[00:27:59.607] Kent Bye: Yeah, and talking to both artists as well as architects, where there's a certain part where you create the thing and you put it out into the world, and then at that point, it's kind of beyond your control as to how people react to it and how they engage with it. And so, there's a certain amount of, as an architect, needing to be aware of the phenomenological experiences or the anthropological perspectives or the sociological contexts and dimensions that you're trying to cultivate in some way and that in some ways with the virtual spaces you're able to maybe have a more of a feedback loop cycle of designing things in these imaginal virtual spaces and to see how people in the virtual spaces start to move around interact and then maybe start to have more of a feedback loop cycle in the design process because it does take so long but that there seems to be like this phenomenological and sociological turn of being able to see how the spaces that are created, how people react to it, and that there's a bit more of an iterative approach to that that's a little bit less of the waterfall approach where you design it and build it. It's very linear of actually building it out, but you're able to take a little bit more of that game design approach. And so it's fascinating to me to see how that sort of ethos of the iterative, agile mindset that comes from software development into game design is now moving into architecture.
[00:29:15.628] Alex Coulombe: Absolutely. And one thing that I'm really excited about in the future is when we start to have more thought and consideration put into virtual architecture. So that, of course, already happens with all the social platforms. Jessica Outlaw and I have had a lot of conversations about this, as well as Kim Bauman-Larsen, Christopher Nichols from Chaos Group, who makes V-Ray. That is to say there's a bunch of us who are really excited about when architects can start paying the same consideration to virtual buildings the same way they do for real buildings because at a certain point we'll reach that point of mass adoption where enough people are going to be inside a VR chat or a high-fidelity space either for personal reasons or like you know big-screen VR where you're watching a movie together or for conferences which is kind of what like high-fidelity is pivoting towards more of this enterprise you know professional use case and The human body, phenomenologically, doesn't care if you're in a real space or a virtual one. It's still going to react the same way to compression and release and light and dark. And it's really exciting to think of the ways that we might start to bridge certain, I want to say, limitations of the real world and start to create spaces that are going to be really excellent encouragers and motivators for different kinds of interactions.
[00:30:25.896] Kent Bye: Yeah, when I was talking to the different architects at the Architectural Association, there was a bit of this debate in terms of like, what are you losing in not having the physicality of the building with not knowing the materials and the heat transfer that happens of being in spaces, because that's a big part. of designing the spaces, but at the same time, there's a certain amount of puzzle solving that happens with an architecture that is very constrained by these real-world constraints that kind of start to go away when you do these virtual architectures. And so, in a lot of ways, you're able to explore the aesthetics and the beauty and things that go beyond the limitations of physical gravity, even, where you can start to create architectures that would be impossible to create in real life, not only from a financial standpoint, but like a physics standpoint. There seems to be this tension, though, in terms of, you know, as architects start to design and do a little bit both of designing physical spaces and these virtual spaces, that there's these variety of different trade-offs where you almost have to do, like, this context switch of, like, am I designing this for reality or designing this for a virtual space? But I'm talking to an architect that is doing that. She told me that she has to do this, like, context switch into designing from reality or designing in the virtual space. for you with your company that you have. I'm just curious to hear how you're navigating this intersection of what type of projects that you're doing, because you have this capability to go into the more artistic, aesthetics, beauty, storytelling, immersive theater side, but yet there may not be as many resources. Then there's the whole other architecture and virtual architecture, but they're kind of like separate worlds and different intersections that you're in the middle of at this intersection. And so I'm just curious what you're doing with your company.
[00:32:05.040] Alex Coulombe: Yeah, that's a fantastic question and we're still definitely figuring that out because the way Agile Lens came about was it grew out of Fisher-Dax Associates which designs theaters all over the world and you know we started using virtual reality just to test sight lines and put different things on the stage of the theaters we were making and play with you know now we're up in the catwalk and checking the sight lines of the lights and all that and then we just realized basically through my moonlighting efforts that there was a lot of opportunity to go beyond just using VR for theaters and go into architecture and other theatrical use cases. One of the first theater productions I got to do VR for was Kenneth Branagh's production of Macbeth at the Park Avenue Armory here in New York City in 2014, where it was transferring over from Manchester, England, and they knew what they wanted for the set design elements and kind of the general audience layout, but the theater was a very different shape, so it was great to use the DK1 to actually prototype, you know, here's a few different seating layouts, here's the blocking of the actors, oh, we're gonna have a hundred extras coming from that east door, but wait a minute, we can't see any of them, okay, let's change how they're gonna move, and then to cycle through those different options inside virtual reality and see, you know, option A, B, C, oh my god, C is by far the best, and the way that is relevant across Design spectrums like architecture and theater was really cool to see so I'm kind of dodging your question But I want agile lens as it develops to be finding these opportunities for emerging Problem-solving that maybe there isn't as much risk taking with it right now, but when people start to say hey, yeah, we really want a an architect who's going to build a virtual space, or, hey, we really want designers who are interested in putting on live productions using Magic Leaps or something like that, that we've built up enough of a foundation, because we're always doing R&D. We're always playing with the newest devices and testing out different use cases, both ones that are very clearly pragmatic, as well as ones that are a little bit more fanciful, a little bit more imaginative, but always with this hope that we're going to have the opportunity to work on never-before-done innovative projects.
[00:34:03.348] Kent Bye: And so for you, what are some of the either biggest open questions that you're trying to answer or open problems you're trying to solve?
[00:34:10.655] Alex Coulombe: Oh, good question. I'd say for problems we're trying to solve, I think we are looking at ways that immersive technology can to put it bluntly, to solve old problems in new ways. And so we're always looking for places where people have done certain things in a certain way for a very long time, and that's just the way they do it. But then we can kind of swoop in and say, hey, did you ever think of doing it this way? And kind of have a light bulb go off in people's heads. Another early project I got to have this moment with was working with Bravo Media. for the late show of Stephen Colbert, where Bravo Media was tasked with putting in these projectors for the dome in the Ed Sullivan Theater. And, you know, there's a very tried-and-true method for doing that, but they were very tight on time. So we worked together to develop, this was a Gear VR app, again in 2014, that allowed us to just kind of look around and place the projectors along the grid in different locations. And then just very quickly get kind of a general idea of here's some good spots to put the projectors. And now the more refined pass can be done on paper in the traditional sense. So I think there's this assumption a lot of people have where when I present some of our VR use cases, they're worried that we're trying to say, everything you've done in the past is irrelevant. And going back to what we were saying earlier about theater and architecture, no, what I'm really trying to say is these are new tools in your toolbox. Everything you've already done is still relevant and in many ways will help catapult you into new workflows and ways of doing your craft using these new technologies. So I guess you could say that we're really interested in finding ways to use virtual reality and augmented reality to solve problems more efficiently, build confidence in how everyone is feeling about a certain design problem because of that ability to rapidly iterate and view it at different stages of the design process, and just ultimately give everyone a better, I want to say, sense of accomplishment and pride once a designed experience, let's say architectural or theatrical, is finished because it's actually turning out the way they expected it to with as few surprises as possible.
[00:36:03.582] Kent Bye: Great. And finally, what do you think the ultimate potential of immersive technologies are and what they might be able to enable?
[00:36:12.335] Alex Coulombe: I'm really excited about just being able to allow everyone to have their own form of creative expression that can start to permeate between us. And this goes back to that somewhat dated notion of VR's empathy machine. But I have friends who are the most brilliant, artistic, strange-thinking people in the world. And I want to be able to put on a Magic Leap or any kind of immersive technology device. And I want to see the world through their eyes, whether they have made it so that you move through the world with a different color palette for everything, or if they have placed their artwork all around New York City, and they've made certain buildings invisible, and they've changed the way Central Park looks, and they've added extra kinds of plants. That kind of ability for someone to create the world in a way that they are excited about, and then invite other people to then also perceive the world from their perspective, I think is a really exciting potential of where this could all be going.
[00:37:05.879] Kent Bye: Awesome. Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:37:10.382] Alex Coulombe: I'm just so, so happy that you and your podcast exist, Kent, and I just want you to keep doing exactly what you're doing forever and ever and ever. Thank you.
[00:37:19.526] Kent Bye: Awesome. Great. Well, thank you so much. Thank you. So that was Alex Coulombe. He is the creative director of Agile Lens, and he's looking at the intersections of architecture and theater. So I have a number of different takeaways about this interview is that, first of all, Well, the both times that I had a chance to catch up and talk with Alex, he was doing demos, guerrilla style, setting up his laptop and just showing different experiences that he had been working on. That was the first time I talked to him at Magic Leap LeapCon. He was doing that. And here again at the XR for Change, he was showing this little demo prototype that he put together. And it was really just exploring the different types of volumetric capture solutions. And it was just nice to be able to see the different fidelity of what they look like. and the Magic Leap headset and the different approaches and philosophies of volumetric capture with a variety of different trade-offs that are involved with all of those. And super impressed by the iPhone capture. It's called the Face Cap Motion Capture. That was the project that he couldn't remember the name of, but he sent me the link to it. And it's an iPhone app that uses the depth sensor camera and is able to allow you to do this like facial capture that gets translated into blend shapes and exported into these immersive environments. It actually looked really, really great. And I think probably one of the highest fidelity ways of being able to actually capture, you know, facial expressions and dynamics. And so I highly recommend people check it out and to be able to see what you're able to do with, you know, just an iPhone and be able to capture different blend shapes and start to create different immersive experiences with, you know, these different facial expressions. So I think a big takeaway for me here was just to look at all these different ways of interdisciplinary collaborations and to see how you could take a variety of different design approaches and bring together those different philosophies and start to have these different competitions to solve different problems. And so it sounds like Alex has been attracted to this process of interdisciplinary design for a long, long time. and so he's again at these intersections of architecture and theater and seeing how there's like these established practices and his big message is that hey you know people from the theater world or people from architecture that there's a lot of tried and true methods for design that completely translate into being able to do spatial design within these immersive technologies and so I think he was just advocating for people to see how they can start to participate and collaborate with these other technologists. I think anybody who's in the realm of theater or an artist, finding collaborations that you can start to do with people who have the technical skills to be able to do different aspects of virtual reality. I think, you know, bringing those teams together of both that artistic inspiration, but also the technical aptitude to be able to pull it off. And there's a lot of magic that happens when you start to bring those two together. And it was also just interesting to hear how Alex was, you know, using VR technologies to walk into a lot of these more traditional theater productions and to start to, you know, use a gear VR to look at different sight lines for projectors and to, you know, take these problems that have very established potential solutions, but to be able to take a first pass and to rapidly iterate. And to be able to get people into these spatial experiences to make these gut decisions, and then to from there, go back into the traditional methods of the tried and true methods for how things usually get done. But it's at least able to narrow the scope of solving different problems using these immersive technologies. And so. It sounds like with his company agile lens, he wants to be a bit on the bleeding edge of trying to be early adopters of all these technologies and to really push the limits of what's possible, the technology, but hopefully leading to different ways to be able to collaborate and participate in these different productions. And so that was definitely what I saw with what he was showing at both magically bleep con as well as XR for change was a little bit of just like a quick rapid prototype, but it was done and it was able to actually show. very important differences in terms of the different affordances of these different volumetric capture solutions and for allow people to be Attracted to one solution or over the other and we will actually make a decision and then from there Start to actually play with a lot of these different volumetric capture solutions So again, whether it's bringing together interior designers or industrial designers, architects, landscape designers, technical theater designers, sounds like a number of different things that he was involved with the school of interdisciplinary design or this world championship of gainful architecture, that there is this fusion that I think is happening between architecture and interactivity, a lot more both from the game engines, but also just the need and desire to create these immersive and engaging interactive experiences that with people, you know, making choices and taking action. And so this fusion of game design with architecture is a theme that I think I was getting a lot from the series of different conversations that I was having with people that are trained in the architectural background, but just to see how these real-time game engines and this movement towards interactive agency, immersive theater, and to see how all these different worlds are starting to like fuse together. And from Alex, he's at this very interesting intersection, going to these things like the Theater Communications Conference to be able to talk to all these different theater makers around the world, and then to also just be advocating and evangelizing the potential of what these immersive technologies can start to be able to do. and to start to leverage these different skill sets that people have within their theater productions because there's actually a very clear one-to-one translation into spatial computing technologies, especially what Andy Serkis was showing with The Grinning Man, which we both saw at South by Southwest at the same time. which was essentially doing this motion capture of all these different actors and it was pretty impactful for me at least to see five or six or seven different actors all at the same time at this tabletop experience and to see the level of fidelity that you could get from the motion capture data and be able to to show that within the full augmented reality experience and You know, you could certainly do that a little bit with the volumetric capture, but you start to get into some really unwieldy territory of how big the file sizes are. And there's a certain amount of capacity that you just wouldn't be able to do that level of scale with the volumetric capture. And you really need to get the efficiency of the mocap data to be able to show what they were able to do with the Imaginarium and the Grinning Man. So yeah, just to see the theater space and what the National Theater is doing, what the Royal Shakespeare Company is doing. I actually have an interview with one of the directors there, the Royal Shakespeare Company, and they're doing all sorts of things about looking at the future of immersive entertainment and collaborating with folks like Marshmallow Lizard Feast and Punchdrunk and you know, just doing these different projects and experiments. And yeah, there's a lot that's happening in the theater space. And I know that Alex is also kind of in the frontier of seeing where all that is going. And I'm super excited to see what they've been able to put together as a part of the project that was directed by Akira Binzing that is going to be at Venice Film Festival that I'm going to be going to off next week. So I'm off for a week and then I'll be back after September 2nd. So hopefully I've been diving into some of the, you know, either future of neuroscience or some of the other podcasts that I definitely want to get out before Oculus Connect 6 is coming up here on like September 23rd. And so I'll be able to either get out one or two more different series before I go off to Oculus Connect 6. And I'll be going off to Venice to be able to see as many of the different experiences as I can, be participating on some panel discussions. A hot topic within the world of immersive storytelling is the nature of critique. What is the value of having these different critical theories and be able to interrogate experiences and to critique them in certain ways. And so I'll be talking to a legendary experimental musician, Laurie Anderson, who's got a piece that's going to be showing there at the Venice Film Festival, but she's also on the jury. and also just being in discussion and the panel discussion talking about, you know, what's the future of critique and being able to analyze and to pick apart and to be able to speak about with specific language about these immersive stories that are being created. I think there's different conversations I had with Shelly Sherman, who is really advocating for having a little bit more of a critical dialogue within the immersive technology community. And I think there's just this process of trying to actually put language and words into something that is so ephemeral and difficult to really pin down, but to perhaps be able to notice the different subtle aspects of the human experience and be able to put language around that and to see the different combinations of all the things that happen in an experience to be able to isolate what is happening and how that kind of gets translated into the different levels of human experience. It sounds like this is something like it's an age-old question that architects have been starting to really try to figure out how to deal with, but perhaps with these immersive technologies and to be able to create these spatial experiences on demand, then maybe we'll be able to help to refine both our own direct experiences of these and be able to articulate and create these different critical design frameworks to be able to more critically analyze a lot of what's happening within these experiences. So that's some of the different discussions they're going to be having there in Venice coming up next week. So that's all that I have for today. And I just wanted to thank you for joining me on this exploration of the future of immersive architecture, not only on the internet, but these immersive experiences. It's a topic that I think I got a lot out of and talking to all these different architects and I I look forward to engaging more with the architectural community. Hopefully the Architectural Association will have another gathering next year to bring together even more architects from around the world and to have a whole demo day and show different experiences, what the different great examples are of immersive architecture. And yeah, just to talk more about where this is all going and what the architects have to say about it. So, if you enjoyed this podcast, then please do spread the word, tell your friends, either on social media, send architects you know, this series, let them listen. And I'll be doing a whole tweet stream of each of these different episodes. You can send them to those tweet streams and let them dive into these different episodes. And, you know, if you'd like to support this podcast, I really do rely upon donations from listeners like yourself. this is a listen supported podcast. And so I do rely upon donations from my listening audience. And if you'd like to donate to this podcast, then just $5 a month is a great amount to give. If you can give more, that's also great too. It just helps me to be able to keep on doing this podcast and to keep this oral history going. And I've got a lot of podcasts I've already recorded that I'm looking forward to process and getting out. And I plan on continuing to show up to these different conferences and just help capture the history as it's unfolding. I just feel like a moral obligation that somebody needs to be out there doing it. And it'd be great to be able to get enough support to allow me to continue to allow me to continue to do this, but also to eventually grow and expand. So you can donate and become a member today at patreon.com slash voices of VR. Thanks for listening.
