Colored is a HoloLens 2-based, augmented reality immersive story installation for 3-10 people that talks about the forgotten history of Claudette Colvin. “This experience plunges the audience into the Deep South during segregation. In the course of that journey, we meet the young Claudette Colvin, 15 years old who, on March 2, 1955, refused to give up her seat on a bus to a white passenger. Nine months later, Rosa Parks repeated this act of defiance and became the icon that history remembers to this day.”
Writer Tania de Montaigne discovered Colvin’s story, and wrote a book named “Noire: La vie méconnue de Claudette Colvin – collection “Nos héroïnes” (The Unknown Life of Claudette Colvin: “Our Heroines” Collection). Then Stéphane Foenkinos discovered the story and adapted into a play, and then Pierre-Alain Giraud joined with Foenkinos to co-direct and co-produce an immersive AR story adaptation with them.
This is one of the more compelling AR stories that I’ve seen so far since they’re using a series of benches in a stark black space that is transformed by AR point-cloud overlays to change contexts into a bus, into a church, and then into a courtroom. They used Volumetric capture studio in Taiwan to asynchronously capture each of the performances, and then added spatial sound, additional archival film footage, added theatrical lighting effects, and wind machines to add additional haptic feedback. Overall, it was really compelling use of mixed reality to go on a spatial journey with two other people at the Tribeca screening (up to 10 people could see it at a time at it’s initial showing at World Premiere of “Noire” at the Centre Pompidou in Paris from April 21 to May 29.
I had a chance to speak with de Montaigne, Foenkinos, and Alain-Giraud at Tribeca Immersive to talk about the translations from book to play to AR installation, their experiential design process, and interdisciplinary fusion of storytelling techniques to tell this forgotten history of Claudette Colvin.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the potentials of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So, continuing on my series of looking at different immersive stories from Tropica Immersive, today's episode is with a piece called Colored, which is a mixed reality experience that's exploring the life of Claudette Colvin, who is a figure who's very similar to Rosa Parks, who refused to stand up on the bus in the context of segregation before Rosa Parks, but her case ended up getting dismissed, and so she was a figure that was lost to history, so to say. So Tania de Montaigne is a writer who came across the story of Claudette Colvin and decided to write a book about it. And then Stefan von Kinos read the book and wanted to translate that into a theatrical play. And then Pierre-Alain Giraud was able to translate these previous two medias and then do a mixed reality performance of this. And so they had different film elements that were produced in the context of the play, and they were able to repurpose some of those and put them into a mixed reality experience. So this was a piece that was actually done in the HoloLens 2. And so you're in a big black room with some objects like benches that get repurposed to get turned into a bus, to get turned into a courthouse, get turned into a church. And so you're in one consistent spatial context with some physical things to sit on, and then they're able to use the augmented reality to overlay these point cloud representations and to change the spatial context as they're telling the story of Claudette Colvin. using some volumetric capture of each of these different characters walking through this space. And so it's a piece that you're able to see with three other people at the same time, at least in the version that was showing at Jarbeka. It first premiered in France with up to 10 people at the same time in a much larger space as well. And it's one of the best augmented reality stories that I've seen and quite innovative and effective use of the medium of augmented reality to be able to tell a story. So that's what we're covering on today's episode of the Wasteless VR Podcast. So this interview with Tanya, Stephan and Pierre happened on Monday, June 12th, 2023 at Tribeca Immersive in New York City, New York. So with that, let's go ahead and dive right in.
[00:02:20.443] Tania de Montaigne: Hi, I'm Tanya de Montaigne. I'm the writer of the project, Call It, and one of the actors too.
[00:02:27.930] Stephane Foenkinos: Hello, I'm Stéphane Fuenquinos, I'm the co-director of Collard.
[00:02:32.555] Pierre-Alain Giraud: Hello, I'm Pierre-Alain Giraud, and I'm co-directing Collard with Stéphane and also co-producing with Emma.
[00:02:39.341] Kent Bye: All right, so why don't you each give a bit more context as to your backgrounds and your journey into this space of immersive storytelling?
[00:02:47.498] Tania de Montaigne: Actually, I'm a writer, fiction and non-fiction, and Collared is my first non-fiction book. After that, Stéphane, who is right next to me, decided to make it become a play. And in the play there are videos, and this video was made by Pierre-Alain Giraud, who's just over there. And Pierre-Alain has already made a piece in augmented reality before. And he tells us, OK, I think we can do something with this story and augmented reality. It can be interesting to mix that. So that's the beginning for me of the journey. I didn't know nothing about that. And at the beginning, I can't make the difference between AR and VR just to make you think where I was about XR technology.
[00:03:41.794] Stephane Foenkinos: As a film director, this is mostly what I was dealing with, stories and actors. And also, I worked as a theater director with Tanya, mostly. So my background is mostly 2Ds, or like one or two-dimensional. But I never envisioned anything XR until I worked with this guy here, with Pierrana. And for me it was like opening a new world, opening a new perspective. And what is interesting, I think it's quite rare from what I gathered, to have an existing piece as a book, as a play, becoming an XR experience. So I think for me augmented reality, this is really what happened here because it augmented the reality of our story and for this I'm very grateful and still in awe of the possibilities.
[00:04:43.573] Pierre-Alain Giraud: And so on my side I did also, before doing this AR piece, I already did one called Solastasia, but I mostly also worked in classical filmmaking and also I did a lot of films that are projected on stage for theatre, so I worked a lot with theatre directors. And that's how I met Stéphane. That's how it started. So when I was working on the play Noir, then I thought it was a good idea to mix AR with that project or to tell the story of Claudette Colvin that way.
[00:05:16.191] Kent Bye: Yeah, and so it sounds like it started as a book and then a play and then an AR experience. So maybe you could give a bit more context for how you came across the story and decided to turn it into a book.
[00:05:28.513] Tania de Montaigne: The book belongs to a collection, a French collection, called Our Heroine. And the principle is that they ask female writers to write about another female who could have been in history and hadn't for a reason or another. And each writer has to find her heroine. So they asked me if I wanted to be a part of this collection. I said yes, but at the beginning I didn't know which woman I could talk about. At a moment I remembered that I had read years before two lines about a teenager who could have been Rosa Parks but hadn't been because she was pregnant. the only thing that was precise and I said, okay, maybe there's something to say about that because I hadn't heard before about this teenager. So I thought it was interesting to go further and that's how I discovered the real story of Claudette Colville, which is nothing about these two lines, but everything about something else. So it was really, really interesting to let people know about her.
[00:06:34.497] Kent Bye: So it started as a book and then how did it get translated into a play? What happened after that?
[00:06:39.420] Tania de Montaigne: What happened is that Stefan Fuenkinos read the book and he said, I think it has to be a play. So that's the idea of this guy.
[00:06:51.446] Stephane Foenkinos: The thing is, I read the book, I've known Tania for a long time, but I was so surprised reading the story, because I thought, being an English major and everything, that I knew about the civil rights movement, but this woman, I never heard. I never heard that name. And more than that, I love the book. I love the way that Tanya's words, for example, just invite you into a story. And I say, that would be a great context for a stage. And then when she said, oh, it's great that you think about that and you want to do this with actors, I say, no, I want to do this with you. And so she said, no. No, no, she was not interested. It was not like being false modesty or anything. I think she literally didn't see herself as an actor. And I said, well, I'm not a stage director, so maybe we can find something together. And little by little, we did some readings and finally, I think she found a place on stage because she was a spokeswoman for Claudette. She was like an intermediary. And we created a kind of documentary theater that I really liked because I also could use images. And then came the most experimented of the whole gang that is this guy here, Pierre Halin, because, yeah, we wanted to do projections. I wanted to use films. I wanted to use archives that you still have in the AR experience. So yeah, for us it's a journey that started eight years ago and the place is still touring.
[00:08:29.192] Kent Bye: So yeah, it sounds like that there's stages of play then on the AR piece that I just saw there's a screen that's showing some of these videos that are presuming some of these videos were shot originally for the play. So maybe you could talk about how you came on to the project and started to either work on the videos and at what point did the idea come up to translate it into a full spatial performance or a spatial story in the context of augmented reality and the HoloLens?
[00:08:54.526] Pierre-Alain Giraud: So the videos that are in the AR piece are not the ones that are in the play, so we made everything new for the AR piece. Yeah, but actually the beginning was the text that Stéphane adapted from the book to the play and then from the text of Stéphane we did a new adaptation for the AR because it's much shorter. And also because in AR we could show some of the actions that were described in the book so we didn't have to tell them anymore. So that's why the text also is shorter. And when that idea came along is two years ago, I think, when we were already finished with the play and I had this other piece that I was working on where I was already working with this idea of putting ghosts in your present and playing with the past, the present and the future, gathering together in the present, basically. I talked to Stéphane and Tania and we started discussing it. Maybe that could be a good idea to do something with that story because we are dealing with a history that has been erased or we have traces of that story and just the form of the holograms that you see in front of your eyes as a real ghost that you can touch or you can really nearly interact with them. Not really, I mean it's a very strange presence in your surrounding. It's not like VR where you don't see what's around you. Here you see always what's around you. You see the sets that were created. You see the other people that are with you in the room. So you really are who you are, you never go somewhere else, but some ghosts are coming to you and to your present and informing your present and your future. So that's the idea we had and we thought to tell the story of Claudette that has been forgotten by history, that's the right form to do it.
[00:11:01.065] Kent Bye: Yeah, well I have to say that I've seen a lot of different virtual reality stories and I think augmented reality, it's been more difficult for me to see how the medium is really going to help tell the story. Up to this point, I've seen some tabletop experiments, I've seen some stuff where you're walking around and interacting with these objects, but I think that both the strength of the story that you have in this piece, but also the holograms and the spatial volumetric capture of these characters and the ghostly-like apparition nature and the point cloud representation and then the haptics and the lighting design and the video. I mean, everything I think really came together in this piece. It's, for me, one of the most powerful augmented reality stories that I've seen and just did a really brilliant job with everything. So, congratulations. on that and I can see the lineage from the book to the play to Experimenting on how to tell the story across these different media. It really shows the depth of the story that's being told here So so congratulations on that, but I guess when I'm talking to Anna Brzezinska She was saying that this piece actually had just premiered in a much larger space somewhere in France I think so maybe give a bit more context for the world premiere of where this first started to show maybe Describe where it world premiered and the process that it took to get it there
[00:12:13.588] Tania de Montaigne: Yeah, first we were at the Pompidou Center in France and what was interesting is that it was a very large space, so gigantic. Pierre-Alain was talking about archives and the past and the future and the present. But what is interesting is that you can see also how different arts are crossing together. They are not erased by another one. So when we were in Pompidou Center, it was very interesting to see how an exhibition is also talking with literature, is also talking about theater, is also dealing with these ghosts. And all these arts are combined to tell the story of this teenager. And everybody is about that when you were saying about this idea that everything has the same goal. And the goal is that we are trying to make this past be yours at the end. And yeah, the larger space permit that. And there was the light, even the light, because here we can't show that. But the light, there is a creation too. So arts are really the king of, that's the subject, how art can deal with history, and how art can tell something, how it's a strong voice, and how do we need art? When there's no art in a society, you can see how it is. There's no humanity at all. So that's the point. We have to deal with this idea that our weapon is art, and what can we do with this weapon?
[00:13:51.388] Stephane Foenkinos: Going to Pompidou Center was, first of all, it's something magical for any of us, because for anyone loving art, as Tania just said, it's like you go to the MoMA, you go to any great museum in the world. And also, it's not until we were there that we were realizing Just like the play becoming the AR experience, we never envisioned that. We never did the play and think, oh yeah, and then we're going to do an XR experience. So being in Pompidou, we never envisioned the fact that we were pioneers in a way, because it's true. We were at the crossing of installation, software, exhibition, theater, film, and all these things are separately displayed at the center, but together, even them, they had trouble really, first of all, understanding it, then promoting it, and really trusting it. Because until you see it, and you've been through it, it's difficult to talk about it. And it's not until we came here, even if it's a smaller form, for us it's amazing just to talk with people who are more experienced and they give us this feedback, wonderful feedback, that you also give us saying that, yeah, we have never seen that, but we never did that thinking, or maybe he did. It has never been done before, so we didn't know what to expect. Even us, while we were doing it, until we watched it, we didn't know what to expect. I knew.
[00:15:38.684] Pierre-Alain Giraud: You had a sense. No, no, it's true that you actually never know because we had something in mind but we never worked with the 4D views technology to this volumetric capture. So we had no idea of the effect it could have on people watching that into the space around you. You know, it was a surprise for us creating it, using those technologies also for the shooting because we never shot into those kind of studios before. So it's a studio where you can shoot only one person at a time with 50 cameras around one person. So we had to have rehearsals with very precise timings because they are complex scenes with people interacting, you know, so we were with a microphone, so you can now stand up and now you go this meter and you have to watch like this little point green so it was like a Yeah, and then we had to recompose the scenes in the space We had to develop the technologies to do that too at the same time. It was a work of creating the work of art but also creating the technologies supporting the art that we were doing. It was a big challenge. Until it was really finished, we didn't know really what to expect because even the last days were critical to just finalize all the timings, the staging, because we had to stage the ghosts around us. So in the location, it's a work that is very new. I think it doesn't really exist. So we had to invent a way to work at the same time.
[00:17:21.686] Stephane Foenkinos: It's interesting, Tania often says that it was like the beginning of cinema. And it's true in a way. The thing that I don't know if we felt exactly like Méliès or people who just started Lumière Brothers, but in a way we were in Taiwan and we were like, what do we do? Okay, how are we going to put this character? And this whole adventure, and maybe we'll talk about that later, but the whole adventure is crazy. but going to Taiwan to tell this story, to find a casting there, and to shoot, as Pierre-Alain just said, because even as a director, you usually have a scene, and you have different actors, and you rehearse with the actors, and you do the scene. Here, it was like something surreal, because you had mostly non-actors. like having to play scenes from a past that they didn't hear about like 20 minutes before and imagining them together in a space later and so it was like a double or triple projection onto something but I think that even if we sweated a lot it was really unique and I really like to think that this uniqueness is what you see now because for us it was also something so surprising it was like there was magic behind this story we sweated a lot we have to say it was not but there's a little magic in this I like to think that
[00:18:51.985] Kent Bye: So, yeah, I mean, I think it all comes together in the end where I feel like you're able to use the affordances of augmented reality where you're in a space that has all black walls, so it allows you to project on all sorts of different holographic images. You start with an exit sign, which was really quite immersive to see, like, this exit sign. I literally had to, like, lift up the HoloLens 2. It just looked so vividly plausible that it was actually there. So you're going through the story and you're having these holograms that you know, you had to shoot them individually But they're creating these cohesive scenes and folks are familiar with Rosa Parks as someone who refused to stand up in a bus But this is the predecessor Claudette Colvin who was a teenager who also didn't stand up and was arrested but it turns out that some of the charges that would have potentially used it as a case to go to change the federal law were dropped and so she ends up being Falsely accused of these violent acts but because of that she's a character that's lost to history So we get to go through her story and go into seeing these spatial reconstructions of like a bus and point cloud representations or her in jail where you're able to see a hologram of a jail you see a volumetric capture of the character behind the jail but you're also able to walk into the jail and see him from the outside or to walk to the wall and her appearance still appears you can see her so she's not completely occluded by these virtual walls so there's all these little design affordances that I think you have in this piece that starts to use the ability to transport me into another space where you could have done this as a VR world where everything would have been completely immersed but I'm there with other people and I think it's with the other people that makes it where the AR is a little bit more interesting because we're all looking at the same spatial context in these holograms but we're able to navigate this space that has benches where we can sit down and act like we're sitting next to someone on a bus or we can sit down and watch a court proceeding or watch a church sermon from Martin Luther King Jr. So we have all of these moments where you have a physical reality and the augmented reality holograms that are all spread around. And I think it all just came together. There's also wind and there's like other aspects with the sound and the Bluetooth. There's a lot of technologies that had to come together. But at the end of the day, I felt like There's a way that being immersed into this black box spatial context that is Changing and morphing around me gave me a sense of a spatial journey, even though I'm not going anywhere, but you're able to create this architecture that's surrounding me and to Create these different scenes where it was very intimate and personal that I felt co-present with these holograms so anyway lots of different stuff that you had to do to pull that off and I just wanted to to recount my own memories of it because
[00:21:38.448] Stephane Foenkinos: OK, can we quote you? Because, Noah, you talk about it so well. The most important thing, I think, that you point out is exactly the viewpoint. You are the own director of your journey during that experience. And that may be the main difference with VR. Because with VR, you can evolve in another space reality. But here, you see the people, but you decide what to watch and where to watch it. For instance, when Claudette is in court and during the trial, you can go and see from the judge's perspective, even look at the notes on his desk, or you can go and sit next to Claudette or behind her. For us, it was so surreal the first time that we really used the HoloLens because we could see the details of someone's head. A few times people ask us, are these characters digitally made? No, they're real persons. And that also adds to the story and adds to the effects and the emotions. They're real people. So you have real people, even if they're ghost-like. and you have your own viewpoint. You decide to go in jail or not. You decide to sit on the bus or not. Some people just stay outside the whole experience. It doesn't mean they don't have a different feeling. But as I always say, coming from filmmaking or doing theater, but when I see people getting out of the experience, it's a face I had never seen before. so I know that the effect of course and the tool that we have with this at the same time it's fascinating as an educational tool it can be also like a narrative tool but also in a way of I'm in the story that's really rare and this is something that you can envision films in the future saying oh wow that could be really interesting to be in the story instead of like being out of it
[00:23:43.743] Tania de Montaigne: And what was interesting when we were talking about the fact that you are not alone to do that. That's very important because the idea is that you are sharing something with people you do not know. And you are experimenting the possibility to build something else. History is like that, but maybe we can invent something else. And that's why AR is a good way to tell this story. And that's the reason why if someone had proposed me to make the same thing in VR, I had said no. Because VR, you're alone. And our problem, I think, is that we are alone. And if you want to change something in a society, you need to know that you are building something others, and this other, you do not know before, but this is society. So the idea was to rebuild something about this idea because I think pandemic has destroyed the possibility that you can trust someone else. And to rebuild that, I think the better way was to propose an experience you have to make with someone else. And we were talking about some people asking us, but why don't you do something that I could do in my home? They wanted to bring colored in their home with a headset. And for me, it would be the worst thing to do. If you're alone with your headset, you're going to be very distant from Claudette. And what was interesting is that when you make the experience, you can see that each group is different. The energy of the group builds something. And some groups are very distant from Claudette, very respectful. And that's beautiful to see them because when she walks, they let her go, just like she was a queen. Some others are very mobile, they are moving around, they are sitting on the bench, and something else is happening, so really the group makes the difference. Maybe if you come five different times, you're going to live five different experiences, because you're going to change the way you're looking at the scene, and because people that will be there will be different.
[00:25:58.717] Pierre-Alain Giraud: Yeah, to be able to propose a piece where people would form a group and would collaborate also as a group. So normally it's ten people together, that's the piece we presented at the Pompidou Center. In Tribeca it's only three people experiencing it together. But we saw, as you were saying, also some students that came maybe from high school and they were at the beginning very turbulent, they were running everywhere and suddenly when there was the trial scene, then silence, total silence. And they start to create something as a group around Claudette. and behave totally differently and follow her and so it was quite magical to see that happening in that space and as a group of people experiencing things together at school and then they experience things together leaving the history of Claudette together. So for us, augmented reality, that way made really sense. We understood that also we were pushing the technology that way and putting all those efforts to do the story of Claudette like that. And I think it's a great tool to speak about history. and to give a perspective and to immerse young and not so young people but to immerse people in history because it creates a link that is very relevant between history and the present you live. That's interesting to deepen that path and there are many other things I think we should try to tell with that technology now. Because now we have something, we have a new form, and I think that's the first project we do with that. But there are many things to tell. And for the architecture, how we thought about it, we decided to have quite a minimalistic set, a place that you can actually relate to because it's benches, it's walls, things that you know. And from those things that are quite common, build virtually with little details that we add, like a lectern that we put on the lectern, some papers, and we put some kind of seats. It becomes something else. It becomes a church, then it becomes a bus, then it becomes a courtroom. And all this changing of architecture, I think, also creates a strong link to the history we are telling and our presence. So you remember it differently, I think, from VR. It's a different thing that happens in your brain and that lasts, I think, differently in the way you experience it and the way you assimilate it.
[00:28:43.503] Kent Bye: I think the fact that I'm able to walk around a full space gives me a different spatial relationship. There have been different VR pieces that do a one-to-one mapping and have had other people, but there is a difference of being able to see people, maybe a little bit of their facial expression, although their eyes are completely occluded by the headset. So I think there's different trade-offs of the different type of social presence you can get in a group cohesion. I actually saw this piece almost two times like it crashed with like five minutes to go and I watched it the second time and normally I'd be really upset or annoyed but this is a piece that the second time I kind of knew where the beats of the story were going to happen and so like you were saying you almost turn into a cinema photographer as you see the volumetric scene unfolding. I'm framing my shot to see like what the scene is like and it gave me an opportunity to try different angles and there was one scene where Claudette's being arrested and The first part of like a four sequence part of her getting arrested actually happened behind me the first time and I missed it. But there's like the sequence of the animations and that freezes and then it goes to the next stage. And so I was able to go to each of the different stages of her being arrested and observe it as if I was shooting it as a cinematographer. And so that was cool to be able to see that because the first time I had seen it, I had missed the sequence order because it was not in my field of view. But the other scene that I wanted to point out was when she realizes that the judge rules against her, she's guilty and she collapses and you have these lighting effects that are coming in. And I actually saw the flash of light on the ground. I was like, wait, is that AR? Is that a lighting effect? And then I was like, I was seeing other lighting effect with a thunder. And then, you know, I have noise canceling headphones on. And so I felt like the wind. And then as I've been doing interviews, it's like, who's doing a vacuum cleaner? And so there's these like, There's like this wind that's happening that I, you know, not aware of until afterwards. But there's, in the experience, I was like, wait, I feel like there's a breeze that's happening. But both times, actually, when that scene happened, I found it deeply, deeply moving. Just moved me to tears to see this moment of all the rain and her collapsing and something about the intensity of that moment along with all the other lighting effects and wind and haptics and the music and everything kind of culminates into this real climax of emotion of the piece. So anyway I'd love to hear any reflections on architecting all the multi-modalities of the wind and the haptics and the music and the lighting as well because I think it's fused all together in a way that As a VR person, I'm sort of noticing some of it. There's probably even stuff that I'm not noticing, but yeah, just a really well architected as everything's coming together.
[00:31:21.477] Stephane Foenkinos: First of all, I always say to Pierre-Alain, I said, I trust you on everything. So I know he's been the brains with the great team that we have. You have to talk about the fact that we have a great team that is building this to make it possible digitally and as an experience, because this is not my background once again. I barely can put a nail in a wall. But the first thing I told him, I said, the technology is amazing. I see what we can do. But it doesn't have to be stronger than the narrative. It has to serve the narrative. But for the rest, it's like a great theater piece. It's like a movie. Of course we love effects. We're using this for an emotion. We have the sound, we have the music, we have an Oscar winner helping us, Nicolas Becker, we have Valger Sigursson who's doing the music. It's essential for the elements that you have collaborators who just enhance everything you do, especially when you do augmented reality. But Pierre and I were so excited, it was like a kid. He would say, oh, we're going to put the wind, and then we're going to have the flashlights when it's the storm. And I said, yes, because it's part of the emotion you want to create. And here, as you say, this is great, because this is what we expected. You don't know where it's coming from. Is it normal that it's happening right now? But it participates. And we try not to, next time we will add smells. But I love, some people really feel like it's raining. I love that. In Pompidou it was bigger and the sound was different. We had different sources because we used speakers. People really felt with the wind and the sound was so precise. People really think that they were wet. But it's part of what you can do with this technology. In a theater, you can do a lot in terms of lighting, in terms of sound, but you don't have that quality of all the emotion at the same time. And just like you, I feel like when it's raining and it's collapsing, this is one of the most difficult moments, even for us. Watching it is like, but we don't want to be hyper sensitive. We never wanted to be I mean, of course we follow Tanya in the fact that we are depriving you from your rights so that you can feel Little by little what this young girl feels even if that could never happen but at least you have all the elements to feel what she feels and what we have as tools is light music sound effects, so we can approach, at least, we will never be here, but we can approach what state she could have felt.
[00:34:16.784] Tania de Montaigne: And what is interesting is that all these elements, even when they are combined, there is space for you. So your imagination is working. And this is the space I wanted to be there because that's the most important thing in the book. That was my goal, not to harass you, but to let you feel what's happening, not to be violent, just to make you feel and after that you're going to work with that and you're going to find a way to be next to Claudette, not up, not down, just equal with her and share something about what she's living. And here what was interesting is that Those two guys find the way that there is everything, there is object, there is a lot of stuff that couldn't take the space, but actually there is this space and there's a name for that, that's poetry. That's the moment that somebody is dealing with all this material and doing something else and bringing something else and this is you, you are bringing you and you are mixing with that and poetry can happen because when this moment precisely I hadn't written this moment so they have to find what kind of images can express that and the first time I saw that it was very moving for me I wanted to cry because that was beautiful to see this little girl realizing that a part of her life has ended and she is there collapsing and in the same time you can see what love could have been hers. So she's dancing all around the room. And I find that really beautiful to show that because life is only projection. You are dealing with what you would like to be, who you are now, what am I going to do. And you can feel, every teenager feels that. segregation apart. This moment that you think that everything is ended and what is interesting is that Claudette Colvin is still alive. So she has done something with this despair. She has done something with this end. She created a beginning with that. The reason why I think she's really an Aryan is because she's there, still there.
[00:36:41.834] Kent Bye: Yeah, so that scene of the rain, you know, the rain is transcending the walls of the boundaries and it's like you're completely immersed in this rain scene and I'd love to hear any of the other effects of the wind and all the stuff that you have to pull off that emotional peak moment of the piece.
[00:36:58.519] Pierre-Alain Giraud: I think also that moment we used that song from Nina Simone, the version of Strange Fruits, which is a very powerful song. And it came together little by little, I think. It's by watching it over and over again. First, we didn't plan on putting Claudette dancing, for example. and even the rain, so we had a scene, we knew she was going to fall down, and then we had this moment with the song and the text of Tanya, and little by little, I said, oh, maybe it's good to add Chloé dancing in a slow motion, in that, and she would multiply, and then the end came really very late. And I said, are you sure? Are you sure? Okay, let's try it. And then we made the rain. First the rain wasn't really working because it was maybe too white and so we had to work on the look of the rain. and also the scale of the rain, how far it would go, from where it would fall, and we worked also on the floor. When it falls down, you see a little impact of the rain, a splash of the rain on the objects too. So we worked on that, and I think it created something quite poetic for this moment. And the lights, we worked with Philippe Bertome, who is a great light designer. And in the original piece, we have actually like a path of light which goes across the whole room. And this path creates the lightning, you know, so it's something that was built little by little with different incrementation of the scene.
[00:38:39.044] Kent Bye: Until we arrived to the point we thought the poetry was there and that was enough for what it was basically Yeah, it's very Shakespearean in a sense how the character may be going through something but the whole world is also transmuting at that moment so having a lightning and a thunderstorm and the collapse and the Nina Simone Strange Fruit song and her collapsing and Yeah, I watched that scene twice back to back and both times it was really deeply moving. So yeah, I think you were able to pull off that poetry. And I think it brings the overall arc of the story that there were real stakes that were happening here and to hear how she was erased from history because of this decision not to charge her for violating these laws of segregation. And without that, it wasn't able to go to the same status of what would later come with Rosa Parks. So we see Rosa Parks later and then near the end we have this transformation of this scene that we've been in where we've seen different artifacts from segregation in the United States of like the difference between water fountains of whether or not people who were white or people who were African American or black or in that context of the time the colored which is the name of the piece. But at the end we have this Expanding out into the scenes of nature and also with the video I think is another component of this where it starts with us watching a video where you're on screen narrating different parts and there's an ability to take us back into the deep south with Video of cotton and farming that was happening with the legacy of the slave trade there So there's other elements are able to add in with the video component which as you're watching augmented reality you're able to have a projector onto the side of the wall that then is able to bring in this whole other video element. So you're bringing in these point clouds, these volumetric captures, the videos and the sound design and this sort of architecture at the end with expanding out beyond the scope of the space to have these scenes of nature that's also transforming. So you're in this single space that has transmuted through all these different iterations over time and Yeah, like I said, I think the whole journey that you take us on through this piece is able to take this deep dive into the bus context, the church context, the judge and the jail and the arrest. And yeah, I think it works quite well to have this transmutation of space while people are walking through it. So yeah, I'd love to hear any other comments as you reflect on all these different scenes and elements and how you end the piece.
[00:41:06.155] Stephane Foenkinos: You mentioned the screens and the archives, it's very important to me because at one point in a world when we never know what's true or not, in a world when we don't have sometimes a real handle about the news for instance, it was very important to show real news, to show footage from that era saying this is not disconnected, this really happened guys, you know, it's not like yeah oh my god it's a terrible story, thank god it's only a movie, no, it's a real story and This image is just to remind, even for us, it's good. We had a lot of young people coming in, and they say, you can hear segregation, you can hear about racism, you can hear about how some people decided that some humans were objects, or were separated, and it was normal, and it was 60 years ago. And of course, we are French people coming to talk about an American story, but for us it's universal, the same way that Tania wanted to do it in the first place. it talks and it appeals to everybody so all this architecture is here to remind you that there's some realness in this and we can generate computer generated images we can use the highest technology But in the end, it's only humans. And the human is what matters. Don't forget the human. So it's great. You can have all these emotions, all these movements, but don't forget, you are human. This young girl was human. And that is also a real story. And when you come out of this, Tania always say that it's important that you, maybe it stays with you. And that's all we want. That's all we all want when we create art, or at least, I don't know if it's art, but a work that stays with you. And this 15 year old girl, which we always say it's like a 10 year old girl today, makes something that huge. We have to honor her. That's it.
[00:43:26.297] Tania de Montaigne: Yeah, and what is interesting in the using of archives is that it shows that it is our memory. We are dealing with that, we are growing with that. The way we are talking is full of this silence, of these sounds, of these words. Our representation is about that, and when you want to change something, you have to know where you come from. and you have to know that this is the ground. So archives is about that. Let's review that and just to make sure that it has happened and it always happened. Fascism is about that, it's about ordinary days and then you move some stuff. You were talking about the bench and everything is the mutation, but this is us. We are full of past, full of these memories, and we are mixing that, and this is present. So, if we want to build a more interesting future, we have to know that. We have to know that this is the way we are living. This is us.
[00:44:31.877] Pierre-Alain Giraud: Yeah, I think what links us, the group, is the relationship to history. I think it's very important for us and we tend to be in a society where we think we are people of instincts and our nature is what represents us and we always forget that we are made of history and in every word we say there's the construction of that word has made centuries and the language the words we say know more about us than we do and that's history So we have to learn about history and we have to make it something fundamental in our present to be able to build the future together. I think that's the point of that work.
[00:45:23.014] Kent Bye: Awesome. And finally, I'd love to hear what each of you think is the ultimate potential of this fusion between writing and theater and immersive storytelling through augmented reality and what that fusion might be able to enable.
[00:45:40.228] Stephane Foenkinos: It gives a lot of ideas. And at the same time, I want to retire. So I don't know. But joke aside, I think I was getting not tired, but I think it's very important that you know that when you start a book, a play, a film, it's going to be two, three, four, five years of your life. So you need to really hang on a subject that really matters to you and means something. I think that if we think of this tool as only recreational, it would be fun, but it's sad not to use this amazing technology, not to do meaningful pieces. That's all I'm saying. Even if it doesn't have to be serious all the time, but yes, it's very inspirational in terms of Wow, there's a new door opening and what do I do with this now? What do I tell? But it has to match at least what we try to do here or to reach here in terms of meaning in emotion.
[00:46:53.298] Tania de Montaigne: Yeah, I'm a newcomer in this world. And what I could have seen when we were in Taipei, we can see a lot of AR and VR pieces. And as a writer, I was a little bit sad because most of them, they were no writer, no narrative, no... It was beautiful technologies, but no story. And I think that the new... interesting thing will be to permit the writer to come in the game, because it has to have a pen and paper, just a side computer, and if there is this movement... What is interesting in our piece is that there's a lot of people that come there. We were talking about the music, and the music is very important in the piece. We were talking a bit about light, and Philippe Bertome creates something special, and that's interesting that it's open. And I think the new goal will be to open the game, the more we can, just to permit to other arts to come in. And now it's going to be interesting, I think, if we can do that, to share that, and to, I don't know, there is a dialogue and style between all these arts together.
[00:48:10.911] Stephane Foenkinos: Very quickly, I just remembered that I always loved interdisciplinary, even as a teacher in my previous job, I would love that the history teacher, math teacher, science teacher, English teacher would work together and to find a common ground because I think Once again, I'm bored with myself. I don't like to do things. Some people, they love to do everything on their own, but it's a choice. But the collaboration is very important. And so interdisciplinary with this kind of tool, wow, it gives another meaning. It sets another level. In the 1930s, you would have choreographer Balanchine working with Paul Poiret with costumes and Picasso and Coco Chanel. And they all work together to create art. I don't compare myself to these people. Even if Tania is probably our Colette. But yes, I think this was a dream of mine to have interdisciplinary art.
[00:49:12.299] Pierre-Alain Giraud: Yeah, maybe I said it already, but I think now we have the beginning of something, of a tool that we have to open to other artists, to writers, to even choreographers. I think there's really something that we can start with that. It's a new form to explore. It's nearly a film that you can walk through. So we now have to share the technology and do more with others.
[00:49:41.913] Tania de Montaigne: It's in front of my eyes the whole time.
[00:49:56.398] Kent Bye: Awesome. Well just a flash of insight a good way to end the interview But I really really appreciated this piece colored and like I said, I think it's the one of the strongest augmented reality storytelling that I've seen and come across and the way you're able to tap it in the history and create this sense of memories through the holograms and allow us to move through the space and yeah, just the blending of all the different theatrical techniques and the sound design and I think it all came together with the storytelling through it all. This transmutation from the book to the play to now this immersive installation. Yeah, so congratulations on the piece. It's a real technological and storytelling achievement. And yeah, I just want to thank you for taking the time to help unpack it all. So thank you.
[00:50:39.583] Pierre-Alain Giraud: Thank you. Merci. Thank you so much. Merci.
[00:50:45.025] Kent Bye: So that was Tanya Dementing, provider of Coloured and bookman Claudette Colvin, also one of the actors, and then Stéphane von Kynos, the co-director of Coloured, as well as Pierre Alangiro, who's the other co-director of Coloured, as well as a co-producer. So I've had a number of different takeaways about this interview is that first of all Well, I think this is one of the most effective immersive stories that I've seen in augmented reality so far and I think part of it is that having a mixed reality component of spatial context where there's places to sit down and just the way that they're able to recontextualize the spatial context around that so to transform it from a bus and into a church, into a courthouse. So there's lots of different ways that they're able to take the existing physical things that are in the environment and then transform it. But also the fact that it's using this installation space where everything's like a black background, it just provides really great context to be able to project any number of different things on top of it that transport you into these different scenes. So it's covering the life of Claudette Colvin and these different scenes and vignettes from her life but also to take the spatial journey through key moments in her life and to show the broader context of what was happening at that time and having different artifacts from that era of American history that are also shown in the context of that world. Yeah. And you're wearing noise canceling headphones. And so you don't hear the engines of the wind machines that are coming up. If you listen closely to a lot of the different interviews that I've been doing at Tribeca Immersive, you'll hear what sounds like this vacuum cleaner in the background. And that's the fans that are turning on to have wind components at the course of the experience. And yeah, just some really powerful moments, especially the emotional climax to the piece when Coletta Colvin collapses down to the ground and they have, raindrops that are falling around you, the spatialized sound around that with the wind that's also coming in, and the Nina Simone's Strange Fruit song that's also playing. Really evocative song and just a really powerful and emotional moment. So really well architected to bring all those things together. They were doing a lot of different spatial capture within the context of Taiwan and they said that they had to cast people who were non-actors and that's probably one of the main complaints that I heard about this piece is that didn't have people that were coming with a deep acting background to play out some of these different scenes. But overall, I feel like that this is a good example of a story that's told across a book, that's told across a play, and then it's told across an augmented reality experience. And they're able in the AR experience to pull in all these different influences. And so you're walking around this room that's black, but you see like a projector that is showing different scenes. And so they're able to set the initial context, but also show different archival footage at some point, but also use that as a way of giving additional information using other media like film. And as you have the HoloLens on, you're able to still enjoy a lot of those things as if you didn't have it on, because it's a pass-through. So they're not having to recreate all this within the virtual context. And having other people there as well, I think is a component where you feel like you're traveling through the spatial context with other people. And based upon where they're at, what they're looking at, because there's so much stuff that's going on, it can help see the wisdom of the crowds and what folks are doing, what they're looking at to help also direct attention as well. I haven't necessarily seen a lot of stories told in augmented reality. I mean, I see a lot more phone based stuff and the stuff that I've seen in augmented reality has been more environmental based or not such a strong narrative of this one. I think there's a strong through line. of a story in this piece that I think is a little bit stronger than what I've seen before, that it's just sometimes harder to tell the full complexity of these different stories. What comes to mind actually is there's a number of different audio-based augmented reality pieces that I think are able to dig into more nuance of some of these different stories. But yeah, I just thought it was overall well done and I was moved by the piece and just thought overall it was really well executed. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast, and if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and I do rely upon donations from people like yourself in order to continue bringing this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
