Fortune is a series of 8 Snapchat and Instagram AR filters that explore the topic of money and our relationship to money. Each project is around 90-seconds in length and uses a variety of immersive storytelling techniques from tabletop-scale animation to interactive facial filters to experiences AR-enabled interactions. At the end of each episode is a question designed to kick off a broader social media engagement campaign and group discussion about money. It should be premiering on Arte’s social media channels later in the fall, and I had a chance to speak to the production team at Tribeca Immersive including Emilie Valentin (Writer of Fortune), Claire Meinhard (Director of Fortune), and Aurélie Leduc (Producer of Fortune working at Atlas V).
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the potentials of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So I'm continuing on my series of looking at different immersive experiences at Tribeca Immersive. Today's episode is about a augmented reality filter piece called Fortune. So this is a piece that was taking an augmented reality story that premiered at Sundance back in 2021 and they were able to reduce it down from like seven minutes down to a minute and a half and creating a six to eight part series that's going to be a number of different Snapchat and Instagram filters so that it tells an immersive story about money and people's relationship to money from a lens of a number of different people and stories and contexts. And it has questions at the end that it asks you so that when they eventually launch this later in the fall, then people will be able to provide an answer and then create a group discussion based upon these different stories that they're providing here. So it's using the augmented reality as a form of storytelling and using both facial filters and kind of tabletop approach of telling these immersive stories. Yeah, just trying to innovate and reduce down all of this different complexity of these scenes down into like four megabytes in the context of one of these Snapchat and Instagram filters. So that's what we're covering on today's episode, the Voices of VR podcast. So this interview with Emilie, Claire, and Earlie happened on Thursday, June 8th, 2023 at Tribeca Immersive in New York City, New York. So with that, let's go ahead and dive right in.
[00:01:46.441] Emilie Valentin: I am Emilie Valentin. I am the writer of Fortune.
[00:01:51.919] Claire Meinhard: I am Claire Mena, and I'm the director of Fortune.
[00:01:55.641] Aurelie Leduc: I'm Aurélie Leduc, and I'm the producer of Fortune, and I work at Atlas 5.
[00:01:59.964] Kent Bye: Awesome. And then you could each give a bit more context as to your background and your journey into working with spatial computing and AR.
[00:02:07.447] Emilie Valentin: I am a documentary director usually, so I came in to find stories that would fit in the project, and I often work on interactive and new ways to tell stories, so this was particularly interesting to me, since talking about money is not a very light subject, but augmented reality makes everything a bit more fun and makes people curious about it.
[00:02:36.606] Claire Meinhard: I'm an art director and I work with augmented reality, also printed media. I come from graphic design and that was a long and new journey for me.
[00:02:49.666] Aurelie Leduc: I've been producing audiovisual and new media content for like five years, six years now. I produced my first VR experience in 2019. It was called Minus 20.7 Degrees. It was directed by Yann Cunhen at the time and then I joined Atlas 5 in 2022.
[00:03:09.139] Kent Bye: Yeah, maybe you could give a bit more of a backstory for how Fortune came about, because I saw the first iteration that was showing at Sundance in 2021. It was a production of NFB and Brett Gaylor that I have an unpublished interview, but maybe you could talk about how that Fortune iteration that was showing at Sundance 2021, how it evolved into this series of Snapchat filters that's showing here at Tribeca 2023.
[00:03:31.110] Aurelie Leduc: So it was actually like a very interesting journey about how to showcase AR. So before it was an AR app that you needed to download on the store, and it was like episodes of seven minutes around, yeah, around seven minutes. And then we had a discussion with Arte, which is our co-producer and broadcaster. And we were like, okay, so it's very difficult to get people to download AR app, especially for narrative project. So we just thought with Arte that the best way to showcase AR was just to use social media like Instagram and Snapchat that have AR at the core of the experience you can have through filters. So we just turned around the mechanic and the format of the filters to try to invent a new form of storytelling. and so use the filters to bring documentary series using AR animations and bringing interactivity thanks to Claire who is really used to do interactive pieces as well. So yeah, that's the basic story of the series.
[00:04:33.230] Kent Bye: Right, so it's a six-part series now, and I know that there was the series with Frank, The World's Greatest Counterfeiter, that was the first episode that was a little bit longer in the first iteration of it, and then you said that you had to shrink everything down to like four megabytes for the snapshot filter, so really getting down to the essence of each of these stories, but I'm wondering where you started with the design process, if it was the writing process or trying to rapidly prototype some of the ideas to see what would work, and so, Yeah, I'd love to hear, where did you begin with this six-part series of Fortune?
[00:05:05.273] Emilie Valentin: I think we began with the stories, because the main goal was to attack that big mountain that is money in our society, but by various angles. So we started with the stories. But at some point, we needed, like, is this story interesting in itself? Is it interesting in AR? So really, we focused on the story that had the most potential in AR. So quickly it became a story about what the art direction, how are we going to fit polygons into that, how can we tell this in a super minimal way but effective way. So we started with a story but we pretty quickly added all the direction of the work and the visual part and chose the story from that angle, from that part on.
[00:05:58.118] Claire Meinhard: The project was shaped with everyone and we tried to adjust every part of the story to fit in a certain way. It depends on the technology we wanted to use on each episode.
[00:06:12.604] Aurelie Leduc: The idea was also like it's going to be a social media project. So we had a lot of discussion with the Arte team about people need to be cached very fast on Instagram and Snapchat. So the goal was to have like very personal stories and the stories needed to fit in 1 minute 30 and be interesting and have a bit of nuance. So it was kind of tricky to find the good stories and then stories that was actually interesting to do in AR. And then that's when Claire came by and said, OK, what are we going to do with AR with this episode? And she came with so many different interactive pieces, hand tracking, turning around objects, face filters, and so many different ideas.
[00:06:52.486] Kent Bye: And I know that with this piece, you have what you would call tabletop, where you have things in a little bit of a small scale, so you see the story unfold. It's an affordance that I saw a lot of early VR stories take, but I think that's an affordance that works quite well with augmented reality storytelling, because then you're able to put these small characters in the scenes that are unfolding. So I'd say probably three or four of the pieces I just saw were in that vein of the tabletop. And then you have the camera facing yourself, and there's some other mechanics that you're exploring. Maybe it's worth going through chronologically from when you produced each of these. So which one did you work on first? And maybe we can talk about each of those.
[00:07:30.298] Claire Meinhard: We worked on Diddy first, that was our crush test and an experimental way to work together. So we started with an episode which is not so interactive, that's more like a mini-theater. And then Frank, which is a more animated one part.
[00:07:49.245] Aurelie Leduc: And then... And then it was Rodrigo, I think the face filter. Yeah. So it was like the first was very narrative and then we thought like we really want to bring interaction because we thought it would be very interesting and it actually started with Rodrigo so the story is about food waste and how Rodrigo is getting into the waste and getting the food back and redistributing to people and that's when Claire came with the idea like okay we need to make that interactive and get people to gather the food and then distribute the food exactly as the character is doing So that's how it started and then I think then all the episodes are interactive except William, that was like the sixth one.
[00:08:33.937] Claire Meinhard: We wanted to make the user move around the object, the virtual object. We wanted them to take their devices and to become an actor. So yeah, that was the main point.
[00:08:47.632] Kent Bye: The episode where I'm turning my head left and right, and that's moving this virtual bicycle on the bottom of the screen that's catching the food, and then it flips, then the bicycle's on the top of the screen, and it's dropping the food across these three different folks. So I thought that was actually a really fun mechanic, because you're turning your head and you're listening to the story, but I guess it was a form of interactivity and I'm not necessarily making a choice that's changing the outcome of the story, but it felt like I'm using my body to have a deeper sense of embodiment as I'm watching it. So it's like a mere neuron effect of me looking at myself. Because a lot of the other ones were 3D models in a spatial context, but this was more of capturing my spatial embodiment. And as I move, then I'm abstractly moving this 2D representation of the bicycle. So anyway, I had a lot of fun with that piece because it felt like it was a little bit different mechanic. And I haven't seen a lot of that using that facial filter mechanism where you're looking at yourself, but you're also seeing how you're moving around is having something happen and got this delightfulness to it. It just made me smile and laugh as I was doing it.
[00:09:51.065] Claire Meinhard: Nice to hear.
[00:09:52.606] Emilie Valentin: Thank you. What's fun to me to watch the process is that every time Claire became a little bit more comfortable with like, OK, this can fit in the filter. This can be a filter. So let's try something new. Let's push the limits. Make it a game. Make it something you turn around. She was always trying to do something different because there was so much to research and develop.
[00:10:17.508] Claire Meinhard: That can be trends for people who are not familiar with our experience that really are subject to us with the social media diffusion to make them like this technology. So we add to a different way to make them like this. Yeah.
[00:10:38.476] Kent Bye: And so are these filters publicly available yet? Or are they just premiering here? Can people at home be able to start to see these already?
[00:10:46.027] Aurelie Leduc: The series is going to be released in October by Arte, so on their Snapchat and Instagram account in French. So for the moment, we have the English version to show at Tribeca and Festival, but the English version is not going to be broadcast yet. So we're looking for partners actually for this one. So yeah, October in France.
[00:11:04.923] Kent Bye: And what was the average runtime for each of these different six episodes?
[00:11:09.267] Aurelie Leduc: A minute and like 30 or 40 seconds. Yeah, under two.
[00:11:13.185] Kent Bye: OK, so each of them around under two minutes, and there's six of them, so around 12 minutes to get through the whole thing. Talk a bit about how you were able to make sure that it was working and the user testing portion of it.
[00:11:26.292] Claire Meinhard: Yeah, as every new technology, that's a bit challenging to users and to us to teach people how they can use. That can be a bit shaky, but we tried our best, for fortune, to be the most user-friendly as we can.
[00:11:42.358] Emilie Valentin: They were using tests within the team, at Arte too, so we changed the call to action, how to make people understand very easily what they had to do, so they would focus on the story and the experience and not on the do I have to move, what do I have to do. So we had sort of a time of tests, but it's the first time we show it to such a broad audience, yet very used to new technologies.
[00:12:09.946] Kent Bye: Yeah, and the last piece that I saw, which I guess was the Margon salary bluff, but the mechanic was that I had to hold my hand out and flip it over. So I'm receiving these different cards and it's a bit of like I'm playing a game of poker and you're dealt different cards. But in her story, she's talking about how she was asking for more money than she was expecting. She actually got it. So the whole story is about asking for what you're worth. But I thought that holding out my hand had this other dimension of I felt like I was also receiving whatever was being given, but also like playing a card game and I see all these chips that are coming in and I'm winning at the game of poker in this context. But yeah, I'd love to hear a little bit more about that process of using the hand interactions to have people feel like they're actually engaged as In some ways, you're listening to the character, but by holding out your hand, you're almost embodying the character in some way. So the way that you're not just as a ghost watching it, but you're actually as an embodied character participating in the story. So I'd love to hear a little bit more about that dimension of telling that story.
[00:13:11.465] Aurelie Leduc: Yeah, I think it's always exactly like Claire was saying about How do we identify with the character? So we really wanted the user to use their body. So this is why you use your hand to embody Morgan. This is why you use your face for Rodrigo. This is why we want people to move around to look at the animation. And for example, like for William, which is our episode about money in prison. So you just have like basically a building prison in front of you and you can turn around and look through the windows to follow the narration. in the storytelling, so the body is really part of how you are going to discover the story. And we actually have two other filters, so it's like an eight-episode series, but the two other ones are not there. And they are also engaging the user in a different way because you have to turn all around you in 360 to find elements in your environment. And we actually have another filter for Lele, where you have to turn around you to trigger the next sequence of the storytelling. So if you don't move, the story is not going to go anywhere. So it's another way to engage the user in the story.
[00:14:19.856] Kent Bye: So showing it here, Tribeca, you have these posters with the QR code, and you're scanning it. And there's six here now, but you're imagining that there's going to be eight total. So how would you imagine that people would watch maybe just one of them? Or if people want to see all of them, how does the user interface work with what would be the best optimal way for people to just watch all of them? And how is that going to work? Is there going to be a website for people to go to and click on it? Or if they're going to have to click on and scan QR codes? Or what's the way to distribute it through Snapchat in that way?
[00:14:50.751] Claire Meinhard: That will be released as a series, so I think that will be twice a week, so that's kind of rendezvous, you know? You can see each episode and that's a proposal, so we don't really need the QR code mechanics for this festival, but we actually don't really use it like that, yeah.
[00:15:12.250] Emilie Valentin: It's going to be directly into the Snapchat account and the Instagram account of artists. They're going to push it through stories and carousel and stuff and it's going to appear within the filter where you can put on makeup on your face or have dog ears or whatever. You're going to have the fortune within it.
[00:15:31.490] Kent Bye: So Arte's account on Snapchat would put out a story and then have a link to the filter. Would you imagine that there would be a clip, just like a short clip, that was showing a segment? So then people would then have to click on the filter, download it, and then they would be able to see the full context?
[00:15:47.173] Claire Meinhard: Yeah. To download a filter in this platform, you have to have a capture of each filter. So there is definitely a sneak peek of everything.
[00:16:00.921] Emilie Valentin: We use the question at the end of every episode as an anchor to start the conversation and also to the filter content.
[00:16:09.928] Kent Bye: So yeah, each of these episodes showing here at Tribeca, it does end with a question. So you're imagining, once it's launched, that people would then take a screenshot of that? Or do you imagine them taking a video? I'm familiar with TikTok, how people will take a clip of something and then respond to it. But I don't know how you imagine if people are going to write text over it, or just create a video of them speaking. if you're trying to set a context for the story and then provide a broader conversation. So I'm just curious how you imagine that conversation playing out on the platform of Snapchat.
[00:16:41.758] Claire Meinhard: We definitely wanted to create a cinematic scene that can be taken in picture, in video. But yeah, the main point for us was to create an experience so we engaged everyone to start with the experience first and then if they want to, obviously, yeah, they can share some capture.
[00:17:03.253] Aurelie Leduc: I think there's two things about it. So first, I think the intention was like, we want people to reflect on their own relationship to money. So that's the first thing why we have questions at the end of the filter. So you can answer it for yourself, or you can just take a video capture of the filter you just did or do a video of yourself answering and then share it on your social media. We hope people are going to do that. That's the main point. Yeah.
[00:17:28.819] Emilie Valentin: I think Arte is going to have some stories and engage with the community to discuss this on the story boxes where you can answer or in the comment section. So hopefully people will answer it their own way through Arte or on their own.
[00:17:45.495] Kent Bye: And do you imagine that these are going to appear on other platforms as well, like Instagram and TikTok? They also have filters. So if you're able to produce it and have one filter go off across different platforms, or if there's enough differences amongst the different platforms that you almost would have to reproduce it. So yeah, curious to hear if it's going to be just on Snap or if there's other platforms you're planning as well.
[00:18:07.345] Claire Meinhard: That will be released on Instagram at the same time as Snapchat.
[00:18:11.947] Emilie Valentin: It's another technology, right? You have to redo it. I don't know how to explain that.
[00:18:16.610] Claire Meinhard: That's two different ways to create filters. So yeah, we did both at the same time. But that will be released on Snapchat and Instagram. TikTok has a different way to create and a different way to monetize everything. So that's not a subject actually, but we don't know. Maybe one day.
[00:18:38.702] Aurelie Leduc: The series was developed for Snapchat, so on Lens Studio, and for Instagram on Spark. So for VR people, it would be like developing the same episode on Unity and then on Unreal. So it's basically you have to do it again.
[00:18:52.694] Kent Bye: But they're short, so at least it's still a lot of work, but it's at least not like a full-length piece.
[00:18:58.640] Claire Meinhard: Yeah, definitely.
[00:19:00.501] Kent Bye: I really enjoyed the, I don't know if it was, is it Estelle that had the carrots? So yeah, and the piece by Estelle, I think that was actually the first one that I saw. There was like a farming scene that's a bit of a tabletop where I'm watching the story unfold and there's tractors going across the farm field, but then I'm asked to plant some carrots and so I have the little button and I'm planting carrots on this physical floor that's in front of me but I'm seeing the augmentation of these plants grow very very quickly so but it gave me this sense of cultivating garden but because most of these different AR experiences that I've seen have been more of this tabletop passive experience that was the first amount of interacting and It is just like pushing a button, but I think the fact that I'm also pointing the camera around these different space, I'm able to move my body around and decide where to put things. And so it gave me a little bit more of an embodied interaction, even though the actual mechanism of the interaction is just pushing a button. So I felt that worked actually quite well in a way that is, again, trying to incorporate my body into the experience. Yeah, I thought that mechanic was something that's different that I don't necessarily see in a lot of different VR pieces and I think works well in AR.
[00:20:11.801] Emilie Valentin: Thank you, we love that part. I mean, the fact that you can grow vegetables all around you and all around the space, in the city, wherever you are, you can make everything green and it's a nice thing to do just for yourself.
[00:20:25.413] Claire Meinhard: That even can be political to say, oh, that's your turn to grow your vegetables. So yeah, really like this episode, I think.
[00:20:33.477] Aurelie Leduc: Yeah, I think it was exactly the same. Like, how do we serve the purpose of the character? And so Estelle's story is about, we don't have any more land for farming because it's too expensive now and people prefer selling their land for building than for farming because they get more money when they sell it for building and not farming. So it's like a political filter in some way as well. So we wanted the user to be able, at the end of the filter, to grow their vegetables, have their lands. So yeah, it serves the purpose of the character in this one as well.
[00:21:06.301] Kent Bye: Yeah. And so if we go back to the very first episode that I saw back in 2021 with Frank, the world's greatest counterfeiter, that was a seven minute piece. And, you know, really diving into that specific story. It's a pretty wild story. And this is a very abbreviated version, but I'm wondering what was the catalyst to take that beginning of that story and to expand it out into this eight part series. What is it about money? that you felt was an important topic to not only cover, but also why was it interesting to use augmented reality with both Snapchat and Instagram filters to explore the topic of money. So, love to hear how the project came about, specifically covering this topic.
[00:21:43.503] Emilie Valentin: Yeah, I can start. Money is a huge topic. We said money is ruling everything. Sometimes it's true, it's taking a huge amount of space in our society. So we wanted to talk about that. But it's such a big story that we had to cut it into little pieces and individual pieces, individual stories that would connect more easily. AR came at that point, I mean, it would turn potentially boring subjects such as poverty or struggle with money into something more light and more attractive and it would be easier for people to get some interest in it.
[00:22:28.972] Aurelie Leduc: Basically everybody has a story to say about money. The series is really how does money rule our lives and how money dictates our choice in life. So this is basically what all the characters are actually saying and it actually really fits the AR purpose. Money is so important in our life and it takes so much part in our daily life. It's really like AR in social media. Every day I'm on Instagram, I watch stories, I watch video. So it was like the perfect way to use AR in your living room and get this story into your daily life as well.
[00:23:05.755] Kent Bye: Yeah, I'd love to hear your reflections on money in this project.
[00:23:09.838] Claire Meinhard: For me that was a way to question the whole society in a different position and in different visions. Because all of our characters talk about money but in a slightly different way. So yeah, for me that's a great landscape, a great window about society.
[00:23:30.852] Kent Bye: Great. And finally, what do you each think is the ultimate potential of spatial computing and augmented reality and immersive storytelling and what it might be able to enable?
[00:23:42.657] Emilie Valentin: Well, I think it's important to try and understand how this technology works and what we can do with it very quickly before it becomes something you don't want it to be. So it's important to have storytelling, to have content, to have reflection and diversity in the way you approach this kind of technology so you can make sure it's a tool of information and entertainment and something good.
[00:24:10.667] Aurelie Leduc: And yeah I think it was AR and mixed realities all over now because of the new headsets of Apple and then the new headset from Meta. So everybody was like yeah so you're doing AR with a phone like it was something old you know and I'm like yes we're doing AR with like the devices people use the most so it's still like the best way to get AR accessible to everybody in their daily life at home. Everybody has a phone. Everybody can use and discover the series. So I think that's how it gets powerful and interesting for the series because we are still using phones. Maybe in 20 years we will not be using phones anymore. But for the moment, if you want an AR series to be actually accessible and seen by people, that's, I think, one of the best ways.
[00:24:53.509] Claire Meinhard: To me, it's like the beginning of this technology on social media. And for us, that was a way to encourage air in Snapchat and Instagram. And I'm pretty sure that if we keep pushing this kind of project, the platform is going with us further.
[00:25:14.869] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:25:20.546] Aurelie Leduc: Nothing special, we're really happy to be at Tribeca this year and to have AR, MR, VR projects, so many different kind of XR format and I think it's very exciting this year.
[00:25:31.751] Claire Meinhard: Yeah, really grateful to be part of all this immersive and very creative project, so yeah, we're really happy about that.
[00:25:42.810] Kent Bye: Awesome. Well, thanks for taking the time to create this project and explain both how it came about and where you see it going. And I think it's using the affordances of Snapchat and the social media. It's something that we have had a lot of these projects at these film festivals, but having it actually distributed and seen by a large number of people, yeah, I'm just glad to see more of that using the existing distribution channels of these social medias to be able to push forward both what's possible with immersive storytelling but also with the technologies of augmented reality, especially after this Apple Vision Pro announcement that just happened within the last couple of days. So anyway, thanks again for taking the time to talk to me today. So thank you.
[00:26:20.273] Aurelie Leduc: Thank you. Thank you. Thank you so much. Thank you very much.
[00:26:24.755] Kent Bye: So that was Emily Valentin, she was the writer of Fortune, as well as Claire Menard, the director of Fortune, as well as Erlie LeDuc, the producer who's working at Atlas 5. So, I have a number of antiquities about this interview, is that first of all, Well, just to see how some of these different immersive story techniques that have been prototyped within the context of virtual reality starting to percolate into the augmented reality and Snapchat and Instagram filter context. There have been a number of different Snapchat and Instagram filters that have been shown at these immersive festivals over the last number of years, although I'd say this is probably the most robust in terms of exploring new structures and forms of storytelling using different techniques of animation and facial filters. And it's really pushing the edge of what's possible with immersive storytelling. But there's also these new affordances that I think are unique to the augmented reality format, especially when it comes to turning on the camera and facing it back at the person to both project your own image and likeness within the context of a story as you're watching something unfold. but adding some interactivity and gamification. And one of the stories where the main protagonist, Rodrigo, is trying to gather up extra food and then redistribute it for people who need it. And so you are moving your head left and right, and that is allowing you to capture the food. And then you flip the bicycle and it's upside down. And so all the stuff that was captured is then falling out of your bike. And then you're on the ceiling going back and forth and trying to redistribute the food. So yeah, just a really fun and novel interactive component, but to add your own identity and image and likeness and to have this way that you are moving your body and using your embodiment to interact with an experience in a story. And overall, they're trying to create the series of these, I think, a total of eight, but they were showing six of these different Snapchat and Instagram filters at Tribeca. And they'll be released in the campaign from Arte. And at the end of each of these different pieces, there's a question that is asking you to answer and reflect upon your own relationship to money. So certainly a very rich and complicated topic with money and a variety of different stories that are exploring this and also just a variety of different storytelling techniques and Using animation and tabletop affordances and using your body to walk around the scene There's another one when you're tapping on the ground to be able to plant crops and then you see over time what happens when those seeds that you're playing to grow into fruition to food as well, so Yeah, just a lot of really innovative ways in which that they're using the medium of augmented reality and the context of these spatial affordances that I think are pretty unique when it comes to both virtual and augmented reality and in this more platform based using the constraints and affordances of these filters. So they have to be super small, super short, bite-sized, but still get the essence of the story and provide enough context for people to respond to and facilitate a whole group discussion. So be very curious to see how that starts to play out as it gets launched here later this fall. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com. Thanks for listening.
