Francesca Panetta is a freelance immersive artist, director, journalist who worked at The Guardian experimenting with new forms of storytelling, then as a creative director at MIT experimenting with new forms of storytelling using AI, and now is an independent artist. We talked about the following projects that have been featured at DocLab since 2014:
- Francesca Panetta’s Portfolio
- In the Event of a Moon Disaster (DocLab 2019)
- Voices of VR #846 DocLab: Deep Fake of a Synthesized Nixon Speech that Never Happened
- Limbo (DocLab 2017)
- Underworld: a Virtual Experience of the London (DocLab 2016)
- Voices of VR #287: Explore the Psychological Impacts of Solitary Confinement in ‘6×9’
- Seven Deadly Sins (DocLab 2014)
This was recorded on Tuesday, December 14th, 2021 as a part of a collaboration with IDFA’s DocLab to celebrate their 15th year anniversary.
If you’d like to support the Voices of VR podcast, then consider becoming a member at Patron.com/voicesofvr.
Thanks for listening!
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
https://dts.podtrac.com/redirect.mp3/d1icj85yqthyoq.cloudfront.net
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. All right. So my name is Kent Bye and I do the Voices of VR Podcast and I'm here back again doing another episode in celebration of the 15th anniversary of the IFA DocLab. And I'm here today with Francesca Panetta. Francesca, why don't you go ahead and introduce yourself and tell me a bit about what you do in the realm of immersive media.
[00:00:31.030] Francesca Panetta: Sure, yeah. I am a freelance immersive artist, director, journalist. I spent a long time at The Guardian experimenting in new forms of storytelling for them. And then more recently worked at MIT as a creative director there, again experimenting in new forms of storytelling. There I was using AI. So done kind of all kinds of different types of techniques over the last 10 years or so. Now independent.
[00:00:59.906] Kent Bye: Okay, great. So maybe you could give a bit more context as to your background and your journey into this type of interactive storytelling.
[00:01:07.858] Francesca Panetta: Yeah, well, my background is in sound. So I trained as a classical musician and then radio producer. So originally at the BBC in radio. And then I moved over to The Guardian in 2006 when podcasting was really just starting. And it was a fantastic place for experimenting in new forms of documentary and journalism. And quite early on, I saw that even though podcasting now doesn't feel like particularly experimental media back 15 years ago, suddenly without the restrictions of broadcast time limits or specific formats, it felt like a very free space. And I began to get a bit of a taste of what working with high level journalists and kind of stories could be when you're unbound from very traditional formats. So we really stripped apart what radio could sound like, what podcasting could be, and began to experiment. And then I moved from there to working with developers and designers and data journalists and filmmakers to begin to make some of these web documentaries that we all remember. Everything from the kind of scrolly newspaper formats to the more kind of glossy NFB kind of formats. And I partnered with a number of different organizations during that period to really see what different types of feature making could look. So partnered with the Imperial War Museum to do a big interactive on the First World War, with the National Film Board of Canada to do a piece about digital morality. We did a piece about climate change, and forest fires in Tasmania. So the topics have always been nonfiction, but on a really wide variety of subjects. And then out of that, I set up a virtual reality department in partnership with Google, which lasted for three years, where we made 13 pieces of VR and 360 content, really trying to experiment in different aesthetics forms, different types of interactivity, with the kind of luxury of like a decent budget and fantastic infrastructure from the Guardian, what could different VR look like for non-fiction?
[00:03:32.890] Kent Bye: Yeah, and so with the ITHA, when was the first time that you came across the ITHA DocLab community?
[00:03:39.394] Francesca Panetta: I can't remember which year now. It feels like a long time ago. I mean,
[00:03:43.983] Kent Bye: It looks like you had the seven deadly sins at DocLab in 2014. I don't know if that was the first project.
[00:03:47.985] Francesca Panetta: So it would have been way before that, you know, a few years before that. I'd been coming for a while. I mean, it's a really, really special community. It's an exciting group of people, both in terms of the DocLab people who put it on, but also the people who come, the audience members as well as the makers. And, you know, it feels very welcoming, but also has a kind of spark and excitement being here. I mean, it's interesting thinking about what makes it unique as a community, because I definitely feel that. And part of the reason to go to DocLab is, of course, for the programming. But I'd say for me, it's about the community. It's a place to be inspired and connect. with people to grow ideas. And, I mean, Kent, this is your series. I wonder if you've managed to think what it is that has made that possible from talking to the people that you have done? Like, are you finding threads about what it is that gives it that kind of magic?
[00:04:53.305] Kent Bye: Well, I think the part of the thing that I've been meditating on a lot lately, at least, is these concepts of potential and how non-interactive media has no potential, meaning that it's already completely authored and there's no participation, there's no requirement for interaction. And just the ways that as people watching these pieces are making choices, then they're able to see something that may be right at the right moment that they need to see it. Or it's something where their participation is somehow changing the outcome of what is made and it's actually in collaboration with the audience in a way. So I feel like the audience of folks at the doc lab are grounded in documentary. It seems to be that there's a reflection of these deeper patterns of character or story that it's grounded in something that has a very established context. And I feel like with fiction, it tends to be a little bit more that you have to build the entire world to be able to tell that story. Whereas with nonfiction, the sort of larger contextual aspects are there and that you're able to then start from there and then start to experiment maybe a little bit more with these other aspects of potential and interaction. So I'd love to hear some of your reflections on that since you've been kind of at this frontier of immersive and interactive media for a number of years now.
[00:06:11.635] Francesca Panetta: Yeah, so you think it's because of the documentary aspect of it that marks it out from the other film festivals, and maybe that's right, that that focus and who that attracts is different. I also think that the team here does a lot of work in community building in the way they program the events, deliberately bring people together. It feels effortless in that it's kind of fun, but it's also, I think, really carefully choreographed to build community as well. So I think it is both the setup of it being documentary and interactive, but also some clever people behind the scenes who are bringing those people together and making sure that those conversations between people happen and that there are those spaces for them as well.
[00:07:00.982] Kent Bye: Yeah, well, maybe it's good to chat about some of the different specific pieces that you've had at the DocLab over the years, and we can maybe unpack some of those different themes from there. So was The Seven Deadly Sins, do you recall if that was the first project that you had at DocLab in 2014?
[00:07:14.110] Francesca Panetta: Yeah, I can't remember, but it sounds about right. That was a project that The Guardian did in collaboration with the National Film Board of Canada. And it was a long project for us to make. and probably my first kind of experience of the difference between newspaper goals and deadlines and production processes and the National Film Board of Canada's. So the kind of desires for the project were quite similar, but it was a real challenge to figure out what that workflow was because I had been working on, I was special projects editor at the time, and I had been given a lot of freedom to have these long, long leads for projects. But even still, it was like nothing like the National Film Board of Canada's processes. They invited me, Lockdown invited me out to Vancouver to do a kind of labbing session for a week. And that was just incredible. Suddenly, I found that the kind of process of ideation and development was very, very different from working in traditional media. and really, really fun. So that was the beginning of that project. And then we kind of went back and forth with different story ideas and different interactions. We were sure that at the heart of it, we were sure that we wanted to be asking audiences, has their sense of ethics changed by coming through the internet? Like when you're on the internet, do you have a different sense of morality and ethics? And we ended up, the kind of core of the piece was a kind of questionnaire, like, is it all right if someone else does this? Is this something that you do? And then you saw the statistics. And then we wrote a number of Guardian stories for it, and we had a wonderful illustrator, and we did a number of videos as well. So it was, in terms of interactivity, it was a relatively simple piece. And in terms of kind of you know, what an interactive documentary can take in form. Not one of the most complicated, but actually it was incredibly successful in terms of the numbers of people who visited it and in terms of how people liked it. Like, I think that it is something that people have been thinking about a lot and the stories were very strong. And it also shows that, you know, at that time, there were extremely complicated interactive web documentaries, you know, these nonlinear ones, which would go in all kinds of places. And I think it also showed that having a kind of clear concept and a clear form can be extremely helpful for audiences as well. That marring something in a huge number of branching tree kind of lines, I mean, it can be great, but it's just not necessary sometimes. And so it was a kind of very clean cut, effective piece.
[00:10:02.911] Kent Bye: Yeah, it's still online. I was watching some of it this morning while I was waiting. And yeah, just to be able to go into one of the seven deadly sins. And so there's different themes that are there. And then you have a video and then there's a number of stories. And then you have these different questionnaires like, are you illegally downloading stuff online? Do you think that's OK? and then you get the survey results. So then you see, okay, this is like a little bit of a sampling to see what's the larger normative standards here for what are people doing and whether or not they think that's okay, whether they are condemning themselves or condemning other people. So yeah, there's that element of choice that you're able to pick what theme is maybe the most interesting to you and then watch or read everything about that or kind of jump around if you're get the gist of one of the topics or not. So Yeah, I think as I think about the principles of that participation, I'm getting reflections of the larger community from that based upon people who that as more people do it, then it becomes a little bit more interesting in terms of the statistics, but also just the be able to make those choices of like what order, because you could sit down and watch that as a linear piece and then read all of it, but then you kind of can control your own journey and then jump around if you want. So you don't have to be a completionist and you can kind of dip in and dip out when you want to. So I thought it was a really nice structure in that sense. But as you reflect on the principles of interaction, just curious as you've gone through your different pieces, maybe some big takeaways in terms of interactivity when it comes to the type of work that you've done over the years.
[00:11:29.770] Francesca Panetta: I mean, I think that each new technology has got different affordances. So trying to compare what type of interactivity works well in VR compared to an interactive web feature or an Alexa is completely different. I feel like the story and developing the interactive design need to be done in parallel, and they kind of talk to each other. So as you're building the pieces, you're permanently experimenting and seeing what works and the language of that technology, the language of the interactivity. So I use this as an example, because I remember this being the first time this was extremely clear to me. So I was building a number of location-based audio pieces. So these are pieces where you were in a specific place, listening to stories in a non-linear way. You can explore a part of a city or a park and hear stories. And I'd come from radio production, and I thought, well, this is fine. I know how to do this. And you put the content. on location and found that none of that content worked at all well because in radio you describe everything and here you are on site in fact you're more like a tour guide and it's a very very different language And at the same time, you're inviting the audiences to be exploring that space, looking around, going and finding things in that space. And so over the course of maybe six months of making the first piece, we developed a whole new language of scripting, but also how what worked in terms of building the rules, like the code of interaction, you know, how pieces stopped and started and faded out, how many pieces you could put on one location, but also what the language was, you know, lots of second-person storytelling, very short clips, very direct language, as if you were a tour guide. And it made me realize that even though this supposedly was the same medium, audio, it had a completely different language which played to that particular platform. And every time I start with a new platform, yes, of course, you bring your storytelling skills from before, but we need to learn the affordances for that particular technology and what is going to work. and how you can develop your practices that you have been using over the years to fit and to work with it. It's like it's got its own character and you need to figure out what that is and what's gonna work with it. And it was the same with VR, trying to pack too much into it, trying to have like dense bits of audio over the top when you're in this place, trying to explore didn't work at all. So that was about trying to figure out what does embodied storytelling look like? What are the interactions that are meaningful? And you're not just kind of putting in because, you know, you've been told at some conference that interactivity is key in VR and you're kind of putting in things for the sake of it, which I've seen also, you know, done and seen a lot of. So what is the language that works both in terms of like the content, but also in terms of the interaction? And I guess when you say what have I learned over the years, it's probably that it just takes a while to explore that and get to know it as new technologies develop.
[00:15:02.648] Kent Bye: Yeah, I remember talking to you at Sundance 2016 where you had six by nine and you mentioning that same thing around the second person language to be able to, rather than having other people talk about their experiences about solitary confinement, you were the one that was immersed in the prison and cell and you were being directed to pay attention to specific things. So that same type of guided tour. aspect, but to really draw out specific details of that room to get people more immersed rather than to listen to someone else's direct experience that may be the result of having people feel less immersed in their own first person perspective of what they're experiencing. So, yeah, that seemed to be a big inclusion as well as you're talking about that. And the next piece that I see from DocLab 2016 was Underworld, a virtual experience of the London sewers. Maybe you could talk a little bit about that project.
[00:15:52.125] Francesca Panetta: Yeah, this is a piece we made for Daydream when Daydream were launching and we just started this collaboration with Google. And we thought it would be interesting to do a piece about urban exploration. So there are these urban explorers who do these kind of incredible physical feats like climbing to the tops of skyscrapers, but also exploring places that are forbidden. And there is, you know, both the excitement of urban exploration, but also a different relationship with how you connect with cities in particular, and how that can relate to the kind of layers of history as well. So this was a piece about a really important part of history of London, which was in the late 19th century, when there was a huge amount of development, both above and under the ground in London. And so, as you are being the urban explorer in the sewers, you're also learning about this period of great development. So it's kind of part game, trying to escape out of the sewers, but also learning about the world of urban exploration and of the history of London at the same time.
[00:17:03.780] Kent Bye: Nice. And are those Daydream experiences still available or are they, if people wanted to see some of those different VR experiences from the Guardian, is it dependent on the Daydream or are they available for folks to be able to check out?
[00:17:14.283] Francesca Panetta: Some of them. So if you go to the Guardian site, you can see all of the pieces there. Some of them we ported over to different platforms. Daydream's no longer available. So some of them are now 360 videos. Some of them we ported over to Oculus. So it slightly depends on the pieces.
[00:17:29.480] Kent Bye: Okay, great. Yeah, I knew you had done 6x9, but I haven't seen some of the other pieces, so I'll have to go check those out. So moving into DocLab 2017, you had a piece called Limbo. Maybe you could tell us a little bit about Limbo.
[00:17:41.155] Francesca Panetta: Yeah, Limbo was another VR piece that we made at the Guardian. This was a 360 video, and it was about asylum seekers in the UK. So the title that we gave it, Limbo, came from the interviews that we did. Almost always when I'm starting pieces, we do a lot of research at the beginning. So quite traditional journalistic research, talking to experts, talking to grassroots people. In this case, we travelled right across the UK talking to asylum seekers about their experience. And this is why we're trying to figure out what the storyline is, what the voices are going to be in the piece, what the angle is going to be. And what came up over and over again was that there is a sense of limbo for asylum seekers. And the reason for this is that they're not allowed to work. They're given a very small amount of money to exist on while they're waiting for an interview. So what happens in the UK is you wait to be given an interview to see whether you'll be given the status to exist, to stay in the UK. So you're given a very small amount of money and you basically are just waiting. waiting for that interview to happen. It's very difficult to interact with the rest of members of your community because there's so little money even for traveling or to do anything. So a lot of people described how they would sit in the rooms that they had been given while they were waiting for the interview. And it felt to them like limbo. And they talked to us about the PTSD that they could be experiencing from having fled from countries like Syria, so the kind of nightmares they'd have, the difficulty it might be to talk to their family back home. I talked to an Iranian doctor who it was too dangerous to talk to his daughter and his wife back home. So he was literally just waiting there, waiting for someone to give him a date for an interview where he would then maybe be let into the country or not. So this piece we decided we'd call Limbo, but also we'd give it the aesthetic as well. They said they didn't feel like they belonged there, didn't feel like home, and yet this now was their new home. So we worked with ScanLab, a great production company in London, who LiDAR scanned the environment. We shot this in Manchester, and we tried to give it this very kind of liminal feeling using point cloud aesthetic. And you're toured through Manchester and into the house that someone is living in. And through this, you hear the voices of the people that we interviewed and the script that we wrote through it, trying to portray this kind of experience.
[00:20:20.402] Kent Bye: OK, was that featured as part of the Museum of Other Realities retrospective piece?
[00:20:24.526] Francesca Panetta: I think so, yeah.
[00:20:26.172] Kent Bye: Yeah, I think I saw it there. It was part of one of the immersive story retrospectives there. So I was able to see that there. Very cool. And we actually had a chance to chat in 2019 at DocLab about in the event of the moon disaster, where you were at MIT doing deepfake explorations of recreating a speech that never happened of Nixon. And that I actually saw that quite a bit over the next couple of years. It's still percolating out from the first time I had seen that. But you spent a number of years at MIT. Maybe you can just kind of reflect on that project, but also some of the other things that you were doing at MIT and artificial intelligence and the intersections between AI and storytelling. What type of specific things you were looking there at that intersection?
[00:21:07.241] Francesca Panetta: Yeah, so this project came out of a number of brainstorms that I was having while I was at Harvard first and then at MIT, and a kind of concern around misinformation and deep fakes. So I'd say a lot of the story ideas that I get probably from the topic matter first, rather than specifically thinking about AI, it was more a concern around misinformation. So the project developed out of one of these kind of regular brainstorms that we were having at MIT and at Harvard, where I was at the time. And it was the 50th anniversary of the moon landing. And we knew of this kind of beautiful speech. And so the speech was if the moon landing had not been successful and Nixon, who was the president at the time, would have delivered this on TV. So yeah, we talked to Casper, the curator of DocLab about this project and we did a bit of fundraising and then I moved to MIT and very quickly needed to put this together for DocLab. I think I'd been at MIT for like four months and we'd got a bit of funding from Mozilla and it was a tight turnaround. I am not a machine learning expert. So we were really in the hands of the two companies that we collaborated with. So Kanye AI and ReSpeech. I would say that, you know, from an artist's point of view, my co-director and I, Holzi Burgund, and I were not sure how realistic this was going to turn out. You know, there is some media where I'm pretty sure what the results are going to be. I would say this was not the case here. We're like, is Nixon's voice going to sound anything like Nixon's? You know, they're going to be like weird lines around the edge where they've put the dialogue replacement part in. But they did an amazing job and it came together in time for DocLab. And then Caspar and his team built this beautiful set because we were, Halsey and I were in Boston. So really all the hard work of sourcing the furniture and finding nice wallpaper and all of the kind of space books they found. I mean, we had regular calls with them looking at different pictures of couches and rugs and things. But they really did a wonderful job of scurrying around Amsterdam, finding all these bits and pieces. And it looked beautiful when we turned up.
[00:23:27.039] Kent Bye: Yeah. And the TV too, it made a big difference because it was of the era as well. So you're watching this piece on a TV from the sixties. So that I think made a big difference as well.
[00:23:36.042] Francesca Panetta: It really does. Yeah.
[00:23:38.523] Kent Bye: Well, I'm curious what you think some of the ultimate potential of all these immersive technologies and immersive storytelling might be and what they might be able to enable.
[00:23:50.180] Francesca Panetta: I think what it does is finds new ways to intrigue and engage with audiences. So we know that audiences have got quite low attention spans at the moment. So I think that things that feel novel and intrigue and draw you in is appealing for audiences. I also think that this space has quite a lot of latitude for storytelling. you know, the potential for making a kind of hybrid piece like we did for Invent a Moon Disaster, that it gives a space for that kind of potentially more creative and artistic space for storytelling, as well as the technology and the types of interaction. It's got a kind of generosity in terms of what you then do with the content. And I think that that means that you can make pieces that can be very emotive, very provocative, very intriguing in a way that maybe more traditional spaces can't do. So I think that, you know, my aim is always to try and engage audiences in really visceral, strong ways that makes them care, that moves them, that makes them care about the subject matters that I really care about and think are really important to society at the moment. So it's about trying to use whatever techniques we can to do that. And I think that this space has a lot of freedom in it. And it means that you get some kind of wacky pieces and wacky experiments. And I've certainly made lots of those myself. which might not really beautifully engage people, but that's like part of the process. You know, it's only through really the kind of trying of the new things that you're going to get to new forms of storytelling and audience engagement. So, you know, that's why I love this kind of forum is that it provides a platform for that kind of work and experimentation. It gives it a validity. It gives it an audience, which, you know, sometimes when you're poking around on your own and trying things that you don't really know who that audience is and you'd have a community to support you in doing that. So these kinds of forums are really useful for that.
[00:25:55.424] Kent Bye: Hmm. Yeah. Is there anything else that's left and said that you'd like to say to the broader immersive community or the doc lab community?
[00:26:04.962] Francesca Panetta: Well, I would thank DocLab for supporting projects like this over the years and providing this space and encourage more of the community, either the audiences to be coming to these festivals and particularly DocLab and making stuff. It's just persevere. This is a tough time for audiences and for makers when we haven't had these physical spaces where we don't know necessarily what audiences want and what those platforms are going to be. My feeling is at the moment, this is a time of kind of supporting each other and persevering. And maybe I'm speaking for myself, but I feel like a lot of the community, these have been hard years and it continues to be so. So that's what I would like to encourage.
[00:26:52.175] Kent Bye: Well, Fran, thank you so much for joining us today. I enjoyed your work over the years and also just the deep insights you have from doing this deep dive of storytelling and really focusing on the stories and seeing what the affordances of all these new media might be through all these different pieces that you've been doing over the years. So, yeah, thanks for taking the time to be able to talk about your own journey and some of your own work and also your experiences at Doclog over the years. So thanks for joining us today.
[00:27:17.006] Francesca Panetta: Pleasure. Thanks for having me.
[00:27:18.769] Kent Bye: So that was Francesca Panetta, a freelance immersive artist, director, and journalist. She spent a long time at The Guardian, experimenting with new forms of storytelling, then as a creative director at MIT, experimenting with new forms of storytelling using AI, and now as an independent artist. This conversation was recorded on Tuesday, December 14th, 2021, as a part of the collaboration with IFAS DAAC Lab in order to celebrate their 15th year anniversary. If you'd like to support the Voices of VR podcast, then please do consider becoming a member at patreon.com slash voicesofvr. Thanks for listening.
