#1123: Interactive Animation of Polarized City “From the Main Square” Wins 2nd Prize at Venice Immersive 2022

From the Main Square tells the story of a polarized city by placing you in the middle of a hand-drawn and animated city as it goes through different waves of conflict and evolution. There are many individual characters who you can trace throughout the course of the experience, but it’s easier to focus on the an overall gestalt of broad shifts and changes over time.

This is the master thesis animation project Pedro Harres, who went back to get a philosophy degree so he could better structure arguments, embed thought experiments, and socio-political provocations, reflections, and commentary into his work. And he’s encoded many references to Brazilian culture, architecture, and political dynamics that have been unfolding there over many years.

While there are some lightweight interactions of zooming into characters and triggering actions, there are many aspects of the experience that are being unconsciously triggered by the audience based upon which quadrant of the experience they’re looking at. This enables a type of fluidity and flow in the experience where there’s also some dynamic action happening at just the right time without you even necessarily noticing. It provides the experience of a natural unfolding of a passive experience, but the very act of your witnessing is unwittingly triggering a linear array of events.

The From the Main Square took home the 2nd place Grand Jury Prize at Venice Immersive, and I had a chance to sit down with creator Pedro Harres to unpack his journey of creating this piece of prescient and timely art that’s tapping into some deeper themes of polarized conflict happening in his home country, but also around the world.

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. It's a podcast that's looking at the structures and forms of immersive storytelling and the future of spatial computing. You can support the podcast at patreon.com slash voices of VR. Today's episode is the second place prize from Venice Immersive. It's the Grand Jury Prize. Normally, the Grand Jury Prize is the top prize at Venice Immersive. It's the second place. From the Main Square is a really amazing animation piece by Pedro Ortiz. So this is This is a piece that is using 2D animation, so drawing animation, but it's built within Unity and it kind of reads as a passive experience, but it's really a lot more interactive than you may realize, even after you've experienced it. I didn't actually realize how interactive it actually was until I actually had this conversation with Pedro to realize that a lot of things wouldn't even trigger unless you're actually moving around and looking at everything. So From the Main Square is a story that's looking at the unfolding of a city over time. And it's actually going to be available for Raindance starting on October 26. So if you have a chance, definitely try to check it out. I believe it's going to be available on Viveport for that month. But if you have a chance to actually watch this piece, I highly, highly recommend watching it before listening too much about the content. If you've already seen it, feel free to listen to it. Or if you really want to listen to this process, it's a really amazing story because Pedro is looking at the different degrees of polarization that's happening, specifically in Brazil, but it's kind of reflective of things that are happening all over the world in terms of polarization. So he's able to capture these interactions at a one-to-one scale, but also of a city. So it's all these individual actors that you can watch and follow throughout the course of the piece, but also this larger political and social dynamic story that's being told through the medium of VR. Talking to Pedro was fascinating because he's got a background both in the animation, but also he went back and got a degree in philosophy, so really thinking about this deep philosophical thought experiment and using the medium to push forward this deeper thinking and reflecting about the nature of what's happening in our world today. I'm just really, really impressed with this piece. And I found it both as a provocative, interactive animation, but the thoughts that came not only from this conversation, but I keep coming back to it as a piece that reminds me of the power of immersive storytelling as well. So thanks for coming on today's episode of the Voices in VR podcast. So this interview with Pedro happened on Sunday, September 4th, 2022 at the Venice Film Festival in Lido, Italy. So with that, let's go ahead and dive right in.

[00:02:44.528] Pedro Harres: My name is Pedro Arres, I'm a Brazilian animation director and I just have done my first VR piece from the main square is the name of it. It's a German production and it's premiering here in Venice this year so I'm really happy about it. I come from this animation background and for me it was like such a big jump to go from a film pipeline so to say and to enter the world of like game engines and interactive content. Actually like there was like this learning curve that was quite steep so to say but at the same time there was like a sense that I was winning freedom because I always like to do films, but the idea that you can do content that is just reactive, it kind of blows your mind on the storytelling possibilities of it. And yeah, that's it.

[00:03:37.921] Kent Bye: Yeah, and maybe you could give a bit more context as to your background and your journey into working with VR.

[00:03:43.360] Pedro Harres: Okay, so my background is that I did a bachelor on film in Brazil and after that I did a bachelor on philosophy and then I worked in animation for several years in Brazil and then there was a moment that I decided to keep on studying and I decided for doing a master on Babelsberg Film University. It's in Potsdam, it's a city just besides Berlin where you have like a whole ecosystem for film production. and the name of this Master is Directing Animation, Animations Regie, and in this Master you have the chance to produce a short film, that's what you should deliver by the end of like the fourth semester, and I stayed for eight semesters and I did a VR piece instead, that is actually like between 17 to 22, 25 minutes, so it's kind of like almost like more than a short film, Animated shorts they are normally around 7 to 8, 11 and so I'm almost like crossing this boundary and for a while they were quite unsure about it but at the same time they see that I was working so much on it and then came the COVID and I got like an extra time I got two semester extra to do it And I'm really thankful for that. At least in my case, it was helpful for the production of this project. And then here I am. But before this project, I always had an interest for video games. When I was a teenager, I was playing Sega Genesis and Super Nintendo a lot. And I always wanted to know how those things work from the inside. And I think that after learning Unity, I finally know. But yeah, I had to learn how to program, I had to learn how to do level design. And the thing is that I had to adapt all my mindset from film. For example, in the beginning I thought that, okay, we need a scenographer for the project, and then we need somebody for, like, editing or montage or whatever it's gonna be and at the end like you should actually fuse these two ideas into level design which is something that comes from the game world so it was like a lot of the process was me and also the crew that are mostly like film students realizing that what we're doing is not actually a film, what we're going to deliver is actually not a file, but a piece of software that can produce content according to a certain input. So what should we change in our mindsets and in our methodology to reach that?

[00:06:08.723] Kent Bye: Yeah, that's really, really interesting. And so, clarify a couple of things. First of all, are you saying that From the Main Square was like your thesis project?

[00:06:15.931] Pedro Harres: Yes, exactly. This is my master thesis project there. Actually, the project started with a short film idea, but then I worked with this illustrator, Dani Eiserich. He's also from Brazil. And he's like such an amazing, he's a graphic genius. But the thing about it is that he's truly rebellious. So sometimes I was like coming to him like with an idea of like, okay, you should lay out, you should create this and that. And he would come with something like completely different. But at the same time, it was so graphically interesting, and I had a feeling that, okay, maybe, like, in a very instinctive way, like, he's feeling the way that things should go, that we went for VR, because what happened is that, at the beginning, the idea of the project, that it was not a city around you, because as a concept, you are inside of the main square of a city that's being built around you, and then you experience the rise and the downfall of this civilization within 19 to 22 minutes. but at the beginning it was like a single building with like two groups like fighting for its structural and facade aspect so to say until it collapsed but then there was one day that I asked him for a concept because we're applying for some funds and he just drew a city instead of a building I say okay Daniel like this looks really great but it should be a single building and he said like Oh man, I'm sorry, like I just had a feeling that it should be like this, like okay, but can't you do another one that said we don't have the time, like the deadline is like in four hours, can't you maybe change the script a little bit? And the script at that time was three pages and I remember that I thought like, oh, Why? Why? Okay, I'll do it, I'll do it. And then I changed the script and then we didn't get this funding because it was like a very hastily changed, like that was done really like on the fly, so like there was like no literary quality for it to make it. but then this project went to the drawer for a long time until I was in the Babelsberg Film University which is a place that I really recommend as an animator if you want to get projects funded because it's such an amazing financial bubble to be honest and they really support you with like high quality equipment and there are like internal applications that have like really low levels of competition in comparison to any application that I've done in Brazil before, like really low in comparison. And then when I was there, I realized that I needed some funding to put my master thesis project together, and I saw that there was a lot of funding for VR and innovative content, and not so many people were applying. And I said, hmm, why not? I have this concept, it would be nice to be in VR, and then why not? And that's like how it started. And then there was like a workshop from the teacher Bjorn Stockleben. He's also my tutor in the project and he was really like the big helping hand inside the institution to make it possible. That's called OmniLab and that's a place that I first started to learn Unity and we did like the first prototype, which is quite funny if you look at it today and compare it with the latest one. and then as I was there first I thought okay let's do it 360 degree but then I start to watch a lot of 360 degree films and I didn't like something about it is that okay you have like this focus this area that you should look at and if you look all the rest it's like a dead zone and so like it say that how you are immersed in the world okay you are but you have no freedom you have just to follow this certain choreography body choreography if you want to get the content if not well that's your loss because we're not gonna make content for all the sides because you know it's too expensive and blah blah whatnot And then I saw many films, maybe some early 360 degree films, maybe it has evolved a bit, and I was unsatisfied by it. Then I started to watch, instead of animation, I started to watch some documentaries. And then I see that the camera, like there was a reportage, a 360 degree reporting piece. about the protests in São Paulo in 2013 against Dilma Rousseff and the camera was just in the middle of the whole demonstration and you could see things happening on the side and the police was coming, people were running and it was such an amazing, exciting madness to experience that that I said, okay, that's the way to do it but how do I do this in an animated way that is like so much more expensive and then I thought Okay, what if we do in a way that actually things are not happening around, but things actually only happen when we look at them. And then at the moment we look at them, we trigger it. And that's the moment that it has become interactive. And that is the moment that we shift to a game engine. And that's where the learning curve went a lot higher.

[00:11:07.693] Kent Bye: Yeah, it's an amazing piece, and I want to dig into it. But before I do, I want to ask one other clarifying question. You said that you had also a degree in philosophy. I'm wondering if you can expand upon what specific branch of philosophy you were studying. And I'm just curious to get a little more context of that.

[00:11:22.391] Pedro Harres: Okay, yes, yes. I studied philosophy and the reason that I went for studying philosophy is that the moment that I finished my bachelor on film, people said to me, okay, now you can go to make films or you can do a master and specialize yourself in something. And I had the feeling that directing films was such a broad activity that you have like to think about so many, you have to put so many things in balance in your mind when you're making your choices. because you have like the budget, the time, the story, what you imagine of the story, what have you shot from it and what is like in your imagination you should replace for the pieces that you've already made and all this like you're shooting out of order and I thought okay for being a director I really need to learn how to think in a more organized and systematic way. And besides that, when I was a film student, I was really into films that have a philosophical side, like Richard Linklater's Waking Life, Ingmar Bergman's Godard, I also like a lot. Lars von Trier is another guy, and even Akira has a lot of philosophy into it. And I thought, okay, really like the type of fiction that drives me is fiction that has like a philosophical view behind it, even if it's like expressed directly through speech or dialogue, or it's just present in the concept. So I say, okay, I cannot escape philosophy, so let's study it. And at this moment, I went to a university in Brazil called the URGS, the Federal University of Rio Grande do Sul. And then I learned a lot about Aristotle, Plato. They're a bit conservative. They go more towards the analytical side. I think that the most useful things that I learned there was Wittgenstein, and I think that the most significant book that I read was The Origin of the Work of Art by Martin Heidegger, because this was something that really made me understand art in a way that was useful to what I was pursuing. And my dissertation was about ethics in Dogville from Lars von Trier. And in philosophy, in contemporary philosophy, you have a brand that's called Philosophy of Film, that was initiated by Stanley Cavell but the guy that I like the most is called Stephen Moore and he made philosophical reading of the Alien series and that was kind of like the book that I use as a model for making my analysis from last one three year and I was using this and the second Wittgenstein the philosophical investigations like when he's really into philosophy of language And yeah, that was it, philosophically.

[00:14:01.863] Kent Bye: Interesting, yeah. As I've been doing the Voices of VR podcast, I've, over time, gotten deeper and deeper into different aspects of philosophy. Because I think VR as a medium challenges our concepts of reality and experience and consciousness. And so, for me, I've leaned towards more of a process-relational approach with Whitehead and Deleuze and other process-relational thinking in that sense. I ask because, you know, this piece seems to be having a dialectical process between these two polar opposite battles that are happening across the course of the film. And I'm curious if there's any philosophical reference or if there's any touch points in terms of what you were drawing from as you were doing this piece.

[00:14:37.908] Pedro Harres: Okay. Yes, there are. There are philosophical, sociological and political basis to it. On a philosophical level, well, actually like there's not an author that represents this. It's just something that I crafted on my own in my view that I really think that humanity in general is like it's a very complicated and a hateful entity, but individuals on the other hand and specifically they can be marvelous. And I think that in this VR piece, I wanted to explore the former side of this assumption. And the story really deals about how humanity tends to conflict. And on the sociological level, the story was really influenced by the rise of the right-wing and alt-right extremism in Brazil, and how this shifts and polarizes society in a very scary way, actually. and I could witness this and it was very funny because in the first version that we had it and it was just one construction the story was really abstract and then when it turned into a CD and the right-wing extremism started in Brazil like it started to become more and more a documentary. And then there's the moment that they went into VR, so it was really a big conceptual shift in the project. And then regarding this sociological level, in Brazil we have like two teams, so to say, that stand on the left and the right side of the political spectrum, that we call them the coxinhas and the mortadelas. and we made kind of like a version of this that we call them the rounds and the squares and if you see the project you're going to see that there's one side that they have like buildings that are more like made of concrete and glass and they're more kind of like square and have more angular designs and it's kind of like a big city so to say And those people, the squares, they tend to be really working-oriented people. They like, I don't know, money. They also tend to prefer authority. They have a taste for authoritarianism, so to say, at least in our piece. They prefer order towards freedom and fruition, so to say. and they're more like oriented and the other side to have what we call the rounds and the rounds they live in those houses that they're kind of like made out of clay and they're like handmade and they're more like organic design and there are people that they have like a certain connection with nature sometimes they can become fanatics as well and they can also like engage in dangerous stuff like there are no saints at all Yeah, but they represent like this other tendency and the story is about how those two tendencies they start to mingle at the beginning in a harmonious way and as like they fight for space and they have like clashing road views how this actually evolves into a civil war conflict. Sorry for the spoiler.

[00:17:33.398] Kent Bye: Yeah, I think the experience as it unfolds, it does feel like it is operating at all these, you know, using the environment to tell a story. So it's like a level of environmental storytelling, but at the scope and scale of the city, but with individual characters that are populating it. So it is this multi-layered story that has the philosophical aspects, the sociological aspects, the political aspects. There was one other political thing I wanted to ask about and clarify because it seemed to be that there was an indigenous population and so I don't know if there was any indigenous philosophy or decolonization thought that was also going into this piece.

[00:18:04.935] Pedro Harres: So that's the thing that like in the beginning in the script we had just the rounds and the squares but the thing is that like as it is a city and there's a space to add like a lot of little things inside Daniel as he was doing the layouts for animation sometimes he felt like drawing something different and we would just like add things to the story that were not in the script And Daniel has a long story of working with different Brazilian nations, among them the Guarani, mainly the Guarani. And then he said, OK, let's put them inside as well. And we saw that although they were kind of closer to the round, thing they were actually like something else and they are inside of it actually like most of the scenes that you see inside of it or at least a good part of them they are based on real things that have happened and many of the buildings that you see there they're actually observational drawings from things that actually exist And I think we didn't think about colonization or decolonization. At least I didn't think about it. But that's the thing that sometimes when you're making a work of art, you cannot really grasp all the aspects and the possible meanings of it. You just use some of them as a conceptual anchor to push your narrative forward and make some decisions. And then it's also like an open metaphor and people can draw it in different directions. Yeah.

[00:19:31.898] Kent Bye: Okay, okay, that's helpful. It was also really helpful to hear how the evolution of how this project came about starting with one building and then spreading out into the entire city. My experience of watching the piece was that there was so much that was happening that you can watch one section of the area. It's a 360 degree, you know, you're immersed in the main square and you're looking at the city from that perspective and that there's things that are subtly and slowly changing. So you could decide to just watch one quadrant of the experience and just watch that as a 2D experience with itself. But I was looking around and so then I would...

[00:20:05.428] Pedro Harres: No, you cannot. Sorry. Oh, because it's triggered? Yeah, exactly. Because what happens is that all the story and all the scenes that you see on the CD, they work more or less as a line of dominoes that they fall after each other. And the way that the content is renewed is that you have to look around. If you don't look around, the story doesn't progress. Even when I submit the project to Venice, I said that the maximum length is like 25 minutes, but that's a leap of faith. like directly one could play it for like two hours if he doesn't turn but it will be boring as hell but one could but from my experience like nobody does that because people start looking around and around because we thought of it like the initial concept of it was from a gaming perspective was that okay it's kind of a small hunt for visual treasures you are like scrolling through this panorama of the city like turning sideways and then you bump into something that attracts your attention and then you go forward with this and then sometimes you have like audios on the other side to trick it But the thing is that the way that we did it without any cuts, it's like a single shot, so to say, is that like, for example, like if you're looking towards east, then on the west, the content is being renewed, and then the content is actually waiting for you to look at it to be animated. And the moment that you do this, then the content on the east will be gone, and then you'll have new things there. And that's the way that the story progress. So it's really like demanding you to turn around. You have to peddle the story with your interest.

[00:21:37.109] Kent Bye: OK, that's really, really fascinating. Because I was looking around, and there is a bit of a change blindness. So you see something, and then you look back, and it's different. But the way that you're able to orchestrate that when you're looking in the opposite direction, that you're using that as an opportunity to do some of those changes. So yeah, I thought that worked quite well. That's interesting to hear that it's architected like that. So, yeah, as you're building this as a piece, it sounds like that the concept art was a real big key, but you've already written some stuff, so talk about that process of going back and forth between the artists and the creation and making of the interactive aspects, and if you were kind of building up to a point and then seeing what it was, and then you were going from that point, or if you felt like you kind of had the full arc of how things were going to unfold before you started to put it together.

[00:22:19.086] Pedro Harres: Well, that was maybe one of the biggest challenges of the project is that like I have a lot of more than 10 years of experience on animation pipelines for films, short films and animated features, especially in 2D. So like I was really comfortable on that side, but I never done a software before. And at the beginning, me and the rest of the crew was kind of like in a certain denial that actually we're not doing a film, we're doing something else. We thought, OK, it's just a film that kind of reacts to people, but it's not really a game. So let's think of it as a film, and what happened is that when we hired the animator Samuel Pathy for the project, for example, he's a great 2D Swiss animator, he won several awards. I think he won the Crystal of Annecy with his last short film. and the moment that I saw his work I thought okay this is a guy that should animate the artwork of Daniel and this is what is going like to deliver it and at the moment that I started talking to him and said okay so it's a VR thing but it's more or less like a film so I booked this amount of time to do this and we're gonna be fine. You just have to deliver me the layouts in a very 4D way, because 2D animation is a very 4D process. You have the storyboard of all the shots, and then you have the layouts, and then you have the animations, and then after that you have composition, and blah, boom, ba-boom, it's ready. And then say, okay, okay, we can do it like this. I'm gonna deliver you one layout every three days, and then we go like this for two months, and then it will be gone. All right. We didn't realize that we have to test it so much. We didn't realize that actually we were like... unsure if it was working or not we were like flying in the dark for a long time because he was delivering one animation every three days and i was always busy designing the layout for the next one and i didn't have the time to put it inside unity to see how it was working or not and i never made even a 360 degree piece before so like for me it was like okay okay this is getting out of control and also that's one of the reasons why it has become so big is that it has so much animation inside is that I was just okay he's doing and a lot of animation for me and I don't know if I have enough or not I don't know if I'm working or not but I know that like just to be safe if I design the layouts of the animations in a way that I can like crush them down into little things and reuse this here and there I can have like even more content and then it's like if I have more content then I'm covered and then like it's gonna work at the end And that's why we have so much animation inside But there was a moment when he did like I don't know like 85% of the animation of the project I said like okay, man You need to stop now like don't do the last 15% because I really need Like one year to like put it all together and then I'll come back to you I don't know if you have everything already or not Maybe we don't need nothing else, maybe we need more, but I really don't know and I need some time to do this. And there was another German animator, Sophia Schönborn, that also did a great work, and she stopped around 75%. No, he did 90%. I had two more animations from him to do in the lesson, from her I had, I think, five. and the ones that I left to the end was regarding the war because the war was like really difficult to lay out because like all the other animations that you see there like all the other little situations or scenes so to say they happen just on one side so you can more or less draw them in a way that like it's a very flat screen and then you just put the character as sprites in the 3d world and give it a little bit of perspective and stuff but the war was just beyond that because we wanted like something like a Teo Angelopoulos shot with these complex choreographies, but actually being truly brutal. And I was like, OK, we don't know even how to think about how to lay out this. So maybe this is the thing that we are living for later. And this was the very thing that I was working on, designing on my last whole month before the submission to Venice. It was a rough month.

[00:26:23.783] Kent Bye: Yeah, OK, so it sounds like that you had the overall themes and arc of the story, but as you were building out and doing these animations, you needed the time to see what the interactive parts were up to a certain point. Then you could start to then figure out how to really wrap it up and do these vast scenes at the end. Yeah, I guess one question around the process of the animations. We're here in Venice, and yesterday there was a Meet the Creators panel discussion that laid out on the table lots of the hand-drawn drawings. And so what was the process of taking these individual drawings? Were they drawn by hand? Did you scan them? How did you go from the artists and the initial drawings and then have them into the animation pipeline?

[00:27:03.023] Pedro Harres: Oh, well, all the animated figures are drawn digitally and all the backgrounds are drawn on paper. And then there was like a moment, like we have actually two art directors in the project. We have Daniel Ezerik and we have Paulo Lange. Because Daniel is really good at doing observational drawings. But the thing is that, as he's a truly rebellious guy, that he does things always in the flow, there was a moment that I said to him, OK, now I need a building that is going to be like this, that I need a window here, I need a door here, and I need this here, and he just doesn't think this way. Because he's going to look at a certain building that has the feeling, and he's going to do it and say, OK, now you changed the animation, you do something like this. And this was difficult, and there was a moment that I just realized, Okay, if I keep on doing this with Daniel, maybe it will never be over. And that's the moment that Paulo Lange entered the project. He's also like a Brazilian illustrator that lives in the same town as Daniel. And he's more like a digital guy that he could receive like really like technically complex layouts. and follow them. And then for the drawings in the background, it was a little bit of like a free experience from Daniel. I just said to him, OK, man, I think that for this side, I think some reference could be some buildings, Anatolia, Cappadocia. But you can put also some bio constructions from the south of Brazil that, you know, and you can also add on this corner, maybe some indigenous hearts from your Guarani friends. And here in the middle, I don't know, you can invent something. I don't know. Do your thing. And then he was like putting it together, you know, it was pretty much like this and then for the other side I think he based it a lot on Sao Paulo like because he was in Sao Paulo for a while and Sao Paulo has like this massive concrete atmosphere but at the same time like it doesn't feel like completely ordered it has like a certain layer of cause to it that it takes a while for you to understand it's one of a kind of elegancy so to say. I learned how to love it as Caetano Veloso. But I know why people look at São Paulo and they get scared sometimes, because it's a really massive city. And there are some buildings from Porto Alegre as well, that's my hometown, and Daniel and Paulo's hometown. And the process was pretty much like that, we defined on the script which were the main ones that we needed for the animation, and these ones had layouts that were a little bit more complex, and those were done by Paulo. And then the other ones on the background that were more kind of like a free thing, they were done by Daniel. But Daniel also did the character design for the figures.

[00:29:37.015] Kent Bye: OK, and it sounds like that the world that you've created actually is embedding a lot of touch points to Brazilian artifacts and architecture. And if a Brazilian was to watch it, do you think that they would get additional meaning from how those pieces of architecture or symbols may be relating to a larger Brazilian culture? I think there's the elements of the story that are clear, but I'm wondering if there's things that are specific to the Brazilian culture that I might not be aware of, but someone from Brazil might pick up on.

[00:30:05.658] Pedro Harres: Oh yeah, for sure. For sure. Yes, yes. I think that most of the references that really strike on the eye, they're actually on the narrative side of, like, many of the situations that you see. For example, like, there's a situation... Okay, how much can I give spoilers here?

[00:30:19.895] Kent Bye: It's up to... I mean, I think I generally give a bit of a warning for people, so it's up to your discretion.

[00:30:26.513] Pedro Harres: Yeah, so like there's like especially some violent scenes inside of it. There are actually like cases that happened in Brazil and they were like documented of like murders and this kind of thing. I'm not going to enter too much in the details because I don't want to give so much spoilers. But even people that are not Brazilians, but they're actually familiar to Brazilian culture, like I have a friend that he's a Chinese guy that I met in Germany and he just saw the concept, like the very concept when the project went from being a a single building to be in a city, and he looked at it and said, oh, this is totally Sao Paulo, I can see it. It was really recognizable to him. And to Brazilian people, I saw that the project has a very powerful meaning. Because the thing is that the story, it was really weird. We wrote the script about this society that was getting more and more polarized and entered civil war. And as we were producing, we could see that the Brazilian government and the situation of the society was actually matching with the script. The script was written and then some months later, okay, now Brazil is like... on the act two of the story and people can feel that like and then at the end okay like I really hope that we don't get to the third act to the climax of it when there is like the real conflict and if there's a place and there's a moment for this conflict to happen is actually the 7th of September now between this and January because that's the moment that Bolsonaro is probably going to lose the next election And like Trump, he's not willing to accept a result that is not in his favor. But to make things worse, he has a lot of military people in his government and many of them are like supporting him. And on this 7th of September, which is a national holiday, he's going to steal that day to make a big protest against the fact that the elections are not going to be fair and that the juridical structure of Brazil is actually pursuing him and blah blah blah but actually like he is the aggressor and this is really strange to live and see like the moment that one of the persons that is really in charge he's like a political delinquent that is actually like damaging the structures of society to see like how far he can go and it's pretty clear to everybody that he could become a dictator that he would like to and everything is about to happen. What day is today?

[00:33:02.638] Kent Bye: I just looked, it's September 4th, so it's like in three days from now.

[00:33:05.000] Pedro Harres: Three days is going to be a huge, a massive demonstration in Brazil about this, and then in November we'll have the result of elections, and then by January the new president, or Bolsonaro, if he wins, there's a chance that he's going to win, and then it's going to be like Orban in Hungary. that this new president is going to assume. And then there's probably going to be a moment of political instability that's going to last for some months, because I think that in such a situation, between the new government taking over and the first three months, there's a chance for them to try something and revert the situation. And I really wanted to do it as a warning for people about the dangers of what was going on and also in a certain way to document it. But I didn't expect it to be premiered in a moment that we are so close to the possibility of its climax become reality. And there was a moment that I showed it to a Brazilian friend. before submitting it to Venice, and he's also a filmmaker, and when he took the glasses off, it was his first or second VR experience, like, he's a film guy, and I could see in his eyes that he was really amazed by it, he really liked it, but he was almost crying, because, like... I'm sorry. Because he could simply feel that... I was talking about his life in it. And my life in it. And for me it was so crazy because... I was experiencing it from Germany, from a certain distance, and watching the news, and being there in the winter, in the COVID times, and doing my things in unity, and seeing how all this violence, this graphical violence present in the thing was also matching with reality and on the existential level as an artist. It was not an easy thing to deal with, but I'm grateful for doing it. I think it was a nice thing. And I don't know, sometimes I like this idea that Bukowski says about writing that, OK, all we have to do is sit in front of the typewriter and open a vein. And I think that's a way to achieve some certain level of artistic truth to it. I don't know.

[00:35:52.637] Kent Bye: Yeah, well... As you... As you elaborate all that, I just see all the parallels that are happening in the United States with Trump and the type of polarization. In all likelihood, Trump was also talking about how the elections were likely rigged, planting the seeds that he knew he might have an opportunity to lose, and if he did, putting in the game plan. There's the whole January 6th committee meetings that have been going on and trying to get people that are testifying, but yet at the same time, there's been all polarization within the United States political establishment where the Republicans are essentially blocking any sort of real interrogation to this insurrection that happened on January 6th. It feels like, as I listen to you talk about the same kind of timeline of this, taking this September 7th holiday in Brazil and take that as an opportunity to protest, but also for the current leader to continue to plant the seeds for that everything's rigged. And, you know, if he has more connections to the military to have some sort of coup that happens. And so in Hungary, it's already kind of played out in that sense. And so this rise of authoritarianism on one hand, but also the polarization that happens more on a human level. That you're showing it as a collective but also a individual level. So yeah in philosophy There's the concept of mere ology, which is the whole and part so things something that's an entity But it's also part of something larger and Arthur Koestler calls it the whole on so it's both a whole and part and so there's like a fractally nested Organisms that you're looking at individual entities, but they're part of a larger whole and I think as a medium of VR You're able to start to connect those individuals into those larger pictures and especially in this piece of how the individuals are fitting into this larger story and that for me this underlying story of polarization is so timely now not only in Brazil, but in the United States and around the world and So yeah, you're really tapping in from an artistic perspective working on this for many many years You know and it's premiering now kind of you're tuned in in your own artistic process to come up to this moment in time Under which this is an issue is top of mind for many people around the world

[00:38:00.143] Pedro Harres: Yeah, but sometimes I have the feeling that artists are really like radars of what is going on. Sometimes they don't really know why they're expressing something, but it actually resonates within the collective somehow. But in terms of road building for this project, something that I would like to mention. I'm sorry if I make a change on the subject. When we were structuring the script, we actually had several timelines from recurring characters that they actually interact with each other and they have things that happen to them. And in this version for Venice, because it's a film festival and you have only 20 minutes to it, like if you don't look at more than once like i think it will be like very difficult for you to grasp all the timelines but we are considering like making a version when we have it on steam vr or something that we would be able like to select a single character and he would be like maybe like Detached from the rest in terms of color or something so that you could recognize and follow the path of a single individual through this collective story and So you actually are having a lots of individual characters that are playing out throughout the entire piece then Yeah, yeah, exactly, exactly. And we had like different voices for each one of them. We have like the punk guy that you see in the beginning and he's talking to some guys and then like he's eating some barbecue and then after that there's a moment that like he's kind of like strangely involved in a fire that's going on and after that he's at some protest and then he's fighting the police and then at the end Well, I'm not going to tell if he survives or not. We actually have like a big Excel sheet of all the characters and we see who made it to the end and who dies. And it was really crazy to make this. There's one thing that's really curious about this project that one day I would like to share the link of my Google Docs Excel sheet. of like all that we did to structure the story because it's so much crazy and it was something like to be done and sometimes not only for me but for other people so we added like actually figures to this excel sheet so you can see the figures and stuff and some of the characters of some situations they are based on real people as I said before like For example, there's a character that is actually filming some people as they fight, and these people fighting, they actually attack the guy filming it. And this was based on a friend of mine, he's a cinematographer in Porto Alegre, and he was filming some protests, and he was attacked. Actually, in reality, he was just attacked by the police. The other people fighting on the other side didn't attack him. And it's really crazy, because he's being beaten, but he doesn't turn off the camera. and you can see the camera shaking as it's being beaten. This was one of the most shocking images that I saw that happened before. It was the beginning of the rise of those protests against Dilma Rousseff at the beginning. And I wanted this to be inside, so this character is pretty inspired in Antonio Ternura is the name of the guy. and he's really like a lovely person and also like there are some indigenous characters that they are like inspired on people that Danielle have met and that's the way it goes like we have some figures inside that are like inspired on people but another thing that was interesting was that we needed to craft the languages for these two groups because we wanted them to be distinct also in terms of language and we didn't want like real dialogue, something to be spoken, but we wanted a feeling, we wanted a pseudo-language, pretty much like, I think it's The Conquest of Fire from Jean-Jacques Annaud, this film of the primitive people that are fighting for fire, and you have the Neanderthals. and other groups and they had like Anthony Burgess the author of Clockwork Orange working on their language and I had just a student of linguistics with me from the Potsdam University that she was like working with me on making those languages. Actually what we did is that We had like a different language for these two groups, the rounds and the squares. And then for the squares that are the people that kind of like live in the city and they like the dictator that's on the rise, we made a mix of Greek, Bulgarian, Swedish. Actually what we did is that like I looked at the scene and I thought, okay, what are you going to say in the scene? And I just thought, wrote it in Portuguese or English. and then I translate it through google translate to these three languages and then we copy paste it into word and like mix the things and change some syllables here and there and and then that's the way that we got into things like and many things like this and then for the round ones we wanted like a feeling more of a like a smooth language that will be more melodic with more vowels and then we use it like Mongolian, Somali, Samoan and a language called Igbo. What's some of that language sound like? Yeah, I can start to see the Wittgenstein influence there Yeah, it goes inside that there was like a actually let the recording of the voices was like really intense because what happened is that We had only like four people to do the voices of all the characters and they say like okay you are going to do the lady scientist and then you're gonna do this other character and then you're going to the the punk and the hooligan and the banker because the banker doesn't interact with the punk but to have like to change the voice a little bit and we had just like single day to record all the voices and some of the people like they really put a lot of like their hearts into it because like we had scenes like people were like begging for their lives or like crying before because they shot somebody and then considering suicide and then like to bring voice talent to this level, it demands a lot of delivery from them. And I'm really grateful to Amishai Muller, Guy Scharaf, Sofia Sarmento and Bruno De Marco. They were the ones that did the voices. And it was such a pleasure to work with them. And also, besides Megan, the linguist that worked with me, Andre Correia, my assistant director. This guy was a little bit like my consciousness cricket through it like that whenever i didn't know where to go to i would like discuss with him like he's a also a bachelor student on animation and he's really like this type of like really skillful artist guy that can do anything like he can Dance he can direct people he can animate and can do so many things and he worked on the preparation of the people as I was recording it like I was recording people on one room and he was like preparing the other ones on the order and Practicing the text with them so that they can get a feeling for the language. And yeah That was really intense and it was like really like a shiny day and this also comes from animation because like on animation like we put so much time into making the animation and making things frame by frame and then there's a moment that you do the voices and then you have like this gush of creation that happens it's just a few hours and then you have like to put like a kind of like a certain soul into it or you have like depends if you are animating before or after the voices like either you're like giving the main lines for the animation to follow or you have to follow what the animation has done and it's such an intense and at the same time fun process and I like it a lot, the devices.

[00:45:28.398] Kent Bye: Yeah, I'd say I definitely want to see the version where you can follow individual characters, because it does feel like a collective story right now, because it is difficult to track individual stories. And so, yeah, it's a type of piece that I think you can revisit, because there's a lot that's going on that it's impossible to catch everything. And then there's enough that is missing that you kind of want to see the evolution of that. And yeah, so finally, what do you think the ultimate potential of virtual reality and immersive storytelling might be, and what it might be able to enable?

[00:45:58.364] Pedro Harres: Final potential of immersive reality in virtual storytelling? That's a good question. I think that we have at the moment things within XR, for example, like augmented reality and virtual reality, that they have quite different potentials. I think that the ability from VR to put you aside from this world and experience something in an immersive way that's completely different, it can be also like a very potential tool for awareness and social change. For example, like to see a documentary about, I don't know, a harsh situation that people are living after a flood in an African country, for example. It can be a lot more powerful if you're actually sitting with them there. And this kind of content, I think it has a real potential for change, for social impact in that sense, and this is helpful. But on the other side, VR has also the possibility of being mere escapism from reality, and people can just be inside of it and become VR junkies, as in this uncanny valley short film from 3D AR. I think you know it. Yeah, I think people that are listening to it also know it and this can also be dangerous. I don't know, I myself like sometimes I had like weird feelings as I was doing VR because I like it was really, I could feel that it was still hurting my eyes. as i was doing it so i felt that like the environment due to the technical gap that we still have it's still not there to give us something that is there like just as reality it is and it's comfortable as reality to be perceived but it's on the way i find it quite interesting how for example like this series carbon Altered Carbon? Yeah, Altered Carbon, exactly this one. How they see VR as a sort of prison in a place that used to torture people. And I find it really funny that, okay, when I'm gonna put you into a VR experience, now I'm gonna strap this to your head. and you cannot move your eyes away from it, it's gonna be with you. There's a certain aspect of sadism into VR, I think. But yeah, it's a tool that has potential to go in different ways. And for AR, I think that AR can also go into this direction, but I'm really afraid that if we enter on a realm of unpaid AR content and we start to have banners under our eyes as we talk to people it will be just a nightmare and this scares me and this also like brings me to like how I think that the whole management of media, social media, internet and even like the information economy should change. I'm a big follower of the ideas of Jerome Lanier especially on his book Who Owns the Future, where he advocates that we shouldn't be producing content for social media, for free, so that they can put ads for us to see, to sell things to us, and we should organize things in a different way. And yeah, and I think that like the path that we are right now on social media, the way that it is, it's like it's quite a tragedy, so to say. And I think that there's a lot of possibilities in VR and AR to extend this. But depending on how we structure it, like it can be a place for like a lot of positive change for humanity, but it can generate a lot of problems. And we probably will have both. And it's up to our level of consciousness as a collective society producing VR and consuming VR to see, okay, in which direction do we want to go with this, you know? And yeah, I think it's a political matter at the very end.

[00:49:56.563] Kent Bye: Awesome. Well, From the Main Square is a really amazing use of VR to be able to tell this kind of vast story across many different scales of the individual and the collective. And obviously, you've done a lot of deep thinking on not only your life, but also in this piece. And I appreciate you taking the time to be able to unpack both the process of you creating it, but also the deeper meaning of what you've created. So thanks so much for this conversation.

[00:50:19.468] Pedro Harres: Yeah, absolutely. It was a true pleasure. And to be honest, like there were things about it that I haven't verbalized during the whole process. And this is the moment that they're kind of being organized into a speech. And this is also like it's a moment that I'm acquiring clarity about the things that I did as this interview was happening. So it was truly a rich process for me. So thanks for that.

[00:50:47.520] Kent Bye: So that was Pedro Arges. He's the creator of From the Main Square, which was the grand jury prize, second place prize at Venice Immersive 2022. So I have a number of different takeaways about this interview is that first of all, I just was really moved by this conversation, not only by what's happening in Brazil and the connections to the polarization, but just really struck by the artistic process for how sometimes artists just follow their own artistic impulse. And as he says, there's like, different acts that you have first act second act third act and he's writing the full act of a full story many many years ago working on this as a piece and then the thing that he was writing it about in brazil is actually as a country moving into that second act and so just the way in that the artist is reflecting different aspects of what's happening in brazil and a lot of different easter eggs that i was not necessarily able to unpack just by watching it because i'm not from brazil i don't recognize all the symbols i don't recognize all the different architecture but I think it also just speaks to the symbolic language of taking different aspects of, if you're from a culture like that, then there's going to be information that's encoded into a piece like this that may not be widely as accessible. And so I think as time goes on, the VR as a medium, I think of it as this visual anthropology, but it's really like a form of symbolic spatial anthropology that's able to, for more of a imaginal space, look at these possible futures and put them out to the logical extreme and see, in this case, you know, ends into the worst aspects of polarization within our society. So yeah, and just the level of interaction, I didn't quite realize I sort of said, Hey, if this is the type of experience, you could just sit and watch. At that point, when I said that, I didn't realize the degree to which that I was triggering things unconsciously. And I think there's a level of conscious and unconscious interaction. And this is a piece that is having all these unconscious triggers that as you move around, there's always something interesting to look at. So if you're just like, really zoning out and there's nothing happening. But if you turn your head and you look around a little bit more, they'll be constantly pushing the story forward. That's why he had a little bit of a target length of, I think, around 25 minutes or so, but can sometimes go faster or sometimes go slower. If you just stare into one quadrant and there's nothing really happening, it really requires you to look around. There was also a thing that we didn't quite get into a little bit. I just wanted to mention that there was another mechanic of having like a magnifying glass to be able to look and zoom in on different characters. And also you would see a character turn yellow or orange and be a sign for you that you can actually engage and click on that as a character. And when you do that, then it unlocks whatever action that that character is going to do. So just have this extra layer of interaction. And it's a little unclear for me as to what point those trigger interactions are getting the experience. If you didn't click on anything, would it still progress? I suspected it would have or maybe not. I don't know. There's different states of things that happen or don't happen. So, again, even the degree of the interactivity is hard to really fully unpack. If you just go through it once, it really requires multiple viewings, and this is an experience that you could watch multiple times because there's just so much that's going on. I was really fascinated to hear that there's a spreadsheet to track individual characters and to just think about, OK, what would the interface be to be able to click on someone and be able to track them over time and get highlights from that specific character? Because there's many different characters that are interacting over the course of this piece. So it's really this complicated, multilinear thing that's happening. You just get immersed in the middle of the square and you're looking at it from more of the collective gestalt level of seeing the architecture changing over time. And so it's really a meditation on time and going through this long, vast period of time and to see this evolution. And yeah, really just pointing out for the most part, polarization between these two parties, but there's also this other indigenous, subtle, unconscious decolonization, I'd say, in terms of just including different aspects of the indigenous folks that are not really necessarily into the circles or squares, I think is what he was referring to the two main factions. And also just the development of entire languages of mashing together different languages and to have a different vibe of interactions and communications that are happening. So Yeah, I really enjoyed this piece and I didn't fully appreciate it to its full extent until after having this conversation and just really appreciated the depth of thought that Pedro is putting into the piece and I'm really happy that it was featured as one of the winners because I do think it's actually one of the more innovative, interactive pieces and narratives, and to also try to bridge the gap between what's happening at a small scale, but also at the large scale. And so this philosophically concept of muriology, of having the holes in the parts, of having these individual entities and characters, but that feeds into these larger aspects that you see play out at the scale of an entire city and these buildings into Yeah, it's just a fascinating use of the medium to really tell this type of story that I don't think would have worked quite as well if it was just a 2D film. Being immersed in the space really gave me this sense of being in that place that he was exploring, which is a reflection of his own home in Brazil. Yeah. And I guess there was an event that happened on September 7th. I don't think there was the degree that a lot of people were expecting of what was going to happen with the president. I think there was expectations for it to go even more. But I think there's still a lot of the dynamics that are going to be unfolding between now and whenever the election is at the top of the year there in Brazil. So keep an eye on that. And if you do have a chance to go to Raindance, it's a virtual conference. And so there should be an opportunity for you to, through Viveport, get access to this as a piece and to be able to watch it. Like I said, it's a piece that's worth watching through again and again, just because it's so rich and there's so much that's going on. So anyway, that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a listener supported podcast. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.

More from this show