ANIMALIA SUM is an animated short VR 360 video / application hybrid with some light, embodied, 3 degree-of-freedom interactions on the Oculus Go. It’s a mockumentary exploring the potential of whether or not humans should consider moving from animals to insects as their primary source of protein.
The visual aesthetic is very striking in that they primarily use hand-sculpted minatures that have been volumetrically-captured with photogrammetry, rigged in Cinema 4D, animated by a mix of hand animations and a Perception Neuron Motion Capture suit, edited in Adobe Premiere, and then imported into a 360 video player wrapped in a Unity application in order to add additional embodied interactions.
The workflow and production pipeline of this piece is unique and extremely impressive as the creators Bianca Kennedy and The Swan Collective were able to achieve a unique, hand-crafted aesthetic that has a lot more character than the typical, purely CGI-generated art has. They’re able to infuse additional layers of personality and charm to add to the already humorous send-up of the classic British nature documentary in the vein of David Attenborough.
It’s hard to fully describe the experience of ANIMALIA SUM, but at the heart, it’s a light-hearted comedy that continues to surprise and delight for the entire 9-minute duration.
I had a chance to catch up with Felix Kraus, founder of The Swan Collective, as well as Kennedy to talk about the evolution of this project, their experiential design process, the difficulties in trying to storyboard and plan a piece like this versus just iteratively creating it, their production pipeline process for this piece, as well as how they were able to overcome the throughput challenges of Sundance New Frontier by having five seats during the ticketed sessions at the Ray.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Here’s a trailer for ANIMALIA SUM that gives you a sense of the visual aesthetic:
Here’s some footage of their Sundance screening installation at the New Frontier at the Ray location:
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye and welcome to the Voices of VR Podcast. So continuing on my series of looking at some of the Sundance experiences, the immersive storytelling innovations, the technological innovations, but also digging into the experiential design process and the workflows for these artists. So today's episode is with Felix Krauss and Bianca Kennedy on the piece called animalia some which it's a very unique piece Very stylistic. It's a stop-motion feel although they're doing something a little different They're actually creating these handcrafted sculptures and then doing photogrammetry and then animating them in cinema 4d and then render it out and edit it and then eventually put it into Unity and have a whole sorts of different aspects of interaction as well. So really innovative way of blending together 360 video with some interaction, but also just a very unique style and workflow for how they were able to even create this piece in the first place. The piece is about eating insects. And so is it a viable alternative to eat insects to get our protein rather than meat? And so that's sort of the premise of the piece and exploring through this mockumentary style Davin Atterborough narration is just this really very funny exploration of this concept which has been put forth by a number of different theorists of just thinking about like in the future is getting our protein from insects going to be a lot more sustainable. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Felix and Bianca happened on Wednesday, January 29th, 2020 at the Sundance Film Festival in Park City, Utah. So with that, let's go ahead and dive right in.
[00:01:52.068] Felix Kraus: So I'm Felix Kraus. I'm an artist working in VR and AR, and I'm also the founder of the artist group, The Swan Collective.
[00:02:00.708] Bianca Kennedy: Hi, I'm Bianca. I'm also working as an artist. I'm working in Berlin, and especially with stop motion and VR and animation. And when we work together, it's all about evolution and future.
[00:02:13.834] Kent Bye: Great. So why don't you each give a little bit more context as to your background and your journey into this immersive space?
[00:02:22.115] Felix Kraus: So we started out basically being very interested in media art and doing films and conventional media art projects, but always trying to combine it with physical artworks. So we paint, we draw, we do sculptures, and then we try to collage them into media artworks. Often. Oftentimes, yeah.
[00:02:46.422] Bianca Kennedy: Yeah, everything I love, if it's drawing or sculpting, I then see that it needs more a time frame and so in the end everything gets like a video or a time-based art form, like for example virtual reality or video art or stop-motion film.
[00:03:03.084] Kent Bye: And so did you study art, or maybe you could talk about your journey into art?
[00:03:06.946] Felix Kraus: So yeah, we both studied in Munich, Germany. But back then, when we studied, there was no such thing as like XR studies or something like that. So we really had to freestyle all of our endeavors and to really learn by doing all that what we had envisioned. And yeah.
[00:03:29.294] Bianca Kennedy: Of course we studied art, but basically we just had free room to do whatever we wanted. There weren't any workshops, but we didn't have to attend them. We could if we wanted. And so we had a lot of time just to do the projects we loved and wanted to do. And that was perfect for everything.
[00:03:47.393] Kent Bye: So maybe you could tell me a bit about what the project is that you have here at Sundance.
[00:03:51.873] Felix Kraus: So the project is called Animalia Sum and that's latin for I am animals and I eat animals so it's like a play on words and the piece is about to identify as an insect but also about eating insects in the future so it's like a dystopic vision that we had based on research that we also did because a lot of wise men and women are proposing to switch to insects for protein and for general food consumption.
[00:04:23.777] Bianca Kennedy: But we didn't just wanted to do a documentary. We decided for doing a mockumentary and using VR where it's useful to get more empathy on other living forms and so it's from the eyes of the insect.
[00:04:37.696] Kent Bye: And so, yeah, maybe you could talk about this mockumentary aspect, because there's certain arguments that when you just would look at it through the numbers, it'd be like, oh, yeah, this makes sense. This is where we're going. But then there's a whole element of the aesthetics and the taste and everything else where, you know, are we really going to, like, move all of the entire world into moving from eating animals and eating insects? Maybe you could just talk about the deeper intention in terms of like if you see that this is actually a viable movement of people to do this stuff or are there other ways to process insects to maybe put it into Food so it can add other flavoring, you know Maybe just kind of dive into more of the backstory of you know How much of this is real and how much of this is just complete farce and it's hard for me to know when that this is actually viable at all so well
[00:05:24.408] Felix Kraus: When we start a project we tend to read a lot of books. So a lot of scientists they propose to basically switch to insect protein and because they say they see a lot of advantages to breed insects because you don't need a lot of water or resources to breed them. You can just stack them into boxes and breed them by the millions and crammed into tiny spaces and killing them by freezing is really easy. So as humans we tend to dismiss that small creatures might also have feelings about this idea and have maybe something might be opposed to an idea or a solution like this.
[00:06:05.835] Bianca Kennedy: Yeah, we just think because we have the intelligence and we can calculate that and scale that. that we are so much more important than the insects. We don't think they have a consciousness, but maybe they just have another system, but we even don't have any words for that. So that was the basic idea to come from that. So we are not really saying, oh, we should all not eating insects or animals or switch to that or eat that. It's just thinking process we wanted to have.
[00:06:35.784] Kent Bye: As part of your research, did you eat any insects?
[00:06:39.064] Felix Kraus: Well, I'm not a vegetarian, I have to admit, so I was eating them to do some research. And we were also doing an AR project where we made a video about how to cook crickets and how to fry them in tasty batter and in dough. And so, yeah, finally we also... I ate them. You had to try once. I tried one. And I mean, I can understand that it might be a solution because they don't taste bad. I mean, we eat shrimp and that's way more disgusting visually, I think, than a crispy cricket in beer batter. So yeah, we tried them and I mean, 50% of humanity is already eating them. Only the Western world, this idea is not really there yet. And yeah, so that is like the main motivation for this project. But then we wanted to go one step further to just think about what the implication might be for the individual beings with the consciousness that are affected by the solution.
[00:07:49.379] Kent Bye: Yeah, well there's a lot of really stylized animation that's happening in here with some light interactivity and different infographics and you're communicating a lot of information in this piece and trying to do it in visual ways and trying to make different points and so maybe you could talk about your creative process in terms of You talk about working with physical objects, but it looks like a CGI piece. And so are you doing these digital scans? And maybe just talk a bit about the mechanics of how you're producing this visual aesthetic.
[00:08:20.450] Bianca Kennedy: It's funny. It's the first time that I hear that it looks like a CGI piece. We built everything. Everything is miniatures. The backgrounds are watercolors. So everything was made analog in the studio. And then we photogrammetried every piece or scanned the background and put it into this 3D room to work further with it.
[00:08:37.907] Kent Bye: I think that's probably what I was picking up on that it had a spatialized experience because you know I've seen other stop-motion like passenger was at Venice Film Festival this year and then there's the Felix and Paul had gymnasia which they also had like a 3d camera that was at scale so they were able to have like a stereoscopic 3d camera but those stereoscopic 3d cameras are very expensive and so you know it's not easy to get a spatialized view like that but to use the photogrammetry on top of the physical art I think makes sense so yeah
[00:09:08.374] Bianca Kennedy: Yeah, that was our problem. When we started out, we really, really wanted to do a stop-motion VR piece. But the problem was the camera, because in our budget, what we can afford is cameras for human beings. So you really need to be far away from them, from the object, that you have a really good stereoscopic view. And with miniatures, the magic happens if you're really close. And those cameras can't do this. So Felix and Paul's stuff is so fantastic. But we just didn't have this camera, otherwise it would be a totally different piece. So after a month of work we decided that this is not what we were going for. This is just not working. And then we thought about how we could do something similar but with different techniques that we can afford and we can do.
[00:09:53.194] Kent Bye: Probably the other big striking character of this experience is the fact that it's being narrated by this David Attenborough type of character with this documentary. Maybe you could talk a bit about the narration and the narration style that you're overlaying to set this larger context.
[00:10:09.805] Felix Kraus: So when we started out we knew that it would be some kind of documentary feel to it. It is rooted in actual science and research so we wanted to make it too sci-fi and to convey the feeling that it's that this could be a real future that might come up. We also wanted to use those elements that also regular documentaries use and this is like, I think David Attenborough and his very uncanny voice is like globally known and you just tend to believe that guy, what comes out of his mouth and to play with that and to mess with that, that was also a lot of fun to do.
[00:10:53.014] Kent Bye: So maybe could talk about some of the major props and scenes and characters like as you go through like maybe just describe as you're telling the story you have lots of different insects and what were the different major things that you were creating just to strike up my memory about different points of the piece as well.
[00:11:09.790] Felix Kraus: So yeah, we started out creating the insects first and like a bunch of them by using regular materials like clay and baking the models in the oven and then hand painting them with watercolors or acrylics. Also for the first scene where you are being swallowed by a human head, we created this head using also fabric and like really raw materials to also create this really human aspect to the models. So to avoid it feeling like a CGI asset that we just downloaded from the store, we need this chaotic element that you can control of paint dripping and stuff also going wrong maybe. Yeah, so there are the insects, there's the head, there's the whale that is like the antagonist in this piece because the insects are advocating to eat whales instead of them. Because one whale is the equivalent, protein-wise, like 70 billion bucks. So there's like this antagonistic thing going on. And we also created some of the scenery. So we built a quite large model where we shot the one stop-motion scene in sequence. And the other environments that we used, like the backgrounds, they are in fact photogrammetry scenes. Maybe you can elaborate on that, where we found them.
[00:12:31.529] Bianca Kennedy: Yeah, everywhere we went from October 2018, we traveled to Brazil, for example, because we had an exhibition there, and we went to Brazil, Toronto, Iceland. We took the spaces that we really liked, that seemed fitting to the piece, we took them with us by the technique of photogrammetry.
[00:12:52.937] Kent Bye: So maybe you could talk a bit about what camera you used for that, what kind of software you used to be able to do some of the photogrammetry mechanics.
[00:13:00.456] Felix Kraus: Yeah, basically we just use our plain old Canon with a 28mm lens to capture the photogrammetry scenes or like real-life landscapes or rooms that we find interesting. And then we process the data with Agisoft Photoscan. I think they changed the name to Metashape, so let's go with Metashape Photoscan and also Reality Capture to get those 3D models calculated by the photos. that we took and then in the end we take those assets and animate them with Cinema 4D to give them like a dynamic system. We rigged our insects after scanning them. We also recorded our own body movements with a motion capture suit that was like Perception Neuron the suit. So a lot of different software and hardware were coming into play there. Also, the VR camera we used for the stop-motion scene was the VUZE XR. So we really tried for every scene to see what technology is apt for our vision.
[00:14:09.682] Kent Bye: And so, with the passenger as well as the gymnasia, they would do the stop-motion, where they would take scene by scene. But it sounds like you're taking a photogrammetry scene and then animating it within the virtual space then, rather than stop-motion?
[00:14:22.863] Bianca Kennedy: It's like 90% of the movie we did it like this, but there's one stop-motion scene left. As we said, we wanted to make a full stop-motion VR piece, but one is left. We did this with the fuse and did a really big model, but it's a 180-degree scene because we had to work around the 360 room and got it collaged so that in the end it still works as 360.
[00:14:48.833] Kent Bye: So when you're in Cinema 4D, is there a timeline or how do you edit? Are you editing it scene by scene or like how do you put each of these scenes together?
[00:14:57.578] Felix Kraus: So we edit scene by scene and the editing takes place in the end in Premiere, Adobe Premiere, with all those wonderful VR tools that they have, fortunately. But in Cinema we really work on 30 second pieces. Because sometimes you have complex transitions from scene to scene, so they are kind of more difficult to achieve. When models are dissolving into the next scene and the background is changing, then you have to really have a storyboard and a linear vision of how the scene should evolve. But we were able to have 12 separate scenes that we then, after rendering them out, only then we started to find the narrative and how to combine those scenes. So it wasn't like a fixed screenplay or even the narrative voiceover. That was like the final step.
[00:15:51.285] Kent Bye: I see. So you're starting with the visuals first and then ending with the script.
[00:15:55.193] Bianca Kennedy: We didn't want to do it that way. We really tried hard to make the storyboard first, but we just couldn't. We couldn't even talk about it and it was so hard, so we just then started building the sculptures and photogrammetry and trying in the Cinema 4D how they could work with each other and learning by doing.
[00:16:16.493] Felix Kraus: To us in the end it was a very important process to do it like that because we noticed we really had to iterate a lot. So we had to test, we had to watch every scene in the headset, I think we rendered every scene like six times until the scale was right, until the elements were not too close or too far away. until the movement was not making you nauseous and so I think being only like a two-person team it's kind of really hard to get a piece like this done in six months time but the advantage is you can go back and forth and if we would have had to write a script and give this to some VR studio, then we wouldn't have had the agency to react to certain outcomes and to adjust and tinker with it. So I think this was what ultimately made the piece really like one seamless experience that we could experience it, we could change stuff and make it perfect in the end.
[00:17:20.874] Kent Bye: So with two people and different bottlenecks along the way, how did you divide the labor amongst the two of you to be able to be able to continue to make progress as you're moving through it?
[00:17:32.063] Bianca Kennedy: We did everything together. It's just we read the books together and we brainstormed and we wrote the text together. We built the miniatures. We just didn't divide anything.
[00:17:42.316] Felix Kraus: Yeah, and sat in front of the computer together and decided on each and every click, which was kind of a challenge as well because we both have strong visions and not always they are congruent. After that project we jokingly said now we need two years off to proceed our individual projects because we not always work together. It's just like every two years we do this big joint venture and then we are off to our own projects.
[00:18:14.488] Kent Bye: Yeah, in computer programming, they call that pair programming. So you're basically having two people, one person who's driving and one person who's watching over. But I think that's actually, they find that you get a lot more efficiencies sometimes because you're able to help each other find different things or help guide or just make the decision making process go faster sometimes. So yeah, it sounds like you're kind of applying that pair programming technique to this relatively small scale project compared to a lot of the other projects here. Yeah.
[00:18:42.561] Felix Kraus: Yeah, I mean basically when you work and live together and spend six months of working on this really weird vision together, then it's also important emotionally to just help each other out. I mean, I think every artist can relate. that there will be situations where you doubt everything and especially after months of maybe even not seeing a lot of people or not going to Venice for the Biennale because you just have to work on this project and to meet the deadline that we had because of an important exhibition in Barcelona. We needed each other to support the other person. I did cry a few times during the making of this project because it is hard to go through all those steps and to basically have to learn everything from scratch. Nothing that's in the project someone taught us. I mean, it's a fun ride, and in the end you are glad that you learned it, but when you have to do it in a very short amount of time, then it might be kind of challenging, yeah.
[00:19:53.783] Kent Bye: And so maybe you could talk about showing it here at the Sundance New Frontier. And I know that there's a couple of locations for the New Frontier. Well, there's three now. There's ones that are out and about around Park City, but there's the two main locations. that have addresses that you can go to specifically. There's the New Frontier Central, which has all the ones that you can go to that are a little bit more open to anybody with a pass, and you can go in and sign up on a list. But then, the New Frontier at the Ray, downstairs in the basement, there's 90-minute sessions with a fixed, limited amount of projects. A lot of people who are rushing in, trying to see as much as they can, as fast as they can. It's like a gold brush. And your project has like five different chairs that are there. It's an eight minute piece. It's pretty easy to see, but yet at the same time, there's all a lot of other projects that you have to basically stand there in line, not seeing anything. And you know, sometimes spend your entire session waiting just to get into a piece. So maybe you could talk a bit about like what you've been seeing or hearing from the audience as you've been showing your piece here at Sundance.
[00:20:58.781] Felix Kraus: I mean, we realized that people were really glad that there's at least one experience that there's no hassle to get into, there's no queue, they just can approach us and they could get into the experience. So this is nice for a change because there are pieces and they are so fantastic, like Scarecrow, we are talking a lot about this project because it's like a one-person theater piece and you spend half an hour with a team of six people behind the curtain to guide you through this very unique experience. They can do seven people a day and we can do literally like 100 people a day. After 10 days of Sundance there are maybe a thousand people who have seen our piece. from the get-go when we decided to work with VR we always had this aspect in mind that it should be like friction free to show and experience the piece. I mean we work a lot with galleries and museums and if they would have to buy a $3,000 computer and an expensive headset so they have one point of access and they have to maybe restart the experience every time a person experienced it and then this is Yeah, maybe not perfect to get your work shown or you give up after maybe one or two exhibitions because you realize, wow, that's just too much effort to show your work. So we wanted to work with mobile headsets like the Oculus Go, for example, which doesn't depend on an external computer. It doesn't depend on external sensors. And we programmed it that way. So when you put on the headset, it just starts and it stops by itself, so it's easy to maintain and supervise. And having this in mind, nevertheless we wanted to push the boundaries of 3DOF, because we're not advocating for 3DOF, it's just what's possible at this moment with this limited technology. But you said, why can't we make a passive 360 experience more interactive? And I said, well, it's not really possible. You either have your pre-rendered stereo imagery or you have a Unity-based game engine piece where it's not that elaborate rendered graphics or that high quality of pre-rendered stuff. But in the end we found a solution of how to combine 360 VR player with interactive elements. So that was really new to us and a nice way to make it more interesting with the Oculus Go.
[00:23:37.601] Bianca Kennedy: VR, especially in the art context, can be so frustrating for the people that are wanting to see it, because there usually is one headset and a long queue, and yeah, especially in the art context, you're not used to that, because you usually have a painting and you can see it, or something else you can just see and not wait for that. So here it's really, really nice that we got five headsets, and usually I think that everyone who wants to see it at the RAE can see it. So this is the first time that we got this fantastic chance. And if we show this piece or other pieces we made in an institution or museum, usually what we try to do is to build an installation around it so that you know what to expect from the piece aesthetically and that even if you're waiting in line that you can see something else and not just the headset and the people wearing them.
[00:24:29.760] Kent Bye: Yeah, I was talking to one of the people I was staying with and I was thinking, you know, like the book existence is one of my favorite experiences that's at the Ray, but it's 25 minutes for two people at a time for a 90 minute sessions. That means like six or maybe eight people tops will be able to see it, but anywhere from five to eight people will get to see that. that's if that they go there and decide to wait there for up to an hour, half hour or an hour, which means that they're standing in line not doing anything for an hour, why not have a way for them to watch a piece if they're waiting for a half hour, watch a 20 minute piece if they're waiting for an hour, have them watch a whole series so that they're able to get an experience of other things while they're waiting and then they get into the queue for that. I mean, it would end up with a lot of people waiting on stuff and it have to be scheduled. I was just trying to think of ways of having like this eight minute piece is something that you could watch that two or three times for if you're waiting in line. And could there be many of those different pieces that people could like sit down in a chair to hold their spot, but yet at the same time, choose to watch a number of different cinematic VR pieces. Just thinking about the throughput issues is so challenging because there is all of this waiting and queuing and how can you make that queuing have less friction and allow people to feel like they're having an experience without having to feel like that they have to essentially stand and wait for after paying all this money you have all the stress and creates this context and environment that I feel like is a lot of pressure but also like isn't a great experience for actually viewing the content.
[00:25:58.809] Felix Kraus: Yeah, I mean I can relate to everyone who's frustrated because he or she only saw like a fifth of the pieces that they paid for in the end. So it's really great that you all also think about how to make it more worthwhile to wait for the actual artworks to experience. And I think also we artists should think about the whole meta aspect of showing a VR piece. It's not about putting a headset there and have your piece. we really should and we already started to think about what to offer while waiting. Maybe you have like a booklet, you can read something, you can watch an accompanying short film about it, you have maybe other experiences by the same artist that can be shown on mobile devices. So there are ways to maybe overcome this big frustration of just not seeing what you want, what you signed up for. We all should really think about that.
[00:26:56.615] Bianca Kennedy: Enjoyment parks, for example, are a great inspiration for that because they have the same problem. You're waiting for a roller coaster and maybe you have to wait in summer during vacation like for an hour and they are doing that with voices that are coming or other people, performers that are doing something or videos that are playing and of course you can't just steal it and maybe it's just not fitting but it's interesting to think about these and I always love to go and steal a little bit.
[00:27:23.983] Kent Bye: Yeah, maybe you could go into a little bit more detail as to the interactive components because you mentioned the whole photogrammetry to Cinema 4D to then export into Adobe Premiere, but that doesn't have the game engine included into that. So then are you exporting into like a 360 video format and then playing that 360 degree format into Unity and then putting interactive elements on top of that?
[00:27:46.495] Felix Kraus: Yeah, so basically we created our own stereoscopic 360 video player that is playing back our pre-rendered content inside Unity. So you have your 360 sphere around the camera and this is playing our MP4 file, so to say, our top-bottom file. But then you are in Unity and then you have all the advantages of real-time elements and interactions, so we could attach dynamic antennae and jaws of an insect to the camera. So when you watch the piece and when you turn your head around you have a dynamic reaction of antennae that are attached to your head and it's like an extension of your own body. You immediately are way more immersed in this foreign creature and you have an easier way to create empathy for those little insects which we ultimately envisioned. And also then in the middle part we have this interactive mini-game where you then are forced back into the role of the human aggressor again. So you don't have your antennae, you are the human attached with some probing stick and suddenly it's the challenge to milk those insects for protein and for antibiotics and other drugs and to have the switch in consciousness and empathy That was really interesting to us and a lot of people of the audience agreed that it was kind of a challenge or at least interesting for them to have the switch to feel sorry for the insects but suddenly they had to exploit them and this raises a lot of questions.
[00:29:30.440] Kent Bye: So just to get the workflow in my mind, sort of what's happening at each phase, the photogrammetry takes the physical object, you get all those images that then creates this 3D scene graph that is the 3D models that have the textures, you put that into Cinema 4D which then allows you to then have like a virtual camera in a virtual space that then create the cinematic elements that allows you to then export into what is presumably a stereoscopic file that then you can put into Adobe Premiere to then edit and then from that edit you put it into Unity to play that file back. So what is missing from Cinema 4D that you have to use Adobe Premiere? Because it seems like you would just want to use the same tool. if you're already doing the cinematic tool, is the editing tools better for domain? What are the things that are missing in that workflow so that you have to actually use those two different tools in order to get to that point of exporting a stereoscopic cinematic video?
[00:30:28.137] Felix Kraus: Well in the end you have linear video with a narration and you have to do sound design. We also did it ourselves in Premiere and we had to place the narration of that David Attenborough impersonator at the right places and also we had to think about where the audience would be looking at at each scene because some scenes started out looking forward but then you maybe end up looking backwards and to make sure that the following scene, you don't miss anything, you had to rotate them as well, those scenes. And this is like this VR editing that's not like just cutting pieces from each clip, but to think about the space and to rotate and think about the experience.
[00:31:15.930] Kent Bye: I see. Great. And so what were some of the biggest challenges and obstacles that you had to overcome in making this piece?
[00:31:24.793] Bianca Kennedy: Especially talking about it in the beginning, when you don't have anything, when there's no puppet made, when you don't have a script because you just can't write it, because you can't decide anything. That was really, really stunning. We had it on our calendars for a month and we talked a lot about that. just deciding that now this is the sculpture and now we are doing the backgrounds with watercolors so in the end if you see it and you have everything then it's clear that it should be like this. But it was a long process and I really didn't like the first two months where we just couldn't decide because we didn't really know what we were going for except the topic and there were many pieces but we couldn't fit them together.
[00:32:14.703] Felix Kraus: Coming from a conventional 2D cinematic background, because our prior projects have also been regular film, it's hard to talk about an idea or to do a sketch. You always end up doing a sketch in a 16x9 frame and everything off-camera, you don't even think about it. we do a close-up onto this bug and then we go there but then when you have to create the scene of the close-up of the bug then suddenly you have to think about what is off-camera because there is no off-camera what what's happening above you what's happening behind you is it still interesting to look around because you can't dismiss in VR to have a every directions people want to explore and you have to deliver a satisfying experience by exploring and looking around. So, yeah, we really had to find a language how to talk about each scene and to expand our preconceived notions of how a scene should be, especially a VR scene, yeah.
[00:33:20.047] Kent Bye: What do you each want to experience in VR?
[00:33:23.275] Felix Kraus: So to me, I always find the things really interesting that you can put into words. So, for example, if there's an experience where shadows are really done right, where a shadow is reacting to the digital assets and you can explore this seemingly random thing that hasn't any storytelling value, but you can just explore this detail and it somehow conveys a pretty archaic feeling to things that stem from reality, but seeing them represented in virtual reality and having the feeling that the creator really understood that it's more about those tiny details, not about like a broader vision to have your assets just running around and going from point A to Z. But really valuing the little things in VR experience, this is where my heart goes a lot faster and I sense that it's not about recreating traditional storytelling devices in VR, but to find a very unique language in VR. You can't just say I'm doing this film, but I'm using VR for it. You have to really understand the medium and to understand what you can do more than that you could do with your regular camera.
[00:34:48.157] Bianca Kennedy: I'm fascinated by the body and also body horror and the body first goes against VR because they use avatars and you don't really have one usually in the pieces when you look at down and you don't have a body and I really like that feeling what you can do with that and also like peeing or eating which isn't a big part of VR yet but I would love to see more stuff like this.
[00:35:14.575] Kent Bye: Peeing in VR? What would that look like?
[00:35:17.406] Bianca Kennedy: I don't know. I really have to think about it. I want to make it.
[00:35:20.309] Kent Bye: So when you go to the bathroom, do you have an immersive experience that would give you some sort of experience that would maybe be amplifying that?
[00:35:26.836] Bianca Kennedy: Yeah, everyone does it. So it would be really interesting how maybe you can pee from your eyes and experience that. Yeah, I want to see that.
[00:35:37.505] Kent Bye: Interesting. Wow. Well, actually, it's a piece where I know there's been some installations where you would go see a piece while sitting on a toilet, but it wasn't necessarily about peeing, but it was more the context. But yeah, I mean, I think as we move forward, there's people spend a lot of time on their phones when they're in the bathroom. So why not have some VR experiences that make it much more exciting?
[00:35:57.289] Bianca Kennedy: That would be so worth it. Yeah. Other people read on the toilet. Why not having a VR experience instead? Yes.
[00:36:04.729] Felix Kraus: Yeah, and your prior piece, We Are All In This Together, is going in the same direction. I mean, you can elaborate on that as well.
[00:36:10.831] Bianca Kennedy: Yeah, the first piece I made, it's more about atmosphere, and it's based on drawings. And you are on a bathtub. From the top, it looks like a cross, and you are sitting in the middle. You have a body there, and you have four bathtubs around you with people, drawn people sitting in them. And so, yeah, I like toilets and bathtubs, it seems like.
[00:36:32.062] Kent Bye: So there was a piece last night called Machine for Viewing which had a number of different film essays but they were in a virtual world that someone was experiencing in a VR headset and it was being projected onto a cinema screen which everybody was watching. So it was a room full of people in a movie theater watching somebody watch a film essay. So everybody was watching the film essay but through the eyes of someone in VR. And so in talking to the creators afterwards, we're talking about what can cinema learn from VR and what can VR learn from cinema. So what do you think are the insights that you can pull from cinema that you're pulling in from VR and what are the aspects that you start to see from VR that you can feel like you can apply to cinema?
[00:37:12.353] Bianca Kennedy: Cinema is a big fascination and inspiration for me and everything art and cinema and usually I put cinema into my art and so I can totally understand that you can use cinema and lighting for your VR piece. But thinking more about how the body thing and being an avatar and putting that avatar back onto the cinema would be really interesting too.
[00:37:36.647] Felix Kraus: Yeah, I mean traditionally we tend to think about how stories have been told for hundreds of years already and I think the main challenge is to now free yourself from those traditions and to be risky to tell a story very differently and to not succumb to the pressure of having to deliver like a story arc or like to try not to deliver the same stories that have been told before and to find your own voice and your own language in VR. I think that is why I love it so much because for the first time in my artistic career I have the feeling that I'm in completely fresh territory. When I was painting before VR I had like hundreds of years of art history in the back of my head and comparing myself to different artists and painters especially. But with VR you suddenly have the feeling of, I mean if you're doing it right, of having the feeling that wow this hasn't been done before and this is like really something new and this is what drives me in the end.
[00:38:46.405] Kent Bye: Great. And finally, what do you each think is the ultimate potential of immersive technologies and immersive storytelling? And what am I able to enable?
[00:38:57.738] Felix Kraus: So to me, as a person and as an artist, it's the ultimate tool to deliver what's inside your head. I mean, I'm a vivid, lucid dreamer and I was researching this for years, also before VR. I have written down my dreams and I was used to write down what was inside my head. And it's kind of difficult to convey in words and to tell someone your dream, it can be quite boring. But you know it's that really rich world and somehow it's stuck in your head. Until VR, I didn't have the right tools to get it out. And when VR came along for the first time, I thought, wow, now I'm able to open up my head and to put my audience inside my head and to really let them experience what I'm feeling and to experience emotions that you can't even describe.
[00:39:54.628] Bianca Kennedy: When you are creative you really want to bring people into your pieces and it doesn't really matter if it's painting or video or VR, but with VR you have so much more opportunities because you have the 360 room, you don't have the wall of the TV where it's hanging and maybe the wall is ugly, but in VR you can totally change that and that's a great, great opportunity we have.
[00:40:19.517] Kent Bye: Great, is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:40:24.267] Felix Kraus: Well, after having listened to hundreds of episodes of you, I was wondering, what do you think is the ultimate potential of VR and what it might be to enable?
[00:40:37.298] Kent Bye: Well, for me, I've got a number of presentations where I go into that for like half hour, but the sort of the short answer is it connects you more to yourself, connects you more to the other, connects you more to the earth, and connects you more to all of reality. I think what VR is going to do is allow us to have this new communications medium that actually I think is going to have all sorts of deeper philosophical implications that are at the root of how we make sense of what reality is. And that goes into the metaphysics of reality. If we see that the metaphysics of substance metaphysics is something that Aristotle would talk about in terms of, it's something from the Greeks that has a specific word that gets translated into substance metaphysics, but that substance could actually have a part of it that's actually a part of being or becoming. So being or becoming could be a same meaning as that substance in that Greek word. And so what does it mean if the underlying nature of reality is being or process? Alfred North Whitehead's process philosophy talks about these concrescence of these moments that arise and then they have a life cycle. So everything's in the process of living and dying through this cycle. So what if this shift towards like the static nature of having concrete objects of books or movies, they're fixed, you know, they have this production process and you see it and everybody that sees it will see the same thing. I think we're moving into a world where it's much more interactive and dynamic and engaging so that people can actually see how their individual agency is expressed within the medium rather than being a passive consumer of something that has been fixed from an authority. I feel like we're in this moment of just being here at Sundance, a lot of things that I'm seeing is this counteraction to the excesses of capitalism and surveillance capitalism specifically in technology companies. in that the way that Lawrence Lessig has sort of mapped out of how there's different dials to shift, to shift collective behavior. He calls it the pathetic dot theory, which is like there's the dot, which is you as an individual, and you have different influences of architecture of code of either the world around you or of technology systems that are there. You have the law from the government, you have the market dynamics of the money, and you have the culture. So you have these different ways in which there's cultural influences, cultural norms, you have the economic incentives of needing to make a living and operating within the economic system, the laws you have to follow, but those larger laws set the broader context for how the companies operate. And then you have the code and the technology. And I feel like VR is a part of this new technological underpinning, which is at the root of cultivating new culture, cultivating new market dynamics, and cultivating new laws. And so I feel like it's this part of this diffusion curve that at the beginning is the architecture and the code, the genesis of the idea, and then it gets dispersed out into enterprise application, bespoke applications, where it starts to then have some market dynamics, usually with the enterprise first, and then it gets into the consumer market, which VR starting back in 1968 and having its first consumer enterprise phase of the first hype cycle back in the early 90s. But now we're in that consumer phase where consumers can actually buy it. It's been a 50 year journey, but it's been a long journey, but it's at that consumer phase now. And the final phase is that mass ubiquity. I feel like there's a natural progression of ideas of that technological architecture and code that then moves to the early market dynamics, the enterprise niche, custom bespoke, and then into the mass market, consumer market, that shifts culture in different ways, and then finally, through the mass ubiquity, then have some walls to be able to give some structure around it. So I feel like VR is able to start to make shifts at each one of those phases, whether it's creating experiences that changes policy, whether it's seeing how you could change your behaviors on certain aspects, or enterprise training, or different market dynamics there, or there's new ways that you can cultivate experience. So, with the printing press, you have this 500-year dominance of books. I feel like we're in a similar phase where we're in a 500-year dominance of computing, and computing technology, and algorithms, and automation, and artificial intelligence, but it all kind of comes back to algorithms and computing. As books were able to democratize access to information and knowledge, VR is able to democratize access to human experience. And so with that, we have this capability of, what are we going to do with this experience machine? What kind of culture do we want to cultivate? What kind of market dynamics do we want to cultivate? And I think some of those market dynamics, actually, if you think about it, most of capitalism is based upon constrained resources. So you have dynamics where you only have a limited amount of resources in order to produce different stuff. But in the digital realm, you have this other thing where Creative Commons, the more people that have it, actually, it's more preserving so that it's more safe to have billions of people have a digital file on their computer than it is to have it just in one place. And so actually, things that propagate out are popular actually are more easy to keep preserved, which has other dynamics of like, is the only thing that gets archived things that are popular, which is not a road we want to go down. But if you think about the existing way that the internet is built through the centralization of the internet service providers, I feel like part of the next big phase as we move forward, part of the new future architecture of the internet is going to be the decentralized web and have decentralized architectures where you can have edge compute, edge networks, and locally shared information that people can start to have experiences that may be site-specific or you have to go someplace to get access to them. And then there's other architectural elements that are just launching now, whether it's WebXR or WebAssembly. The World Wide Web has HTML, JavaScript, and CSS that has an amazing ability to transmit all sorts of information and knowledge. And now they just added a completely new language of WebAssembly to be added to the one of three that we've had to run all of the World Wide Web. We have a fourth that just got launched to be able to run binary code. And that binary code is going to be able to essentially do all the things that a web server could do. So you have the ability to do serverless architecture, so to have these decentralized nodes. So what does that mean to start to invert the dependence on these centralized places that are surveilling us and controlling us and manipulating us in different ways? It's a part of the backlash to the excesses of surveillance capitalism that I think will start to drive this movement toward the decentralized web and to create these new cultures where there may be a little bit more of like a wikipedia style open sharing of culture and knowledge but in a medium that's experience-based. So finding these formats to be able to share our experiences and share our knowledge and share our stories in a way that is not resource-constrained in the same way that other experiences have been resource-constrained. Once you actually have the equipment, then the world is your oyster in terms of what type of experiences you want. I think that's the world we're living into that is going to create new market dynamics, cryptocurrencies, and microtransactions that There may not be an algorithmic solution to capitalism to create a more equitable world. I think it's still going to be an aspect of consciousness of people sharing and redistributing and being conscious about how their resources that they have get spread out. But yeah, that's at least, I think VR, to me at least, is one of the most highly leveraged places that I can be working, people in the industry can be working to be able to bring about a larger change.
[00:48:26.368] Felix Kraus: Great. Do you think we're on the right track? I mean that's a very positive vision and it would rely on a lot of creators being able to deliver that content and also to do it maybe, a lot of them would need to do it non-profit without money.
[00:48:42.475] Kent Bye: We are on the opposite track. we're actually going into this walled garden universe where one company controls everything. It's essentially Ready Player One playing out in real life, where IOI is Facebook and Google and Amazon and Apple and Microsoft. Collectively, there's less than a dozen of technology companies that are controlling our destiny online. And I feel like the affordances of how wealth and capital gets transmitted, that it has had this disparity of these huge billionaires. They have all the power and all the control. But at the same time, the thing that gives me hope is folks like Neil Trevitt, who says that every open standard has a proprietary competitor. So there's a value for something like the Quest. But we need an open-sourced Quest. We need a version that is just as good, but works and doesn't have the same constraints. But Windows existed for a long, long time before Linux came around. And how long has it taken for Linux to have the same user interface and user experience that is as good as Windows? It's taken decades, it takes a lot longer. So it takes longer. So I think it will get there eventually, but it just takes longer. And it's important to have that dialectic between that closed and open. But we need that open alternative to be able to have these alternatives. That's why I'm such an advocate for WebXR just having launched. So people want to have some sort of resistance. And I think it's important for people to continue to build out these tool sets that don't rely upon only a monoculture of just a handful of companies that are controlling everything. Because we actually need the market dynamics to actually give people the experience to show what's possible, and then to see what those limitations are, to see what other content that is out there that may not be curated, or may not be into the walled garden, may not be invited in, and to create this system where there can actually be this collaboration between these polarity points.
[00:50:33.983] Felix Kraus: Really well put. Thank you for an amazing answer.
[00:50:38.544] Bianca Kennedy: I would love that.
[00:50:40.567] Felix Kraus: I might quote you on that. Awesome. Well, thank you so much for joining me today.
[00:50:44.790] Kent Bye: So thank you.
[00:50:45.491] Felix Kraus: Thank you. Thank you, Kent. Thank you.
[00:50:48.254] Kent Bye: So that was Felix Kraus. He's the artist working in AR and VR, and he's the founder of the artist group called Swan Collective, as well as Bianca Kennedy. She's an artist in Berlin working in stop motion, VR, and animation. And their piece that they had at Sundance, New Frontier, was called Anomalia Sum. So I have a number of different takeaways about this interview is that first of all, well, this is just a delightful piece. I'm so glad that I had a chance to see it there at Sundance. It's a lot of fun to go through and the style is so unique and the look and feel that they were able to create. because it really goes beyond a lot of what you've seen in typical computer generated CGI graphics. Now, part of the reason why I felt that it was a little bit more CGI than stop motion is because in stop motion, it's like you can really see the imperfections of the movements. And when I saw this piece, there wasn't any of that stop motion technique. They were actually creating a sculpture and then doing photogrammetry and then rigging it up and then animating it. So then any of the motion that you see has a lot more of that CGI animation that you have. But that workflow, the pipeline, so they have different sculptures, they do photogrammetry using a Nikon 28mm lens. With the Metashapes, the Agisoft Metashape photo scan. they're able to get like a 3d model and then rig it up and then put it into cinema 4d and then render everything out and then potentially even do like the animation with perception neurons so they're able to actually like put their embodiment into these different bugs that they've created and then export that eventually into a video that could then be edited in adobe premiere to be able to do a lot of the sound design in there, and then export that out into a 360 video, and then take that 360 video, create a 360 video player, put it into Unity, and then have another layer of interactions where you're able to use your head movement, because this is an Oculus Go, so it's very limited three degree of freedom. experience, but still able to have like these extensions from your face that makes it feel like you're an insect and be able to interact with a number of different scenes where it feels like you're able to engage with these interactive infographics. And so overall, just a really delightful piece. And so glad that they were able to put that into the new frontier at the Ray, where usually you end up having to wait for at least 15 minutes or so for any of the other experiences. And so. there's a good chance for any of the experiences that you see there. There's about three hours of content there in the 90 minute slot altogether of all the different experiences. And the Animale Assam was like five different seats and you know, you could just walk up and see it. I would like to see in the future. And I talked to Milo, one of the curators, I was telling them that I'd love to be sitting outside of waiting to see a certain experience and then to be able to see some of the experiences like this and not have to go out and spend some of your precious time waiting on some of the other experiences. So as I was making my way through all the other experiences, this is the one that I ended up seeing last, but I'm so glad that I did and be able to kind of unpack the different dynamics of this experience. And so it's a fun piece. I mean, it's type of piece that you really need to see to fully appreciate. But the thing I'm really taking away is this workflow of using photogrammetry to be able to create these handcrafted feel of objects to get away from a lot of the normal blockiness. And so to really have this artistic design that allows you to do this handcrafted feel, but to then to do the photogrammetry to bring in different scenes and. And for Bianca said that they've been traveling around to these different places around the world, and they would be able to do photogrammetry scenes and bring different aspects of the world and places that they were traveling to and bring that into this experience as well. And like I said, it's got these different interactive components, which I haven't seen a lot of experiences on the go that use a lot of really well designed and crafted three 60 videos on top of these different interactive components that you have, because it's wrapped within a unity container. So hopefully you'll be able to see this experience at some point. It's a very delightful experience and yeah, very much enjoyed it and kind of inspired to look into different ways of doing this type of pipeline and, you know, getting stuff from Cinema 4D or maybe even capturing scenes in web VR, for example, and to be able to export that into 360 video. And then from there to create. different prototypes that you could be edited into Adobe, do some sound design, put it up on YouTube. So just to take different experiences and to think of a distribution platform of YouTube. So there's something there for creators as they're starting to get into the medium of virtual reality to think about the minimum viable product and how You could put something together with this type of workflow and then just get something out. And then even thinking about, can this be a 360 video? Does it need to be an interactive piece? So you can at least get it out there and people can have access to it, even if they don't have a virtual reality headset. Obviously experiences like this are way better when you're seeing it in VR, but just to be able to think about how to put things out as a 360 video first, and then think about how to expand it out, especially with a piece like this, where they're able to do a whole lot of things with just the 360 video on its own. but they're adding different interactive components as well that really justify it being an interactive Unity experience on the go. But just seeing how 360 video can be a stepping stone for further explorations of experiential design. And I've seen a lot of different groups over the years who start with 360 video that really understand something fundamental about telling stories in the medium and then eventually move into the fully interactive So I think that's a good workflow, especially if you're coming from more of a film background and are familiar with some of these existing tools. You can start to bootstrap and prototype different types of stories, see how they work, get an intuitive sense of the medium, and then as you move into the more completely CGI or interactive pieces that are driven by either Unity or Unreal Engine, then you can start to use the fully interactive and Six Degree of Freedom type of interactions as well. But a piece like Anomaly of Some really shows how far you can go by just combining stuff like 360 video on top of some of these light interactions that you have as a rapper. One final note is that in both this interview and the next interview that I did, the tables were turned on me and I was asked, you know, what I think the ultimate potential of VR is. And it was interesting for me to do these interviews back to back actually, and then to have them both asked me because I found in this, I was trying to work out this larger conceptual breakdown between the early stages of the genesis of an idea, the custom bespoke model, and then the product, and then eventually going out to mass ubiquity in the mainstream. And just how there's a technology diffusion that happens through each of these different phases, and that there's aspects of virtual reality that are going to start to do a one-on-one substitute where it's just swapping out functionality, and then you're able to augment in some ways and make it slightly better, and then modify, and then eventually completely replace. You're able to do things in VR that you wouldn't be able to do anywhere else. I feel like there's a similar kind of diffusion that's happening with virtual reality as we start to implement it into all these different use cases and contexts that we're kind of going through this one-to-one substitution and doing a slight augmentation and modification. Maybe you're able to do things that are somewhat different and then eventually there's things that you could do that you could not do in anything else other than the medium of VR itself. I feel like that that's going to have a huge impact and also bring about all sorts of deeper philosophical paradigm shifts that after I came back from Sundance, I did a deep dive into really trying to formalize and describe what those specific paradigm shifts are. And in my talk that I gave at the 2020 Education and VR International Summit in Altspace, I started to explore that a little bit. But if you want a book recommendation, I definitely recommend Dynamics of Transformation by Grant Maxwell, who starts to break down some of the philosophical principles and concepts as they start to be added together and how those might form together into a completely new paradigm. I think that's just one model. I think there's going to be lots of other aspects of those concepts from artificial intelligence and the blockchain and decentralized web and category theory and all sorts of other stuff that are not in that book. But I think it actually is a good baseline to start to talk about some of the fundamental concepts that will be familiar for anybody who's listening to the Voices of VR podcast to hear different aspects in that book itself. So that's all that I have for today. And I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast and please do spread the word, tell your friends and consider becoming a member of the Patreon. This is a list of support a podcast. And so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of VR. Thanks for listening.
