One of my favorite experiences from SXSW was Vincent Morisset’s Composition, which is a physical installation of a table with blocks that serves as the interface of a interactive music piece that continually shifts the algorithm for how it generates the music. It’s got some great beats and tunes that manages to always sound great while still giving the users the ability to modulate and shape the music through how they distribute the blocks on the table. It’s a remarkably elegant experience that is able to tap into some deeply immersive and delightfully amazing interactions. There’s a physicality to this experience that goes beyond the pure VR or AR experiences that we’ve been enjoying from the comforts of our home, and also taps into some unique social dynamics as an experience that also works well with more than one person. I talk with the director of the piece, Vincent Morisset about how he developed this as a part of an artist residency at the Phi Centre, and the evolution of this piece to achieve this balance of agency and novelty.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So, continuing on in my series of featuring some of the different XR experiences at South by Southwest, today's episode is with Vincent Morissette of Composition. So, Composition was one of my favorite experiences that I saw this year at South by Southwest because it was such an immersive and transportive experience of agency where just constantly surprised me and just a lot of fun to be able to move these physical building block bricks on this table then then had this projection map to be able to have emergent music composition and other pleasing visual aesthetics that were happening there and just really quite magical, actually. It was one person or multiple people, so it had a lot of ways in which it was the type of experience that you couldn't have remotely. It really needed to have that physical installation with the projection map and everything else. It was just one of those things that you could only really get the true experience if you were there physically to be able to have that type of experience. One of the other things that makes it so visceral is because it shows the difference between what you can do within the virtual reality and what is left out if you have that lack of physicality or that lack of real-time social dynamics can emerge as well. We'll be talking about how this project came about. It's from a residency program at the Phi Center in Montreal. Vincent has a huge background with doing lots of different interactive and immersive experiences over the last two decades. Previously, he did an interview with him just going over some of the different work that he's done in the context of the 15th year anniversary of the InfoDocLab. So I'll be linking that in the show notes if you want to catch that as well. But yeah, just really on the frontiers of experimenting and tinkering on the future of immersive and interactive experiences. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Vincent happened on Tuesday, March 15th, 2022. So with that, let's go ahead and dive right in.
[00:02:03.113] Vincent Morrisett: My name is Vincent Morissette. I'm from Montreal. I'm a director and the founder of the studio Attois. That means yours or your turn. I've been doing interactive projects for the last two decades and some were in headset. And I'm here in Austin at South by Southwest presenting a piece called Composition. It's a piece where you're in front of a table with cubes on it, with little faces sketched on each surface, and you're invited to move them, stack them, and create a composition. So depending on how you arrange the cubes on the table, musical structure, evolve, change, and also the visual. So it's a play on the light and shadows and kind of abstract and figurative elements. And you don't need to have musical skill. It sounds amazing all the time. So it's not presented as a musical tool or interface. It is more kind of engaging with a film and you kind of transform it with your hand a bit like a god-like You can do it by yourself or with someone you know or total strangers as it happened many times during the festival.
[00:03:23.753] Kent Bye: Maybe you could give a bit more context as to your background and your journey into creating this type of interactive and music-based pieces.
[00:03:30.757] Vincent Morrisett: Yeah, I define myself as a director. I've always been fascinated in combining film grammar and interactive video game mechanism and that kind of junction between those two vocabulary. I've done interactive music video for Kid Fire, done a bunch of interactive projects with the National Film Board of Canada and I've always been fascinated by what interactivity brings at an emotional level, how the fact of changing or transforming what you see or engaging with it open at kind of a different emotional window. But the difference with a classic video game is that I'm never into kind of performance or goals. It's often impressionist, poetic take on a journey. And I'm agnostic of platform, of medium. As I said, we've done a lot of web stuff in the past, mobile projects that are kind of a hybrid of books and video capture. And in the past, we've also developed a lot of projects with computer vision, so using the body. as a way to engage with the piece instead of having some kind of interface in your hands or joystick or a mouse. So I'm continuously looking for a new way of engagement and trying to make the interface disappear so that I think the future of engagement with content will be kind of seamless. And I think that's the ultimate promise of XR, kind of forgetting about technology and just be in the moment, in the place. So composition is a bit a take on that, just being in the physicality, the pleasure of stacking cubes, throwing them around and just kind of having a kind of a really reactive, complex, rich outcome of this.
[00:05:33.032] Kent Bye: Yeah, I had a really amazing experience with Composition, I just really enjoyed it, and in fact it's been one of the experiences I've been recommending a lot of people go see, because it's short and sweet, six minutes, but then it's really engaging and interactive, and what I found is that whenever I've played with interactive music pieces, sometimes it's hard to make it sound good, like whatever you're doing, but also to make it seem like it's not just a tool, like that you're just making music. but you also don't want it to be so chaotic as to like to have no agency to have input into what you're doing it's just already kind of pre-programmed or just sounds like noise but it always sounds good but there's always these moments of perverting what my expectations are because you know it starts off as a sequencer but it kind of jumps into maybe a focusing on one block and having a 360 degrees radial scan to see what other blocks are in relationship to it. So it's kind of like a left to right orientation and then kind of like a more of an individual polar coordinates where it's rotating in a circle. So it's kind of jumping in between different coordinate systems.
[00:06:30.188] Vincent Morrisett: And pinballs sometimes where it just kind of jump from one to the other in a kind of a really hypnotic way. And as you say for us it was really important that you didn't need any musical background and actually it empowers people so there's this kind of evolution when they arrive in front of the table without any preamble like no intro and then they put one and then the natural way of approaching it is engaging as a classic sequencer and then at one point there's a point where you kind of abandon and you just let go and just kind of throw the cubes everywhere and experiment with what can be done. And then there's this marvelment about having this kind of ecosystem continuously reacting and generating something beautiful. So it was a hard work. Like we did this project during a residency at the FAI. So we had the luxury of time and a place in a small team like Édouard Langton-Benoît, Caroline Robert and Villoupeur. who is a really talented beat maker. So he's from the hip-hop community. So it's a project that has this kind of underlying beat that is really visceral and satisfying. And then there's this kind of more music score quality to it that makes you feel that you're kind of observing miniature world. So we kept that tension of having some kind of the control and really understanding what each cube does and at the same time having this kind of backbone that brings you into different places.
[00:08:04.528] Kent Bye: There's also, it's a projection mapped experience. So there's a table, you have a number of different blocks that you're moving the blocks around and you have different audio reactions based upon whatever conceit is happening, whether it's going around in a circle or like a more of the scanning line sequencer. And so it's reacting to the physical blocks that you're putting onto the table, but it's also projecting onto the table other images. And so you've also got this other visual aesthetic that's developing. Maybe you could talk a bit about that because it's kind of like, blipping in and out, or it's also unpredictable in a way, but it also adds another quality that breaks you out of this... You're in something beyond just a music, you're in a whole immersive experience that is also showing these other visuals that may or may not be connected to whatever's happening on the music table.
[00:08:49.758] Vincent Morrisett: There's something a bit poetic in it, like we're playing with, there's a lot of trompe l'oeil in a sense that we give a bit with video mapping the illusion that the cubes are generating light. So we play with this kind of casting of shadows and perspective and depth. that is done through a kind of optical illusion in a way and then we integrate a bit of a figurative element like some kind of like arms and legs and since on all cubes there's this little face like you start to kind of create these kind of scenes and choreography of these strange characters. So we kind of flip from really simple, optical, light ballet, and then jump into these kind of figurative characters that dance to the rhythm that you're creating. And that was, again, for us something important that we don't necessarily tell a story and we want people to just kind of kind of dive and there's something almost soothing and meditative at one point about the project. It's almost like a musicotherapy. Like I stayed most of the festival outside of the booth just to kind of discuss with visitors. And most of them were like, either I want this table at home or this is the best thing I've seen this year. And also like just this kind of calm, appeasing, look on their face. And another thing that was quite cool is when two persons that didn't know each other did it, often we would see them continue to do the tour of the exhibition together. There was this kind of bridge, emotional bridge that we created somehow.
[00:10:37.425] Kent Bye: Maybe you could take me back to the beginning of this project. How did it start? Was it an idea? Was it a pitch? Was it a reaction to getting the fellowship at the Phi Center? Maybe you could just describe how this project came about.
[00:10:49.272] Vincent Morrisett: We had a carte blanche at Phi. It's always been kind of a an idea of working together. So we had the support of them and I was thinking of like what about their DNA, like the FI as an entity is a place, is a physical place, but that has been at the forefront of distributing XR experience, so that was kind of the premise of how I could bridge physicality and kind of a communal experience and play with what we've developed over the past years, like kind of build on that vocabulary of music, interactivity, handmade, crafty visuals, and just kind of bring this to new territories. And so this idea of building something kind of an XR experience without any headset was fascinating to me. So in a way that was the idea of the table, the kind of interactive table came quite early and for a long time the project was just called table. So the table was the canva or the theater and it was like, okay, what do we do with that table? So we played around with objects, little robots, our own hands. And then, just by trial and error, and just as an art residency, just kind of the luxury of experimenting and figuring it out, the pieces of the puzzle came together, and it was a collaboration with Edouard, who co-developed this crazy computer vision system, and Caroline, who designed all the mythology. the visual mythology of the project, and VLooper, who, it was his first take on interactive music composition, so there was this really abstract, like, what would we do? How does it work? And slowly, we kind of build our vocabulary. Okay, this kind of work, if we put objects there, and there's timeline, and we hit a beat, that's kind of cool, but it's not enough. If the beat is off, it's ugh, you feel kind of a discomfort. So all these little things kind of, So it's quantized to make sure it sounds good always? Correct, but we designed it so that there is a sense of precision. Even if you move a couple of millimeters, it has an impact and it will change. And if you stack cubes one over the other, the sonorities will change, like another family of beats, for instance. And if you group cubes together, there's another family of more heavy sounds, for instance. So the project was not a clear vision, it was more this intuition or a feeling of what could the experience be lived and then it kind of shaped up with the limitation and the possibility of that premise.
[00:13:40.025] Kent Bye: It seems like this would be a great piece to have in a museum because it's quick, it's short and simple, but it's also highly satisfying because I often talk about this concept of the predictive coding theory of neuroscience, which is essentially saying that your brain is a prediction machine and you're trying to understand and model what you're expecting in the scene and that this is an experience that hits that sweet spot of not being so simple as to be a tool and not being chaotic but in that middle part where it's just enough where you can understand some of the order but it's still unpredictable and novel enough to keep it interesting and engaging. So you're constantly trying to figure out as you engage with this piece how to like use your agency to be able to create something that's new and that it kind of unlocks these other surprising things that is not always necessarily clear what is connected to what you just did and what was already pre-programmed in terms of how the experience itself is unfolding.
[00:14:33.013] Vincent Morrisett: Exactly, and that kind of grey area is interesting because it creates something that is magic and I think you summarize it well to keep the tension between those two and I think in most of my interactive projects that's what you want to achieve, to kind of really feel that you're free and that your own personality is kind of reflected in the actual experience and at the same time as an author I want to kind of accompany them and bring them in some kind of direction, emotional direction. At the moment we present six minutes, but we have 25 minutes of content. So depending, like you've heard something, but you could have been four different time into what have been different sound chapters, if I would say. And here is with headset, but you could also have a a setup where two person or more are manipulating the cubes and we could have speakers overhead and other people could just watch it because at one point I often go and people will throw the cubes around and then don't touch at anything for like one or two minutes just zoning out and just kind of wow what did I do like it's amazing and just don't want to touch anything because it's too beautiful And then they start to mess around with the cube again, but you want to be, and then at one point you almost want to crystallize that moment. And I think the fact that it's satisfying also to watch is a bit like a twitch, like you don't necessarily need to engage to kind of appreciate it.
[00:16:03.210] Kent Bye: So you said 25 minutes of content but 6 minute experience, does that mean that in order to experience it all you'd have to do it like a little over 4 times and then, but even then you'd have to unlock things because that content's gated between certain actions a user has to do like either stacking or other things like how, if I wanted to experience all 25 minutes of the content, what would I have to do?
[00:16:24.697] Vincent Morrisett: You would have to come at a really quiet moment and then we put the infinite mode because it's not an unlocking mechanism it has this linear backbone of Scenes and then you kind of unfold sonorities and beats in these sound universe but you're not necessarily kind of unlocking or branching the thing so There's no real way to cover exactly the full content as a visitor in the six-minute slots. It's more kind of a randomness to it. But yeah, this is a particular context since there are a lot of pieces in the same venue. We don't want to parasite with sound.
[00:17:06.029] Kent Bye: I see, so you would take 25 minutes in the infinite mode, and then there would be a fairly linear unfolding of all those things, but still have the opportunity for the user to have the control to be able to modulate what aspects of that are being revealed.
[00:17:18.913] Vincent Morrisett: Yeah, totally. It is still really different from individual to another. When I go, I never get bored because it's so unique and beautiful to see how each person engages. Some are really analytic and do long lines with cubes, some are really chaotic. some individual will start a bit like this and then just let go and create chaos. So you see the familiarity like if you've done the project a couple of times, but still there is this expansive quality to it where it's like you really understand that, you know, if I just put two cubes on the table, like you really kind of say, okay, that this is kind of a really minimalist structure, and then add more, add more, and then you create this kind of orchestra.
[00:18:07.296] Kent Bye: Yeah, when I did it, I did that thing where I was putting him in a straight line, and then the sequencer line was going throughout, and then there was this context switch where then it was focused on one block, and then doing a line that was rotating around that, and so it was like almost changing the rules in which that you're operating, and that once you understand the rules, then it changes again, so it feels like there's a certain amount of not being fixed to just one dimension, but playing with both dimensions, And so I guess a follow-on question is that, like, when you're laying them down, are you using, like, pitch on the vertical axis at all, or is it because there's a certain rhythmic aspect when you have on the horizontal axis at the beginning, but what happens if you put things on the y-axis up and down and the vertical axis on the context of the table, if that's doing anything, if there's any positional information of where the blocks are, if it's more kind of one dimension or another?
[00:18:55.802] Vincent Morrisett: It's a bit of both. And that was the interesting part was that in the process of the residency, we kind of understood that having too many control or option made the people be in a mindset of like, OK, I'm having a sound software in front of me and then a bit like the Reactable that was used by Bjarke. in the early 2000s where you were controlling objects to kind of make guys a DJ. I didn't want to have the people in the mindset of like, OK, I will change the pitch, like sonorities change depending roughly where you put it on the table, how you stack the cubes or if they're in group or individual. But there's also another aspect that is not homogenized but that is kind of constant through the thing so like it hit a cube it does like a beat and then if you move it a bit the sonority will be different but it's kind of part of a sequence that sounds good so that the melody continues so it was kind of really finding the right balance and depending of the chapters some of the rules change a bit so that as you said like it continued to be interesting and surprising yeah so we took out a lot of the features that we could have tried to take out the intellectual pitch of like, OK, we could do this, this, this. At first, we were really excited about like, oh, we could do all these things. But then when we bring it back to the user experience, we're like, oh, no, we need to kind of make something where it's less cerebral and more emotional.
[00:20:38.275] Kent Bye: Yeah, and that's easier said than done, I imagine, because it's very easy to make it as a tool where you're like, oh, I'll turn this dial or move this block in this direction, and it's going to have this effect. But it gets into the space where it's a little mysterious, where I found myself trying to understand the algorithm of what was happening, and what can I do, and how can I control it, and how can I express my agency. But yet, then when I would maybe figure something out, then it switches, and then it kind of put me into that, like, well, I guess it doesn't matter what it is, I'm just going to act in a more chaotic or random manner in a sense not being so overthinking or deliberate about my movements but just throwing blocks around because I just want to hear the different variation and so in that sense as my expression my agency becomes more chaotic because I don't understand what I'm doing or why I'm doing it but it puts me into that more poetic mindset that is able to receive whatever is happening in a way that I can feel like I'm engaging with something but not overthink it and just immerse myself. So in that way, I felt super immersive and fun, but being left with this kind of poetic mystery about what I did and what it meant. And so talking to you and understanding like I didn't discover, for example, to cluster things versus putting them to individuals or just I did try stacking them, but I couldn't hear anything. at that moment and so it's a process of kind of doing these little experiments and trying to see what happens and see how you can interact with it without knowing what the rule set is and leaving it open-ended enough for people to figure it out and experiment for themselves but still at the end of it having a bit of not knowing what it was happening in a mystery but feeling extremely satisfied that it was a fun experience and so it's kind of like this interesting balance there that I've done a lot of immersive experiences and I don't often walk away from
[00:22:17.434] Vincent Morrisett: Experience is feeling that so I know that it's not an easy thing to do No, I'm really I'm really a pizza the second time that we're presenting the projects We presented it at the Phi Center in Montreal after the residency and this is the first public context where I have the opportunity to see people meet the people and it's a real gift to kind of being able to exchange and see how they felt about it and it is really this kind of marvelment about the Also this kind of, how did you do that? Why did you do this? But not in a cryptic or snub. I think it's a really approachable project that can go beyond language. A kid could really enjoy this project. So I think it's accessible, but at the same time it has its aura of magic and a bit of mystery and for me like this project is also kind of a Really good proof of concept of being able to expand into other Engagement with physicality and just imagining that now we have cubes on a table But we could imagine just a larger canvas a venue and instead of cubes it's human being in a venue and then the visitors are becoming the musical score and you're putting through light and visuals and this kind of play head triggering each person a sound so then it's this kind of new ritual about live music about how you engage with music how you are in relation with others and it's kind of a beautiful icebreaker like I'm a bit of a shy person and just having this kind of premise of just kind of oh there's this kind of thing this animation that put in relation to people that don't know each other it's kind of a beautiful I'm looking forward to kind of experiment with this kind of a larger scale kind of okay this could be a way of experimenting music in a venue like becoming the musical score
[00:24:19.644] Kent Bye: Yeah, it feels like after talking to you I'm a bit surprised to hear how linear the piece is and how authored that musicians would have the ability to create an arc or an experience but then have the user be able to still participate and have their own freedom to feel like they're participating in that composition. And as I think about this as a piece, when people are asking like, well, what should I see here while I'm still here at South by Southwest? I was like, composition, absolutely. Because like, this is not something that's going to translate well into like a virtual medium. I was just thinking about why is this a piece that's better in physical reality versus like, I think there's aspects of this piece that could work in virtual reality. However, Two big things is first of all the haptics of actually grabbing the blocks and feeling the blocks and placing them in a way that there's something about that experience that the lack of haptics or even if you have like a haptic glove I think would still be maybe a little too clumsy to give me the experience of actually picking up blocks or a handful of blocks you know and to like move them around. But also the quality of the light and the projection map is something that is difficult to recreate with an immersive experience in the sense that first of all the table is big enough that my field of view is pretty much fully taking up with what I'm seeing and I'm not seeing any occlusion from my field of view but also the quality of the light is something that is difficult to explain virtual light versus actual light that I'm seeing. I can still perceive a difference between I know when I'm in a VR experience and I'm seeing light being represented versus actually seeing light being reflected. And there's something different that I don't exactly know all the physics or what exactly is happening that's different, but it definitely looks different and more precise. I don't see any pixelation or more vibrant or luminous or, you know, there's something about seeing a sunset or sunrise versus taking a picture of it where there's something about the quality of light that's lost. So the quality of light but also the haptic experience of this feels like those two things together make it something that's much more viscerally immersive than something that would be mediated either through AR or VR.
[00:26:23.797] Vincent Morrisett: And also like seeing the other person physically and seeing the other hands and your own hands. Like we're playing also with projection of hand, arm, silhouette on the table as a part of the visual vocabulary. And it becomes a bit of a choreography of four hands. And I think the hands over the table also is part of that choreography and also of the animation of the movie, if we would describe it. as a linear piece. But yeah, I agree, I wouldn't think, and it was designed, like that was the main purpose and probably the first original seed of the residency was like, how can we think of experience that could live in the headset? enhance it in a way that you have the quality of the interactivity, the reactiveness and the magic, but then make it even more special or take advantage of the physicality of the real world and just kind of be at that sweet spot of the best of both worlds. And that's what is interesting me in this kind of mix, I don't know if it's the right word, a mixed reality or AR, I don't know what would be the good term, but I like that idea of blurring lines between the two.
[00:27:45.195] Kent Bye: Yeah, mixed reality experience or projection map is how I tend to think about these types of experiences. So what's next for Composition?
[00:27:52.636] Vincent Morrisett: This is an opportunity to meet people and to make the project travel. Obviously, like we want to have the Composition be seen by other people. And as you say, it's a physical installation. So we're looking for different contexts. And I think it's a context that can live in venues or museums that are open to a large, broad audience. but it's kind of a bit as the theme itself. I think it's a piece that could be in different kind of concept like science museum, fine art museum. It has a kind of a broad spectrum to it and again to come back to this idea of a venue like I'm continuing to think and see how the project could be adapt and there would be Like some work to do about human behavior dynamics that are completely different from one person controlling the cubes having a bunch like yesterday we tried to go to a restaurant like and we were ten people and Just trying to know where we would go and what we would eat. I was like, oh my god. Yeah humans It's difficult to kind of a crowd of person, but since the project sounds good, whatever you do, I have a good feeling that whatever would be ever, we could pull it out. But for me, it's interesting to see how we can add a bit as a maestro like be able to orient a bit or give some cues to the audience to do certain aspect of thing. Of course there will be some rebels and we know about this but then play a bit with how a crowd could behave and play with this and I think there would be some kind of excitement about be able to kind of move around and do stuff and have a direct feedback on the music that we collectively hear. And I think it could be a post-pandemic kind of interesting experience. Like a lot of people I met this week were like, oh, this is the first live show I've seen in ages. And like, they're just like, oh my God, this felt so good. So I think the next year or two will be kind of rediscovering the pleasure of collective experience. So I'm excited about that.
[00:30:13.404] Kent Bye: And finally, what do you think the ultimate potential of virtual reality and these types of interactive experiences, interactive stories might be and what they might be able to enable?
[00:30:24.731] Vincent Morrisett: I think it's again like for me is just to have a way like to refine and to get more and more closer to natural behavior. I think the headset make it kind of react to the way we were just looking in a natural environment and that kind of mimics that. So that aspect It is now figured out and for me now it's kind of how to the rest of the engagement that is kind of interesting to me to kind of okay how the human body, how our gestures, how the way we behave can also be having an impact on the environment or the situation we're in front of. So I think there's still a lot of room for this kind of experimenting and developing project where the interactivity becomes more and more refined and natural and intuitive and less about kind of, okay, I'm triggering an interface like a tongue with a remote control or even your hand, but in a bit of a goofy way. I think this is what excites me, like just kind of making it more and more natural.
[00:31:36.637] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader Immersive community?
[00:31:41.859] Vincent Morrisett: It's an exciting time, I think, to do things. I think we need to continue to be curious and I think it's also cool to kind of expand beyond our own universe and create bridge with other community medium and do interesting collaboration. the community will win to kind of infiltrate other medium so that it becomes a way of engaging with content.
[00:32:16.484] Kent Bye: Awesome. Well, Vasant, thank you so much for joining me here on the podcast. I really enjoyed the composition. And yeah, just I feel like it's one of these pieces that, like you said, it brings back that physicality and the collective experimentation of what you can do in physical reality with other people co-located in the same place and time. And yeah, I really appreciated the piece and the opportunity to be able to break it down a little bit with you at the South by Southwest. So thank you so much. Thank you again. Bye. So that was Vincent Morissette. He's from Montreal and a director and founder of ATOAA, A-A-T-O-A-A, and doing lots of different interactive experiences for the last two decades. And the piece that was at South by Southwest this year is called Composition. And again, one of my favorites from the show and the exhibition, just again, like it was the type of piece that you can only really have while you're physically there. and just a great exploration of projection map, interaction, agency, interactivity, and this process of emergent music composition. And again, it's a very linear piece, but you give this agency to be able to really modulate and control it in a way that you can really leave your signature and personality on the table. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you could become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
