The WebXR Design Summit was a 9-hour series of talks about immersive design & experiential design that was organized by Ben Erwin and his WebXR Polys Awards team on October 12th. I was the host and moderator for the day, but also helped to curate a few sessions including a fireside chat with two immersive design professors Doug North Cook, Program director & Assistant Professor of Immersive Media at Chatham University, as well as Robin Hunicke Game Designer and CEO of Funomena as well as Full Professor at UC Santa Cruz, and the founding Director of the Art + Design: Games & Playable Media program.
I wanted to hear about some of the latest trends of how they’re teaching immersive technologies and experiential design, and a common theme was interdisciplinary collaborations. North Cook was talking about a Design Dialogue practice that was sort of like a telephone game where an architect would create a blueprint, which was then created into a 3D model, transformed into an immersive audio file, turned into a water color painting, and then created into an immersive VR experience. The goal was to see what kind of design practices translate well into other modalities and affordances, but also to escape the normal production pipelines for each media.
The theme of interdisciplinary collaboration also showed up for Hunicke, who announced that UC Santa Cruz’s Art + Design: Games & Playable Media Department is creating a newly shared department with her colleagues in Theater Arts that blends game design, game production, game art, with dance, theater, performing arts, dramaturgy, and critical practice to create a shared hybrid curriculum combining the affordances of play, performance, design. She mentioned that A.M. Darke is creating an Open Source Afro Hair Library, micha cárdenas is looking at embodiment & performance in terms of environmental challenges, Ted Warburton is looking at the combination of the environment, dance, and teleawareness, and dramaturg Michael Chemers is looking what insights Japanese Puppet Theater can provide to the location of embodied performances of puppeteered avatars and virtual beings.
North Cook and Hunicke also talked about what insights that web design and WebXR could provide to the overall design practices for immersive technologies and experiential design talking about aspects of telepresence and the fusion of lots of data into experiences, but also the uncertainty around the economic sustainability of the open web and what the new models to support a diverse and robust ecosystem of immersive experiences.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So last Tuesday was the WebXR Design Summit. That was over nine hours of content featuring a number of different half-hour panels and 15-minute talks. It was produced by the WebXR Awards. They have the Poly Awards, but they've also been producing a number of different summits. They had the Developer Summit, they had a Business Summit, and this was the Design Summit. So I was asked to moderate the design summit, but I also helped to curate a number of different sessions, including this one that I'm going to be airing on my podcast, which was a couple of immersive design professors, Doug Northcook at Chatham University, as well as Robin Honecke at UC Santa Cruz. And each of them are teaching students around the principles of immersive design and experiential design, coming from each of their different backgrounds and disciplines. But I wanted just to get their take in terms of the overarching trends that they see within the academic spheres with teaching students immersive design. And what ended up coming up was a lot of themes around interdisciplinary collaboration and the fusion of lots of different design disciplines. Well as what they thought in terms of how the web and web design is going to potentially start to influence the larger Immersive design and experiential design, so that's what coming on today's episode otherwise the VR podcast So this conversation with Doug North cook and Robin Hanake happened on Tuesday, October 12th 2021 so with that let's go ahead and dive right in and
[00:01:34.780] Doug North Cook: I'm Doug Northcook. I'm the Program Director and Assistant Professor of Immersive Media at Chatham University here in Pittsburgh, where I run an undergraduate program focused on immersive technology design. I started working with VR when the Oculus DK1 showed up at a friend's house back in 2013, and I put it on my head and I was like, this is really weird. We should just do this now. So I stopped doing all the other stuff I was doing back then, which was mostly like web development and some real-time 3D work, and I've been doing that ever since.
[00:02:19.465] Robin Hunicke: I guess I'm a game designer and a CEO now. I've been running my own company, Phenomena, for nine years since September 26th, which is technically we call it awesome day. It was the first day that we were ever given actual funding to make our own actual things and weren't living off of our savings. co-founder Martin Middleton and I met while working on Journey and then decided to build a game company that was as joyful and diverse and inclusive as the titles that we like to make. So it's actually been really an amazing process going from just the two of us to now over 30 people. We have done a sort of unusual thing. We bootstrapped the business but decided to focus on building technology for new platforms, games for new platforms, basically, because we realized that it was going to be kind of difficult to keep getting permission to do the weird thing on the standard gaming publishing platforms. And so we moved initially into VR, started working with Oculus when it was When it didn't even have hand-track controllers and then shipped Luna on the Oculus, through our research with hand-track controllers, really got into that notion of embodiment. And I've actually created several games for that platform, two of which are currently unreleased, but hopefully released this coming year. And then, in addition, we worked on Wattam with Kid Takahashi, so making games about connecting and creativity and thinking about how just new technologies enable new ways of connecting. And so, in the last year, we've actually done a lot of work on the newest platform in our portfolio, which is Roblox. So we're spending a lot of time thinking about the way that people can play when they're also engaged as creators and the ways that you can build experiences that cross a variety of brand experiences, new creative experiences, new IP, educational experiences. creativity platforms essentially inside those environments. And my ultimate goal is that eventually I'll be able to play one of my Roblox games in VR. So we'll see how that all works out. But I'm a lifelong learner. I'm a Buddhist. So I'm just going to do this. We can all just hold some space for having a conversation and being able to be with each other virtually, even though we're not together in person, which is much more fun. So, yeah, that's me.
[00:04:38.032] Kent Bye: Awesome. Well, this is the WebXR Design Summit. And when I thought about the designers that I look to and always enjoy talking to, you two immediately came to my mind and are both engaged in teaching and taking stuff that is at the frontiers of not really having things settled yet, but yet you still need to somehow convey to students these ideas. And I know Doug, you had a number of different people come in and give a series of different seminars. I know I gave a lecture there at Chatham University. And so maybe we could go into the theory of design and immersive design and experiential design. And what were some of the things that you took away after listening to all these different immersive creators come and give a speech about the frontiers of experiential design?
[00:05:21.839] Doug North Cook: Yeah, so we ran this new course at the university last spring, which was intended to be an open public lecture series, but COVID had its way with us and we pivoted it and made it just on ground for our students, but I still brought in all of the same people. which created a pretty unique experience for them to have these really intimate encounters with people who, you know, are kind of working out at the edges of this kind of work. But the intention of that seminar was to bring people not just who are doing virtual reality games or even working in VR at all, but to bring designers from different disciplines that can speak into that work. So that meant bringing in architects and sculptors and theater designers and also a couple of VR people as well, but really trying to understand that, you know, when we're creating virtual reality experiences, we're doing exactly that. We're designing an experience. We're not just creating a piece of software. It's not It has this different capacity to put people in a place and to create memories and to create connections. I know a lot of software developers, and I wouldn't say that all of them are the best people at doing that work of building connections. I think there are certainly developers who are doing that, but people that have a long history of doing that exist in other disciplines that have this rich, deep history of designing for the human experience. So a lot of the work that I've been doing and a lot of the work that I do with my students is, how can we go out and find these other places where the thinking already exists? And how can we bring it in and adapt it and reuse it and remix it? in a way that's really going to make the experiences that they're working on feel really connected, really grounded, and really impactful. And I've been seeing that with my students. We did this great research project over the summer where I got some funding for some of my undergraduate students to work on a design dialogue project where we had an architecture student and an immersive media student working on the same projects together and just handing designs back and forth from their medium and having to retranslate them back into other mediums. So taking an architectural drawing and turning it into a 3D model and then turning that into an audio file and then turning that into a watercolor and then turning that into a VR experience and doing that like dozens and dozens of times over weeks and weeks and And the work that came out of that was like nothing that any of us had seen because it was filtered through all of these different artistic and design mediums, which normally in the development process you don't have the luxury of daydreaming for weeks on end and doing watercolors. So yeah, so that's kind of some of what we've been getting into. I mean, I think there's an exciting opportunity in academia to let students like really create and really explore the boundaries of what these technologies can be without, you know, not that we want them to go so far out that they're not prepared to go into the workforce in a meaningful way, but that they really do have the freedom to do some of the work that professionals don't get to do. And I think that's been really exciting to watch.
[00:08:44.729] Kent Bye: Yeah, that's true. And Robin, I know that being not only on the frontiers of making video games and all these new emerging platforms, but you've also been involved in helping to create some game design theory from the mechanics, dynamics, and aesthetics. So the mechanics, a lot of the different rules, but also the different aspects of your solving puzzles and mental social presence, the dynamics, a lot of your agency and active presence, how things change over time, and then the aesthetics with the emotional core. And I know that you've centered a lot of things within the emotions first and then build out everything around that. And so I'm wondering if over time, you've also started to add in elements of the environment and embodiment into this overarching framework of game design, because it seems like game design is going to be blended with other aspects of architecture and theater and cinematic storytelling and all these different disciplines. And so, yeah, I'm just curious to see how the MDA framework, so mechanics, dynamics, and aesthetics, if that's been evolving and what kind of things you've been seeing at the frontiers of teaching your students these immersive practices.
[00:09:51.230] Robin Hunicke: Yeah, actually, so, you know, it's relatively exciting to be able to announce it, but we just collaborated on a huge departmental shift for arts, games and playable media at UC Santa Cruz. So after operating the program for seven years independently at the level of the division, I've actually formed a shared department with my colleagues in theater arts. And so we have a new faculty now, which is a blended faculty of dance, theater, performance, dramaturgy and critical practice that's founded in both games and theater and performance arts. And so we have already now the undergraduate major in game design and game production and game art. And now we also have this undergraduate major in theater arts. And we're basically creating a new hybrid shared curriculum to engage with both the undergraduate and graduate programs that we both sponsor, that's really going to be looking at the affordances of play, performance, and design as we create these new technological spaces. So I'll give you a couple of examples of where that research is headed. We have people that are working on open source libraries for African-American hair cells, you know, AM Dark. We have people working in the space of embodiment and performance in terms of environmental challenges, Misha Karanas. We have faculty that really look at embodiment and dance and actually also tele-awareness So, you know, people who have engaged in performance that goes well beyond the space of the individual dance theater. Ted Warburton, my colleague and also the acting chair in my division for many years recently, is a dancer who engaged in one of the first collaborative hybrid practices with the dance company in another location through telecommunications and video. And so what we're looking at is kind of exploding the notion of performance and performance spaces. So we have a black box theater. that we're currently gonna do a capital fundraise for to advance and be able to include cameras, virtually racted, essentially, actors and performances through green screen that then can be projected into a theater space where the audience can engage with the theater. We have Michael Chemers, my chair, is a dramaturg who works on looking at how ancient performance styles like Japanese puppet theater really help us reposition ourselves in terms of thinking about what is really a performance and who the performer is in a performance. You know, when you're sitting in all black, silently puppeteering with three other people, like mom and shunts, essentially, you know, is the same thing. Who is the performance? Are the actors the performance? Are the puppets the performance? Where does the performance live? And what do we choose to see and not see? So we're really trying to take all of those aspects of looking at critical race theory, thinking about feminist pedagogies and deconstructing the notion of performance and the performance of self. and building out new platforms for engagement. So my own work in Roblox and in AR performance is really also engaging fundamentally new curricula. So I just came from a curriculum meeting for a new curriculum that we're proposing that is aiming to create a safe space for people that want to make a difference in technology and the way that it engages with people that are typically underserved. So looking at the ways in which creative technologies perform on us through the curated flows and the things, the games that we play and the things we see on our Instagram and our Facebook, and also how we perform ourselves in those spaces, like when I'm on TikTok and I'm presenting myself or I'm reenacting something or I'm reinterpreting a piece of work, how is that creating a sense of me, a sense of us, a sense of our identity as a collective? I think that games are really an interesting part of that space, but they're a small part and they're actually relatively new. So what I've been doing a lot is just thinking about how ancient technologies, like the stuff that Chalmers has been studying as a dramaturg for 20 years, can inform my own practice and how the way my younger colleagues who are really working at the center of race and ethnic studies and feminism with respect to these media can help improve my ability to sort of advocate for change and position different bodies and different performances of self in the space of AR, VR, and more traditional gaming. So I think it's a really rich time for students. I think you can really take these things in almost any direction and make a positive impact that really changes the way we see the media itself.
[00:14:15.047] Kent Bye: That's really amazing and it's really exciting to hear because it's kind of a further elite.
[00:14:21.333] Robin Hunicke: We're hiring a tenured professor. So if you're out there and you're tenured and you want a really great job, it's Current.
[00:14:28.118] Kent Bye: Yeah, well, it's the blending of all these different practices together that I think is really at the frontier of what I see just generally happening with these immersive technologies is this interdisciplinary fusion between all these different practices. And I know, Doug, we've talked previously about practices of industrial design and ergonomics and architecture in general has come up again and again today, just generally as another practice that is about the designing of space. It also has been a very interdisciplinary fusion of all these different practices and is putting in more levels of interactivity and agency. And so, you had started to mention this blending of these different modalities and almost playing a telephone game between starting in one place and going in and see where you end up. But I'd just be curious to hear Generally, if you see a similar trend there at Chatham University of this getting the different departments to start to collaborate in some ways. And what I think of as kind of the spatial medium of XR is almost like the lingua franca of getting boiled down to the human experience, but all these different design disciplines that are kind of feeding into that. And what you're seeing from your perspective at Chatham University when it comes to that
[00:15:37.765] Doug North Cook: Yeah, I mean, it's interesting. I think we've, you know, we brought on two more full-time people into our department this past year, one of whom is mostly working on open source tools, mostly around volumetric capture and some VR design practices, and the other of which, Joseph Amodi, who's an amazing activist and designer, most of their work is focused on doing healthcare activism, like focusing on vaccine hesitancy and some of these things. So part of what I've really tried to do is to encourage faculty to come in to our program, whether it's full-time or part-time people who are already taking an interdisciplinary approach and are practicing in spaces that I'm not necessarily connected to or working in. I don't really have a foot in healthcare or in doing that kind of work, but But then, I mean, you know, my colleagues at the university, part of the magic of working at a university is that you get to work with all of these people in a way that you don't get to do if you're working at a software company. You know, like my colleagues are architects and biologists and poets and social workers. And they come from this huge range of disciplines and backgrounds and experiences and ages. And our department is the newest, the youngest. You know, and so we're trying to do a lot of new things. We're trying to do a lot of new things very quickly, and we've been there for about four years now, and we're at the point now where a lot of my colleagues have started to show up, and they're like, what is this? How do we do this? Which has been really great. So we've had some really exciting interdepartmental collaborations that we've already done. We have new ones that we're talking about with the sciences, looking at science education, have a really great historic women and gender studies institute at the university which Chatham was founded over 150 years ago as a women's college and so we have this long history of feminist politics and thought and philosophy that have kind of been born out of the university and so we've taken a lot of that into our department and We have an ongoing artist residency that we run now that's a collaboration between the Institute, the art department, and us, where we provide full funding for an immersive artist from an underrepresented gender identity to just come and get paid to produce work. with no restriction from us. We don't tell them what to do. We just give them space and equipment and mentorship. And, you know, we're in this unique position, both of getting to kind of push where the conversation goes, but also create opportunities for people that, you know, maybe Adobe or Google wants to give them a residency. And that's cool. And they should definitely do that. But that comes with a lot of strains. You know, you can't necessarily produce like really out there queer art if you're doing it under the guise of Google. So we've really tried to create that space, but also, you know, really trying to bring in the arts, but also trying to invite the sciences and our faculty in healthcare who have gotten really excited about these technologies. We actually put together a dual degree program where students from our program can go right into a one-year master's in healthcare informatics, which is like healthcare data science. where you could go and work on really complex healthcare software and data problems, if that's what you want to do, right? And trying to forge some of those connections. Yeah, but that's why I think seeing what is possible with these technologies at the university level is so exciting because you get the chance to kind of do little prototypes and experiment with how these technologies can interact with different disciplines in a way that's just, it's almost impossible to do as an individual company because you have to build those connections with those industries. But when you're at a university, you already have those people, you already have scientists, you already have actors and theater designers and industrial designers, you know, all of that already exists. So it's been really exciting. And I mean, you know, realistically, at this point, there aren't very many universities still that are doing anything beyond, you know, a few courses And it's exciting, I think, to be engaging with this at the degree level and bringing in research faculty and really kind of letting these things take on a life of their own.
[00:20:01.906] Robin Hunicke: Yeah, also the libraries could get really involved now. Like, you know, the University Librarian at Santa Cruz, Elizabeth Cowell, has spent the last five years doing capital fundraising for digital research commons that all students have access to where they can play Luna, you know, or any other VR title on a headset that they just check out and take into a room. There's a green screen there. There's a wall, a data wall, you know, they're currently building out a really fantastic fabrication space in the science library. And so it's like, She had the foresight to see that libraries need to engage on these platforms and give people who wouldn't normally even consider majoring in games an opportunity to create experimental content for classes that can tell their stories in a better way. We had a dean's student recently that was awarded for creating a game maker title that has essentially helped her refine her arguments around where food deserts come from and what they mean for people of color in certain urban environments where you're wandering around and you're engaging in the practice of planting seeds and then harvesting plants and eating those things and sharing them with your families and your friends and looking at what are the systemic issues that come from living in a place like that. And that was out of a class that was taught in a college system by AM, you know, where most of the students would never have considered using gaming technology to create a more engaged argument or an engaged discussion about these, these issues. And so I think when you look at the university context now is the time as a, as a teacher, wherever you are, like, you know, Christine, I really appreciate, like, we need more perspectives on these locations of activism and coming to terms with the cultures that we've created inadvertently by innovating without really looking ahead. And so I think it's it's a really great time to be in the system of a college, a university, whether it's a private school, a public school or a community college. There are so many ways to create access and then share that access in a way that I think will really revolutionize the kinds of content and opportunities that we see in the next 20 years.
[00:22:11.759] Kent Bye: Yeah, it's really an excited time. And I know that both of you are really at the stage of a lot of the new stuff that's happening for the first time in a lot of cases. But I wanted to pull in some of the different aspects of WebXR and the web, because most of the stuff that's happening in the immersive industry has been in Unity, Unreal Engine, which are at the heart game engines. They're not designed to be data visualizers or pulling in different aspects of open web APIs or pulling in stuff from the web. And maybe Robin, looking at something like Roblox, which has different aspects of, even though it's still an app, there's still some aesthetics and interaction designs that are still kind of in that 2D mode of, you're in a 3D world, but there's still a lot of 2D affordances that may not actually translate over into VR if you were to do it one-to-one. And so it feels like Roblox actually could be one of the spaces where you kind of have this fusion, where you're pulling in aspects of chat or accessibility options. I'm just curious to hear from each of your perspectives of how you see what the web and web designers have to provide this larger immersive XR design disciplines, which if all the different disciplines are probably the things that's going to be, you know, not at the frontiers, but pulling in, but still like, what is it about the web that is going to change the different design practices that we could see with where things go with WebXR.
[00:23:35.864] Robin Hunicke: Yeah, I mean, even as you look at, say, the pivot that Magic Leap did, right, they went from basically being an entertainment content company to a working tools for productivity and access company, right, in their pivot over the last year, right. They brought in this new CEO to re-envision what does a headset like the Magic Leap that's going to still be cost prohibitive for an everyday consumer, what does it need to do in order to perform in this space economically? What I think is so interesting is that there is an economic reason to do that pivot, but there's also a fundamental reason, which is that we're all working from home now. And like, it sucks. Like, you know, the 2D screen environment is exhausting. You know, even just the notion of space and volume of voice and the ways in which we position ourselves in three space and a co-working space versus when we're all in like, I mean, I'm in meetings sometimes at school, there's like 20 people on the meeting or 30 people on the call. And what you see is the hand raising and the chat and like, it's just so difficult even to teach your class. Like one of my classes is 180 kids, you know, it's like, how am I gonna teach 25 teams of five how to be making video games together when I can't walk around the room and play their games, right? I really work to innovate on the classroom side and then losing the classroom and then having to go into a screen has been really difficult because you just get so much more data when you can sit down and play the game and talk to the team. and feel it on the controller or when you're playing a role-playing game and you see how everyone's behaving at the table. It's just so hard to get that out of the screen. So I think what WebEx and UX designers really have an opportunity to do right now is to be asking themselves, what aren't we thinking about and how can we recreate these flows, embodiment, support the idea that a certain number of people are not going to want to have their camera on. So what are the ways in which they engage? that feels embodied and part of the conversation, but also gives them the space to not be doing this. I mean, some people just don't like to look at themselves. You know, some people don't have the privacy of a home. You know, you see a lot of people in my community who have kids, their screens are blurred out and then suddenly it's a child's face or a dog, you know, or a grandparent, you know, a caretaker. the idea of really thinking about what kinds of questions can we be asking ourselves about the affordances that are in this space. And then if we do get to a place where I have a headset that's comfortable that we can wear, when do we need to be using it fully? When do we need to be really up and about doing work? I think the thing I miss most about collaboration and my classroom spaces and my office right now is overhearing and being able to become part of a conversation that I want to become part of. It's so hard. to have a party online. It's so hard to get a casual private conversation going between someone about a third person in the room. It's like 90% of our socialization and learning comes from having a dialogue about other things that are happening in this space. And you can't do this in this medium. And so I think really paying attention to where we feel fatigued, looking at what we miss, You know, I just went to see Crangvin at the Red Rocks in Colorado, and they did a seven minute jam that was a tribute to Houston hooks and rap music. But they started it off with Elton John, Benny and the Jets, the first opening chords of that song. And when you're in a concert hall, and you hear those coming at you at full volume, and they're shaking your body, you're like, Damn, Elton John was good. Like, he knew the venue, right? He knew the embodied feeling of hearing those chords on the keyboard, right? And we've created all these spaces for so long to be able to experience that, and now we're trapped in this, you know? And so I think it's like, we really have to be asking ourselves, like, fuck, like, what are we missing, you know? And how are we going to get that back if this becomes the new normal as it seems to have done?
[00:27:28.958] Doug North Cook: Kent, you trapped us here, you did this.
[00:27:32.785] Robin Hunicke: I mean, we're starved for this, right? Like, it's so great to see you, but I really wish we were having a beer at GDC, you know? I really do, you know? I'm not gonna lie.
[00:27:42.492] Doug North Cook: But yeah, I mean, I think with the web, you know, the web is in this interesting place where it's become so cannibalized and like moved in so many directions, right? Like we saw this with mobile. The real question is like, what even is the web now? I think when most people are talking about the web, they're talking about like things that I access through my web browser. It's like, cool, but. Really, I think what most people are actually talking about is all of the data that exists behind all of that. And most of that is just these massive databases, right? Whether it's a social media platform or Amazon, but all of that doesn't just exist on the web anymore, right? It's getting delivered on mobile apps, tablet apps, native desktop applications, and in some cases now, immersive applications, right? So there's always going to be space for flat apps. Right? Like no one's going to want to just like read a book in three dimensions, right? Like, what does that mean? You know? So really, I think for web designers, and even, you know, when we're talking about like web VR and some of the really great work that the team that's been putting A-Frame and some of the other really great frameworks together have been doing, right, is what is the role of the web browser as we increasingly are not using the web browser to access most of our content, right? Like if you look at the overall data usage for the average mobile phone user, the majority of time is not spent in the web browser. It's been in dedicated apps, accessing content that can also be accessed on the web. Right. And not all of those things are going to translate into what we would think of as an actual true immersive application, right? Like TikTok. is a great phone app, right? It would be a terrible VR app, right? Because it's just scrolling video. I don't want to watch scrolling video in VR. Maybe I do. Probably not, you know? So I think it's really thinking about what are the things that exist on the web now that would be better in an embodied and three-dimensional experience, right? I think the team at Shopify is doing amazing things, building these really innovative VR and AR tools for e-commerce, right? And that's an easy thing where it's like, oh, I want to buy an object that is three-dimensional. It would be great if I could go and look at it and walk around and maybe see what it looks like in my house, see what it looks like in a different house. Maybe I would like to buy right like there's so many of those experiences but you know there's lots of things that shouldn't be moved to an immersive application and some things that just are going to be a lot longer to figure out but. But I think, I mean, realistically, you know, if you're still a web designer and you're, and you're making a good living doing that, that's cool. I don't know how long that's going to last. You know, we've seen this massive shift in the last 10 to 15 years of the majority of, especially backend web development is getting outsourced to every other country in the world. That's not in the U S because everybody has really great high speed internet and they are as good or better than we are at most of this work. Right. So.
[00:30:45.509] Robin Hunicke: There's also the problem of the curation, right? Like I was reading an article from a colleague of mine who is essentially like a growth architect at larger companies. So she's worked at places like Etsy, you know, like trying to think about how does the product grow as it becomes more of a product that people use online. And she was sort of saying, like, we've all had the experience of using a web technology that seemed like a service and now just feels like it was designed by robots, like it's not for humans to use anymore. Like, you know, the other day, my phone replaced Google as my main search link when I type in the Google search bar at the top of my phone. So suddenly I was in Yahoo search all the time, and I was like, wait a minute, what happened to all my searches? Why are they all bad? The volume of data that goes through Yahoo versus Google is so small that the searches are all borked. It's just like, what's going on? It's really like my phone was not made for me. It was taken away from me by the apps and the flows and the curation and the desire to drive traffic. based on agreements that I did not participate in. And I think that that's the other thing to really be thinking about. It's like, how are you providing an authentic experience to people, one that really does serve them, where they're getting what they need and not what you think they should buy? You know, where not everything is a ranked, stacked, order, search, SEO battle. And I think that's going to be a big struggle for people working in straight web and where entertainment content can kind of be a source of relief.
[00:32:11.061] Doug North Cook: And I think it's the big, one of the biggest questions with immersive technology in general is how is it going to be affected by the rapid decline of web ad sales and profitability? Who knows what's going to happen, right? Is everybody going to wear AR glasses in 10 or 15 years? Maybe. But if they do, there's definitely going to be a really difficult economic structure that's going to be built up around it, like the web, right? Like Google searches are not curated. They are bought, right? Like your placement and your success, you can just buy it on the web. Currently, that's not quite the same for any closed ecosystem like the PlayStation or the Oculus Quest or any of these things, right? But inevitably, that's going to have to shift dramatically.
[00:33:02.353] Robin Hunicke: In some ways it kind of is, right? Like, I mean, you know, in the end, like the access to those platforms, it continues to shrink as larger and larger service-based games eat up all the space. You know, when you look at the top 10 titles on a PlayStation or the Xbox over the last 10 years, There has not been a lot of movement there because those are evergreen titles now. Like The Sims, they've learned that expansion packs and community engagement will keep you at the top of your game for a very, very long time. And so how are we innovating? When you look at Roblox, it's essentially like a mall. You know, you walk through it, you have conversations with your friends about what you're doing. You know, it's like it's kind of like going to Disney World with your own fake money and then having a good time with your friends. And that feels really fresh, you know, in the same way that TikTok felt fresh compared to Instagram or Facebook. a couple of years ago, as those platforms become normalized and then competitors proliferate, what are we going to be doing in the future? I'm really actually quite curious to see where the web goes because it's getting a little bit oatmeal right now.
[00:34:00.242] Kent Bye: Yeah, I have one last conceptual question and maybe we'll have some final thoughts at the end. But the thing that I come across again and again and again when I talk to different design disciplines is that there seems to be some competing approaches. Let's say like the game design approach, rapid iteration that you have to continuously play test and you don't really know what the reaction is going to be until you get it into the player's hands. And something very similar with say web design, which has a very scrum iterative approach, minimum vial product. this agile approach, which is very iterative. Now the opposite approach of say, like making a film or building a building, which is much more of an architectural process that has pre-production, production and post-production that is much more linear. And there seems to be this combination of aspects of storytelling, which come from that, or building spaces and architecture, that's being blended into this more iterative approach. And so I'm curious how you've started to see these almost competing design philosophies of rapid iteration versus designing everything out beforehand, how you see those two worlds clashing together from architecture and design, theater, story, and game design.
[00:35:08.007] Robin Hunicke: Oh, God, I have so many contemporary experiences right now and working in Roblox, where we're working with partners who are traditionally in print media, you know, people that are traditionally doing activations in stores, for example, you know, and when you're working with like a fashion brand, and you're communicating, like, how does a game get developed? And when does it hit its peak viewability? And like, what is a rough look like versus A layout for an advertisement might just be the photograph without treatment and some tag copy that you're working through. It's not that you're literally building the models that are going to be showing the clothes in the picture. What I see is that there's a huge opportunity for people who understand digital content creation and rapid iteration to move into industries that were currently closed before to these kinds of perspectives, because they just didn't work in Roblox. Even a year ago, if you had said to somebody, one of the most successful experiences in Roblox is going to be a Lil Nas X concert, and the next most successful one is going to be a Gucci experience where you walk through a garden and then buy a purse, that you wear on your character outside, like literally no one would have believed you. Like, I know because I was fundraising for it at the time. No one believed me. And now it's like, there's just this huge gap. And like, there are going to be so many industries moving into the space of digital content, engagement through VR, the metaverse, this kind of thing. You said the metaverse two years ago while fundraising, people would literally lock you out of the room. They were like, whatever, you know, have a good time with that. Sounds really cool. So is virtual currency. And now it's like, everything is NFTs in the metaverse. So like, I think you're really looking at, if you're in this space as a student, there is a huge opportunity to break into an industry like fashion or cars, you know, where you could be building content that would be like literally taking Ferrari from nothing to amazing in the metaverse in the next five years. I think it's actually a really great time to be in the space and to have the access to the tools that you have.
[00:37:02.797] Doug North Cook: Yeah, I completely agree. I think, you know, I think that's where, like, it's been really exciting seeing what my students want to do with these tools. So, like, I had a student who built a fashion experience last year. I have a student who's building a tarot deck. that's a VR and AR app that also has a physical deck, and it's taking physical products and putting them into digital spaces. And I think there's what Robin's talking about, which is people from these disciplines going into these other places. And I think that's what is going to upend a lot of these, what we would consider to be like more auteur disciplines, like architecture. And I saw you had Andrea on a panel earlier today. who I think understands this better than anyone, that the ability to iterate with architecture when you can put people inside of a full 3D embodied rendering of a space, that is something completely new. And if the auteur architects really understood what that means, which some of the really great firms do, that they're already doing that, right? Like Zaha Hadid Architects has been doing all of their rendering and most of their work in Unreal Engine for like the last three or four years, because they know that that's where they need to be. And they're one of the most cutting edge firms in the world. And they're really digging in and doing that. And I think the people that are going to last this transition, I think, are the companies that are going to not just embrace immersive technology, but also companies that are making really significant contributions and moves on the environment and on climate change that, you know, it's not just about embracing new technologies and new ways of working, but also embracing the realities that we just can't keep living. in the way that we used to and immersive technology and all of this brings up so many interesting opportunities to, you know, solve some of those things. I mean, like Oculus's big tagline for their developer conference like two years ago was defying distance, which seems very strange now in hindsight, right? That like, it's like, oh God, like if only these tools were like five years more mature two years ago. Maybe we could have been doing this exact thing in an embodied VR space and not on the web.
[00:39:19.645] Robin Hunicke: like a traditional way. It's actually been really amazing. I mean, I know, Kent, when we first started talking, I read The Uninhabitable Earth in like 2017, you know, and went through this very long and involved process of introspection about phenomena and the program at school and like, what could I be contributing to the earth as we rapidly were approaching the targets that people said we were only going to approach in 20 years, right? In the last six months, as we've seen climate disasters kind of like take over the space, It's just become more and more my conviction that we're working in the space where we have to assume adoption of these technologies. We need to assume that we're going to be working, co-working across distance to solve really massive social problems, pop-up housing, migration, social justice, ways of voting, ways of engaging locally when we're in disaster, because national or international systems can't help, right? We need to start to innovate on these kinds of collaborative, collective actions in order to survive. And it's the generation coming through right now that's going to have the most investment in and control over those outcomes. Like, there's never been a better time to be doing this. It's so, so critically important that we use the tools that we have to create lasting change. And we don't just pass up this moment. of a new medium and just continue to do what we have been doing, which is monetize and productize eyeballs. It's just not going to work. When you read about the laying down movement that's happening in China right now, where students are leaving school, moving to the remote areas, and just doing what they need to get by because they don't want to participate in an economy which is destroying their ecosystems and shortening their lives, literally. I think you're going to see more and more and more of that. How can we be harnessing this stuff for a climate corps that's actually repopulating trees and helping fish and wildlife living locally, but being entertained globally by games, concerts, and other kinds of media that are really going to keep our spirits up when we're done at the end of the day working in situ in our environments, trying to curb some of the damage that we've done? I think this is a really, really powerful question, and it's been great to be here to talk about it.
[00:41:25.469] Kent Bye: Yeah. And with that, we're coming to the end of our 45 minutes and our discussion here. So, uh, Robin to end.
[00:41:32.919] Robin Hunicke: I thought that was an amazing message to end with. Thank you so much, Robin and Doug. That was a really awesome. Thank you. Yeah. Thanks for having us.
[00:41:43.177] Kent Bye: So that was Doug Northcook. He's the Program Director and Assistant Professor of Immersive Media at Chatham University, as well as Robin Koenigke, who is a game designer and CEO of Phenomena, as well as a full professor at UC Santa Cruz and the founding director of the Art and Design Games and Playable Media program. So this was a part of the WebXR Design Summit that was taking place on Tuesday, October 12th. And if you haven't watched it, there's a whole nine-hour YouTube livestream that has all the different sessions. And I highly recommend checking it out because WebXR as a design practice is still very nascent. It's at the very beginning. What the Design Summit ended up being was a lot of folks from other disciplines and other modalities, a lot of people that are just creating in different platforms, but just talking about the design principles generally and the whole stack of the 3D production line, everything from the low level to the high level. So it was just some really rich conversations from lots of different people from around the industry. But this conversation, I thought, was particularly interesting, just to see that there was these common threads of this interdisciplinary collaboration. Robin actually announced that there's going to be this combining of the art and design, games and playable media, creating a shared department with her colleagues in the theater arts department. including dance, theater, performance, dramaturgy, and critical practice that's founded in games, theater, and performing arts. So kind of mixing the practices of game design, game production, game art, and the theater arts, and they're going to be creating a new hybrid shared curriculum to engage with undergraduate and graduate students looking at the affordances of play, performance, and design. So again, this is really on the frontiers of what I also see, which is this fusion of different aspects of embodiment. So theater having lots of embodiment and lighting and practices like dramaturgy, which is this combination of looking at both the critical theory and analyzing the text and looking at all the elements of drama, but then to translate that into all the other components of experiential design with the lighting and the staging and the costumes and the Performances and you know just a lot of the nuances of the larger context and connecting the dots between What the playwright had created and all the other contextual aspects, but also working with the director and working with playwrights I mean dramaturgy was something that I did a bit of a deep dive just to see okay. I Not a lot of theater productions have a dramaturge, but it seems to be what I would think of as this interdisciplinary Experiential designer or architect who's looking at all these other contextual aspects and making sure it all coheres so you take lots of different film production roles and the theater roles and blending those together with the aspects of game design and playable media and a interactivity and agency and play and mashing that up with all the aspects of performance. Same thing with Doug Northquook in terms of the kind of like telephone game, what he calls the design dialogue project, which, you know, would take something from architecture, make a 3D file, then an audio file, and then translate that into a VR experience. And so going from one modality to the next, and there's not really a lot of good reasons why most workflows would do that. But in terms of really trying to look at the different design practices and the inspirations, I'm very curious to see this continuation of these different types of design dialogues and the fusion of all these different design disciplines together into what I call experiential design or immersive technology design. So, for me, that's just exciting to hear that there's this fusing together all these different design practices. And to actually have the game design department more in direct collaboration with the theater department, I think, makes a lot of sense because a lot of the motion capture technologies and performance and other elements of how to construct the narrative parts of a story. And immersive theater has already been doing a lot of this sort of fusion of game design elements, but, you know, to have the full university setting collaborations is going to bring more and more insights from the theater realm. Also, there's a lot of other aspects that Robin said in terms of the critical race theory, feminism, and decolonization. And Misha Chardonnay was mentioned doing embodiment and performance in terms of environmental challenges. She's actually got a book coming out in January that I'm really excited to learn more about. It's called Poetic Operations, Trans of Color, Art and Digital Media. So taking a lot of algorithmic analysis of cutting, shifting, and stitching, and taking these aspects from trans studies and applying them to what's happening with immersive media. in the blending and mixing of realities. So lots of really interesting aspects of critical theory that are fusing into experiential design and the intersection of everything from decolonial theory and intersectional feminism and trans studies and being able to look at and critique and analyze what's happening with some of these trans of color artists and what they're doing with augmented reality and virtual reality media. What Robin said, also, was that Michael Chimmer has written a book about the ghostlight dramaturgy, but also looking at Japanese puppet theater, and so looking at the Japanese traditions of operating a puppet and how that translates into puppeteering these avatars in these performances and where does the performance lie. So, yeah, lots of really interesting intersections there. And also, the open-source libraries for African-American hairstyles from A.M. Dark. So, lots of really interesting shout-outs that Robin gave, and I'm really curious to see how this continues to evolve and just this fusion of all these different practices is, I think, still very early in terms of the combination and drawing insights from all these different practices as more and more of these folks from different disciplines start to create immersive experiences. The other points I just wanted to bring out is thinking about and reflecting what does the web have to give to overall aspects of experiential design. I think a lot of the discussion ended up going towards what do people do on their computers when it comes to Zoom and these big social media platforms and the overall Centralization and monopolization of a lot of the aspects of the web that has happened over the years and so it's like this deeper question of like What is the web and what has the web translated into because we have native apps and then we have the the web apps So what is the real difference between these for me? I think the web is a lot about that and decentralized, being able to have everybody publish. One of the things that Doug Northquick was talking about was how, at the back end, there's a lot of these databases and data, and how showing that data and revealing that data, depending on whatever the context is. If you're on a phone, you may be on a native app, and you may be still looking at that same data, but there's still a lot of aspects of that data that are being shown on different aspects of the web. So I think there are going to be specific insights that are going to be coming from a web designer perspective, which is much more around the mashing up and mixing up of different APIs and blending in different information together and revealing it in different ways. So I think there's going to be a lot of things that are going to be made available that are going to go above and beyond what we've seen in, say, Unreal Engine or Unity. And there's just a lot of design practices that I think have been blending together this traditional pre-production, production, and post-production, and the more iterative aspects of web design, where you're incrementally implementing different aspects and kind of building stuff out slowly. And also the responsive design and progressive enhancement and graceful degradation concepts of being able to design one code base that works on a tablet or works on a phone or works on a PC, and it's able to be very fluid and have this responsive design. Well, when you add another dimension, there's going to be a three-dimensional responsive design. And I know that Trevor Flower is Potassium framework that starts to look at the different portals and the different ways that you're designing and all the different design Considerations that you have to take into account it sort of expands exponentially I have an interview with him that I've done before that really digs into a lot of the nuances of that But just I think in general this concept of being able to have a virtual world and experience But have a different access points into it whether or not you have a fully immersive VR experience or an augmented reality experience or maybe you're on a 2d phone and you have a portal experience and There's different social VR applications everything from VR chat and rec room rec room in particular that is able to really create everything from a totally immersed VR experience on the quest or a PC VR version from Steam or looking at PS VR or Xbox with sort of the console approach or the Android or iOS to be able to have portaled windows in and they're able to have an experience that has this cross-play with all these different people having different access points into these experiences. I actually have an interview with Cameron Brown where I'll be digging into a little bit of the history and evolution of Rec Room. I hope to be airing that here early next week. But, yeah, I do think there's going to be a lot of other different design practices that are going to be starting to be blended into the overall aspects of experience design, especially when it comes to human-computer interaction and the different affordances of web design We already have a lot of the 2D menus and whatnot that we've seen, but as we move forward and we have things like the Vive Flow, where you have your phone that actually becomes a device, I do expect that we're going to start to see more tablet interfaces that are going to start to be blended together with virtual reality. And this is something that non-fatiguing interfaces that Rob Lindemann has been studying for many, many years. But there's going to be certain ways that you're going to be able to draw in the different affordances of, say, a tablet or phone and those touch interfaces and start to fuse them in together. with these immersive experiences and it might be that the web designers are able to really push some of that forward or it's sort of a blending of Pairing up your VR headset with your phone again starting to see that with these foreign factors like the vive flow that was announced this past week and There's never been an example of a technology that completely supplants the previous technology. So there's a lot of things that you can do on your phone and tablets that are always going to be better than what you can do in a fully spatial virtual reality environment. But there's still going to be different aspects of this non-fatiguing interfaces that are going to be seamlessly blended into the affordances of spatial computing and virtual and augmented reality as well. So that's still at the frontiers for where that's going to go. And I do think that WebXR is going to be maybe on the frontiers of some of that experimentation. Anyway, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
