#1482: Mixed Reality Gameplay Innovations & Design Insights from “Starship Home” Co-Directors Mark Schramm & Ashley Pinnick

I interviewed Co-Directors of Starship Home Mark Schramm, Director of Engineering at Creature, and Ashley Pinnick, Art Director at Creature, at Meta Connect 2024. See more context in the rough transcript below.

Here’s the Launch Trailer for Starship Home:

This is a listener-supported podcast through the Voices of VR Patreon.

Music: Fatality

Rough Transcript

[00:00:05.458] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So on Thursday, September 26, 2024, was the launch of Starship Home, which is one of the first mixed reality games that's being launched on the MetaQuest platform. So I had a chance to previously talk to Doug Northcook, who was one of the founders of Creature, talking a little bit about Starship Home, but I had a chance to actually sit down with both of the co-directors of Starship Home, the director of engineering at Creature, Mark Schramm, as well as the art director at Creature, Ashley Penick. And so we talked a little bit about how this project came about, but also how a lot of this mixed reality project was this going back and forth with designing a lot of the different system level things that are needed to even run a mixed reality experience within the context of the MetaQuest platform. And so it's a very cutting edge pioneering project in that sense. And yeah, we just talk about some of the different design process and some of the different specific considerations for how to develop a mixed reality game and all the new and different design considerations that they had to take into account. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Mark and Ashley happened on Thursday, September 26th, 2024 at MetaConnect in Menlo Park, California. So with that, let's go ahead and dive right in.

[00:01:36.004] Mark Schramm: Hi, I'm Marc Schramm, I'm Director of Engineering at Creature. We're launching Starship Home today, this crazy mixed reality game for the Quest 3 family. And I've been around the VR space since the early DK1 days. I did Gravity Lab, Nighttime Terror for Gear VR, and then I worked at Superhot for the longest time. And now working on this crazy mixed reality game.

[00:01:55.523] Ashley Pinnick: Hi, I'm Ashley Penick. I'm art director at Creature. And like Mark, I have been around since the DK1 days, starting as an independent developer, working on my own projects, then being a tech artist and art director for Tilt Brush for a long period of time, and then being independent, working on a number of projects as an artist and designer, working on things like Cosmo High, Slime Ball, and now working with Mark on launching this mixed reality game, Starship Home.

[00:02:26.038] Kent Bye: Maybe you could each give a bit more context as to your background and all the disciplines you're pulling in. A little bit more about your journey into the space.

[00:02:33.344] Mark Schramm: Yeah, I think being a VR developer and indie developer for a long time, my focus is mostly on the engineering side. But I'm also just, I was co-directing the game Starship Home with Ash and co-game designing the game. And yeah, just doing pretty much everything other than anything visual.

[00:02:50.077] Ashley Pinnick: Yep, and similar but opposite with Mark. So I like Mark starting out as an indie developer. For me, like straight out of art school, just interested in games and art and interactive media and how all these things can come together for like storytelling and expression. and I think that for Starship, like Mark said, we shared sort of director duties, but my focus was on all things visual, so leading an art team, developing the art style, and trying to make something that was beautiful, tactile, and performant in mixed reality. So yeah, that was essentially my focus together with Mark, so yeah.

[00:03:27.677] Kent Bye: I had a recent conversation with Doug Northcook and he gave a bit of a context for the development of creature, but maybe you could each talk a little bit about your journey into coming to work with Doug and the broader creature entity to be able to do these different types of mixed reality experiences and games and like where that relationship started.

[00:03:45.356] Mark Schramm: Yeah, I think for context is that Creature is like a really young studio. This is our first game. Creature is also a game label that's run by Doug. And the way we started working on Starship Home was that about two years ago Doug approached all of us, I think, and just asked us if we wanted to work on this crazy idea. And I think we just all said yes. And then the way this actually happened is that he showed us a pitch deck with the game idea and asked what we think. And we all loved the idea. We loved the idea of plants and space and we all wanted to work on a cool mixed reality game. And then I think he approached Meta and essentially told them about the pitch, told them about the people. And I think what happened is that Meta said, oh, this looks great, but all these people that are mentioned here in this pitch deck, they're all placeholder, right? Like, how can you have people from Superhot, people from Tilt Brush, people from Space Pirate Trainer? How can you have all these people? And Doug just said, well, I asked them and they said yes. And I think that's how we all came working together. I would have never thought I'd work with someone who worked on Tilt Brush. So I was very pleasantly surprised when Doug told me about all the other amazing people that work on the project. So, yeah.

[00:04:53.683] Ashley Pinnick: Yeah, I think Mark summed it up really well. Doug had reached out to me because he knew through mutual friends, other people who worked on Tilt Brush, that I was independent. I had been working as an independent art director on multiple projects, different things that are happening in games and VR. for a few years and so through that I got connected with Doug and yeah just like Mark said Doug showed us the pitch we talked through the creative possibilities of the game and I was really interested in like this intersection between something organic and cozy for lack of a better word with the idea of you know raising alien plants and then like the psychedelic possibilities of their dreams And of course the mixed reality possibilities of, you know, trying to create a full fledged game with this new tech stack that Meta was supporting and trying to understand what kind of experience we could create together.

[00:05:51.530] Kent Bye: Yeah, well, I think being in the realm of VR, each of you are comfortable with uncertainty and being able to see what's possible, but also bump up against the edge of what the platform is doing and pushing the edges of possibility, but also dealing with the constraints of those new affordances. And talking to Doug, he was saying that this project actually started on the Quest 2, so before a lot of the Quest 3 and even the meshes that were in the room scan. So it sounds like this is an idea that you've been incubating and prototyping for quite a long time. in terms of pushing the edge of what's possible for mixed reality gaming. And maybe you could start for where you began with this project before you had all of the platform tools from MetaSight. Because I imagine there might have been some back and forth of this type of creative project driving what even the platform's able to provide for the types of tools that you needed to even do this game. So I'd love to hear where that process began.

[00:06:41.107] Mark Schramm: Yeah, I think the biggest issue we had at the beginning with Quest 2 about two years ago is that mixed reality wasn't a thing. There was just a pass-through API. And then I think the first thing they had was adding the bounding boxes around objects that the user had to define. And that one also had this weird thing where if you turn off the Guardian, if you go close to an object, the bounding box fades in. We have this big blue box that appears around an object that we have no control over. It renders on top of our screen. And if you've seen our game, our game really uses all the surfaces in your room. It really wants you to go up to a table and just place a plant on top and just play with it. But at that point, that bounding box would just render on top of the plant. It really would discourage you from going to any surface in your room, your walls, your tables. And it was really meant to be like, you're in the middle of your room and we'll just have something like the old guardian that just prevents you from running into something. Working together with Meta meant that we had input for them on things like these. And then with many conversations and us showing the use case of what cool mixed reality actually be, I think we together with Meta have been shaping the platform and shaping a lot of the features that are currently available.

[00:07:53.171] Ashley Pinnick: Yeah, we've been giving them a lot of feedback and as the game has evolved, you know, so has the platform, right? So I think one thing from a visual perspective that was definitely a shift from working on Quest 2 to Quest 3 is of course having color pass through and some of the other features and things that we didn't have access to before that really allowed the game to feel richer and deeper and more connected with your space. Something that we gave them a lot of feedback on were things like the bounding boxes, which are clearly designed from a UX perspective for safety and not for experiential use, where you can come up to a surface and interact with it. And we designed our props and objects in the game in order to respond to where they were placed in your space. That's something that in sharing with Meta, I think that they started to understand the language that we were trying to develop. And all of that came together through prototyping and design and discovery as a team. But being able to share that feedback with Meta and have them understand why certain aspects of the design of their platform were at odds with the idea of making a game that's full pass-through, super helpful to be able to give them that feedback.

[00:09:04.076] Kent Bye: Right. So it sounds like that starting for Quest 2 was before even you have all the existing room scans and meshes and everything that's kind of built in. And so talk about the iterative process of designing the core bones of an experience where it feels like you're on a train track kind of laying down the rails as you're building or- You're flying, and you're trying to finish the plane, but you're already in the air. And this is kind of like this paradoxical back and forth between pushing what's possible and then building stuff out. So yeah, just talk about that design process of trying to get the core gameplay loops of what you wanted to try to do. you know i had a chance to play it here in palo alto in the hotel room for a couple hours and you know for me it was like super transportive of being able to like feel like i was transforming the hotel room and like creating these new memories and feeling like i was going on a journey even though i wasn't going anywhere so yeah it's really at the cutting edge of mixed reality gaming and gameplay but i'm just wondering like where in the process where you began of trying to figure out what were the core affordances of what mixed reality could do and how you wanted to try to turn that into these different gameplay loops that you have

[00:10:08.128] Ashley Pinnick: I think like one of the first things that Mark did was, you know, to the point of what you're talking about with transforming the space. I think that that's something that, you know, was part of the conversation from the beginning. And one of the first things that Mark did is he got like an asset from somewhere online. that could be a window that had depth into space. And we were already, one of the main things we were starting to fill out was, you know, extensile tech and having windows out into other environments and cutting out your space different ways. And then Mark put in the window that had like all this depth and little greebles and things in it. It was very like hard sci-fi, so not the visual style of our game, but in spirit, it added this literal and visual depth to what we were doing and that I think helped unlock from my perspective a certain type of quality that we wanted the objects to have so that they don't feel like objects that are from the real world but they have a kind of mechanical design and a plausibility that really lends to the kind of game that we want to create so that it can be cartoony but it has like a level of depth that allows you to sit in this fantasy and belong in that space and I think that that's something that We discovered early on through prototyping, through Mark putting some of the different ideas with windows and things inside, and us doing some experiments even just with a plant that looks at you and responds to you, which ended up in the final game. All of those things, I think, came down to this idea of presence and transporting you and making use of your space, giving you the ability to... like transform it, but also have input in what that is for you. So I think that there was a lot of early prototyping and exploration that really helped us double down on having things that respond to you, things respond to your input, and things that feel like they are tangible, tactile extensions of your space.

[00:12:06.452] Mark Schramm: You were mentioning the analogy of being on a runway trying to build a plane before it takes off. But in MR, it was more like you're on what looks like a runway and you think you might be building a plane, but you're not sure. You might actually be on a train track trying to build a train or you're on a highway trying to build a car. You don't know what you're building yet. You know that you're going at 100 miles per hour. and you have to make a decision at some point. And I think we had a lot of experimentation with things. And one of the things that we thought was obvious and then realized that's completely wrong was just like giving you lots of objects that we can put in your room and like fill it up with. And we thought that would be fun because like, why wouldn't it be fun to give you like all the Starship objects, like all of the things like a reactor core and like engines and many, many more weird stations. I think at some point we had like a station for like plant care itself with like putting soil and all of that stuff. And what we realized is that we need to be respectful with the player's space. Because a normal room, unless you live in an Ikea catalog, it's not tidy. It's full of stuff. And then if we give you 50 different items, that have various sizes, we just cramp your room even more. And then rather than playing a fun game, it becomes a chore, where you need to now organize this room even more than you had to do before, what you were probably trying to escape from in the first place. So what we realized is that we wanted to build more rich and delightful objects, and that's why our objects and our plants, they feel like nice little self-contained fun little things that you can play with, rather than just giving you just a bunch of decoration that you place across your room. And in the same way, we also realized that with anything in our gameplay, we were trying to straddle that fine line between immersion and intrusion. Immerse the player enough into the experience that they actually buy into the fantasy and they're having a good time and they're feeling transported, but never go so far as we're actually intruding onto your space and making you feel not at home anymore.

[00:14:03.635] Kent Bye: Yeah, and it reminds me of the storage unit you put on the ground, and you pull it up, and you are able to have a shelf to put away things so you can keep proper feng shui. So it's not too cluttered, but you still have access to different objects that you're interacting with, which I think was a really great affordance that you have in there. That's actually reminding me of a game that was at Magic Leap LeapCon back in 2018. It was a mixed reality game. And it was all about plants and cultivating plants. And it feels like that, you know, a lot of games are a lot about agency and expressing your will into the world. And that this game overall feels like it's a nurturing, more of a yen, trying to cultivate and grow things in a way that You've got really nice icons that are on the plants that you're taking care of and it's like the plants are telling you what it needs because you can't necessarily communicate but it's got this iconography and the different types of things that you have to learn how to interact and play with the plants but then like put the plant on another object that then enables even more gameplay with these plants so it felt like layered first order signals to say okay here's something that you can do and then the the second order where you're interacting with other objects in order to have other gameplay loops throughout the course of this game but overall it's like this more cultivating experience and like you're making a home and taking care of things this is like one of the first mixed reality games and it feels like this is in some ways a new genre of trying to figure out what are the different types of experiences in games that are unique to mixed reality and that feels like this game is kind of like the starship home like you're making the home but you're also going on a journey so it feels like it's a unique mix of all these things so in terms of genre and in terms of other inspirations i'm wondering like what other games or experiences were you looking to to get some inspiration for what this experience could be

[00:15:45.367] Ashley Pinnick: That's a great question. I think that, you know, to what you're talking about, you know, we had a lot of back and forth and a lot of prototyping at the beginning about like, well, what is this game? What actually works? And like Mark said, you know, the first instinct of creating a lot of objects that weren't very rich, but were there in order for you to like have 25 different things in place in your space. I think that was a huge learning that had us be thinking about a lot of other things. I think that, you know, something that I'm always inspired by is like the elegance and simplicity of like Nintendo games and like the clear UX in a lot of games like that where you are being told what your goalposts are, you are being communicated to in a way that is like delightful and is accessible regardless of like what language you speak. That type of thing where like the visual affordance is something that is part of the design from the beginning. And a lot of the prototyping that we did, I think, you know, emphasized being able to connect gameplay, visuals, mixed reality all together in order to try and communicate to the player because we're always competing for your attention with your actual space, with like your dog that's running circles in the corner and like your... sink that's leaking and dripping you know like there's always some other thing happening and so we have to be respectful of your space and also try and communicate in a way that is going to capture your attention where we don't have control over what's around you or where things are relative to each other even so If you place all of your plants in a corner across the room from something else, we can't guarantee that you are going to see like a notification on a screen that you've placed across the room, that type of thing. So I think like elegance is something that you're like always trying to achieve. And I think part of that is an understanding that like UX is not just about notifications, but also about visuals and about audio and about haptics and like that richness so that you can So that's one area that I think that I drew inspiration from and that we talked about from a visual perspective. The inspirations I took from were also like Y2K era visual design, like something that's organic and friendly and not like a cold sort of sci-fi that you would normally see. Looking for something that's cozier and softer and more welcoming. as well as psychedelic art and things that could help you get into this weird world that we're building. So those are the things that come to mind for me.

[00:18:32.389] Mark Schramm: Yeah, I think all of us, we're huge sci-fi nerds in a way, some more direct than others, but that's why in the initial assets that I just brought in, just the prototype stuff, they were hard sci-fi. And by doing that, we also realized, no, that doesn't feel right. And in a way, also, having these assets in your games or something that looks more realistic that someone knows from a style from the Star Wars movies or whatever, or any sci-fi movie, it doesn't gel well with your space. Because these objects exist in your space with your tea kettle in the background or your lamp, your pot plant that's also in the background. So we wanted something that feels very different from the typical hard sci-fi. And then you actually came up with this great art style of everything just being soft and round and yet have this plasticky Y2K feel of the atomic purple N64 and all of those things. And yeah, just having this in your space makes it feel cozy. And it also doesn't feel like it's out of place, even though they are distinctively alien, distinctively weird. And I think because they're so different, you just accept them as like, okay, well, now this is an alien tech that's in my room, but it just makes sense now. And I think one of the largest things that we had to figure out during development is trying to find that right balance of how interactive those objects are. If they weren't interactive enough, they would feel stupid. They just feel they take up space. But if they're being too complex, that's really hard to teach. So all our objects have really large levers with a bright red handle, and it's very clear how you interact with them. And then they're fun because we have really good haptics and really good audio design and really good special effects on these objects and they just feel delightful to use.

[00:20:17.010] Ashley Pinnick: I think that that was definitely something that we talked a lot about in the visual style and in the way that we want to express gameplay and how visuals and gameplay have to be in sync with each other. Making that lever that makes sense so that it fits the interaction and having all the mechanical design follow what the game design and what the feel of the game should be. And I think that, to Mark's point about the... visual style that we created is creating cohesion across these different objects was also something that was really important for having this language that like all of the objects could share and so I think that was something that once we started having multiple objects like we have the windows we have the front of the ship where you have all your controls and the plants themselves by giving them all like a similar shape language and a color palette that they can share then they all relate to each other and it's not quite as jarring if that makes sense for you to have these like very cartoony very stylized things in your space because they together gel and everything in your space it can be there you can notice it but it can fade into the background a little bit i think what what also helped us

[00:21:30.093] Mark Schramm: Going back to, are we a plane, a train, a car? Well, at some point we realized we were a spaceship. But to actually build that, what really helped us was drilling down what genre our game is. We had all of these little individual pieces that were a little bit puzzly, a little bit sandboxy. We had a star chart we can go around. But then what we really found out of all the pieces that we liked was the characters. And that includes our plants, we see them as characters. Internally we have names for them, but they're not being shown to the player really. They are our characters, and then also the characters that are talking to you, which is the starship, there's a character called Vondrim, there's a character called Jeff. And they become your buddies along the way. And we had little test recordings from Bonnie, our voiceover actress, And they were just amazing. And we just felt like, oh, we want to hang out more. At that point we had written very little dialogue for the characters and we were thinking we were more like a sandboxy experience. But we wanted to just drill down on the things that were fun. And that was just like hanging out with cool little aliens and cool little alien plants. So Chris Henney from Space Pirate Train of Fame and I, we wrote the story with the help of Xavier Nelson, who did El Paso Elsewhere just recently. And we wrote this fantastical narrative of traveling across the galaxy and figuring out this weird mystery of why plants are falling ill across the galaxy. And things happen and the player gets to resolve that.

[00:22:58.789] Kent Bye: Yeah, I think there's a dual purpose for those messages, which is for one, have those narrative arcs and the bits to give you a little bit more context for why you're doing these things in the context of a larger story and a larger narrative. But also just kind of more functional, like here's the next thing that you're supposed to do. And so I found myself after I got to 37%. and ran into some game breaking bugs that we can get into. I think there were probably more on the platform side than what was happening on the actual Starship. But I noticed as I was playing it that sometimes I would see all the visual language for what I needed to do after I figured out some of the gameplay loops and some of the different tasks. And there would be an incoming message, and then I had an option to stop the action that I was doing in the moment of caring for these plants or to go over and to like get the message and sometimes I felt well I kind of like know maybe what that message is going to tell me like what to do next if I get lost but there could be also narrative components so there's this like trade-off I had to be like okay I'm going to be present with these plants but then there's like this notification system like it's like you go over and to accept the message that is kind of either disruptive what were you doing or it actually helps progress you to have you do the next thing and so It felt like sometimes I could ignore those messages and sometimes it would go away and sometimes I would go over and get it and just get like more parts of the story. So I'm just wondering how you like navigate that in terms of trying not to hold the hand of people too much while at the same time not have them be completely lost as to what to do next.

[00:24:22.844] Mark Schramm: Yeah, I think that was a big thing that we worked on, trying to make sure that, because at the very beginning we had almost zero guidance for you and then a lot of players felt lost and then we had a lot of guidance for you and then it felt very preachery, right? Like, it's like a teacher telling you exactly what to do every step along the way. So we ended up on a system where you can do things and if there's a message, you can accept it. It doesn't autoplay, it's an incoming transmission, so it doesn't interrupt what you're currently doing, but you can decide actively to go there and listen to it. But in most cases, these are just little helpful messages that will remind you what to do in case you have forgotten what to do. Because a use case that we also see is that you play for a little bit and then life happens and you don't play for another week or two or a month. and like the worst feeling that i have when i come back and play a game that i love it's like i come back and i have completely forgotten what i'm supposed to do so that's why we have these messages once in a while where like one character chimes in and says hey you know we should probably try watering the plant and then we have several systems that help you as well we have a little bot that's flying around that can give you guidance and it's just a plant that's in desperate need of like fertilizer or water or something it'll just park itself next to it has a little light on top And then almost like a little billboard that's rotating around it and telling the player, hey, this plant needs water. And then we also have a manual, which is like an explorer's guide for Starship and Galaxy explorers. And it contains information on how to take care of plants, how to go down to planets, how to space travel, and all of these things.

[00:25:51.863] Ashley Pinnick: Yeah, I think that something that I saw a lot in playtesting and I think was echoed with other playtesting that folks did with friends and coworkers and things like that was when we first started to really add a lot more guidance after the game had been very open-ended. The question I had gotten a lot from people was, oh, when's it going to slow down? Because I just want to like hang out with these plants. I want to have a good time. When is it going to let the foot off the gas? You know, and that type of feedback, super helpful and something that we tried to address. address by giving you things you can opt into and reminders but not trying to force you to like follow everything in the exact path in the exact way that really handholds you throughout the experience because I think that something that we wanted and that we hope that players will experience is the ability to just relax and be able to spend time with the plants and experiment with how they interact with you with how they interact with each other being able to take friends into the game and show them like their favorite plant dream if they want and so I think that the ultimate balance that we would like to strike and hope that players will experience is being able to take away across the galaxy journey to rescue and heal all of these plants, but also to be able to just spend time in your space with them and chill if you want as well. Hmm.

[00:27:22.833] Kent Bye: Yeah, and I think I fell into that. At a certain point, I was very obedient to always accepting every message that came in. And at some point, I was like, okay, I kind of know what they're going to say, so I just surrendered myself to doing whatever I felt like and getting into that more of open-ended play and also just trying to solve the next puzzle to unlock the next narrative bit or to progress the experience. So there were a couple of game breaking bugs that I hit that I suspect in talking to Doug that he said, look, there's a lot of stuff that's happening at the platform level that is beyond our control. So one thing that I found is that for some reason I stopped and started, I don't know why, but when I restarted, the plant was gone and then you have to fertilize the plant at some point in order to progress. But because it was gone, I couldn't progress. And so despite everything that I had done, it was like, okay, I think this is broken because I can't do anything. I was like fertilizing the little floating holographic bot that was there. And it's like, wait, that's not what I'm supposed to be doing. So then I reset and there's this conceit where there's a box where you put all these different objects in the world to kind of make the space and transform your home into a starship. So the home that's at the front and like the different windows, there's like stuff you could put on the floor that are also windows, but also the storage unit and this like fertilizing creatures space. and the airlock that is interfacing with the outside world and the watering can. So basically all these objects that I'm interacting with. And so sometimes when I would stop and restart, there'd be things that would disappear, like the plant one time. And another time the airlock was embedded into the wall and I couldn't get access to it. So then I had to like shut it down and replace everything. So I felt like stopping and starting feels like high risk. And talking to Doug, he said like there's certain probability of things like one in 200 chance of something like going wrong But it sounds like there's a unique challenge of trying to figure out if there's something that's wrong with your game or something that's wrong at the platform level. And at this point, because it's like the first game, it's a challenge of knowing what any one of those number of things that could go wrong with the game because the platform is still developing that your experience takes a hit or having some sort of game breaking bug that's like beyond your control.

[00:29:26.408] Ashley Pinnick: what I was just going to say is it's also a challenge of like what data do we have access to as developers because there's a lot that's hidden away on the platform level that you know for good privacy reasons that we don't have access to I know that Mark is probably desperate to answer this question so I'll let him continue but I will say that yeah it's also an issue of like How much do we actually have access to? And I will say that some of the issues it sounds like you ran into, we are hoping to be able to push an update to make some of those things easier for you. Like the scene drift thing is something we don't have control over, but is something that you could potentially run into. And we have ideas of how to help mitigate that so that you can like force grab an item out of a wall and just... not worry about it for me when objects have disappeared. Sometimes they're like outside in the next room, right? Which doesn't help if you're in a hotel room. So I hear you.

[00:30:19.795] Mark Schramm: Yeah, I think, I mean, as the platform has evolved, we've all been working like very fast and very hard on all these issues. And we are one of the first large scale mixed reality titles on a platform that's also just still actively evolving. And yeah, so I think we're going to take the hit on like some of the platform level issues. And for example, spatial anchors can drift. And especially if you have like many, many rooms in your headset stored, spatial anchors can get confused as to which room they belong to. So if you had a plant and you put it somewhere and then you reload the game, that spatial anchor might now think it belongs to a different room and then that plant is gone, right? So that's why we have some options in the game, like just getting a new box delivered with all the items and sounds like that has saved you a few times. But yeah, it's not a great user experience and we know it. And I hope the players also know that, and I hope the players have the patience with the platform and with our game. And they can always come to our Discord or anywhere we are where you can find us, and we have some easy solutions. It sounds stupid, but have you just tried restarting the game? Have you tried turning it off and on again? Not to be pedantic about it, but sometimes the scene API just requires sometimes a second start to come back with all the spatial anchors, right? Sometimes it's all just like something has gone into the wall and then when you come back it just spatializes again correctly. Then there are other options like resetting of the box and then we have a debug menu that there's like a way to get into that we can tell players if something has gone horribly wrong. which allows them to have access to some of the tools that we internally use to fix some of these issues. But then we're also just monitoring everything now and seeing the issues that people are having, the issues on the new device or new firmware versions. And all of these issues change every time there's a new firmware update, every time there's a new SDK update. Sometimes it gets much better, sometimes it gets a little bit worse. But overall, I think the platform is becoming more mature over time and hopefully players see that as well in our game. And yeah, if players have an issue, just please feel free to reach out. We're all in this spaceship together in a way and traveling into the future and traveling across the galaxy and hopefully we'll all find a good time.

[00:32:27.000] Ashley Pinnick: I think definitely having feedback because of the infinite variability of a space, because of the funky shapes your room could be, because of the different choices you might make setting up your space, there's definitely a huge need for any feedback from players, good, bad, weird. confusing any of it because we've been testing in our bathrooms, testing in our largest room in our house, like testing in low light, testing in bright light, trying to understand what people might run into and trying to cover as many cases as possible. But it's always something that, you know, you never know what might happen when a firmware update drops. And we've been doing our best to try and fast follow all of that with our development to say, OK, like what issues might the player run into? What are some fixes? And hopefully being able to continue to address those in the future and make it a stable, fun, relaxing experience and not a frustrating one.

[00:33:27.327] Mark Schramm: In a VR game, they very clearly tell the player, hey, you need a seated space or a standing space or like a six-by-foot space, right? And then within that space, you can test your game to the fullest extent. Like, there's a floor space and the game needs to be playable in that. And as long as the player has that, that game can be played. In mixed reality, we can't just say, yeah, it works in most rooms, but just not your room. Just get a new room, right? We can't do that. So, yeah, we're really trying to understand what players' rooms are, how they are shaped. And then also I think a lot of our gameplay revolves around the player positioning things. So in a way we get around some of these issues from a conceptual level. But yeah, there might be weird rooms and we have rooms with sloped ceilings and that's something that the presence platform currently doesn't support. Someone might have a weird pillar in the middle of the room that is right where something happens in the game, right? Stuff like this is hard and then sometimes it even helps just doing a new room scan and maybe just scanning it quicker, scanning it slower, having more definition or less definition can sometimes help. All of this stuff is very new and I hope we can evolve the platform with Meta as well based on the feedback we can give them and we hope that players will have a good time.

[00:34:36.892] Kent Bye: Yeah, in my hotel room, there's also a mirror. So I don't know if that was also throwing things off. But one of the things I noticed was more of a platform level when I'm doing a room scan, there's kind of this timer that is moving from left to right that in some ways, I read it as this is how much percent that you're done. And then when I first did Thomas's laser dance, I was scanning things. And I was trying to get every single corner and mesh correct. And then at some point, it just said, OK, you're done. Here's your mesh. And then it had half the room not even scanned. And so I found like, okay, I'm in this big room and I knew that that was a pattern. So then I scanned it. I was like, okay, I'm getting all the little corners and then it crashes. And then like, okay, now I have to restart. And I had to do it like five times, but I was like, okay, I know at some point something's going to go wrong where it's going to crash for whatever reason. I just want to get a good enough mesh that's not perfect, but at least is good enough for my room. So there seemed to be like this weird pattern. timing like I always would want more time to actually like fill out the mesh but I've run into a couple of mixed reality games where at the platform level I felt rushed and it was also crashing and like not always getting like the scan right the first time.

[00:35:42.417] Ashley Pinnick: Definitely like what I noticed at least in the flow that my headset has who knows with like whatever the latest firmware update that's been changing some of those flows in subtle ways but the most common thing that I saw is that bar comes up and it's like okay you're good to go but you could actually continue scanning. And so if you just ignore it and let the notification float in front of your face, you could sit there and continue scanning. But most people, 99% of people are going to go, oh, oh, I'm done. And then they keep going. Right. And so I think that I totally understand that feeling of being rushed. And you are not the first person who's mentioned that to us because it definitely, I think, is some conflicting information. And I think that with scanning spaces, the thing that I've been doing is middle of the day with all of my lights on and moving things out of the way to scan just what I want. Like I'll go to the nth degree to try to get the right thing that I want. So it's like that my cup doesn't end up getting scanned that I will inevitably move just so that then I have something clean to work off of. But yeah, I've definitely run into a lot of those issues that you've mentioned about crashing or it saying you're done when you've only scanned half the room. Hopefully as the platform continues to mature, Meta will find the right balance to like something that doesn't feel like there's a ton of friction or time to set up, but also something that feels like it gets you the optimal result. So I think it's a hard problem, but they know they're continuing to iterate on it and hopefully they can find the right balance.

[00:37:14.821] Mark Schramm: Yeah, and I think as more people will experience mixed reality, there will be just more feedback from meta as well that helps them inform decisions around like how the room scanning works. As a developer right now, we don't know if you scanned your full room or you've only like scanned one corner and then sort of timed out or that progress bar has filled up, which I'm also not 100% sure if that's just like an amount of like surface area that you scanned or something, right? But as a developer, we just get a room mesh and we have no information if an area in this room mesh is accurate or not. So to an extent, we just have to live with what we get. So that's why I also think it's at the very beginning of the game on the starting screen, we kind of show you the room mesh that we've detected. It says like, oh, can you just please check if that looks okay to you? And then there's a start button and you launch the game. I know players see a big button and they punch it without reading what it says above it. And that's totally fine. But it's just another checkpoint where a player can like, oh, wait, this room looks weird. Maybe I go and rescan it. And there's a button that brings them back to the OS level room scanning part of the quest where they can rescan their room if they think it does look weird.

[00:38:19.784] Kent Bye: Yeah, for sure. I definitely restarted a few times just to get the room mesh correct because it was a nice sanity check to have that there. But I think overall, when I was playing the experience, I had an amazing time playing it. And I felt like there was ways that you're using screen space shaders and outlining different worlds and taking me into psychedelic trips. And I felt like there was a lot of ways that you were... able to make it feel like I was going on a journey even though I wasn't going anywhere and it felt like I'm staying in this hotel room so like when I go back in there now I have these memories of this space and this experience and like yeah so when I go back home it'll be a challenge of seeing like if I can scan an existing room and like if it stays at the 37% and like if I can continue to progress if I'm changing the context because it you know it's maintaining some of those different states

[00:39:05.441] Mark Schramm: Yeah, I mean, like you were mentioning, you formed new memories in that room that you were in and maybe it's like for that hotel room, that hotel room is now forever your starship room, right? Like that's where you went on to adventure. But it's hard to explain VR to people and it's even harder to explain MR to people, right? Like this mixed reality game, like how does it feel being in your living room and going into space? And I don't know if that makes sense to people, but it's the sentence that I thought I would never say. It feels significantly different bringing your living room into space than your kitchen. It's a very different context. It feels like in my living room, I want to chill. In my kitchen, I want to go and work and flip burgers and stuff. It feels very different.

[00:39:43.055] Kent Bye: Yeah, and there were certainly some moments that I felt like I had this really deep sense of presence where there's an airlock and the airlock actually has a space that's like a portal that's into a wall. And like I found myself putting it up against the hotel wall. But when I tried to put one of the objects into the airlock I actually tried to put my fist through the wall and like punch the wall and I was like oh it's like one of those moments where you're trying to lean on a table that's not actually there or you know you get so immersed into an experience where I felt like there's something around like having my physical body in the space and having the room and having all these stylized cartoon objects But yet my brain felt like it was plausible because I was able to interact with them They were responsive to what I was trying to do and it just felt like it had this sense of plausibility and also like a sense of Environmental presence that tricked me and I think there's those moments of being tricked where it kind of unfortunately breaks the illusion but at the same time like Reinforces like how strong my suspension of disbelief actually was in that moment. So

[00:40:45.087] Ashley Pinnick: That's super cool to hear that you were having that suspension of disbelief. Although I'm sorry for your hand that you hit on the wall. It has definitely happened. And some of the things that we were trying to do with designing objects and having this plausible mechanical design is also trying to prevent some of those scenarios as much as possible. I think the airlock is definitely the one where you can still be like, oh, I'm going to reach inside. and pull this little guy out. But then also, it's part of the reason why when you open the airlock, the plant or the thing that's inside the airlock comes out of the drawer with it. So try and help just a little bit to be like, okay, now touch it here, not there. Or for our front of the spaceship where you have the controls and things like that, by pushing those further out on a virtual sort of ledge so that then you aren't hitting your knuckles when you push the hyperspace lever forward and all that type of thing. It's definitely something we were trying to think about and design for as much as possible, but really wanting those objects to feel rich and interactive and interesting to play with.

[00:41:52.193] Mark Schramm: Yeah, and I think the interesting thing is that these are not photorealistic objects, right? And I think a lot of MR games fall into this trap of trying to render photorealistic graphics because you're literally competing against the photorealism that you see through the pass-through. But we also realized that just also from a performance budget, we cannot render photorealistic objects. We cannot design photorealistic objects. We don't want to design photorealistic objects. So what we do instead with the system tools that we have of like coloring the path through and having like edge detection in the path through and just aligning the light in the virtual and in the real, we're trying to bring down the realism of the real world a little bit while still like building like physically plausible, nice looking, nice shaded objects. And then they kind of meet in the middle. Like we bring down the realism a little bit and we push our virtual objects as much as we can. And then when they meet in the middle, That moment of presence happens, where it's like, oh yeah, they mix. They meet in the middle and you accept them as one world that you can live in. And yes, it's still your living room or your hotel room, but it's also not. It's also like the spaceship that's going somewhere, but it's still your room. and it's like that conflict of like it's my room but it's not my room but it's okay like that is like a really interesting moment like the moment we reached it with like the color pass through and the color lots that we have and the coloring of the you know the edges and our objects and the lighting when that all matches it just feels magical it feels like the first time when we put on like a you know a cv one and we felt presence we just have that that feeling all over again definitely

[00:43:21.418] Ashley Pinnick: I think that something that we intentionally did from a shaders and visuals perspective is trying to create these objects where they have this like we focused a lot on like specularity and like variation and roughness and that type of thing to give you something that felt physical something felt tactile and comparing an object that was like flat shaded or smooth shaded with objects that have like a bit of pattern or a bit of variation and bit of that surface that you can differentiate between you know something that feels like like a hard plastic versus like anodized metal that brought a level of presence with objects where it's not a physically accurate material it's not we're trying to do it's not what we wanted to do but something that has that reaction to being moved in your space something that feels like it has a plausibility I think added a lot to the overall suspension of disbelief because your brain believes it and that's enough.

[00:44:26.906] Kent Bye: Yeah, I noticed the tinting of the room. So it's interesting to decrease the fidelity of the physical reality and then increase the plausibility of those virtual objects. So yeah, I definitely felt like it met in the middle. And yeah, I just had a lot of fun playing it. And I'm looking forward to playing through more. And I think there's these various different frustrating platform bugs or things that were just like, as I get home and try to play through the rest of the experience, I was having a lot of fun as I was going through these different gameplay loops and trips and journeys. I had a conversation earlier with Karistel right before I went to go play it from Gorilla Tag. And he was just talking around the way that they built the locomotion system so that your body is intimately connected to your environment around you. And I feel like the mixed reality has a natural affordance of your body being connected to your natural environment. There's stuff as you move around and you're able to touch different surfaces, but you're interacting with virtual objects and there's no hand tracking. So it's just mostly having the controller to kind of fill in with those haptic feedback, I think is those little subtle things that were helping to sell the plausibility of these interactions and how you're using the haptics of the controllers to help fill in the gaps of interacting with the virtual objects that my brain was able to just surrender to the process of like interacting with everything. And at a certain point, you know, like I said, getting so immersed so that I kind of forget that I'm in a virtual space.

[00:45:44.332] Mark Schramm: That being said, we also support hand tracking. And we want to do that especially as an accessibility thing. Because as you know, you own a VR headset or a MR headset, and you might be the only ones of the people you know around you that have one. And you want to share a cool experience with your friends. And the hardest thing is getting a headset on, but then you always have this awkward thing where they have the headset on, and you have to hand them the controllers. And it's like, oh, this is the left controller, and this is the right controller. And it's like, oh, this is the grip trigger and all of that. So we also support hand tracking for the experience but also for the dreams of plants with the specific intent that hey you can just quickly hand someone a headset and they can have five minutes of good fun and don't have to worry about like controllers and everything yeah.

[00:46:25.563] Ashley Pinnick: I would like, to the point of haptics, you know, I'd love to shout out the person that really focused on haptics in the game, Carson Custer, who just really employed a lot of love and care with bringing those objects to life. And it was something where when we would compare an object that had been fully set up with the haptics and the sound effects and our visual effects that were created by some wonderful people on our art team with something that didn't have it, the ability to believe what was happening was crazy than the difference between something that had no haptics and something that did have it and I think that that adds a lot to like you said you believing it especially something like being able to touch the plants like stuff like that makes a huge difference I think in being able to just fall into the world so yeah I guess finally as we start to wrap up I'd love to hear what each of you think the ultimate potential of mixed reality might be and what it might be able to enable I'll go first. So it's a great question. I think that one thing that I think is really fun about mixed reality is that because it's a blend of virtual objects and the real world, that it's not quite as isolating. as it could be for you to have a face computer on where you are like divorced from what's happening around you. And I think that that will allow for different types of experiences. And with what we've tried to do with Starship Home, I think that We wanted to create a game that has that chill, that has that transportive factor, but also doesn't intrude or overtake from your space and having respect for the space that you have and having respect for the choices that you make. I think is an interesting and open question for how people choose to design and develop mixed reality experiences. So I think there are interesting opportunities with taking advantage of and blending the two spaces that I really hope that people explore and find some new types of gameplay with.

[00:48:34.727] Mark Schramm: Yeah, as an engineer, I'm sort of stuck deep inside the code. But I always felt like I want to be creative, but I can't draw, I can't sing or play an instrument. But what I did on Starship was, with Chris, write the story. And I'm really excited for the storytelling aspects of mixed reality. Which kind of brings the question, why is Starship Hope not a VR game? Why couldn't the story be told in VR? In VR you're always inhabiting some other character. You're an astronaut or a spy or a super soldier, right? But for the narrative that Starship is trying to tell, which is overall like a climate change allegory, we felt that the player needed to be themselves. But you're never really yourself in VR, you're always just puppeteering someone else. But in Mixed Reality, just by default, you are you. And we really wanted the player to feel like that their actions have an impact on the universe and that there's an effect on the actions that they can actually affect the universe and affect change. And Mixed Reality, when used for storytelling, we are where we try to bring a player to different worlds. In Mixed Reality, we can bring those worlds back to the player and that feels super impactful.

[00:49:47.694] Kent Bye: Great. Is there anything else that's left unsaid or any final thoughts that you have for the broader immersive community?

[00:49:53.596] Ashley Pinnick: Great question. I just really look forward to seeing what new creative things can come out of people exploring mixed reality. And I think that if anybody has any questions or any feedback, we would love to hear any thoughts from the community, from players, from people who are interested or critical of mixed reality, and just hear about what you think. We have a Starship Home Discord. We're all on, you know, the various internet websites. But I would love to hear people's feedback and know what their experience has been if they play the game and what they're making with Mixed Reality.

[00:50:29.434] Mark Schramm: Yeah, and if you're a VR developer, MR developer, and have any questions like how we've done these things, like, yeah, as Ash said, you can find us on the internet. I think we're very easy to find. And just shoot us a message. And we're always happy to help. We want more people making mixed reality games. We want more people creating cool experiments in this space. the more we can help them, the more fun this space is. Right now it's a very lonely place to be, for us right now, being the first really long mixed reality game, but we hope to see more. And to all the players that pick up Starship Home, if you run into any issues, we really appreciate your patience with us and the platform, and just reach out to us on all the places you can find us and we'll get you sorted.

[00:51:12.711] Kent Bye: Yeah, and I think the Starship Home, my experience of it, is that it's a really, truly cutting edge and creative breakthrough in terms of trying to really find the affordances of the medium of mixed reality. But also, it's been one of those experiences that has helped to shape the platform itself and all the different features that you as developers are needing to even have an experience like this. And so in that respect, I feel like it's well worth anybody who's in the industry to check it out just to see the experience and what resonates with folks, but also to push the edges of finding the bugs and reporting back to you and then trying to get overall the platform better for not only you as creators of this specific experience, but for everybody in the industry to help co-evolve and develop this platform as it moves forward. But I certainly had a lot of fun playing through it. I'm looking forward to getting back home setting up my other starship and i think i'm gonna set up so that i have a little bit more of a cinematic 180 view of the helm where i can like sit back and get a fully immersive like view and arrange it a little bit different way after playing through about a third of it and strategies for how to properly arrange my plans and so i've got plans for my next starship home as i move forward but i had a lot of fun playing it so far and it was a real pleasure to get a chance to talk to you to hear a lot more about your process and uh how it all can be and yeah just thanks for sharing all your journey with me so thanks thank you ken it's been great talking with you Yeah, thank you, Kent. Thanks again for listening to the Voices of VR podcast. And I would like to invite you to join me on my Patreon. I've been doing the Voices of VR for over 10 years, and it's always been a little bit more of a weird art project. I think of myself as a knowledge artist, so I'm much more of an artist than a business person. But at the end of the day, I need to make this more of a sustainable venture. Just $5 or $10 a month would make a really big difference. I'm trying to reach $2,000 a month or $3,000 a month right now. I'm at $1,000 a month, which means that's my primary income. And I just need to get it to a sustainable level just to even continue this oral history art project that I've been doing for the last decade. And if you find value in it, then please do consider joining me on the Patreon at patreon.com slash voices of VR. Thanks for listening.

More from this show