Living Cities is a digital twin XR startup announced on May 26th, 2022 with an article titled “Reality is Scarce…and The Metaverse is infinitely abundant.” that shares a list of “9 principles for connecting the real & virtual.” The founders of Living Cities include XR luminaries Matt Miesnieks (who sold his previous startup 6D.AI to Niantic), John Gaeta (who did visual effects on The Matrix, worked on the HoloLens, co-founded Lucasfilm’s ILMxLAB, & was a senior Vice President at Magic Leap), as well as Dennis Crowley who co-founded the geospatial social networks Dodgeball and Foursquare.
I had a chance to catch up with Miesnieks & Gaeta on July 6th to unpack each of their principles for connecting the virtual and the real, including how they’re trying to capture the spirit of a place and lore of specific locations digitally allowing the physical and virtual realms to be combined in unique ways. They’re still working on their initial demo, and so there is a lot of reading between the lines of their guiding philosophical principles to understand what exactly it is that they’re building. But they have a lot of deep ideas for what the next steps should be in building out an AR Metaverse that blends world scanning technology with various social XR communication features and self-expression tools.
See below for the audio interview and a full rough transcript.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
MIESNIEKS: I’m Matt Miesnieks, and I’m the CEO and founder of a company called LivingCities.xyz. We are building a digital twin of the real world and working to figure out how to make that a living copy of a real place. It’s a new company. We haven’t really launched or said anything about our product as yet. And the company kind of builds on what I have been doing over the last 12 years or so in augmented reality.
Most recently, my company, 6D.AI was acquired by Niantic and 6D had developed some technology to crowdsource a 3D map of the world so people can capture 3D scenes on their phones and build them into a map. And Living Cities is kind of taking that one step further in the sense of what happens when you’ve captured this map. You know, what can you see? What can you do and how do you actually use it?
GAETA: My name is John Gaeta, and (laughs) what do I do in the realm of spatial computing? Well, I do a lot of personal computing in space. And I think a lot about how other people might be doing that. I think today we’re here to talk about, you know, how to harness the potential of people in space and project that into new forms. So it’s a long answer to that one is perhaps another chat.
BYE: Okay. Yeah, maybe for each of you, you could give a bit more context as to your background and your journey into doing this work that you’re doing now with Living Cities.
MIESNIEKS: Sure. My background goes, you know, I’ve been in tech my whole life. I started just out of college or in college working on the Internet like a while on the Internet and helped travel around Asia, building that out. Then I spent about a decade in Mobile for the company that invented the mobile web browser, you know, started out technical, ended up commercial.
And about 12, 13 years ago now, I was thinking about what comes after mobile and landed on augmented reality as being the direction that everything was going to head, which I still feel is correct, but I was definitely way, way too early. I probably should have waited ten years or so before jumping in. And yeah, you know, during AR, you know working in AR I’ve been very interested in infrastructure and enabling technologies and what are a lot of those platforms and technologies that are going to make all these amazing AR experiences possible?
GAETA: My background is I began in cinema known for designing visuals and concepts in the original Matrix trilogy, where I worked with a number of colleagues in some methodologies that at the time were pretty experimental. But were created in a way that we thought might be relevant in future times when things like virtual reality might, might actually be plausible.
After Matrix Trilogy, I did more cinema with the Wachowski’s and then eventually started to see a lot of colleagues leave to move towards real time graphics and more experimental media. I was intrigued by a lot of what they were doing. Some of them were going into gaming. Some were starting to go into the labs of Silicon Valley and such.
So I started a period of time where I was experimenting with new media. I worked on things before the Kinect, but human interface. And then following that, the HoloLens — the pre-HoloLens with Microsoft. And after that I went back to film sort of entertainment worked for Lucasfilm as they were about to launch a next generation of Star Wars.
I helped begin something called ILMxLAB, which is one of the first immersive entertainment labs that was pushing some boundaries in real time graphics. And there was a lot of exploration on behalf of Disney and Future Disney ideas and Star Wars universe. Following that, I was SVP at Magic Leap for a couple of years, which was interesting, and I could write probably two books about that particular experience. It was like living in the future in a way, and very interesting sort of early foundational thinking from there.
And that’s around the time just around that Lucasfilm and Magic Leap time, Matt and I became friends. We’re exchanging a lot of theories and appreciation of each other’s work and vision. And since that time, we found reasons to come together along with Dennis on some ideas that were early in those days, and they seem right for us now. So I guess I’ll stop there.
BYE: And you just mentioned Dennis Crowley, one of the co-founders of Foursquare, which was a geospatial located, check-in app that was a pioneer in the space of trying to map out the social layer on top of where people were located. So Matt maybe could pick up — since Dennis is not here — how you and John and Dennis collectively like — a little bit of the origin story of [yeah] where you’re at with 6DAI and with John’s whole journey from Matrix to Magic Leap. So yeah maybe you can pick up what was a catalyst for Living Cities?
MIESNIEKS: Yeah, sure. I think what’s been awesome about working with John and Dennis is all of us have spent decades really, you know, looking at the same problem, but from three very different perspectives, you know. So, you know, in Dennis’ case, I met him through a mutual friend a couple of years ago and was really just interested in talking to him as a potential advisor to whatever I was going to do next.
You know, just as someone I was curious to know and as I described, you know, what we were hoping to achieve in living cities. He was really excited, said, Look, I’m negotiating my exit from Foursquare right now. I will remain as chairman, but I won’t have any day to day roles. And I’m thinking about what I want to do next.
And what you’re talking about is kind of the direction I always hoped, you know, Foursquare would get to. And he’s like, How can I be involved? And I’m like, “Do you want to join us and run product?” And he was like, “Yep, I’m keen.” So you know, it took about a year from that conversation until we finally had a company, but it was just great to tap into his — like he didn’t know really anything about AR or he hadn’t lived in that world at all.
But he’d spent his whole career figuring out how software intersects with the real world and how that can shape and influence our real world behaviors. And, you know, he has a sort of line about, you know, he wants people to get online in order to get offline. And that the way that, you know, these online services and experiences can actually — it’s not about getting sucked up into it.
You live your life online. It’s all about, you know, we have this whole life which is partly online, partly offline. And the online pieces should support and leverage and encourage also your offline part of your life. So philosophically, that was just so aligned with what we’re doing and obviously had a lot of experience running a large company and it was just amazing sort of having someone else on the team who’s kind of been through that as an entrepreneur.
And it’s just nice, you know, from my point of view, to have his experience behind us. And then in terms of how the three of us came together, it was, you know, like what John said, we’d been talking for five or six years, you know, about the same sorts of ideas, same sorts of vision. We talked about working together a little bit at 6D just before Niantic acquired us and stayed in touch and realized that, you know, we all wanted to do the same thing.
We just needed to turn that into a company and then put some product focus around it so that we can actually bring something out. And that idea of what happens when you’ve got a digital copy or a digital layer of the real world with, you know, I called it the AR cloud back in the 6D days. Nowadays it could be one little facet of the metaverse, which is that part of these virtual worlds that are mirror worlds of reality and the potential for all different sorts of novel interactions, novel economies, novel use cases start to get unlocked when you realize that a virtual avatar and a physical person can be in the same place at the same time.
BYE: Yeah. And if my memory serves me, I remember going to the Magic Leap LeapCon in like 2018. And I don’t know if it was you, John, that was on stage with Rony talking about the Magicverse.
GAETA: Yep.
BYE: Magic Leap was basically like ten startups all in one, doing lots of different things ahead of its time in many ways as an independent entity, trying to push forward the state of the art on so many different levels, and this idea of bringing the augmented reality out into the world and having these different layers of what Rony was calling the Magicverse.
But I’d love to hear some of your reflections of what got you excited about that Magicverse and how you see that you may be continuing that idea of trying to bring — or what was it about the Living Cities that was recapturing your imagination for where you wanted to go?
GAETA: I mean, yes, there’s a thread line through all of it. I mean, all of us I would say as long as we’ve been curious in these spaces and realms, each chapter, each relationship kind of leads you a little closer towards something. The idea of the magic verse, you know, even before the magic verse, Neal and I — well, Neal was on stage with us, right?
And so magic verse was kind of a product of Neal and Rony and myself thinking about stuff. And even before then, when I was experimenting at Lucasfilm, we got deep into trying to understand destinations, virtual destinations that are based off of places with real histories. You know, because we were thinking about the, the universe of Star Wars and how each place is special and has its own history and contains many different types of characters and relationships
it carries. It just goes quite deep. And the greatest of fictitious universes tend to do that. And fans thrive and engage deeply into those things. But oddly, those things often are based upon real world places and people and one sort of extracts from those things. So most fiction represents something that’s real and true in terms of people, places and events and such.
At any rate, you know, I sort of came into Magic Leap with a lot of thinking along those lines. If we could try to understand a sense of place and a destination based on the lore. I then went on to Magic Leap and had the luck of working with someone like Neal, you know, thinking about things like that. And Rony, who is a real deep thinker and incredibly instinctual and intuitive on these things as well, where it was pretty quick to imagine that you could have any one layer over the world at any given time and they would have to intertwine, which leads towards spatial computing is like in what way do you intertwine?
How do you intertwine? And the layer, of course, could be anything the mind could conjure from entertainment and creative to pure utility. You know, like, here’s my health layer, here’s my tourism layer on and on, right? So these could be endless layers created by infinite amounts of people over time. So the layers thing is coming. It’s coming still.
And the combination of those two things is stuck inside of me. And Matt was talking about many of the same things. You know, when we met years back, he understood the way that reality or the real world was going to couple to a virtual layer, was going to — it would need to be fused by way of understanding of not just the shapes of things, but the ongoings, what’s happening at that time.
So essentially the kinds of elements you would need to feed into a simulation of a place, right? What’s happening there? What is the light like? You know, what is the purpose of the place and the people within the place? And we always brought this back up as we talked over the years. What we do right is like people were wondering about augmented reality, to what end, you know, could it benefit people?
Could it amplify people in interesting ways? Could it lead to heightened capabilities? You know, in terms of expression? So we dragged all of that stuff from even before Magic Leap during the 6D days, the Magic Leap days into today. And I’ll let Matt sort of riff on this. But I think that there’s a lot of confusion, of course, about what the metaverse is.
We could talk about that endlessly. But the one thing that’s true is that a virtual container with no purpose or meaning or history or not defined by way of the people inside it and or community. And their purpose is like this is sort of a desert to wander through. But as soon as you have an understanding of place, which is defined by people, then suddenly the value starts to show itself.
And I guess I’ll throw it back to Matt with regard to that.
MIESNIEKS: Yeah. And Kent, you got any questions? You want to jump in with others? I can just ramble on for a good hour or two on this.
BYE: So right now — you published an article back on May 26 called “Reality is Scarce: And the Metaverse is Infinitely Abundant.” So you’re starting to be a part of this larger metaverse conversation, which John just spoke in some sense. And when I think about the differences between VR and AR, I think about is that in AR you’ve in the center of gravity of whatever existing context you’re in, that you’re using the virtual information to either modulate or subtly shift the context or maybe try to change the context of what you’re in.
But with VR, it’s a lot easier to do a complete context, which I could be at home. I go into an immersive experience. I could be at my doctor’s office, I could be on a date with my partner, I could be visiting family. And so the context there is much more of a stark context shift. So I see that there’s something with being grounded in this gravity of the existing context and being able to use the virtual layers to either connect people on a deeper layer or to maybe subtly shift whatever the context is and maybe create a new context, kind of a liminal space that doesn’t have an established context. So I’d love to hear —
MIESNIEKS: Yeah.
BYE: Some of your thoughts on where you start with the Living Cities and what context you bring in and how you start to iterate there?
MIESNIEKS: Yeah, well. One thing I’m really trying to avoid, I guess is the semantics like this is “Yeah, this is VR, this is AR, this is XR, this is spatial –.” We’re really trying to think about what is the user experience? And what is the potential of this product we’re trying to make? And you know, the device you use to look into this place could be anything, you know, it could be a phone, a browser, a VR headset or an AR headset.
What we think is most interesting for us is kind of what John was alluding to there. Like if you build a virtual place just from scratch, it’s really, really difficult. And John sort of educated me just how difficult it is to create a universe, a history, a law, a culture, all these things that this isn’t just something that exists for like VR space creators.
It’s everyone who’s ever made a film or written a novel or anything that sort of create this world. And it’s very, very difficult to do that from scratch. Nearly every time someone does it, John said before. It’s rooted in reality, somehow in the human condition somehow. So what we’ve found really amazing, I guess, is that if you go to a real place and say, look, we’re going to somehow replicate this place and bring it online, you get all of that history and culture and the law and the types of clothes you wear there and the type of music you listen to and the type of people that go there.
Kind of all of that you get for free, you know, because it’s a real place. You know, it’s had – hopefully you pick a place that’s got a lot of history and it’s interesting, you know, in reality. And when you can then bring that online, you get this kind of different thing. Like it’s not quite AR. It’s not quite VR. It’s definitely aligned and connected with reality and it’s essentially being updated in real time by what’s going on in the real world.
But it could be experienced entirely, virtually, and it could be tweaked and modified and adapted and changed around with all the tools that are available to the virtual world creation. So that sort of concept of like bringing the world online in that way is kind of where we’re zeroing in on and building out. It’s because we think that if you can do that, you have all seen like photogrammetry captures of places or 360 videos of real places and you go in there and you look around for a bit and that’s it.
You’re kind of done. But this idea of like, how do you bring that place to life? How do you make it feel like you’re really there? How do you tap into that spirit of the place? How do you connect to the people that are in that place? All these aspects of it, the magic that no one’s ever really – we haven’t seen anyone do this before and we think it’s something potentially really magic and big that could be unlocked if we can solve it.
BYE: Yeah, I guess the question that comes up is the matter of scale because I think of something like Google Earth VR, which has replicated all of the entirety of the Earth with different types of coverage that is even in Google Maps. It can’t be the same resolution universally everywhere because resource limits and don’t need high resolution things if it’s just an empty cornfield.
So you have the whole range of the entire world. So where do you start with creating a digital twin? Do you start with urban cities? Do you try to recreate an entirety of the urban cities? Do you try to take, like an Ingress approach where you pick areas of interest and start to organically build out based upon whatever the early adopter users are? Or how do you start to boil the ocean in that sense?
MIESNIEKS: Yeah, well. You, you, you try not to is the main thing. You know, one problem with building an AR product, any type of software product, is this idea of population density, you know, like Pokémon or like Niantic with these global games, how do you put a Pokémon on every street corner in every town, and how do you get more than one player in your neighborhood?
You know, that’s a really difficult problem to solve. And so much of just if you take this idea of AR is something that I look through something and I see something digital in my physical world right now, you’re going to have that population density issue. We’ve kind of flipped that on its head and we’re thinking about, rather than figuring out How do I get content to everyone on Earth, or how do I get everyone on Earth to the content?
And so we’re consciously choosing a starting location that’s in an urban environment that is very diverse and creative and reasonably well known globally, and tapping into the specialness of that place and trying to bring that to the web, to the metaverse. You know, one thing I often talk about when anyone brings up scale and AR in the same paragraph is, you know, in all my years of like, I’ve never met anyone who has ever said my AR app is too popular on iOS.
How do I put it to Android? Or It’s so successful here I’m struggling to scale. Like everyone’s always had the problem of how to actually get engagement and somewhat to come back to it repeatedly. And that’s the problem we’re really going after. We think that if we can get that working in one very small constrained location, the question of like bringing that to multiple locations, whether they’re public spaces or private spaces, we can then start replicating that.
And then, as I know from my 6D experience, the potential to crowdsource and build those maps of the places, you know, 3D realistic maps is still not quite there today to do that in the highest possible quality, but it’s coming pretty fast and a lot of the mapping infrastructure is already in place with everything from open street maps to every major platform that’s out there.
And they’re all working towards building 3D versions of their maps, but no one knows what to do with those maps once they exist exactly.
BYE: And John, did you have any thoughts on that?
GAETA: Of course, I mean, we’ve been — We’ve been inside those thoughts for a lot of it here. Yeah. I mean to, again, to reinforce some things that Matt just said, trying to boil the ocean is really going to be a slow, incremental exploration. And it probably is the domain of the giant map companies. Right, to try to do that.
But what we’re talking about is more of a strategy, a creative strategy, social strategy of going compact but deep. Back to trying to understand, you know, what’s inside the fabric of a universe? What is actually the spirit of a place? We use that term a lot. And the spirit of a place generally is in this place, these types of people congregate to do these things, and they’ve done that for, you know, in the case of the real world many, many years ago.
So in more older parts of the world, it could be like a lot of years, centuries even, right? But to try to understand what happens in a compact area. And it’s mostly really about knowing the people. And we don’t want to just sort of suggest that our interest is creating a copy, precise copy, because, you know, interesting places and people and events, you know, appear in books and in movies and all sorts of other kinds of expressions of the same place.
You know, that we don’t have any rules among ourselves about how precise we really feel like replicating. We know that it’s possible to take a picture of you and your family, you know, in a place. And that’s an expression of you in a place. Right. And it’s framed by you. And you made a choice in how you did that.
So there are a lot of ways that one could reflect what’s happening, reflect the spirit of a place. There’s different media forms that could happen in. And so our interest is essentially reflecting the real world up into some form of itself, right? A virtual form of itself. But the form factor of the things being reflected could potentially fall anywhere on the spectrum of totally real and volumetric to totally expressive.
And to use that Star Wars example again, like if, for example, a bazaar in Tatooine is really based off of a similar type of place in Morocco, for example, right? You could look at it as like an abstraction of that place in Morocco, right? It’s a sort of a fanciful sort of abstraction of that place. But underneath it, you see the elements — right? — of the real place.
And that’s interesting, right? So you can also think about it as you can fall on a spectrum of like it’s completely real to it’s an expressive or abstraction of the real. So these things are all in balance, I think. Right? And interesting, just like all different types of social media that exist today. I mean, you can put something literal up there and, you know, sort of something you’re sharing about yourself. Or you could be creative with it and you can abstract on it, but it’s still an expression of yourself.
So we love this idea that future social media will probably again still be people expressing themselves. But in this new virtual destination type of container. And potentially, again, interesting people in places export things outward like, “Hey, this is what they do in Shibuya, Japan.” It’s very interesting. I see it in videos and on the Internet and pictures and all that stuff.
Hey, that’s really cool. I’ll be influenced by that and I’ll make some art or create something, and it was influenced by that. And I put that out there now. Right? And so there’s sort of a progression, right? So the influence came in or the expression came to you, you did something and then you’re trying to participate in a way right from afar, out of appreciation or inspiration.
That stuff can happen. So there is this relationship that can happen between those that are there at the source of something and those that are everywhere else. That’s a big thing that we’re wondering about, right? How do things reflect in both directions?
BYE: Yeah. And in the Medium article, I know, Matt you had written up nine different principles that I want to bring up and dig into. And John just mentioned one of them, the spirit of the place, and also talking about the virtual and the real reflection of each other. But before we dig into those principles and values, I did have one clarifying question, which is as we’re talking about this, the question for me that comes up is, is this something that you’re starting with recreating these different spaces just from the outside, publicly accessible places?
Or are you actually doing any internal depictions of these spaces, which is there’s these boundaries between public property and private property. And if you are sticking with stuff that’s from publicly accessible view or if you’re going inside of any?
MIESNIEKS: Yeah, we’re definitely focusing on a public place to start with. You know, one of the things that we found is we’re still finding, you know, is that when we’re trying to describe what we’re doing in words, particularly to anyone who’s not familiar with VR or AR or even sometimes if they are, it’s a very abstract, amorphous concept for people to get their heads around.
And so we’re working to soon, you know, have a demoable example of what we mean. And so able will look at it go, “Oh, I get it. That’s what you’re talking about.” If we were to go to a private place, you know, a high-profile concert venue or theme park or stadium or something, we’d need to be able to show them what we do before they would really buy into what we would want them to buy into.
So this first site is as much I wouldn’t call it a proof of concept. It’s definitely gonna be a product, but it’s very much a stake in the ground of saying, Look, this is what we can do, and once it’s working here, we can do that for other places as well. If you’re in a private place or even a different country, you know there’s different laws around what you can capture and record in real time of what’s going on in that place.
You know, everything from security cameras to swiping your wrist on entry doors and all that sort of stuff. Potentially we could take all of that data in and use it to create a very accurate, real time simulation of everything that’s going on in that real place. But the ability to do that is shades of gray from completely public in an environment that’s got pretty strict laws to a private place where potentially you could do anything.
BYE: Yeah, that makes sense. I think up until the point where I’m able to see it, there’s a lot of unanswered questions and stuff that I’ll probably understand a lot more once you have created that proof of concept and released it. So I think with the time we have remaining, it might be worth just going through some of these principles that you’ve written, because I think this article you wrote out, that was part of the reason why I reached out, because I thought some deep thinking about trying to think about the underlying philosophical principles of what’s going to be really driving what you’re going to be moving forward.
And so I’d love to hear some extrapolation of these different principles. We already talked a little bit about the spirit of place, but I’d love to hear a little bit more of a riffing on some of these nine principles that you brought up here.
MIESNIEKS: Sure. Yeah. I mean, either of us can sort of go into these. Let’s start with reality is scarce. You know, everyone’s thought that it’s a big world. It’s is the huge world we live in. But what’s interesting to me is if you go to a place in the real world, you go to the center of Times Square in New York. In the physical world, there’s only one thing in one place at a particular time.
And when you’re looking at a virtual space, you can potentially have infinite things represented in that same space. You know, that’s all the different layers that we’ve talked about. So that idea of scarcity and how that connects to the abundance of being online is really interesting — from an economic point of view because we’re not quite sure exactly how that will play out.
But the idea of one physical person in a place, maybe 100,000 virtual people in the same place gives a really interesting sort of imbalance between what power does the virtual person have that the physical person doesn’t have, and vice versa. And that interplay between scarcity and abundance and value is ripe for us to explore. I don’t know, John, did you want to pick another one? Just to…
GAETA: Okay, we can play ping pong. So, you know, “people are the killer app.” That’s always been true, isn’t it? Since the beginning of time before they use the word app. We can experiment and explore all sorts of fantastical things to do, places to be in layers of reality. But at the end of the day — I think it’s proven time and time out — what matters is who you’re with or who you’re interacting with and what is it that is happening between either, right?
So at the end of the day, it’s got to be the center, right? It’s the sun is the center of the universe. And so we need to appreciate and understand that if people are going to stay engaged in experimental realities and mirror worlds and fantastical things like this, it has to begin and end with the relationship to people, the engagement of people.
So we’re trying to orient things in that particular way because the idea of the metaverse is an abstraction. It can be like infinite universes. They still are just places, but they’re empty places. They’re empty places until they’re defined by the people within them, and the things that they’re doing. So that’s the point of that. People are the sun at the center of the universe. We’re focusing on that and that’s why we feel assured in how we’re prioritizing things.
MIESNIEKS: And it also one thing I just add to that, it’s probably my biggest mistake, the thing I missed with AR and that I’d been working for years, you know, looking at world-facing AR, like look through the phone or through the glasses to see the content in the world. And what was successful was, you know, face filters and it was that ability to kind of how do I point the camera at myself?
And it’s about people and how do I augment people, not rooms, you know? And so, yeah, that really drove home to me that people should be the center of all the interactions that we’re exploring. We talked a bit about Spirit of Place. We talked a bit about how lore was built into real places, but the next one about communication, social and self-expression being the same thing.
What we see, I guess you’ll see today is nearly everything that is at least the virtual world half of the metaverse as opposed to the crypto off of the metaverse. Nearly all of those products are entertainment or gaming, but they’re hard and you know, that’s good. But something I always believed is that when you look at markets and market opportunities and people and what we do with our time and our lives and our energy, like we like to be entertained, but we far more than that, we like to communicate with each other.
And, you know, we spend far more money on communicating, you know, everything from our mobile phone bill to traveling than we do entertaining ourselves. And so I think when we look at our piece of the metaverse that we’re trying to create, we really wanted to tap into this idea of how do you enable people to communicate? Because that is social.
You know, Facebook is a communication platform. They just rebranded it as a social network. And a lot of that communication is just how do we express who we are? And when you go through a platform shift, that use case doesn’t change. All that changes is that you now have some new forms of media, you know, in this case 3D media, to start doing those same things.
And what is this new form of media? How does that let you express yourself? Will communicate in a way that wasn’t possible before. So we’re farmers and that as a use case than we are in building like a game like Niantic has done.
GAETA: Which is super exciting because we’re like in the midst of a near paradigm shift in capture and generation of things. So self-expression is really going to evolve and move fast in the next year or years. And it’s a very exciting area for us. And we’re lucky, right, to have begun now, right? Because we don’t have a lot of things that we’re bound to a lot of legacy, that we’re bound to with regard to what’s the new form of social media and self-expression.
So we’re kind of free. Oh, we’re lucky. “AR is the wrong place to start.” Just to mention that. Like, basically, I don’t think anybody who listens to this incredible program of yours, Kent, would argue, right, that augmented reality is really going to itself be a paradigm shift in the way that we perceive and consume content. But right now, it’s a window.
It’s the way we’re looking at AR and VR or any device, any screen. Is that they’re windows upon the actual thing that matters. And the thing that matters is what it is that people are coming to and engaging with and that’s where our focus has to go, because we’ve all been spending a lot of time, you know, joyfully inventing powerful windows onto the thing that matters.
So AR will come. VR will come. You know, the spatial web is going to come. But it matters what’s actually before our eyes at all. So that’s what that means.
MIESNIEKS: Yeah. The next one with “The real world being a dynamic living place.” We’ve talked a bit about that. Like if you want to capture this feeling, you know the spirit of the place, if you want to tap into the people that are inhabitants of that place, this digital twin — like the idea of a digital twin, like in architecture, engineering, whatever, is always like a static model.
And we think that’s just a first step. Like it’s nowhere near enough. So you need to somehow bring all those dynamic aspects of reality up into the virtual twin. Not much else to say there, but it’s closer to a simulation than it is to a digital twin.
GAETA: Yes. And a simulation that can have its parameters tweaked at some point. So “Get on the metaverse to get off the metaverse.” Again, like the honeymoon of virtual worlds and all of this stuff at a certain point, most people will realize that real life was always so much more interesting and so much more stranger than fiction and beautiful than any digital experience that we could have.
However, what we were sort of implying there is that like great art or something that can catalyze your interest in pursuing something, you know, that is a function that can be served by things like the metaverse. So this is more about the remote co-presence aspect. We could get a taste of a place and all that goes with it. And really, hopefully what it would cause is a drive for us to sort of get to the real place.
So the metaverse could serve to catalyze greater appreciation and engagement of the real world and reality itself. And the weird thing about essentially working in these areas, in these mediums is that the deeper in you go — and visual effects did this before world stuff — but the deeper you go into trying to replicate reality, the more it makes you actually appreciate and look at reality and engage reality harder.
Perceive it like to actually not just sort of let it go by, but actually try to use your senses to perceive what’s happening around you and appreciate it in a deeper way. So that’s hopefully what will happen is as we dabble in the metaverse, it’s going to make us get out of the metaverse and really take in our real lives or real places a lot more than we realize.
Not for all. Maybe some people get lost. But I do think a lot of people will suddenly look at reality freshly after dabbling in the metaverse.
MIESNIEKS: And the last phrase is about reflections. That’s a word we use a lot. Like how do this virtual world in the real world like reflect into each other? And I think the phrase mirror world is kind of a misnomer in that it kind of implies like a perfect literal twin of the real place, like looking in a perfect mirror at yourself.
And, you know, it’s not too difficult to imagine once you go down that path, it’s like it’s impossible to do. How do you get every blade of grass? Perfect. So the interesting thing is, if you think about reflections as a term, it has a much broader meaning than just a literal mirror, and it can mean anything, you know, like a funhouse mirror can give you like a slightly warped reflection.
You can have a pond or a lake where you get a reflection that’s kind of affected by the ripples and the color of the water right through to something like an impressionistic painting, like a Monet painting of his water lilies that is totally impressionistic, but it is definitely like you go to the place that he painted and you can tell it’s the same place and it is an interpretation of that real place.
And so that broader definition of reflections is really what we’re leaning into, and we’re not going for the whole pixel perfect real time copy of the Atom. But more a sense of what does it feel like? And how does it feel similar? But it can also be a little bit different as well. And create something that’s more interesting and more engaging or more fun or more peaceful, you know, whatever you feel it needs to be. Once, you know, all these creative tools are out there.
BYE: Awesome. Well, I think that actually gives me a really good sense of your intentions and your philosophical principles that you’re basing your company on and from there and extrapolating out and building it and really excited to see where it goes as it continue to evolve and progress down the path of building the Living Cities. So I guess just to wrap things up, I’d love to hear from each of you what you think the ultimate potential of these immersive technologies of virtual reality, augmented reality, or just the blending of the virtual and the physical together, what you think that might be able to enable?
GAETA: Well I’ll go first and Matt can like land the plane. How long have you been doing these podcasts, Kent?
BYE: It’s been over eight years now.
GAETA: Yeah. They’re amazing, and I’ve heard many of them. And so much has happened and changed and evolved. And it’s like, this is a road, right? We’re like trying to get on a road. And the road is a long road. It might not ever end. You know, imagine this road 20 years from now, 50 years from now. Right? But the idea of the road we’d like to get on is one that’s actually intertwining the real world and reality with the virtual, a sort of an extension and amplification of the real.
And these technologies that we have — it’s just remarkable and impressive, you know, what’s actually been done in the last decade. Incredible, really. Everything from like computer graphics depictions of real things to ways of seeing and being inside and touching these things in a sense. But we’ve literally just gotten a bunch of colors of paint and some brushes just in the last few years, couple of years.
You can make an argument that like this year and the next few years are like the beginnings — the beginning now. We’re finally almost ready to get going. So the answer to the question is it’s a road we’re stepping on, I think everyone’s going to step on it. And we’re really looking forward to collaborating with people, having others really tell us what they want to do, what we should do. I mean, we just have a general sense, right, of the elements of the formula of this alchemy. But we absolutely need people to sort of mix these things together with us and learn from them.
MIESNIEKS: Yeah, for me, yeah again, if I think long term and get away from products and devices and things. It really is about giving people superpowers to let them influence and change their perception of reality. That’s kind of at the heart of it. And hopefully those powers are used for good or at least incentivized for good, and they make the world a better place.
BYE: That’s awesome. Well, is there anything else that’s left unsaid that you’d like to say to the broader immersive community?
MIESNIEKS: We want to talk all about our product, but we can’t yet. We’re just going to wait until we get something to show off that we’re really looking forward to starting to show it off internally. It’s super exciting. Just in the last week, you know, we had some milestones where everything kind of is hooked together now and we’re getting a sense of what it’s going to look and feel like. So, you know, for me, I just can’t wait to start talking about that.
BYE: And well, John and Matt, thanks so much for joining me to help give a little bit of a sneak peek. I know it’s still early days and we’ll have a lot more to talk about once you are able to show the world what you’ve been creating. But I think I love to just to hear your journeys up to this point and to get a little more context as to what’s inspiring you and the different principles that you’re building upon. So really looking forward to seeing where you take this. So yeah, thanks again for joining me here on the podcast.
MIESNIEKS: Yeah, thank you. Thanks for having us.
GAETA: Of course, it’s great to be here.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So today's episode, I'm covering Living Cities XYZ and the Medium article that they published on May 26, 2022. The title is Reality is Scarce and the Metaverse is Infinitely Abundant. And the startup has got some big hitters within the context of XR. Matt Meissnick, he's one of the founders of 60AI, which was acquired by Niantic. You have John Gata, who did special effects within The Matrix. He then went on to work at Microsoft on some of the Mahalo and stuff, and then one of the co-founders of IMLX Lab, on into Magic Leap as the Senior Vice President. And then Dennis Crowley, who co-founded Foursquare, which is a pioneering geospatial-located application where you do check-ins. So, from their perspective together, they all come together to create this digital twin of physical reality. And what is the representation of that reality that they're trying to capture? and device agnostic in terms of how you're accessing all these different layers of lore and the spirit of a place. We're trying to capture not just a digital twin, but actually a living simulation that's dynamic and is trying to, in some ways, create this social layer on top of reality and give multiple ways of people access it. It's a little bit ambiguous as to exactly what they're building yet, since they haven't released any demos, but they released this paper giving all the different principles, and through the course of this conversation, trying to suss out about where they're at now and where they're going. Also, I happened to record this interview right before I went on vacation for nine days into Alaska. It was a family trip. I was thinking a lot about going into these different places and thinking about what is the spirit of this place, and it just really influenced my thinking about how do you define a place and the spirit of a place and the character of a place. and what the stories and lore are behind that. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Matt and John happened on Wednesday, July 6th, 2022. So with that, let's go ahead and dive right in.
[00:02:06.691] Matt Miesnieks: I'm Matt Misenecks. I'm the CEO and founder of a company called livingcities.xyz. We are building a digital twin of the real world and working to figure out how to make that a living copy of a real place. It's a new company. We haven't really launched or said anything about our product as yet. And the company kind of builds on what I have been doing over the last 12 years or so in augmented reality. Most recently, my company 6D.ai was acquired by Niantic and 6D had developed some technology to crowdsource a 3D map of the world so people can capture 3D scenes on their phones and build them into a map. And Living Cities is kind of taking that one step further in the sense of what happens when you've captured this map. You know, what can you see? What can you do? And how do you actually use it?
[00:03:01.858] John Gaeta: My name is John Gaeta. And what do I do in the realm of spatial computing? Well, I do a lot of personal computing in space. And I think a lot about how other people might be doing that. I think today we're here to talk about how to harness the potential of people in space and project that into new forms. So a long answer to that one is perhaps another chat.
[00:03:30.563] Kent Bye: Okay. Yeah. And maybe for each of you, you could give a bit more context as to your background and your journey into doing this work that you're doing now with Living Cities.
[00:03:39.751] Matt Miesnieks: Sure. Um, my background goes, you know, I've been in tech my whole life. I started just out of college or in college working on the internet, like the wireline internet and helped travel around Asia, building that out. Then I spent about a decade in mobile for the company that invented the mobile web browser, started out technical, ended up commercial. And about 12, 13 years ago now, I was thinking about what comes after mobile and landed on augmented reality as being the direction that everything was going to head, which I still feel is correct, but I was definitely way, way too early. I probably should have waited 10 years or so before jumping in. And yeah, during AR, working in AR, I've been very interested in infrastructure and enabling technologies and what are a lot of those platforms and technologies that are going to make all these amazing AR experiences possible.
[00:04:32.919] John Gaeta: My background is I began in cinema, known for designing visuals and concepts in the original Matrix trilogy, where I worked with a number of colleagues in some methodologies that at the time were pretty experimental, but were created in a way that we thought might be relevant in future times when things like virtual reality might actually be plausible. After Matrix Trilogy, I did more cinema with the Wachowskis and then eventually started to see a lot of colleagues leave to move towards real-time graphics and more experimental media. I was intrigued by a lot of what they were doing. Some of them were going into gaming. Some were starting to go into the labs of Silicon Valley and such. So I started a period of time where I as well was experimenting with new media. I worked on things before the Kinect, but a human interface, and then following that, the HoloLens, the pre-HoloLens with Microsoft. And after that, I went back to film, sort of. Entertainment worked for Lucasfilm as they were about to launch a next generation of Star Wars. helped begin something called ILMxLAB, which is one of the first immersive entertainment labs that was pushing some boundaries in real-time graphics. And there was a lot of exploration on behalf of Disney and future Disney ideas. and Star Wars universe. Following that, I was SVP at Magic Leap for a couple of years, which was interesting. And I could write probably two books about that particular experience. It was like living in the future in a way, and very interesting, sort of early foundational thinking formed there. And that's around the time, just around that Lucasfilm and Magic Leap time, Matt and I became friends. we're exchanging a lot of theories and appreciation of each other's work and vision. And since that time, you know, we found reasons to come together along with Dennis on some ideas that were early in those days, and they seem right for us now. So I guess I'll stop there.
[00:07:01.480] Kent Bye: And you just mentioned Dennis Crowley, one of the co-founders of Foursquare, which was geospatial located check-in app that was a pioneer in the space of trying to map out the social layer on top of where people were located. So Matt, maybe you could pick up, since Dennis is not here, how you and John and Dennis collectively, like a little bit of the origin story of who you are. where you're at with 60 AI and with console journey from the matrix to magically. So yeah, maybe you can pick up what was the catalyst for Living Cities. Yeah, sure.
[00:07:34.108] Matt Miesnieks: I think what's been awesome about working with John and Dennis is all of us have spent decades really, you know, looking at the same problem, but from three very different perspectives, you know. So, you know, in Dennis's case, I met him through a mutual friend a couple of years ago and was really just interested in talking to him as a potential advisor to whatever I was going to do next, you know, just because someone I was curious to know. And as I described what we were hoping to achieve with Living Cities, he was really excited. He said, look, I'm negotiating my exit from Foursquare right now where I'll remain as chairman, but I won't have any day-to-day roles. And I'm thinking about what I want to do next. And what you're talking about is kind of the direction I always hoped Foursquare would get to. And he's like, how can I be involved? And I'm like, do you want to join us and run product? And he was like, yep, I'm keen. So, you know, it took about a year from that conversation until we finally had a company. But it was just great to tap into his, like he didn't know really anything about AR or hadn't lived in that world at all, but he'd spent his whole career figuring out how software intersects with the real world. and how that can shape and influence our real world behaviors. And he has a line about he wants people to get online in order to get offline. And the way that these online services and experiences can actually, it's not about getting sucked up into it and you live your life online. It's all about, you know, we have this whole life, which is partly online, partly offline, and the online pieces should support and leverage and encourage also your offline part of your life. So philosophically, that was just so aligned with what we're doing. And obviously you had a lot of experience running large company, and it was just amazing sort of having someone else on the team who's kind of been through that as an entrepreneur. And it's just nice from my point of view to have his experience behind us. And then in terms of how the three of us came together, it was like John said, we'd been talking for five or six years about the same sorts of ideas, same sorts of vision, talked about working together a little bit at 6D just before Niantic acquired us. and stayed in touch and realized that we all wanted to do the same thing. We just needed to turn that into a company and then put some product focus around it so that we can actually bring something out. And that idea of what happens when you've got a digital copy or a digital layer of the real world with, you know, I called it the AR cloud back in the 60 days. Nowadays, it could be one little facet of the metaverse, which is that part of these virtual worlds that are mirror worlds of reality. And the potential for all different sorts of novel interactions, novel economies, novel use cases start to get unlocked when you realize that a virtual avatar and a physical person can be in the same place at the same time.
[00:10:36.995] Kent Bye: Yeah. And if my memory serves me, I remember going to the Magic Leap LeapCon in like 2018. And I don't know if it was you, John, that was on stage with Rony talking about the magic verse. Yeah. Magic Leap was basically like 10 startups all in one doing lots of different things ahead of its time in many ways as an independent entity, trying to push forward the state of the art on so many different levels. And this idea of bringing the augmented reality out into the world and having these different layers of what Roni was calling the magic verse. But I'd love to hear some of your reflections of what got you excited about that magic verse and how you see that you may be continuing that idea of trying to bring, or what was it about the living cities that was recapturing your imagination for where you wanted to go?
[00:11:21.768] John Gaeta: I mean, yes, there's a thread line through all of it. I mean, all of us, I would say, as long as we've been curious in these spaces and realms, each chapter, each relationship kind of leads you a little closer towards something. The idea of the magic verse, you know, even before the magic verse, Neil and I, oh, Neil was on stage with us, right? And so magic verse, was kind of a product of Neil and Rhody and myself thinking about stuff. And even before then, when I was experimenting at Lucasfilm, we got deep into trying to understand destinations, virtual destinations that are based off of places with real histories, you know, because we were thinking about the the universe of Star Wars and how each place is special and has its own history and contains many different types of characters and relationships. It just goes quite deep and the greatest of fictitious universes tend to do that and fans thrive and engage. deeply into those things. But oddly, those things often are based upon real world places and people, and one sort of extracts from those things. So most fiction represents something that's real and true in terms of people, places, and events and such. At any rate, you know, I sort of came into Magic Leap with a lot of thinking along those lines, if we could try to understand a sense of place and a destination based on the lore. And then went on to Magic Leap and had the luck of working with someone like Neil, you know, thinking about things like that. And Roni, who is a very deep thinker, incredibly instinctual and intuitive on these things as well. It was pretty quick to imagine that you could have any one layer over the world at any given time, and they would have to intertwine, which leads towards spatial computing. It's like, in what way do you intertwine? How do you intertwine? And the layer, of course, could be anything the mind could conjure from it. entertainment and creative to pure utility. You know, like here's my health layer. Here's my tourism layer on and on. Right. So these could be endless layers created by infinite amounts of people over time. So the layers thing is coming. It's coming still. And the combination of those two things is stuck inside me. And Matt was talking about many of the same things. You know, when we met years back, we understood the way that reality or the real world was going to couple to a virtual layer was going to, it would need to be fused by way of understanding of not just the shapes of things, but the ongoings, what's happening at that time. So essentially, the kinds of elements you would need to feed into a simulation of a place, right? What's happening there? What is the light like? What is the purpose of the place and the people within the place? We always brought this back up as we talked over the years, what we do, right? It's like people were wondering about augmented reality to what end, you know? Could it benefit people? Could it amplify people in interesting ways? Could it lead to heightened capabilities, you know, in terms of expression? So, you know, we dragged all of that stuff from even before Magic Leap, during the 60 days of Magic Leap days, into today. And I'll let Matt sort of riff on this. But I think that there's a lot of confusion, of course, about what the metaverse is. We could talk about that endlessly. But the one thing that's true is that a virtual container with no purpose or meaning or history or not defined by way of the people inside it and or community and their purposes, like this is sort of a desert to wander through. But as soon as you have an understanding of place, which is defined by people, then suddenly the value starts to show itself. And I guess I'll throw it back to Matt with regard to that.
[00:15:42.499] Matt Miesnieks: Yeah. And now, Kent, you got any questions you want to jump in for others? I can just ramble on for a good hour or two on this.
[00:15:50.550] Kent Bye: So right now you published an article back on May 26 called Reality is Scarce and the Metaverse is Infinitely Abundant. And so you're starting to be a part of this larger metaverse conversation, which John just spoke in some sense. And when I think about the differences between VR and AR, What I think about is that in AR, you're in the center of gravity of whatever existing context you're in, and that you're using the virtual information to either modulate or subtly shift the context, or maybe try to change the context of what you're in. But with VR, it's a lot easier to do a complete context switch. I could be at home. I go into an immersive experience. I could be at my doctor's office. I could be on a date with my partner. I could be visiting family. And so the context there is much more of a stark context shift. So I see that there's something with being grounded in the center of gravity of the existing context and being able to use the virtual layers to either connect people in a deeper layer or to maybe subtly shift whatever the context is and maybe create a new context, kind of a liminal space that doesn't have an established context. So I'd love to hear some of your thoughts on where you start with the living cities and what context do you bring in and how you start to iterate there.
[00:16:59.993] Matt Miesnieks: Yeah. One thing I'm really trying to avoid, I guess, is the semantics. Like this is AR, this is VR, this is XR, this is spatial. We're really trying to think about what is the user experience and what is the potential of this product we're trying to make. And, you know, the device you use to look into this place could be anything. You know, it could be a phone, a browser, a VR headset, or an AR headset. What we think is most interesting for us is kind of what John was alluding to there. Like if you build a virtual place just from scratch, it's really, really difficult. And John sort of educated me just how difficult it is to create a universe, a history, a lore, a culture, you know, all these things that this isn't just something that exists for like VR space creators. It's everyone who's ever made a film or written a novel or anything that's got to create this world. And it's very, very difficult to do that from scratch. Nearly every time someone does it, John said before, it's rooted in reality somehow, in the human condition somehow. So what we've found really amazing, I guess, is that if you go to a real place and say, look, we're going to somehow replicate this place and bring it online. you get all of that history and culture and the lore and the types of clothes you wear there and the type of music you listen to and the type of people that go there. All of that you get for free because it's a real place. Hopefully you pick a place that's got a lot of history and is interesting in reality. And when you can then bring that online, you get this kind of different thing. Like it's not quite AR, it's not quite VR. It's definitely aligned and connected with reality and it's essentially being updated in real time by what's going on in the real world. but it could be experienced entirely virtually and it could be tweaked and modified and adapted and changed around with all the tools that are available to the virtual world creation. So that sort of concept of like bringing the world online in that way is kind of where we're zeroing in on and building out. It's because we think that if you can do that, We've all seen photogrammetry captures of places or 360 videos of real places, and you go in there and you look around for a bit and that's it, you're kind of done. But this idea of how do you bring that place to life? How do you make it feel like you're really there? How do you tap into that spirit of the place? How do you connect to the people that are in that place? All these aspects of it are the magic that no one's ever really, we haven't seen anyone do this before. And we think it's something potentially really magic and big that could be unlocked if we can solve it.
[00:19:47.303] Kent Bye: Yeah, I guess the question that comes up is the matter of scale, because I think of something like Google Earth VR, which has replicated all of the entirety of the earth with different types of coverage that is even in Google Maps. It can't be the same resolution universally everywhere because resource limits and don't need high resolution things if it's just an empty cornfield. So you have the whole range of the entire world. So where do you start with creating a digital twin? Do you start with urban cities? Do you try to recreate an entirety of the urban cities? Do you try to take like an ingress approach where you pick areas of interest and start to organically build out based upon whatever the early adopter users are? Or how do you start to boil the ocean? In that sense.
[00:20:27.925] Matt Miesnieks: Yeah. Well, you try not to, is the main thing. You know, one problem with building an AR product, any type of software product, is this idea of population density, you know, like Pokemon or like Niantic with these global games. How do you put a Pokemon on every street corner in every town? And how do you get more than one player in your neighborhood? You know, that's a really difficult problem to solve. And So much of just, if you take this idea of AR as something that I look through something and I see something digital in my physical world right now, you're going to have that population density issue. We've kind of flipped that on its head and we're thinking about rather than figuring out how do I get content to everyone on earth, or how do I get everyone on earth to the content? And so we're consciously choosing starting location that's in a urban environment that is very diverse and creative and reasonably well-known globally, and tapping into the specialness of that place and trying to bring that to the web, to the metaverse. One thing I often talk about when anyone brings up scale and AR in the same paragraph is, in all my years of AR, I've never met anyone who has ever said, my AR app is too popular on iOS, how do I port it to Android? Or it's so successful here, I'm struggling to scale. Everyone's always had the problem of how do I actually get an engagement and someone to come back to it repeatedly? And that's the problem we're really going after. We think that if we can get that working in one very small constrained location, the question of like bringing that to multiple locations, whether they're public spaces or private spaces, we can then start replicating that. And then as I know from my 6D experience, the potential to crowdsource and build those maps of the places, you know, 3D realistic maps, is still not quite there today to do that in the highest possible quality, but it's coming pretty fast. And a lot of the mapping infrastructure is already in place with everything from open street maps to pretty much every major platform that's out there. And they're all working towards building 3D versions of their maps, but no one knows what to do with those maps once they exist.
[00:22:51.432] Kent Bye: Exactly. And John, did you have any thoughts on that?
[00:22:54.875] John Gaeta: Of course. I mean, we've been inside those thoughts for a lot of this year. Yeah, I mean, again, to reinforce some things that Matt just said, trying to boil the ocean is really going to be a slow incremental exploration, and it probably is the domain of the giant map companies, right, to try to do that. What we're talking about is more of a strategy, a creative strategy, social strategy of going compact, but deep back to trying to understand, you know, what's inside the fabric of a universe? What is actually the spirit of a place? We use that term a lot in the spirit of a place generally. is in this place, these types of people congregate to do these things. And they've done that for, in the case of the real world, many, many years. So in more older parts of the world, it could be like a lot of years, centuries even. But to try to understand what happens in a compact area. And it's mostly really about knowing the people. And we don't want to just sort of suggest that our interest is creating a copy, precise copy, because interesting places and people and events appear in books and in movies and all sorts of other kinds of expressions of the same place. that we don't have any rules amongst ourselves about how precise we really feel like replicating. We know that it's possible to take a picture of you and your family in a place, and that's an expression of you in a place, right? And it's framed by you, and you made a choice in how you did that. So there are a lot of ways that one could reflect what's happening, reflect the spirit of a place, There's different media forms that it could happen in. And so our interest is essentially reflecting the real world up into some form of itself, a virtual form of itself. But the form factor of the things being reflected could potentially fall anywhere on the spectrum of totally real and volumetric to totally expressive. To use that Star Wars example again, like if, for example, a bazaar in Tatooine is really based off of a similar type of place in Morocco, for example, you could look at it as like an abstraction of that place in Morocco. It's a sort of a fanciful sort of abstraction of that place. But underneath it, you see the elements of the real place. And that's interesting. So you can also think about it as, you can fall on a spectrum of like it's completely real to it's an expressive or abstraction of the real. So these things are all in balance, I think, right? And interesting, just like all different types of social media that exist today. I mean, You can put something literal up there and, you know, sort of something you're sharing about yourself, or you could be creative with it and you can abstract on it, but it's still an expression of yourself. So we love this idea that future social media will probably, again, still be people expressing themselves, but in this new virtual destination type of container. And potentially, again, interesting people in places export things outward, right? Like, hey, this is what they do in Shibuya, Japan. It's very interesting. I see it in videos and on the internet and pictures and all that stuff. Hey, that's really cool. I'll be influenced by that. And I'll make some art or I'll create something. And it was influenced by that. And I put that out there now, right? And so there's sort of a progression. Right? So the influence came in or the expression came to you, you did something, and then you're trying to participate in a way, right, from afar, out of appreciation or inspiration. That stuff can happen. So there is this relationship that can happen between those that are there at the source of something and those that are everywhere else. That's a big thing that we're wondering about, right? How do things reflect in both directions?
[00:27:29.867] Kent Bye: Yeah, and in the Medium article, I know, Matt, you had written up nine different principles that I want to bring up and dig into. And John just mentioned one of them, the spirit of the place, and also talking about the virtual and the real reflection of each other. But before we dig into those principles and values, I did have one clarifying question, which is, as we're talking about this, the question for me that comes up is, is this something that you're starting with recreating these different spaces just from the outside publicly accessible places, or are you actually doing any internal depictions of these spaces, which I guess there's these boundaries between public property and private property. And if you are picking with stuff that's from publicly accessible view, or if you're going inside of any.
[00:28:12.766] Matt Miesnieks: Yeah, we're definitely focusing on a public place to start with. One of the things that we found, I guess we're still finding, you know, is that when we're trying to describe what we're doing in words, particularly to anyone who's not familiar with VR or AR, or even sometimes if they are, it's a very abstract, amorphous concept for people to get their heads around. And so we're working to soon, you know, have a demo-able example of what we mean. And so people can look at it and go, oh, I get it. That's what you're talking about. If we were to go to a private place, you know, high profile concert venue or theme park or stadium or something, we'd need to be able to show them what we do before they would really buy into what we would want them to buy into. So this first site is as much a, wouldn't call it a proof of concept, it's definitely going to be a product, but it's very much a stake in the ground of saying, look, this is what we can do. And once it's working here, we can do that for other places as well. If you're in a private place or in a different country, you know, there's different laws around what you can capture and record in real time of what's going on in that place. You know, everything from security cameras to swiping your wrist on entry doors and all that sort of stuff. Potentially, we could take all of that data in and use it to create a very accurate real time simulation of everything that's going on in that real place. But the ability to do that is shades of gray from completely public, you know, environment that's got pretty strict laws to a private place where potentially you can do anything.
[00:29:44.903] Kent Bye: Yeah, that makes sense. I think up into the point where I'm able to see it, there's a lot of unanswered questions and stuff that I'll probably understand a lot more once you've have created that proof of concept and released it. So I think with the time we have remaining, it might be worth just going through some of these principles that you've created. Cause I think this article you wrote up, that was probably the reason why I reached out. Cause I thought some deep thinking about trying to think about the underlying philosophical principles of what's going to be really driving what you're going to be moving forward and So I'd love to hear some extrapolation of these different principles. We already talked a little bit about the spirit of place, but I'd love to hear a little bit more of a riffing on some of these nine principles that you wrote up here.
[00:30:24.042] Matt Miesnieks: Sure. Yeah. I mean, either of us can sort of go into these. I'll start with reality is scarce. You know, everyone says, look, it's a big world. It's a huge world we live in. But what's interesting to me is if you go to a place in the real world, you go to the center of Times Square in New York, In the physical world, there's only one thing in one place at a particular time. And when you're looking at a virtual space, you can potentially have infinite things represented in that same space. You know, that's all the different layers that we've talked about. So that idea of scarcity and how that connects to the abundance of being online is a really interesting from an economic point of view, because we're not quite sure exactly how that'll play out, but the idea of one physical person in a place and maybe 100,000 virtual people in the same place gives a really interesting sort of imbalance between what power does the virtual person have that the physical person doesn't have and vice versa. And that interplay between scarcity and abundance and value is ripe for us to explore. I know, John, do you wanna pick another one just to?
[00:31:35.241] John Gaeta: Okay, we can play ping pong. So yeah, people are the killer app. That's always been true, isn't it? Since the beginning of time, before they used the word app. We can experiment and explore all sorts of fantastical things to do and places to be in layers of reality. But at the end of the day, I think it's proven time in and time out. What matters is who you're with or who you're interacting with. And what is it that is happening between either, right? So at the end of the day, it's got to be the center, right? It's the sun at the center of the universe. And so we need to appreciate and understand that if people are going to stay engaged, in experimental realities and mirror worlds and fantastical things like this, it has to begin and end with the relationship to people, the engagement of people. So we're trying to orient things in that particular way, because the idea of the metaverse is an abstraction. It can be like infinite universes. They still are just places, but they're empty places. They're empty places until they're defined by the people within them and the things that they're doing. So that's the point of that. People are the sun at the center of the universe. We're focusing on that. And that's why we feel assured in how we're prioritizing things.
[00:33:14.741] Matt Miesnieks: Yeah. And it also, one thing I'll just add to that, it's probably my biggest, not mistake, but thing I missed with AR in that I'd been working for years, you know, looking at world-facing AR, like look through the phone or through the glasses to see the content in the world. And what was successful was face filters. And it was that ability to kind of, how do I point the camera at myself? And it's about people and how do I augment people, not rooms? And so, yeah, that really drove home to me that people should be the center of all the interactions that we're exploring. We talked a bit about spirit of place. We talked a bit about how law was built into real places, but the next one about communication, social and self-expression being the same thing. What we see, like you still see today, is nearly everything that is at least the virtual world half of the metaverse, as opposed to the crypto half of the metaverse, nearly all of those products are entertainment or gaming at their heart. And, you know, that's good, but something I'd always believed is that when you look at markets and market opportunities and, and people and what we do with our time and our lives and our energy, like we like to be entertained, but we far more than that, we like to communicate with each other and, You know, we spend far more money on communicating, you know, everything from our mobile phone bill to traveling than we do entertaining ourselves. And so I think when we look at our piece of the metaverse that we're trying to create, we really wanted to tap into this idea of how do you enable people to communicate? Because that is social. You know, Facebook is a communication platform. They just rebranded it as a social network. And a lot of that communication is just how do we express who we are? And when you go through a platform shift, that use case doesn't change. All that changes is that you now have some new forms of media, you know, in this case, 3D media, to start doing those same things. And what is this new form of media? How does that let you express yourself or communicate in a way that wasn't possible before? So we're far more interested in that as a use case than we are in building like a game like Magnetic is doing.
[00:35:34.663] John Gaeta: Which is super exciting because we're like in the midst of a near paradigm shift in capture and generation, right, of things. So self-expression is really going to evolve and move fast in the next year, years. And it's a very exciting area for us. And we're lucky, right, to have began now, right, because we don't have a lot of things that we're bound to a lot of legacy that we're bound to with regard to what's the new form of social media and self-expression. So we're kind of free and we're lucky. AR is the wrong place to start. Just to mention that, like, basically, I don't think anybody who listens to this incredible program of yours, Kent, would argue right, that augmented reality is really going to itself be a paradigm shift in the way that we perceive and consume content. But right now, it's a window. The way we're looking at AR and VR or any device, any screen, is there windows upon the actual thing that matters. And the thing that matters is what is it that people are coming to, engaging with, and that's where a focus has to go because we've all been spending a lot of time, you know, joyfully inventing powerful windows onto the thing that matters. So AR will come, VR will come, you know, the spatial web is going to come. but it matters what's actually before our eyes at all. So that's what that means.
[00:37:22.763] Matt Miesnieks: Yeah. The next one about the real world being a dynamic living place. We've talked a bit about that. If you want to capture this feeling, this spirit of the place, if you want to tap into the people that are inhabitants of that place, this digital twin, like the idea of a digital twin, like in architecture, engineering, whatever, it's always like a static model. And we think that's just first step, like it's nowhere near enough. So you need to somehow bring all those dynamic aspects of reality up into the virtual twin. Not much else to say there, but it's closer to a simulation than it is to a digital twin.
[00:37:59.610] John Gaeta: Yes. And a simulation that can have its parameters tweaked at some point. So get on the metaverse to get off the metaverse. Again, like the honeymoon of virtual worlds and all of this stuff. At a certain point, most people will realize that real life was always so much more interesting and so much more stranger than fiction. and beautiful than any digital experience that we could have. However, what we're sort of implying there is that like great art or something that can catalyze your interest in pursuing something, that is a function that can be served by things like the metaverse. So this is more about the remote co-presence aspect. We could get a taste of a place and all that goes with it. And really, hopefully, what it would cause is a drive for us to sort of get to the real place. So the metaverse could serve to catalyze greater appreciation and engagement of the real world and reality itself. And the weird thing about essentially working in these areas and these mediums is that the deeper in you go, and visual effects did this before world stuff, But the deeper you go into trying to replicate reality, the more it makes you actually appreciate and look at reality and engage reality harder, perceive it, like to actually not just sort of let it go by, but actually try to use your senses to perceive what's happening around you and appreciate it in a deeper way. So that's hopefully what will happen as we dabble. in the metaverse, it's going to make us get out of the metaverse and really take in our real lives or real places a lot more than we realize. Not for all, maybe some people will get lost, but I do think a lot of people will suddenly look at reality freshly after dabbling in the metaverse.
[00:40:09.705] Matt Miesnieks: And the last phrase is about reflections. That's a word we use a lot. How do this virtual world and the real world reflect into each other? I think the phrase mirror world is a bit of a misnomer in that it implies a perfect literal twin of the real place, like looking in a perfect mirror at yourself. And it's not too difficult to imagine once you go down that path, it's impossible to do. How do you get every blade of grass perfect? And so the interesting thing is if you think about reflections as a term, it has a much broader meaning than just a literal mirror. And it can mean anything, you know, like a funhouse mirror can give you like a slightly warped reflection. You can have a pond or a lake where you get a reflection that's kind of affected by the ripples and the color of the water right through to something like an impressionistic painting, like a Monet painting of his water lilies that is totally impressionistic. But it is definitely like you go to the place that he painted and you can tell it's the same place. And it is an interpretation of that real place. And so that broader definition of reflections is really what we're leaning into. And we're not going for the whole pixel perfect real time copy of every atom, but more a sense of what does it feel like and how does it feel similar, but it can also be a little bit different as well and create something that's more interesting or more engaging or more fun or more peaceful, you know, whatever you feel it needs to be once, you know, all these creator tools are out there.
[00:41:50.683] Kent Bye: Awesome. Well, I think that actually gives me a really good sense of your intentions and your philosophical principles that you're basing your company on. And from there, kind of extrapolating out and building it. really excited to see where it goes as we continue to evolve and press down the path of building the living cities. So I guess just to wrap things up, I'd love to hear from each of you what you think the ultimate potential of these immersive technologies, virtual reality, augmented reality, or just the blending of the virtual and the physical together, what you think that might be able to enable.
[00:42:23.884] John Gaeta: Well, I'll go first so Matt can like land the plane. How long have you been doing these podcasts, Ken?
[00:42:30.985] Kent Bye: It's been over eight years now.
[00:42:32.406] John Gaeta: Yeah. They're amazing. And I've heard many of them. And so much has happened and changed and evolved. And it's like this is a road, right? We're like trying to get on a road. And the road is a long road. It might not ever end. Imagine this road 20 years from now, 50 years from now. Right. But the idea of the road we'd like to get on is one that's actually intertwining the real world and reality with the virtual sort of an extension and amplification of the real and these technologies that we It's just remarkable and impressive, you know, what's actually been done in the last decade. Incredible, really. Everything from, like, computer graphics depictions of real things to ways of seeing and being inside and touching these things, in a sense. But we've literally just gotten a bunch of colors of paint and some brushes just in the last few years, couple years. You can make an argument that, like, this year and the next few years are like the beginning now. We're finally almost ready to get going. So the answer to the question is, it's a road we're stepping on. I think everyone's going to step on it. And we're really looking forward to collaborating with people, having others really tell us what they want to do, what we should do. I mean, we just have a general sense of the the elements of the formula of this alchemy, but we absolutely need people to sort of mix these things together with us and learn from them.
[00:44:19.032] Matt Miesnieks: Yeah, for me, yeah, again, if I think long term and get away from like products and devices and things, it really is about giving people superpowers to let them influence and change their perception of reality. That's kind of at the heart of it. Hopefully those powers are used for good, or at least incentivized for good, and they make the world a better place. Awesome.
[00:44:46.528] Kent Bye: Well, is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:44:51.361] Matt Miesnieks: We want to talk all about our product, but we can't yet. We just got to wait until we've got something to show off, but we're really looking forward to starting to show it off internally. It's super exciting. Just in the last week, you know, we hit some milestones where everything kind of is hooked together now and we're getting a sense of what it's going to look and feel like. So, yeah, for me, I just, I just can't wait to start talking about that.
[00:45:13.995] Kent Bye: Well, John and Matt, thanks so much for joining me to help give a little bit of a sneak peek. I know it's still early days and we'll have a lot more to talk about once you are able to show the world what you've been creating, but I think I'd love to just to hear your journeys up to this point and to get a little bit more context as to what's inspiring you and the different principles that you're building upon. So really looking forward to seeing where you take this. So yeah. Thanks again for joining me here on the podcast.
[00:45:37.382] Matt Miesnieks: Yeah. Thank you. Thanks for having us. Great to be here.
[00:45:40.588] Kent Bye: So that was Matt Meisnix. He's one of the co-founders of Living Cities XYZ, previously created 6D AI, which was acquired by Niantic. So was John Gata. He has done lots of different special effects within the film industry, worked on The Matrix, then went on to work on some HoloLens stuff at Microsoft, and one of the co-founders of IMLX Lab, and then eventually was a senior vice president at Magic Leap. and is now one of the co-founders of Living Cities XYZ. So, a number of takeaways about this interview is that, first of all, well, I was really thinking about this conversation a lot when I was on my vacation to Alaska, just because when you go on to a place, you try to learn about what is the stories, what is the lore, what is the spirit and the essence of the character of that place. And a lot of that character of the place is defined by not only the people that are there, but also the geographic locations shaping in this feedback loop based upon what the space is doing and what the people that are there were able to create a unique culture within that place. And so, how do you start to capture the essence of that and then try to give people remote access to it in some ways? So it's a little vague as to what exactly they're going to be doing with their application. I'm really looking forward to see where they take it. But they're trying to solve this problem of, you know, you have a digital twin, it's very static, a concrete object of something that's not really changing or moving, but they're trying to make it into a living simulation that is dynamic, but also have people involved in somehow. So what if you actually go to a place like in Times Square? Are you able to have access to people who are having a virtual representation or simulation of Times Square? And how do you interface and relate between the people who are in the physical reality and the virtual reality? A lot of times, both Matt and John were using the word real as a contrast to virtual. For me personally, I prefer physical versus virtual because I think that It's creating this hierarchy of things that are in the physical reality being more real than stuff that is happening in the virtual reality because as I was saying you may actually have infinitely more layers of history, the past, or what are the access to the information when you have someone that's in the virtual representation versus someone that's in the physical representation. And for me the big difference is the physicality of the haptics, the touch, the taste, the smell, things that you don't Captured but also the living dynamic nature of things that are emerging in the moment obviously there's gonna be a lot more vibrancy of what's happening in the physical reality but in the virtual reality you may actually have things that are different and more agency to be able to control and tune the different Representations they were talking about reflections as opposed to something. That's just a one-to-one mirror world Matt was saying that that The phrase is something that is a little misleading because it's a little bit more of a representation or a reflection of that world rather than something that is trying to, for one-to-one, match everything that's happening in that world. But when you have that representation, what kind of things that are new and different that give you access to many dimensions of that place that would not necessarily be available to people who are there? But if they are there, you might have an augmented reality portal into that. So thinking about how these two realms interface is a lot of what they're trying to do with the Living Cities XYZ. I just wanted to recap the nine principles because we covered a couple of them throughout the course of the conversation. The first one is reality is scarce. People are the killer app. The spirit of the place. Lore is built into real places. Communication, social, and self-expression are all the same thing. AR is the wrong place to start. The real world is a dynamic living place. Get on the metaverse to get off the metaverse. And the real and the virtual are reflections of each other. So yeah, just some really interesting reflections of where to start with the metaverse And I think that there's been a lot of talks about the metaverse my previous interview with Matt ball that I just published Was all about the metaverse and the future of that you know Matt ball is also taking a device agnostic approach to this and saying that this is going to be a much larger shift from the 2d to 3d and Thinking about all the aspects of creating these real-time 3d virtual simulations, and so there's nowhere better to start than to look at what's already happening in the physical world and then adding layers of augmentation on top of that. But then when you start to create a digital twin, then can people have virtual access to that? And then if you have them co-present with each other, then what kind of stuff can you start to do? I just think about Times Square as an example, because it seems like the most obvious place to potentially start. I would love to go into a virtual representation of Times Square and see what's on all billboards. I'm not sure, in terms of privacy, whether or not I want to see all the different people who are there, but he's saying that there are certainly events and different advertising that's happening at a certain time each night. In Times Square, from 11.57 to 12, they are featuring some sort of artistic work of an artist each night. So, there's lots of different things that are happening, just to see all the billboards and advertising that is changing in a dynamic, and when you're actually in the physical Times Square, you can see it. But what if you have a virtual representation of that? Would you be able to go there? And then would that be a hangout spot that would be able to have things happening in that virtual space that were reflecting the spirit of that place? And I think that's the idea of what they're trying to go for. So really interesting to see where they take it, especially with the combination of all their experience with Matt from Liz decades plus of experience within augmented reality with John Gata with all of his special effects and working in the Star Wars universe and one of the co-founders of IMLX Lab which I had a chance to interview him at Sundance 2016 when they had a project there at Sundance New Frontier and Dennis Crowley who's a founder of Foursquare so be really curious to see where they take it and look out for whenever they start to announce some of their first demos like I said they're still an Stealth mode, they've announced that they've raised $4 million in funding, but haven't been able to see what actually they're producing. So when you think about the long road of the metaverse, as John was talking about, and how it could be that when you start to think about the metaverse and building it out, what if the augmented reality is going to help kickstart a reason for why people are going into these different virtual spaces? Try to solve that problem that Matt was talking about in terms of once you have an augmented reality app, what's going to bring people back into that app? And it seems like time and time again, connections and communications to other people and self-expression has been a key part of what has been driving a lot of those engagements. If you look at what's happening in Snapchat and VRChat and Breakroom and all these other social VR places, or even the more metaverse apps with Fortnite and Minecraft and Roblox, all of these things have social dynamics within them as well. They've got a lot of really solid principles and different philosophical implications and very much looking forward to where they take this in the future. So that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
