Matt Miesnieks is creating an AR cloud with 6D.AI, which aims to “synchronize 3D computer vision data across devices, time and space” in order to enable “persistent content, occlusion, and real-time shared experiences.” Miesnieks has founded a number of AR start-ups including Dekko, and he’s also a partner with the early-stage AR fund Super Ventures. I had a chance to catch up with Miesnieks back in October 2017 after the Google Pixel 2 announcement to get an update on the state of the AR industry. He gives a brief survey of the AR ecosystem, compares other AR solutions to what the Microsoft HoloLens has accomplished, and lists some of the open problems left to be solved in the AR space (including some of the things that 6D.AI are working on).
LISTEN TO THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Support Voices of VR
- Subscribe on iTunes
- Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So when I started the Voices of VR Podcast in 2014, at the time, virtual reality was the thing that was the most popular. over the years we've actually seen different dimensions of augmented reality start to come into play and I think as we move forward the Voices of VR is going to be covering both the VR and AR and a lot of people are calling that XR for X being a variable for all sorts of dimensions on that mixed reality spectrum from augmented reality to virtual reality to whatever flavor of reality that we come up with and whatever we call it. But all in all, it's mixed reality, virtual reality, augmented reality. I'm going to be covering it all here on the Voices of VR podcast. And these technologies represent these new paradigms of spatial computing. And it's going to probably take a while before it's reaching a stage of mass ubiquity. But The affordances and the technological roadmap kind of represents the most interesting problems to be solved in technology. Combined with, it's the things that you could apply to the human experience and just have all these amazing different potentials and applications. For augmented reality, it's really starting in the industry and specific applications that you can solve at work. But there's also things around identity. So for Snap and Snapchat, it's being able to modulate and mediate your expression of your identity and to embody these different virtual avatars. That, in turn, changes how you express yourself. But I had a chance to talk to Matt Mycenyks, who's been involved in augmented reality for a long, long time. And he's got 6D AI. It's trying to figure out the infrastructure to be able to essentially create the world as an operating system. In order to do that, you have to be able to actually know where the objects are in the world. And so they are creating this AR in the cloud system with 6D AI. I had a chance to talk to him back in October. At that time, they were still stealth and they weren't really talking too much about their product roadmap. But you can kind of get a sense of some of where they're going now in the course of this interview. And coming up here, next couple of weeks, I'm actually going to be going to F8, which is Facebook's developer conference, and then VRLA, and then I'm going to Microsoft Build, which will be a lot more augmented reality news, and then Google I.O., which I expect to see perhaps some more announcements around ARCore as well as other technologies that are kind of into the ecosystem of augmented reality. I see that these are technologies that are not going to be going away, and Matt is somebody who has just vast amounts of insights and depth about some of the unique affordances of augmented reality. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Matt happened on Wednesday, October 4th, 2017 in San Francisco, California. So with that, let's go ahead and dive right in.
[00:03:08.787] Matt Miesnieks: I'm Matt Misenecks. I currently do two things in AR. One is I'm a partner at Superventures, where we invest in pre-seed AR startups. and the other thing I do is I'm CEO and founder of a AR startup called 6D.AI and we're still in stealth but we're working on solving some of the hard technology problems around the AR cloud and connecting people and devices to each other.
[00:03:34.570] Kent Bye: Yeah, one of the things that I noticed if you look at Simon Wortley's technological evolution model, there's like these different phases that technology goes through where there's ideas that are proved out in academia with duct tape prototypes and then there's bespoke applications that are released and then it moves into consumer grade where it's more available to the masses and maybe even consumer available and then Eventually, it's just everywhere and ubiquitous. But right now, we're kind of going from the bespoke applications of AR into these consumer launches with both the HoloLens and ARKit and ARCore. And so you're someone who's been in that bespoke world for a number of years. And so maybe you could talk a bit about your background from AR and then kind of what you see as the new possibilities of what's available now with consumer available, but also kind of like this reinventing of the wheel of what may have already been done within the bespoke context.
[00:04:27.301] Matt Miesnieks: Yeah, there's a bunch of stuff to answer there. So, yeah, I started working in AR about nine years ago. I moved my family to Amsterdam to work for a company called Layer. They did the first sort of mobile AR browser, one of the first. And I came from about 10, 15 years in mobile before that and saw a lot of the same sorts of things that led to the smartphone becoming what it is today. with AR. It was definitely too early to jump into it, but it feels good that it seems to be the right decision. From there, I went and started my own company called Deco here in San Francisco. We did 3D computer vision, like markerless SLAM, markerless real-time 3D reconstruction, and some 3D content in there. So I guess what Magic Leap had been calling mixed reality, we were doing it on an iPad 2. And then I spent a couple years at Samsung, really looking a lot more at the hardware side of things and prototyping a lot of, not so much the hardware, prototyping interactions and use cases for consumers and getting a sense of the timing and what are the pieces that need to exist before that's all going to work. And then about a year and a half ago, I went into super ventures and sort of led to doing what I'm doing now. So, when I started in AR it was very much, it wasn't even bespoke, it was like academia and the products like Layar were very, I mean, trivial, like technically trivial to build and they're exciting because of the ideas that they kind of embodied more than the actual user experience they delivered. And seeing so much of the pieces come together lately has been really exciting. Like you said, some of the pieces are consumer ready, particularly the tracking and odometry that's in ARCore and ARKit. HoloLens is the first and still the only example of what you can do when all the pieces are in one thing. And I think, I think though that I'm, you know, I sort of tend to be more skeptical than most people in the AR community, mostly because I've been through the AR hype cycle twice already, so I sort of recognize that this is sort of the beginnings of another one. And also know what, you know, what's possible with what works today, which isn't that much. You know, it's great to, To build some experiments to learn the learning curve is really steep But I think until we get that sort of widespread everywhere type adoption that that's still several years away and I guess the challenge and the opportunity is between now and then what can be built with the constraints with the products today and I I draw a lot on my time in the smartphone space, where I started working for the company that invented the mobile phone browser, OpenWave, and saw the first WAP phones right through to the iPhone. That was about a six or seven year period. in AR today, we're in the early stages of that cycle. You're not, so to say, like the HoloLens is like the Nokia communicator, like those big clunky WAP phones with a keyboard. And between now and whatever that iPhone, Android of AR, that pervasive device that everyone's gonna wanna buy, probably be a pair of glasses of some sort, we've gotta figure out what's the BlackBerry of AR? What's the iMode in Japan of AR? Palm Pilot of AR, and they're going to be these products that can be really successful, but they're really just markers on the journey, and figuring out that is the fun at the moment.
[00:08:11.671] Kent Bye: Yeah, I see it a number of different ways. I just got back from the Google press conference here on October 4th, and they were announcing a lot of their ambient computing devices, so developing conversational interfaces. They released Pixel Buds, which start to have a Bluetooth-enabled device that you put in your ears. You're able to start to have conversational interfaces with your phone, so it's paired with your phone. moving away from looking at this screen so much and both with the air pods from Apple where again your Have these conversational interfaces to be able to engage with technology and listen to music So music driving a lot of the use of ambient computing as well as the photo sharing applications for people to be able to drive You know adoption of some of these latest phones that have AI assistance and whatever but I see it like this moving away from the screen and becoming more and more immersive and As well as companies like Snapchat, to me, I think are probably one of the leaders in terms of driving innovation, in terms of allowing people to alter their own identity and change the way that they express themselves by having these filters of augmented reality on top of themselves. having the iPhone have a front-facing depth sensor camera to me was an indication that Snapchat has had more influence in driving technological decisions at a company like the iPhone to have the depth sensor camera focused on your face rather than focused out into the world, which I think a lot of people that were probably in AR would want to have that. Outward facing depth sensor camera, which would enable whole sorts of other applications But yet that's not on the leading edge of driving consumer behavior So I feel like there's there's consumer behavior that's happening but there's also like enterprise applications of augmented reality with a the HoloLens, and it seems like those more bespoke applications is probably where I would imagine where SuperAdventures is probably investing in more of those companies that are maybe a little bit more of less of the mass consumer, but more of like bespoke. I'm curious if you could kind of talk about that.
[00:10:06.913] Matt Miesnieks: Yeah, yeah. Again, there's like a whole bunch of things I can answer there. I'll start with the depth camera one like you mentioned, and Snap definitely popularized that use case, and for a long time I used to go around telling people that the most successful AR application in the world is Snapchat. Two years ago that used to get a lot of blank looks, but I think people get it now. Although I think a lot of AR people were hoping and dreaming for a front-facing or world-facing depth camera, that just technically can't be done yet. There's a whole lot of issues with just the range of those cameras, the power, the interference with sunlight. So I think Apple definitely was smart in the way that they built the hardware. Then, I guess backing up a little bit to your comment about Google and their shift towards immersive computing. I think that some, you know, when we think about what our AR product is going to be, like the hardware product, I think we're still thinking about it the way we think about our smartphones in that we've got, you know, a little bit of hardware that we keep in our pocket. That's a self-contained product in itself. And then we have, you know, the cloud, which is a different thing. and some of our apps upload and download to and from the cloud, but with AR and all the surrounding types of experiences, that distinction between what is the hardware product that I carry or wear and what is the software operating system or ecosystem, I think that the software side of things is going to be very much a blurred line between what's the cloud, what's the client. You know, like Google Voice, some of it that runs locally and some of it that's done in the cloud, but it's all transparent to us. Even Google Photos, same sort of thing at the moment. And the hardware that we use, whether it's something we put in our ear or a speaker on our bedside table or glasses, those will all just be different ways to interface this single entity of software. So then shifting gears to enterprise and super ventures, you know, we up to now, I guess we have We've stayed away from anything that was consumer applications on AR. We looked at a lot of those missing pieces of the roadmap that need to be built and invested in a number of those, whether they're types of input or types of computer vision or types of analytics. They are B2B, but they're not that enterprise type of service. On the enterprise side, I'm really looking for the salesforce.com of AR. At the moment, every AR deployment in a factory is a custom-built application. They might all just use ODG or Vuzix hardware or Epson hardware, but the application is custom. So I think someone's going to figure out, here's the hospital worker's product, or here's the field service engineer's product, ScopeIR is an example of a company working towards that, but no one's kind of solved it completely yet. But I think now, just in the last month, the potential for a big audience for your application is real. And so we're looking at those sorts of companies. But what we're, you know, the filter we're looking at is, you know, why should this application be in AR at all? Like, why shouldn't it just be a regular smartphone? And does it work, you know, effectively within the constraints of what, you know, version one of ARKit and ARCore can can serve up? Yeah.
[00:13:52.673] Kent Bye: Yeah, I've been thinking a lot about immersive computing, spatial computing, and the unique affordances of virtual reality, and now as augmented reality is starting to have phone-based AR, as well as the HoloLens, starting to incorporate that into the same realm of spatial computing, immersive computing. just thinking about these fundamental questions about what are the unique affordances of this medium. And for me, I think about it in terms of experiential design, where you kind of have to think about things in a holistic way, but also spatially, you know, what are the unique affordances of space and being able to actually use your body in a natural, intuitive way. And so for you, what do you think are some of the unique affordances of augmented reality as a medium?
[00:14:33.879] Matt Miesnieks: Yeah, it's a good question. My wife runs Adobe's design lab and sort of figuring out tools. She co-founded Deco with me and has been working as a designer and from a visual arts background in this space. Pretty much everything I've learned is just by either listening to her or arguing with her about something and sort of figuring out how this will work. And she started using the term sensory design to talk about a lot of these interactions because they are You know, although we talk about immersive, that's kind of a catch-all phrase, but at the end of the day, we're trying to engage our senses in ways that you can't do with just a phone, whether that's sight, sound, haptics, you know, all that sort of stuff coming together. And designing for, you know, I'll stick to AR, we learned, like I founded Deco with Silke, who's a designer, and then my team at Samsung was co-founded with a designer as well. And so, you know, I don't know anyone that's been putting design at that same level as a technology for as long as we have. And it teaches you a lot, a lot that a lot of people still haven't come close to figuring out. And things like app designers generally don't get it, at least not initially or not intuitively, because going from a 2D interface to 3D is completely That's a huge change. And game designers who might know 3D, they struggle as well because they don't understand trying to solve problems for real people in their real world, in their day-to-day lives. Games are completely self-contained. So the type of designer that I've found is just like gets it the easiest are people with an industrial design background. You know, product designers for physical products. Specifically if there's sort of some complexity to it, like more than a chair, but designing a coffee machine or a vacuum cleaner or something like that. Because they know how to think about problems in the real world that we have in living our day-to-day life and they think about them physically and spatially and just naturally and all they're doing is doing what they already know how to do except the end product just remains digital. They don't send it off to the printer. So that sort of approach is really effective in AR. The affordances, I mean, the biggest one is just, I guess is more than one, but the biggest benefit we found comes from helping the user understand and believe that the content is really in the world. Not that it's real, like you don't want, you know, photorealism or, you know, something that looks like it's a real thing is far less important than, for example, having, you know, rock-solid tracking so it doesn't kind of creep around in your peripheral vision because that just freaks you out. Getting things like shadows and lighting right, like even Even if it was a sort of cartoon image of, you know, Bugs Bunny or something sitting on the sidewalk here, if the shadows and the lighting were right and the tracking was solid and there was some awareness of the structure of the 3D world for physics, you know, those affordances suspend my disbelief more than, you know, a photorealistic object and that was kind of surprising to us how you know, just putting a drop shadow underneath something just completely changed how people, you know, believed it. And, yeah, people were quite happy to believe that this fantastic, you know, fantastical type of creature was really there.
[00:18:11.027] Kent Bye: Yeah, and imagine that just like in the virtual reality community where someone has designed a 2D game and tried to switch it over into a 3D context, doing a direct port usually doesn't work. You kind of have to design it from the ground up. So I feel like there's these new workflows and processes that need to be really cultivated to go from what a whole line and lineage of user experience, user interface, and product design, product management, all these ways of being able to visualize things in 2D have existing workflows, but this is kind of representing entirely new workflows and paradigm shifts as well to really think about it. And I'm, yeah, just curious to hear your thoughts there.
[00:18:48.005] Matt Miesnieks: Yeah, it's actually exponentially more difficult than VR. Like that change from like a 2D app into VR, was difficult. But VR, the developer controls every pixel, and the user is assumed to be in a big, open, physical space. With AR, you don't control the scene. You kind of get that given to you by wherever the user happens to open their app. It could be with a nice, flat table in front of you, but they could be sitting on the bus. They could be walking down the sidewalk. And, you know, people don't even understand what the problems are yet to think about what the tools need to be to think about how do we solve these problems. And, you know, ARKit and ARCore just give you a very basic ground plane, no real structural awareness of the scene. And unless, you know, there's a handful of people that have built, you know, complex Tango apps or complex HoloLens apps, that's only when you start to realize how difficult it is. And, I mean, as an example, we tried to build a game at Deco and We realized that the whole concept of a game needed to change, you know, from a video game. We, you know, had things like, well, there's going to be some progression, you know, from starts off easy and gets harder. And yet, because we couldn't control the structure of the scene, the very first level that the user opens up could be in this really complicated, you know, natural environment, and all of a sudden the game is really, really hard, you know, through nothing that we could control. and we ended up having to kind of abandon that type of thinking and completely shifted you know over to the more of an idea like toys and free play and like here's a here's an object that has some properties you know it could just be physics like a ball that bounces or it could be something that's intelligent like a you know robot or a character and then how do we play with it you know how do we you know kick a ball and bounce it off the wall or how do we have a little you know we ended up using a virtual RC car that you could drive around and and do jumps and crash into things and just sort of have fun. And that sort of paradigm shift into structurally what is a game or what is an app, what is an interface, we're barely even understanding. Definitely as a broad industry, we don't even know that those are problems yet, let alone trying to have a sense of how to solve them.
[00:21:19.143] Kent Bye: Yeah, I just had an interview with Clay Bavor of Google and was asking him about WebVR and WebAR. And I think that one of the things he said is that the whole concept of an app is going to likely kind of go away in the sense of like, it's going to be much more about the computer vision and AI algorithms being able to detect where you're at and to figure out that location as well as the context. and then have things that are made available based upon your own preferences. So it's a little bit like thinking about the world as an operating system and how there's going to be different functions and applications that you may want to have that's based more upon where you're at in that world rather than the app model that you have now, which is opening up your phone and opening up a specific application. So do you have any thoughts on that?
[00:22:03.255] Matt Miesnieks: I love that phrase, the world is an operating system. That was part of our pitch at Deco in 2012. We were going around saying that, and no one had a clue. It was just a crazy choice of term back then. But the metaphor, I think, is becoming more and more understood in that an app does go away. You don't load up your desktop and click on a little button to launch an app. It's going to be just as you happen to walk by a store, or as you happen to walk by a You know your your stereo system at home as I happen to look at you versus looking at someone else That's going to trigger You know we used to call them like switches of context like my context changes as I you know focus on different things and so instead of an app where you switch context by like closing email and opening opening slack I that context switching needs to be very seamless and lightweight in the background. When you start to think about that and dig into it, all of a sudden the challenges of scale just hit you, because with ARCore you can have a little app that runs on six feet in front of you, but if you want to do it on a whole city, or let alone the world with thousands of people all at the same time, The technology just doesn't really exist to do that yet and the web has got a great head start in that it sort of works at big scale, but it doesn't really do this natural 3D type of stuff. I think there's a really, you know, greenfield, exciting opportunity there as to what are these large-scale applications and how do we deploy them and how do we interact with them, and then how does a user choose to engage with them or not. That's one of the most fascinating problems, I think, in AR at the moment.
[00:24:02.307] Kent Bye: Yeah, and when I look at the virtual reality ecosystem, you have this, well, when I look at the video game industry, there's kind of three major segments, which is the mobile gaming, console gaming, as well as the PC gaming. And I see that right now, if you look at the analog for immersive computing for virtual reality, we have a lot of the PC gaming, we have the mobile VR, and then maybe some standalone units that may be kind of trying to figure out their unique use case of what you can do with maybe if you have tellerless VR. And actually, it used to be in video games, there used to be location-based entertainment where you would go play video games. And I think that actually VR is going to have a whole other location-based entertainment where there's going to be a certain level of haptics and using space and mixed reality environments that go beyond what you can do at home. there seems to be those different dimensions. And when I look at augmented reality, I also see that maybe there's a similar kind of ecosystem developing with phone-based AR with what you can do with your mobile phone. And then the HoloLens is maybe on the other extreme of the head-mounted capabilities. And I'm just curious to hear from you, the other big players in that ecosystem that you see as maybe the highest end and possible for augmented reality. and where Magic Leap may fit into that, and also, yeah, just if that kind of maps over from what I see developing virtual reality kind of mirroring over to augmented reality.
[00:25:30.659] Matt Miesnieks: Yeah, it's a good sort of analogy. One of the, I think, challenges to AR kind of taking that similar sort of bifurcation in products is AR is kind of pointless if you're just sitting still, like if you're in one place. It's really, The benefit of it comes from actually engaging with other people or other things or other places. So that ability to do like a tethered PC powered type of AR, you know, like say what Meta is pursuing. The use cases around that. are challenging to find. Having a big virtual office is what Meta is going after, but it's hard to think of too many more. And to get AR that can be experienced out and about, The phone version of AR is really only good for very small, quick interactions. No one just wants to hold their phone up and walk down the street looking through it. So that's always going to be a bit of a challenge for that sort of cardboard, gear VR type of AR product. So, I'm not sure. The answers there I tend to think are more just how you design your products for that really quick type of interaction. Google Lens is a great example. Pull your phone out, visually query the scene and get a response or play a little game or something. To get to the head-worn type AR that everyone's pursuing, that's more where I think we're going to get this stratification. You're going to have, on the light end, products like Snap Spectacles. I've just been so impressed by the way Snap tried to solve one of the hardest problems in AR and went about it Without complicating things and that that's like how do you how do you get a consumer to wear a camera? Today so far only GoPro has had any success at that and I saw Google tried They've got this little clip-on thing, you know, but that problem just by itself is Incredibly, you know to date it's been unsolvable. So I think We're going to see you know these lightweight products that are you know glasses or headworn or something that really only just focus on one problem at a time. I heard about an early prototype of the Apple glasses that all they had was a display. They looked great, but they really had almost no processing power, just a display, no camera, and they served up your notifications. I don't know whether that's what they're thinking about for their end product or not, but they could have just been testing a particular set of use cases in this form factor. But again, that discipline of just paring it right back to say, look, there's a problem that's never been solved, which is how do you put a display in eyewear and deal with form factor and fashion and prescription optics and all that sort of stuff. And that's all that product had to solve. I kind of think we'll see a lot of experiments like that with these lightweight type of devices, lightweight glasses. And then on the other end are going to be the devices that you're paid to wear, because you'd never wear them for any other reason. And they're going to be big, powerful, quiet, fully featured. deliver like measurable ROI to enterprises, you know, you will Measurably reduce the training time for someone to get up to speed on the on the factory floor You'll measurably reduce errors in some complex procedure because you have the checklist You know running through you and have little arrows saying, you know, press this button not that button Those sorts of You know more complex products will you know, they'll be too expensive for consumers for for quite a long time But the benefits are really there and they're being quantified and you know people are being paid to wear them So yeah at some point it'll all it'll all merge But it that's a few years away, you know, three four at least. Yeah
[00:29:40.337] Kent Bye: Yeah, when I went to the Microsoft Build conference where I got to see a lot of the HoloLens developers show off what the different use cases that they have, it was a lot of those enterprise applications where it was either something to do with spatial relationships, so whether or not they were doing architecture, engineering, design, and having collaboration. Or I also saw a lot of people doing sales. So being able to walk into somebody's room with, and maybe in a doctor's office, and then start to customize the different medical gear that they would be able to fit within that context and give them sense of how this would lay out and so you have these very specific use cases of anything that has to do with people who are trying to sell things that have a spatial relationship to it or deal with visualization of spatial content of anybody that uses CAD or 3D modeling seem to be a lot of those use cases. So that's sort of in the enterprise and education. And then there's Magic Leap that is maybe going after this entertainment. I have my own, I guess, concerns and worries about whether or not Magic Leap will be able to find the storytelling and immersive unique qualities of what it means to have spatial stories and entertainment. So I don't know, where do you see Magic Leap kind of fitting on this? Because it seems like they're going after a very specific high-end thing that may be not enterprise, but geared into giving people these immersive entertainment experiences.
[00:31:14.510] Matt Miesnieks: Yeah, so definitely, like you mentioned earlier, the VR and AR as a medium are really effective at helping you simulate, educate on complex, expensive things, whether that's a piece of medical equipment to preview or training you on how to fix an oil rig or something. So that type of stuff is really a good fit for this medium where You know, as we learned, games and entertainment are actually probably the most difficult type of experience to create. It's almost back to front to VR and where, you know, it's easy to take a 3D game but in AR because of all these issues with the structure of the world, it's really hard. So, you know, Magic Leap, from what I understand, they are still very much focused on a consumer product. Very much entertainment is kind of the leading use case. But there's no science fiction involved. The problems that they're working on are the same problems as everyone else. As far as we know, they haven't broken the laws of physics or invented something that's impossible. So, you know, what I expect, you know, is that their product will come out. It'll be very much positioned as a version one, you know, product of a journey. And there'll be a couple of things that's better than other products in, and there'll be a whole bunch of stuff that it's worse. And it won't be like, whoa, this is like the new generation. I think Over the last couple of years people working in AR have kind of, it's dawned on them how far ahead Microsoft was with, or is with HoloLens. I mean that was released two years ago and there still isn't a product out there that does what the HoloLens does. And, you know, if Magic Leap ships something that does what the two-year-old Hololens does, that in itself would be a remarkable achievement. I still think they get punished by the media and the market for doing that, but in terms of the difficulty and complexity of building something like that, it would be something impressive. So, yeah, I just don't really want to talk too specifically about what I know about this stuff, but it is a difficult problem. They've got some good people. They've bitten off probably more than they realized. When it started, it was all based on display technology. They had this retinal projection system with depth of focus, variable depth of focus. And then they sort of said, we're building an AR system. And I went, well, that's like saying that you've invented an OLED screen and claiming you're going to build an iPhone. There's a lot beyond the screen that you have to build. And they obviously have figured that out and raised a lot of money in trying to do it.
[00:34:20.742] Kent Bye: Yeah, I see, like, when I look at virtual reality and augmented reality, I see this as a new spatial immersive computing platforms and capabilities that are not going to go away. We're on this trajectory, and I think the question of when it's going to flip over into mass ubiquity is probably the biggest open question. I tend to have the date of maybe 2025 as a day where a goal where I see like maybe mass ubiquity when it comes to maybe head-mounted augmented reality displays and then virtual reality just being kind of a blending of the mobile technologies and what you can do on the desktop and then having something that is available enough but also the use cases and what the consumer drive is going to be starting with a lot of ambient computing now with being driven by both the music and photo sharing applications of how people are changing the way that they're maybe taking a selfie with an augmented reality filter through Snapchat and, you know, a way of connecting to their friends but also having conversational interfaces and the evolution of artificial intelligence to be able to do the computer vision that's necessary. So that's sort of what I have in my mind and I'm curious to hear your perspective as well as the kind of technological roadmap and milestones of things that you see that we'll need to go through in order to kind of, you know, get there technologically and sociologically in terms of the fear of privacy, as well as cameras, I think is this whole separate other issue that, like you said, Snap is able to solve that in a way that, you know, Google Glasses wasn't able to. But I think that sort of cultural use of these technologies is a whole other question, but yet, I see like just having these computers with our home, being able to talk to them, they're starting to break down that ambient computing and calm computing in a way. But I'm curious to hear your view on that sort of roadmap of what you see needs to happen for us to get there.
[00:36:16.396] Matt Miesnieks: I think your 2025 date for sort of massive options, you know, probably not too far off. Again, sort of drawing on my smartphone days, you know, there were products that everyone got excited about. But if you take, you know, the iPhone release in 2007 is like the starting gun. Even that, you know, wasn't a good time to kind of expect mass ubiquity. It took about three years after that before I think it was the 3GS and iPhone 4 that really sort of kicked into overdrive because, you know, we had to wait for people to understand it. We had to wait for their carrier contracts to expire and roll over to a new phone and they had to fix the problems in version 1 and add copy and paste and add an app store and all that sort of stuff. So I think even if we get to a you know, a pair of AR glasses that we all want to buy, you know, by, I don't know, 2021, 2022. It won't be, it'll still be a couple of years after that before there's like 100 million people using them and you can really, you know, see that it's going to replace smartphones. So, forgot the second half of your question.
[00:37:23.950] Kent Bye: Oh, just sort of the roadmap in terms of things that we need technologically.
[00:37:28.455] Matt Miesnieks: Yeah, in terms of problems still to be solved. So there's a whole bunch of... Just wait for the truck. There's a whole bunch of problems around just how does the phone or the camera, I guess, perceive the world in 3D. You know, we need to basically have a 3D model of the real world captured in real time. You know, it can't just be something from last week that's downloaded. And then we need to sort of serve up to the application, look, this is what things are. This is a chair, this is a sidewalk, this is a couch, you know. That problem's still challenging, you know. There's research-type solutions, but that's, you know, still a couple of years, I think, before we start seeing good solutions. The social problems around or challenges around fashion, around what are our norms, all the panic around Google Glass and people worried about getting recorded. I lived through exactly the same thing when camera phones were new in 2004 or something and everyone freaked out. They're going to sneak into bathrooms and take photos of my kids and stuff. you know, all the OEMs voluntarily added like a click sound just to calm everyone down. And now, you know, no one ever even thinks about that as a fear. So there is going to be a lot of those sorts of things. Privacy of you know, wearing a camera and actually capturing data as you walk around. Even when you, you know, if I invited you over to my house and you sort of walked down my hallway just looking for the bathroom and glanced into my kid's bedroom, you'd sort of leave my place with a partial 3D model of my kid's room. So how do I feel about that? I don't know. You know, there's these questions that I don't know what the answers are. Interestingly, I think Startups have probably got a better idea, a better chance of solving those problems because you can take more risks, you know, like with sincere best intentions, maybe you get it wrong, like your shade of grey is just on the wrong shade of grey. And a startup can quickly course correct and you're not sort of big enough to be worth suing. So you can run a lot more experiments and kind of learn where that boundary is. where if Apple or Google just can't afford to even get one person's private information wrong and it's a national scandal. So I think that's going to be interesting to see how the startup ecosystem competes against the big guys with solving some of these less technical challenges. Apart from other technical problems around optics, I think input is nowhere near solved. That's probably the most underestimated difficult problem, because it sounds easy. It sounds like you just talk, you just point, gestures, tap, whatever, but tying all that together and making it contextual so that the system takes the correct input at the correct time to determine your intention, That's ultimately what it has to do. It has to read your mind and say, well, you really intended to do this. And take advantage of all the signals, you know, my body language, my, you know, where I am, who I'm with, you know, what I'm talking about previous to what it may think is input. And all that stuff is a long way from being solved. And to be honest, I think solving, when input is really solved, that is the starting gun for the next product. So that's what I'm really looking at.
[00:41:09.942] Kent Bye: And for you, what do you want to experience in AR? In AR?
[00:41:13.385] Matt Miesnieks: That's a good question. I'm really excited by the communication potential of AR. The ability to communicate aspects of who I am. Potentially different people can see a different dimension of me at different times. You know, someone at work might look at me and I'll have a whole bunch of work augmentations there, and someone socially might look at me at the same time and they'll see a different view of me. And I love that, you know, with everyone, not just me expressing myself, but seeing other people and how they communicate. Along with the ability to, you know, a bit like VR with whatever, you know, social sharing, holoportation, you know, that ability to feel like that other person is really there with you when you're somewhere else is, I think, gonna be probably the most powerful end use case of any of this stuff. All the information search and entertainment, that's kind of interesting and useful, but that's not what's going to really change my life and emotionally draw me in.
[00:42:21.967] Kent Bye: Great. And finally, what do you think is the ultimate potential of augmented reality and what it might be able to enable?
[00:42:30.444] Matt Miesnieks: Yeah, I look at AR and VR as two sides of the same coin. And if VR is very much about ultimately taking you somewhere else, AR is much more about how do I improve or enhance or augment where I am right now, who I'm with, where I am, what I'm engaging with, and almost give me a superpower that I can see with eyes that can see things that I can't see already today. feeling of I can do things that were unimaginable and I can do them without even thinking about it is Ultimately the potential of AI Awesome.
[00:43:11.438] Kent Bye: Well, thank you so much. Yeah. Thank you So that was Matt my sneaks. He's one of the co-founders of 60 AI as well as a partner at super ventures So I have a number of different takeaways about this interview is that first of all, Matt is in a position to be able to get a lot of deep insight into the augmented reality ecosystem. I mean, not only is he a part of a venture capital firm, SuperVentures, to help sort of scout what kind of projects are out there to help invest into these different projects. He's also creating his own startup with 60AI, which is like AR in the cloud. And he's also just been involved in the industry for over a decade now. And so he's been in this mobile space, really thinking deeply about a lot of these issues. I think augmented reality, there's like still a lot of the key component parts that have to really fit into the entire like, you know, hardware and ecosystem, everything else like that. He said that the Microsoft HoloLens is actually like far and above and beyond what anybody else has done for at least like two years now. The prototype they have released with the HoloLens is technologically just so much more advanced. Also the sociological barrier of putting a camera on your face snap just announced recently their second iteration of the spectacles camera and you know, I think that the numbers I think that they sold maybe 220,000 but if you look at like kind of going towards 2025 and looking at all the other sociological barriers that we have to get through in order to actually be comfortable with people walking around with cameras on their face. If you look at like what Matt said is when these cell phone cameras came out, people were also having this freak out of like, oh my gosh, people are going to be able to take photos of us in the bathroom. I think there's going to be a similar hesitation to that, especially now that we've got these surveillance based marketing companies like Google and Facebook who are trying to capture all this information about other people. I mean, the big risk, I think, with augmented reality is that you have the cameras that can detect other people's faces and Facebook essentially has the biometric data of everybody's like face and to be able to do face recognition. And so imagine being able to like, you know, walk around and essentially not have any anonymity. If all of this data is being sort of ingested into these augmented reality glasses and you are just happening to be walking around, you could be identified and located in specific spaces and then kind of fed into a massive surveillance database within something that's like in Facebook or Google. And the implications of that, I think, are the thing that is the most scary. Since the time I had this conversation with Matt, the conversation around privacy and Cambridge Analytica and Facebook has just absolutely exploded. And so it's, again, sort of a much more sensitive topic as to how we're going to actually deal with some of these privacy issues. So I think it'll be interesting to see as we move forward. I think that it's actually possible that only augmented reality headsets are used for industrial applications. So when you're at work, but when you're kind of walking around, are you going to have something that's going to be mediating your environment that you're wearing on your face? I expect something like the Magic Leap as well as Apple just also recently was reported on by 2020 they're expecting to have some type of like augmented reality product that they're going to be releasing. Apple tends to be a little later to waiting for everybody else to release whatever they're going to do and then they kind of come in and then release something that just has everything really dialed in in terms of the user experience and so If 2020 is the timeframe where Apple is going to be releasing something that might be equivalent to kind of like the first version of the iPhone, then as we move forward to like 2025, I think that that's still a pretty good target for when I expect all of this immersive technology to be completely massively ubiquitous, both with like virtual reality and augmented reality and whether or not they can kind of go in between each other. One of the other really important points that I think Matt made is that he's looking to people who are coming from industrial design and product design as the people who are best well-suited to be able to start to think about designing for augmented reality. These are people who have already been thinking about space and spatial design and affordances of human ergonomics. And a lot of the web developers, we've been coming from this 2D paradigm. And there's people who have been designing for a 3D spatial paradigm for a long time. It just so happens that a lot of skill sets have been going into products that have been actually made and manufactured and created within physical tangible space. But those same skill sets are going to be extremely important and valuable when it comes to designing augmented reality. And so I've just found that people who are coming from a product design background, architecture, they are thinking about things spatially in a way that people who maybe are coming from a technology background have been thinking about things in 2D. So I think some of the open problems in terms of like the input control, you know, since the time of this interview, we've had the Leap Motion, North Star, which is going to be a tethered application to start off with. One of the things that Matt said is that augmented reality to really be useful and to be able to change context in your environment and to have the world as an operating system and to be able to kind of mediate these different applications based upon where you're at. Well, if you're tethered to a computer, you're not going to be able to get that affordance of being able to just have that spatial computing overlaid into all of different dimensions of reality. And so I do actually think that now that both Meta and the Leap Motion have the tethered applications, we're going to see higher resolutions and maybe specific applications that are geared towards people like doing work at a computer. and other industrial applications. And then if you want to go out and see these spatial information overlaid on top of reality, let's say you're a construction worker and you need to go out to the place that has, you know, in the process of being built and you want to see the information overlaid on top of that, then you'll use something like the HoloLens or some other sort of like tetherless head-mounted display for augmented reality. So I think it's still early days within the realm of augmented reality. You have pretty much all the major technology companies involved with it at some degree. Apple with the ARKit, Google with ARCore, we have Microsoft with the HoloLens, and we'll see what is announced at F8. I expect there might be some augmented reality glasses that are announced that Oculus has been working on. So it'll be interesting to see if Facebook is going to be coming out with some sort of what would be equivalent to like a snap where you're maybe walking around and capturing different dimensions of your life, your personal life. Or I have a hard time seeing how Facebook's going to be making like an industrial AR application move and play because, you know, Microsoft has really got the enterprise market really dialed in. And even when it comes to virtual reality, I think that the Windows Mixed Reality what I see is Microsoft is really making a hard push for both medicine as well as the industrial and enterprise applications, whereas Facebook's obviously been focusing on gaming through Oculus, and HTC Vive is probably also the other industrial applications that have been used, and they're probably, in terms of their tracking and the lighthouse and everything, being able to add extra trackers into the, I think, location-based entertainment, and more the highest end of virtual reality is where HTC Vive has been really focusing. But if you need something that's simple and cheap and works at your desktop, then the Windows Mixed Reality, I think, is going to be something that is also going to be really playing into that ecosystem. So I'm excited to go to F8, VRLA, Microsoft Build, as well as Google I.O. over the next kind of week and a half to be able to see all the latest news, both in augmented reality and virtual reality. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a donor to the Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
