#148: Soren Harner on Meta’s Augmented Reality Glasses

Soren-HarnerSoren Harner is the chief product marketing officer at Meta, and he talks about the Meta 1 developer kit and augmented reality eyewear. They have an SDK to create experiences within Unity which includes different gesture interactions to use your hands to select buttons.

Meta was originally incubated by YCombinator, they went on to have a successful Kickstarter for the Meta 1 developer kit, and they recently raised a $23 million dollar round of funding.

Soren talks about some of the differences between augmented and virtual reality, and some of the outstanding technical and cognitive challenges for AR. The goal of virtual reality seems to be to achieve a sense of presence in a virtual environment, but there are different ways of looking at what constitutes a successful augmented reality experience.

Soren says that Meta is focused on working on different use cases to help solve specific problems. Ultimately, a good AR experience is one that is usable, helps solve a problem, is convincing, gains traction in the marketplace, and integrates well with the human perceptual system. Rather than gaming or entertainment, Meta is targeting professional applications, data visualization, and working with 3D models and Building Information Modeling management.

There are privacy and social implications with using AR in public places, and Soren expects that there will be a period of climatizing to the new form factors before people become comfortable with the AR headsets. Meta is focusing on indoor use cases since their AR headset requires it to be tethered to a computer for now. Soren sees that Meta will use digital content to facilitate collaborations, and that they’re leaving a lot of the content up to developers.

Reducing latency seems to be a big area for Meta, and Soren admits that latency is an issue that they’re continue to work on it. Oliver Kreylos called out the latency in this hands-on review of Meta at SVVRCon.

The gesture controls that Meta has implemented include tracking your hand as a point cloud so that you can use the highest point on your hand as a tapping motion for selecting buttons.

Meta has some prominent members of the wearable computing community including the original cyberman Steve Mann, Columbia University Professor Steven Feiner, and Jayse Hansen, who worked on the sci-fi user interfaces for a number of movies including Iron Man, Ender’s Game and Avengers. Soren said that sometimes a user interface that looks great for Hollywood may actually not be a very practical human computer interaction model, and that’s where Feiner’s insights can help bring Hansen’s visions into something that is usable and practical.

Finally, Soren sees that augmented reality will help integrate and digital and physical worlds in a new way so that we can understand the world better, help teach people things, provide new communication tools, and help manage information about the world around us.

Become a Patron! Support The Voices of VR Podcast Patreon

Theme music: “Fatality” by Tigoolio

Subscribe to the Voices of VR podcast.

Rough Transcript

[00:00:05.452] Kent Bye: The Voices of VR Podcast.

[00:00:12.094] Soren Harner: I'm Soren Harner. I'm the Chief Product Officer at Meta. Meta is an augmented reality startup. We create augmented reality eyewear. We have a product shipping called the Meta One and an SDK that's based on Unity. And using it, you can build augmented reality applications that allow you to see objects in 3D and interact with them gesturally using your hands.

[00:00:32.107] Kent Bye: I see. And so, tell me a bit about the origin of how this company came about.

[00:00:36.571] Soren Harner: So, Meta started actually while our founder, Meron Gribbets, was a student at Columbia University. He was taking one of Steve Feiner's human-computer interaction courses, and he got introduced to the idea of augmented reality from Steve. And Steve is rumored to have coined the term augmented reality. He's been thinking about it since way before it had a name or any of us have been doing it. And sparked on that interest, the meta team started to form. They got into Y Combinator, came from New York City to the West Coast to join Y Combinator, got funded out of that, and then the rest is history.

[00:01:11.901] Kent Bye: I see. So you did have a Kickstarter, so maybe talk a bit about the Kickstarter and then delivering on that and then sort of what's happened since then.

[00:01:17.747] Soren Harner: Yeah, so the business has been really built on our Kickstarter campaign. We had a couple hundred Kickstarter backers, and then we got an additional number of pre-orders. That allowed us to fund a lot of the development of our first product, the MetaOne, which we just started shipping to customers in last December.

[00:01:35.313] Kent Bye: And you've also raised some funding since then?

[00:01:36.793] Soren Harner: Yeah, that's right. Off the back of shipping our product, we announced our $23 million Series A. And so we're proud that that's been led by some strong investors like Horizons Ventures.

[00:01:48.210] Kent Bye: Since virtual reality is sort of on the leading edge in terms of within this year or next year starting to have like big mass consumer products, it seems like AR is sort of lagging behind in terms of both the technology and as well as the uses. So how do you see that kind of playing out with sort of the virtual reality and augmented reality playing off of each other?

[00:02:08.048] Soren Harner: Yeah, well let me take that in two parts. So first, the technology. The technology is different in the sense of AR eyewear because it involves a see-through display. And that means that if you want to create like a stereoscopic or 3D image, you have to figure out a way to make the display to project that in a way that's transparent. Also, there are other challenges involved in registering content to the environment. So when you're projecting the content into the environment, you want to be able to anchor it or make it aware of what's happening in the environment. So it brings you into a lot of other things more than just creating a 3D display like you have in VR and doing some head tracking. richer position tracking, you're anchoring stuff to the environment. There are a lot of technical challenges that need to be solved to do it in a way that's really convincing and natural and comfortable for people. There are also a lot of cognitive challenges to be thinking about as well in terms of how it interacts with the human perceptual system. So all of that and more is what keeps us busy every day at Meta. Now, second was in terms of like the use cases and how those use cases are evolved. Well, the clear use cases for VR are entertainment, like cinematic VR in gaming. And those exists for AR as well, but they're different. You have to think differently about how you create content for AR. And also, AR traditionally has had roots in a lot of other use cases that are applicable in various industry verticals and in productivity and other areas. And so, in fact, at Meta, our primary interest is not in entertainment and gaming. Our interest is more around productivity, more around things that people can use to do their work better and more effectively every day. We believe that AR is really going to be transformative there.

[00:03:53.631] Kent Bye: There's a mixed reality spectrum that ranges from having no augmentation to augmented reality to augmented virtual reality all the way up to virtual reality. And so some might say that the closer you get to VR, it's all about creating a sense of presence and immersion and taking you into another world. And so what would you say the intent of augmented reality would be then? How do you measure a successful AR experience?

[00:04:16.437] Soren Harner: Yeah, I think that's exactly right. There is a continuum from VR to AR. And the way you measure success in any experience, so first of all, are you doing something that people actually need or want? You know, and that comes down to use cases. And second of all, does it really integrate in with the human perceptual system in a convincing way? Does it make people frustrated? Are they comfortable? Is it understandable to them? And it's the fundamental way of measuring success with any new computing paradigm. Is it usable? Is it convincing? And is it natural for people? And does it get adoption and traction?

[00:04:53.259] Kent Bye: I see. And so what's the primary demographic for Meta then?

[00:04:56.508] Soren Harner: But the primary demographic for us at the moment are developers, because we're producing a developer kit. So we're really attracting developers onto our platform. And for what they do with it, really, that's for them to decide. They should be building the apps that they think are going to be most meaningful for their users and their customers. But we're adapting our platform in particular for things that people need for professionals, for developers, for people doing technical work. We see a lot of demand for data visualization, which surprised us a bit. But data visualization becomes more powerful when you can do it along a third dimension in 3D. We also see a lot of interest in 3D models. and being able to understand how parts fit together in mechanical assemblies or in architecture, in engineering, the BIM space, the Building Information Management, where you're trying to get lots of different people in different areas on the same page in terms of understanding how project timelines and components fit together in a massive building. So those are all different sorts of, yeah, I guess use cases in that area.

[00:06:02.415] Kent Bye: With Google Glass, I guess there was throwing augmented reality technology out into the wide world with, you know, there's all sorts of privacy implications and a lot of backlash that happened with that. What type of considerations is Meta taking in terms of, you know, is this something that you expect people to use in private situations or do you expect people to sort of take it out and onto the streets and the implications of some reactions to that?

[00:06:25.665] Soren Harner: Yeah, there are a lot of social implications around both AR and VR, and that's something that we're taking careful steps towards understanding. You know, we're doing focus groups, we're really thinking about how we describe the use cases, but to directly answer your question, we're focused on indoor use cases between, say, small groups of people standing around a table, really communicating and using digital content to facilitate that communication. And so that gets around some of the social things that we need to think about, but you can still imagine, like the CEO of a large company, what's he going to want to put on his head when he's in a meeting with other people doing telepresence or something else? There really are social considerations and acceptance to think about. And there's going to be a period of climatizing to these new form factors before people become comfortable with them. You know, Google Glass is an example of, you know, maybe where it got a little bit ahead of itself and the social implications became a bit of a backlash. And that could certainly be true, so we're watching that in what we're doing, so we're watching that carefully. But by taking the steps of doing, say, indoor use cases, letting people slowly get comfortable, you know, seeing themselves with a headset, an augmented reality headset on, we can avoid a lot of that.

[00:07:39.257] Kent Bye: And because it's an indoor use case, is it tethered to a computer or is all the computer on board to the glasses that are self-contained then?

[00:07:46.298] Soren Harner: Yeah, for the developer kit, it is certainly tethered to a computer. And, you know, and really, I think what everybody wants long term is to have as much freedom and mobility as possible. So, you know, as the technology permits, you know, and as costs come down and processing improves and computer vision gets on the chip, we can do more of that.

[00:08:06.533] Kent Bye: And one of the specs that Oculus VR had said that they're aiming for is like 20 milliseconds of latency, meaning that when you're moving your head around, that if it's lagging behind, it sort of breaks the sense of reality or presence or immersion. And so are you measuring your latency? And what sort of latency numbers do you see with your system?

[00:08:24.685] Soren Harner: Yeah, we are measuring our latency, and our latency is certainly a lot higher than 20 milliseconds. We believe people can perceive to around 30 or 40 milliseconds, so we're working towards that range. But we have a bit of a different problem, because since we have a see-through display, most of what is in your visual field is in fact the real world. you know, the augmented reality content is not taking up the majority. And so, we actually have not had problems with nausea or people having problems due to latency, except in rare cases. But there is, you know, latency is an issue, and it's really the end-to-end performance from, in our case, sensors that are tracking your hands and tracking the world, and that making its way through the computer system, through the computer vision algorithms, back out through registering content to the environment and the display. And so that's something that can make people uncertain about their interactions in an application. So that's obviously a huge goal for us to reduce that latency as much as we can.

[00:09:24.587] Kent Bye: And what kind of gesture controls do you have implemented in this first dev kit for Meta?

[00:09:29.245] Soren Harner: Well, we're doing something very simple. We're tracking basically your hand as a point cloud. And you can use the highest point on your hand, often your finger, as like a mouse cursor to point and click. And in our SDK, we have user interface elements. You can create buttons, screens. You can bring full web browsers into the world and interact with them just using your fingers and hands. And if you want to move them around, you grab. You can grab and then move something around. You can pinch. The various other things, but we want to stay clear a bit from introducing some whole language of gestures that somebody has to learn. We want somebody to be able to interact with it in a very intuitive natural way and we believe that is modeling the human hand as realistically as possible and using physics within the SDK to have actual collisions between your digitized hand and digital content. And that's going to be a much more natural interaction. Still, what's missing, of course, is haptic feedback, some real sense of touch. That's obviously not something that's easy to do directly with digital content, but it's something to always experiment with. And hopefully, we'll see some ways of simulating that.

[00:10:39.682] Kent Bye: And maybe you could talk about some of the other people that you have on your team. You've got quite a roster of augmented reality rock stars.

[00:10:46.924] Soren Harner: Yeah, well, we've been pulling a lot of people from Steve Feiner's lab at Columbia, from Steve Mann's lab in Toronto. We're right up the hill from Stanford, so we give guest lectures in computer vision and other things from time to time there, and are pulling a bit from that crowd. Really, we've got a lot of different skills that are needed to do this. We have experts in optics, we have people building sensor arrays, we have extensive computer vision team looking at problems around tracking the environment. And so, yeah, we've got a dev team, and we're going to be growing it even more to build that out, and we have a hardware team as well. And the important thing is to keep them as integrated as possible, to always be thinking about the end-to-end experience. So, yeah, in addition to Steve Mann and Steve Feiner, we have Jace Hansen as our creative director, and he's the Hollywood artist behind a lot of the graphics you see in movies, like Iron Man and Ender's Game and, you know, a whole number of others.

[00:11:43.489] Kent Bye: Yeah, I just imagine a lot of this sort of sci-fi user interfaces where, you know, the Iron Man. So the person that was actually doing some of those sort of design work is working here actually implementing it in AR, it sounds like.

[00:11:55.251] Soren Harner: Yeah, yeah, exactly. And Jace has been thinking about actually how not only what we want the user interfaces to look like, but since it's Hollywood, how you look while you're interacting with it. And that's important too. And then of course, Steve Feiner will come along and say, yes, but if you do that, you're going to get gorilla arm. And that's where like Hollywood thought meets like practical human computer interactions. And, and so that's why, you know, you would say that it's this interplay between these thinkers in these different areas that makes it really come together.

[00:12:25.407] Kent Bye: And for people who don't know or are not familiar with the work of Steve Mann, you know, who is Steve Mann and what has he done in this field?

[00:12:30.591] Soren Harner: Yeah, so Steve Mann is, you know, many would say the originator of wearable computing. Back in the 70s when he was at MIT, he was wearing head-mounted displays. He was essentially the first Cyberman in many ways. And he's been innovating in that field ever since. He's been, he's an inventor, he's very interested in extending what he calls humanistic intelligence in terms of like really how we can extend and transcend our own abilities. And that's really kind of at the heart of the vision at Meta, you know, to help people be more productive than we would otherwise be, help us manage more information, help us see better in low-light environments. And AR has a lot of potential in all of those areas and that comes from Steve Mann's vision.

[00:13:14.801] Kent Bye: So what do you see as the ultimate potential for augmented reality then?

[00:13:18.475] Soren Harner: Yeah, I haven't thought about it that way, the ultimate potential, because I see potential in so many areas. But I'd say the ultimate potential is something that helps us manage and find information, that helps us integrate the physical world and the digital world in a more dynamic way. I would love for AR to be just, you know, a massive conduit into the human brain. You know, we've got a lot of processing that happens in our visual system, and AR can really help us just digest information in much more higher quantities and help us, you know, manage more information. That's, of course, a really long-term vision, but we think that it can be a tool to help us think better, to help us be smarter, to help us transcend limitations. And really, it helps in teaching. There are a whole number of areas around education where we see it taking off. So augmented reality fundamentally enables you to bring together the digital world and the physical world. And you can use that to understand the world better, if you can pop up information around the objects in the environment. You can use it to teach people things, to show you how to put things together, take it apart. And so it's as a teaching and a communication tool, and then as a thinking tool to really help extend your working memory and other things. Great.

[00:14:29.844] Kent Bye: Well, thank you so much. Thank you. And thank you for listening! If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voicesofvr.

More from this show