Matt Bell of Matterport on their 3D camera for capturing physical spaces. The Matterport camera and software seems like a great solution if you wanted to quickly convert a physical space into a 3D model to use within a game context. Here’s the 3D scan that Paul Grasshoff from Matterport captured of the SVVRCon 2014 exhibition space, which was then imported into VR Chat.
At the moment, the primary use cases for the camera have been for real estate, home improvement, remodeling, construction, insurance, documentation of spaces, and crime scene investigation. But since they provide a textured mesh in the form of an *.obj file, then you can scan a room and within a half hour be able to get the file and import it into a VR experience that you’re developing.
Matterport is primarily a software company, and they use an iPad camera to be able to control their $4500 professional camera. One thing to note is that they are charging a monthly fee ranging from $49 to $149 to be able to scan, process and host a number of different files — and so there does appear to be a recurring fee to be able to actually use their camera.
You can either host the 3D model with Matterport if you need to have other people look at it online, or you can just export the textured mesh and start manipulating it for your game.
It’s also able to measuring distances within these models, and it’s dimensionally accurate down to around 1%, which is good enough for remodeling and furniture placement — and certainly good enough to quickly create 1:1 scale environments for VR. The current camera has a range limit of about 30 feet, but cameras with larger ranges will be coming in the future.
Matt also talks about how Matterport is developing some apps for Google’s Project Tango & Intel’s RealSense mobile 3D cameras, as well as lighting considerations when doing a 3D scan of a physical space.
Reddit discussion here.
TOPICS
- 0:00 – Intro of Matterport. Solving content creation problem of making 3D models of physical spaces. Place a Matterport camera in a space, take a number of shots, and it’ll build a model within ~30 minutes.
- 0:42 – What are people using this for? Real estate, home improvement, remodeling, construction, insurance, documentation of spaces, crime scene investigation
- 1:03 – What is the technology behind Matterport? Primesense 3D sensors that capture depth and color. Matterport software puts together the 3D puzzle pieces to create a coherent model of the space that’s dimensionally accurate.
- 1:57 – Using an iPad for the software end to control the camera. Uses the CPU to align the pieces & GPU to display 3D models.
- 2:25 – What’s the cost of a Matterport camera? Aimed at professionals at $4500. Writing apps for Google’s Project Tango and Intel’s RealSense, mobile 3D cameras. Built a demo for project Tango to scan a room
- 3:21 – What’s the output from Matterport? Textured 3D mesh. They allocate the polygons to describe complex models and how they’re optimizing the 3D model.
- 4:21 – What are some applications for how people are using Matterport? Scanning of famous monuments and world cultural treasures, and using the Oculus Rift to have an immersive experience of a space. Take spaces you care about and start to experiment with them. Make a model of your house, and you can play a game in your house or do remodeling with drag-and-dropping of furniture.
- 5:48 – Measuring distances within these models. Dimensionally accurate down to around 1%, which is good enough for remodeling and furniture placement.
- 6:24 – What type of file is it, and can you alter the file? Most people just leave the models on the Matterport platform and it’s embed code. You can download the model as a *.obj and then edit the textured mesh just like any other 3D file.
- 7:15 – Considerations for lighting? We have the real world as our rendering engine. Generally light the room that looks pleasing to you as you walk around it. What if you wanted to turn off a light later? Could get the geometry an paint it later
- 8:28 – Have people used it to scan people and faces? Not the focus. More focused on spaces. Mobile app will be optimized for more use cases.
- 9:15 – Is there a scanning pattern to avoid occlusion issues? Not really, just be sure that camera can see all parts of the room.
- 9:49 – In a room with high ceilings? What is the range? ~30ft high is the current limit. There are nextgen sensors that have a greater range. Matterport is primarily a software company.
- 10:30 – Matterport.com is where you can see more models and see their 3D cameras.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:12.008] Matt Bell: I'm Matt Bell, founder of Matterport. Our company's goal is to solve the content creation problem for platforms like Oculus. We make a camera that makes it incredibly easy to build a 3D model of any space. You simply place the camera in a few different locations in that space, and it'll automatically build that model in roughly about half an hour of cloud processing time. And then you can view that model online and soon on mobile and on Oculus.
[00:00:41.850] Kent Bye: I see. What are people using this application for? Sure.
[00:00:45.593] Matt Bell: So biggest markets right now are real estate and home improvement and remodeling. We also have people in construction and insurance and documentation of spaces, as well as a range of other things like even crime scene investigation.
[00:01:03.967] Kent Bye: I see. And so maybe talk a bit about the technology. What is actually going on to be able to create this 3D model of a room?
[00:01:11.392] Matt Bell: Sure, so our system uses 3D sensors, actually the same PrimeSense sensor that's in the Microsoft Kinect. And those are different from regular cameras in that they give you distance information as well as color. Now, those images with depth in them, you can think of them as like 3D jigsaw puzzle pieces. As you move our camera around the space, it produces all of these different 3D puzzle pieces. What our software does is figure out how those pieces fit together. So that as you move the camera around in real time, it's madly stitching all those pieces to create a coherent model of the space. So this is then full color, dimensionally accurate. It's essentially the same level of immersive 3D asset that you'd get in a video game.
[00:01:57.313] Kent Bye: I see. And so it sounds like you are also using an iPad to be able to facilitate that transfer. Maybe talk about that software side of things.
[00:02:05.735] Matt Bell: Sure, so you use the iPad to control the camera. The iPad has a beautiful display, a great touch interface, and plenty of both CPU for doing the alignment of the puzzle pieces, as well as GPU for rendering the model. And so it's a fantastic platform for controlling our camera.
[00:02:25.472] Kent Bye: I see. What is the cost for people to be able to use this?
[00:02:29.456] Matt Bell: Sure. So our first generation camera is primarily aimed at professionals. So it costs $4,500. So it's right in line with what they'd spend on other professional photography equipment. That said, we are working on a consumer version. You may know about things like Google's Project Tango or Intel's RealSense. Essentially what those companies are providing is a first-generation mobile device with a built-in 3D sensor. So our software stack is very similar whether it's running on our standalone professional camera versus running on a cell phone or tablet. And so we've already built a demo on Project Tango that lets you scan rooms using a Tango device in the same way that you can scan a room or a house with Matterport with our professional camera.
[00:03:20.924] Kent Bye: I see. So what is the actual model that you're getting? And I guess I'm thinking in contrast is if you were to sit down and design a model, it would have information that's more vector based and perhaps very efficient in terms of the file sizes. So I'm just curious about what you're doing in terms of optimizing this 3D model down into like the core elements.
[00:03:41.424] Matt Bell: Sure, so right now we produce a textured 3D mesh. We do allocate the polygons to provide much more detail in areas that have high geometric complexity. So things like chairs or house plants, things like that end up getting a lot of polygons, whereas floors and walls end up getting a much lower number.
[00:04:02.529] Kent Bye: How are you able to detect that in terms of the relative depth from the camera?
[00:04:06.979] Matt Bell: So a lot of the processing for optimizing the model actually happens fairly far downstream from the camera. It comes from analyzing a very high-resolution model that we produce first, essentially looking for areas that have certain geometric properties.
[00:04:22.005] Kent Bye: And so what are some applications that you see people using this Matterport camera that get you really excited about what you're doing?
[00:04:30.343] Matt Bell: Oh, all kinds of things. Personally, I am partial to the scanning of famous monuments. Being able to go into world cultural treasures and capture those in 3D and let anyone anywhere in the world experience them. Especially when you think about mating this with a platform like Oculus, where you have the immersive experience of being in a place. Now we're essentially able to take these famous locations that could be thousands of miles from where you are and bring it alive for you where you just put on a headset and suddenly you're there. And to me that's incredibly exciting. Other areas I find very interesting are that you can take the spaces you care about and start to experiment with them. So for example, if you live in a house somewhere, you can make a model of this house and then you could have a game that you play in a house where there's some digital creature that you're chasing around the house. Or you could go remodel the house by just dragging and dropping furniture in there. So there's this whole layer of applications that's going to get built on this easily created 3D content, whether it's for fun or whether it's for more serious planning purposes. And basically seeing what people do with that, You know, this whole application ecosystem that's developing, that to me is very exciting.
[00:05:47.768] Kent Bye: And can you talk about measuring distances in these models and what people are able to do with those accurate measurements?
[00:05:54.668] Matt Bell: Sure, so the models are all dimensionally accurate down to around 1%. We're able to get absolute scale from the PrimeSense sensor, and so that carries all the way through to the final model. So an accuracy of 1% is more than good enough to make sure that components are properly placed in a construction project, or if you're contemplating buying a new piece of furniture, you can tell how it's going to fit and how good it looks before you end up making an expensive purchase.
[00:06:22.580] Kent Bye: And so when you're getting this output from Matterport, can you talk about what type of file it is and how easy it is to go in and, let's say, just take a table and start moving it around?
[00:06:33.491] Matt Bell: Sure, so in terms of the general user experience, most users just leave the models on the Matterport platform. So it's similar to YouTube in that essentially once you create the content you can just put it up there and then it's on the web, you can send people links to it, you can embed it on other websites. So we've made it very lightweight so you never have to think about 3D models if you don't have to. But if you are the sort of user who wants to scan a house and put it in a game or something like that, you're able to download the model as an OBJ, and it's essentially a textured mesh. So you can do anything with it that you could with any other 3D asset.
[00:07:12.188] Kent Bye: And so maybe you could talk a bit about special considerations for lighting, for example. Do you want to have a well-lit room when you're scanning it? And then what kind of lighting do you do when you put it into a virtual environment?
[00:07:24.188] Matt Bell: Yeah, so it's interesting that we essentially have the real world as our rendering engine, right? It's an amazingly good rendering engine. So everything works perfectly without you adding any lighting. So generally, you'll want to light the room in a way that looks pleasing to you as a person walking around in it. And then the model will essentially reflect all of that lighting.
[00:07:46.112] Kent Bye: So I guess if you have no lighting, can you then generate all the lighting inside of the game, let's say if you wanted to turn off a light?
[00:07:52.475] Matt Bell: So you could scan a room in complete darkness and you'd get all the geometry. The thing you'd be missing though is the appearance of the objects from a color perspective. There are a number of different directions you could go with that, but you do need to have the lights on at least a little bit if you want to know what color everything is.
[00:08:09.773] Kent Bye: I see. So if people are really good at creating their own meshes, then they could get the geometry and then do that later, I guess.
[00:08:16.516] Matt Bell: Exactly. If you wanted to paint everything a completely different color than it is in reality, you can totally do that. It's your 3D model and you can manipulate it in the same way that you could any other 3D model.
[00:08:28.461] Kent Bye: And so have people used this Matterport camera to scan a human face or to actually get avatars of people?
[00:08:35.258] Matt Bell: That hasn't been the focus of the first camera we built. The camera is much more optimized for scanning spaces. That said, the next generation hand scanning that we're working on with some of these consumer mobile devices is optimized for a range of use cases. I will say that scanning human faces is kind of tricky because we all have machinery deep inside our brains for analyzing faces very, very carefully. So if one side of the face is an eighth of an inch higher than the other side, your brain will notice that and call it out as being something off or tricky. So in a way, face scanning, it requires a lot of special effort to get it just right.
[00:09:16.474] Kent Bye: And if you have like a square room, is there a certain pattern that you have to put the camera in or is it a straight line? I'm just kind of wondering about the occlusion issues that you may run into.
[00:09:26.906] Matt Bell: Yeah, you don't have to follow any specific pattern. You do just need to make sure that the camera is able to see all the major areas you care about. So, for example, if there's a big couch in the middle of the room, you'll want to make sure that you put the camera both in front of the couch and behind the couch. Otherwise, the camera won't actually know what's on the back of the couch. It can't see through walls.
[00:09:49.077] Kent Bye: And so we're in a room now that maybe has a ceiling that's maybe 20 feet, 30 feet, I'm not sure. What are the range of limits in terms of how high the volume that this camera can capture?
[00:10:00.130] Matt Bell: Yeah, so with the range of the PrimeSense sensor being roughly 30 feet, you can capture ceilings up to about 30 feet high. Beyond that, with the current sensor, it's not possible. There are, however, next generation sensors that are under development right now that will have greater range. And to us, we're primarily a software company. So anyone who makes a great 3D sensor, we can write a software stack that will turn data from that sensor into a high quality 3D model.
[00:10:28.949] Kent Bye: Great. And finally, what's the best place for people to get more information about what you've been working on?
[00:10:33.933] Matt Bell: That would be Matterport.com. You can see a number of different sample Matterport models there. You can also download some sample models if you want to play around with them in a CAD package. And then you can also order cameras directly on our site.
[00:10:47.964] Kent Bye: Great. Well, thank you very much.
[00:10:49.506] Matt Bell: Thank you.
