One of the most magical AR demos I’ve ever had a chance to see was Tilt Five at Augmented World Expo 2021. Jeri Ellsworth stumbled upon a complete paradigm shift towards AR while she was at Valve and accidentally had a beam splitter turned around and it was shooting beams of light that happened to hit some retroreflective material another Valve engineer was experimenting with. She saw a beautiful, high-contrast image that inspired her to continue to develop this idea into a fully-fledged, tabletop gaming focused AR headset that she was able to acquire from Valve after getting fired, go through another startup cycle of beginning and ending of castAR with too broad of a focus, and then eventually into Tilt Five, which has been much more laser-focused on tabletop gaming.
I had a chance to catch up with Ellsworth during a busy AWE showing, where she shared quite a lot of details about her journey into XR really starting with helping to bootstrap Valve’s hardware division, some of the internal dynamics there, her journey with castAR, and then finally with her latest efforts with Tilt Five. They should be shipping out their Kickstarter units here soon. It’s a completely different and unique approach to AR, and it’s also one that creates some pretty magical experiences when focused on a table-top gaming retroreflective material. It really feels like this is a device that is going to bootstrap quite a lot of innovation with AR gaming and the affordances of tabletop holograms, and I look forward to see how it continues to develop. But definitely keep an eye on Tilt Five, and try it out for yourself to see how they’ve been able to bring the magic of holograms to live with their Tilt Five glasses.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Here’s a previous interview with Ellsworth conducted by Valve News Network’s Tyler McVicker that I mentioned in this interview.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. So the demo that blew me away the most at Augmented World Expo was TELT 5. I had never had a chance to try it out before, and it was absolutely incredible what Jerry Ellsworth and the rest of our team at TELT 5 have been able to do. So it's a little bit of a paradigm shift for how you create augmented reality rather than having these waveguides where you're bending light at the limits of physics creates the smaller field of view. She's actually creating these projectors that are inside of the glasses that are shooting out beams of light that are then getting reflected from this retro reflective material and then back through like the series of different lenses that then give you this real depth of this augmented reality world. It's completely magical what they've been able to achieve with this and it's a complete paradigm shift from rather than using augmented reality glasses where you're walking around in the world, it's the bounded use case of sitting in front of a tabletop with a specific retroreflective material and you have this amazing depth of field and just real vital experience and just totally blown away with it. Also, Jerry Ellsworth has quite an epic history and story in terms of her journey into all of this. She was actually responsible for helping to start the hardware lab at Valve, which then created all these different augmented reality and virtual reality projects, which then, you know, eventually led to the creation of the HTC Vive and the Valve Index, but also Michael Ebrash and other folks that are there. which she subtly refers to initially as her arch-nemesis because Abrash and some of his other team are actually responsible for getting Jerry fired from Valve on February 12th, 2013. Then, Abrash jumped ship over to Oculus, and that probably pissed off a lot of people at Valve to the point where I don't know if they were actually going to have the commitment to build different VR hardware. But after that, it was a catalyst to really have Valve collaborate with HTC and to release the HTC Vive back in March of 2015 at Mobile World Congress. And then the first demo was this show at GDC, where I was in 2015. So till five, quite an epic story for how it went from valve and to cast AR. And that was another company that came and went and, um, was able to then get the assets and then eventually create the tilt five, which is just now going to be shipping the Kickstarter units any moment now. So that's what we're covering on today's episode of the waste of VR podcast. So this interview with Jerry happened on Thursday, November 11th, 2021. So with that, let's go ahead and dive right in.
[00:02:33.168] Jeri Ellsworth: My name is Sherry Ellsworth. I'm a co-founder and CEO of a company called Tilt 5. We're building augmented reality glasses that are specialized for tabletop entertainment. That's board games as well as video games. Our main focus is to bring people together so that you can have a group experience, something that people are really hungry to do. And if your friends can't be with you, we can link our game board system together and you can play over long distances and still feel like you're connected.
[00:03:03.172] Kent Bye: So yeah, maybe you could give me a bit more context as to your background and your journey into this space of immersive AR gaming.
[00:03:10.277] Jeri Ellsworth: Oh boy, this could be a long story. You've probably heard my story before, so I'll try to keep it brief. I was a very strange kid who ripped everything apart. I think I was an engineer from an early age. Got into early home computers. Loved them. Dreamed about holograms back when the original Star Wars movies came out. In my teen years, I became kind of a rebel and started racing cars. So I did quarter mile dirt track cars and sprint cars. And it was sponsored by British Petroleum. And I had a quite successful career racing, which was kind of interesting. I went from that to what I do now. In the 90s, after I got tired of the rough life of racing cars, I opened a chain of retail computer stores right around 1995, when Windows 95 was coming out and AOL disks were being mailed in the megatons worth. And it was a great experience. I opened a chain of retail computer stores, learned a lot about running businesses and about products, and did that until 2000, when the computer store market crashed right after Y2K. This entire time, ever since I was a kid, I had always been doing electronics as kind of a hobby. And in the 90s, I got into working with these custom chips that you could emulate ASICs with it, or full custom chips called FPGAs. So I decided even though I'm a high school dropout, never went to college, I'm going to come to Silicon Valley and see if I can brute force my way into startups and start working on circuit boards and FPGAs and doing engineering work. It was really rough. I came to events very much like this at AWE where at this point I was pretty broke because my stores had failed. I had to sneak into conferences and shake a lot of people's hands and show them all these little circuit boards I designed and try to convince them to let me do designs for them. Eventually I got a break or two and then developed this reputation as the person you go to if you need to. solve a tough problem and pull a team together and it kind of started to snowball from there. But what really put me in people's eyes is a toy company contacted me and they're like, we hear you can make custom chips and you have this passion for old computers and we see that you've reverse-engineered some of these old computers for fun. Can you make a chip that emulates the Commodore 64 all in a chip? I'm like, I've never done a custom chip at this point. And I was just like, sure, I can do that. Had to do it in a year. It was really rough. I've never worked harder in my life to get that done, but it was one of my most satisfying projects ever. So a little team of us, I did the chips, I did the circuit boards. Some folks did the software to load up all your favorite 1980s video games, and we did it. We got it on the market. It was a huge viral hit before I even knew what viral hits were. And they sold out immediately. In two weeks, they sold out everything they could produce. And then from there, I was able to springboard to all kinds of other exciting projects. So now I'd cut my teeth in doing custom chips, so I wanted to do more of that. So I worked on a reconfigurable processor. Back in the early 2000s, it was supposed to be for machine learning. I didn't even know what machine learning was all about. It never really took off. It was too early for machine learning. I look back and I realize, like, wow, we were working on something pretty cutting edge, but the timing was wrong. I designed some chips that went into TiVo to do video compression. It was really exciting. I did low Earth orbit rocket navigation systems and telemetry systems. Just did all kinds of really exciting projects. But really, what brings me to my passion for AR gaming is I was recruited by Valve Software. They actually stalked me. They thought that I was perfect to run their hardware lab, and they were sending me emails, contacting me through social media, and I was ignoring them because I don't want to go work for a video game company doing hardware. I just didn't believe they had the conviction to do it, so I was just dismissing them. They got more and more desperate. They start showing up. I collect pinball machines. So I'd go to these vintage pinball events. And here I'd be playing a pinball machine and someone would step up to the pinball machine next to me like, hey, you're Jerry, right? Hey, we're from Valve. I'm like, what the heck is wrong with these vowel people? They showed up at Maker Faire. Finally, Gabe Newell called me up, and he's like, I just want to take you out to lunch. I'm going to fly to Portland, where I was living at the time, and take you out to lunch. He came down. We had lunch. And he's like, we're doing really exciting things. You should really consider us. And I'm like, no, no, no, no, no. He's like, well, please, just fly to Seattle. This isn't an interview. Just spend an afternoon with us. It was a total lie. I got up there. They put me in this room. It was like a panel interview. So there's like 10 people in there and they're asking me like, we want to make a Steam controller. How would you do it? I'd be like, I'd go to this JDM and we'd do this. We want to make a Steam box. Oh, I just do this and that. And it was kind of really fun. And then I swear, Gabe Newell must have done a, like, no signal. And everyone got up and then left. And he's like, come with me. And he took me to the fourth floor. It's like, this whole fourth floor of our building is yours. Unlimited budget. Hire all your friends and the best people that you can get. And this is the mission I want you to think about. How can you bring the whole family together to play games? Right now, games have a bunch of little islands, and it's not approachable by everyone in the universe. Solve that problem. And I'm like, wow, that's a pretty ambitious goal. And he's like, stay another night here. I'm like, well, I just flew up for the afternoon. I don't even have clothes to stay here. And so he's like, well, we can take care of that. So someone takes me to their swag room and gives me like a valve software t-shirt. And then they take me out or whatever to a store and get underwear and toothpaste and everything else I need. And so next day I show up at their office wearing a valve shirt, you know, wandering around, just let me wander the whole building just to get to know people. And I guess I'm in like, But I was working at another company, pretty happy with the project I was working on. Well, I've got to finish this project. It's going to be like a couple months. And they're like, we want you now, like right now. Can we contact that company and just pay them to let us have you earlier? I'm like, I will not do that. I have to finish this project. But I came up with an arrangement where I'd work halftime for them and weekends for them to get things off the ground and then finish my other project. Pretty amazing.
[00:10:01.567] Kent Bye: Sorry, that was really random. What time frame did you actually go up to Bellevue and to Valve? When was that approximately?
[00:10:09.971] Jeri Ellsworth: I think it was 2010. I'd have to go back and look. It could have been around 2009, maybe. It was quite a long time ago.
[00:10:19.775] Kent Bye: Okay, yeah, that's really I didn't know a lot of that sort of evolution in history I watched an interview that Tyler McVicker did with you to get some of the aspects of that story and I guess because you were there maybe the next phase is your time at valve and then what happened to then you getting fired for what what happened and What was that period that happened at Valve? Because clearly there was something about the seed that was planted there, that provocation from Gabe, that is the spiritual successor being Tilt 5, that started with that. And so give me a bit more context of what happened at Valve after you got there fully and you're done with this other project, then what happened?
[00:10:56.701] Jeri Ellsworth: Valve is a very amazing, interesting place. So they pride themselves in not having a management structure. And so good projects will bubble to the top just because you can attract people to your project and other people within the company believe it's good. However, there is a few people in there that can influence things, like Gabe. He's like, I'm doing hardware. Now we're doing hardware. And so in the early days, the first couple years, it was absolutely amazing. So I was working there. We had a great team. We hired some of the best people around and we're just doing pure research, like understanding how people play games and like why other people don't play games. We had psychologists on board just understanding we're doing all kinds of testing. It was hardware and gameplay mechanic testing. It was just amazing, but we didn't have any direction yet. And so some of the projects we worked on, we were trying to read people's emotions and feed those back into games to see if we can heighten the gaming experience. Now I think back about ethics and I kind of realize I'm like a typical engineer that doesn't think about the implications of what you're doing. If you take like biofeedback signals and put it into game, you could make very highly addictive game mechanics that maybe isn't the best. But anyway, we were doing that and it was exciting. We were doing VR, we were doing AR, we were throwing ungodly amounts of money to just see what it would be like. So we had like military headsets and we got medical tracking devices and hooked them up and we had software resources that were just pushing the graphics cards to their limit to be like, okay, what's it feel like to have presence in VR? What's it feel like to have presence in AR? What's the best feeling interactions? And so our crew, there was like a core crew. We would get in to work at 10 and we'd be there until like 2 in the morning. We were so excited about what we were working on. It was just pure magic. And then so you have to productize things after a while. And I come from a product background. So I started to focus on AR. I'm like, this is the mission that we're going after. This is really the tools that will bring people together. You can't transport people off into a virtual world and have that same intimate connection with them across the table. Grandma's probably not going to want to do that. And so I started putting all my efforts in there. How can we productize that? And so we were trying to solve some of these really tough problems in AR. The biggest problem, and people usually don't talk about it, is displays and resolution and all that technology is easy. The problem is the world combiner. How do you combine the real world and your virtual graphics that you're putting in there? And so there's things like waveguides. So we were doing waveguides. We were doing holographic optical combiners where we're projecting off axis into holograms. You know, we're doing pen light stuff. We were doing folded prism displays. We just were doing tons of Experiments to try to solve this world combiner problem so that you could see the world and have a good Graphics experience and on top of that a lot of people were also on the team working on like tracking and slam and We had a room called the marker PTAM room, which was insane it was just like every single square inch had a April tag on it and
[00:14:24.164] Kent Bye: That's the Valve demo room, which I had a chance to be in once. But yeah, that's where they did the fiducial markers, right?
[00:14:30.307] Jeri Ellsworth: Yeah, yeah. I mean, we were so slap happy. Sometimes we'd be in that room staring at these fiducial markers. And we'd start to see things in the patterns in them. I really loved the AR group. We had Pooping Bird, which was the center of our universe. Which, if you look closely, it looked like a bird pooping. ghost fire trash can or something. We had dog bone. Anyway, it was a fun bunch. We started all this research in AR, and a VR group spun off. And then those tensions really started to form in the group. There was a group that's like, VR is it. We're going to start doing video passthrough in the future, and we'll do AR that way. And it's like, I was receptive to it. We were doing lots of experiments, like, can we do AR with video pass-through that was good, which we could never manage to do. There's just so many problems with it. With all these depth of focus issues you have with fixed cameras and depth of focus issues with VR systems, you always end up in this, like, very uncomfortable, uncanny universe where things just don't feel right ever. So I was like, I don't think that's a path forward for us. And so actually there was like a person running that group, then we were doing this other stuff. And so one day I was to the tilt 5 angle, like how did I find the technology to do the tilt 5 stuff? This was a pure accident. So I was trying to do a near-eye display, a direct retinal scan system with a beam splitter in it. And I put a beam splitter in backwards. And so instead of projecting into my eye, it was projecting this laser light out into the room. One of my colleagues had hung a piece of this material called a retroreflector on the wall to bounce lasers off of for a tracking experiment. And I looked at this retroreflector 30 feet away in the room, and I saw this beautiful, high-contrast image. I was like, Holy cow, that's weird. And then I flipped my beam splitter around and kept doing my other research. Months went by, and I just kept thinking about it. And I'm playing with this retroreflector. I'm like, there's just something here. It just seems like it solves some problems. And then what if it was in a game board type form factor where you just flip it open on the board and slip your glasses on? So I prototyped it. And I'm like, oh yeah, this is working. So our first prototypes were ginormous. It was like a welding helmet, like ratchet thing, like band, with these giant projectors attached to it to generate the images. And I remember this other gentleman that was in the VR group that was my maybe arch nemesis, we'd say. came over to give it a try and he just did not like that I was like pushing so hard in the AR space and like doing this kind of weird non-intuitive AR display. His comment on it is like, this is like gravel for the eyes. I'm like, OK, OK. So we kept developing. I kept pushing. And I actually went to Gabe a few times. I'm like, hey, Gabe, I think I'm onto something. How do I navigate this weird environment inside Valve? He's like, you know our beliefs. Always do the right thing for the customer. And if you have conviction, advocate for it. And just remember, some of the best things that have happened inside Valve happen with people with arrows sticking out of their back. not very comforting. So I kept pushing, right? And at the same time, we had a lot of other controversial things happening. Carmack was coming through, he was at Oculus at this point, they'd just done their Kickstarter, and this other group, the VR group's like, well, we should seed those guys with all our technology. And the AR folks are like, no, we're building products here. Why are you giving away all our tracking technology, telling them about wands and like all these haptics that we're working on? And became really, really a pressure cooker. I came into work one day and I'm heading towards the elevator and someone comes out of the elevator and they like, you know, have you heard what they did to Dave? I'm like, no, they fired Dave today. I'm like, my mechanical engineer? What the F? And so I get in the elevator and I go upstairs to the hardware lab. And it's just like, who died in here? It's just like, everything's in turmoil. And I'm like, how could they fire my mechanical engineer? And someone's like, you're getting fired, too. Just wait for the email. And I'm like, what? How could this be? Eight months ago, Valve does this trip where all the employees and friends and families of the employees get flown over to Hawaii for a one week vacation. So I was over there, Gabe Newell had me stand up in front of everybody and he proclaimed, Jerry is the most valuable hire we've ever made. Right? And then I'm coming in like maybe eight months later, like you're getting fired. And so, you know, the email comes through. It's like, you know, Gabe wants to see you at like three o'clock. And it's like 10 or something or 11 o'clock in the morning. So we're just all sitting around like everyone knows who's getting fired and like leaked through the org. So like all the fired people are sitting around just like grumbly. We all go out to the bar across the street, you know, waiting for our time to. to get fired. It was the most bizarre layoff I've ever seen. We were free to just roam the building. And I had people coming up to me like, oh my god, you brought such a playful air to the company. Like, I can't believe it. It was just the weirdest thing. And so I'm preparing what I'm going to tell Gabe. So I march up to Gabe's office. When it's my time, I walk in the door. I'm going to just tube his butt out. And I'm like, so this is it, right? And I'm just going to try to be super aggressive. And then I quickly devolved into tears. I can't believe you're doing this. You're killing my project. It's going to be so good. And his response is like, I don't know. People say you're too abrasive to work with, and you don't fit the culture. So you've got to go. I will always be a fan. And I'm like, oh, I can't believe this. And as I was walking out the door, I just turned to him. I'm like, Gabe, you should just sell me that technology. And he's like, OK. Boom, $100 and a lot of probably $10,000 worth of legal paperwork. I was able to take everything.
[00:21:22.056] Kent Bye: Wow. Yes. Well, I think I mean I've heard glimmers of that with your conversation with Tyler and as somebody who's been in the XR industry Tracking the evolution of the medium. I see that there was this dialectic that was happening between Valve and what at the time oculus really going back and forth and different stuff and you know Blake Harris came out with a whole history of the future which is really telling the story through the oculus side and anybody that was at valve that went over to oculus aside and
[00:21:50.063] Jeri Ellsworth: That's not the truth. Yeah, when I read that I'm like, that's not the truth. I even talked to Blake I'm like man, that's not right. You need to get more perspective of the valve people and I was really sad There wasn't like the other side of the story. Of course, we all look through our own prism. I
[00:22:05.498] Kent Bye: Well, as someone who's also been covering this story, Valve is notoriously not willing to really have some of those discussions. I know that Steam Dev Days, Robin Walker gave a talk about, here's our philosophy for how we communicate with the world. And it amounts to community-driven interactions, where they're really focused on the products in the community. But that's been part of my frustration of covering the history, is that I know so much of what you helped to start there at Valve, and then has continued, has been a huge part of shaping the overarching Aspects of the entire industry of XR especially with room scale and all these things if it would have been just left up to the devices of oculus we would still be using controllers
[00:22:41.990] Jeri Ellsworth: have a funny story about that. So Carmack comes through. He's now aligned with Oculus. And so I was always the person that did the whole hardware lab tour. So VR, I'd just take them around the different areas. And the AR group was big into wands. Wands and haptics, that was what we were all about. And Carmack's like, well, Don't think gamers are gonna like wands in room scale. I'm like oh You are really wrong about this. We had this like pretty heated debate. I'm like this is really you want room scale He's like no I think they're gonna want to sit on the couch and just enjoy games very similar to how they used to and so I'm fired from Valve. OK, here's another side story on this. This person that was my maybe arch nemesis there, jump ship with the entire VR group, goes to Oculus after pitching all the technology over the fence to them, and then gets a huge payout when Facebook acquires them one month later. I was furious. And then I started getting calls from some of the key folks at Valve. It didn't help the pain at all. There were apologies, like, you were right, you were telling us, like, there were some, like, bad actors here. And, in fact, some of those people are invested in Tilt 5 now. So that's, you know, another kind of interesting story about exiting Valve is, like, my story is really important to me. Like, I never agree to have hush orders or something on my story. So when I left, they had an exit package and it was very lucrative. It was $40,000, $50,000, but I had to agree that I would never talk about anything inside Valve. And I'm like, no, I'm not going to do that. And the same day, I just tweeted a picture of, like, I brought, I'm a pinball collector, so I had scattered pinball machines throughout Valve. And I took a picture of all my pinball machines with the head boxes closed up so you couldn't actually play them. Because I'm like, no one gets free pinball on my machines, because it's going to take me a few days to get the machines out. If I'm fired, no one gets to wear my machines out for free. But I took a picture, put it on Twitter, like, got fired today. And I was like, this was terrible PR for Valve because there was so much hype of me going there in the first place and then getting fired like three or four years later was just like, what the hell happened? And then the rumors got out that like they fired like 30 or 40 people and it was like the biggest like cleansing of Valve ever. And it was all about like people that didn't culturally fit, right? And so Folks from Valve, especially on the HR side, are like, what are you doing? You're talking to the press. Oh my god. Sign the papers and stop talking. And I'm like, you can't. It's insulting, the amount of money that you're offering first. I can be bought out. OK, fine. I can be bought out at a price, but not at that. And so I talked to the press, and I'm still talking to the press saying, this is what Valve is like inside. Meanwhile, I'm negotiating to get this AR technology out of the company. It was bizarre. Some people were not happy with me, yet I was able to push it through.
[00:26:00.443] Kent Bye: Wow. Wow. want to put a close to some of that because, you know, we could go on and on and on. I have so many questions. I mean, just one comment, which that your arch nemesis is likely Michael Abrash, who you were talking about. But maybe let's transition into, OK, so now you're fired. You're able to speak freely, which I'm happy that you've done that. You've done conversations with time. Vicar, I've been wanting to sort of catch up with you to get some of this story, just because it's part of the history that's so important. But you managed to maintain your autonomy to be able to speak freely, which is amazing. somehow also get all the technology that you're working on, which then has now come and turned into Tilt 5. So what happened then after you're fired and then you have access to all this intellectual property? Where do you pick up from there? And to where you're at now with doing the Kickstarter and launching? So what happened before you launched the Kickstarter? What was the process for you to take the technology to then get it to the point where you'd able to actually go around to different communities and start to demo it?
[00:27:03.241] Jeri Ellsworth: Yeah, that was interesting. So I'd worked at startups for a decade and a half or maybe two decades, right? And I experienced what a startup is like. And I'm like, well, I can do this, right? I can do this. But I was a little scared. So Oculus had just done a pretty decent Kickstarter. I'm like, well, maybe we should just bootstrap this. Some of the folks on the AR team came over to what we call CastAR. Well, first, actually, the company was called Technical Illusions. And we're like, well, that's not a good name for our product. So we were going to call the product CastAR. And then eventually, we dropped the Technical Illusions name and just called the company CastAR. So there's that piece of the evolution. And that happened over maybe six months. I was outside of Valve. We had the gravel-for-the-eyes prototype when we left the company, so we had to actually push it far enough that people, if they squinted hard enough, they could realize what we were trying to do. In my living room, I started building little tiny projectors, which are the core of our system, and put that together. built some prototypes Got those working my co-founder Rick. He was working on the tracking system So he got some tracking going for us and then we started going to events. We went to Maker Faire We went to Penny Arcade and we started showing like what year was this that you're actually showing it was it like 2013 that sounds right gosh, I
[00:28:26.194] Kent Bye: Oculus was released in March of 2013. Oculus was bought by Facebook on March of 2014. You must have been fired 2013.
[00:28:37.423] Jeri Ellsworth: before the Oculus acquisition, but not maybe four months or three months after I was fired, because I was furious. Like, my dream had been ripped away from me, and then the people that were, like, lobbying, saying that I was a bad influence inside Valve, jump ship and take the technology with them. So I just remember, like, having phone calls with some of the head folks at Valve. Like, FN knew it. I knew it. So yeah, yeah, it checks out. That's about the right time. Yeah, yeah. And there was lots of hot glue and zip ties. And there's pictures out there. These things are god-awful ugly. But they worked. And it was a very magical experience. It worked good enough. Then we did a Kickstarter. We did quite well. We sold like a million bucks on Kickstarter. And it's like, OK, we're doing it. But I had a serious imposter syndrome issue going on at that point. I should have been the CEO of the company, but I was too scared. I'd never been the leader of a company like that before, besides my little computer stores that I had in Oregon. So I hired an external CEO to come in, who was quite good. He was a scrappy startup CEO and helped us raise money. And things were so frothy in the investment community at that point, Andy Rubin came in and dropped $15 million on our laps. And it's like, whoa, this is really happening. And so we start working on the product. We were about ready to go into manufacturing for the Kickstarter units and send them out. And then I got pulled into a meeting with our investors. And they're like, that Kickstarter thing is a waste of time. You need to be going for a moonshot here. And at this point, Magic Leap was pushing out tons of unrealistic hype. You know, whales jumping out of the floor and like stuff that's not even possible in AR for another 40 years, right? And so that spooked our investors and like you got to do a moonshot like right away. You got to do something bigger. Stop doing the Kickstarter thing. Just refund all their money. So we figured out how to do that, which was good. We ended up winning a lot of hearts by like giving everyone their money back. And then we started working on an all-in-one AR system. And then there's all kinds of drama like started. So Andy's connected in the industry with various chip manufacturers. So now we have to give up these chips that we were going to use. We had to pivot to different chips that invested into Andy Rubin's incubator that we were at. And so big resets in the hardware. And then they're getting more and more spooked about Someone said Apple's coming out with AR glasses next week, right? There's always been a boogeyman for us like definitely next week Apple's coming out with AR glasses that are as small as yours and more immersive than an oculus and Which is not true. And they're like, I'm patting my fist on the table. No, no, no, it's like it's impossible physics physics You can't bend light that fast So then they got upset with my CEO because they're telling him, like, blanket the earth with CastAR systems. And he's like, I don't know how to fund this. Like, you're telling me to ship a million units the first year. You know, that's like a $400 million, like, investment we need. I can't go from a $15 million investment to a $400 million when we're still, like, resetting the design. We don't have any customers yet. And they're like, he's out. So the board votes him out. Then they start bringing in a string of, I can't believe they actually use this one time, we're bringing in a rainmaker for you. These are all personally fine people. But running startups, being an executive from Disney or Sony, they don't have the chops to do scrappy startup. And so the company became very schizophrenic. It was really a bizarre thing to experience. We went from a pretty narrow focus. We, in general, thought games were what we were going to do. But we didn't have laser focus at the time. Now we're gonna do medical imaging, education, and it was just like layers and layers and layers and layers upon layers of what we were going to do with the system. Some of them weren't appropriate for the system, right? But we were gonna do them. And burned through this money like crazy, and they were hiring all their buddies, they acquired a couple game studios to try to fluff the company up, and boom, we're out of money before we know it. Meanwhile, I was just shoved into a corner, like, just sit over there, be quiet. At one point, I was on the board of directors and I had voting rights. They tried to get me to sign papers to say that I would only have voting when there was a tie to break. I'm like, there is no way I'm signing that. That's a horrible signal to send to a founder that you're so unimportant you can't be making decisions. Anyway, crazy. I learned a lot. I mean, the CastAR time frame, I learned so much about running a startup and what not to do that it's made me so much better at running till five. So OK, we make a boom. crater in the ground, $15 million up in smoke, and so many funny stories around that, but we don't have time for the last days. But I'm sitting in the office, my professional executives have exited stage left, and I have to figure out how to get the company into liquidation. You know, they're such experts, they just left that to me, right? But I'm sitting there, and I get this random phone call. And I'm like, hello, Nolan Bushnell. I didn't even know you knew who I am. This is the founder of Atari. He just randomly got my number and called me. He's like, what you were building, I got to see it at an event. You're going to change the world. You know, it's magic. And this is coming from someone who's messed up a lot of startups. there's always a way to figure this out. So I'm just challenging you with a, just go figure it out. And so I'm sitting there like crying in my beer and like get this pep talk from founder of Atari. I'm like, well, maybe there is a way. So I reached out to all my mentors that are in the kind of business space and like, can I do this? Like I just had this huge failure. Like, is it possible? They're like, yeah. In fact, in Silicon Valley, blowing a crater in the ground is a rite of passage. Everyone does it. It's not a negative. It's a positive. Just think about everything you did wrong. Figure out how you're not going to do that same thing next time. Have those answers. And just spend a bunch of time figuring out exactly what you're going to do this time. Don't just grab a bunch of money and go in random directions. And so I pulled in some of the core CastAR team, the most loyal and talented people. It was three of us.
[00:35:32.067] Kent Bye: At that point, did CastAR dissolve completely? And then you were able to then acquire all the IP? Or is that what happened?
[00:35:40.173] Jeri Ellsworth: Yeah, it was a very interesting situation. So all the assets, patents, we had tons of super valuable patents and stuff, go up on the auction block, but it's a blind auction. And so there's this process, and one of my mentors told me how to game the process a little bit. So I had to be around, and some of my personnel needed to be around to explain the technology that was going to be sold to these different companies. And they're like, well, you don't have to explain it any better than just the most bland way possible. So we made the most bland executive summary. I mean, so bland. It was like just trying to make it seem like the most unappealing offering. And the bank, because we had some bank debt along with this. So the bank actually owned it. So here's our executive summary. We were the most boring, bland people whenever the big companies came through. And we had this really interesting situation. We knew who was going to bid and who wasn't going to bid just by the way that they were engaging with us. And we were actually outbid for a time being by way more money than we could scrape together. Way more money. And the CEO of this particular company really kind of like falling in love with my kind of methodologies of doing stuff. And he's like, if we buy this stuff, will you come to my company? I'm like, no, no. If you buy this stuff, you're on your own because I'm going to do this no matter what without that IP. And he's like, OK, how about I just invest the money we're going to pay for the IP into your company? We'll have more upside. You really sound like you're going to do this come hell or high water. And the bank folks were so mad. They're like, what did you do to scare those people away? I'm like, I didn't do anything. I was just being myself. So, I mean, that's ethical. I don't know. But it's true. I wasn't going to go there. I was going to do this. So we won the assets. And we had this first investment, which was pretty big. And it's like, that was incredible. And so my co-founder Jamie and I sat down and we're like, OK, what are we going to do? Like, that really sucked at the last startup. And he's like, well, the biggest problem is you weren't our leader. You've always had the vision of how this was going to go together. Like, even when we had other CEOs in the company, the personnel would come up to you to get guidance. So you have to be CEO. OK. I can figure this out. The other thing is we have to be laser-focused. Like, we can make compromises in our system to nail one type of experience, and we have to resist all these other shiny temptations out there and just nail this first thing. Then we can layer the onion down the road. And so that's what we've done. We really focus the company as, like, we can do video games on the table, top-down perspective. They're amazing. You can play with your friends. It's magical. They're holographic. You can add new features like hand tracking in or a wand and manipulate things in 3D space. It's the things we've dreamed about since Star Wars. We can do board games, right? So that's an easy one. There's lots of ways we can get board games on this and add like, you know, a new magical experience to it. Save games, set games up instantly, solve all these tough problems for people that want to play games in the round. And that's what we focused on. And it's really resonating. Like, we are so backordered on kits. It's going to take us into probably January of next year to get caught up now that we're shipping units.
[00:39:16.153] Kent Bye: So when did you do the new Kickstarter? Because you did the CastAR Kickstarter, and then you refunded the money, and then you did a new Attil 5. So when did the Attil 5 Kickstarter happen then?
[00:39:26.176] Jeri Ellsworth: Oh, this is kind of an interesting story. Maybe boring for your listeners, but from my growing a startup, it's interesting, I think. So I was pretty successful with this new conviction going out in this clear vision of what we were going to do. I raised about a million and a half, maybe it was a million six, to bootstrap the company for two years. So there were these things we wanted to do to our headset to make it really a slick experience. We wanted to have all this upscaling of the frame rate in the headset and all the tracking to be a tight loop in the headset so you could plug into a phone, a PC, a laptop. and have a variety of compute platforms to use on it. So we worked really hard and we, with our investors, we set goals of like, we'll have this done by this time, we'll have the plastics and the industrial design by here, and we were progressing. And we were getting little tranches of money from different investors and having success, but we reached kind of the end of the road of how far we could go without customers. So every investor we'd go talk to and they would be giggling and having a good time with our system. Like, this is great, this is great, but you don't have any customers, sorry. And so we had this one investor that was pretty keen on us. They're like, if you could prove you have customers, we're in. Go do a Kickstarter. And I'm like, oh man, I don't want to do a Kickstarter again. Kickstarters are one of the hardest things you can do. There's all this lead-up marketing you have to do and after doing one of them I knew that I had to do a ton of that so we had to get on to social media and start to like figure out how to have a voice and I'm glad we did it though I mean it was like this is exactly what we needed to do. We probably would have been struggling more today if we hadn't like figured out how to talk to the community and refine our message. So we started doing that three months or four months in advance. We started being controversial so that people would listen to us, like throwing oculuses in the trash can, like, this is going into the trash. You're going to want our stuff instead. And just trying to see what resonated in the community. And then we did the Kickstarter. At this point, I had talked to enough press. I kind of knew how to line up press. We had a lot of really good press lined up and embargoed. And they all came through and saw that it was legit and works. And Kickstarter themselves came through. They're like, hey, you fucked up that last thing. But you did it in a nice, clean way. So maybe let's get together and let's do this a little cleaner this time. And let us be involved. So Kickstarter was involved and very helpful to make sure that Everyone was cool along the way. We did a lot of bloodletting along the way, we called it. Because in customers' minds, we had the stain of CastAR, so we had to get the story out. This is what happened with CastAR. We fucked up. We admit it. We were dumb. We were naive. We're going to do better this time. All that got done before we launched our Kickstarter. Now we had about $40,000 in the bank. There was only three of us in the company at the time. We all went on furlough, and it was like a poker chips moment. This is our marketing money. I pushed all of our money into marketing as much as I could, and that was it. That was as good as we could do, and our conversion rate on marketing was amazing. Out of $40,000, we raised $1.8 or $1.9 million. Largest ever AR project on Kickstarter. If I only had more marketing money, that was the most agonizing thing. I was pulling out my credit cards to put them into Twitter for more marketing dollars and maxing credit cards out. I was just like, if I had another 5,000, I could probably sell another 100 units.
[00:43:06.503] Kent Bye: Yeah, I remember at the Oculus Connect 6, which is the last sort of in-person gathering that was happening with the VR conferences. I remember I was trying to sync up with you and you were busy at that point. I think the Kickstarter may have finished or completed. I wanted to get like the story of where you were at at that point. And I hadn't at that point been able to try it out. And so the pandemic happens. There's lots of different stuff happening with parts not being as easy to manufacture, to go through all that. And then we're here at Augmented World Expo, which is kind of like the return to the XR industry, which had really started at Silicon Valley Virtual Reality Conference 2014, which was the first professional gathering. And then that's when I started my podcast. And so I've been sort of tracing it up into that gap. And then we're coming back, and it's like everybody coming here, what's happening. And then it feels like getting a chance to see the demo, it felt so magical just to be able to feel like it was like such a, it was a lot wider field of view than I was expecting because of there's kind of a marker that you see on the table, and it cuts off there, but it's still able to feel like I'm really immersed into that space. And I think the other aspect is the controller, which is, most of the controllers are either like a 3DOF controller or even using your hands, but to really put all the buttons that you need to have proper games and having different orientations. For me, my experience was that, you know, the wand of being able to interact with it, but also the depth that you can get from what you're seeing. It started to see how, just like the Oculus DK1 started with games, this feels like really tuned to have people open up their minds for what's possible for augmented reality gaming with their co-present with other people. And I feel like as all the different stuff that I've seen here at Augmented World Expo, I was like, wow, this is like the most mind-blowing thing. And to really focus on that context and to really nail it. So yeah, it feels like a celebration of people coming back together, but to kind of get their mind blown again to the point where there's incremental progress that happens, but something that really is just coming out from a totally different perspective and really picking a very specific context and use case and being able to really optimize all the different trade-offs that you have to do. And I think what you were able to do with the Tilt 5 is just completely nail all those things.
[00:45:13.240] Jeri Ellsworth: Oh, you just made my day. Thank you for that. I'm so proud. I was just telling one of my co-workers recently, it's probably been eight years of this, right? I've been rugged, yanked out from underneath me multiple times. And to finally get this into the universe, either the world's going to love it, be neutral or hate it, I don't care. Like I've got to see it through once and see what happens and I'm so happy because I saw this glimmer of light at valve Even though it was gravel for the eyes I just knew where it could go and if we just kept refining it and you know maybe this like heroes journey I went through was necessary for Timing to line up in the industry for us to develop like we added some light field technology to it So you can focus at different depths like we didn't have that in the early days You know just to have all the pieces come together this reprojection so no matter what frame rate the game is running at you always get like this perfectly buttery smooth and tracking and image lock and being able to refine the controller so that it can be like a Nintendo style controller or a pointy jabby thing. Yeah, it's part of the evolution. I have a funny story. So in the death throes of CastAR, I got an opportunity to pitch the company because they're trying to flip us to some of the big companies. We had Bill Gates come through, super delightful. When he came through, had the demo, he was like deadpan the entire time. I'm like, oh my god, we're blowing it so bad, so bad. And then he turns to me, he's like, this is the best AR I've seen. It's better than some of the stuff that we're doing at Microsoft. It's very good. And then he starts going into radians per degree of this and that. And we had this really interesting technical talk. I'm like, wow, OK. You just can't get the wow moment like you can out of almost anybody else. Pretty chill experience. The next week or two, all of a sudden, all right, Zuckerberg's coming through. Security crew comes through. We have to, like, isolate people away from Zuckerberg. It was, like, this really weird air of, like, royalties coming in. And it was like, oh, boy. So Zuckerberg comes in, gets a demo. And he's like, hmm, oh, yeah, you can get a couple of giggles out of him and stuff. And then he's like, I don't understand. Why did you make the field of view bigger than the board? I was like, well, that's what you want. You don't want your head locked in one place. You want it out of the corner of your eye when you're looking at your friends, still see your game off to the side. Like, that's why. And then he said something kind of snarky, like, well, what Abrash is working on is going to be cooler. And I'm like, F you. Oh, I didn't say it, but I was thinking like, F you, dude.
[00:48:00.372] Kent Bye: Did he know the back story for what happened between you and Abrash at all?
[00:48:03.633] Jeri Ellsworth: I have no clue, like the whole interaction with Zuckerberg was like, he has handlers and it was like really prescribed and I'm like, that was my second time running into Zuckerberg, I've had bad experiences with this guy. I was at CES and his handlers came through, I was at the Epson booth looking at their AR glasses and like the gentleman I was talking to was like, oh my god this is great, Jerry Ellsworth's here, like I love your work at Valve, whatever. And this was way back. And then his handlers come up, like Zuckerberg wants to take a look and step between me and the guy I'm talking to. I'm like, OK. And then Zuckerberg comes up and looks through the Moverio glasses or whatever. And this engineer guy's like, oh my god, I got Zuckerberg and Jerry here. And it's like, he didn't even acknowledge me. I'm like, needless to say, I don't have a lot of respect.
[00:48:55.191] Kent Bye: Well, one of the things I really respect with your approach that you've taken is that it seems so community driven going to many different types of communities I mean, there's the tech conferences, but also going into like tabletop gaming conferences I don't know if that was a part of your Kickstarter strategy, but also just getting the word out but really being in tight conversation with what is presumably going to be the end customers and so Did that fit into the marketing part of the Kickstarter? Or that seemed to be a key part of just getting the word out, just saw lots of social media of you going all over the place. And also a diversity of different types of places to show this, both in tech conferences, but also more tabletop gaming conferences.
[00:49:32.552] Jeri Ellsworth: Yeah definitely, I mean it's our whole team like if you say there's a culture for a team like our whole team like really at our heart believe this you could probably ask anyone on our team like you know what are you about and they're like oh bringing people together making their life delightful like that's us that's what we're going to do and so Anywhere we can take this where we can delight people like we want to take it there We want that audience and we think that's a huge broad audience. It's hundreds of millions of dollars billions of dollars of opportunity out there You know I think back like talk about Nolan Bushnell and the Atari 2600 You know and the pong games. You know I've talked to Al Corn and Nolan recently and like what was it like and When you wanted to bring a Pong machine into a hundred million homes, they said everyone thought they were nuts. Like they couldn't get anyone to buy it. None of the retailers would buy it. And they, I believe the story was something like they got Sears and Roebuck or Montgomery Ward to like stick it in some crazy place in their store because they so didn't believe that anyone would want to like play Pong in the living room. And it kind of feels like that. It's like we know people want to do this. We know trends show. If we look at how people play video games today, like 70% of people that play games now play to connect with their friends. It's all multiplayer. And it's kind of a broken experience. Like in my living room right now, to have a multiplayer experience, I have two big televisions, two PlayStations, two Xboxes, two Switches. And it's kind of broken, like I have to face forward, I can only play certain games where it doesn't matter if I see my friend's screen. You know, I'm not really bonding with them very well, just kind of maybe elbow to elbow on the couch or something. What we're striving for is just like, oh, flip the game board up and now we're going to be like face to face. Sometimes we're going to be co-op and we're going to be like solving a problem. Sometimes we're going to be against each other and I'm going to be able to give you the evil eye and it's going to be so fun because we can do that.
[00:51:38.757] Kent Bye: Technical question is that I see the tilt 5 headsets here are plugged into a computer Does that mean for everybody that has a tilt 5 glasses will they need to have their own dedicated computer? That's running it
[00:51:50.050] Jeri Ellsworth: Yeah, so we can work on tablets, Android tablets and phones. We can do PCs for Windows and Linux right now. On PCs, laptops we think is probably the use case that you're going to see someone throw a laptop on their dining room table and do this. We believe most laptops will be able to do two headsets. It's usually down to like how many USB hubs are in your, because we chew up a lot of bandwidth on USB and then you're sharing it with other stuff in the computer.
[00:52:17.235] Kent Bye: Okay, so you can run one Unity game and drive multiple headsets, maybe even two or three or four, depending on how many USB ports there are?
[00:52:24.792] Jeri Ellsworth: Yeah, and how powerful your graphics card is and stuff. You can run multiple instances of Unity games as well, like our system, we can just pair it off to different instances. But, you know, all these games now are networked. A lot of our games, we have about 40 games in development, about 30 developers, you know, working with the system on the game side. And the genres of games are there's sandbox games, like there's painting and sculpting things. There's puzzle games. You probably played one of them the mummy puzzle game thing. There's action games We have like battle planet and some of these things where you're like co-op and you're running around like shooting stuff up We have some RTS is we pretty much have a little bit of everything in every genre You can imagine then on the board game side. We have a couple platforms that we ported over to the system So we have tabletopia It has 140 board games, so the public domain ones that you would think of like checkers and chess and poker and stuff like that. Then they have a ton of licensed content like Euro-style games in their platform. That one's exciting because it's free until you save your game. You know, they get you there. And then for people that want to do role-playing, where you want to have your minis on the table, we have a couple kind of sandbox-y map generation tools so you can make your dungeons and your maps. And then you can scroll them back and forth so you have this very vibrant 3D world that you can put your minis down on the table and do your D&D campaign. So, it's interesting as I hired Hans to be my business development guy and I told him from day one, you're the most important person in this company. We are selling fucking holograms. Our shirt says, ready set fucking holograms. That's what we're selling. We're not selling a piece of plastic with sensors or a wand with buttons. Like, that was part of our problem at CastAR. We were like so excited about Ooh, we generated this new IMU sensor fusion thing. The geeks loved it, but 100 million homes don't care about what sensors you put in. So yeah, my DevRel team and business development team are the most important people. I'm like, forget with those software people. They always think they're pre-Madonna's and the best. But you guys, because you're getting all the holographic content. They've done a great job. We have some really exciting news that's going to be in the next four months or so. We signed some big titles.
[00:54:44.987] Kent Bye: Yeah, the ecosystem that you have, I think, is really quite impressive. I'm here at Augmented World Expo, do lots of what are essentially tech demos that aren't really that interesting. I mean, it sort of shows the technology, but till 5, there's these games and, you know, just crowds and crowds and crowds of people trying to get a sense of what the future is headed, because you've been able to hit a use case that is able to really use the affordances of the technology in a unique way. One question around the multiplayer aspect, because you did say about and the intention trying to bring people together in the same physical location, but they're, I imagine, have aspects of have people in remotely or is that something that you're not focusing on? Is it mostly having games with people that are in the same place or also adding aspects where people can be at home and being able to share a tabletop experience like in a virtual experience?
[00:55:31.639] Jeri Ellsworth: Both, both. So fortunately for us, all the game engines, Unity and Unreal, have amazing network stacks on them. So that problem's been solved a couple decades ago. And so if your friends can't come over to play the game with you, you can just link your game board to theirs. It happens naturally. So I would say probably at least 50% of our multiplayer games are games that you can do over the network. And it's just seamless. It's just like how you launch a game on Steam, and you find your buddy, and they launch it, and you connect together.
[00:56:02.670] Kent Bye: How do you do the audio? Do you have a virtual representation of the avatar of people? Or are they invisible, and you just hear them? Or do you see their wand? What kind of presence do you have with the other people when you're watching some of these experiences?
[00:56:15.085] Jeri Ellsworth: Yeah, it depends on what the developer wants to do. So you could do just wand interactions and just teleport some kind of representation of their wand. Voice chat, right now we're just leveraging Discord or whatever you want to fire up in the background and use for voice chat. We'll probably integrate something a little closer to the system down the road when we're a little bit bigger. But yeah, it's up to the game developer what they want to do on a lot of this. It's really cool when you start thinking about how you can have symmetrical and asymmetrical views into the game. And this kind of goes back to how I was talking about how things are broken if you try to have a multiplayer experience in your home, or maybe even remotely. But sometimes you want all players to be privy to all information so you can have a shared game space and everyone gets to see everything. But say a real-time strategy, there's going to be a fog of war. So your friend sitting across the table is not privy to where you're placing your military troops down until the fog of war lifts and then you have your epic battle in the middle of the table. And so that's a really fun dynamic that developers get to play with on our system. It's like, you know, what's shared, what's not shared, what's symmetric, and also what's positional and what's not positional. So I mentioned Battle Planet. This is a game where you run around a sphere. It's very much like Robotron. You can have multiplayer in it. This game, even though you're sitting across from each other on the table, you don't have any direct one-to-one spatial position because one player might be on the bottom of the planet, the other might be on the top, and you're running all over this sphere shooting the bad guys. Then we have other games like Tabletopia, where it's like a board game, where you want to have spatial relations on everything, so the poker chip is the same place for each player. They may be looking at the back side of the poker chip and you're looking at the front side of the stack or something, but it's all spatially located. And it's super easy to do. We've created these really easy tools in Unity, in Unreal, to do real-time editing. So you can have Unity open, and you can be testing these asymmetrical and symmetrical views in real-time. You wear the glasses, you move something on your computer screen, you see it move instantly out on the table as you're making your adjustments. Or like, I want to test occlusion on this one so that Player 2 can't see it. You can just click a button. It's not present. That's another thing. I work with a lot of mentors to help me through this crazy process of bootstrapping a startup. And one of my mentors is like, if something's difficult for the developer, just make a tool to make it easier. And so that's kind of our mantra is like, we run into something that's difficult. A lot of XR development is you have to build it in Unity, then download it to the device, then look at it, and that cycle is really slow, so we put a lot of emphasis on what can we do real-time to speed up development. Most of our games come from third-party developers. These are existing games. that have an aesthetic that we want or a gameplay mechanic and we think, well, if we add wand or we add some special effects to it, it's going to be even better. And so we can take their code base, drag and drop our Unity SDK into it. Within five minutes, most of these people have it rendering on the table. Now there's just like, Well, I have these menus. Where do I want to stick them? Are they going to float? Are they going to follow you when you walk around the table? Just a few things like that. Oh, I want to use the wand. So do I want to use the positional? Or do I want to use it like the Nintendo game controller configuration? They get to kind of tune it up.
[00:59:50.147] Kent Bye: Yeah, that's amazing. And I wanted to make two comments. One comment is accessibility in the sense of I've done a lot of demos here at AWE. A lot of times if you're someone like myself who wears glasses, you can't really even get a good experience of these headsets. Lenovo think reality a3 or even the spectacles or magically both times they have special lenses So aside from the hollow lens one and two which was probably one of the more accessible AR headsets But everyone else is going this weird direction where you need to wear contact lenses in order to have any experience with their AR glasses But with tilt 5 I was able to wear it with my glasses and not only Was it comfortable to wear with my glasses? But the field of view seemed to be way better than all the other things. So there's something there I don't even... What's that?
[01:00:39.236] Jeri Ellsworth: We just crush everybody on those features. So the thing that I recognize by turning this optical system inside out and having this board aspect to it is this solves the fundamental problems everyone's chasing after in one swoop. You want a light field so you can focus at different depths so that foreground objects go out of focus when you focus on something up close. It's natural so you can wear it for hours because it's not straining your eyes. You want to have an eye box so big that even if the glasses are tilted on your head you can almost have them like at a 45 degree angle and you still have a great experience because our eye box is actually bigger than our lenses eye relief. So if you wear glasses like so many of these Optical systems like when you try to magnify an image really big from a really tiny display I box and I relief go down to like almost infinitesimally small when you get in these high magnifications because we don't have to do that we actually make the light take a path down to this retroreflector and back again and You can actually hold the glasses maybe four inches away from your face and still get a stereoscopic image if you line it up with your eyes, because our eye relief is so deep. And then we obsessed on the form factor, just obsessed. I have a book, I should share it with you sometime. We printed off about 400 pages, the entire evolution of how we got to our industrial design, and a lot of thought went into it. Got to be light. We obsessed on every ounce of weight. I'm still pissed. I could have made it lighter, but we didn't have enough time. Approachability. So what's the shape? Some of our team are more into aggressive look. And they're like, oh, we want angles and black. And like, no, maybe we should keep it neutral, white, curves, friendly. Grandma can pick it up. It doesn't feel like a piece of tech device. And a lot went into it. No straps, no ratchets. That was important. You just slip it on. It should stay on your head. You can shake your head around. You should try it when you go do another demo. You can shake it around. It's almost impossible to knock off your head, but it still feels like a pair of glasses going on. All that was so important and really hard to get right. Honestly, we haven't got it right yet. We still could improve.
[01:02:58.258] Kent Bye: So what is the technology you're using? I mean, I'm not an AR technology journalist. I hear people talk about waveguides, but is this waveguides? Are you rendering digital light fields? Or how do you describe what this display technology is then? Is it something that you've invented and created that no one else is doing?
[01:03:14.665] Jeri Ellsworth: That's right, and we patented it to the hilts. Yeah, so when we start with light fields, that's kind of a hot topic. Light fields, you can do digital light fields, that's really difficult. You have to render every single focal depth and you have to have some kind of display element that generates every single one of those independently. The Looking Glass is a really great example of that. They kind of condense the problem into a small space, only do light fields in the X and Y, they don't do Z, which, you know, you're squaring your problem when you try to make it work in every angle. That's a digital way of doing it. What we discovered, and this is actually recent, really recent, and something I'm pretty proud that I was able to discover, So the light that leaves the glasses goes through a pinhole, which makes it infinite depth of field when it goes through a pinhole. And then I made this realization that I could actually diverge the rays to closely mimic actual light fields just by having a secondary corrective lens after this pinhole. But there were a lot of challenges around that efficiency. I had to come up with an illumination system that could get as much light through a little tiny hole as possible and then diverge it out. And so we get that for free through a hunk of glass that we stuck on there, a really cleverly placed hunk of glass. And so because we know the constraints, we're not trying to boil the ocean and put this everywhere. We know you're going to be within a meter or two of the game board. you know, we can tune it up so it works naturally. So now you can sit there and play for hours without eye fatigue because it looks like these are real objects that are really sitting there at the right accommodation, vergence, distances and focus.
[01:04:58.189] Kent Bye: Does it solve the vergence accommodation conflict where you're able to have different depths of field where you're focusing and then as your eye focuses you're able to actually see different depths?
[01:05:05.821] Jeri Ellsworth: Yeah, yeah, yeah. That was the Eureka moment a few years ago. It was like, if we just make this small tweak to this projection system, it's like, within our constraints of tabletop, it works. It won't work if I try to project 100 feet away. But that's, who cares? That's not our experience. So that was really neat. Some of the challenges we had to do is we had to make these projectors small, light, high performance, 180 frames per second, low cost. Like, we had to make the system so low cost that it's just a no-brainer. You can buy like 10 of our systems for the price of a HoloLens or a Magic Leap. It's because we're not using $600 waveguides. We cleverly turned the optics inside out. We used this $1 per square meter retro-reflective material.
[01:05:56.995] Kent Bye: So usually, I know Tom Furness had a patent on being able to shoot I-beams into the eye, but this is taking that same idea of digital light fields and then inverting it and projecting it outwards. My understanding of digital light fields is it's sort of like a quantum wave of potential, but that you are collapsing it. In some ways, you have to render it. So in a 2D projection, so you're projecting it onto the frame of the virtual camera. So is that essentially what you're doing? Are you having an intermediary digital light field, taking something from Unity and converting that into digital light field? Or is it just rendering it like a regular Unity camera?
[01:06:29.886] Jeri Ellsworth: It renders it just like a regular Unity camera. But because the light is traveling the right distances, we get to take advantage of light leaving the glasses and traveling the right distance. So an object up close, light rays don't diverge as far as light rays that go to the far end of the board. And so when you converge back there and focus on something at the back of the board, it's the proper divergence of the rays. So it's like zero compute. Another zero compute thing we get for free. I've been obsessed with retroreflector for eight years because it solves so many of these problems. Occlusion is a huge problem. It's really difficult. You have to machine vision. You have to be really good at recognizing things like your hand or your wand. your miniature, whatever you want to blend into the real world. Ours, we get that for free. If you put your hand out in front of the experience, the light passes right through your fingers, or it passes right by the wand, and we don't have to compute any of that. Now, the downside is there's always trade-offs, right? So if I have a miniature on the table or something and I want to have a virtual character come stand in front of it, well, my miniature is blocking part of the retroreflector, so you can't have, say, a virtual character in a real-world miniature or something standing in direct line with your eyes. But there's some other interesting, like, human perceptual things that is really neat. Like, if you have something occluding part of your retroreflective screen and you just move past, in front really fast, like if a bullet goes by, your brain just fills in. It just looks seamless. Your brain's like, yep, that bullet just passed in front of that real-world miniature. If you ever get to come by the office, you know, this stuff's not easy to, like, demo in a frantic AWE experience, but you can come by and I can show you some of this stuff. And this is, these are the exciting techniques that future developers that get the kits are going to be able to exploit and make some really amazing experiences.
[01:08:24.783] Kent Bye: Okay, I think I understand what may be happening where I didn't it was like a mysterious thing because I go through a lot of these demos here at AWE you put on these glasses and you have this tiny little window and it's so frustrating because it's like you want it to be fully immersive and so you have this retroreflective material on the board and and that I felt like if I were to really lean forward, it would fill up my entirety of my field of view. But it was occluded at the retroreflector, so it sounds like you have the Tilt 5, there's a projector that's shooting out light beams from the glasses onto the retroreflector, and then reflecting off of that onto my eyes.
[01:08:58.362] Jeri Ellsworth: Yeah, yeah, that's exactly how it works. There's two projectors, one for each eye. And you see these silver lenses in the front. You just think it's a reflector. But this is actually like eight layers of special films to control polarization. If you just had a regular reflector there, you'd only get 25% of the light back to your eyes because there's so much waste in it. Ours gets 85% of the light that's projected out of the glasses back to your eyes. But what's great about it is it floods the entire Aperture this whole area where the silver lens is and so your eye can be anywhere in there That's why we have such a massive eye box and because the light rays are actually coming from the correct angle. They should be So if the glasses are tipped on your head for instance, or you have them up or down or something You don't notice like any IPD problems. You don't have any chopping off of the image because The light rays would have come from that location in real life if those were real world objects running around the table. It's one of those things when you start like grabbing a pen of paper and drawing it out, you're like, holy cow, this like changes everything about how to do AR. Because one of the challenges, like wave guides, people have been promising wide field of view wave guides forever. And it's just like, when you start to do the math on how fast you can bend light inside of a slab of glass, it's just like, there's no way to do it. It's like you'd have to like, have thousands of displays all around the waveguide and have light coming in from all kinds of different angles and perfectly stitch them together. Someday, someday they'll do that. But that's why we're seeing like 30 degree field of view or like best case like 40 degree field of view and some of these birdbath ones. You know, sometimes you might like the meta, the old meta stuff. Not the new meta, the old meta. The AR meta, yeah. AR meta, the true meta. They had a kind of clever thing with some, you know, kind of gnarly trade-offs. It's like really distorty, but they had this giant, like, visor they put out in front of you and this compound reflector that, you know, it's fixed focus, it has distortions, but it does pretty good.
[01:11:07.701] Kent Bye: Okay. Yeah, it's just was very baffling to see it. So I think I understand it I guess that question would be if you were to just expand out a huge Retroreflective material would it make it feel even more immersive then because you have a pretty small Size, but you have projectors. Is it really tuned for that size? Or could you expand out the size to get even wider? Experiences because the field of view is kind of bounded by the end of that retroreflective material if you expand it Does that then make it wider or is it then start to break because it's limited by the projector
[01:11:38.408] Jeri Ellsworth: No, our field of view fills the frames of the lenses, so we have the giant lenses in there and we fill that entire area so you don't see any of the drop-off. So we just tuned our projectors to fill as much as you can see through these giant lenses. And so we sell two kits. Actually, we're probably not going to sell the economy kit. We're showing the economy kit because it's easier and smaller and compact for our little tiny booth. But we sell a pro kit which has a long rectangular board and it has a cool feature to it where you can kick the back of it up so that you can have this infinite view out into the distance. So you can either play it flat like if you're doing a D&D game where you want to have like this giant map in front of you with all your miniatures on it. Or if you want to play a video game where it's like has some kind of action that's going off into infinity just flip up the kickstand and then you just look out across.
[01:12:31.467] Kent Bye: Yeah, I think that Bad VR had that, where they had the little section there that was able to kind of give it, it was an additional portal. So, okay, that helps to explain all that stuff. So, yeah, as we're moving forward here, you have the development kits from the Kickstarter shipping here soon? It's like, what happens next after AWE?
[01:12:48.314] Jeri Ellsworth: So I'm really excited to share. We've shipped out hundreds of dev kits to developers, and that's how we have all this third-party content coming. We just went through a beta cycle with our Kickstarter backers. These are folks that paid extra to get janky, early stuff. And so they helped us through USB connectivity issues and projectors shutting off when a barbecue lighter or something is struck across the room. Weird things, lots of weird things. And so this actually started last week. They started mass production in our factory in China. First units are going to come to our office next week for 100% inspection. And then they're going to go out to our first Kickstarter backers. And over the next two months, we're going to try to fulfill all the Kickstarter backers. We have a pre-sale button on our website. You can go pre-sale for like $5. We have such a backlog. We're probably going to be pretty late into January before we get caught up at this rate. It's been a real challenge. You asked how long ago the Kickstarter was. It was like two years and some change. We would have been late anyway, but not as late as we would have been with the COVID situation and not being able to go to China to work in the factory. We had to invent some technology so that we could do this stuff remote. So our glasses are designed to be low cost, so they have to go together quickly without any kind of adjustments by the factory operators. So we have this whole robotic line of calibration and inspection machines, and we had to really make those robust because we couldn't be there to work with them. So we have all these remote access robotic machines, air quote robotic, it's just a bunch of things swinging the glasses and wands around, but pretty proud of the team putting together a system that we can monitor manufacturing like on a glasses by glasses basis as they go through the factory.
[01:14:42.342] Kent Bye: Wow, wow. I'm so impressed with everything that you're doing here. And congratulations on this showing. And yeah, I'm just curious for you what you think the ultimate potential of these immersive technologies, I guess more augmented reality for you, and what they might be able to enable.
[01:14:59.025] Jeri Ellsworth: Well, I think augmented reality and XR technology is going to be the future. We're going to probably not use our phones in like 20, 30 years. It's probably going to be some kind of XR technology. I can't predict the future, what it's actually going to look like. But I just know it's the next evolution. I'm excited to be on the forefront of it. And I'm excited that we get to work on AR somewhere that makes people's lives wonderful. It's counter to Meta and some of these other companies that want to just rip you out of your world and stick you into completely virtual. I don't think that's the right step. I believe that families are going to want to be engaged with each other in their home and not being transported off somewhere else in the early days. But I think as AR and XR evolves like this, entire spectrum of what is VR and what is AR is going to be completely blended together. We'll probably have glasses in 40 years that can do everything, you know, so when I want to play a board game with grandma, I do that in AR mode, but when I want to have amazing VR type experience, it just occludes the whole world and I play my VR game that way. When I was at Valve, another anecdote, I was probably stomping around the hardware lab like, VR comes after AR. VR comes after AR, which is probably why they called me abrasive. But that's my belief. VR comes after AR. MARK MANDEL-WALDAU Awesome.
[01:16:28.814] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[01:16:32.959] Jeri Ellsworth: I am so excited that we're starting to have events again. I wish we had gotten a giant booth here where we could have just had, like, 20 tables set up with our third-party developer games there and just let people sit down and play for an hour. Yeah, so I'm excited for, like, as these events happen, we start to do those things where we can have, instead of all the little quick demos where we have to shuffle people through, where people can really sit down and use it. And I guess the other thing, too, that maybe we haven't talked about, That's important. Every one of these kits, when you install the drivers, you get all the developer tools with it. And they're super easy to use. So if you're a professional developer, you'll appreciate the tools. If you're just a hobbyist, you can have a lot of fun with it. I do. You saw our forest demo running through there. That was a prototype I put together in like two hours just to dog food or SDK. And the team liked it so much they hired an external company to add a little polish to it because it was Jerry code. But I'm not a coder. I'm like an optics electronics person. But if I can do it, anybody can create an experience on our system.
[01:17:41.905] Kent Bye: Yeah, it really reminds me of the early days of the Oculus DK1 where it was really the people that were developers just being able to make stuff. So I feel like that's the equivalent in the AR space now where I'm just excited to see what comes out of it.
[01:17:55.020] Jeri Ellsworth: I hope that we can democratize AR. I mean, it's really unfortunate, like, awesome systems like HoloLens and Magic Leap cost $3,000 because there's a lot of cool things you can do with them, but it's out of reach for most people. So, you know, here's a way for a few hundred bucks, you can have a really magical experience or learn how to do XR experiences.
[01:18:20.221] Kent Bye: Awesome. Well, Geri, thank you so much for taking the time to share your story. It's been epic.
[01:18:25.869] Jeri Ellsworth: I hope it wasn't too rambly. I feel bad when I start going off on all these war stories. And back when I was three years old, I...
[01:18:35.372] Kent Bye: I think it really helps inform where you're at now, because you wouldn't be here without going through each of those phases. And someone who's trying to capture a lot of the oral history, I think you've filled in a lot of the gaps. And there's more gaps to always be filled in, but it's a good sense of what you were involved with and helped to really kickstart at Valve, has continued to move into what's happening in the VR industry. excited to hear that you're able to then take that seed and maintain control over it for this long to be able to be at the point you are now to be able to launch this into the world. So congratulations on this amazing showing here at AWE, and just thank you again for taking the time to help unpack it all here on the podcast.
[01:19:11.284] Jeri Ellsworth: Thanks for having me. It was super fun.
[01:19:13.340] Kent Bye: So that was Jerry Ellsworth. She's the co-founder and CEO of TELT5 and working on augmented reality glasses that are specialized for tabletop entertainment to be able to bring people together so that you can have a group experience within AR. So I've a number of different takeaways about this interview is that first of all, well, holy crap, that was. Quite an amazing, epic hero's journey of all the things that Jerry has gone through. Not only just getting more context about her life growing up and then all these different contract works and sneaking into different conferences and making ASICs and FGPAs, but eventually creating a digital clone of the Commodore, 64, and then eventually getting stalked by Valve to be able to go there and essentially start the hardware group. That hardware group there has been able to continue to do some of the most innovative work of all the XR industry. A big part of the history of VR itself has been this conversation that happened between Oculus and Valve. For Jerry's point, she was extremely skeptical that they should be collaborating so closely with these other companies, which ended up burning Valve in the end in terms of It being sold to at the time Facebook now meta for two to three billion dollars and essentially was the catalyst to have valve Double-down and to really commit to producing some of this different hardware that they've been tinkering with and at the time augmented reality was squashed and there's actually a lot harder problems overall with the computer vision and everything and I think, actually, the approach that Jerry is taking is such a paradigm shift that it literally could happen now, rather than all the other stuff. The same type of stuff with augmented reality that we're seeing with the Qualcomm XR2 and XR1, all these different devices that you see on the floor of Augmented World Expo, they all have a certain approach, and they're all doing a waveguide approach where they're bending light. But this is just a completely different paradigm shift. It's very constrained. It's not the type of thing you could be walking around out in the world. It's something that is going to be very private. tethered to either a phone, PC, or tablet that you're able to have one or two meters. And because of that, they're able to really tune the trade-offs and to really create maximum field of view and have these wands that are the sixth off ones that also can be turned around to be used as an antenna controller. So lots of different ways of engaging and interacting with these different environments. And yeah, it just feels like a magical world and just the lights that you're seeing. It's hard to fully describe. It does feel like a hologram. that they've been able to pull that off by shooting off these light beams into these two projectors above your eyes and so that you don't have to like really account for the IPD because your eyes are able to see where the lights are going and having proper depth of focus to be able to shoot these light beams and you get it back and your mind just synthesizes it all as these immersive 3D worlds. And just all the different things she had to hack together and build this. She's basically like a MacGyver of technology, just to be able to do stuff that no one has ever really figured out yet. It all started with having a bean splitter turned inside out, and I happened to have other people at Valve doing retroreflective materials and playing with lasers, probably with the lighthouses or whatnot. Lots of different tracking technology were laser-based there. This specific material, which is pretty cheap, is able to reflect the light back and it just looks amazing. One thing that I would note is that when I was at the Bad VR, some of the retro-reflective material had little scratches on it, and so when you do have scratches or marks on the retro-reflective material, it's sort of the equivalent of when you get a scratch on one of your eye lenses or a mark with some dead pixels on here. headset within virtual reality so just making sure that the retro-reflective material is pristine and not have any scratches that it starts to then mess with the light as it's being reflected back and it breaks things a little bit so I think as people move forward just being careful with not getting it too scuffed up but it seems cheap enough that you can start to replace it and I think there are some fiducial markers that are in there just to help stabilize the scene as well At the edges of the retroreflective material, it starts to cut off and get occluded. But if you start to lean in, it starts to fill up the entire field of view, which feels amazing when you actually start to play with it. I do think that this is going to be a system that is going to catalyze quite a lot of innovation when it comes to tabletop AR gaming. And to take the existing systems that are built in unity Start to add in the plugins and be able to convert some games that already have been built And played out and start to just you know have it so that people are playing around in a tabletop and just the other aspects of You'll be able to actually have the retroreflective material and be able to shoot things from different angles and still have people have their own view So it's not always just like this shared view So I don't know if that has to be 180 degrees apart 90 degrees You know what the limit is and where if you do start to ever get some type of conflict I didn't get to test any of that out or to ask more specific questions about that. But I imagine that they've been able to figure it out to have multiple people around the same area where they all have their headsets, which means that they have different ways of each having their own view into the world to either do cooperative play or non-cooperative play. And so there's ways that you can see what their actions are, but not know exactly what they're doing. And so, yeah, just the different types of symmetry and co-op versus competitive play within that. Yeah, just overall super blown away with the tilt 5 and really excited to see where this goes because I think Jerry's really on to something here with Creating what was originally called by Michael a brush as gravel for your eyes but it's really quite a magical portal now when you start to see it and I I'm excited to see where this goes and to see what folks are able to create with it. There were some other data visualization stuff there. BadVR had some demos there where they're starting to do geospatial visualizations that you can navigate around. I think there's probably going to be a lot of other use cases above and beyond gaming, but really starting with the gaming first to be able to start to prototype what's possible with the technology. I think that's where you get the most excitement, people coming together. Whether it's a D&D campaign, where you're able to start to have these imaginal worlds that you have a window into, or some of these other real-time strategy games where you're playing with them. I think those are the two probably big game genres that you're starting with. Just starting to play with what kind of portals you can create into these immersive worlds that have some depth of focus. So yeah, excited to see where the larger community takes it. And this is quite a unique approach and, uh, yeah, just a epic, epic story. And so glad I had a chance to sit down with Jerry during AWE, which was probably the hit of the show with so many people really excited to be able to see it for the first time, or just to see the latest developments and all the other tunings that they've been able to do over the last couple of years during the pandemic. So, that's all I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
