I had a chance to participate in a panel discussion exploring “The Intersection of AI and the Metaverse: What’s Next?” on the main stage of Augmented World Expo on the opening day on May 31, 2023. The discussion was moderated by Amy LaMeyer (Managing Partner of WXR Fund), and also featured Tony Parisi (Chief Product Officer, Lamina1), Alvin Graylin (Global VP of Corporate Development , HTC and President of China for HTC). Graylin and I engaged in a bit of a dialectal debate on a number of issues at the end, which was a bit of a continuation of our lively conversation in episode #1185 at the start of SXSW 2023 exploring our differing perspectives about AI. You can check out the video recording our AWE panel, or tune into the podcast version which provides some additional context and the beginning and end (or also check out the rough transcript below).
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. It's a podcast that looks at the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So continuing on my series of looking at the intersection between XR and artificial intelligence, today's episode is a panel discussion that I was on at Augmented World Expo, looking at the intersection of AI and the metaverse, what's next? So this is a panel discussion that was moderated by Amy Lemire, a managing partner at the WXR Fund, and Alvin Wayne Graylin, who's the Global VP of Corporate Development for HTC and the President of China for HTC. And originally it was going to be Joanna Popper, who unfortunately was not able to make it. And so Tony Parisi stepped up, who's the Chief Product Officer for LaminaOne, to participate in this discussion as well. So I wanted to include this panel discussion in this series just because Alvin and I do Get into a little bit of a dialectical debate at the end and I think there's different perspectives of the artists and maker and creator perspective of trying to push forward the technology and I guess there's other ethical implications of the technology that also deserves trying to think about how to be in my relationship to all the ethical and moral implications of the technology and it's a complicated thing because I think And as I go through this series and I reflect upon all the different conversations I had with artists and makers and creatives, a lot of them are on that bleeding edge of pushing forward the potentials for artificial intelligence. And I think in some ways Alvin's advocating for that perspective to continue to push forward to see what the potentials of these technologies are. And I think where I get a little bit hesitant is when there's advocating for, we can figure out ethical issues later, and that there will be different processes that need to be decoupled from the potentials of the technology. And that, you know, solving those different ethical issues may take different means, whether it's through regulation or having different teams who are trying to red team these different technologies. What are the different responsible innovation infrastructures that we need to have within companies to be able to? Address some of the arms that are being brought about from the technology So anyway, yeah, we sort of discuss some of those different perspectives here at the end of this conversation But yeah I think this is also just a good overview of some of the different Potentials for not only the underlying technology of what even makes modern XR technologies with computer vision and everything else I mean there's so much machine learning and artificial intelligence that's embedded at all the different levels of XR and There's so much about these virtual worlds and XR that are driving innovation and AI as well So I really see them as these sibling technologies and I'd refer you back to a couple episodes ago where I was able to dive into My plenary thoughts on artificial intelligence that I included in this series as well to recast both the history but also a landscape of the different folks that I'm listening to to be able to understand the conceptual frameworks for sense-making around artificial intelligence So that's what we're covering on today's episode of the Voices of VR podcast. So this panel discussion with Amy, Tony, Alvin, and myself happened on Wednesday, May 31st, 2023 at the Augmented World Expo in Santa Clara, California. So with that, let's go ahead and dive right in.
[00:03:13.358] Amy LaMeyer: Hi, everybody. So nice to see you out there. Looking forward to introducing the panelists today. First we have Tony Parisi, 25 plus year veteran in this space. Now the chief product officer at Lamina One. Alvin Graylin, GM and president of China for HTC. And last but not least, Kent Bye, who's recorded over 2,000 episodes of Voices in VR. And also, fun fact for those that may not know, but he did a couple of years of Voices of AI as well in 2016 to 2018. So today we are indeed talking about AI and the metaverse and what's next. So if you're looking for what's already happened, you can go somewhere else because we're going to talk about the future. But there are two hot topics that are being ever clumped together more and more these days. And I think we'll start off by, you know, every one of these people on stage could do 45 minutes themselves on a keynote on these topics. So you're really lucky to be here, to hear the nuggets of wisdom from all three of them, to hear the discussion that we're gonna have. And there's lots of public content as well that if you wanna dive in deeper. And so as you would hear in a Kent by podcast, let's dive right in. First of all, let's define the metaverse. That was the big topic last year, so we should do that just to make sure we're all aligned on what the definition is. I was going to go with Matthew Ball's definition, and you've written principles on the metaverse, but earlier today, Roni said the spatial internet, and I thought, oh, that's a nice, easy way to define it. So I'm going to go with spatial internet, unless any of you want to add to that.
[00:05:05.822] Tony Parisi: It's pithy. No, that sounds good. It hits the spot, right? Yeah, great. It's simple.
[00:05:10.344] Amy LaMeyer: So we're gonna move on to AI and how to define AI and maybe a little bit of a history of AI as well, since that's a newer topic, at least for this particular conference. We've seen a lot of change in generative AI, particularly in the last six months. AI is a broader technical set of anything from computer vision, natural language processing, machine learning, You know, Ori says XR is the interface for AI, and Rony said AI is what XR has been waiting for. Do you agree with these statements, and what would you add to the definition of AI?
[00:05:45.182] Tony Parisi: I mean, these are massive topics, these are massive problem domains. I could start with a pet peeve of mine, which is that I think a lot of people think that AI is the solution to all of our content creation and authoring problems in XR, and I would say that's a somewhat lazy way to think about it. Clearly, we're seeing a lot of generative AI assisting in authoring, but it's not the whole equation. You look at game engines, that's how most people have created XR for the last little while, but there's a whole slew of authoring techniques, capture, tools, easy drag and drop, that folks have already forgotten about because AI has hit the scene now and they're like, oh, we're just gonna put some text into some magical thing and worlds are gonna come out of it. Well, that's a piece of the whole equation. So I think we start there when we think about content creation and AI is a good marriage. I don't know what our other panelists think about that, but you can't oversimplify it to say you're just gonna you know, speak some words and worlds are going to happen, right? Yeah, not yet.
[00:06:41.024] Alvin Graylin: Maybe in a few years. Yeah, I think XR actually will benefit greatly from AI. You know, there's this myth that AI is here, now XR is dead. That's completely the opposite. There will definitely be a lot of improvements and efficiencies from AI making content creation better, which I actually think right now is the biggest roadblock to XR adoption is the lack of quality content. I mean, you saw in all the speeches this morning, there's so much hardware coming out, things are getting thinner and lighter and cheaper and higher resolution. But we still, on the standalone platforms, there are hundreds of pieces of content. On the PC VR, we're talking about thousands of pieces of content, compared to your phone has millions of content. So, we need to be able to get to millions of pieces of XR content for that space to take off. So, I think XR will definitely bring the costs trending towards zero. It won't be automatic, but it's definitely trending that way. So, I think it will definitely be.
[00:07:36.913] Tony Parisi: Well, in fact, it powers, you know, capture technology and other things like photogrammetry anyway, right? I mean, so those techniques are actually AI, you know, machine learning based anyway, right?
[00:07:45.041] Alvin Graylin: Absolutely. In fact, I mean, what made the internet and apps work is because everybody with a phone or social networks work is everybody's phone becomes a creator. Pretty soon, everybody with a phone and with an XR device will be a world creator, which I think will make the metaverse a lot more exciting.
[00:08:01.819] Kent Bye: Yeah, and for me, I see them more as like sibling technologies where they're co-evolving together. And if you look at the evolution in history of even like virtual reality, where originally you have the PC and it's tethered, but then you have the computer vision innovations and machine learning to do simultaneous location and mapping to be able to actually have the standalone VR headsets. And then on the same time you have at SIGGRAPH 2017, the video was showing that they had these virtual environments where they were teaching a robot to be able to have these interactions and then they had that same neural network that was deployed out to that actual robot. And if you look back in the history of how a lot of these deep learning algorithms were even developed, you go back into like these Atari simulations where you have a very empirical number and benchmarks that a lot of the AI were trained on these virtual worlds. So you have this back and forth between how VR is helping AI develop, and AI is helping VR develop. And you have it both at the technological scale of just the hardware itself, but you have up until the stack into content creation. And as I go out and talk to a lot of artists and creators, right now AI is sort of still in the pre-production mode. Like there's a lot of stuff that hasn't been shipped out to production. Although I think that's actually gonna change at Tribeca this year, where there's gonna be an entire film that's made with AI. But right now I see a lot of stuff that's not quite production ready, I love all virtual, there's actually using Whisper AI from OpenAI to be able to do conversational interfaces, so you can actually speak out into a virtual world, talk to a chatbot, and then have that send back to OpenAI and send back. And it feels like you're chatting with ChatGBT, but you're doing a conversational interface, but you're immersed into these virtual worlds. So I expect to see a lot more of that type of user interface and conversational interfaces with the open source version of Whisper that's able to allow you to just open up these new vistas of interfaces in UX. So I think it's going to be both the hardware and also the experiential dimension as well.
[00:09:54.926] Alvin Graylin: because we talk a lot recently about generative AI and content creation, but I think it's important to understand that AI has actually been part of XR usage for a very long time, right? Including, you're talking about the SLAM and hand tracking, voice interactions, full body tracking, scanning rooms, all of that is actually AI. We didn't really focus on it as much before, and pretty soon we're gonna have AI going into these systems being used to power the NPCs. And so you go into a social world and there could be thousands of people and maybe 10% of them are real people, but 90% are probably in AI. So those are the kind of things that are going to change. So it's not just about creating the world, it's about all the interactions that it's going to enable or have enabled so far.
[00:10:39.637] Amy LaMeyer: Yeah, it seems like things have been changing daily. We've been on a chat thread that's keeping ourselves up to date. But in particular, there was the news yesterday, and I wanna dive into this a little bit. It was the statement on AI risk, and this was from the Center of AI Safety. And I'll read it for you, for those of you that may have missed it. Mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemic and nuclear war. So, given that statement has been signed by Sam Altman and a lot of leaders in the technical space and government leaders in general, where do you stand on that statement? Tony.
[00:11:18.458] Tony Parisi: Me first, really? Recently, Jaron Lanier wrote a piece for The New Yorker, which I loved because his main point in that, it's a great piece, he's actually pretty excited about the tech, I think, in general, is that we're mystifying AI. We're sort of unconsciously making it some other thing that gives it magical powers when really it's technology that we're creating. And if we just acknowledge that and start demystifying it, we won't think of it as this other thing that's coming for us. So, I absolutely agree with that. That it's actually human-driven. It's human-driven. We're making this, at least for now. I mean, maybe in a few years that might not be the case anymore. So, I mean, I think that's kind of the good news and bad news because that means that really what we need to look at is people behind this and humans behind this. So I'm a little bit in the camp of that piece that came out only because really what we have is potentially unbridled avarice. No offense, because you're an investor, but unbridled avarice meets a low emotional intelligence tech founders. No offense, all of you tech founders. I'm one of you. and what could possibly go wrong, and we've seen the movie with social media, and you can make arguments that there were some actual existential threats with social media in the last few years that we're just coming to grips with, and we're like, oh yeah, we'll break it, then we'll fix it. Are we going to unleash some genie out of the bottle, you know, unintended consequences? I'm not worried about chat GPT mid-journey, I use them every day. What comes after that? What startup's getting funded right now that's going to be Skynet and destroy us all? So that's my paranoia. I'm also an old middle-aged man, so I may just not be able to keep up with this stuff, but that's how I feel about it.
[00:12:57.533] Alvin Graylin: Okay. Somebody talk me dead. I kind of heard both sides from what you were saying. One side you were saying you're not worried about it, but the other side is that it's going to be existential risk. So which one is it?
[00:13:09.582] Tony Parisi: Oh, I'm worried because it's people who are doing this. Ah, okay.
[00:13:13.237] Alvin Graylin: So maybe I'm on a slightly different camp than you actually, because I actually have been doing XR and AI research for 37 years, and we are actually using more black box models now versus the more prescriptive or procedural models in the past. So we are not actually defining it, it is self-learning, and unfortunately self-learning on stuff that humans made, and we've not been a really good example. for how we treat each other and how we treat animals in the world and so forth. So if it learns from us and is trying to be more like us, I would be very worried. Now, I would also say that there's two kinds of risks that is going to be associated with AI right now. One is more the misuse of AI that is, not that it becomes sentient and wants to kill everybody, but Even existing AI can be misused by people and create real risk. The other risk is that AI becomes super intelligent, much more than us, and then finds that we're either bad for society or it just wants to preserve itself, and then kills us. So I think the second risk is actually not as high, because so far, there hasn't been any significant data that shows that when it becomes smarter, that it's going to become evil, even though all the dystopian novels say so. But the fact that if we get something that will be a thousand times smarter than us, and there's people behind it that will misuse it, that actually can be very dangerous, whether from manipulation or using it to set off some chain of bad events. The post that came out yesterday, I think, has validity. And it's not just founders and investors that are in there. There's people like Bengio and Hinton, who are the fathers of this space, who are actually, I think, pretty sanguine about the risk. For them to have put their name on it, it does say a lot about the potential real risk. So we do need to be conscious of it, but I don't think we need to panic.
[00:15:05.641] Amy LaMeyer: not nuclear level.
[00:15:22.084] Kent Bye: We need to have things like the AI Act that's happening in the European Union to address some of the different harms and the tiers of harms. And so there's real mechanisms that are happening that are very important to be able to ensure that there's not undue harm that's being caused by some of these systems. The problem that I see is that the type of rhetoric that's being used by casting AIs as superintelligence turns AI into this god, and also it overstates some of the different capabilities of what AI is able to actually do, and it ends up being this mechanism of using fear to promote AI in this kind of weird way. Brian Merchant and Justin Hendricks did a whole podcast on that. There's been a lot of AI ethicists who have been on the wings critiquing a lot of these things for many, many years. Timothee Cabrew, and you have companies like Google that are just firing all these ethicists. You have Meta who's eliminating all their responsible innovation team. They just fired a lot of their policy folks. And so you have the people that are supposed to be the checks and balances between the engineers who are trying to create and make something and then the ethicists who are trying to say, are we doing this in an ethical fashion, are we deploying this in a way that is responsible? And, you know, there's this kind of, like, once a product has been set, there's an intent to have it out in the world. There's no red button for any of the ethicists to be able to push. A lot of that responsibility has been put onto the engineers. So the people who are making it are now, at the same time, have two masters of whether or not they're going to meet their deadline to ship it and whether or not it's going to be ethical. And you can't have one person do that. It needs a dialectical process with all these people that are coming in to do that. And all those people just got fired. So there's a way in which that if you search on Twitter for like hashtag AI hype, Emily Bender, you have folks like Ryan Calo, you have Yann LeCun, who's in charge of AI research at Meta. All this stuff is sort of overstating that we're so far away from the cognitive architectures that are going to actually have like superhuman level intelligence of AI. A lot of it is this autoregressive larger language models, which is really what has been called by a lot of these and it's a lot dumber than it is for people who aren't able to interface with it, or, you know, the documentary coded bias goes into, like, facial recognition as an example, where people who are black don't always get their face properly recognized. And when it comes to cops using these artificial intelligence systems to automatically identify people, and when it goes wrong and they go to jail, then that's the type of level that the European Union is saying we need to just outright ban these levels. So I think the EU and their legislation with these tiered models is a great approach. And I feel like the EU is like five to 10 years ahead of this conversation and they've been looking at for a long time. And I feel like we're here in the United States reinventing the wheel, thinking that we're going to have these AI ethics statements going to be a solution to any of this.
[00:18:16.912] Tony Parisi: Okay, first off, yeah, if we're going to rely on U.S. regulators, good luck. But what's really disturbing me is hearing that those jobs, those people are being let go at those larger companies doing this, because remember that dinner we had at South by Southwest? We were talking about this a lot at dinner. We need more humanities, we need more ethicists, we need more guided development from inside the industry. We can't expect government to regulate this. They can barely keep up with the technology that's going on, right? So it's very disturbing to hear that this is becoming so little.
[00:18:47.365] Alvin Graylin: there is an intent to do regulatory capture by the big companies, because they keep saying, oh, this is really dangerous and we're going to help you manage it. But if they're the ones helping set the rules that will manage themselves, and it's only the biggest guys that are on that table, it's a way to reduce competition from any smaller labs. But the second thing is when you're setting your own rules to manage yourself, that's actually a formula for something bad to happen.
[00:19:12.289] Amy LaMeyer: What are we seeing in China in terms of regulation?
[00:19:14.710] Alvin Graylin: Yeah, so about a month ago, China just came out with a new set of regulation that I thought was actually pretty comprehensive. Everything from defining that everything that's released by generative AI has to be watermarked, so it has to be noted, there must be a clear I guess, origin of information, of where everything came from. The people who release the models are responsible for ensuring the quality of it, and if anything's wrong, they have to fix it within a certain amount of time. And even the users have to have some level of responsibility if they misuse it, that there will be consequences for misuse. So I felt like they've actually done a pretty good job that the U.S. could follow as a way, you know, because I know usually people, you know, say, oh, whatever China is doing is bad and wrong and oppressive. But in this case, I actually think that the way that they've approached it is fairly methodical.
[00:20:03.377] Amy LaMeyer: And so we were talking about this a little bit earlier, too. Who do we think bears the responsibility if something goes wrong in this space? We're saying a lot could go wrong. It's human-driven, ultimately, even though it's machines. But where does the responsibility lie in terms of something goes wrong?
[00:20:20.947] Alvin Graylin: I mean, according to at least these kind of proposals right now, and even in the U.S. proposals that are recently coming out, the makers of the models should take some level of responsibility. And I think they have to. Otherwise, what's happening today with this race condition where Google releases something, then, you know, OpenAI has to release something faster, and then they have to integrate more, and everybody is just racing to get to that finish line, which we don't know if it's a good finish line.
[00:20:45.693] Amy LaMeyer: Yeah, I'm wondering as we see this technology change and as we see regulation or guidelines be developed, will that really protect consumers? Because ultimately, those are the people that we need to protect. Are you seeing that? I had a couple of interviews.
[00:21:04.584] Kent Bye: So there was a Stanford Cyber Policy Center. They had a whole existing law and extended reality symposium that was at the beginning of the year. And I was seeing who had a beat on not only things like biometric data, because that's a whole other aspect of this is We need to advocate for new human rights for cognitive liberty that protects our aspects of self-determination, our freedom of thought, and our mental privacy. So each of these different fundamental human rights has to be defined at an international law level. And then from there, that needs to ripple out into changing definitions of biometric data and GDPR. That's one example of how, once things get set at an international legal level, then they filter out into all these existing regulations. In some extent, there's the privacy aspect of it, and then there's the other aspect, which is deploying these AI systems and large language models out at scale, and what type of harm comes from doing that. And so, that's where I think the AI Act is really trying to take a tiered approach, where you have the stuff that is so dangerous that you should just outright ban it, and they have a number of those different types of applications. And then you have the high risk, which is things that they are then obligated to report to the European Union on what are the different ways that you're making sure that this is safe. And then there's the other at the low risk, which is more of like, you need to disclose to the user that you're using AI. So an example of like, if you're talking to a chatbot, you're answering the phone, there's gonna be a requirement that you're actually either talking to a human or not. And if you're not talking to a human, it needs to disclose that you're actually interacting with an AI, interactive agent at that point. And so, The European Union, like I said, is at the forefront of that, and they're probably five to ten years ahead of what the United States is in thinking about this. And so, I would look to see what they're doing, because that is going to, just like GDPR, help to shape the architectures of privacy in these big companies that they had to, if they wanted to do business in Europe, they're going to have to change and build their systems to oblige by these laws. So anybody that is trying to create these metaverse applications that are international and global, then you're gonna have to address the hardest legislation first, and then hopefully we'll have some of those protections that are happening in European Union in five to 10, 20 years or maybe at some point, come to the United States.
[00:23:24.141] Amy LaMeyer: At least there's a guiding light. I'd like to talk about creators a little bit, shift the topic, and before going there, maybe giving a quick shout out to the XR Guild, speaking of ethics and privacy and biotech. Yay! Nice. And yeah, for those of you that don't know about the XR Guild, you can go to xrguild.org, and also there's a group from there that's meeting tomorrow at 1pm, but It's a way to get involved whether you're a creator or a builder or an advocate. There's a set of principles that are making sure that we're building a solid metaverse and set of technologies that's safe for everybody. But on the creator side, I want to jump in there a little bit because we're talking about AI and we're talking about the metaverse and what's next. And on the way in today, I was listening to Kent, your most recent Voices of VR, which everyone should listen to. It's my favorite episode ever and I've listened to a lot. It's on the Fatboy Slim Engage VR experience that came out a couple months ago. But I was thinking about it because as part of that interview, you asked how long it took and they said it took a year or like a year and a half to create this 45 minute piece, which is amazing. And again, one of the most fun music experiences I've done in VR, but that's a long time. So, my question to all of you is, will AI help us create in the metaverse faster, better? Like, what is that intersection of AI and the metaverse?
[00:24:54.255] Alvin Graylin: Yeah, I mean, I think we kind of briefly talked about it earlier, but essentially, you can now speak environments into place, you can describe, you have a text, create a picture, then that picture becomes a 3D model, that 3D model, then you place into the world. So the workflow is definitely going to get faster. And the quality of content that's going to be created by less experienced developers will get better. In fact, I was talking to some developers recently who've been able to get 2 to 3x increase in efficiency. Reduce their artists, which may not be a good thing, but they've reduced their artist count and still able to maintain quality of production. So that's definitely going to happen. And it's going to be even more automated going forward, because every day, like you're saying, there's a new release of something that now automates that process even more, I think. NVIDIA just released in their presentation, What's Possible? And it's going to get to that place where what was described by Ori this morning, where you stick a book in and you get something out. I mean, we're a few years from that, but I think we will get there.
[00:25:56.280] Amy LaMeyer: So you're saying it's going to get faster, it's going to get better, and we're going to replace, we won't need as many builders.
[00:26:02.002] Alvin Graylin: I think we'll still need senior creators, senior designers, but you're going to need less and less of the junior people. 100%, yeah.
[00:26:10.813] Kent Bye: Well, I just watched the finale of Succession, and I just want to give a shout-out to writers and storytellers, because I don't think AI is going to ever... I think the differentiating factor as we move forward is going to be storytelling. So, that's a big reason why I cover things like the Sundance New Frontier, and South by Southwest, and Tribeca, and Venice Immersive, and IFA DocLab, because that's where I see the frontiers of artists. And so, I always, like Shari Frillo says, she always listens to the artists and what the artists are doing. So what are the artists doing? Well, there was Whipped Cream, they're using AI to do concept art, and then do the concept art, and then they actually have the artists go out and actually make it. Or there's a couple pieces at Laval Virtual that were using ChatTBT to do chatbot, and so in one of the experiences, it was said that when you go into a virtual world, you have this place where if there's nobody there, you need to have something to start it. So if there's someone there chatting with a chatbot, maybe that'll then help gather people so that once you get enough critical mass of people, then you stop talking to AI and you talk to each other. But you have this problem where you have these empty rooms, and that may be a way of using AI to sort of kickstart some of these different social dynamics. And so for me, when you said, Alvin, that we are gonna have a world where we go in with 90% chatbots, I thought, oh my God, that sounds like hell because unless it's a really well-designed characters that you feel like you're actually pushing forward a story and there's a kind of a Westworld-esque, you feel like you're in a coherent narrative. But as of right now, the AIs aren't able to produce that type of coherent narrative. So I'm seeing more of immersive theater actors that have actual live actors doing stuff that AI can never do. And so there's this element of the liveness of alive, of things that are always going to preserve for that reactivity and being able to adjust to those dynamic interactions that are going to be difficult for AI to ever replace. So I feel like there's going to be the storytellers that I see that are using the technology, but also finding the limitations of what that technology can't do yet.
[00:28:03.060] Amy LaMeyer: So a skilled creator using the technology to do things at a different level.
[00:28:07.843] Alvin Graylin: We don't want to limit ourselves to what it is today or what it is in the next six months. But if you look at the trajectory of change and the level of realism you can get to, and honestly, for most of the people you're talking to are the senior designers and creators, right? If you went and talked to the people who are doing graphics design for some art shop, they're going to have a tough time keeping their job. But for the really creative people, which are the most ones that you're going to interview, they're going to be hard to be replaced by AI. So we don't want to, I guess, extrapolate that for all creators, they're going to be safe.
[00:28:40.599] Tony Parisi: Yeah. And can I jump in with the writer analogy here? Because, yeah, I mean, that's a very timely conversation with what's going on with the writer's strike right now and this conversation of the writer's room shrinking. I think, by analogy, what Alvin was saying about visual artists, I think we're going to see the same thing with screenwriters, too. I think the more senior ones will be just fine, if you will. The writer's rooms will get smaller, though, because of these same factors that you can now use AI as a creative tool. to help with a lot of this sort of drudgery, or just sort of like, let's do the ideation, let's do the concept phase, with something like a chat GPT. Can I tell my story? Yes. It's a good time for this. So, I also make music, and I'm doing music NFTs, and I'm hanging out with all these wonderful Web3 musicians, and one of these artists, a woman named Emma Miller, who's a Scots singer-songwriter, was doing a Twitter space. The actual metaverse where we all hang out is just like Twitter spaces and discords all day. And she was talking about the SCOBY she was growing. Did people in the audience know what a SCOBY is? Who knows? OK, like five minutes. It's a symbiotic culture of bacteria and yeast that you put into sweet tea, and out comes kombucha. Eventually it helps it ferment. And most of us in this Twitter space are like, Emma, what the hell is a SCOBY? And she described it, and I looked at the picture on Wikipedia, and I was horrified. And I said, I'll be right back. And this is while Twitter space was going on. And I went to chat GPT, and I asked it to write me a 15-minute scene in final draft format about a SCOBY that grows out of control and threatens a neighborhood. And it spat back this scene to me that was absolutely hilarious. Bizarrely, the protagonist was named Emma. I did not put that in the prompt. I don't know what's going on there. And I shared this with this Twitter space that turned into a week-long series of Twitter spaces where a bunch of us got together and did table reads of an entire horror film franchise of Scobie movies that went five main episodes, a fan fiction handwritten one, a spin-out episode from one of the characters, and then a franchise reboot with the SCOBY, mated with Jellyfish, and the season took over the world. All because we were farting around in chat in VPT. But it led to this super creative, interactive theater, wonderful art experience, just as a lark. So I don't completely hate it, I'm not completely horrified by it. There's a wonderful creative tool there, but, you know.
[00:30:59.230] Amy LaMeyer: And I don't know that it's junior or senior, like I want to push back a little bit to say maybe it's the creative people that learned how to use the tool that are going to be the ones that ultimately... There was also, I did an interview with Violetta who did a piece called Prison Acts, but
[00:31:15.187] Kent Bye: She's actually been liberated in terms of, she's not a coder, and she's self-described as being on the spectrum, and so she has difficult communicating, and so she's been using ChatGBT to actually, like, and she speaks multiple languages, which for her says is actually a benefit when using these conversational interfaces, and so it's actually opened up these whole new vistas for her, to push the next edge of what she's able to do in these augmented reality scenes and the art that she's creating that is augmenting her, and she's not a senior level, but she's a junior level that's able to actually, previously she had to work with a whole team, but now she's able to be more self-sustaining in a way. So I feel like there's people who may not have access to some of the ways that they need to really focus their attention or interface with the technology, and this is providing a new conversational interface with the technology that is even producing these code snippets that are kind of opening up these new ways. And obviously, a lot of the code that you produce doesn't always work, and you still have to sometimes use chatGBT to fix itself. Or if you can't, then with like WebGPU, I did an interview with Brandon Jones, and this is a whole new shader language called Wixel. It's the WebGPU shader language. There's no existing examples of this, which means that we need a lot of human labor to be able to produce that. So things like these large language models don't start to work until you have a corpus of human labor that is being built upon. So you always will need these humans and if you get rid of all the humans and just have it kind of feed upon itself, it's just going to have this feedback cycle where it's garbage in, garbage out type of situation at some point. So, I get a little afraid of as we start into this new era where we've been scraping the internet to train these chat GPT, now there's going to be all this chat GPT nonsense and garbage and hallucinations that are out there and it's going to be, you know, and that's why Yann LeCun says large language models are doomed in that Mathematically, he doesn't see that they have a lot of utility, but in terms of their hallucinations, he just doesn't see that it's gonna be a viable solution, that you need to have other architectures to actually have these human levels of interaction. And so it's not that they obviously are surprisingly effective at what they do, but yet there's a lot of limitations that, from his perspective, are not gonna be overcome. So I think we have to understand what it can and cannot do, and as we navigate this, then we all have this new level of media literacy when it comes to interface.
[00:33:26.017] Alvin Graylin: I have to say people are overstating the frequency of hallucinations and also the lack of control over it, because they actually have done fine-tuning systems where if you're, let's say, in the medical field, you can actually train your data on medical data and the level of hallucination goes way down. You can then also add in additional procedural-based checking to validate the data. So the level of hallucination isn't generic for everything, everywhere. It can be tuned. And by combining it with cognitive systems or secondary systems, you can get pretty good content. Now, the other thing is hallucination may only be bad in certain use cases, but not necessarily in creative use cases. So if you're writing a story and whether you're having a Scooby or having a slime mold, it doesn't really matter. They can get creative in things that we didn't even think about that could be super cool. So hallucination in that context could actually add creativity. I don't know if I would say that it's a stochastic parrot type of a model today because they've actually recently just came out about two weeks ago with some new tests on theory of mind and you know chat gbt or gbt3 was about a level grade three level gbt 3.5 was around a sixth grade level in terms of theory of mind and now with gta4 we're actually getting they've gotten over 90 percent or 100
[00:34:47.221] Kent Bye: The problem, though, is that they don't disclose what their architecture is, they don't know what the training data is, and so there's no way to, and also, tests are not a great way of assessing what intelligence is. So, like, all these things are not, there's a lot of problems with these assertions.
[00:34:58.587] Alvin Graylin: I'm not saying that tests are perfect, but what it is showing is that the level of theory of mind capabilities of these models are getting better and better in a very quick way. So I think to say that they will never be able to reach these certain levels is probably a little bit too conservative.
[00:35:15.974] Amy LaMeyer: Do you think that we'll go to smaller data sets that are more reliable, perhaps? Like, let's say we took all voices of VR data and it was just, you know, AI-based.
[00:35:27.098] Alvin Graylin: They've actually done that. They've actually trained it on specific smaller sets. And essentially, all the hallucination goes away.
[00:35:33.840] Kent Bye: Well, when I asked Chad DBT who I was, it gave me a lot of reasonably plausible information that was half of it was completely made up because it was based on the entire Internet.
[00:35:42.251] Alvin Graylin: Right. So it did a training based on your interviews. I bet it would be a lot more accurate.
[00:35:47.405] Kent Bye: Well, I did actually, over this past couple of weeks, I used WhisperX, which is an offshoot of Whisper, and I went back and got transcripts for all 1200 plus Voices of VR podcast episodes, and even with diarization, and so I've been starting to play with trying to actually get the transcripts, and once you have the transcripts, then it opens up new possibilities, but I do think there is this other aspect of ownership and consent and who owns that data and you know, it's sort of like as we go into these new realms, to what degree have people consented to using that data and a lot of data that's been stolen. So there's a lot of like deeper ethical issues for how these data sets are even being trained. And so I'm cautious with even the archive that I've helped to curate, you know, what happens to that and what people start to do that. There's a lot of exciting possibilities, but at the same time, it's like, what is this model of shared consent as we move forward for whether people can opt in or opt out to being a parts of these data sets, which I think.
[00:36:38.055] Alvin Graylin: I think we need to separate the data provenance and ethics versus the capabilities. Why?
[00:36:43.762] Kent Bye: No, it's the same. It's a relational aspect.
[00:36:48.768] Alvin Graylin: I'm saying that the capabilities of these products, of these systems will get increasingly better. But it's up to us how we want to manage the provenance or how we want to manage the rights of the data where it came from. But I'm saying you can't say because you're not sure whether this data came from the right place that this system will never get smart. I mean those are completely two separate issues.
[00:37:11.665] Kent Bye: It's a utilitarian argument that you're making, that essentially the benefits for doing that, it doesn't matter what kind of harms you bring, because you don't have to be in relationship to those people, and I think that's just a wrong assertion.
[00:37:22.270] Alvin Graylin: No, no, no, that's not what I'm saying. What I'm saying is that those are two different questions, and we don't... Just because the rights isn't properly managed today doesn't mean they won't be properly managed in the future. And just because the systems are not fully smart today doesn't mean they won't be fully smart in the future. So those are two separate questions and with two separate ways of solving.
[00:37:47.038] Kent Bye: One last thing I want to say is that I feel like these technologies, both virtual and augmented reality, are catalyzing us to have these fundamental paradigm shifts, both philosophically, ethically, and even from a level of substance metaphysics, where you have this object-oriented way of having this separation. And I think as we move into more of a relational ontology, we have to think about how these things are related to each other. We can't separate them, so that's why I'm disagreeing. is that you have a different sort of metaphysical assumption there that is allowing you to make this, you can have these harms, and because you have this benefit that you found, it's okay to have those harms, but if you don't think about those relationalities from the very beginning, then how are you gonna do that later? And that's my point, is that we have to, as we move forward, think about how we become into a right relationship to all these technologies as we move forward, and as these things that you can just sort of like say, oh, we'll deal with that later, is just a type of way that you're, I don't know, it feels like you're out of relationship at that point.
[00:38:39.298] Amy LaMeyer: Well, if we think about the Internet, right, there was a lot of hope in the beginning that the Internet would connect people and it would get information to people that didn't have it. And all of those things were true. But there were a lot of things about the Internet that we didn't realize back in late 90s, 2000, that did come true. So my hope is that as we're continuing to build this new computing paradigm, these new emerging technologies as they're integrating with each other, that we can all go out, depending on whether we're consumers, users, builders, CEOs, that we can think about how are we using this technology and hope that we are using it from a good place. We don't have a lot of time left, but everybody has interesting things to say, and you guys have all published really interesting pieces. Can you give a shout-out to just a few things for those that are in the Kent camp versus the Alvin camp in particular? I've had a chance to listen to them, and can you just say just a few pieces that you...
[00:39:33.824] Kent Bye: So I recently did a whole talk at San Jose State University, where it was like a preliminary thoughts on AI. And so it's a talk on YouTube that I did. It's on my YouTube channel. And we actually, Alvin and I, had a conversation that I'd point folks to. And hopefully, I'll relaunch some of the voices of AI at some point, since it's coming back.
[00:39:49.810] Amy LaMeyer: Yeah, and you were at Harvard recently.
[00:39:51.230] Alvin Graylin: Yeah, so I spoke at the Harvard XR Conference about a month ago, and then also at the Athens Hellenic World XR Conference as well about two weeks ago. And Lewis Rosenberg and I are working on a book about AI and XR, so wait for that in about six months.
[00:40:05.020] Amy LaMeyer: Yeah, and Tony, you're doing some really exciting things with LaminaOne and other spaces, so...
[00:40:09.898] Tony Parisi: Yeah, so, you know, Lamina was founded by a novelist, and a lot of people on the executive team are creators in one form or another, like myself as a musician. And Neil's co-founder, Peter Vesson, is very, very keen on making sure that blockchain technology like LaminaOne has is being used to help creators with provenance, to help people get paid out on a large scale fairly. technologies like that, so I'm very sympathetic with that, that's why I work there. My personal writing these days has been a lot more creator-centric, about musicians in particular, because that's my passion. The real goal here for technologies like this, including AI, I think should be to enable that creativity, but make sure the people who contributed to it, which you started alluding to, can actually then get paid, or can opt out of this machine learning system if they don't want to be part of that machine. Nani Dela Pena had suggested, you know, maybe there needs to be a creative common style licensing and attribution where people get to opt in, they get to say what level they want to participate in this machine. And so, you know, that's where a lot of my writing is going these days, and my thought pieces are around how people can use these tools to create, but also participate in the future economy fairly. And again, our company is very much looking at solutions like that.
[00:41:18.452] Amy LaMeyer: Yeah, well, I want to thank every one of you for being on stage. There's a lot to discuss. I would love to continue to have this conversation over the years. It's going to continue to shift over the coming years. And thank you all for attending and listening to what's next in AI and the metaverse.
[00:41:37.156] Kent Bye: So that was a panel discussion called The Intersection of AI and the Metaverse. What's next? That happened at Augmented World Expo at the opening day on the main stage. And it was moderated by Amy LeMayer, the managing partner at WXR Fund, Tony Parisi, the chief product officer at Lambda One, Alvin Rank Graylin, the global vice president of corporate development at HTC, as well as the president of China for HTC, and myself, Kent Bye, of the Voices of VR podcast. So I have a number of different takeaways about this conversation is that, first of all, So in the moment of this conversation, there was certain aspects of the contradictory nature that I think that Alvin and I had that is a bit of a carryover from our previous conversation. If you go back to episode number 1185, I had a deep dive debate about the future of AI with Alvin. So at the beginning of South by Southwest that we recorded back on March 10th, 2023, very first conversation as I had landing into Austin, we went to have just a very brief conversation. And then we kind of got into the topic of AI and went on and on and on. But I think that Alvin's really pushing forward that the artists and the creators and the makers and the investors are trying to push forward what's possible with these technologies. And there's this creative impulse that As I do this series, I have to admit that there's a lot of different artisan makers and creators who are on that side of just trying to push forward what's even possible with the technology. And at Tribeca, there was a piece called In Search of Time that was using different generative AI systems. And one of the co-directors, Matt Tierney, said to me in that interview that he feels like that right now the community's at the stage that feels very much like the inception of hip-hop, where people would take their parents' old jazz records or 70s soul, sample it, resample it, and then construct something new. And that it took a while for the industry to catch up and go, wait, wait, wait, we need to provide royalties for all these samples. So it's sort of this creative impulse where you're pushing forward what's possible and then trying to do it in more of an ethical and right relationship way later after you see what the potentials are. And I think that in some ways is a part of the natural process. And I guess one of the things that I was pushing back on from Alvin was that he was really advocating that we need to completely decouple that innovation process from the ethical process. And I feel like it's already sort of decoupled. There isn't really a lot of regulation that's happening in the United States. It's already just kind of totally unregulated and people are just doing whatever they want. If anything, there's things that are happening within the context of the European Union with the AI Act that is trying to at least set some bounds of a tiered system of like, here are the different applications of AI that should be just outright banned. Here are the high risk ones that have some different reporting obligations. And then here are the medium or normal tier risk that You have to disclose it to the users. And so there's already ways in the context of the European Union that anybody that's doing a global company is going to have to abide by some of these regulations in the EU that have to have different reporting obligations and reporting different aspects of harm. And so that's already starting to happen within the European Union. Very similar to GDPR is driving the bleeding edge of privacy engineering and privacy architectures So folks are gonna have to think about some of these different things as they move forward And so I think one of Alvin's points is that you know That's a separate process that maybe requires something like regulation or it requires really robust responsible innovation teams who are trying to red team these and It's difficult to look at an AI system and then to understand the full capabilities and gaps. I mean, in some ways, it needs to have certain levels of transparency of what the data is being trained on and then given to different researchers. And it takes a lot of eyes and a lot of folks who are poking at these things to understand some of the different potential harms that are done. Just hearing different anecdotes of people being able to create poisonous mustard gas from some of these different large language models that are scraping the internet and have all sorts of information that could be used by terrorists to create biological weapons that if someone who is at a company, they may not even be thinking about that that's even a possibility. But if they're shipping out these different projects without really understanding the full harm, then that's the types of stuff that can happen. You know, Alvin was mentioning a lot of stuff around large language model has been demonstrating like human level capabilities and, you know, passing such and such standardized test. And so, you know, some of the pushback that I hear from AI ethicist is that some of these standardized tests, first of all, are not really a great indicator of intelligence. And secondly, we don't know what a lot of the different training data is from a lot of these different models. And so if some of the different training data is coming from some of these different tests, just because it's passing that test may be just that it's recalling some of that information that was used in the training data set. But in the absence of having full access to some of the different training information, then we have no idea of how to actually evaluate what these large language models and these systems are capable or not capable of doing. I think one of the points that Alvin was trying to make is that rather than trying to scrape the entirety of all the Internet, if you start to have more limited data sets, then you can have a little bit more reliability when it comes to hallucinations. But I guess part of my pushback was hearing folks like Yann LeCun who was given a speech at the philosophy of deep learning Giving more of a mathematical argument that he just sees that given the number of tokens and the possibilities for getting things wrong that a large language model architecture within its own context without any other cognitive architectures that are looking at a real deep understanding that is being exhibited rather than just kind of like this stochastic parrot autoregressive model of just predicting the next word. So there's all these different types of limitations that Yann LeCun speaks about, and has an opinion that these large language models are doomed when it comes to the long term of this path towards artificial general intelligence. That's not to say that they were going to come up with other architectures. But part of my pushback in the moment was just understanding and hearing some of those different limitations from Yann LeCun, thinking about different aspects of hallucinations, And yeah, there's large disagreements and debates around all of these things. I mean, it isn't like there's a way to mathematically informally prove any of these things. It's basically like trying to model some of these different behaviors and make these different predictions based upon those models. And there's always going to be limitations of these models. They're never going to be perfect. And so I think that's part of the challenge of trying to understand not only the philosophy of these deep learning entities, but also the possibilities and also the constraints and limitations. And so, yeah, I guess I try to listen to a lot of these different AI ethicists, Timnit Guru, Margaret Mitchell, Emily Bender, Ryan Calo, all these different people are looking at a little bit more critical and skeptical take on some of these different Claims of hashtag AI hype that I think when I have had a couple of conversations with Alvin now I found myself pushing back and being contradictory on that and I guess as I go through some of these different conversations Sometimes when people are too pessimistic on AI sometimes I push back on that so I guess maybe there's just a dialectical and contradictory nature when it comes to some of these different things and Because certainly throughout the course of this series, you'll be hearing from a lot of other artists who are much more on the side of Alvin when it comes to just pushing forward the possibilities and not having complete answers as to how to address some of the broader ethical implications of these things. And it may be that we need to have some of these different government regulations step in to be able to handle some of these things, because they really are beyond the scope of any one developer or artist or creative, and that we need to have more of a comprehensive approach of reining in some of these different systems. I guess some of my pessimism around that as a strategy is that like Alan was saying that a lot of these companies are trying to go towards regulatory capture of trying to say, Hey, we need to regulate ourselves, or we need to set the laws of what the regulation should be such that that's only these large, huge companies that can even put out some of these different AI systems. And talking to different artists and creators, there's a lot of excitement of being able to open source a lot of these different tools so that people can modulate them and use them in a lot of different contexts and not always have a paywall behind everything. I mean, open AI, open source, the Whisper, and I've been using the WhisperX to be able to do transcriptions for all of my different podcasts. And so there's a certain amount of utility that I get from that. What are the different ethical implications of that? I don't know. Maybe there's a loss of other people who have been professional transcribers who are being put out of business. You know, for me, there is a bit of utilitarian thinking that I use myself that just to say like the benefit of having transcripts on my site, maybe outweighing any potential lost income from these transcribers that already it was in the tens of thousands of dollars to be able to pay for all these transcripts and I wouldn't have the transcripts. So yeah, I think it's one of those things of AI is to splicing labor and jobs and chat to BT, maybe trained on a lot of data that they didn't have complete consent for. And there's still a lot of oblique nature for what the data sets even are. So anyway, this was just a good opportunity for me to do a little bit of a extended discussion on some of these different ethical and moral issues that I think that are subtly coming up. And I had a chance to ask a lot of different folks about them. And, you know, there's a wide range of different thoughts. And I think at the end of the day, we're going to need some type of regulation to really address some of the biggest issues of some of these different models and some of the deepest harms. But I guess I still stand by the fact that we can't just sort of rely upon the regulation and push forward with some of these things without taking a step back and thinking about some of the deeper implications, especially as just as an example, if you are training a model with certain data that you don't have the rights to consent to, and then all of a sudden you use that model to be able to displace the jobs of people that you train the data using stolen data or data that wasn't fully consented to. So it's those type of ethical and moral dilemmas that I think is the underlying nature for a lot of these different technologies. But throughout the course of this series, we'll be diving much more into the possibilities and the use cases that a lot of these different XR developers and artists are pushing forward when it comes to using these different technologies. And so I guess now we'll be getting back to our regularly scheduled programs of looking at the intersection of XR and AI. And yeah, thanks for sticking around and listening to a little bit more extended discussion of some of these different ethical conversations You know, because at the end of the day, I think that, you know, a lot of the points that Alvin is making are correct. I guess I also just wanted to put forth some of those other counterpoints to live into the idealized version where we're also trying to be in right relationship and think about some of these things. proactively and ahead of time and not creating a situation where we're going to just fix it in post or fix it later or move fast and break things. And then all of a sudden we've moved too quickly when it comes to some of these things without really thinking about some of the implications, because that's sort of the default behavior and that's sort of what's already happening. So, anyway, thanks again for listening to the Voices of VR podcast, and if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a message-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
