We are in the middle of a hype cycle peak around AI as there are a lot of hyperbolic claims being made about the capabilities and performance of large-language models (LLMs). Computational Linguist Emily M. Bender and Sociologist Alex Hanna have been writing academic papers about the limitations of LLMs, as well as some of the more pernicious aspects of benchmark culture in machine learning, as well as documenting some of the many environmental, labor, and human rights harms from both the creation and deployment of these LLMs.
Their book The AI Con: How to Fight Big Tech’s Hype and Create the Future We Want comprehensively deconstructs the many of the false promises of AI, the playbook for AI Hype, and the underlying dynamics of how AI is an automation technology designed to consolidate power. Their book unpacks so many vital parts of the Science and Technology Studies narrative around AI including:
- How big technology companies have been using AI as a marketing term to describe disparate technologies that have many limitations
- How we anthropomorphize AI tech from our concepts of intelligence
- How AI Boosting companies are devaluing what it means to be human in order to promote AI technology
- How AI Boosters and AI Doomers are two sides of the same coin of assuming that AI is all-powerful and completely inevitable
- How many of the harms and costs associated with the technology are often out-of-sight and out-of-mind.
This book takes a critical look at these so-called AI technologies, deconstructs the language that we use as we talk about these automating technologies, breaks down the hype playbook of Big Tech, restores the relational quality of human intelligence that is often collapsed by AI. It also provides some really helpful questions to ask in order to interrogate the hyperbolic claims that we’re hearing from AI boosters. We talk about all of this and more on today’s episode, and I have a feeling that this is an episode that I’ll be referring back to often.
This is also the 100th Voices of VR podcast episode that explores the intersection of AI within the context of XR, and I expect to continue to cover how folks in the XR industry are using AI. Being in right relationship to every aspect of the economic, ethical & moral, social, labor, legal, and property rights dimensions of AI technologies is still an aspirational position. It’s not impossible, but it is also not easy. But this conversation helps to frame a lot of the deeper questions that I will continue to have about AI. And Bender and Hanna also provide a lot of clues to the red flags of AI Hype, but also some of the core questions to ask that help to orient around these deeper ethical questions around AI.
I’ve also been editing unpublished and vaulted episodes of the Voices of AI that I did with AI researchers at the International Joint Conference of Artificial Intelligence that I did back in 2016 and 2018 (as well as a couple of other conferences), and I’m hoping to relaunch the Voices of AI later this summer to look back at what researchers were saying about AI 7-9 years ago to give some important historical context that’s often collapsed within the current days of AI Hype (SPOILER ALERT: this is not the first nor the last hype cycle that AI will have).
I’ll also be engaging within a Socratic Style Debate where I’ll be mostly arguing critically against AI on the last day of AWE (Thursday, June 12th, 2:45p) after the Expo has closed down, and before the final session. So come check out a live debate with a couple of AI Boosters and an AI Doomer. Also look for an interview that I just recorded with Process Philosopher Matt Segall diving more into a Process-Relational Philosophy perspective on AI, intelligence, and consciousness coming here soon. Segall and I are going to explore an elemental approach to intelligence, which is based upon concepts that I explore in my elemental theory of presence talk.
Intelligence, privacy, and identity are also very contextual, and so if you’d like to hear more of my thoughts on the overlap between the Ethics of AI and XR, then be sure to check out my paper on Privacy Pitfalls of Contextually-Aware AI: Sensemaking Frameworks for Context and XR Data Qualities published as a part of the proceedings of the Existing Law and Extended Reality Symposium, and also my 20-minute talk on The Landscape of XR Ethics and my SXSW talk on both the Ultimate Potential of XR including both the promises and perils — many of the same contextual domains I covered about XR are also concerning AI.
I know that I’ll be referring back to this conversation with Bender and Hanna often as they’re taking a needed critical look and deconstructing the AI Hype. They have rigorous scholarship on this issue with lots of detailed footnotes that can send you off down many rabbit holes (one of my favorite ones was with Dr. Jonnie Penn’s Ph.D. thesis on Inventing Intelligence looking at the early history of AI). Even though it covers lots of the darker aspects of AI, their ridicule-as-praxis tone is playful and light-hearted as they skewered the more hyperbolic and ridiculous claims of AI Boosters. If you’re interested in moving beyond the marketing hype of AI, then I can’t recommend The AI Con book highly enough as it is an excellent primer that will also provide many new avenues of research into other scholars and journalists who have been providing comprehensive critical takes on AI.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Podcast: Play in new window | Download
Rough Transcript
[00:00:05.458] Kent Bye: The Voices of VR podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR podcast. It's a podcast that looks at the structures and forms of immersive storytelling in the future of spatial computing. You can support the podcast at patreon.com slash voicesofvr. So on the Voices of VR podcast, there's been around 100 interviews that I've done over the last decade that have looked at the intersection between artificial intelligence and XR technologies. That's virtual reality, augmented reality, mixed reality. And so we're in the middle of a hype cycle with artificial intelligence, with all these big companies making all these announcements, making all these hyperbolic claims. There's this leaderboard mentality where they are putting up these numbers and saying, we're the best at this or that. But there's a lot of problems with this methodology. And so on today's episode, I'm going to be deconstructing some of this AI hype by talking to the authors of a book called The AI Con, Emily M. Bender and Alex Hanna. They do a great job of looking at all this AI hype and starting to deconstruct from the computational linguistics and sociological point of view, looking at the limitations of these large language models. And, you know, just generally looking at the way that power is being used with artificial intelligence and just a point to bring this home. Within the past week or so, there's been congressional Republicans that have been wanting to pass a law that says that we should ban all the states here in the United States from passing any laws that are constraining or regulating artificial intelligence for the next decade. And so that's just one example of how a phrase like artificial intelligence, which is not well defined, and it can mean all sorts of things. And so if there's legislation that's passed, then that could basically mean that there's going to be put a moratorium on all tech regulation for the next decade. Which to me seems absolutely absurd, but this is the kind of hyperbolic hype cycle stage that we're in. For anybody that's familiar with the Gartner hype cycle, you have different stages where there's like emerging technologies. You have the peak of inflated expectations. That's the peak of the hype cycle. And then it goes into the trough of disillusionment, which is like you realize that all this stuff that's been hyped up doesn't actually live up to the hype that's being talked about. And then you go into the slope of enlightenment and then eventually the plateau of productivity. But it seems like in this AI hype cycle, we're kind of in this weird state where a lot of people have realized the limitations and constraints and they don't like the AI being shoved down their throat. But at the same time, we have people who are like true believers and all these companies are continuing to develop all these technologies. And so this is the type of discussion that we're going to have today to kind of break down all these different dimensions of AI. AI hype and even critiquing the word of artificial intelligence and having some alternatives like synthetic text extruding machines. So we'll be coming all that and more on today's episode of the Voices of VR podcast. So this interview with Emily and Alex happened on Wednesday, May 21st, 2025. So with that, let's go ahead and dive right in.
[00:02:57.287] Emily M. Bender: My name is Emily M. Bender. I'm a professor of linguistics at the University of Washington, and my main specialization actually is computational linguistics. And under that umbrella, in addition to working on computational syntax and semantics, I have been looking at the societal impacts of language technology since late 2016.
[00:03:14.949] Alex Hanna: Yeah, and I am Alice Hanna. I'm the Director of Research at the Distributed AI Research Institute. We're a nonprofit research institute that focuses on both the harms of artificial intelligence and new computational technology, but also tries to envision new technological futures. My work is a lot on the data behind new computational methods and the way that data exacerbates race, class, and gender inequality.
[00:03:43.902] Kent Bye: Great. Maybe you could each give a bit more context as to your background and your journey into this space of taking a critical look at emerging technologies.
[00:03:51.977] Emily M. Bender: Yeah, so for my part, I was minding my own business doing grammar engineering, which you can think of as automatic sentence diagramming, but across languages, when my field got taken over by a craze for language models. And that happened a little bit earlier in computational linguistics slash NLP than the rest of the world, because this is technology that comes out of our field. And in the late 20 teens, the people from the computer science side of the field started claiming that language models understand. And I said, hang on, from the point of view of a linguist, I can tell you that can't possibly be true because languages are systems of signs. There's the form and the meaning. And all there is in the training data for the language model is the form. And you can't expect your machine learning system to learn what's not in the training data. So I got involved in a lot of debates around that online and then got frustrated with the unending supply of people who wanted to argue with me about it. And my colleague Alexander Kohler said, hey, let's just write the academic paper and settle this debate. That paper came out in 2020. It's sometimes referred to as the octopus paper. And I'm sorry to report it did not end the supply of people who wanted to come argue with me.
[00:04:59.551] Alex Hanna: Yeah, and I got into this field because I'm a sociologist and a sociologist that's focused on the social impacts of technology and have been since I entered the field as a grad student in 2009. And so much of what I've been focusing on had been the effects of social media, in particular on social movements, and have a body of scholarship on that. Then was using some computer computational social science methods, especially kind of text processing and machine learning, to do some analysis of newspaper data on social movements and then kind of in about nine or ten years ago at this point had been paying attention a lot to this kind of development in fairness around machine learning and the fairness, accountability, transparency conference that had come up around that time. and then was looking a lot more critically in machine learning models and their impacts on society. And then around that time, there was, as Emily said, everybody started losing their mind around large language models and then started looking a lot more around that. So Emily and I connected a lot around the data that was being used to evaluate large language models, wrote a few academic papers around that and started focusing on that more intently.
[00:06:10.762] Kent Bye: Yeah, and as an experiential journalist covering emerging technologies, I do see this kind of factionalization where people are either true believers in a lot of this stuff or they're complete skeptics. And, you know, I try to find the path in the middle, but maybe you could start to lay out this argument for how there are some more delusional aspects into what's happening with the AI hype around artificial intelligence and how you start to lay all those arguments out in your book called The AI Con.
[00:06:40.410] Emily M. Bender: Yeah, so I think a key part of it has to do with how language works and how we work with language. So I've already said that the only training data available to a language model is the form of language. And it's a little bit hard to experience the form of language without the meaning behind it when we are looking at or listening to something in a language we speak. But, you know, imagine yourself traveling abroad to some country where you don't speak the language. You can sort of set up that experience of I can tell people around me are talking and communicating or I can see there's a bunch of signs that have something to say, but it doesn't mean anything to me because I don't know that language. And so it's worth keeping in mind that that is the input to a language model. However, the language models have gotten so big and there's some clever architecture in there and have so much training data that it is very possible now to see very coherent seeming text that comes out. And when we see that coherent seeming text, we do what we always do with language, which is we attempt to make sense of it. And the way we make sense of language is by imagining the mind behind the text. Somebody said that. I'm going to keep track of everything I think they have in common with me in terms of common ground. They must have guessed about my state of mind or the state of mind of their reader if they didn't know it was me. And then ask myself, what would they be trying to convey by picking those words in particular? And that's how we always interpret linguistic signals. And so it's very hard when we see the output of something like ChatGPT to keep in mind that all of that's actually on our end. If it makes sense, it's because we're making sense of it. That parlor trick then gets dressed up further with the way the interface is built. There's absolutely no reason for a ChatGPT or a Claude or whatever to use I and me pronouns because there's no I inside of there. That's absolutely a design choice on the part of the programmers. but it is designed to make it feel like you are talking to an entity. And then once you have that in place, that sets up the possibility for further cons in the AI con that maybe I'll hand it over to Alex to talk about.
[00:08:40.078] Alex Hanna: Yeah, so I think the thing, and I mean, the way you're framing it is there's sort of this very big hype extreme, and then at the other end, there are the skeptics. And I think, you know, like, I understand the impulse to kind of go in the middle Because that is also my impulse is sort of like, where is the truth here? And I think the thing about it is, I mean, I think it's gone to such an extreme. When you start looking behind the surface, you, I mean, for me, became skeptical very quickly. So what is the intention of these systems? And what is the intention of what they're doing? So because of what these systems are doing is fundamentally trying to Generating the most probable output or sequence of tokens from an input sequence of tokens, then there is no way in which this architecture gets us to a place where we're going to ever resolve the quote-unquote hallucination problem. So that's one of them. And we don't like to call them hallucinations because that imbues them with a particular sort of anthropomorphizing. And what it also signals is that where the data is coming from, is this set of texts which had not been given with consent. In fact, it's sort of anything that's not nailed down on the internet. And then the third part of it is the sheer amount of environmental damage going into the training of the models. And so all four kind of seemingly realistic sounding texts for the purpose of what it seems to be replacing many humans in their labor and in social services at scale. And so it seems to be like, you know, if you're going to say what technologies do and things that I think also deserve ridicule, things like nfts or blockchain or all these different things which haven't really found a suitable use case what are these large language models attempting to solve and someone put it very pithy on blue sky which is the problem they're trying to solve is wages and so when it comes to that it sounds like well it doesn't really sound like there's a middle position here in this case actually i think the middle position is sort of saying well there's a problem with these and it's a question of power and they're manipulating the form of language for the purpose of subverting power of most people in the world.
[00:11:04.075] Kent Bye: Yeah. I wanted to dig into a bit of the benchmarks because that's part of the work that you've both worked a lot on. And I was just watching the Google IO keynote yesterday, and there's a moment where they put up a slide of like, look, we're number one on this leaderboard, giving some perceived empirical validation that these models are using some scientific method of research when they're actually there's a lot of holes in terms of how the data are collected but also like the limitations of these different types of evaluations and are they being trained on the data there's no transparency all these things that you're starting to unpack and critique in terms of this number go up leaderboard type of mentality when i went to the international joint conference of artificial intelligence back in 2016 did a bunch of interviews and one of the comments i got from a researcher was that the empirical results were outpacing the theoretical ones And so what that meant to me was that, hey, they're getting results, but they have no idea why. And I think it's part of this mentality of creating a benchmark and then chasing that number that is collapsing context of how to actually evaluate these systems. So I'd love to hear some of your work on looking at and deconstructing the benchmarks and this whole kind of leaderboard mentality that AI has.
[00:12:14.045] Alex Hanna: Yeah, I mean, so we read a few papers on this, and so we wrote two papers together on this topic in particular. So one of them was called AI and the everything in the whole world benchmark, which talks about this notion of kind of a general purpose benchmark. And in particular, we focused on one that was considered a general image understanding data set. So that's ImageNet. And then another one that was for natural language understanding or natural language inference.
[00:12:43.198] Emily M. Bender: Understanding.
[00:12:43.959] Alex Hanna: It's understanding. Yeah. Yeah. So that one's considered understanding because it's glue. General language, understanding something.
[00:12:50.686] Emily M. Bender: Evaluation, I think.
[00:12:51.527] Alex Hanna: Evaluation.
[00:12:52.688] Emily M. Bender: It was actually only for English. So the E should stand for English. For English.
[00:12:56.072] Alex Hanna: Yeah. Yeah. Never. The classic Bender rule, which is, you know, you need to ask what language it is. And it's probably English. So we focused on that and then we had another paper called Data and its Discontents. And then I had a few other papers on ImageNet in particular and the focus on what ImageNet is and how it came to be and telling kind of a critical history of ImageNet. And then we also had another paper kind of on a survey of benchmark data sets, in particular image computer vision data sets, which are images. And so the really interesting thing with this is that the way that benchmarks get cast as doing everything. So that's one thing. I mean, that was one of the points that we made in the AI and the everything in the whole wide world benchmark paper, in which we're focusing on this notion of generality is this thing that's, you know, it's not a goal that can be met in particular. And it comes from this children's book, which is the, what is it? The Grover and the everything in the whole wide world museum or something. Yeah. And so in which the Sesame Street Grover looks at everything else in this museum and looking at all these different things. And then there's a door labeled everything else and it's the outside. Right. And so this is sort of a tale about external validity and face validity of saying that there is a general benchmark for anything, which is not possible because, you know, you can't actually have a general benchmark. And there's some great work, too. And I think it was DeVries. It was a team that's actually an old paper that came out of Facebook research, and it was showing effectively that image data sets in particular are highly represented from images in the West. And then there's a dearth of images from anywhere in South Asia and Southeast Asia. and Africa. And so there's this kind of validity problem of the benchmarks. But then it's also this kind of you also rightly pointed out that then there is this double problem of the leaderboard culture. And before Jeff Hinton went completely existential risk, he was also even decrying this and a sort of idea that like scaling itself has become this issue. You know, you make number go up, that's good enough. And there's not really any novel kind of methods emerging within the various disciplines that fall under the cover of AI. And so you're trapped in this very particular sort of epistemic culture of doing work, and there's very little methodological innovation. And so there's this double-edged thing. There's an idea that these AI researchers are very careful, they're very methodical, they're very concerned with metrics, but at the same time, the kind of research itself is actually quite sloppy, and it's not really thinking about what it means to do more precise science and engineering.
[00:15:38.817] Emily M. Bender: So you mentioned that someone said to you that the empirical results are outstripping the theoretical results, which sounds to me like there's actually no results. And you can see this in this culture of leaderboard chasing or soda chasing when people get very upset if they can't post their papers right away on archive because then they are going to cease to be relevant because they're going to get scooped. And basically, if your paper isn't going to be interesting in a month, why would you think it's interesting now, right? And this is because just making the number go up is what it takes to get the cookie in this race right now. But if you just make the number go up and you don't have anything else that you've learned in the process, then there's no durable research results there.
[00:16:20.677] Kent Bye: Yeah, one of my favorite footnotes that I discovered in reading your book was Dr. Johnny Penn's Inventing Intelligence, going into the history of the creation back to the 1956 Dartmouth Conference on Artificial Intelligence. And so you're making the argument that artificial intelligence as a term is kind of a marketing term that's always meant to draw from the human capabilities and make the technology way more powerful than it actually is. And that you are making a number of different arguments in terms of there's a devaluing of humans that are reducing people down to the computability, quantification and rationality aspects, but also like drawing on these more race science and eugenics and hierarchies of value that are also very problematic in the history, but are also tied into this. So I'm wondering if you could just comment on, you know, this calling it intelligence and the metaphors that we use and the ways that you're trying to fight against those metaphors, but also trying to ground it in other language, but trying to also prevent this kind of anthropomorphizing that people can have with AI that is projecting way more capabilities than are actually there.
[00:17:27.697] Emily M. Bender: Yeah. So there's two problems. I'm going to start with the metaphors, Alex, and I'll let you talk about the history of intelligence. And so the two problems are sort of it oversells the technology. And I want to say a bit more about that. And then there's also the fact that it is just completely embedded in this deeply racist project that you can't escape from. But On the metaphor side, before we get into the really dark stuff, if you call something artificial intelligence, you're doing two things. One, you're lumping together a bunch of technologies that are actually disparate. And so every time ChatGPT gives you an output that you find pleasing or the autofocus in your camera does the right thing, if you think that that's actually one technology, then you're building a mental model of what it is that's completely inaccurate. You're sort of lumping it all together, and then you're more likely to accept, you know, again, so-called artificial intelligence in consequential decision-making systems. everything from social services to peer review and science, because look, it can do this, it can do that. It's one thing, it's not. So there's the lumping aspect of it. But then there's also the fact that intelligence, a very poorly defined term that Alex will say more about, indexes things that people can do. So rather than talking about automation, where you can say, I made a system to do specifically this, and I can evaluate how well it does specifically that thing. Instead, the project becomes, I'm making a system that is generally amorphously intelligent, and so therefore I'm going to evaluate it by just throwing it at these different leaderboards and, you know, number go up. So that's already bad enough. But then if you look into the history of it, it just becomes completely awful.
[00:18:55.120] Alex Hanna: And I appreciate you going down. I think I saw you say something about this online and tracing down Johnny Penn's dissertation, which is a fascinating artifact, right? Because what he does is he ends up tracing down kind of what is meant by artificial intelligence by the different people involved with the original 1956 Dartmouth project, right? And so he traces down what Minsky was working on and what McCarthy was working on and what Rosenblatt was working on and Herbert Simon and all these different people that had been part of this project. And the interesting thing as artificial intelligence is now, and as Emily laid out, the same way in which there's not really a coherent set of technologies today, it's the same way as it was back then. And so the kind of technologies are a bit as disparate as they are today in which they include everything from linear regression to a transformer model, right? And so back then it was kind of everything from the perceptron to automatic programming, or rather what we'd call today programming languages. And so there's this huge variation on this, and there's kind of this idea that relies so much on the metaphor of intelligence and the metaphor of a computational brain, or at least that's outwardly the project or the mythos that Minsky And to some degree, McCarthy kind of perpetuated through that history. And then just talking about the dark history of intelligence, you know, this is, I think, a little more well-trodden territory. But this kind of notion of intelligence as one singular metric that can be optimized for this kind of notion of IQ. And one of the fun things, I mean, not so fun things, but one of the things that we've found throughout this process is The AI researchers that we cite in the book and that we describe in terms of leading these particular organizations have no definition of intelligence. They like to say it's kind of hotly debated. They tend to focus on, especially when they talk about artificial general intelligence, and that's where the equivocation gets very, very strong. but intelligence itself as a singular metric has this dark history that's connected with IQ testing and this history of what Stephen Jay Gould calls reification of intelligence that is tied to both positive and negative eugenicist projects, negative insofar as discouraging populations from procreation. In the UK, that probably looks like these draconic programs against people who are mentally disabled, and the US is primarily along the lines of race and race ordering. That's well established from people who are well embedded in Silicon Valley as a geography. So Stanford is very much implicated in that. Some of the major leaders in the development of the Valley are very much implicated in that. And that reliance on this notion of singular intelligence really persists throughout the project.
[00:21:51.377] Kent Bye: Yeah. And this collapsing of the human value where humans get collapsed into computability, quantification, rationality, where things that can be maybe reflected in the AI automation technologies. And that a theme that I saw come up again and again was this feminist concept of situated knowledges and how Important it is to have a very specific context and perspective and how knowledge is actually a mix of all these different paradigms, whether that be science and the process of science from these different paradigms through art, or if people are working in the context of medicine, they're in a community of practice and they're in relationality and there's something around the AI technologies that are collapsing that relationality. And kind of stripping out that context where it's almost like, you know, girdle incompleteness, where you're trying to get all of those perspectives into one unified one and eliminating all the diversity of those situated knowledges. So there seems to be like some fundamental, like philosophical problems there when you're taking all these diversity of perspectives and then collapsing into a uniform perspective without maintaining the individual unique perspectives of each of those perspectives. So I'd love to hear some of your thoughts on that, because it seems to be a common theme was the collapse of that context and the ignoring of that relationality.
[00:22:59.738] Emily M. Bender: Yeah, so relationality, I think, is super important. And Abebe Burhane has some wonderful writing on that. She talks about relational ethics in the context of machine learning and AI. And I think you're right that the idea that we can get to a view from nowhere by feeding everybody's data into the machine and letting the machine reflect it back out. loses a lot of the humanity behind the speech acts that are being collected and a lot of the relationships between them. I mean, if you think about how information exchange goes, that's also inherently relational. Who said it? Who said it to whom? What is our relationship to each other? But I think it's also not just relationships that get lost, but also what it is to do the thinking and the relating behind the words when the people selling these technologies say, a simulation of the conversation that a tutor has with a student is a good enough relationship replacement for actual tutoring. They're basically saying all that matters is the words and not the thought and the experience and the relationship behind them.
[00:24:00.339] Alex Hanna: Yeah. And I appreciate you pointing out this kind of notion of situated knowledge. And one of the sources that we cite in the text is Lisa Masseria and MJ Crockett. And so in particular, they're two feminist anthropologists of science. And one of the things that they're really noting here is kind of the notion of like they're in particular attacking this idea of AI for science. So this idea that you can have something that they call the sum total of human knowledge or human science, which is on the face of it already an absurd concept or anything to say, but then also noting that there's a way in which there's a recombination that you are recombining or finding new types of ideas by recombining and doing reforming the word forms of scientific texts, which is not at all how science works. That is not, you know, science is the most basic formulation of a scientific method is, you know, the formulation of a, not that it happens like this. And of course, this is criticized from top to bottom by many people, but the kind of development of hypotheses and a series of experiments and sort of a testing of that. And moreover, the development of a kind of a research program that focuses on developing ideas and working within research programs and trying to also developing new methods and methodologies of challenging those research programs. And so this area and Crockett do a great job at going after this in the AI for science domain and saying, you know, like, Well, many of these things depend on communities of practice. They depend on, you know, having discussions, the different process of ideation. It is not simply a matter of kind of recombining and trying to suggest that you have the quote unquote best science and meshing these together into kind of a way that this makes sense. And I think one of the things that the AI for Science crowd really misses is that they are It misses the idea that there are distinct epistemic cultures and that actually having certain epistemic cultures disrupt other epistemic cultures is actually part of the process of developing new ideas and has led to a lot of different innovations in practice. And so I think that's something that then gets taken to the extreme by many of the tech companies who are developing these products.
[00:26:18.181] Kent Bye: Hmm. Yeah, when I was doing an interview with Daniel Lufer of Access Now, we were talking about the AI Act, and he was critiquing some of the utilitarian nature of a lot of the philosophy of ethics where as long as it works for the perceived majority, then you're going to have this move fast and break things where you ship things out because it is whatever your target demographic might be is a perception that it's working for most of them. And for the people who it's not working for are already these marginalized communities who are then having all of these automated systems do more and more amplified harm. So I think a film like Coded Bias does a great job of elaborating that dimension. And so Lufer was saying that you really need to take a human rights approach, which is a lot of how the AI Act was developed. But we still are fighting against this culture of you're shipping things without any checks and balances, you know, not even responsible innovation anymore. It doesn't seem to be any strong regulatory approach to AI here in the United States. You know, as you document in your book, they're being bombarded by both AI doomerism and AI hype, which the function of that is to delay any functional action. So I think the EU is taking some of the most credible action in terms of regulating AI. But in terms of the harms, it seems like there are these marginalized communities that are suffering from the impacts of these large models. And I think a big part of your book is also trying to document a lot of these costs that may be hidden from people if they're just using it for vibe coding or image generation, or whatever it may be for their constrained use case, where they're finding some perceived utility, there's all these other hidden costs that are happening. So I'd love to hear you just describe like, how do you start to talk around the harms and some of these marginalized communities? And how do you tell that story?
[00:28:01.300] Alex Hanna: I will say that I do want to push back because the EU AI Act ended up like it's got some things that are good, but it's got a lot of things that are not good. And I was actually looking at the things that Luther had said and more recently and, you know, had posted a criticism of some of the carve outs for human rights. In particular, there had been some carve outs for biometric mass surveillance. And then as well as people on the move, people who are refugees in particular places, because those are places in which the EU AI Act, really falls short. And I know Access Now and Amnesty Tech and Human Rights Watch had posted about this. And we have, if you want to hear more about that, our most recent episode of our podcast on episode 57, we talked with Petra Molnar, who talks about this in her book, The Walls Have Eyes. And so just getting back to your original question, thinking about the hitting costs and the costs that people aren't as top of mind. And one of the things that we talk about in the book are the costs in terms of labor and labor displacement, but also labor, the people who are in the AI supply chain. So people who are moderating the content, who are red teaming who are encountering toxic content, gore, child sexual abuse material. So they're in this process of doing moderation, both in training data, but also in moderation, and sometimes actually pretending like they're chatbots themselves. So that happens more often than one would imagine, but still does. And the one case that we cite in the book is the CEO of Cruise, which was operating as an automated taxi in San Francisco until a few accidents there. He admitted on Hacker News that these things were being driven by humans two to four percent of the time. And then we also talk about the environmental concerns, which I think has gotten a lot more attention lately, especially with the struggle that's happening in Memphis right now, where Elon Musk is attempting to build a quote unquote supercomputer, which I'm just interpreting to mean a data center powered by 35 unlicensed methane gas generators and is polluting this area of Box Town in southwest Memphis, which is predominantly black and poor. And so there is this idea where these things are kind of out of sight, out of mind because they are in the cloud. And this is where training happens and where this is where inference happens. But these things have real material impacts in the world. The cloud is here now.
[00:30:27.881] Emily M. Bender: Yeah. And all of that is hitting those who are already marginalized the hardest and first. So there's the direct pollution landing on Box Town. And there's also the contribution to the climate crisis and the communities that are bearing the brunt of that first are largely in the majority world or in parts of the West that have less resources. On top of that, we find in the application of these automated systems, you've got Virginia Hewbate's book, Automating Inequality. So using automatic decision systems to keep people in what she calls the digital poorhouse. There was a case we talked about in the book of a health insurance company, United Health, that used an algorithm to basically automatically deny people access to sufficient care in rehab centers and so on. And so we're sold this promise of, oh, it's going to be more efficient. We're going to have this things that are available to everyone in the face of scarcity. And instead, it's like, how can we more quickly automate the denial of services and denial of care and denial of access, denial of freedom in the case of pretrial sentencing to people that are already marginalized and that those in power would like not to offer those things to.
[00:31:37.740] Kent Bye: Yeah, and I think another big aspect of the AI systems that we have right now, these big, large language models, is that it's all built by stolen data. There's sort of a colonial nature of these companies seizing all this information that is available on the internet. And as I talk to people, I often get the argument of Kirby Ferguson and everything is a remix argument, which is like, hey, artists and humans are all about, you know, remixing things. And it feels like that there's something different around this at the level of scale it is, the level of consent. And there's something I think that's different in terms of what is happening with these automated systems and large language models and what is the normal creative practice.
[00:32:17.630] Emily M. Bender: Yeah.
[00:32:17.830] Kent Bye: So I'd love to hear you kind of deconstruct. How do you deal with people who cite the everything is a remix argument?
[00:32:24.493] Emily M. Bender: It's such a dehumanizing approach to what's going on. But the first thing I wanted to do is to link back to what we were saying before about regulation and this sort of what the big tech companies are saying about how they shouldn't be regulated. And we're starting to hear discourses of, well, we can't possibly build this technology if we can't just go grab everybody's everything. And that argument seems to come from a place that they have an inherent right to build the technology. And no, they don't. Right. I think it's really important to sort of start our refusal there. Like, we don't have to accept that this is a necessary part of the world and then like contort what we're doing and contort our interpretation of, say, copyright law to make it possible. But against this, everything's a remix argument. It basically is saying that when a company steals data and feeds it into a machine and remixes it, the machine is doing the same thing as a person who reads or views or listens to something and experiences it and is inspired by it and then makes reference to it and is in relationship to it in their own art. And the machine can't do that because the machine is not the kind of thing that can have experiences, that can be in relationship or can have any accountability.
[00:33:30.106] Alex Hanna: Yeah. And I mean, I think there's I haven't seen the video that you're referenced, but I'm familiar with the argument that I think a few people are making. So just to add on what Emily is saying, I mean, first, the dehumanization aspect of it. First, it depends on that you would have to take seriously the metaphor that the brain is a sort of computer and that they're doing the same thing in operation. So that's not happening on the face of it. Right. The second part of it is thinking a kind of conception of what art is and what inspiration is, which I think is much more about thinking about who you're in relationship with. You know, even something if you're not in a relationship with a creator, your favorite creator, but you're writing fanfic, even fanfic draws on some kind of a community of practice of other people who are writing fanfic or drawing on inspiration for a piece of IP that somebody respects and enjoys, you know. And so there's that aspect of it. And that's not what's happening there. I mean, to produce a single image in a diffusion model, one must rely on thousands, if not millions of other images that are taken without consent or that need to be absorbed into a system's training data set or whatever, right? And so these are typically I mean, there's smaller models that are not necessarily doing that on the image side, but those are not what's happening with mid Germany or stable diffusion. And so There's a very technical argument that's almost there that's sort of like, that's not what's happening both in practice because the fusion models don't work like that, but then the artists who are fighting against this are saying like, you do not have our permission to do that, right? You know, you could imagine a particular sort of litigious organization like Disney saying, like, you can't make fanfic of our thing. But that's not what's happening here. It's artists like Carla Ortiz or Greg Rutakowski or Kelly McFadden who are saying, like, you are creating things in the style of us and is taking our data without our consent.
[00:35:39.409] Kent Bye: And it seems like that in your book, you go quite a lot into how the AI doomers and AI boosters are basically the two sides of the same coin, that the functional results of AI doomers is that they're still putting AI onto this pedestal, deifying and almost treating it like this god, that it's way more powerful than humans, but that the end result is that it's still this hyperbolic AI hype that is... not really paying attention to the existing harms that are happening right now and so in part of deconstructing the ai doomers you make a couple arguments in the sense of on one they're kind of having this circular references where they're not having peer review they're not having good citational practices they're kind of writing these blog posts and referencing each other But then there's other arguments that there's actually funding and they're creating like a completely separate epistemic communities that are completely isolated and maybe having more academic rigor, but still they're not really in communication with the broader community. So I love to hear a little bit of deconstructing of the AI Doomer and how it's maybe serving the same function of the AI booster.
[00:36:46.112] Emily M. Bender: I have a way of saying it that I've been honing over the last couple of weeks of talking about this. So the boosters and the doomers like to present themselves as like the extreme ends of the continuum of possible positions on this topic. And a lot of the media picks that up as well. And then they say, are you somewhere in the middle? And in fact, as you said, they are two sides of the same coin. And you can see that very clearly when you say, okay, well, the AI boosters say AI is a thing. It's inevitable, it's imminent, it's going to be super powerful, and it's going to solve all of our problems. And the AI doomers say, AI is a thing, it's inevitable, it's imminent, it's going to be super powerful, and it's going to kill us all. And so it's just that last little turn at the end that makes it different. And in fact, in that closed epistemic community, those people talk to each other. The citation networks that are often citations to non-peer-reviewed things, you'll get people who say, well, if people cite it a lot, that is a kind of peer review. That's absolutely not how peer review works. When people cite something, it's very frequently, I have an idea, I want this part of it to be supported, I'm going to go find a paper that supports that. where peer review, when it's working well, is someone actually looking very critically at something and saying, does this argument actually go through? Have they done the experiment in a reasonable way? And a lot of things that we see in this space and then being cited in this space don't even say, like, what the data was that they use, like specifically, what was the input, what was the output, and then it'll get picked up and cited. And so you end up with these very shoddy paper shaped objects that are paper shaped in the sense of having a bibliography. But if you look into the bibliography, it's just archive paper after archive paper after company blog post.
[00:38:23.976] Alex Hanna: Yeah, and I just want to talk a little bit. There's some really nice paper by Shazadeh Ahmed and Amy Weinkauf and a few others that's about the particular sort of epistemic community, particularly around AI safety. And there's a sort of way in which there's like a really hyper focus on, you know, there's kind of already an agreed upon set of things that don't really touch many of the concerns that have been raised by other people on AI harms. It's really insular. There's not only the archive problem. I mean, it's getting worse and so far as people don't even post things on archive anymore. That's that's even a bridge too far. I mean, people are just saying OpenAI or Anthropic, they're just posting papers on their website. Right. And so there's that problem. But there's just like the citation networks are focusing on. This is the problem that we're focusing on. And it's the citation is of this and there's you know if there is a citation that's outside of the network it's other people have said this but we're going to decide to focus on these other things we're talking about this with regards to the stochastic parents paper which is you know these people are focusing on this thing but we're not going to talk about that we're just going to talk about this other thing so there's A particular sort of epistemic community that is very closed, there's also the funding networks that come with it. And I think this is also well-referenced in the test real paper by Tanit Yabrou and Emil Torres, in which it focuses on many of the same kind of sites and projects, like 80,000 hours is kind of one of these effective algorithms. projects. There's also these AI safety awards and some of these very large conferences like NeurIPS or ICML, where they focused on, if you work on the safety project, you can get like $50,000 to $100,000, which is just this incredibly wild prize that you would have. And so they're very well-funded in addition to these particular closed epistemic communities.
[00:40:18.720] Kent Bye: And I guess an associative link to some of the talk around the doomers in the boosters is artificial general intelligence, super intelligence, which, you know, is still speculative. There's no proof that that's even going to be possible. But also the thing that I keep coming back to is the role of context and how there's this kind of like context independence where they want to have something that is going to be uniform across all these different contexts. And Amy, I know in language, there's like pragmatics or the role of context that changes meaning, or at least I feel like there's something around like intelligence and context where there could be a contextual dimension to intelligence where it does have like bounded applications, more narrow that has a more specific context rather than something that's way more general. So I'd love to hear any like reflections on the role of context relative to both the AGI and superintelligence?
[00:41:10.322] Emily M. Bender: Yeah. So first of all, I'm going to take issue with you saying that these things are still speculative because that seems to entail that at some point they might not be. And I wouldn't even go that far. They are purely speculative in the sense of speculative fiction, not in the sense of financial speculation. And I think that when you talk about context on the one hand you can say we can build useful applications that are built for a specific context and then we can evaluate them in their context and we can decide do we want to use this do we have enough information out of that evaluation to decide that it's a good tool for our use case but if you want to talk about something that is supposedly general then you would need a system that is adaptable to different contexts and i think that um To get something that is adaptable to different contexts, you either have it doing one thing that is very specific. So people will sometimes reference electricity as a general purpose technology. Electricity is not a technology, it's a natural phenomenon, but there's technologies around it that have certain specifications to them, right? This wire carries the current in this way. I'm not an electrician, but like you can get the components and they have a specific set of affordances. And then someone who is building a system out of them knows that they can build on that. So that's one way to get something that works in different contexts. It is what it is. It's not varying, but what it is is well-defined. And so somebody can use it in different contexts. If you want something that of its own accord is adaptable to different contexts, I think you would need something that only exists in the realm of science fiction at this point. The problem with the synthetic text extruding machines is that they look like they're doing that, right? They look like because the text that comes out is sort of prompted or instigated by the text that's gone in, and then that draws on different aspects of the training data, it looks like it's very flexible in different contexts. but it's none of that, it's just an illusion. And one of the things that people will sometimes say is they'll say, well, you know the meaning of a word by the company it keeps, right? This distributional semantics idea. And the problem with that is that it's being misinterpreted. We certainly do make sense of words in context and we learn what they mean over time and those meanings change over time based on how they're used. So meaning is use, but use isn't just what other word forms are around it, but what someone used it to convey. And so if we're gonna get to something that can actually flexibly work across contexts, that thing would have to be something that could stand in relationship to other people and have a positionality and have accountability. And that doesn't describe a machine.
[00:43:44.778] Alex Hanna: Yeah, I mean, I think this idea of context is such an interesting word because I think it has so many different disciplinary meanings. And so a hilarious thing. There's a great essay by Dick Seaver, which is called The Funny Thing About Context Is That Everybody Has It or something of that nature. It's got this great kind of it's just one of these cheeky anthropologically titled papers. And it's sort of like the idea of. the certain kind of idea about the flexibility of the term context, because computer scientists use context in a very different frame than how anthropologists do, than how sociologists do, and sociolinguists probably do, but I know very little about, you know, I know less about sociolinguistics. And so it's an interesting idea about how context even gets used in terms of how training is done and how there's a context window about a particular text. Whereas as if context could be gleaned like particular in terms of just the characters around a certain token. And so I think when we talk about context, it's a little harder to really think about it. It makes more sense to talk about something like tasks or maybe even like situations. But I think a lot more about human properties and more about What are the nationalities? Who are the people using it? Who are these people? What is their gender? What is their ethnicity? What is their socioeconomic class? What are they going to be using it for? What does this supplant in terms of other technological use? What does this improve? And when you think about that in a case, how could anything called AGI be formed that is nothing but a hegemonic use case? Because those things are going to be considered marginalized or edge cases. And that's not very helpful for people in those social positions, right? They want technology that's going to be working for them for their particular situation. And instead, you know, that's, of course, antithema to the way Silicon Valley operates, because you kind of have a one size fits all thing. And it tries to get people, you know, come up with very interesting, cool ways to adapt. But that is not really how these things are thought of. And that's certainly not how AGI is being thought of to be such a thing or to ever exist. I mean, it's not being created with those people in mind.
[00:45:52.583] Kent Bye: I have this experience of having conversations with people, and I'll often have people that are using some sort of generative AI, maybe vibe coding, image generation, something where they are receiving some valuable utility that they're able to do their job better by using it. they're not seeing any perception of anything that's wrong. And so I'm wondering how you deal with those different types of situations. If you try to like really analyze the inputs and outputs and see if there's maybe something that they're missing or there's like a lack of AI literacy to know like the limitations of the technology and just kind of uncritically using it. And then if there's nothing immediate that you can see that they may be perceiving that, then there's like maybe a second order argument, which is talking about the hidden harms that are not being seen. And so just curious to hear a little bit about your strategy of how you negotiate these conversations to try to reveal how they may be immediately being harmed by using this. But then if that's not clear or not buying that argument. than if you go into like the other levels of harm that are used in these technologies and how you negotiate that on the scale of a one on one conversation.
[00:46:58.496] Emily M. Bender: Yeah. We were actually on another podcast recently where one of the hosts was just adamant that his use of these things was valuable and fine. And we would say, but here's all these harms are here. So one of the technologies was this thing that recorded everything he said and heard all day and then wrote a diary entry for him at the end of the day. So something that is completely useless. And he was evenly admitting that at least half the stuff in the diary entries was incorrect. And he was like, I was very sycophantic, but he somehow, it was something he actually physically wore. And so we pointed out that that was, you know, a violation of privacy of the people he was talking to. And he kept saying, oh, I completely agree with you. And then like continuing to say that he was going to use it. So We don't always succeed in those one-on-one conversations. I do think that it is worth sort of stepping people through, okay, so in this situation, what are you doing with the output? And then sort of going from there to, they say, well, I can check it. Well, really like how much time do you put into that and what are the costs if it's wrong another direction is well it's helping me work faster well do you get the benefit of that or is the company you're working for getting the benefit of that because they're just going to ask you to do more work i do think it is really worthwhile talking about the harms that are not to the user themselves but to other people i think it's really important people to know about that i have yet to see somebody who was excited about using the technology who was convinced by that argument
[00:48:23.670] Alex Hanna: Unfortunately, so the kind of strategy I take, I think a lot of people really aren't convinced by the environmental and labor stuff, which is a little upsetting to me. But, you know, I think those are harms which are often so diffuse and over there. It's unfortunately probably not that compelling. the things that i think are a little more compelling are the kind of things in which are about the uses of these technology uh technologies of labor displacement and so the focus that you know you're using this thing now and it might be offering this but there might be a case in which this is a thing that is not offered as something as a quote-unquote collaborator as it's being sold now but is one that is meant to be something that disciplines labor and so there are a lot of people that come up to us that are saying i have to use this as part of my job and i don't like that and so i think that kind of way and taking away people's own control of what they're doing is can be pretty compelling because people don't like control taken away from them who would have thought so that's one thing i think another compelling argument is often the one from thinking and critical thinking in particular So one of the arguments that we offer in the book very briefly, and it's helpful that there's some research behind it now too, is that the process of writing and the process of creating in terms of diffusion models, even drawing or doing things in a kind of a digital or graphic design format, is that that is the process of thinking itself. So that's the argument we call the argument from craft in terms of if you are doing this, this is the process. The process is not the ideation. We have ideas. you know, I have 100 ideas and that and 450 will get me a cup of coffee. Right. So that's the kind of aspect of that, which I think is like the actual work is actually writing this down and working through this, working through your argument, working through your work product. And there's been some research and I don't think this has been peer reviewed. So take it with a grain of salt. But I mean, some of the people who are more trustworthy of LLMs and use them extensively have less critical thinking skills. And so that kind of muscle is atrophied a bit. And so that might be a compelling argument to, I think, some folks. And then I think there's also maybe just like now at this point, I've seen this argument proffered and I don't know how compelling it is, but some people are like, well, these things are becoming so pervasive. It might actually give you an edge, right? It might actually give you an edge on the job market. If you're saying like, I don't use these technologies, I don't vibe code, you know, I know how to think through a problem, you know, and that itself may become its own sort of credential. And in a market where it seems like people have let this set of critical thinking or problem solving muscles atrophy out, And I'm seeing more of that argument being proffered because there's been so many things about students using these and we don't have like a systematic survey on this. I think the last survey I heard on another program was something like 25% of all students use them. So there might be something offered from especially a student level offered and saying, well, I don't use these things. I think through my own work and I have that critical thinking muscle.
[00:51:43.699] Emily M. Bender: We're also seeing students who are getting upset and leaving coding boot camps when they discover that the company has laid off all the teachers and replaced them with access to ChatGPT. And the students are like, if I wanted to ask ChatGPT, why would I pay a company to let me ask ChatGPT, right?
[00:51:59.176] Kent Bye: Yeah. Yeah, and I think in the last section where you start to go through some of the questions that you're asking in terms of interrogating, some of them around the models, I wanted to, before we go into some of the specific questions, I wanted to ask more of a general question around like this kind of open source washing that we've seen within the community where They'll claim that something is open source, but I think, you know, your metrics for something being open would be like, what are the algorithms you're using? What's the data? And then, you know, kind of the more transparency, like the hugging face example, where there's model cards, where there's more transparency around the data, what's there, how it was gathered, where the data practices around that. So when you think around openness and transparency around these models, what are the things that you would want to see in order to really consider something to be truly open?
[00:52:45.796] Emily M. Bender: I mean, I think you've basically covered it there. So not just what's the source code. And sometimes I'll say, look, if you don't have access to the data, it's not open source. And on Mastodon, I get the smartasses replying, well, the data is not part of the source code. It's like, well, actually, in an interesting way, it is, right? Because the resulting trained model is about the interaction between the data that went in, the learning algorithm, and the learning so-called curriculum, how they applied the learning algorithm. And if you don't have access to all of that, you don't have complete information about how it was built. And I think that the original ethos behind open source for many people is, can I just use this myself? And so people in that mindset are going to be satisfied with the weights. But the question that I want to ask is, is this an appropriate thing for me to use in my use case? And if I don't have access to the data, then I can't make that decision. And if I don't have access to information about how the data was collected, then from an ethical point of view, I can't make that decision. So I need the full set of information before I'm gonna call something open.
[00:53:45.242] Alex Hanna: Yeah, and I mean, I think Emily covered most of it. I mean, the other aspects of it are having access to not only the training data, also the reinforcement learning with human feedback data, having a data sheet, having other types of documentation, having the access to the training data itself allows there to be particular sorts of audits in terms of how secure a particular model is. Is there sufficient safeguards in terms of if there is PII in the training data, which is alarming, is it going to sufficiently not leak that? And there's some work from folks like Nicholas Carlini have showed that when he was looking at a model that had open training data called the pile. And so he was actually able to assess whether there was PII leakage, right? And you can't do that. And there's also these claims around things like emergence, which is very silly, of this idea that there's emergent capabilities. Like when Sundar Pichai claimed that, I think it was Gemini or Bard had learned Bengali when it had not. Bengali was in Common Crawl, the one of the datasets that we knew that the dataset had been using in the training data. And so we can actually assess those particular types of claims.
[00:54:56.779] Kent Bye: Hmm. Yeah, and so a lot of the questions around the model or some of the questions that you're asking, two of the other ones that I just wanted to have you maybe elaborate on are what is being automated and who benefits, as well as are these systems being described as human? And so as I was reading through the book and also looking at Johnny Penn's PhD thesis, There seems to be this idea of AI being a technology of power and the consolidation of power and who is benefiting from that consolidation of power. And so I'd love to have you just elaborate on as you're starting to think critically around these automation technologies, asking you these questions, and then how do you start to think around the power dynamics of them and why that's important?
[00:55:36.308] Alex Hanna: Do you want to talk about the asking of questions, Emily?
[00:55:38.350] Emily M. Bender: Sure, yeah. So it's important that these are questions that we ask, but they're questions that we're urging our readers to ask. So this isn't something that you have to have a PhD to be able to do. These are questions that anybody, when faced with some automation, can ask. And you might not get full answers, and sometimes the lack of an answer is enough to say that I'm not going to use it. So, you know, what is being automated if someone says to you, so there's this ridiculous company called Hippocratic AI that provides and putting scare quotes here, nurses with different specialties that are basically chat bots attached to a speech to text thing and in terms of the who benefits thing their advertising copy says that you can hire these in quotes nurses for nine dollars an hour and the idea that you have an hourly rate for an automated system is already absurd like that sort of is this anthropomorphizing move right but what's being automated here well on one level you might say communications with patients and it's all about like sort of pre-op and post care that they're trying to automate with this so at one level that's what they're attempting to automate but what they're actually automating like what's the input what's the output well the input is something the patient said and the output is some text that comes back out of the chat bot and then we can say okay is that really a good match for our use case well clearly not and if we're looking at it in that very hard nose what's the input what's the output mindset you can see how it is not in any sense a nurse and then who benefits well it's the people who are making money off of our medical system and would like to cut costs is who benefits there is it being described as human absolutely and one of these systems being described in human terms is a clear red flag that it is being oversold and so that's something i think is really easy for people to be on the lookout for
[00:57:21.308] Alex Hanna: And with regards to thinking about the different power dynamics of this, one of the aspects of the generality argument is that it gives us a sense that there's a tool that can do everything for everyone, which is certainly not what the case is. And especially when it comes to these models of this size, to be able to train these at scale and to have the improvement at scale, you have to have the ability to have huge amounts of compute, have huge amounts of data collection capacity, and then the ability to have the size to do inference at scale. And so that already is an argument basically that you need centralization to do this with any kind of, you know, to have the state of the art model, right? And so that already makes it a tool of power because the providers are the ones that only can do this on a technical level. But then there's the element of Especially when thinking about what it means to replace or to replace someone at their job. That means that the people who are controlling the technology then are the ones who are using it to automate away certain positions and certain kinds of human labor. And then that then becomes the central conceit of them. And so in some ways, LLMs are another form of bossware. Bossware tends to focus on tools of surveillance and tools of micromanagement. And this comes a little bit from a report that a coworker.org developed which is called little tech which focuses on many of these small surveillance companies but in this case this is mass you know kind of industrial automation in that sense that is meant to basically be a cudgel for labor and so that also is one place where power really is at play and so i think that's kind of in addition to the questions which i think are a very helpful individual tool they're also a very helpful collective tool to be used by collectivities or trying to push back against that imposition of power.
[00:59:24.588] Kent Bye: And the final question that I usually ask all of my interviewees around like the ultimate potential of all these kind of emerging technologies, but that kind of leads you into like kind of a tech solutionism or kind of an assumption that we are going to have these technologies. And I think a big point of your book is that we can't resist and that this is not inevitable, that there could be other pathways. And so I'd love to hear you maybe elaborate on your vision for as we move forward, what you think the most exalted potential for people kind of coming together in the having that resistance or if you see this more symbiotic of like using it in a way that is ethical or makes sense or that there may be more conditions that are there that would mean an ethical use of the technology and just serve your final reflecting thoughts on what your vision is for that in the future.
[01:00:11.807] Emily M. Bender: so i think we have to again unpack you said using it in a way that's ethical but it isn't one thing so i see no beneficial use case for synthetic text extruding machines on the other hand pattern matching at scale if it is done by people who have control over their own data or communities that have control over their own data that can be great and there's an example we like to point to in tehiku media which is an outfit in ateroa new zealand that produces language technology for the Te Reo Māori language, where the company doing this, Te Uku Media, sees itself as part of the community and in guardianship of the data, as opposed to extracting it from the community to make money off of it. And so that's great. They have been able to produce language tools that are beneficial in their language reclamation process. So there are certainly positive examples out there. But I also want to just quickly speak the idea of, you know, do we have a use case of people coming together to resist or use case a potential future
[01:01:07.310] Alex Hanna: i think absolutely yes and there's a lot we have to say in the book about both individual and collective modes of resistance yeah and i think just to put a finer point on it too i mean thinking about the ways in which people have really control over technologies and to think about alternative technological futures is very helpful i mean what's going to work for particular people we'd love to talk about tehiku media as this organization that it serves the maori people in eritrea and really thinking about the thing that i love about that is thinking about indigenous data sovereignty and the ways in which certain data can be used for training and models and certain data is not is to be respected and honored and whatnot. And so, you know, when I think really thinking about that and bringing that approach to technology as a whole, I think technology and what we're building is really nice. And so it's thinking about a lot of things that maybe get paper over or ignored and other types of movements like open source, which I think focuses you know, there's the distinction between, you know, free as in gratis versus free as in speech. And I mean, I think that also that is one approach to it, but then there's also really thinking about what it means to kind of honor the people who created the data or the people who were represented in the data as well, as well as keeping the compute in the hands of people who are doing that training and ensuring that that's all happening in a way that's respectful and for community.
[01:02:33.957] Kent Bye: Awesome. Well, Emily and Alex, thanks so much both for joining me here today on the podcast to unpack a lot more of your book, The AIcon. I think it does a lot of really great deconstruction of a lot of the unsubstantiated claims and the delusional nature of just the language, providing new alternative language, but also deconstructing the metaphors that we are using to talk around these technologies. And Yeah, I find myself still kind of using AI because I've been using it for so long, but I think really questioning how I use it and maybe being more specific around the technologies that I'm referring to rather than feeding into that larger AI hype by just even looking at the language that I'm using. Yeah, it's a real comprehensive look at a lot of these things. And I just really enjoyed reading not only a book, but all the comprehensive footnotes that you have. It's a whole other kind of journey to go on some of the rabbit holes that you have with referencing some of the scholarship and folks that are pushing back on this. And so, yeah, just really enjoyed not only reading through it, but also having that chance to have this conversation today. So thanks again for joining me here today on the podcast.
[01:03:32.491] Alex Hanna: Thank you. Thank you. And I'm glad that you are always reading the footnotes.
[01:03:38.778] Kent Bye: So that was Alex Hanna. She's a sociologist. And Emily M. Bender, she's a computational linguist. And they wrote a book called The AI Con, How to Fight Big Tech's Hype and Create the Future We Want. So I have a number of different takeaways about this interview is that, first of all, Well, I highly, highly recommend that you check out this book, The AI Con, just because there is a lot of hyperbolic claims that are being made. And I found this book really helpful to try to understand the methods that are being used to construct the hype, the ways that they're comparing the technology to humans and human replacements, which are devaluing of humans and reducing them down into like computational machines. beings, but also just the way of artificial intelligence is like an ill-defined term. Even when I went to the International Joint Conference for Artificial Intelligence and talking to AI researchers, one of the comments that was made to me was that there's a lot of technologies that for a while are artificial intelligence, and then once they mature or realize that it's not going to reach this kind of mythical place of breakthroughs of intelligence, then it just becomes a part of computer science and that it's no longer considered a part of AI research. there's always this ill-defined way in which that artificial intelligence as a term is kind of a marketing term that even from the founders was used to anthropomorphize the technology in a way that is to do this magic trick of making us think that technology is a lot more powerful than it actually is because it's leaning on our ways that we think that humans are acting and behaving. And I think with the language, Emily was pointing out a number of times of how there's a structure and a form of language, and then it's taking the form Without that deeper structure or meaning or relationality that's happening with what is the context and how is this relating to how these communities are interacting and what is the relational nature to this. So whenever you're like compressing down humans into these machines, then you start to think that, oh, well, you can just use these AI technologies to replace humans. And so there's this push towards automation, which has been around since like all of technologies have started. different ways that it could be automating or displacing human labor but here we have these systems that are basically taking all this data that is using human labor and then reducing that down and then using those same machines to replace that labor and it's doing it at such a scale that i have to really ask like the power dynamics there's all these billionaires who are like pushing these technologies but also all these limitations that we seem to be in this weird situation where people are either like true believers are not really critically questioning some of these different things, or they're overlooking some of these different harms or overlooking some of the limitations, or people who are on the other side who are just completely resisting it and fighting back and trying to fight against this narrative of it being inevitable. And I think that's a lot of the camp that both Emily and Alex are on but I think there's also like potential utility use cases with certain conditions where you're in right relationship to both the data and the process there is a way to ethically do this and they reference these indigenous communities are doing it to kind of reclaim their language or from the voices VR podcast I'm using whisper acts to be able to do automatic transcriptions there's probably a lot of other manual labor that I need to do just to kind of verify things but I do have like the time code that's in there so that if you do want to check the actual source, then you can click on the timecode and it'll play that little clip. But, you know, at the same time, just by lumping together all these different disparate technologies under the banner of artificial intelligence seems to build into the hype and create this allure that is probably not warranted that these things are actually just well-defined scoped contextual applications of tools rather than putting a term like intelligence onto it which we then project all these human qualities and anthropomorphize it and treat it like this totally magical god-like entity that is going to take over all of the earth I found it really fascinating to really break down the artificial and general intelligence through the lens of context, just because like privacy and identity is so connected to contextual domains. And so when you start to also look at intelligence, I think that there's certain intelligence that's also highly contextual. And so when you start to do like these artificial and general intelligence, then you're basically trying to strip out all those different contextual specificity and create these things that are not well scoped, not well defined. You don't really know how to measure it or even know if you've achieved it or not. And so you've got this mentality where you've got these benchmarks that already have a lot of problems and limitations for, you know, you create a benchmark. So what's to prevent a large language model from changing? taking all the data that was used to create that benchmark and to include it into the model, then it's just repeating that information rather than actually showing any sort of understanding. And so with all these things, there's not like a lot of transparency for these types of researchers to actually evaluate these independently. And so you end up having these closed circular systems where they're checking themselves against these benchmarks, throwing out these numbers saying we're number one, and then giving you the sense that there is some actual scientific rigor that's happening behind this, but it ends up being like very thin when you start to really unpack some of the different citational practices and research that's behind it. And that's a lot of what you get through reading this book and reading through the footnotes is to kind of deconstruct all of this magic trip of AI that's happening right now. So I've done like 122 interviews with AI researchers back at the International Joint Conference of Artificial Intelligence back in 2016, 2018. I went back and I've been editing a lot of those and I'm hoping to relaunch those at some point. But it's helpful for me to have conversations like this just to help get a framing and a grounding. I'm really taking away this framing of power and automation and how AI is really a technology of power and how it consolidates power. That's a little of the lens that we have to look at it right now, especially when you start to see this type of legislation where government entities that are proposing legislation to put a 10 year moratorium on states being able to regulate this thing. You basically are creating a system for all these technology companies to have like a mitigated power. And it's that consolidation of that power. So I think we really have to look critically at the larger context under which this is happening and the colonial nature of like seizing all this data and not having the right relationship and yeah all the human labor that's there that is being reflected within these and so there's like all these claims around emergent properties but we don't have like the transparency of these data sets then you know like the example that the google ceo sundar prinshai is basically saying that you know this automatically kind of just learned bengali but yet at the same time the large language model used data sets that included the bengali language so it didn't just like automatically just learn it it's actually just trained on the data so These are the types of things where you kind of have these anthropomorphized intelligence and really when you look at it, there's some gaps and holes in there. So anyway, like I said, this AI at the point is like this hype cycle that continues to persist and I expect it to continue to persist where you have this kind of factionalized like true believers versus people who are critics. And within the XR community, there is quite a lot of people that are using the technologies and using them in bounded use cases that are I'd say have some utility, but I think the challenge is, like we were talking about in this conversation, where how do we really look at weighing that against both environmental costs, the other harms, the data later harms, the ways that this is consolidating power. My concern is around surveillance capitalism and ways that these AI technologies may be used to continue to gather lots of psychographic information around us? And then how does that personal identifiable information, as it gets fed into these AI models, how does it keep from being leaked out or remembered or violating our contextual privacy? So there's a whole paper that I wrote around the problems with contextually aware AI as it's being combined with all these XR technologies. So definitely go check out the book, The AI Con by Emily M. Bender and Alex Hanna. And I'm actually going to be at Augmented World Expo on a panel discussion in the Socratic dialogue, arguing against artificial intelligence. I'm going to try to carry the spirit of a lot of these kind of more skeptical views of AI, but also more of the perspectives of the AI researchers, which, you know, there's kind of a machine learning approach where you're just doing this kind of empirical approach. But there are other types of AI algorithms, whether it's like the constraints and planning knowledge representation and reasoning there's robotics along with all these other machine learning and deep reinforcement learning deep learning you know all these things are also there as well and so the international joint conference of ai is trying to look at these holistically and so the philosophy of deep learning discussions that were happening were you know really deconstructing that like these large language models on their own really aren't enough that you really need these other types of cognitive architectures to have more sophisticated ways of dealing with reasoning and understanding and other modeling of the world that goes beyond what we're perceiving the large language models to be able to doing. But as Emily calls them, these synthetic text extruding machines, they're not really capable of doing the full extent of what people are saying they're able to do. So anyway, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
