49 ms·

What Emily Bender meant by "stochastic parrots"

177 points by digital55 3mo ago

deleted 3mo ago
[deleted]
noduerme 3mo ago
Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
- DanielHB 3mo ago
  Pattern matching machines seems more appropriate.
  - diego_sandoval 3mo ago
    For humans?
  - otabdeveloper4 3mo ago
    LLMs do not match patterns. They predict one statistically most likely token (only one!) given a history of some N previously known tokens.
    - beardedwizard 3mo ago
      Is that prediction not based on matching previous patterns, whose frequencies are more or less encoded in the weights?
      - snapcaster 3mo ago
        you're really reaching for no apparent reason. Just move on from pattern matching machines it's not a good mental model for LLMs
    - lennoff 3mo ago
      afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
      - tsimionescu 3mo ago
        I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
    - lioeters 3mo ago
      > statistically most likely Isn't that pattern matching essentially?
- bunderbunder 3mo ago
  Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
  - beardedwizard 3mo ago
    Explain it to me
  - jeremyjh 3mo ago
    Which direction is the correlation? I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written. After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features. There has been a lot of subsequent research that really calls many of the statements in this article into question.
- root-parent 3mo ago
  Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI. Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding. [1] https://chrisclay.substack.com/p/what-is-superposition-in-neural-networks https://chrisclay.substack.com/p/what-is-superposition-in-ne...
  - marshray 3mo ago
    What is "real understanding", and what question can we ask ChatGPT to determine whether it has it?
    - noduerme 3mo ago
      Ask it how to prevent Spotify from automatically playing every time you get in your car. The answer will involve a bunch of Android settings that don't actually exist, cobbled together from a bunch of bad advice in online forums. Explain to it how it's wrong. Then clear your cache and ask it the same question again from scratch, and get the same garbage. Repeat until it's clear that it doesn't understand anything.
      - marshray 3mo ago
        OK, so I asked ChatGPT "How do I prevent Spotify from automatically playing every time I get in my car?" The answer looks entirely reasonable. I can't check the Android settings, but I was able to confirm that all of the suggested iPhone and Spotify and settings exist. Most of them I had already configured that way, some I didn't know about before.
- snerbles 3mo ago
  The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones. Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
- dkdbejwi383 3mo ago
  What's wrong with "large language model"?
  - cwillu 3mo ago
    Seems like a lot of people are upset about other people calling both apples and oranges “fruit”.
  - marshray 3mo ago
    Naming things using qualifiers like "large" has never aged well when transistors were involved. For example, consider the term "short wave" radio which refers to wavelengths of at least 10 meters. Today's mobile communications use wavelengths 100x - 10,000x shorter.
- scarmig 3mo ago
  Which frame inspires a more productive research program? Which has better predicted the trajectory of capabilities over the past five years?
  - PaulDavisThe1st 3mo ago
    There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
    - scarmig 3mo ago
      There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development. But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020? The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
  - Planktonne 3mo ago
    > Which has better predicted the trajectory of capabilities over the past five years? By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god. People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
- amiga386 3mo ago
  Spicy autocomplete
- fxwin 3mo ago
  I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts). It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence. (fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
  - xgulfie 3mo ago
    Understand is a pretty imprecise term. What does it mean for a computer to understand? Does an H264 decoder understand Eraserhead.mkv?
    - fxwin 3mo ago
      Sure, but that's the nature of language (which is also why i put "understand" in quotation marks. I usually follow it up with "whatever that means" lol) . I think in this case, it carries with it implicit properties (abstract semantic representation) that i think models possess, which a decoder doesn't.
- paulcole 3mo ago
  > Stochastic Parrot Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it. So it's a great phrase because in just about every case I can ignore what someone says afterwards. Similar to "glorified autocomplete."
  - andrewla 3mo ago
    At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial. From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective. I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
    - paulcole 3mo ago
      > one could say that all of physics and creation is "glorified autocomplete" Exhibit A.
    - delis-thumbs-7e 3mo ago
      That’s not how language works https://www.telelib.com/authors/J/JoyceJames/prose/finneganswake/finneganswake_0101.html https://www.telelib.com/authors/J/JoyceJames/prose/finnegans...
      - andrewla 3mo ago
        I dunno, man, I looked at that text and I see one word after another. Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
- delis-thumbs-7e 3mo ago
  This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things. What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are. [1] Coined by Marvin Minsky: https://www.thekurzweillibrary.com/consciousness-is-a-big-suitcase https://www.thekurzweillibrary.com/consciousness-is-a-big-su...
bubbi 3mo ago
[dead]
jongjong 3mo ago
The term is not very useful since most humans are stochastic parrots... At least most of the time. Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
- lennoff 3mo ago
  i think it actually makes sense, an LLM just imitates human communication, which happens to be useful from time to time.
- elysianfields 3mo ago
  It sometimes feels same as with the models, especially in corporate: - Lots of Haiku around, many mistakes unless process is very clear - Some Sonnets, still do mistakes but can adapt - Some Opus, able to improvise and think outside the box. But even the Human Opus/Mythos are hilariously wrong sometimes.
- GolfPopper 3mo ago
  Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
  - jeremyjh 3mo ago
    Nothing in that term implies sentience.
- TonyAlicea10 3mo ago
  > most humans are stochastic parrots There's a lot more happening behind the scenes when a human repeats phrases than what's happening in an LLM. Sociological phenomenon. The desire to be liked, successful, or popular. The feeling that those phrases brings up. LLMs are not experiencing any of that. As far as we know, neither is a parrot.
baggy_trough 3mo ago
'Stochastic parrots' is a great term, but reading it now, it's quite apparent how bad this paper is.
SpicyLemonZest 3mo ago
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension. I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
- Diogenesian 3mo ago
  This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
  - throwaway7356 3mo ago
    > yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability Where can I access such a Lisp expert system? If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
- PaulDavisThe1st 3mo ago
  > But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
crancher 3mo ago
[flagged]
iwontberude 3mo ago
I respect you and parrots, please don’t use parrots as an insult.
- angry_octet 3mo ago
  Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed. So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
Aurornis 3mo ago
> in part because Google fired two of the authors, Timnit Gebru I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since. The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
- utopiah 3mo ago
  True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise. She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
  - Aurornis 3mo ago
    I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
    - marshray 3mo ago
      Never underestimate the power of a catchy title that resonates with the intuitions and preconceptions of people who will never read the paper. Cf. https://machinelearning.apple.com/research/illusion-of-thinking https://machinelearning.apple.com/research/illusion-of-think...
  - simianwords 3mo ago
    Timnit got popular because she was part of Woke 1.
platypii 3mo ago
[flagged]
petergs 3mo ago
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it). In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems". > it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
delis-thumbs-7e 3mo ago
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at. On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-here-are-eight-examples-how.html https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...
- genxy 3mo ago
  The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus. Now if they had said, "Imagine your average American ..." (/s)
- Sharlin 3mo ago
  It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time. Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
  - rob74 3mo ago
    It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
  - andrewflnr 3mo ago
fsckboy 3mo ago
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
- bigstrat2003 3mo ago
  It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
  - DangitBobby 3mo ago
    It turns out that determinism isn't what makes computers useful in the first place.
  - naasking 3mo ago
    1. Determinism is a very small subset of what makes computers useful. Non-determinism like stochasticity is literally everywhere, like random seeds. 2. LLMs are detemrinistic. They have a parameter to tune how stochastic they are.
    - TacticalCoder 3mo ago
      Sorry at first I downvoted you instead of upvoting you.
  - TacticalCoder 3mo ago
    What do you mean by determinism here? That you ask the computer 2+2 and it gives 4 as you expected or that if you ask the computer 2+2 and it hallucinates 5, you want it to always hallucinate 5? Which one it doesn't do for you? Does it sometimes answer 4, sometimes 5? There are definitely models that will always give 100% of the time the exact same answer, bit-for-bit, given the same input and seed. There are generative image models you can run locally doing just that. But you can also run some the SOTA chinese LLMs at "temperature 0" and, given the same input, they'll always give you the exact same output. Because it's just a machine doing computation. In the beginning of LLMs some "engineers" have tried to hand-wave non-sensical explanation as to why LLMs couldn't possibly be deterministic but: the open-weights models that can be run in a 100% deterministic way are way more powerful than the SOTA models of back then, so those explanation were pure rubbish bollocks. Now of course if you run a complex chain of events, with LLMs doing calls to other LLMs, where some of them go fetch infos on unreliable networks, with infos that may have changed, then, logically, you won't always get the same answer.
themgt 3mo ago
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do > Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives. This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer. The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
ayhanfuat 3mo ago
Here is what Jeff Dean said about the firing at the time: https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQqz7ScqxhSIxeYGrWjK0/preview?pru=AAABdlOOKBs*gTzLnuI53B2IS2BISVcgAQ&tab=t.0#heading=h.aplcvu32myqt https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQ...
- f33d5173 3mo ago
  > resignation I appeciate short letters like this that get straight to the point...
- viccis 3mo ago
  She was probably repeating behaviors she learned in academia. These kinds of extremely toxic "don't just apologize for disagreeing with me but also give me the name of every person involved for collective punishment" is a classic move for academic tyrants throwing weight around. The understanding there is that they will then move to cut off every named person from power or access to academic resources. Google did the right thing by protecting their people. I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure. edit: And it looks like Mitchell immediately left an angry comment about being called a "linguist" despite the article never doing so. Starting to notice a pattern of confrontation, ego, and escalation among some of these people.
- guelo 3mo ago
  You can always count on management to tell it like it is /s
dwa3592 3mo ago
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
- CamperBob2 3mo ago
  Those costs have to be compared to the way things are currently done without AI. They never are. Ever.
  - supern0va 3mo ago
    >They never are. Ever. And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time. It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop. The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet. It's slightly maddening.
- azakai 3mo ago
  My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation. It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways! Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
dekhn 3mo ago
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community. I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
andrewla 3mo ago
For context, here's the main quote: > Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text. Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text. This I think is clarifying, from the same paragraph in the text: > ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
- SAI_Peregrinus 3mo ago
  I don't accept that it applies to parrots. Certainly not to Congo African Grey parrots.
wolttam 3mo ago
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is". I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context. What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it. What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference. I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
- Diogenesian 3mo ago
  "Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
  - josh-sematic 3mo ago
    > Transformer ANNs are dramatically dumber than cockroaches Source?
    - Diogenesian 3mo ago
      My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects. I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
NyxWulf 3mo ago
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
- gpderetta 3mo ago
  They are just token generators. It is just that 'just' does a lot of lifting!
- AlexandrB 3mo ago
  Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen. Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition. > The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly. Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.
waffletower 3mo ago
> when OpenAI imposed ChatGPT on the world... OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
- magicmicah85 3mo ago
  I agree with a lot of her points but that word really is revealing of her thoughts about OpenAI.
- delis-thumbs-7e 3mo ago
  > A large, monied cross-section of the world I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
  - waffletower 3mo ago
    $20.3B is far from the trillion+ investments that are causing enormous societal contention now.
- tootie 3mo ago
  I think "imposed" is a pretty fair word. LLMs already affect everyone's lives whether you use them or not and their footprint is still growing. It's like I don't have to buy a car, but cars are absolutely imposed on everyone anyway.
softwaredoug 3mo ago
I don’t see a problem with the “stochastic parrot” label. It just turns out stochastic parrots are incredibly useful. At a minimum it’s probably more accurate than “AI”.
- TaupeRanger 3mo ago
  Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
  - delis-thumbs-7e 3mo ago
    Why is it not indicative of what LLM’s do?
    - TaupeRanger 3mo ago
      Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
      - delis-thumbs-7e 3mo ago
        Yes, I am: https://en.wikipedia.org/wiki/Alex_(parrot) https://en.wikipedia.org/wiki/Alex_(parrot)
      - softwaredoug 3mo ago
        The "stochastic" is the key modifier to "parrot". > "Stochastic" means having a random pattern or variable that can be analyzed statistically but not predicted precisely
    - naasking 3mo ago
      Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?
skeledrew 3mo ago
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
- dekhn 3mo ago
  Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
  - skeledrew 3mo ago
    Useful for regurgitative pattern generation I can get. The core definition though is it doesn't understand what it's generating. Meanwhile I'm seeing LLMs constantly perform tasks which I'd say requires understanding, like reading the manual for a tool it's never encountered before and then going on to effectively use that tool. That's 2 different kinds of useful.
    - dekhn 3mo ago
      I'm of the general belief that something can be unaware and not have any understanding (in the subjective experience of consciousness), while also appearing to do so, and be useful. That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems). I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.
blululu 3mo ago
This all sounds like a lot of backpedaling and “well actually” kind of stuff. “Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”. Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here. This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
- theragra 3mo ago
  Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
  - supern0va 3mo ago
    Honestly, I'm puzzled that people don't intuit that "understanding" itself is merely ideas in relation to other ideas.
lowbloodsugar 3mo ago
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
SonOfLilit 3mo ago
[dead]
mediaman 3mo ago
Bender's paper had this to say about stochastic parrots: "Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot." This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot. But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment. At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.) RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper. RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models. So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
- emtel 3mo ago
  Even if pre-training was the only training step, it still wouldn't necessarily follow that the only thing the model is doing is stitching words together probabilistically, unless you expand the definition of "probabilistically" to the point that it becomes meaningless. This kind of thinking assumes that design of the training process and the "design" of the artifact that training produces must be similar.
troelsSteegin 3mo ago
Bender's linked May 12, 2026 post "Frequently Unasked Questions", https://medium.com/@emilymenonbender/stochastic-parrots-frequently-unasked-questions-49c2e7d22d11 https://medium.com/@emilymenonbender/stochastic-parrots-freq... , was a better read.
benlivengood 3mo ago
What I look forward to after research like https://arxiv.org/abs/2603.02491 https://arxiv.org/abs/2603.02491, which demonstrate the necessity of world-modeling capability to achieve satisfactory performance on certain goals, is a refractor the SoTA test suites to demonstrate how much world-modeling is necessary in various task distributions. There have been a few years now of arguments about the level to which transformers do or do not have a world model (v.s. being purely stochastic parrots like early pre-trained LLMs) and now we have some tools to actually make quantifiable determinations.
- thomastjeffery 3mo ago
  But the stochastic parrot (LLM) is the world model, isn't it? What's the difference?
  - simianwords 3mo ago
    Yeah… LLMs clearly already have a world model
    - thomastjeffery 3mo ago
      I think it's a good distinction to make between having and being, which seems to be what the whole "stochastic parrots" bit was intended to make all along. It doesn't make sense to say a model is in possession of its self. That's exactly the sort of poetic anthropomorphization that Bender was criticising here, and a good reason to not refer to an LLM as "an AI".
      - simianwords 3mo ago
        Wow this is exactly kind of pedanticism that annoys me with Bender. Glad that this kinda thing is becoming unpopular.
        thomastjeffery 3mo ago
        There is more here then pedantry, but if you aren't interested in it, then go right ahead and live your life.
a34729t 3mo ago
If your field (NLP/computational lingustics) has evaporated the next best thing to do is hit the pundit circuit.
- kilpikaarna 3mo ago
  No you see, she says right there in here "Frequently Unasked Questions" (lol) that she is in fact not at all bitter about her field being effectively solved by someone else, thank you very much! Her preferred approach is in fact alive and well, more important than ever and still very very interesting. One wonders if her PhD students feel the same. The authors of the paper make some good points (in it and elsewhere), but I've seen precisely nothing that suggests they aren't awful people who resort to playing the victim card, inciting cancel mobs and in general behaving like the worst stereotypes of "leftist academia" in response to any criticism.
losvedir 3mo ago
She discusses it more in this debate (?) discussion (?): https://www.youtube.com/watch?v=MwfSCCo6jXs https://www.youtube.com/watch?v=MwfSCCo6jXs . I found it interesting but ultimately thought much less of her and her argument afterwards.
stefffff 3mo ago
[flagged]

177 points by digital55 3mo ago

deleted 3mo ago

[deleted]

Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.

DanielHB 3mo ago
Pattern matching machines seems more appropriate.
- diego_sandoval 3mo ago
  For humans?
- otabdeveloper4 3mo ago
  LLMs do not match patterns. They predict one statistically most likely token (only one!) given a history of some N previously known tokens.
  - beardedwizard 3mo ago
    Is that prediction not based on matching previous patterns, whose frequencies are more or less encoded in the weights?
    - snapcaster 3mo ago
      you're really reaching for no apparent reason. Just move on from pattern matching machines it's not a good mental model for LLMs
  - lennoff 3mo ago
    afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
    - tsimionescu 3mo ago
      I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
  - lioeters 3mo ago
    > statistically most likely Isn't that pattern matching essentially?
bunderbunder 3mo ago
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
- beardedwizard 3mo ago
  Explain it to me
- jeremyjh 3mo ago
  Which direction is the correlation? I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written. After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features. There has been a lot of subsequent research that really calls many of the statements in this article into question.
root-parent 3mo ago
Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI. Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding. [1] https://chrisclay.substack.com/p/what-is-superposition-in-neural-networks https://chrisclay.substack.com/p/what-is-superposition-in-ne...
- marshray 3mo ago
  What is "real understanding", and what question can we ask ChatGPT to determine whether it has it?
  - noduerme 3mo ago
    Ask it how to prevent Spotify from automatically playing every time you get in your car. The answer will involve a bunch of Android settings that don't actually exist, cobbled together from a bunch of bad advice in online forums. Explain to it how it's wrong. Then clear your cache and ask it the same question again from scratch, and get the same garbage. Repeat until it's clear that it doesn't understand anything.
    - marshray 3mo ago
      OK, so I asked ChatGPT "How do I prevent Spotify from automatically playing every time I get in my car?" The answer looks entirely reasonable. I can't check the Android settings, but I was able to confirm that all of the suggested iPhone and Spotify and settings exist. Most of them I had already configured that way, some I didn't know about before.
snerbles 3mo ago
The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones. Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
dkdbejwi383 3mo ago
What's wrong with "large language model"?
- cwillu 3mo ago
  Seems like a lot of people are upset about other people calling both apples and oranges “fruit”.
- marshray 3mo ago
  Naming things using qualifiers like "large" has never aged well when transistors were involved. For example, consider the term "short wave" radio which refers to wavelengths of at least 10 meters. Today's mobile communications use wavelengths 100x - 10,000x shorter.
scarmig 3mo ago
Which frame inspires a more productive research program? Which has better predicted the trajectory of capabilities over the past five years?
- PaulDavisThe1st 3mo ago
  There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
  - scarmig 3mo ago
    There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development. But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020? The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
- Planktonne 3mo ago
  > Which has better predicted the trajectory of capabilities over the past five years? By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god. People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
amiga386 3mo ago
Spicy autocomplete
fxwin 3mo ago
I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts). It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence. (fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
- xgulfie 3mo ago
  Understand is a pretty imprecise term. What does it mean for a computer to understand? Does an H264 decoder understand Eraserhead.mkv?
  - fxwin 3mo ago
    Sure, but that's the nature of language (which is also why i put "understand" in quotation marks. I usually follow it up with "whatever that means" lol) . I think in this case, it carries with it implicit properties (abstract semantic representation) that i think models possess, which a decoder doesn't.
paulcole 3mo ago
> Stochastic Parrot Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it. So it's a great phrase because in just about every case I can ignore what someone says afterwards. Similar to "glorified autocomplete."
- andrewla 3mo ago
  At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial. From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective. I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
  - paulcole 3mo ago
    > one could say that all of physics and creation is "glorified autocomplete" Exhibit A.
  - delis-thumbs-7e 3mo ago
    That’s not how language works https://www.telelib.com/authors/J/JoyceJames/prose/finneganswake/finneganswake_0101.html https://www.telelib.com/authors/J/JoyceJames/prose/finnegans...
    - andrewla 3mo ago
      I dunno, man, I looked at that text and I see one word after another. Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
delis-thumbs-7e 3mo ago
This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things. What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are. [1] Coined by Marvin Minsky: https://www.thekurzweillibrary.com/consciousness-is-a-big-suitcase https://www.thekurzweillibrary.com/consciousness-is-a-big-su...

bubbi 3mo ago

[dead]

jongjong 3mo ago

The term is not very useful since most humans are stochastic parrots... At least most of the time. Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.

lennoff 3mo ago
i think it actually makes sense, an LLM just imitates human communication, which happens to be useful from time to time.
elysianfields 3mo ago
It sometimes feels same as with the models, especially in corporate: - Lots of Haiku around, many mistakes unless process is very clear - Some Sonnets, still do mistakes but can adapt - Some Opus, able to improvise and think outside the box. But even the Human Opus/Mythos are hilariously wrong sometimes.
GolfPopper 3mo ago
Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
- jeremyjh 3mo ago
  Nothing in that term implies sentience.
TonyAlicea10 3mo ago
> most humans are stochastic parrots There's a lot more happening behind the scenes when a human repeats phrases than what's happening in an LLM. Sociological phenomenon. The desire to be liked, successful, or popular. The feeling that those phrases brings up. LLMs are not experiencing any of that. As far as we know, neither is a parrot.

baggy_trough 3mo ago

'Stochastic parrots' is a great term, but reading it now, it's quite apparent how bad this paper is.

SpicyLemonZest 3mo ago

> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension. I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.

Diogenesian 3mo ago
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
- throwaway7356 3mo ago
  > yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability Where can I access such a Lisp expert system? If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
PaulDavisThe1st 3mo ago
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do. So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".

crancher 3mo ago

[flagged]

iwontberude 3mo ago

I respect you and parrots, please don’t use parrots as an insult.

angry_octet 3mo ago
Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed. So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.

Aurornis 3mo ago

> in part because Google fired two of the authors, Timnit Gebru I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since. The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.

utopiah 3mo ago
True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise. She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
- Aurornis 3mo ago
  I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
  - marshray 3mo ago
    Never underestimate the power of a catchy title that resonates with the intuitions and preconceptions of people who will never read the paper. Cf. https://machinelearning.apple.com/research/illusion-of-thinking https://machinelearning.apple.com/research/illusion-of-think...
- simianwords 3mo ago
  Timnit got popular because she was part of Woke 1.

platypii 3mo ago

[flagged]

petergs 3mo ago

I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it). In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems". > it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.

delis-thumbs-7e 3mo ago

> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at. On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-here-are-eight-examples-how.html https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...

genxy 3mo ago
The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus. Now if they had said, "Imagine your average American ..." (/s)
Sharlin 3mo ago
It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time. Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
- rob74 3mo ago
  It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
- andrewflnr 3mo ago

fsckboy 3mo ago

it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)

bigstrat2003 3mo ago
It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
- DangitBobby 3mo ago
  It turns out that determinism isn't what makes computers useful in the first place.
- naasking 3mo ago
  1. Determinism is a very small subset of what makes computers useful. Non-determinism like stochasticity is literally everywhere, like random seeds. 2. LLMs are detemrinistic. They have a parameter to tune how stochastic they are.
  - TacticalCoder 3mo ago
    Sorry at first I downvoted you instead of upvoting you.
- TacticalCoder 3mo ago
  What do you mean by determinism here? That you ask the computer 2+2 and it gives 4 as you expected or that if you ask the computer 2+2 and it hallucinates 5, you want it to always hallucinate 5? Which one it doesn't do for you? Does it sometimes answer 4, sometimes 5? There are definitely models that will always give 100% of the time the exact same answer, bit-for-bit, given the same input and seed. There are generative image models you can run locally doing just that. But you can also run some the SOTA chinese LLMs at "temperature 0" and, given the same input, they'll always give you the exact same output. Because it's just a machine doing computation. In the beginning of LLMs some "engineers" have tried to hand-wave non-sensical explanation as to why LLMs couldn't possibly be deterministic but: the open-weights models that can be run in a 100% deterministic way are way more powerful than the SOTA models of back then, so those explanation were pure rubbish bollocks. Now of course if you run a complex chain of events, with LLMs doing calls to other LLMs, where some of them go fetch infos on unreliable networks, with infos that may have changed, then, logically, you won't always get the same answer.

themgt 3mo ago

What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do > Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives. This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer. The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?

ayhanfuat 3mo ago

Here is what Jeff Dean said about the firing at the time: https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQqz7ScqxhSIxeYGrWjK0/preview?pru=AAABdlOOKBs*gTzLnuI53B2IS2BISVcgAQ&tab=t.0#heading=h.aplcvu32myqt https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQ...

f33d5173 3mo ago
> resignation I appeciate short letters like this that get straight to the point...
viccis 3mo ago
She was probably repeating behaviors she learned in academia. These kinds of extremely toxic "don't just apologize for disagreeing with me but also give me the name of every person involved for collective punishment" is a classic move for academic tyrants throwing weight around. The understanding there is that they will then move to cut off every named person from power or access to academic resources. Google did the right thing by protecting their people. I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure. edit: And it looks like Mitchell immediately left an angry comment about being called a "linguist" despite the article never doing so. Starting to notice a pattern of confrontation, ego, and escalation among some of these people.
guelo 3mo ago
You can always count on management to tell it like it is /s

dwa3592 3mo ago

I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".

CamperBob2 3mo ago
Those costs have to be compared to the way things are currently done without AI. They never are. Ever.
- supern0va 3mo ago
  >They never are. Ever. And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time. It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop. The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet. It's slightly maddening.
azakai 3mo ago
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation. It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways! Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.

dekhn 3mo ago

Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community. I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.

andrewla 3mo ago

For context, here's the main quote: > Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text. Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text. This I think is clarifying, from the same paragraph in the text: > ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?

SAI_Peregrinus 3mo ago
I don't accept that it applies to parrots. Certainly not to Congo African Grey parrots.

wolttam 3mo ago

I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is". I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context. What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it. What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference. I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).

Diogenesian 3mo ago
"Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
- josh-sematic 3mo ago
  > Transformer ANNs are dramatically dumber than cockroaches Source?
  - Diogenesian 3mo ago
    My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects. I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.

NyxWulf 3mo ago

After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.

gpderetta 3mo ago
They are just token generators. It is just that 'just' does a lot of lifting!
AlexandrB 3mo ago
Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen. Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition. > The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly. Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.

waffletower 3mo ago

> when OpenAI imposed ChatGPT on the world... OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.

magicmicah85 3mo ago
I agree with a lot of her points but that word really is revealing of her thoughts about OpenAI.
delis-thumbs-7e 3mo ago
> A large, monied cross-section of the world I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
- waffletower 3mo ago
  $20.3B is far from the trillion+ investments that are causing enormous societal contention now.
tootie 3mo ago
I think "imposed" is a pretty fair word. LLMs already affect everyone's lives whether you use them or not and their footprint is still growing. It's like I don't have to buy a car, but cars are absolutely imposed on everyone anyway.

softwaredoug 3mo ago

I don’t see a problem with the “stochastic parrot” label. It just turns out stochastic parrots are incredibly useful. At a minimum it’s probably more accurate than “AI”.

TaupeRanger 3mo ago
Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
- delis-thumbs-7e 3mo ago
  Why is it not indicative of what LLM’s do?
  - TaupeRanger 3mo ago
    Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
    - delis-thumbs-7e 3mo ago
      Yes, I am: https://en.wikipedia.org/wiki/Alex_(parrot) https://en.wikipedia.org/wiki/Alex_(parrot)
    - softwaredoug 3mo ago
      The "stochastic" is the key modifier to "parrot". > "Stochastic" means having a random pattern or variable that can be analyzed statistically but not predicted precisely
  - naasking 3mo ago
    Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?

skeledrew 3mo ago

Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?

dekhn 3mo ago
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
- skeledrew 3mo ago
  Useful for regurgitative pattern generation I can get. The core definition though is it doesn't understand what it's generating. Meanwhile I'm seeing LLMs constantly perform tasks which I'd say requires understanding, like reading the manual for a tool it's never encountered before and then going on to effectively use that tool. That's 2 different kinds of useful.
  - dekhn 3mo ago
    I'm of the general belief that something can be unaware and not have any understanding (in the subjective experience of consciousness), while also appearing to do so, and be useful. That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems). I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.

blululu 3mo ago

This all sounds like a lot of backpedaling and “well actually” kind of stuff. “Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”. Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here. This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.

theragra 3mo ago
Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
- supern0va 3mo ago
  Honestly, I'm puzzled that people don't intuit that "understanding" itself is merely ideas in relation to other ideas.

lowbloodsugar 3mo ago

Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.

SonOfLilit 3mo ago

[dead]

mediaman 3mo ago

Bender's paper had this to say about stochastic parrots: "Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot." This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot. But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment. At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.) RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper. RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models. So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.

emtel 3mo ago
Even if pre-training was the only training step, it still wouldn't necessarily follow that the only thing the model is doing is stitching words together probabilistically, unless you expand the definition of "probabilistically" to the point that it becomes meaningless. This kind of thinking assumes that design of the training process and the "design" of the artifact that training produces must be similar.

troelsSteegin 3mo ago

Bender's linked May 12, 2026 post "Frequently Unasked Questions", https://medium.com/@emilymenonbender/stochastic-parrots-frequently-unasked-questions-49c2e7d22d11 https://medium.com/@emilymenonbender/stochastic-parrots-freq... , was a better read.

benlivengood 3mo ago

What I look forward to after research like https://arxiv.org/abs/2603.02491 https://arxiv.org/abs/2603.02491, which demonstrate the necessity of world-modeling capability to achieve satisfactory performance on certain goals, is a refractor the SoTA test suites to demonstrate how much world-modeling is necessary in various task distributions. There have been a few years now of arguments about the level to which transformers do or do not have a world model (v.s. being purely stochastic parrots like early pre-trained LLMs) and now we have some tools to actually make quantifiable determinations.

thomastjeffery 3mo ago
But the stochastic parrot (LLM) is the world model, isn't it? What's the difference?
- simianwords 3mo ago
  Yeah… LLMs clearly already have a world model
  - thomastjeffery 3mo ago
    I think it's a good distinction to make between having and being, which seems to be what the whole "stochastic parrots" bit was intended to make all along. It doesn't make sense to say a model is in possession of its self. That's exactly the sort of poetic anthropomorphization that Bender was criticising here, and a good reason to not refer to an LLM as "an AI".
    - simianwords 3mo ago
      Wow this is exactly kind of pedanticism that annoys me with Bender. Glad that this kinda thing is becoming unpopular.
      - thomastjeffery 3mo ago
        There is more here then pedantry, but if you aren't interested in it, then go right ahead and live your life.

a34729t 3mo ago

If your field (NLP/computational lingustics) has evaporated the next best thing to do is hit the pundit circuit.

kilpikaarna 3mo ago
No you see, she says right there in here "Frequently Unasked Questions" (lol) that she is in fact not at all bitter about her field being effectively solved by someone else, thank you very much! Her preferred approach is in fact alive and well, more important than ever and still very very interesting. One wonders if her PhD students feel the same. The authors of the paper make some good points (in it and elsewhere), but I've seen precisely nothing that suggests they aren't awful people who resort to playing the victim card, inciting cancel mobs and in general behaving like the worst stereotypes of "leftist academia" in response to any criticism.

losvedir 3mo ago

She discusses it more in this debate (?) discussion (?): https://www.youtube.com/watch?v=MwfSCCo6jXs https://www.youtube.com/watch?v=MwfSCCo6jXs . I found it interesting but ultimately thought much less of her and her argument afterwards.

stefffff 3mo ago

[flagged]