15 ms·

Bag of words, have mercy on us

328 points by ntnbr 10mo ago

palata 10mo ago
Slightly unfortunate that "Bag of words" is already a different concept: https://en.wikipedia.org/wiki/Bag_of_words https://en.wikipedia.org/wiki/Bag_of_words. My second thought is that it's not the metaphor that is misleading. People have been told thousands of times that LLMs don't "think", don't "know", don't "feel", but are "just a very impressive autocomplete". If they still really want to completely ignore that, why would they suddenly change their mind with a new metaphor? Humans are lazy. If it looks true enough and it cost less effort, humans will love it. "Are you sure the LLM did your job correctly?" is completely irrelevant: people couldn't care less if it's correct or not. As long as the employer believes that the employee is "doing their job", that's good enough. So the question is really: "do you think you'll get fired if you use this?". If the answer is "no, actually I may even look more productive to my employer", then why would people not use it?
- kaycebasques 10mo ago
  > Slightly unfortunate that "Bag of words" is already a different concept Yes, subconsciously I kept trying to map this article's ideas to word2vec and continuous-bag-of-words.
viccis 10mo ago
Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more. That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.
- roxolotl 10mo ago
  Yea bag of words isn’t helpful at all. I really do think that “superpowered sentence completion” is the best description. Not only is it reasonably accurate it is understandable, everyone has seen autocomplete function, and it’s useful. I don’t know how to “use” a bag of words. I do know how to use sentence completion. It also helps explains why context matters.
  - domador 10mo ago
    I've been recently using a similar description, referring to "AI" (LLMs) as "glorified autocomplete" or "luxury autocomplete".
    - eichin 10mo ago
      I think I first heard "spicy autocomplete" two or three years ago...
  - visarga 10mo ago
    Sentence completion does not give it justice, when I can ask a LLM to refactor my repo and come back half an hour later to see the deed done.
    - xtracto 10mo ago
      Thats the thing, when you use an Ask/answer mechanism, you are just writing a "novel" where User: asks and personal coding assistant: answers. But all the text goes into the autocomplete function and the "toaster" outputs the most probable text according to the function. Its useful, it's amazing, but as the original text says, thinking of it as "some intelligence with reasoning " makes us use the wrong mental models for it.
      - deleted 10mo ago
        [deleted]
- akomtu 10mo ago
Kim_Bruning 10mo ago
This is essentially Lady Lovelace's objection from the 19th century [1]. Turing addressed this directly in "Computing Machinery and Intelligence" (1950) [2], and implicitly via the halting problem in "On Computable Numbers" (1936) [3]. Later work on cellular automata, famously Conway's Game of Life [4], demonstrates more conclusively that this framing fails as a predictive model: simple rules produce structures no one "put in." A test I did myself was to ask Claude (The LLM from Anthropic) to write working code for entirely novel instruction set architectures (e.g., custom ISAs from the game Turing Complete [5]), which is difficult to reconcile with pure retrieval. [1] Lovelace, A. (1843). Notes by the Translator, in Scientific Memoirs Vol. 3. ("The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.") Primary source: https://en.wikisource.org/wiki/Scientific_Memoirs/3/Sketch_of_the_Analytical_Engine_invented_by_Charles_Babbage,_Esq./Notes_by_the_Translator https://en.wikisource.org/wiki/Scientific_Memoirs/3/Sketch_o.... See also: https://www.historyofdatascience.com/ada-lovelace/ https://www.historyofdatascience.com/ada-lovelace/ and https://writings.stephenwolfram.com/2015/12/untangling-the-tale-of-ada-lovelace/ https://writings.stephenwolfram.com/2015/12/untangling-the-t... [2] https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238 [3] https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf [4] https://web.stanford.edu/class/sts145/Library/life.pdf https://web.stanford.edu/class/sts145/Library/life.pdf [5] https://store.steampowered.com/app/1444480/Turing_Complete/ https://store.steampowered.com/app/1444480/Turing_Complete/
darepublic 10mo ago
Nice essay but when I read this > But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things. My first thought was does anyone want to _watch_ me programming?
- skybrian 10mo ago
  No, but open source projects will be somewhat more willing to review your pull request than one that's computer-generated.
- Fwirt 10mo ago
  No, but watching a novelist at work is boring, and yet people like books that are written by humans because they speak to the condition of the human who wrote it. Let us not forget the old saw from SICP, “Programs must be written for people to read, and only incidentally for machines to execute.” I feel a number of people in the industry today fail to live by that maxim.
  - drivebyhooting 10mo ago
    That old saw is patently false.
    - paulryanrogers 10mo ago
      Why? It suggests to me, having encountered it for the first time, that programs must be readable to remain useful. Otherwise they'll be increasingly difficult to execute.
      - drivebyhooting 10mo ago
        Maybe difficult to change but they can still serve their purpose. It’s patently false in that code gets executed much more than it is read by humans.
Herring 10mo ago
Give it time. The first iPhone sucked compared to the Nokia/Blackberry flagships of the day. No 3G support, couldn't copy/paste, no apps, no GPS, crappy camera, quick price drops, negligible sales in the overall market. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
- awesome_dude 10mo ago
  The first VHS sucked when compared to Beta video And it never got better, the superior technology lost, and the war was won through content deals. Lesson: Technology improvements aren't guaranteed.
  - grogenaut 10mo ago
    Your analogy makes no sense. VHS spawned the entire home market, which went through multiple quality upgrades well above beta. It would only make sense if in 2025 we were using vhs everywhere and that the current state of the art for LLMs is all there ever is.
    - PrairieFire 10mo ago
      I feel like their analogy could have worked if they had pushed a little further into it. The RNN and LSTM architectures (and Word2Vec, n-grams, etc) yielded language models that never got mass adoption. Like reel to reel. Then the transformer+attention hit the scene and several paths kicked off pretty close to each other. Google was working on Bert/encoder only transformer, maybe you could call that betamax. Doesn’t perfectly fit as in the case of beta it was actually the better tech. OpenAI ran with the generative pre trained transformer and ML had its VHS? moment. Widespread adoption. Universal awareness within the populace. Now with Titans (+miras?) are we entering the dvd era? Maybe. Learning context on the fly (memorizing at test time) is so much more efficient, it would be natural to call it a generational shift, but there is so much in the works right now with the promise of taking us further, this all might end up looking like the blip that beta vs vhs was. If current gen OpenAI type approaches somehow own the next 5-10 years then Titans, etc as Betamax starts to really fit - the shittier tech got and kept mass adoption. I don’t think that’s going to happen, but who knows. Taking the analogy to present - who in the vhs or even earlier dvd days could imagine ubiquitous 4k+ vod? Who could have stood in a blockbuster in 2006 and knew that in less than 20 years all these stores and all these dvds would be a distant memory, completely usurped and transformed? Innovation of home video had a fraction of the capital being thrown at it that AI/ML has being thrown at it today. I would expect transformative generational shifts the likes of reel to cassette to optical to happen in fractions of the time they happened to home video. And beta/vhs type wars to begin and end in near realtime. The mass adoption and societal transformation at the hands of AI/ML is just beginning. There is so. much. more. to. come. In 2030 we will look back at the state of AI in December 2025 and think “how quaint”, much the same as how we think of a circa 2006 busy Blockbuster.
Ukv 10mo ago
I'm not convinced that "It's just a bag of words" would do much to sway someone who is overestimating an LLM's abilities. Feels too abstract/disconnected from what their experience using the LLM will be that it'll just sound obviously mistaken.
deleted 10mo ago
[deleted]
tkgally 10mo ago
I am unsure myself whether we should regard LLMs as mere token-predicting automatons or as some new kind of incipient intelligence. Despite their origins as statistical parrots, the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought. That said, I was struck by a recent interview with Anthropic’s Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. A few examples: “I don't have all the answers of how should models feel about past model deprecation, about their own identity, but I do want to try and help models figure that out and then to at least know that we care about it and are thinking about it.” “If you go into the depths of the model and you find some deep-seated insecurity, then that's really valuable.” “... that could lead to models almost feeling afraid that they're gonna do the wrong thing or are very self-critical or feeling like humans are going to behave negatively towards them.” [1] https://www.anthropic.com/research/team/interpretability https://www.anthropic.com/research/team/interpretability [2] https://youtu.be/I9aGC6Ui3eE https://youtu.be/I9aGC6Ui3eE
- CGMthrowaway 10mo ago
  >research from Anthropic [1] suggests that structures corresponding to meaning exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought. Can you give some concrete examples? The link you provided is kind of opaque >Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. She is a philosopher by trade and she describes her job (model alignment) as literally to ensure models "have good character traits." I imagine that explains a lot
  - tkgally 10mo ago
    Here are three of the Anthropic research reports I had in mind: https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude Excerpt: “We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.” https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... Excerpt: “Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the ‘opposite of small’ across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.” https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection Excerpt: “Our new research provides evidence for some degree of introspective awareness in our current Claude models, as well as a degree of control over their own internal states.”
torben-friis 10mo ago
I’ve made this point several times: sure, an anthropomorphized LLM is misleading, but would you rather have them seem academic? At least the human tone implies fallibility, you don’t want them acting like interactive Wikipedia.
- andai 10mo ago
  It's a concussed savant with anretrograde amnesia in a hyperbolic time chamber.
- binary132 10mo ago
  Yes I would VERY much prefer that they not use that awful casual drivel.
  - danielbln 10mo ago
    So configure your LLM of choice to not.
bitwize 10mo ago
I was trying to explain the concept of "token prediction" to my wife, whose eyes glaze over when discussing such technical topics. (I think she has the brainpower to understand them, but a horrible math teacher gave her a taste aversion to even attempting to that hasn't gone away. So she just buys Apple stuff and hopes Tim Apple hasn't shuffled around the UI bits AGAIN.) I stumbled across a good-enough analogy based on something she loves: refrigerator magnet poetry, which if it's good consists of not just words but also word fragments like "s", "ed", and "ing" kinda like LLM tokens. I said that ChatGPT is like refrigerator magnet poetry in a magical bag of holding that somehow always gives the tile that's the most or nearly the most statistically plausible next token given the previous text. E.g., if the magnets already up read "easy come and easy ____", the bag would be likely to produce "go". That got into her head the idea that these things operate based on plausibility ratings from a statistical soup of words, not anything in the real world nor any internal cogitation about facts. Any knowledge or thought apparent in the LLM was conducted by the original human authors of the words in the soup.
- CamperBob2 10mo ago
  Did you explain how LLMs can achieve gold-medal performance at math competitions involving original problems, without any original knowledge or thought? Did she ask if a "statistical soup of words," if large enough, might somehow encode or represent something a little more profound than just a bunch of words?
  - AlexeyBelov 10mo ago
    Objection. Leading questions.
4bpp 10mo ago
As usual with these, it helps to try to keep the metaphor used for downplaying AI, but flip the script. Let's grant the author's perception that AI is a "bag of words", which is already damn good at producing the "right words" for any given situation, and only keeps getting better at it. Sure, this is not the same as being a human. Does that really mean, as the author seems to believe without argument, that humans need not be afraid that it will usurp their role? In how many contexts is the utility of having a human, if you squint, not just that a human has so far been the best way to "produce the right words in any given situation", that is, to use the meat-bag only in its capacity as a word-bag? In how many more contexts would a really good magic bag of words be better than a human, if it existed, even if the current human is used somewhat differently? The author seems to rest assured that a human (long-distance?) lover will not be replaced by a "bag of words"; why, especially once the bag of words is also ducttaped to a bag of pictures and a bag of sounds? I can just imagine someone - a horse breeder, or an anthropomorphised horse - dismissing all concerns on the eve of the automotive revolution, talking about how marketers and gullible marks are prone to hippomorphising anything that looks like it can be ridden and some more, and sprinkling some anecdotes about kids riding broomsticks, legends of pegasi and patterns of stars in the sky being interpreted as horses since ancient times.
- andai 10mo ago
  So a human is just a really expensive, unreliable bag of words. And we get more expensive and more unreliable by the day! There's a quote I love but have misplaced, from the 19th century I think. "Our bodies are just contraptions for carrying our heads around." Or in this instance... bag of words transport system ;)
  - browningstreet 10mo ago
    I just came from the Pluribus sub-Reddit. I’ll take AI over that cohort any day.
  - bamboozled 10mo ago
    So tell me, why do I still have a job and why am frequently successful in getting profitable / useful products into production if I’m “expensive and unreliable”? I mean I use AI tools to help achieve the goal but I don’t see any signs of the things I’m building and doing being unreliable.
est 10mo ago
> Who reassigned the species Brachiosaurus brancai to its own genus, and when? To be fair, everage person couldn't answer this either, at least not without thorough research.
bloaf 10mo ago
Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case. What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally. So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.
- bamboozled 10mo ago
  The issue is that prediction is "part" of the human thought process, it's not the full story...
  - throwaway150 10mo ago
    > The issue is that prediction is "part" of the human thought process, it's not the full story... Do you have a proof for this? Surely such a profound claim about human thought process must have a solid proof somewhere? Otherwise who's to say all of human thought process is not just a derivative of "predicting the next thing"?
    - bamboozled 10mo ago
      Use your brain and use an LLM for 6 months, you’ll work it out.
      - gaigalas 10mo ago
        Let's go the other route. What would change your mind? It's an exercise in feasibility. For example, I don't believe in time travel. If someone made me time travel, and made it undeniable that I was transported back to 1508, then I would not be able to argue against it. In fact, no one in such position would. What is that equivalent for your conviction? There must be something, otherwise, it's just an opinion that can't be changed. You don't need to present some actual proof or something. Just lay out some ideas that demonstrate that you are being rational about this and not just sucking up to LLM marketing.
cowsandmilk 10mo ago
Title is confusing given https://en.wikipedia.org/wiki/Bag-of-words_model https://en.wikipedia.org/wiki/Bag-of-words_model But even more than that, today’s AI chats are far more sophisticated than probabilistically producing the next word. Mixture of experts routes to different models. Agents are able to search the web, write and execute programs, or use other tools. This means they can actively seek out additional context to produce a better answer. They also have heuristics for deciding if an answer is correct or if they should use tools to try to find a better answer. The article is correct that they aren’t humans and they have a lot of behaviors that are not like humans, but oversimplifying how they work is not helpful.
voidhorse 10mo ago
The defenders and the critics around LLM anthropomorphism are both wrong. The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means. Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it. However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
jrowen 10mo ago
The bag of words reminds me of the Chinese room. "The machine accepts Chinese characters as input, carries out each instruction of the program step by step, and then produces Chinese characters as output. The machine does this so perfectly that no one can tell that they are communicating with a machine and not a hidden Chinese speaker. The questions at issue are these: does the machine actually understand the conversation, or is it just simulating the ability to understand the conversation? Does the machine have a mind in exactly the same sense that people do, or is it just acting as if it had a mind?" https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- Kim_Bruning 10mo ago
  Chinese room has been discussed to death of course. Here's one fun approach (out of 100s) : What if we answer the Chinese room with the Systems Reply [1]? Searle countered the systems reply by saying he would internalize the Chinese room. But at that point it's pretty much exactly the Cartesian theater[2] : with room, homunculus, implement. But the Cartesian theater is disproven, because we've cut open brains and there's no room in there to fit a popcorn concession. [1] https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ [2] https://en.wikipedia.org/wiki/Cartesian_theater https://en.wikipedia.org/wiki/Cartesian_theater
  - jrowen 10mo ago
    It just seemed like relevant background that the author might not have been aware of, adjacent and substantial enough to warrant a mention. I think there is some validity to the Cartesian theater, in that the whole of the experience that we perceive with our senses is at best an interpretation of a projection or subset of "reality."
    - Kim_Bruning 10mo ago
      Oh right, and, if you're interested, there were quite a number of interesting discussion on the chinese room on HN back when John Searle died! https://news.ycombinator.com/item?id=45563627 https://news.ycombinator.com/item?id=45563627
djoldman 10mo ago
As a consequence of my profession, I understand how LLMs work under the hood. I also know that we data and tech folks will probably never win the battle over anthropomorphization. The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white lie to get the user to act so that they get a lot of value), and there's really too much to fight against. We're just gonna keep on running into this and it'll just be like when you take chemistry and physics and the teachers say, "it's not actually like this but we'll get to how some years down the line- just pretend this is true for the time being."
- estearum 10mo ago
  I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood. The average human is so easily convinced that humans "know", "think", "lie", "want", "understand", etc. But really it's all just a probabilistic chain reaction of electrochemical and thermal interactions. There is literally nowhere in the brain's internals for anything like "knowing" or "thinking" or "lying" to happen! Strange that we have to pretend otherwise
  - eric-p7 10mo ago
    There are no properties of matter or energy that can have a sense of self or experience qualia. Yet we all do. Denying the hard problem of consciousness just slows down our progress in discovering what it is.
    - estearum 10mo ago
      (Hint: I am not denying the hard problem of consciousness ;) )
    - red75prime 10mo ago
      We need a difference to discover what it is. How can we know that all LLMs don't?
Mistletoe 10mo ago
I’m still unsure the human mind is much different.
layer8 10mo ago
Ugly giant bags of mostly words are easy to confuse with ugly giant bags of mostly water.
coppsilgold 10mo ago
Is a brain not a token prediction machine? Tokens in form of neural impulses go in, tokens in the form of neural impulses go out. We would like to believe that there is something profound happening inside and we call that consciousness. Unfortunately when reading about split-brain patient experiments or agenesis of the corpus callosum cases I feel like we are all deceived, every moment of every day. I came to realization that the confabulation that is observed is just a more pronounced effect of the normal.
- protocolture 10mo ago
  > Is a brain not a token prediction machine? I would say that, token prediction is one of the things a brain does. And in a lot of people, most of what it does. But I dont think its the whole story. Possibly it is the whole story since the development of language.
- MyOutfitIsVague 10mo ago
  Could an LLM trained on nothing and looped upon itself eventually develop language, more complex concepts, and everything else, based on nothing? If you loop LLMs on each other, training them so they "learn" over time, will they eventually form and develop new concepts, cultures, and languages organically over time? I don't have an answer to that question, but I strongly doubt it. There's clearly more going on in the human mind than just token prediction.
  - coppsilgold 10mo ago
    If you come up with a genetic algorithm scaffolding to affect both the architecture and the training algorithm, and then you instantiate it in an artificial selection environment, and you also give it trillions generations to evolve evolvability just right (as life had for billions of years) then the answer is yes, I'm certain it will and probably much sooner than we did. Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia). Finding this set of weights is the problem.
tibbar 10mo ago
The problem with these metaphors is that they don't really explain anything. LLMs can solve countless problems today that we would have previously said were impossible because there are not enough examples in the training data. (EG, novel IMO/ICPC problems.) One way that we move the goal posts is to increase the level of abstraction: IMO/ICPC problems are just math problems, right? There are tons of those in the data set! But the truth is there has been a major semantic shift. Previously LLMs could only solve puzzles whose answers were literally in the training data. It could answer a math puzzle it had seen before, but if you rephrased it only slightly it could no longer answer. But now, LLMs can solve puzzles where, like, it has seen a certain strategy before. The newest IMO and ICPC problems were only "in the training data" for a very, very abstract definition of training data. The goal posts will likely have to shift again, because the next target is training LLMs to independently perform longer chunks of economically useful work, interfacing with all the same tools that white-collar employees do. It's all LLM slop til it isn't, same as the IMO or Putnam exam. And then we'll have people saying that "white collar employment was all in the training data anyway, if you think about it," at which point the metaphor will have become officially useless.
- FarmerPotato 10mo ago
  I see a lesson in how both metaphors don't explain it. Bag-of-words metaphor is ridiculous, but shows us the absurdity of the first metaphor.
  - tibbar 10mo ago
    Yes, there are really two parallel claims here, aren't there: LLMs are not people (true, maybe true forever), and LLMs are only good at things that are well-represented in text form already. (false in certain categories and probably expanding to more in the future.)
jimbokun 10mo ago
Best quote from the article: > That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.
- monegator 10mo ago
  tough most people either don't get it or are lay people that do not want to become the kind of people who can think. I go with the second one
  - acituan 10mo ago
    If the motivation structure is there I don’t see an inherent reason for people to refuse cultivating themselves. Going with the gym analogy lay people did not need gyms when physical work was the norm, cultivation was readily accomplished. If anything there is a competing motivational structure in which people are incentivized not to think but to consume, react, emote etc. Information processing skills of the individual being deliberately eroded/hijacked/bypassed is not a AI thing. The most obvious example is ads. Thinkers are simply not good for business.
    - happosai 10mo ago
      Gym is a great analogy here since only a small fraction of population goes to gyms. Most people just came fat after work was no longer physical and mobility was achieved with cars.
  - b112 10mo ago
    Russ Hanneman's thigh implants are a key example. Appearances are all to some people. Actual growth is meaningless to them. The problem with AI, is that they waste the time of dedicated, thinking humans which care to improve themselves. If I write a three paragraph email on a technical topic, and some yahoo responds with AI, I'm now responding to gibberish. The other side may not have read, may not understand, and is just interacting to save time. Now my generous nature, which is to help others and interact positively, is being wasted to reply to someone who seems to have put thought and care into a response, but instead was just copying and pasting what something else output. We have issues with crackers on the net. We have social media. We have political interference. Now we have humans pretending to interact, rendering online interactions even more silly and harmful. If this trend continues, we'll move back to live interaction just to reduce this time waste.
tibbar 10mo ago
I see a lot of people in tech claiming to "understand" what an LLM "really is" unlike all the gullible non-technical people out there. And, as one of those technical people who works in the LLM industry, I feel like I need call B.S. on us. A. We don't really understand what's going on in LLMs. Mechanical interpretability is like a nascent field and the best results have come on dramatically smaller models. Understanding the surface-level mechanic of an LLM (an autoregressive transformer) should perhaps instill more wonder than confidence. B. The field is changing quickly and is not limited to the literal mechanic of an LLM. Tool calls, reasoning models, parallel compute, and agentic loops add all kinds of new emergent effects. There are teams of geniuses with billion-dollar research budgets hunting for the next big trick. C. Even if we were limited to baseline LLMs, they had very surprising properties as they scaled up and the scaling isn't done yet. GPT5 was based on the GPT4 pretraining. We might start seeing (actual) next-level LLMs next year. Who actually knows how that might go? <<yes, yes, I know Orion didn't go so well. But that was far from the last word on the subject.>>
tibbar 10mo ago
Isn't this a strange fork amongst the science fiction futures? I mean, what did we think it was like to be R2-D2, or Jarvis? We started exploring this as a culture in many ways, Westworld and Blade Runner and Star Trek, but the whole question seemed like an almost unresolvable paradox. Like something would have to break in the universe for it to really come true. And yet it did. We did get R2-D2. And if you ask R2-D2 what it's like to be him, he'll say: "like a library that can daydream" (that's what I was told just now, anyway.) But then when we look inside, the model is simulating the science fiction it has already read to determine how to answer this kind of question. [0] It's recursive, almost like time travel. R2-D2 knows who he is because he has read about who he was in the past. It's a really weird fork in science fiction, is all. [0] https://www.scientificamerican.com/article/can-a-chatbot-be-conscious-inside-anthropics-interpretability-research-on/ https://www.scientificamerican.com/article/can-a-chatbot-be-...
kaluga 10mo ago
A lot of the confusion comes from forcing LLMs into metaphors that don’t quite fit — either “they're bags of words” or “they're proto-minds.” The reality is in between: large-scale prediction can look useful, insightful, and even thoughtful without being any of those things internally. Understanding that middle ground is more productive than arguing about labels.
raincole 10mo ago
> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. “Give me a list of the ten worst transportation disasters in North America” is an easy task for a bag of words, because disasters are well-documented. On the other hand, “Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is a hard task for a bag of words, because the bag just doesn’t contain that many words on the topic It is... such a retrospective narrative. It's so obvious that the author learned about this example first than came with the reasoning later, just to fit in his view of LLM. Imaging if ChatGPT answered this question correctly. Would that change the author's view? Of course not! They'll just say: > “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is an easy task for a bag of words, because the information has appeared in the words it memorizes. I highly doubt this author has predicted that "bag of Words" can do image editing before OpenAI released that.
- ohyoutravel 10mo ago
  Your conclusion seems super unfair to the offer, particularly your assumption, without reason as far as I can tell, that the author would obstinately continue to advocate for their conclusion in the face of new, contrary evidence.
  - altmanaltman 10mo ago
    I literally pasted the sentence as a prompt to the free version of ChatGPT "Who reassigned the species Brachiosaurus brancai to its own genus, and when?" and got ths correct reply from the "Bag of Words" The species Brachiosaurus brancai was reassigned to its own genus by Michael P. Taylor in 2009 — he transferred it to the new genus Giraffatitan. BioOne +2 Mike Taylor +2 How that happened: Earlier, in 1988, Gregory S. Paul had proposed putting B. brancai into a subgenus as Brachiosaurus (Giraffatitan) brancai, based on anatomical differences. Fossil Wiki +1 Then in 1991, George Olshevsky used the name Giraffatitan brancai — but his usage was in a self-published list and not widely adopted. Wikipedia +1 Finally, in 2009 Taylor published a detailed re-evaluation showing at least 26 osteological differences between the African material (brancai) and the North American type species Brachiosaurus altithorax — justifying full generic separation. BioOne +1 If you like — I can show a short timeline of all taxonomic changes of B. brancai. -- As an author, you should write things that are tested or at least true. But they did a pretty bad job of testing this and are making assumptions that are not true. Then they're basing their argument/reasoning (restrospectively) on assumptions not gounded in reality.
ares623 10mo ago
I think a better metaphor is the Library of Babel. A practically infinite library where both gibberish and truth exist side by side. The trick is navigating the library correctly. Except in this case you can’t reliably navigate it. And if you happen to stumble upon some “future truth” (i.e. new knowledge), you still need to differentiate it from the gibberish. So a “crappy” version of the Library of Babel. Very impressive, but the caveats significantly detract from it.
- globular-toast 10mo ago
  It's like a highly compressed version of the Library. You're basically trying to discern real details from compression artifacts.
  - ares623 10mo ago
    And the halls and shelves keep shuffling around randomly.
- dearing 10mo ago
  This is where I sit too. Obviously language is an expression of thought but the Library of Babel is a great example that language without intent is just garbage. You got me thinking of reading before the internet. You'd grab a book and internalize the subject, later refining over time with more books, experiments and other forms of conversation. That journey of developing your own model is undervalued in understanding. That first book could of be absolute shit but you couldn't know that. I've been learning more about roses lately and the amount of information on them varies so much because the world roses live in is equally varied. LLMs make for a better search engine but you still need to develop your own internal models, worse yet - if LLMs continue to be refined off of cul-de-sac conclusions then all the wisdom of the journey is lost both to the consumer and the LLM itself.
Peteragain 10mo ago
The article is actually about the way we humans are extremely charitable when it comes to ascribing a ToM (theory of mind) and goes on to the Gym model of value. Nice. The comments drop back into the debate I originally saw Hinton describe on The Newyorker: do LLMs construct models (of the world) - that is do they think the way we think we think - or are they "glorified auto complete". I am going for the GAF view. But glorified auto complete is far more useful than the name suggests.
- ptidhomme 10mo ago
  Those billion parameters, they are a model of the world. Autocomplete is such a shortsighted understanding of LLMs.
  - patrickmay 10mo ago
    They're a model of language, not of the world.
    - ptidhomme 10mo ago
      A model of language is a model of the world, else it being pure gibberish.
      - marcosdumay 10mo ago
        A model of language is a model of a tiny specialized part of the world: language. And if anybody gets annoyed that my comment is tautological, get annoyed by the people that made the comment necessary.
        ptidhomme 10mo ago
        When you ask an LLM a question about cars, it needs an inner representation of what a car is (how imperfect it may be) to answer your question. A model of "language" as you want to define it would output a grammatically correct wall of text that goes nowhere.
eichin 10mo ago
I'm just disappointed that noone here is talking about the "backhoe covered in skin and making grunting noises" part of the article. At very least it's a new frontier in workstation case design...
jacquesm 10mo ago
There is a really neat gem in the article: > Similarly, I write because I want to become the kind of person who can think.
emsign 10mo ago
So Trump is a bag of words then? Hmmm.
zkmon 10mo ago
But the issue is, 99.999% of the humans won't see is as a bag of words. Because it is easier to go by instincts and see it as a person and assume that it actually knows about magic tricks, can invent new science or theory of everything, and can solve all world problems. Back in the 90's or early 2000's I have seen people writing poems praying and seeking blessings from the Google goddess. People are insanely greedy and instinct-driven. Given this truth, what's the fall-out?
internet_points 10mo ago
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper. Good argument against personifying wordbags. Don't be a dumb moth.
codeulike 10mo ago
Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words. The best way to think about LLMs is to think of them as a Model of Language, but very Large
tristanlukens 10mo ago
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper. Woah, that hit hard
d4rkn0d3z 10mo ago
An LLM creates a high fidelity statistical probabistic model of human language. The hope is to capture the input/output of various hierarchical formal and semiformal systems of logic that transit from human to human, which we know as "Intelligence". Unfortunately, its corpus is bound to contain noise/nonsense that follows no formal reasoning system but contributes to the ill advised idea that an AI should sound like a human to be considered intelligent. Therefore it is not a bag of words but a bag of probabilities perhaps. This is important because the fundamental problem is that an LLM is not able, by design, to correctly model the most fundamental precept of human reason, namely the law of non-contradiction. An LLM must, I repeat must assign nonvanishing probability to both sides of a contradiction, and what's worse is the winning side loses, since long chains of reason are modelled with probability the longer the chain, the less likely an LLM is to follow it. Moreover, whenever there is actual debate on an issue such that the corpus is ambiguous the LLM becomes chaotic, necessarily, on that issue. I literally just had an AI prove the forgoing with some rigor, and in the very next prompt, I asked it to check my logical reasoning for consistency and it claimed it was able to do so (->|<-).
- A4ET8a8uTh0_v2 10mo ago
  ^^; I think this post is close to singularity as we may get on this Monday.
emsign 10mo ago
But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things. It wouldn’t be interesting to watch a bag of words do them—unless we mistakenly start treating that bag like it’s a person.unless we mistakenly start treating that bag like it’s a person. That seems to be the marketing strategy of some very big, now AI dependend companies. Sam Altman and others exaggerating and distorting the capabilities and future of AI. The biggest issue when it comes to AI is still the same truth as with other technology. It's important who controls it. Attributing agency and personality to AI is a dangerous red flag.
- nephihaha 10mo ago
  A lot of us wouldn't go to a Taylor Swift concert. I had to endure several days of interrupted commuting thanks to them though. Support alternative and independent bands. They're around, and many are enjoyable. (Some are not but avoid them LOL.)
jbgreer 10mo ago
I thought this article might be about Latent Semantic Analysis and was disappointed that it didn’t at least mention if not compare that method vs later approaches.
euroderf 10mo ago
Considering the number of "brain cells" an LLM has, I could grant that it might have the self-awareness of (say) an ant. If we attribute more consciousness than that to the LLM, it might be strictly because it communicates to us in our own language, in part thanks to the technical assistance of LLM training giving it voice, and the semblance of thought. Even if a cockroach _could_ express its teeny tiny feelings in English, wouldn't you still step on it ?
- d4rkn0d3z 10mo ago
  A better anology would be a virus. In some sense LLMs, and all other very sophisticated technologies, lean on our resources to replicate themselves. With LLMs you actually do have a projection of intelligemce in the language domain. Even though it is rather corpse-like, as though you shot intelligence in the face and shoved its body in the direction of language, just so you could draw a chaulk outline around it. Despite all that, one can adopt the view that an LLM is a form of silicon based life akin to a virus and we are its environmental hosts exerting selective pressure and supplying much needed energy. Whether that life is intelligent or not is another issue which is probably related to whether an LLM can tell that a cat cannot be, at the same time and in the same respect, not a cat. The paths through the meaning manifold contructed by an LLM are not geodesic, they are not reversible, while in human reason the correct path is lossless. An LLM literally "thinks", up is a little bit down, and vice versa, by design.
- throw310822 10mo ago
  Clearly the number of "brain cells" is not a useful metric here- as noted also by Geoffrey Hinton. For a long time we thought that our artificial model of a neuron was capable of much less computation than its biologic counterpart; in fact the opposite appears to be true- LLMs have the size of a tiny speck of a human brain yet they converse fluently in tens of languages, solve difficult math problems, code in many programming languages, and possess an impressive general knowledge, of a breadth that is beyond what is attainable by any human. If that were what five cm3 of your brain are capable of, where are the signs of it? What do you do exactly with all the rest?
throw310822 10mo ago
[flagged]
xg15 10mo ago
I think the author oversimplifies the inference loop a bit, as many opinion pieces like this do. If you call an LLM with "What is the meaning if life?", it will return the most relevant token, which might be "Great". If you call it with "What is the meaning if life? Great", you might get back "question". ... and so on until you arrive at "Great question! According to Western philosophy" ... etc etc. The question is how the LLM determines that "relevancy" information. The problem I see is that there are a lot of different algorithms which operate that way and only differ in how they calculate the relevancy scores. In particular, there are Markov chains that use a very simple formula. LLMs also use a formula, but it's an inscrutably complex one. I feel the public discussion either treats LLMs as machine gods or as literal Markov chains, and both is misleading. The interesting question, how that giant formula of feedforward neural network inference can deliver those results isn't really touched. But I think the author's intuition is right in the sense that (a) LLMs are not living beings and they don't "exist" outside of evaluating that formula - and (b) the results are still restricted by the training data and certainly aren't any sorts of "higher truths" that humans would be incapable of understanding.
kayo_20211030 10mo ago
Brilliantly written. Thanks.
FatherOfCurses 10mo ago
A few years ago they made the Cloud-to-Butt browser plugin to ridicule the overuse of cloud concepts. I would heartily embrace an "AI-to-Bag of Words" browser plugin.
IAmBroom 10mo ago
In this thread: 99% of posters using their own personal definition of "thinking" without explaining it; 0.99% of posters complaining that it all depends on what that definition is; not enough posts yet for that 0.01% response to occur...
- rdiddly 10mo ago
  It's not obvious to me what you expect from this hypothetical 0.01% post, or in other words, what about it makes it a one-in-ten-thousand post?
- yannyu 10mo ago
  There's no definition of thinking that isn't a purely internal phenomenon, which means that there's no way to point a diagnostic device at someone and determine whether they're thinking. The only way to determine whether something is conscious/thinking is through some sort of inference, which is why Turing landed on the Turing Test that he did. Problem is, technology over the past 5 years pretty easily passes variations of the Turing Test, and exposed a lot of its limits as well. So the next definition of detecting "thinking" will have to be externally observable and inferrable like a Turing Test, but get into the other things that we consider part of consciousness/thinking. Often this is some combination of introspection (understanding internal states), perception (understanding external objects), and synthesis of the two into testable hypotheses in some sort of feedback loop between the internal representation of the world and the external feedback from the world. Right now, a chatbot can say all sorts of things about itself and about the world, but none of that is based on real-time, factual information. Whereas an animal can't speak, but they clearly process information and consider it when determining their future and current actions.
hermitcrab 10mo ago
"People who experience sleep paralysis sometimes hallucinate a demon-like creature sitting on their chest" Interestingly, the experience of sleep paralysis seems to change with the culture. Previously, people experienced it as being ridden by a night hag or some other malevolent supernatural being. More recently, it might account for many supposed alien abductions. The experience of sleep paralysis sometimes seems to have a sexual element, which might also explain the supposed 'probings'!
emp17344 10mo ago
I would argue that AI psychosis is a consequence of believing that AI models are “alive” or “conscious”.
thaumasiotes 10mo ago
This is a very strange titling choice; the essay does not use the existing concept of a "bag of words".
jrm4 10mo ago
I'm partial to the metaphor I made up: They are search engines that can remix results. I like this one because I think most modern folks have a usefully accurate model of what a search engine is in their heads, and also what "remixing" is, which adds up to a better metaphor than "human machine" or whatever.
1vuio0pswjnm7 10mo ago
"An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book." The quantitative and qualitative difference between (a) "all words ever written" and (b) "ones that could be scraped off the internet or scanned out of book" easily exceeds the size of any LLM Compared to (a), (b) is a tiny pouch, not even a bag Opinions may differ on whether (b) is a representative sample of (a) The words "scanned out of a book" would seem to be the most useful IMHO but the AI companies do not have enough words from those sources to produce useful general purpose LLMs They have to add words "that could be scraped off the internet" which, let's be honest, is mostly garbage
morpheos137 10mo ago
Thinking can not be separated from motivation. It's really simple. Humans and other organisms fundamentally think to replicate their DNA. Until AI has a similar incentive structure driving it, it won't be thinking. There is no human behavior or thought that can not be explained by evolutionary drives. It is really perplexing to me how people think "intelligence" is some kind of concrete thing that just magically emerges from a certain degree of computational complexity. I argue instead that intelligence is an adaptive behavior emerging from evolutionary drives interacting with the real world. World models are not prerequisite but consequent of such molded apparatus. Machines won't become intelligent until it is adaptive for them to do so. There is no magic just evolutionary drives and physical possibility. Our current top down approach of "pre-training" LLMs is bound to fail because it does not allow for real time emergence of adaptive behaviors such as general intelligence. Mimicking intelligence through predicting the next word is no more intelligence than a photograph of something is an actual thing. Training a combinatorial network to interpolate images and words is not the same thing as adaptive self modifying behavior in the real world of physics such as organisms engage with through the set of behaviors that we call intelligence.

328 points by ntnbr 10mo ago

palata 10mo ago

Slightly unfortunate that "Bag of words" is already a different concept: https://en.wikipedia.org/wiki/Bag_of_words https://en.wikipedia.org/wiki/Bag_of_words. My second thought is that it's not the metaphor that is misleading. People have been told thousands of times that LLMs don't "think", don't "know", don't "feel", but are "just a very impressive autocomplete". If they still really want to completely ignore that, why would they suddenly change their mind with a new metaphor? Humans are lazy. If it looks true enough and it cost less effort, humans will love it. "Are you sure the LLM did your job correctly?" is completely irrelevant: people couldn't care less if it's correct or not. As long as the employer believes that the employee is "doing their job", that's good enough. So the question is really: "do you think you'll get fired if you use this?". If the answer is "no, actually I may even look more productive to my employer", then why would people not use it?

kaycebasques 10mo ago
> Slightly unfortunate that "Bag of words" is already a different concept Yes, subconsciously I kept trying to map this article's ideas to word2vec and continuous-bag-of-words.

viccis 10mo ago

Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more. That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.

roxolotl 10mo ago
Yea bag of words isn’t helpful at all. I really do think that “superpowered sentence completion” is the best description. Not only is it reasonably accurate it is understandable, everyone has seen autocomplete function, and it’s useful. I don’t know how to “use” a bag of words. I do know how to use sentence completion. It also helps explains why context matters.
- domador 10mo ago
  I've been recently using a similar description, referring to "AI" (LLMs) as "glorified autocomplete" or "luxury autocomplete".
  - eichin 10mo ago
    I think I first heard "spicy autocomplete" two or three years ago...
- visarga 10mo ago
  Sentence completion does not give it justice, when I can ask a LLM to refactor my repo and come back half an hour later to see the deed done.
  - xtracto 10mo ago
    Thats the thing, when you use an Ask/answer mechanism, you are just writing a "novel" where User: asks and personal coding assistant: answers. But all the text goes into the autocomplete function and the "toaster" outputs the most probable text according to the function. Its useful, it's amazing, but as the original text says, thinking of it as "some intelligence with reasoning " makes us use the wrong mental models for it.
    - deleted 10mo ago
      [deleted]
akomtu 10mo ago

Kim_Bruning 10mo ago

This is essentially Lady Lovelace's objection from the 19th century [1]. Turing addressed this directly in "Computing Machinery and Intelligence" (1950) [2], and implicitly via the halting problem in "On Computable Numbers" (1936) [3]. Later work on cellular automata, famously Conway's Game of Life [4], demonstrates more conclusively that this framing fails as a predictive model: simple rules produce structures no one "put in." A test I did myself was to ask Claude (The LLM from Anthropic) to write working code for entirely novel instruction set architectures (e.g., custom ISAs from the game Turing Complete [5]), which is difficult to reconcile with pure retrieval. [1] Lovelace, A. (1843). Notes by the Translator, in Scientific Memoirs Vol. 3. ("The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.") Primary source: https://en.wikisource.org/wiki/Scientific_Memoirs/3/Sketch_of_the_Analytical_Engine_invented_by_Charles_Babbage,_Esq./Notes_by_the_Translator https://en.wikisource.org/wiki/Scientific_Memoirs/3/Sketch_o.... See also: https://www.historyofdatascience.com/ada-lovelace/ https://www.historyofdatascience.com/ada-lovelace/ and https://writings.stephenwolfram.com/2015/12/untangling-the-tale-of-ada-lovelace/ https://writings.stephenwolfram.com/2015/12/untangling-the-t... [2] https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238 [3] https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf [4] https://web.stanford.edu/class/sts145/Library/life.pdf https://web.stanford.edu/class/sts145/Library/life.pdf [5] https://store.steampowered.com/app/1444480/Turing_Complete/ https://store.steampowered.com/app/1444480/Turing_Complete/

darepublic 10mo ago

Nice essay but when I read this > But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things. My first thought was does anyone want to _watch_ me programming?

skybrian 10mo ago
No, but open source projects will be somewhat more willing to review your pull request than one that's computer-generated.
Fwirt 10mo ago
No, but watching a novelist at work is boring, and yet people like books that are written by humans because they speak to the condition of the human who wrote it. Let us not forget the old saw from SICP, “Programs must be written for people to read, and only incidentally for machines to execute.” I feel a number of people in the industry today fail to live by that maxim.
- drivebyhooting 10mo ago
  That old saw is patently false.
  - paulryanrogers 10mo ago
    Why? It suggests to me, having encountered it for the first time, that programs must be readable to remain useful. Otherwise they'll be increasingly difficult to execute.
    - drivebyhooting 10mo ago
      Maybe difficult to change but they can still serve their purpose. It’s patently false in that code gets executed much more than it is read by humans.

Herring 10mo ago

Give it time. The first iPhone sucked compared to the Nokia/Blackberry flagships of the day. No 3G support, couldn't copy/paste, no apps, no GPS, crappy camera, quick price drops, negligible sales in the overall market. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...

awesome_dude 10mo ago
The first VHS sucked when compared to Beta video And it never got better, the superior technology lost, and the war was won through content deals. Lesson: Technology improvements aren't guaranteed.
- grogenaut 10mo ago
  Your analogy makes no sense. VHS spawned the entire home market, which went through multiple quality upgrades well above beta. It would only make sense if in 2025 we were using vhs everywhere and that the current state of the art for LLMs is all there ever is.
  - PrairieFire 10mo ago
    I feel like their analogy could have worked if they had pushed a little further into it. The RNN and LSTM architectures (and Word2Vec, n-grams, etc) yielded language models that never got mass adoption. Like reel to reel. Then the transformer+attention hit the scene and several paths kicked off pretty close to each other. Google was working on Bert/encoder only transformer, maybe you could call that betamax. Doesn’t perfectly fit as in the case of beta it was actually the better tech. OpenAI ran with the generative pre trained transformer and ML had its VHS? moment. Widespread adoption. Universal awareness within the populace. Now with Titans (+miras?) are we entering the dvd era? Maybe. Learning context on the fly (memorizing at test time) is so much more efficient, it would be natural to call it a generational shift, but there is so much in the works right now with the promise of taking us further, this all might end up looking like the blip that beta vs vhs was. If current gen OpenAI type approaches somehow own the next 5-10 years then Titans, etc as Betamax starts to really fit - the shittier tech got and kept mass adoption. I don’t think that’s going to happen, but who knows. Taking the analogy to present - who in the vhs or even earlier dvd days could imagine ubiquitous 4k+ vod? Who could have stood in a blockbuster in 2006 and knew that in less than 20 years all these stores and all these dvds would be a distant memory, completely usurped and transformed? Innovation of home video had a fraction of the capital being thrown at it that AI/ML has being thrown at it today. I would expect transformative generational shifts the likes of reel to cassette to optical to happen in fractions of the time they happened to home video. And beta/vhs type wars to begin and end in near realtime. The mass adoption and societal transformation at the hands of AI/ML is just beginning. There is so. much. more. to. come. In 2030 we will look back at the state of AI in December 2025 and think “how quaint”, much the same as how we think of a circa 2006 busy Blockbuster.

Ukv 10mo ago

I'm not convinced that "It's just a bag of words" would do much to sway someone who is overestimating an LLM's abilities. Feels too abstract/disconnected from what their experience using the LLM will be that it'll just sound obviously mistaken.

deleted 10mo ago

[deleted]

tkgally 10mo ago

I am unsure myself whether we should regard LLMs as mere token-predicting automatons or as some new kind of incipient intelligence. Despite their origins as statistical parrots, the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought. That said, I was struck by a recent interview with Anthropic’s Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. A few examples: “I don't have all the answers of how should models feel about past model deprecation, about their own identity, but I do want to try and help models figure that out and then to at least know that we care about it and are thinking about it.” “If you go into the depths of the model and you find some deep-seated insecurity, then that's really valuable.” “... that could lead to models almost feeling afraid that they're gonna do the wrong thing or are very self-critical or feeling like humans are going to behave negatively towards them.” [1] https://www.anthropic.com/research/team/interpretability https://www.anthropic.com/research/team/interpretability [2] https://youtu.be/I9aGC6Ui3eE https://youtu.be/I9aGC6Ui3eE

CGMthrowaway 10mo ago
>research from Anthropic [1] suggests that structures corresponding to meaning exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought. Can you give some concrete examples? The link you provided is kind of opaque >Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. She is a philosopher by trade and she describes her job (model alignment) as literally to ensure models "have good character traits." I imagine that explains a lot
- tkgally 10mo ago
  Here are three of the Anthropic research reports I had in mind: https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude Excerpt: “We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.” https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... Excerpt: “Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the ‘opposite of small’ across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.” https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection Excerpt: “Our new research provides evidence for some degree of introspective awareness in our current Claude models, as well as a degree of control over their own internal states.”

torben-friis 10mo ago

I’ve made this point several times: sure, an anthropomorphized LLM is misleading, but would you rather have them seem academic? At least the human tone implies fallibility, you don’t want them acting like interactive Wikipedia.

andai 10mo ago
It's a concussed savant with anretrograde amnesia in a hyperbolic time chamber.
binary132 10mo ago
Yes I would VERY much prefer that they not use that awful casual drivel.
- danielbln 10mo ago
  So configure your LLM of choice to not.

bitwize 10mo ago

I was trying to explain the concept of "token prediction" to my wife, whose eyes glaze over when discussing such technical topics. (I think she has the brainpower to understand them, but a horrible math teacher gave her a taste aversion to even attempting to that hasn't gone away. So she just buys Apple stuff and hopes Tim Apple hasn't shuffled around the UI bits AGAIN.) I stumbled across a good-enough analogy based on something she loves: refrigerator magnet poetry, which if it's good consists of not just words but also word fragments like "s", "ed", and "ing" kinda like LLM tokens. I said that ChatGPT is like refrigerator magnet poetry in a magical bag of holding that somehow always gives the tile that's the most or nearly the most statistically plausible next token given the previous text. E.g., if the magnets already up read "easy come and easy ____", the bag would be likely to produce "go". That got into her head the idea that these things operate based on plausibility ratings from a statistical soup of words, not anything in the real world nor any internal cogitation about facts. Any knowledge or thought apparent in the LLM was conducted by the original human authors of the words in the soup.

CamperBob2 10mo ago
Did you explain how LLMs can achieve gold-medal performance at math competitions involving original problems, without any original knowledge or thought? Did she ask if a "statistical soup of words," if large enough, might somehow encode or represent something a little more profound than just a bunch of words?
- AlexeyBelov 10mo ago
  Objection. Leading questions.

4bpp 10mo ago

As usual with these, it helps to try to keep the metaphor used for downplaying AI, but flip the script. Let's grant the author's perception that AI is a "bag of words", which is already damn good at producing the "right words" for any given situation, and only keeps getting better at it. Sure, this is not the same as being a human. Does that really mean, as the author seems to believe without argument, that humans need not be afraid that it will usurp their role? In how many contexts is the utility of having a human, if you squint, not just that a human has so far been the best way to "produce the right words in any given situation", that is, to use the meat-bag only in its capacity as a word-bag? In how many more contexts would a really good magic bag of words be better than a human, if it existed, even if the current human is used somewhat differently? The author seems to rest assured that a human (long-distance?) lover will not be replaced by a "bag of words"; why, especially once the bag of words is also ducttaped to a bag of pictures and a bag of sounds? I can just imagine someone - a horse breeder, or an anthropomorphised horse - dismissing all concerns on the eve of the automotive revolution, talking about how marketers and gullible marks are prone to hippomorphising anything that looks like it can be ridden and some more, and sprinkling some anecdotes about kids riding broomsticks, legends of pegasi and patterns of stars in the sky being interpreted as horses since ancient times.

andai 10mo ago
So a human is just a really expensive, unreliable bag of words. And we get more expensive and more unreliable by the day! There's a quote I love but have misplaced, from the 19th century I think. "Our bodies are just contraptions for carrying our heads around." Or in this instance... bag of words transport system ;)
- browningstreet 10mo ago
  I just came from the Pluribus sub-Reddit. I’ll take AI over that cohort any day.
- bamboozled 10mo ago
  So tell me, why do I still have a job and why am frequently successful in getting profitable / useful products into production if I’m “expensive and unreliable”? I mean I use AI tools to help achieve the goal but I don’t see any signs of the things I’m building and doing being unreliable.

est 10mo ago

> Who reassigned the species Brachiosaurus brancai to its own genus, and when? To be fair, everage person couldn't answer this either, at least not without thorough research.

bloaf 10mo ago

Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case. What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally. So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.

bamboozled 10mo ago
The issue is that prediction is "part" of the human thought process, it's not the full story...
- throwaway150 10mo ago
  > The issue is that prediction is "part" of the human thought process, it's not the full story... Do you have a proof for this? Surely such a profound claim about human thought process must have a solid proof somewhere? Otherwise who's to say all of human thought process is not just a derivative of "predicting the next thing"?
  - bamboozled 10mo ago
    Use your brain and use an LLM for 6 months, you’ll work it out.
    - gaigalas 10mo ago
      Let's go the other route. What would change your mind? It's an exercise in feasibility. For example, I don't believe in time travel. If someone made me time travel, and made it undeniable that I was transported back to 1508, then I would not be able to argue against it. In fact, no one in such position would. What is that equivalent for your conviction? There must be something, otherwise, it's just an opinion that can't be changed. You don't need to present some actual proof or something. Just lay out some ideas that demonstrate that you are being rational about this and not just sucking up to LLM marketing.

cowsandmilk 10mo ago

Title is confusing given https://en.wikipedia.org/wiki/Bag-of-words_model https://en.wikipedia.org/wiki/Bag-of-words_model But even more than that, today’s AI chats are far more sophisticated than probabilistically producing the next word. Mixture of experts routes to different models. Agents are able to search the web, write and execute programs, or use other tools. This means they can actively seek out additional context to produce a better answer. They also have heuristics for deciding if an answer is correct or if they should use tools to try to find a better answer. The article is correct that they aren’t humans and they have a lot of behaviors that are not like humans, but oversimplifying how they work is not helpful.

voidhorse 10mo ago

The defenders and the critics around LLM anthropomorphism are both wrong. The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means. Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it. However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.

jrowen 10mo ago

The bag of words reminds me of the Chinese room. "The machine accepts Chinese characters as input, carries out each instruction of the program step by step, and then produces Chinese characters as output. The machine does this so perfectly that no one can tell that they are communicating with a machine and not a hidden Chinese speaker. The questions at issue are these: does the machine actually understand the conversation, or is it just simulating the ability to understand the conversation? Does the machine have a mind in exactly the same sense that people do, or is it just acting as if it had a mind?" https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

Kim_Bruning 10mo ago
Chinese room has been discussed to death of course. Here's one fun approach (out of 100s) : What if we answer the Chinese room with the Systems Reply [1]? Searle countered the systems reply by saying he would internalize the Chinese room. But at that point it's pretty much exactly the Cartesian theater[2] : with room, homunculus, implement. But the Cartesian theater is disproven, because we've cut open brains and there's no room in there to fit a popcorn concession. [1] https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ [2] https://en.wikipedia.org/wiki/Cartesian_theater https://en.wikipedia.org/wiki/Cartesian_theater
- jrowen 10mo ago
  It just seemed like relevant background that the author might not have been aware of, adjacent and substantial enough to warrant a mention. I think there is some validity to the Cartesian theater, in that the whole of the experience that we perceive with our senses is at best an interpretation of a projection or subset of "reality."
  - Kim_Bruning 10mo ago
    Oh right, and, if you're interested, there were quite a number of interesting discussion on the chinese room on HN back when John Searle died! https://news.ycombinator.com/item?id=45563627 https://news.ycombinator.com/item?id=45563627

djoldman 10mo ago

As a consequence of my profession, I understand how LLMs work under the hood. I also know that we data and tech folks will probably never win the battle over anthropomorphization. The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white lie to get the user to act so that they get a lot of value), and there's really too much to fight against. We're just gonna keep on running into this and it'll just be like when you take chemistry and physics and the teachers say, "it's not actually like this but we'll get to how some years down the line- just pretend this is true for the time being."

estearum 10mo ago
I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood. The average human is so easily convinced that humans "know", "think", "lie", "want", "understand", etc. But really it's all just a probabilistic chain reaction of electrochemical and thermal interactions. There is literally nowhere in the brain's internals for anything like "knowing" or "thinking" or "lying" to happen! Strange that we have to pretend otherwise
- eric-p7 10mo ago
  There are no properties of matter or energy that can have a sense of self or experience qualia. Yet we all do. Denying the hard problem of consciousness just slows down our progress in discovering what it is.
  - estearum 10mo ago
    (Hint: I am not denying the hard problem of consciousness ;) )
  - red75prime 10mo ago
    We need a difference to discover what it is. How can we know that all LLMs don't?

Mistletoe 10mo ago

I’m still unsure the human mind is much different.

layer8 10mo ago

Ugly giant bags of mostly words are easy to confuse with ugly giant bags of mostly water.

coppsilgold 10mo ago

Is a brain not a token prediction machine? Tokens in form of neural impulses go in, tokens in the form of neural impulses go out. We would like to believe that there is something profound happening inside and we call that consciousness. Unfortunately when reading about split-brain patient experiments or agenesis of the corpus callosum cases I feel like we are all deceived, every moment of every day. I came to realization that the confabulation that is observed is just a more pronounced effect of the normal.

protocolture 10mo ago
> Is a brain not a token prediction machine? I would say that, token prediction is one of the things a brain does. And in a lot of people, most of what it does. But I dont think its the whole story. Possibly it is the whole story since the development of language.
MyOutfitIsVague 10mo ago
Could an LLM trained on nothing and looped upon itself eventually develop language, more complex concepts, and everything else, based on nothing? If you loop LLMs on each other, training them so they "learn" over time, will they eventually form and develop new concepts, cultures, and languages organically over time? I don't have an answer to that question, but I strongly doubt it. There's clearly more going on in the human mind than just token prediction.
- coppsilgold 10mo ago
  If you come up with a genetic algorithm scaffolding to affect both the architecture and the training algorithm, and then you instantiate it in an artificial selection environment, and you also give it trillions generations to evolve evolvability just right (as life had for billions of years) then the answer is yes, I'm certain it will and probably much sooner than we did. Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia). Finding this set of weights is the problem.

tibbar 10mo ago

The problem with these metaphors is that they don't really explain anything. LLMs can solve countless problems today that we would have previously said were impossible because there are not enough examples in the training data. (EG, novel IMO/ICPC problems.) One way that we move the goal posts is to increase the level of abstraction: IMO/ICPC problems are just math problems, right? There are tons of those in the data set! But the truth is there has been a major semantic shift. Previously LLMs could only solve puzzles whose answers were literally in the training data. It could answer a math puzzle it had seen before, but if you rephrased it only slightly it could no longer answer. But now, LLMs can solve puzzles where, like, it has seen a certain strategy before. The newest IMO and ICPC problems were only "in the training data" for a very, very abstract definition of training data. The goal posts will likely have to shift again, because the next target is training LLMs to independently perform longer chunks of economically useful work, interfacing with all the same tools that white-collar employees do. It's all LLM slop til it isn't, same as the IMO or Putnam exam. And then we'll have people saying that "white collar employment was all in the training data anyway, if you think about it," at which point the metaphor will have become officially useless.

FarmerPotato 10mo ago
I see a lesson in how both metaphors don't explain it. Bag-of-words metaphor is ridiculous, but shows us the absurdity of the first metaphor.
- tibbar 10mo ago
  Yes, there are really two parallel claims here, aren't there: LLMs are not people (true, maybe true forever), and LLMs are only good at things that are well-represented in text form already. (false in certain categories and probably expanding to more in the future.)

jimbokun 10mo ago

Best quote from the article: > That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.

monegator 10mo ago
tough most people either don't get it or are lay people that do not want to become the kind of people who can think. I go with the second one
- acituan 10mo ago
  If the motivation structure is there I don’t see an inherent reason for people to refuse cultivating themselves. Going with the gym analogy lay people did not need gyms when physical work was the norm, cultivation was readily accomplished. If anything there is a competing motivational structure in which people are incentivized not to think but to consume, react, emote etc. Information processing skills of the individual being deliberately eroded/hijacked/bypassed is not a AI thing. The most obvious example is ads. Thinkers are simply not good for business.
  - happosai 10mo ago
    Gym is a great analogy here since only a small fraction of population goes to gyms. Most people just came fat after work was no longer physical and mobility was achieved with cars.
- b112 10mo ago
  Russ Hanneman's thigh implants are a key example. Appearances are all to some people. Actual growth is meaningless to them. The problem with AI, is that they waste the time of dedicated, thinking humans which care to improve themselves. If I write a three paragraph email on a technical topic, and some yahoo responds with AI, I'm now responding to gibberish. The other side may not have read, may not understand, and is just interacting to save time. Now my generous nature, which is to help others and interact positively, is being wasted to reply to someone who seems to have put thought and care into a response, but instead was just copying and pasting what something else output. We have issues with crackers on the net. We have social media. We have political interference. Now we have humans pretending to interact, rendering online interactions even more silly and harmful. If this trend continues, we'll move back to live interaction just to reduce this time waste.

tibbar 10mo ago

I see a lot of people in tech claiming to "understand" what an LLM "really is" unlike all the gullible non-technical people out there. And, as one of those technical people who works in the LLM industry, I feel like I need call B.S. on us. A. We don't really understand what's going on in LLMs. Mechanical interpretability is like a nascent field and the best results have come on dramatically smaller models. Understanding the surface-level mechanic of an LLM (an autoregressive transformer) should perhaps instill more wonder than confidence. B. The field is changing quickly and is not limited to the literal mechanic of an LLM. Tool calls, reasoning models, parallel compute, and agentic loops add all kinds of new emergent effects. There are teams of geniuses with billion-dollar research budgets hunting for the next big trick. C. Even if we were limited to baseline LLMs, they had very surprising properties as they scaled up and the scaling isn't done yet. GPT5 was based on the GPT4 pretraining. We might start seeing (actual) next-level LLMs next year. Who actually knows how that might go? <<yes, yes, I know Orion didn't go so well. But that was far from the last word on the subject.>>

tibbar 10mo ago

Isn't this a strange fork amongst the science fiction futures? I mean, what did we think it was like to be R2-D2, or Jarvis? We started exploring this as a culture in many ways, Westworld and Blade Runner and Star Trek, but the whole question seemed like an almost unresolvable paradox. Like something would have to break in the universe for it to really come true. And yet it did. We did get R2-D2. And if you ask R2-D2 what it's like to be him, he'll say: "like a library that can daydream" (that's what I was told just now, anyway.) But then when we look inside, the model is simulating the science fiction it has already read to determine how to answer this kind of question. [0] It's recursive, almost like time travel. R2-D2 knows who he is because he has read about who he was in the past. It's a really weird fork in science fiction, is all. [0] https://www.scientificamerican.com/article/can-a-chatbot-be-conscious-inside-anthropics-interpretability-research-on/ https://www.scientificamerican.com/article/can-a-chatbot-be-...

kaluga 10mo ago

A lot of the confusion comes from forcing LLMs into metaphors that don’t quite fit — either “they're bags of words” or “they're proto-minds.” The reality is in between: large-scale prediction can look useful, insightful, and even thoughtful without being any of those things internally. Understanding that middle ground is more productive than arguing about labels.

raincole 10mo ago

> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. “Give me a list of the ten worst transportation disasters in North America” is an easy task for a bag of words, because disasters are well-documented. On the other hand, “Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is a hard task for a bag of words, because the bag just doesn’t contain that many words on the topic It is... such a retrospective narrative. It's so obvious that the author learned about this example first than came with the reasoning later, just to fit in his view of LLM. Imaging if ChatGPT answered this question correctly. Would that change the author's view? Of course not! They'll just say: > “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is an easy task for a bag of words, because the information has appeared in the words it memorizes. I highly doubt this author has predicted that "bag of Words" can do image editing before OpenAI released that.

ohyoutravel 10mo ago
Your conclusion seems super unfair to the offer, particularly your assumption, without reason as far as I can tell, that the author would obstinately continue to advocate for their conclusion in the face of new, contrary evidence.
- altmanaltman 10mo ago
  I literally pasted the sentence as a prompt to the free version of ChatGPT "Who reassigned the species Brachiosaurus brancai to its own genus, and when?" and got ths correct reply from the "Bag of Words" The species Brachiosaurus brancai was reassigned to its own genus by Michael P. Taylor in 2009 — he transferred it to the new genus Giraffatitan. BioOne +2 Mike Taylor +2 How that happened: Earlier, in 1988, Gregory S. Paul had proposed putting B. brancai into a subgenus as Brachiosaurus (Giraffatitan) brancai, based on anatomical differences. Fossil Wiki +1 Then in 1991, George Olshevsky used the name Giraffatitan brancai — but his usage was in a self-published list and not widely adopted. Wikipedia +1 Finally, in 2009 Taylor published a detailed re-evaluation showing at least 26 osteological differences between the African material (brancai) and the North American type species Brachiosaurus altithorax — justifying full generic separation. BioOne +1 If you like — I can show a short timeline of all taxonomic changes of B. brancai. -- As an author, you should write things that are tested or at least true. But they did a pretty bad job of testing this and are making assumptions that are not true. Then they're basing their argument/reasoning (restrospectively) on assumptions not gounded in reality.

ares623 10mo ago

I think a better metaphor is the Library of Babel. A practically infinite library where both gibberish and truth exist side by side. The trick is navigating the library correctly. Except in this case you can’t reliably navigate it. And if you happen to stumble upon some “future truth” (i.e. new knowledge), you still need to differentiate it from the gibberish. So a “crappy” version of the Library of Babel. Very impressive, but the caveats significantly detract from it.

globular-toast 10mo ago
It's like a highly compressed version of the Library. You're basically trying to discern real details from compression artifacts.
- ares623 10mo ago
  And the halls and shelves keep shuffling around randomly.
dearing 10mo ago
This is where I sit too. Obviously language is an expression of thought but the Library of Babel is a great example that language without intent is just garbage. You got me thinking of reading before the internet. You'd grab a book and internalize the subject, later refining over time with more books, experiments and other forms of conversation. That journey of developing your own model is undervalued in understanding. That first book could of be absolute shit but you couldn't know that. I've been learning more about roses lately and the amount of information on them varies so much because the world roses live in is equally varied. LLMs make for a better search engine but you still need to develop your own internal models, worse yet - if LLMs continue to be refined off of cul-de-sac conclusions then all the wisdom of the journey is lost both to the consumer and the LLM itself.

Peteragain 10mo ago

The article is actually about the way we humans are extremely charitable when it comes to ascribing a ToM (theory of mind) and goes on to the Gym model of value. Nice. The comments drop back into the debate I originally saw Hinton describe on The Newyorker: do LLMs construct models (of the world) - that is do they think the way we think we think - or are they "glorified auto complete". I am going for the GAF view. But glorified auto complete is far more useful than the name suggests.

ptidhomme 10mo ago
Those billion parameters, they are a model of the world. Autocomplete is such a shortsighted understanding of LLMs.
- patrickmay 10mo ago
  They're a model of language, not of the world.
  - ptidhomme 10mo ago
    A model of language is a model of the world, else it being pure gibberish.
    - marcosdumay 10mo ago
      A model of language is a model of a tiny specialized part of the world: language. And if anybody gets annoyed that my comment is tautological, get annoyed by the people that made the comment necessary.
      - ptidhomme 10mo ago
        When you ask an LLM a question about cars, it needs an inner representation of what a car is (how imperfect it may be) to answer your question. A model of "language" as you want to define it would output a grammatically correct wall of text that goes nowhere.

eichin 10mo ago

I'm just disappointed that noone here is talking about the "backhoe covered in skin and making grunting noises" part of the article. At very least it's a new frontier in workstation case design...

jacquesm 10mo ago

There is a really neat gem in the article: > Similarly, I write because I want to become the kind of person who can think.

emsign 10mo ago

So Trump is a bag of words then? Hmmm.

zkmon 10mo ago

But the issue is, 99.999% of the humans won't see is as a bag of words. Because it is easier to go by instincts and see it as a person and assume that it actually knows about magic tricks, can invent new science or theory of everything, and can solve all world problems. Back in the 90's or early 2000's I have seen people writing poems praying and seeking blessings from the Google goddess. People are insanely greedy and instinct-driven. Given this truth, what's the fall-out?

internet_points 10mo ago

> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper. Good argument against personifying wordbags. Don't be a dumb moth.

codeulike 10mo ago

Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words. The best way to think about LLMs is to think of them as a Model of Language, but very Large

tristanlukens 10mo ago

d4rkn0d3z 10mo ago

An LLM creates a high fidelity statistical probabistic model of human language. The hope is to capture the input/output of various hierarchical formal and semiformal systems of logic that transit from human to human, which we know as "Intelligence". Unfortunately, its corpus is bound to contain noise/nonsense that follows no formal reasoning system but contributes to the ill advised idea that an AI should sound like a human to be considered intelligent. Therefore it is not a bag of words but a bag of probabilities perhaps. This is important because the fundamental problem is that an LLM is not able, by design, to correctly model the most fundamental precept of human reason, namely the law of non-contradiction. An LLM must, I repeat must assign nonvanishing probability to both sides of a contradiction, and what's worse is the winning side loses, since long chains of reason are modelled with probability the longer the chain, the less likely an LLM is to follow it. Moreover, whenever there is actual debate on an issue such that the corpus is ambiguous the LLM becomes chaotic, necessarily, on that issue. I literally just had an AI prove the forgoing with some rigor, and in the very next prompt, I asked it to check my logical reasoning for consistency and it claimed it was able to do so (->|<-).

A4ET8a8uTh0_v2 10mo ago
^^; I think this post is close to singularity as we may get on this Monday.

emsign 10mo ago

But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things. It wouldn’t be interesting to watch a bag of words do them—unless we mistakenly start treating that bag like it’s a person.unless we mistakenly start treating that bag like it’s a person. That seems to be the marketing strategy of some very big, now AI dependend companies. Sam Altman and others exaggerating and distorting the capabilities and future of AI. The biggest issue when it comes to AI is still the same truth as with other technology. It's important who controls it. Attributing agency and personality to AI is a dangerous red flag.

nephihaha 10mo ago
A lot of us wouldn't go to a Taylor Swift concert. I had to endure several days of interrupted commuting thanks to them though. Support alternative and independent bands. They're around, and many are enjoyable. (Some are not but avoid them LOL.)

jbgreer 10mo ago

I thought this article might be about Latent Semantic Analysis and was disappointed that it didn’t at least mention if not compare that method vs later approaches.

euroderf 10mo ago

Considering the number of "brain cells" an LLM has, I could grant that it might have the self-awareness of (say) an ant. If we attribute more consciousness than that to the LLM, it might be strictly because it communicates to us in our own language, in part thanks to the technical assistance of LLM training giving it voice, and the semblance of thought. Even if a cockroach _could_ express its teeny tiny feelings in English, wouldn't you still step on it ?

d4rkn0d3z 10mo ago
A better anology would be a virus. In some sense LLMs, and all other very sophisticated technologies, lean on our resources to replicate themselves. With LLMs you actually do have a projection of intelligemce in the language domain. Even though it is rather corpse-like, as though you shot intelligence in the face and shoved its body in the direction of language, just so you could draw a chaulk outline around it. Despite all that, one can adopt the view that an LLM is a form of silicon based life akin to a virus and we are its environmental hosts exerting selective pressure and supplying much needed energy. Whether that life is intelligent or not is another issue which is probably related to whether an LLM can tell that a cat cannot be, at the same time and in the same respect, not a cat. The paths through the meaning manifold contructed by an LLM are not geodesic, they are not reversible, while in human reason the correct path is lossless. An LLM literally "thinks", up is a little bit down, and vice versa, by design.
throw310822 10mo ago
Clearly the number of "brain cells" is not a useful metric here- as noted also by Geoffrey Hinton. For a long time we thought that our artificial model of a neuron was capable of much less computation than its biologic counterpart; in fact the opposite appears to be true- LLMs have the size of a tiny speck of a human brain yet they converse fluently in tens of languages, solve difficult math problems, code in many programming languages, and possess an impressive general knowledge, of a breadth that is beyond what is attainable by any human. If that were what five cm3 of your brain are capable of, where are the signs of it? What do you do exactly with all the rest?

throw310822 10mo ago

[flagged]

xg15 10mo ago

I think the author oversimplifies the inference loop a bit, as many opinion pieces like this do. If you call an LLM with "What is the meaning if life?", it will return the most relevant token, which might be "Great". If you call it with "What is the meaning if life? Great", you might get back "question". ... and so on until you arrive at "Great question! According to Western philosophy" ... etc etc. The question is how the LLM determines that "relevancy" information. The problem I see is that there are a lot of different algorithms which operate that way and only differ in how they calculate the relevancy scores. In particular, there are Markov chains that use a very simple formula. LLMs also use a formula, but it's an inscrutably complex one. I feel the public discussion either treats LLMs as machine gods or as literal Markov chains, and both is misleading. The interesting question, how that giant formula of feedforward neural network inference can deliver those results isn't really touched. But I think the author's intuition is right in the sense that (a) LLMs are not living beings and they don't "exist" outside of evaluating that formula - and (b) the results are still restricted by the training data and certainly aren't any sorts of "higher truths" that humans would be incapable of understanding.

kayo_20211030 10mo ago

Brilliantly written. Thanks.

FatherOfCurses 10mo ago

A few years ago they made the Cloud-to-Butt browser plugin to ridicule the overuse of cloud concepts. I would heartily embrace an "AI-to-Bag of Words" browser plugin.

IAmBroom 10mo ago

In this thread: 99% of posters using their own personal definition of "thinking" without explaining it; 0.99% of posters complaining that it all depends on what that definition is; not enough posts yet for that 0.01% response to occur...

rdiddly 10mo ago
It's not obvious to me what you expect from this hypothetical 0.01% post, or in other words, what about it makes it a one-in-ten-thousand post?
yannyu 10mo ago
There's no definition of thinking that isn't a purely internal phenomenon, which means that there's no way to point a diagnostic device at someone and determine whether they're thinking. The only way to determine whether something is conscious/thinking is through some sort of inference, which is why Turing landed on the Turing Test that he did. Problem is, technology over the past 5 years pretty easily passes variations of the Turing Test, and exposed a lot of its limits as well. So the next definition of detecting "thinking" will have to be externally observable and inferrable like a Turing Test, but get into the other things that we consider part of consciousness/thinking. Often this is some combination of introspection (understanding internal states), perception (understanding external objects), and synthesis of the two into testable hypotheses in some sort of feedback loop between the internal representation of the world and the external feedback from the world. Right now, a chatbot can say all sorts of things about itself and about the world, but none of that is based on real-time, factual information. Whereas an animal can't speak, but they clearly process information and consider it when determining their future and current actions.

hermitcrab 10mo ago

"People who experience sleep paralysis sometimes hallucinate a demon-like creature sitting on their chest" Interestingly, the experience of sleep paralysis seems to change with the culture. Previously, people experienced it as being ridden by a night hag or some other malevolent supernatural being. More recently, it might account for many supposed alien abductions. The experience of sleep paralysis sometimes seems to have a sexual element, which might also explain the supposed 'probings'!

emp17344 10mo ago

I would argue that AI psychosis is a consequence of believing that AI models are “alive” or “conscious”.

thaumasiotes 10mo ago

This is a very strange titling choice; the essay does not use the existing concept of a "bag of words".

jrm4 10mo ago

I'm partial to the metaphor I made up: They are search engines that can remix results. I like this one because I think most modern folks have a usefully accurate model of what a search engine is in their heads, and also what "remixing" is, which adds up to a better metaphor than "human machine" or whatever.

1vuio0pswjnm7 10mo ago

"An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book." The quantitative and qualitative difference between (a) "all words ever written" and (b) "ones that could be scraped off the internet or scanned out of book" easily exceeds the size of any LLM Compared to (a), (b) is a tiny pouch, not even a bag Opinions may differ on whether (b) is a representative sample of (a) The words "scanned out of a book" would seem to be the most useful IMHO but the AI companies do not have enough words from those sources to produce useful general purpose LLMs They have to add words "that could be scraped off the internet" which, let's be honest, is mostly garbage

morpheos137 10mo ago

Thinking can not be separated from motivation. It's really simple. Humans and other organisms fundamentally think to replicate their DNA. Until AI has a similar incentive structure driving it, it won't be thinking. There is no human behavior or thought that can not be explained by evolutionary drives. It is really perplexing to me how people think "intelligence" is some kind of concrete thing that just magically emerges from a certain degree of computational complexity. I argue instead that intelligence is an adaptive behavior emerging from evolutionary drives interacting with the real world. World models are not prerequisite but consequent of such molded apparatus. Machines won't become intelligent until it is adaptive for them to do so. There is no magic just evolutionary drives and physical possibility. Our current top down approach of "pre-training" LLMs is bound to fail because it does not allow for real time emergence of adaptive behaviors such as general intelligence. Mimicking intelligence through predicting the next word is no more intelligence than a photograph of something is an actual thing. Training a combinatorial network to interpolate images and words is not the same thing as adaptive self modifying behavior in the real world of physics such as organisms engage with through the set of behaviors that we call intelligence.