7 ms·

The Timmy Trap

181 points by metadat 1y ago

harpiaharpyja 1y ago
A lot of negative comments in this thread, but I think the article makes one really good point, that has had far-reaching consequences for for society before and beyond AI: humans are pretty bad at assessing intelligence.
deleted 1y ago
[deleted]
sobiolite 1y ago
The article says that LLMs don't summarize, only shorten, because... "A true summary, the kind a human makes, requires outside context and reference points. Shortening just reworks the information already in the text." Then later says... "LLMs operate in a similar way, trading what we would call intelligence for a vast memory of nearly everything humans have ever written. It’s nearly impossible to grasp how much context this gives them to play with" So, they can't summarize, because they lack context... but they also have an almost ungraspably large amount of context?
- kayodelycaon 1y ago
  They can’t summarize something that hasn’t been summarized before.
  - naikrovek 1y ago
    they can, they just can't do it well. at no point does any LLM understand what it's doing.
    - kayodelycaon 1y ago
      It can produce something that looks like a summarization based on similarly matching texts. Depending how unique the text is determines how accurate the summarization is likely to be.
    - kblissett 1y ago
      If you think they can't do this task well I encourage you to try feeding ChatGPT some long documents outside of its training cutoff and examining the results. I expect you'll be surprised!
  - timmg 1y ago
    About a year ago, I gave a film script to an LLM and asked for a summary. It was written by a friend and there was no chance it or its summary was in the training data. It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.
    - pc86 1y ago
      I'm not as cynical as others about LLMs but it's extremely unlikely that script had multiple truly novel things in it. Broken down into sufficient small pieces it's very likely every story element was present multiple times in the LLM's training data.
      - Spivak 1y ago
        I'm not sure I understand the philosophical point being made here. The LLM has "watched" a lot of movies and so understands the important parts of the original script it's presented with. Are we not describing how human media literacy works?
        pc86 1y ago
        I'm not making a philosophical point. The earlier comment is "I updated a new script and it summarized it," I was simply saying the odds of that script actually being new is very slim. Even though obviously that script or summaries of it do not exist in their entirety in the training data, its individual elements almost certainly do. So it's not really a novel (pun unintended?) summarization.
        BobaFloutist 1y ago
        The point is that if you made a point to write a completely novel script, with (content-wise, not semantically) 0 DNA in it from previous movie scripts, with an unambiguous but incoherent and unstructured plot, your average literate human would be able summarize what happened on the page, for all that they'd be annoyed and likely distressed by how unusual it was; but that an LLM would do a disproportionately bad job compared to how well they do at other things, which makes us reevaluate what they're actually doing and how they actually do it. It feels like they've mastered language, but it's looking more and more like they've actually mastered canon. Which is still impressive, but very different.
  - originalcopy 1y ago
    I'd like to see some examples of when it struggles to do summaries. There were no real examples in the text, besides one hypothetical which ChatGPT made up. I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.
- tovej 1y ago
  You can reconcile these points by considering what specific context is necessary. The author specifies "outside" context, and I would agree. The human context that's necessary for useful summaries is a model of semantic or "actual" relationships between concepts, while the LLM context is a model of a single kind of fuzzy relationship between concepts. In other words the LLM does not contain the knowledge of what the words represent.
  - neerajsi 1y ago
    > In other words the LLM does not contain the knowledge of what the words represent. This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation. If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.
- btown 1y ago
  It's an interesting philosophical question. Imagine an oracle that could judge/decide, with human levels of intelligence, how relevant a given memory or piece of information is to any given situation, and that could verbosely describe which way it's relevant (spatially, conditionally, etc.). Would such an oracle, sufficiently parallelized, be sufficient for AGI? If it could, then we could genuinely describe its output as "context," and phrase our problem as "there is still a gap in needed context, despite how much context there already is." And an LLM that simply "shortens" that context could reach a level of AGI, because the context preparation is doing the heavy lifting. The point I think the article is trying to make is that LLMs cannot add any information beyond the context they are given - they can only "shorten" that context. If the lived experience necessary for human-level judgment could be encoded into that context, though... that would be an entirely different ball game.
  - entropicdrifter 1y ago
    I agree with the thrust of your argument. IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops. For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.
- ratelimitsteve 1y ago
  I think the differentiator here might not be the context it has, but the context it has the ability to use effectively in order to derive more information about a given request.
- jchw 1y ago
  I think the real takeaway is that LLMs are very good at tasks that closely resemble examples it has in its training. A lot of things written (code, movies/TV shows, etc.) are actually pretty repetitive and so you don't really need super intelligence to be able to summarize it and break it down, just good pattern matching. But, this can fall apart pretty wildly when you have something genuinely novel...
  - gus_massa 1y ago
    Humans too. If I were too creative writing the midterm, most of my students would fail and everyone would be very unhappy.
    - card_zero 1y ago
      That's exams, not humanity.
    - BobaFloutist 1y ago
      That's because midterms are specifically supposed to assess how well you learned the material presented (or at least directed to), not your overall ability to reason. If you teach a general reasoning class, getting creative with the midterm is one thing, but if you're teaching someone how to solve differential equations, they're learning to the very edge of their ability in a given amount of time, and you present them with an example outside of what's been described, it kind of makes sense that they can't just already solve it. I mean, that's kind of the whole premise of education, that you can't just present someone with something completely outside of their experience and expect them to derive from first principles how it works.
      - throwway120385 1y ago
        I would argue that on a math midterm it's entirely reasonable to show a problem they've never seen before and test whether they've made the connection between that problem and the problems they've seen before. We did that all the time in upper division Physics.
- usefulcat 1y ago
  I think "context" is being used in different ways here. > "It’s nearly impossible to grasp how much context this gives them to play with" Here, I think the author means something more like "all the material used to train the LLM". > "A true summary, the kind a human makes, requires outside context and reference points." In this case I think that "context" means something more like actual comprehension. The author's point is that an LLM could only write something like the referenced summary by shortening other summaries present in its training set.
  - jjaksic 1y ago
    But "shortening other summaries from its training set" is not all an LLM is capable off. It can easily shorten/summarize a text it had never seen before, in a way that makes sense. Sure, it won't always summarize it the same way a human would, but if you do a double blind test where you ask people whether a summary was written by AI, a vast majority wouldn't be able to tell the difference (again this is with a completely novel text).
- cainxinth 1y ago
  This can be solved with prompting. You can say: “summarize this document, but don’t just recap, give me the big picture” or anything to that effect.
tovej 1y ago
Good article, it's been told before but it bears repeating. Also I got caught on this one kind of irrelevant point regarding the characterization of the Matrix: I would say Matrix is not just diguised as a story about escaping systems of control, it's quite clearly about oppressive systems in society, with specific reference to gender expression. Lilly Wachowski has explicitly stated that it was supposed to be an allegory for gender transition.
- kayodelycaon 1y ago
  The character Switch was supposed to have a different gender in the matrix vs real life. It’s really a shame that didn’t happen.
- altruios 1y ago
  There is nothing more free than the freedom to be who you really are. Going to rewatch the Matrix tonight.
- xg15 1y ago
  It wasn't. Switch was intended to be genderfluid, but the Matrix itself, or "logging out" of it was apparently not meant as an allegory for transitioning (though she doesn't mind the interpretation) : https://www.them.us/story/lilly-wachowski-work-in-progress-season-two-showtime https://www.them.us/story/lilly-wachowski-work-in-progress-s...
ChrisMarshallNY 1y ago
That's a great article. Scott Jenson is one of my favorite authors. He's really big on integrating an understanding of basic human nature, into design.
snozolli 1y ago
Regarding Timmy, the Companion Cube from the game Portal is the greatest example of induced anthropomorphism that I've ever experienced. If you know, you know, and if you don't, you should really play the game, since it's brilliant.
- ChrisMarshallNY 1y ago
  I'm on a Mac, and would love to see Portal 2 (at least) ported to M-chips. I would love Portal 3, even more.
- generationP 1y ago
  The cube doesn't work, or at least it didn't for me. The goggly eyes really do make a difference.
- bitwize 1y ago
  That's a matter of informed anthropomorphism. A lot of people don't become attached to the Companion Cube, but are informed that their player character is so attached.
- andrewla 1y ago
  It is a brilliant game and the empathy you develop for the cube is a great concept. But arguably much deeper is the fact that nothing in this game, or any single-player game, is a living thing in any form. Arguably the game's characterization of GLaDOS hits even harder on the anthropomorphism angle.
- aidenn0 1y ago
  Floyd in Planetfall was that for those of us that are older.
School-Cotton 1y ago
The article claims (without any evidence, argument or reason) that LLMs are not intelligent, then simply refuses to define intelligence. How do you know LLMs aren't intelligent, if you can't define what that means?
- energy123 1y ago
  It's strange seeing so many takes like this two weeks after LLMs won gold medals at IMO and IOI. The cognitive dissonance is going to be wild when it all comes to a head in two years.
  - oytis 1y ago
    I've seen these claims, and Google even published the texts of the solutions, but it still didn't published the full log of interaction between the model and operator.
  - aprilthird2021 1y ago
    IBM Watson won Jeopardy years ago, was it intelligent?
    - perching_aix 1y ago
      > Rather than being given questions, contestants are instead given general knowledge clues in the form of answers and they must identify the person, place, thing, or idea that the clue describes, phrasing each response in the form of a question. [0] Doesn't sound like a test of intelligence to me, so no. [0] https://en.wikipedia.org/wiki/Jeopardy https://en.wikipedia.org/wiki/Jeopardy!
      - aprilthird2021 1y ago
        Why? Computers also won chess years ago, but they're not intelligent either? Why is winning a math competition intelligent but a trivia competition or a chess competition not intelligent?
naikrovek 1y ago
I've mentioned this to colleagues at work before. LLMs give a very strong appearance of intelligence, because humans are super receptive to information provided via our native language. We often have to deal with imperfect speakers and writers, and we must infer context and missing information on our own. We do this so well that we don't know we're doing it. LLMs have perfect grammar and we subtly feel that they are extremely smart because subconsciously we recognize that we don't have to think about anything that's said, it is all syntactically perfect. So, LLMs sort of trick us into masking their true limitations and believing that they are truly thinking; there are even models that call themselves thinking models, but they don't think, they just predict what the user is going to complain about and say that to themselves as an additional, dynamic prompt on top of the one you actually enter. LLMs are very good at fooling us into the idea that they know anything at all; they don't. And humans are very bad at being discriminate about the source of the information presented to them if it is presented in a friendly way. The combination of those things is what has resulted in the insanely huge AI hype cycle that we are currently living in the middle of. Nearly everyone is overreacting to what LLMs actually are, and the few of us that believe that we sort of see what's actually happening are ignored for being naysayers, buzz-kills, and luddites. Shunned for not drinking the Kool-Aid.
- hnfong 1y ago
  You're ignored not because you're right or wrong, but because your unsolicited advice is not useful. For example, I can spin up any LLM and get it to translate some English text into Japanese with maybe 99% accuracy. I don't need to believe whether it "really knows" English or Japanese, I only need to believe the output is accurate. Similarly I can ask a LLM to code up a function that does a specific thing, and it will do it with high accuracy. Maybe there'll be some bugs, but I can review the code and fix them, which in some cases boosts my productivity. I don't need to believe whether it "really knows" C++ or Rust, I only need it to write something good enough. I mean, just by these two examples, LLMs are really great tools, and I'm personally hyped for these use cases alone. Am I fooled by the LLM? I don't think so, I don't have any fantasy about it being extremely intelligent or always being right. I doubt most reasonable people these days would either. So basically you're going about assuming people are fooled by LLMs (which they might not be), and wondering why you're unpopular when you're basically telling everyone they're gullible and foolish.
codeulike 1y ago
Well I, for one, can't beleive what that guy did to poor Timmy
- beezle 1y ago
  When I saw the post title I immediately thought of Timmy from South Park lol
hackyhacky 1y ago
> LLMs mimic intelligence, but they aren’t intelligent. I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? I see two possibilities: 1. We define "intelligence" as definitionally unique to humans. For example, maybe intelligence depends on the existence of a human soul, or specific to the physical structure of the human brain. In this case, a machine (perhaps an LLM) could achieve "quacks like a duck" behavioral equality to a human mind, and yet would still be excluded from the definition of "intelligent." This definition is therefore not useful if we're interested in the ability of the machine, which it seems to me we are. LLMs are often dismissed as not "intelligent" because they work by inferring output based on learned input, but that alone cannot be a distinguishing characteristic, because that's how humans work as well. 2. We define "intelligence" in a results-oriented way. This means there must be some specific test or behavioral standard that a machine must meet in order to become intelligent. This has been the default definition for a long time, but the goal posts have shifted. Nevertheless, if you're going to disparage LLMs by calling them unintelligent, you should be able to cite a specific results-oriented failure that distinguishes them from "intelligent" humans. Note that this argument cannot refer to the LLMs' implementation or learning model.
- dkdcio 1y ago
  > I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? the ability for long-term planning and, more cogently, actually living in the real world where time passes
  - hackyhacky 1y ago
    > the ability for long-term planning and, more cogently, actually living in the real world where time passes 1. LLMs seem to be able to plan just fine. 2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
stefanv 1y ago
What if the problem is not that we overestimate LLMs, but that we overestimate intelligence? Or to express the same idea for a more philosophically inclined audience, what if the real mistake isn’t in overestimating LLMs, but in overestimating intelligence itself by imagining it as something more than a web of patterns learned from past experiences and echoed back into the world?
- justinlivi 1y ago
  I think AI skeptics have a strong bias to assume that human intelligence fundamentally functions differently from LLMs. They may be correct, but we don't have a strong enough understanding of human cognition to make the claim in as uncertain terms as the skeptical argument is unusually made. The training methods between human learning and machine learning are obviously fundamentally vastly different as are the infrastructure-level mechanics. These elements are likely never going to align, though with time the machine infrastructure may start to increasingly resemble human bio hardware. I bring this up because these known vast differences may account for a significant portion of the differences in expected output from human and machine processing. We don't understand the fundamental conceptual "black box" portions of either form of processing well enough to state definitely what is similar or dissimilar about those hazy areas. Somewhere within that not-well-understood area is what we collectively have vaguely defined "intelligence." But also within that area are all the other aspects that both humans and now machines are quite good at - prediction, fluency, translation. The challenge of lexicon and definition is potentially as difficult a task as is sharpening the focus of our understanding of the hazy black-box portion of both machine processing as well as human processing. Until all those are better defined I don't think we have a good measure for answering the question of machine intelligence either way.
- gowld 1y ago
  LLM's fail because their input data is limited in dimension compared to humans (text, pictures, audio, video) and because their capacity to rewire their own brains is limited to the transformer architecture.
pbw 1y ago
LLM's can shorten and maybe tend to if you just say "summarize this" but you can trivially ask them to do more. I asked for a summary of Jenson's post and then offer a reflection, GPT-5 said, "It's similar to the Plato’s Cave analogy: humans see shadows (the input text) and infer deeper reality (context, intent), while LLMs either just recite shadows (shorten) or imagine creatures behind them that aren’t there (hallucinate). The “hallucination” behavior is like adding “ghosts”—false constructs that feel real but aren’t grounded. That ain't shortening because none of that was in his post.
- deleted 1y ago
  [deleted]
- pitpatagain 1y ago
  I can't decide how to read your last sentence. That reflection seems totally off to me: fluent, and flavored with elements of the article, but also not really what the article is about and a pretty weird/tortured use of the elements of the allegory of the cave, like it doesn't seem anything like Plato's Cave to me. Ironically demonstrates the actual main gist of the article if you ask me. But maybe you meant that you think that summary is good and not textually similar to that post so demonstrating something more sophisticated than "shortening".
  - pbw 1y ago
    Yes, GPT-5's response above was not shortening because there was nothing in the OP about Plato's Cave. I agree that Plato's cave analogy was confusing here. Here's a better one from GPT-5, which is deeply ironic: A New Yorker book review often does the opposite of mere shortening. The reviewer: * Places the book in a broader cultural, historical, or intellectual context. * Brings in other works—sometimes reviewing two or three books together. * Builds a thesis that connects them, so the review becomes a commentary on a whole idea-space, not just the book’s pages. This is exactly the kind of externalized, integrative thinking Jenson says LLMs lack. The New Yorker style uses the book as a jumping-off point for an argument; an LLM “shortening” is more like reading only the blurbs and rephrasing them. In Jenson’s framing, a human summary—like a rich, multi-book New Yorker review—operates on multiple layers: it compresses, but also expands meaning by bringing in outside information and weaving a narrative. The LLM’s output is more like a stripped-down plot synopsis—it can sound polished, but it isn’t about anything beyond what’s already in the text.
AndrewKemendo 1y ago
Who are you going to lodge your complaint to that the set of systems and machines that just took your job isn’t “intelligent?” Humans seem to get wrapped around these concepts like intelligence consciousness etc. because they seem to be the only thing differentiating us from every other animal when in fact it’s all a mirage.
kbaker 1y ago
Seems like this is close to the Uncanny Valley effect. LLM intelligence is in the spot where it is simultaneously genius-level but also just misses the mark a tiny bit, which really sticks out for those who have been around humans their whole lives. I feel that, just like more modern CGI, this will slowly fade with certain techniques and you just won't notice it when talking to or interacting with AI. Just like in his post during the whole Matrix discussion. > "When I asked for examples, it suggested the Matrix and even gave me the “Summary” and “Shortening” text, which I then used here word for word. " He switches in AI-written text and I bet you were reading along just the same until he pointed it out. This is our future now I guess.
- gowld 1y ago
  Shades of Jonathan Lethem's classic "The Ecstasy of Influence: A plagiarism" https://harpers.org/archive/2007/02/the-ecstasy-of-influence/ https://harpers.org/archive/2007/02/the-ecstasy-of-influence...
foobarian 1y ago
The LLMs are like a Huffman codec except the context is infinite and lossy
Isamu 1y ago
You can compare the current state of LLMs to the days of chess machines when they first approached grandmaster level play. The machine approach was very brute force, and there was a lot of work done to improve the sheer amount of look ahead that was required to complete at the grandmaster level. As opposed to what grandmasters actually did, which was less look ahead and more pattern matching to strengthen the position. Now LLMs successfully leverage pattern matching, but interestingly it is still a kind of brute force pattern matching, requiring the statistical absorption of all available texts, far more than a human absorbs in a lifetime. This enables the LLM to interpolate an answer from the structure of the absorbed texts with reasonable statistical relevance. This is still not quite “what humans do” as it still requires brute force statistical analysis of vast amounts of text to achieve pretty good results. For example training on all available Python sources in github and elsewhere (curated to avoid bad examples) yields pretty good results, not how a human would do it, but statistically likely to be pertinent and correct.
- gowld 1y ago
  One key difference is that an LLM really is better than any human at almost everything, simply because the LLM is kind of OKish at millions of things that each single human is just terrible at or completely ignorant of. The common LLMs are not trying to be a better human than a single human. They are trying to be more useful per $ cost than a bevy of humand doing some basket of human tasks. Making a replacement for an individual complete human, an effort that was popular before the Internet made that idea seem quaint, is very different challenge, and more niche, because it's not as economically efficient.
xg15 1y ago
I feel this article should be paired with this other one [1] that was on the frontpage a few days ago. My impression is, there is currently one tendency to "over-anthropomorphize" LLMs and treat them like conscious or even superhuman entities (encouraged by AI tech leaders and AGI/Singularity folks) and another to oversimplify them and view them as literal Markov chains that just got lots of training data. Maybe those articles could help guarding against both extremes. [1] https://www.verysane.ai/p/do-we-understand-how-neural-networks https://www.verysane.ai/p/do-we-understand-how-neural-networ...
- mattgreenrocks 1y ago
  Previously when someone called out the tendency to over-anthropomorphize LLMs, a lot of the answers amounted to, “but I like doing it, therefore we should!” I’ll be the first to say one should pick their battles. But hearing that over and over from a crowd like this that can be quite pedantic is very telling.
  - tempodox 1y ago
    This very comment thread demonstrates how utterly hopeless it is trying to educate the believers. It has developed into a full-blown religion by now.
nojs 1y ago
Even stronger than our need to anthropomorphize seems to be our innate desire to believe our species is special, and that “real intelligence” couldn’t ever be replicated. If you keep redefining real intelligence as the set of things machines can’t do, then it’s always going to be true.
- safetytrick 1y ago
  Yes, I agree, we seem to need to feel "special". Language is really powerful, I think it's a huge part of our intelligence. The interesting part of the article to me is the focus on fluency. I have not seen anything that LLMs do well that isn't related to powerful utilization of fluency.
ArnavAgrawal03 1y ago
> They had known him for only 15 seconds, yet they still perceived the act of snapping him in half as violent. This is right out of Community
- WorkerBee28474 1y ago
  Clip from s01e01: https://www.youtube.com/watch?v=z906aLyP5fg https://www.youtube.com/watch?v=z906aLyP5fg
vcarrico 1y ago
I might be mixing the concepts of intelligence and conscience etc, but the human mind is more than language and data; it's also experience. LLMs have all the data and can express anything around that context, but will never experience anything, which is singular for each of us, and it's part of what makes what we call intelligence (?). So they will never replicate the human mind; they can just mimic it. I heard from Miguel Nicolelis that language is a filter for the human mind, so you can never build a mind from language. I interpreted this like trying to build an orange from its juice.
- hackyhacky 1y ago
  > LLMs have all the data and can express anything around that context, but will never experience anything, On the contrary, all their training data is their "experience".
nataliste 1y ago
The author's argument is built on fallacies that always pop up in these kinds of critiques. The "summary vs shortening" distinction is moving the goalposts. They makes the empirical claim that LLMs fail at summarizing novel PDFs without any actual evidence. For a model trained on a huge chunk of the internet, the line between "reworking existing text" and "drawing on external context" is so blurry it's practically meaningless. Similarly, can we please retire the ELIZA and Deep Blue analogies? Comparing a modern transformer to a 1960s if-then script or a brute-force chess engine is a category error. It's a rhetorical trick to make LLMs seem less novel than they actually are. And blaming everything on anthropomorphism is an easy out. It lets you dismiss the model's genuinely surprising capabilities by framing it as a simple flaw in human psychology. The interesting question isn't that we anthropomorphize, but why this specific technology is so effective at triggering that response from humans. The whole piece basically boils down to: "If we define intelligence in a way that is exclusively social and human, then this non-social, non-human thing isn't intelligent." It's a circular argument.
ticulatedspline 1y ago
- LLMs don't need to be intelligent to take jobs, bash scripts have replaced people. - Even if CEOs are completely out of touch and the tool can't do the job you can still get laid off in an ill informed attempt to replace you. Then when the company doesn't fall over because the leftover people, desperate to keep covering rent fill the gaps it just looks like efficiency to the top. - I don't think our tendency anthropomorphize LLMs is really the problem here.
andoando 1y ago
This, along with a ton of commentary on LLMs, seems like its written by someone who has no technical understanding of LLMs.
Joeri 1y ago
LLMs mimic intelligence, but they aren’t intelligent. They aren’t just intelligence mimics, they are people mimics, and they’re getting better at it with every generation. Whether they are intelligent or not, whether they are people or not, it ultimately does not matter when it comes to what they can actually do, what they can actually automate. If they mimic a particular scenario or human task well enough that the job gets done, they can replace intelligence even if they are “not intelligent”. If by now someone still isn’t convinced that LLMs can indeed automate some of those intelligence tasks, then I would argue they are not open to being convinced.
- shafoshaf 1y ago
  They can mimic well documented behavior. Applying an LLM to a novel task is where the model breaks down. This obviously has huge implications for automation. For example, most business do not have unique ways of handling accounting transactions, yet each company has a litany of AR and AP specialists who create semmingly unique SOPs. LLMs can easily automate those workers since they are simply doing a slight variation at best of a very well documented system. Asking an LLM to take all this knowledge and apply it to a new domain? That will take a whole new paradigm.
  - quesera 1y ago
    Absolutely agreed, but I suspect that a whole lot of what humans do every day can be reduced to pattern-following. If/when LLMs or other AIs can create novel work / discover new knowledge, they will be "genius" in the literal sense of the word. More genius would be great! (probably) . But genius is not required for the vast majority of tasks.
  - andrewla 1y ago
    > Applying an LLM to a novel task is where the model breaks down I mean, don't most people break down in this case too? I think this needs to be more precise. What is the specific task that you think can reliably distinguish between an LLM's capability in this sense vs. what a human can typically manage? That is, in the sense of [1], what is the result that we're looking to use to differentiate. [1] https://news.ycombinator.com/item?id=44913498 https://news.ycombinator.com/item?id=44913498
0x457 1y ago
> A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. How is that a summary? It reads as a one-liner review I would leave on Letterboxed or something I would say, trying to be pretentious and treating the movie as a work of art. It is a work of art, because all movies are art, but that's an awful summary.
- pwdisswordfishz 1y ago
  Review? Where does it say anything about the quality of the film?
  - 0x457 1y ago
    * * * * (4/5) A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. ----- Do I need to explain more?
deleted 1y ago
[deleted]
intalentive 1y ago
Good point about the Turing Test: >The original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. By the original meaning of the test it's easy to tell an LLM from a human.
jhanschoo 1y ago
I disagree with the author in a big way. '25's LLMs are designed to echo material already out there on the Internet if it exists because we value that more. If I want a summary of The Matrix, I prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary that requires a justification as to its deviation. In fact, the example provided by the author is a great illustration of this: > A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. The words here refer back to the notions "free will" that is prominent in Western discourse from St. Augustine through Descartes and thereafter and similarly of "sci-fi". These are notions an uneducated East-Asian with limited Internet use and pop culture fluency will simply not understand. They would in fact prefer the latter description. The author and this hypothetical East-Asian live in very different zeitgeists, and correspondingly experience the movie differently and value different summaries of the film. They each prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary (relative to their zeitgeist) that requires a justification as to its deviation. On the other hand, if you asked LLMs to explain material and concepts, one modality in which it does is use formulaic, novel, and unorthodox analogies to explain it to you. By formulaic and novel, I mean that the objects and scenarios in the analogies are frequently of a certain signature kind that it has been trained with, but it is novel in that the analogies are not found in the wild on the internet. If you have frequently used LLMs for the purpose of explaining concepts, you will have encountered these analogies and know what I mean by this. The analogies are frequently not too accurate, but they round out the response by giving an ELI5-style answer. Ironically, the author may have succumbed to LLM sycophancy.
justsomedumbass 1y ago
Intelligence is a social construct? Interesting thought, but denies the innovator credit for their efforts. Does technology naturally evolve in an 'intelligent' society? I don't think so. Now without somebody going deep into the weeds to find gold. Without individual genius, we would be far behind in our collective progress.
jstgunderscore 1y ago
Firstly, I think a lot of commenters here should ask themselves "Do you believe that ANY machine could EVER be intelligent?" Unless you believe in the magic sauce of a supernatural soul/mind/etc, our brains function as deterministic biological machines. There's essentially nothing that separates the processing and memory potential of silicon from neurons. And if the building blocks can be made analogous, then all the same emergent properties are possible. There's no reason to believe a circuit couldn't be made to behave in exactly the same way as a human brain. I'm not saying that's where LLMs are; only that it is theoretically possible. So if you imagine such a machine, and you deem that it is not intelligent, you have reserved intelligence as an exclusive human trait and this entire discussion is meaningless. Secondly, although I'm not in either pro or anti LLM-intelligence camp, I find a lot of the arguments against machine intelligence disingenuous and/or unbalanced. For instance the "Can't process information it's not familiar with" argument. Another commenter stated the case of scientific papers that it doesn't have any reference for, that it may hallucinate a garbage interpretation of the paper. Not surprising, but guess what, a human would do the same thing if they were forced! Imagine holding a gun to someone's head and telling them to explain a concept or system they've never heard of. That's essentially what we're doing with LLMs; obviously we don't need to threaten, because we haven't given them agency to say no. Another example is the "Can't be novel, unique, or create something completely new." First of all, difficult to prove, but okay let's take it as given that an LLM can't be novel. Can you prove that a human can? We make all these assumptions on how intelligent and creative we are as humans, and how original our thoughts can be... but how original are they, and can we prove it? How do you know your original thought, or Beethoven's 5th, or the fast inverse square root trick was completely separate from any prior influences? Or... was that "original thought" the conglomeration of a thousand smaller inputs and data points that you trained on, that became part of your brains subconscious processing system, and came together in a synthesis that looks like brilliance. Finally, whenever this discussion comes up with friends I ask them to think of the least intelligent person they know. Then imagine how many of those there are in the world (likely millions). Could you imagine any conceivable test of any length or depth which would designate all the those humans as intelligent, and all the LLMs as not? I certainly can't. I highly doubt there could be anything approaching 100% accuracy at this point. Ultimately I think we should ditch both the intelligence and consciousness questions. We can't define them in ourselves, we certainly can't define them in another entity. Let's just come to terms with the fact we're highly functioning biological machines who are both scared and excited to have created something so similar to us.
jjaksic 1y ago
There's a lot of moving the goal posts here. What would it mean for an AI to be considered "intelligent"? At various points it meant different things (chess, jeopardy, go, human language, math, etc). The most well-established benchmark was the Turing test, which LLMs now pass spectacularly -- in fact the only way to reliably tell that you're talking to an AI is usually because it responds too fast and too perfectly. Sure, AI often makes mistakes, but so do humans. Ask a random person a random question and I guarantee that at least 90% of the times an LLM will produce a better answer. So if knowledge or reasoning mistakes are a benchmark, then humans are not intelligent either. What is the difference then? I feel that whoever claims that AI is not intelligent simply feels uncomfortable thinking that they are, and is going out of their way to rationalize that, by using various double standards. Like, AI making occasional mistakes counts as evidence against it, but the fact that the vast majority of people are incapable of answering some of the most basic questions somehow doesn't count against human intelligence. And then there are various contrived "definitions of intelligence", such as "living in the physical world". These have nothing to do with cognitive intelligence and are just made up to make humans fit the definition and AI to not fit. Doing "truly novel things"? How many people have ever come up with anything truly novel? What about the other 99.99%, are they intelligent? I think there's one thing that's tripping people up. Humans do have a very high COLLECTIVE intelligence. Collectively (and given enough time) we're capable of accomplishing great things, much more than current AI. But each individual human on their own is no more intelligent than a caveman (we have essentially the same brain).

181 points by metadat 1y ago

A lot of negative comments in this thread, but I think the article makes one really good point, that has had far-reaching consequences for for society before and beyond AI: humans are pretty bad at assessing intelligence.

deleted 1y ago

[deleted]

sobiolite 1y ago

The article says that LLMs don't summarize, only shorten, because... "A true summary, the kind a human makes, requires outside context and reference points. Shortening just reworks the information already in the text." Then later says... "LLMs operate in a similar way, trading what we would call intelligence for a vast memory of nearly everything humans have ever written. It’s nearly impossible to grasp how much context this gives them to play with" So, they can't summarize, because they lack context... but they also have an almost ungraspably large amount of context?

kayodelycaon 1y ago
They can’t summarize something that hasn’t been summarized before.
- naikrovek 1y ago
  they can, they just can't do it well. at no point does any LLM understand what it's doing.
  - kayodelycaon 1y ago
    It can produce something that looks like a summarization based on similarly matching texts. Depending how unique the text is determines how accurate the summarization is likely to be.
  - kblissett 1y ago
    If you think they can't do this task well I encourage you to try feeding ChatGPT some long documents outside of its training cutoff and examining the results. I expect you'll be surprised!
- timmg 1y ago
  About a year ago, I gave a film script to an LLM and asked for a summary. It was written by a friend and there was no chance it or its summary was in the training data. It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.
  - pc86 1y ago
    I'm not as cynical as others about LLMs but it's extremely unlikely that script had multiple truly novel things in it. Broken down into sufficient small pieces it's very likely every story element was present multiple times in the LLM's training data.
    - Spivak 1y ago
      I'm not sure I understand the philosophical point being made here. The LLM has "watched" a lot of movies and so understands the important parts of the original script it's presented with. Are we not describing how human media literacy works?
      - pc86 1y ago
        I'm not making a philosophical point. The earlier comment is "I updated a new script and it summarized it," I was simply saying the odds of that script actually being new is very slim. Even though obviously that script or summaries of it do not exist in their entirety in the training data, its individual elements almost certainly do. So it's not really a novel (pun unintended?) summarization.
      - BobaFloutist 1y ago
        The point is that if you made a point to write a completely novel script, with (content-wise, not semantically) 0 DNA in it from previous movie scripts, with an unambiguous but incoherent and unstructured plot, your average literate human would be able summarize what happened on the page, for all that they'd be annoyed and likely distressed by how unusual it was; but that an LLM would do a disproportionately bad job compared to how well they do at other things, which makes us reevaluate what they're actually doing and how they actually do it. It feels like they've mastered language, but it's looking more and more like they've actually mastered canon. Which is still impressive, but very different.
- originalcopy 1y ago
  I'd like to see some examples of when it struggles to do summaries. There were no real examples in the text, besides one hypothetical which ChatGPT made up. I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.
tovej 1y ago
You can reconcile these points by considering what specific context is necessary. The author specifies "outside" context, and I would agree. The human context that's necessary for useful summaries is a model of semantic or "actual" relationships between concepts, while the LLM context is a model of a single kind of fuzzy relationship between concepts. In other words the LLM does not contain the knowledge of what the words represent.
- neerajsi 1y ago
  > In other words the LLM does not contain the knowledge of what the words represent. This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation. If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.
btown 1y ago
It's an interesting philosophical question. Imagine an oracle that could judge/decide, with human levels of intelligence, how relevant a given memory or piece of information is to any given situation, and that could verbosely describe which way it's relevant (spatially, conditionally, etc.). Would such an oracle, sufficiently parallelized, be sufficient for AGI? If it could, then we could genuinely describe its output as "context," and phrase our problem as "there is still a gap in needed context, despite how much context there already is." And an LLM that simply "shortens" that context could reach a level of AGI, because the context preparation is doing the heavy lifting. The point I think the article is trying to make is that LLMs cannot add any information beyond the context they are given - they can only "shorten" that context. If the lived experience necessary for human-level judgment could be encoded into that context, though... that would be an entirely different ball game.
- entropicdrifter 1y ago
  I agree with the thrust of your argument. IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops. For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.
ratelimitsteve 1y ago
I think the differentiator here might not be the context it has, but the context it has the ability to use effectively in order to derive more information about a given request.
jchw 1y ago
I think the real takeaway is that LLMs are very good at tasks that closely resemble examples it has in its training. A lot of things written (code, movies/TV shows, etc.) are actually pretty repetitive and so you don't really need super intelligence to be able to summarize it and break it down, just good pattern matching. But, this can fall apart pretty wildly when you have something genuinely novel...
- gus_massa 1y ago
  Humans too. If I were too creative writing the midterm, most of my students would fail and everyone would be very unhappy.
  - card_zero 1y ago
    That's exams, not humanity.
  - BobaFloutist 1y ago
    That's because midterms are specifically supposed to assess how well you learned the material presented (or at least directed to), not your overall ability to reason. If you teach a general reasoning class, getting creative with the midterm is one thing, but if you're teaching someone how to solve differential equations, they're learning to the very edge of their ability in a given amount of time, and you present them with an example outside of what's been described, it kind of makes sense that they can't just already solve it. I mean, that's kind of the whole premise of education, that you can't just present someone with something completely outside of their experience and expect them to derive from first principles how it works.
    - throwway120385 1y ago
      I would argue that on a math midterm it's entirely reasonable to show a problem they've never seen before and test whether they've made the connection between that problem and the problems they've seen before. We did that all the time in upper division Physics.
usefulcat 1y ago
I think "context" is being used in different ways here. > "It’s nearly impossible to grasp how much context this gives them to play with" Here, I think the author means something more like "all the material used to train the LLM". > "A true summary, the kind a human makes, requires outside context and reference points." In this case I think that "context" means something more like actual comprehension. The author's point is that an LLM could only write something like the referenced summary by shortening other summaries present in its training set.
- jjaksic 1y ago
  But "shortening other summaries from its training set" is not all an LLM is capable off. It can easily shorten/summarize a text it had never seen before, in a way that makes sense. Sure, it won't always summarize it the same way a human would, but if you do a double blind test where you ask people whether a summary was written by AI, a vast majority wouldn't be able to tell the difference (again this is with a completely novel text).
cainxinth 1y ago
This can be solved with prompting. You can say: “summarize this document, but don’t just recap, give me the big picture” or anything to that effect.

tovej 1y ago

Good article, it's been told before but it bears repeating. Also I got caught on this one kind of irrelevant point regarding the characterization of the Matrix: I would say Matrix is not just diguised as a story about escaping systems of control, it's quite clearly about oppressive systems in society, with specific reference to gender expression. Lilly Wachowski has explicitly stated that it was supposed to be an allegory for gender transition.

kayodelycaon 1y ago
The character Switch was supposed to have a different gender in the matrix vs real life. It’s really a shame that didn’t happen.
altruios 1y ago
There is nothing more free than the freedom to be who you really are. Going to rewatch the Matrix tonight.
xg15 1y ago
It wasn't. Switch was intended to be genderfluid, but the Matrix itself, or "logging out" of it was apparently not meant as an allegory for transitioning (though she doesn't mind the interpretation) : https://www.them.us/story/lilly-wachowski-work-in-progress-season-two-showtime https://www.them.us/story/lilly-wachowski-work-in-progress-s...

ChrisMarshallNY 1y ago

That's a great article. Scott Jenson is one of my favorite authors. He's really big on integrating an understanding of basic human nature, into design.

snozolli 1y ago

Regarding Timmy, the Companion Cube from the game Portal is the greatest example of induced anthropomorphism that I've ever experienced. If you know, you know, and if you don't, you should really play the game, since it's brilliant.

ChrisMarshallNY 1y ago
I'm on a Mac, and would love to see Portal 2 (at least) ported to M-chips. I would love Portal 3, even more.
generationP 1y ago
The cube doesn't work, or at least it didn't for me. The goggly eyes really do make a difference.
bitwize 1y ago
That's a matter of informed anthropomorphism. A lot of people don't become attached to the Companion Cube, but are informed that their player character is so attached.
andrewla 1y ago
It is a brilliant game and the empathy you develop for the cube is a great concept. But arguably much deeper is the fact that nothing in this game, or any single-player game, is a living thing in any form. Arguably the game's characterization of GLaDOS hits even harder on the anthropomorphism angle.
aidenn0 1y ago
Floyd in Planetfall was that for those of us that are older.

School-Cotton 1y ago

The article claims (without any evidence, argument or reason) that LLMs are not intelligent, then simply refuses to define intelligence. How do you know LLMs aren't intelligent, if you can't define what that means?

energy123 1y ago
It's strange seeing so many takes like this two weeks after LLMs won gold medals at IMO and IOI. The cognitive dissonance is going to be wild when it all comes to a head in two years.
- oytis 1y ago
  I've seen these claims, and Google even published the texts of the solutions, but it still didn't published the full log of interaction between the model and operator.
- aprilthird2021 1y ago
  IBM Watson won Jeopardy years ago, was it intelligent?
  - perching_aix 1y ago
    > Rather than being given questions, contestants are instead given general knowledge clues in the form of answers and they must identify the person, place, thing, or idea that the clue describes, phrasing each response in the form of a question. [0] Doesn't sound like a test of intelligence to me, so no. [0] https://en.wikipedia.org/wiki/Jeopardy https://en.wikipedia.org/wiki/Jeopardy!
    - aprilthird2021 1y ago
      Why? Computers also won chess years ago, but they're not intelligent either? Why is winning a math competition intelligent but a trivia competition or a chess competition not intelligent?

naikrovek 1y ago

I've mentioned this to colleagues at work before. LLMs give a very strong appearance of intelligence, because humans are super receptive to information provided via our native language. We often have to deal with imperfect speakers and writers, and we must infer context and missing information on our own. We do this so well that we don't know we're doing it. LLMs have perfect grammar and we subtly feel that they are extremely smart because subconsciously we recognize that we don't have to think about anything that's said, it is all syntactically perfect. So, LLMs sort of trick us into masking their true limitations and believing that they are truly thinking; there are even models that call themselves thinking models, but they don't think, they just predict what the user is going to complain about and say that to themselves as an additional, dynamic prompt on top of the one you actually enter. LLMs are very good at fooling us into the idea that they know anything at all; they don't. And humans are very bad at being discriminate about the source of the information presented to them if it is presented in a friendly way. The combination of those things is what has resulted in the insanely huge AI hype cycle that we are currently living in the middle of. Nearly everyone is overreacting to what LLMs actually are, and the few of us that believe that we sort of see what's actually happening are ignored for being naysayers, buzz-kills, and luddites. Shunned for not drinking the Kool-Aid.

hnfong 1y ago
You're ignored not because you're right or wrong, but because your unsolicited advice is not useful. For example, I can spin up any LLM and get it to translate some English text into Japanese with maybe 99% accuracy. I don't need to believe whether it "really knows" English or Japanese, I only need to believe the output is accurate. Similarly I can ask a LLM to code up a function that does a specific thing, and it will do it with high accuracy. Maybe there'll be some bugs, but I can review the code and fix them, which in some cases boosts my productivity. I don't need to believe whether it "really knows" C++ or Rust, I only need it to write something good enough. I mean, just by these two examples, LLMs are really great tools, and I'm personally hyped for these use cases alone. Am I fooled by the LLM? I don't think so, I don't have any fantasy about it being extremely intelligent or always being right. I doubt most reasonable people these days would either. So basically you're going about assuming people are fooled by LLMs (which they might not be), and wondering why you're unpopular when you're basically telling everyone they're gullible and foolish.

codeulike 1y ago

Well I, for one, can't beleive what that guy did to poor Timmy

beezle 1y ago
When I saw the post title I immediately thought of Timmy from South Park lol

hackyhacky 1y ago

> LLMs mimic intelligence, but they aren’t intelligent. I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? I see two possibilities: 1. We define "intelligence" as definitionally unique to humans. For example, maybe intelligence depends on the existence of a human soul, or specific to the physical structure of the human brain. In this case, a machine (perhaps an LLM) could achieve "quacks like a duck" behavioral equality to a human mind, and yet would still be excluded from the definition of "intelligent." This definition is therefore not useful if we're interested in the ability of the machine, which it seems to me we are. LLMs are often dismissed as not "intelligent" because they work by inferring output based on learned input, but that alone cannot be a distinguishing characteristic, because that's how humans work as well. 2. We define "intelligence" in a results-oriented way. This means there must be some specific test or behavioral standard that a machine must meet in order to become intelligent. This has been the default definition for a long time, but the goal posts have shifted. Nevertheless, if you're going to disparage LLMs by calling them unintelligent, you should be able to cite a specific results-oriented failure that distinguishes them from "intelligent" humans. Note that this argument cannot refer to the LLMs' implementation or learning model.

dkdcio 1y ago
> I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? the ability for long-term planning and, more cogently, actually living in the real world where time passes
- hackyhacky 1y ago
  > the ability for long-term planning and, more cogently, actually living in the real world where time passes 1. LLMs seem to be able to plan just fine. 2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.

stefanv 1y ago

What if the problem is not that we overestimate LLMs, but that we overestimate intelligence? Or to express the same idea for a more philosophically inclined audience, what if the real mistake isn’t in overestimating LLMs, but in overestimating intelligence itself by imagining it as something more than a web of patterns learned from past experiences and echoed back into the world?

justinlivi 1y ago
I think AI skeptics have a strong bias to assume that human intelligence fundamentally functions differently from LLMs. They may be correct, but we don't have a strong enough understanding of human cognition to make the claim in as uncertain terms as the skeptical argument is unusually made. The training methods between human learning and machine learning are obviously fundamentally vastly different as are the infrastructure-level mechanics. These elements are likely never going to align, though with time the machine infrastructure may start to increasingly resemble human bio hardware. I bring this up because these known vast differences may account for a significant portion of the differences in expected output from human and machine processing. We don't understand the fundamental conceptual "black box" portions of either form of processing well enough to state definitely what is similar or dissimilar about those hazy areas. Somewhere within that not-well-understood area is what we collectively have vaguely defined "intelligence." But also within that area are all the other aspects that both humans and now machines are quite good at - prediction, fluency, translation. The challenge of lexicon and definition is potentially as difficult a task as is sharpening the focus of our understanding of the hazy black-box portion of both machine processing as well as human processing. Until all those are better defined I don't think we have a good measure for answering the question of machine intelligence either way.
gowld 1y ago
LLM's fail because their input data is limited in dimension compared to humans (text, pictures, audio, video) and because their capacity to rewire their own brains is limited to the transformer architecture.

pbw 1y ago

LLM's can shorten and maybe tend to if you just say "summarize this" but you can trivially ask them to do more. I asked for a summary of Jenson's post and then offer a reflection, GPT-5 said, "It's similar to the Plato’s Cave analogy: humans see shadows (the input text) and infer deeper reality (context, intent), while LLMs either just recite shadows (shorten) or imagine creatures behind them that aren’t there (hallucinate). The “hallucination” behavior is like adding “ghosts”—false constructs that feel real but aren’t grounded. That ain't shortening because none of that was in his post.

deleted 1y ago
[deleted]
pitpatagain 1y ago
I can't decide how to read your last sentence. That reflection seems totally off to me: fluent, and flavored with elements of the article, but also not really what the article is about and a pretty weird/tortured use of the elements of the allegory of the cave, like it doesn't seem anything like Plato's Cave to me. Ironically demonstrates the actual main gist of the article if you ask me. But maybe you meant that you think that summary is good and not textually similar to that post so demonstrating something more sophisticated than "shortening".
- pbw 1y ago
  Yes, GPT-5's response above was not shortening because there was nothing in the OP about Plato's Cave. I agree that Plato's cave analogy was confusing here. Here's a better one from GPT-5, which is deeply ironic: A New Yorker book review often does the opposite of mere shortening. The reviewer: * Places the book in a broader cultural, historical, or intellectual context. * Brings in other works—sometimes reviewing two or three books together. * Builds a thesis that connects them, so the review becomes a commentary on a whole idea-space, not just the book’s pages. This is exactly the kind of externalized, integrative thinking Jenson says LLMs lack. The New Yorker style uses the book as a jumping-off point for an argument; an LLM “shortening” is more like reading only the blurbs and rephrasing them. In Jenson’s framing, a human summary—like a rich, multi-book New Yorker review—operates on multiple layers: it compresses, but also expands meaning by bringing in outside information and weaving a narrative. The LLM’s output is more like a stripped-down plot synopsis—it can sound polished, but it isn’t about anything beyond what’s already in the text.

AndrewKemendo 1y ago

Who are you going to lodge your complaint to that the set of systems and machines that just took your job isn’t “intelligent?” Humans seem to get wrapped around these concepts like intelligence consciousness etc. because they seem to be the only thing differentiating us from every other animal when in fact it’s all a mirage.

kbaker 1y ago

Seems like this is close to the Uncanny Valley effect. LLM intelligence is in the spot where it is simultaneously genius-level but also just misses the mark a tiny bit, which really sticks out for those who have been around humans their whole lives. I feel that, just like more modern CGI, this will slowly fade with certain techniques and you just won't notice it when talking to or interacting with AI. Just like in his post during the whole Matrix discussion. > "When I asked for examples, it suggested the Matrix and even gave me the “Summary” and “Shortening” text, which I then used here word for word. " He switches in AI-written text and I bet you were reading along just the same until he pointed it out. This is our future now I guess.

gowld 1y ago
Shades of Jonathan Lethem's classic "The Ecstasy of Influence: A plagiarism" https://harpers.org/archive/2007/02/the-ecstasy-of-influence/ https://harpers.org/archive/2007/02/the-ecstasy-of-influence...

foobarian 1y ago

The LLMs are like a Huffman codec except the context is infinite and lossy

Isamu 1y ago

You can compare the current state of LLMs to the days of chess machines when they first approached grandmaster level play. The machine approach was very brute force, and there was a lot of work done to improve the sheer amount of look ahead that was required to complete at the grandmaster level. As opposed to what grandmasters actually did, which was less look ahead and more pattern matching to strengthen the position. Now LLMs successfully leverage pattern matching, but interestingly it is still a kind of brute force pattern matching, requiring the statistical absorption of all available texts, far more than a human absorbs in a lifetime. This enables the LLM to interpolate an answer from the structure of the absorbed texts with reasonable statistical relevance. This is still not quite “what humans do” as it still requires brute force statistical analysis of vast amounts of text to achieve pretty good results. For example training on all available Python sources in github and elsewhere (curated to avoid bad examples) yields pretty good results, not how a human would do it, but statistically likely to be pertinent and correct.

gowld 1y ago
One key difference is that an LLM really is better than any human at almost everything, simply because the LLM is kind of OKish at millions of things that each single human is just terrible at or completely ignorant of. The common LLMs are not trying to be a better human than a single human. They are trying to be more useful per $ cost than a bevy of humand doing some basket of human tasks. Making a replacement for an individual complete human, an effort that was popular before the Internet made that idea seem quaint, is very different challenge, and more niche, because it's not as economically efficient.

xg15 1y ago

I feel this article should be paired with this other one [1] that was on the frontpage a few days ago. My impression is, there is currently one tendency to "over-anthropomorphize" LLMs and treat them like conscious or even superhuman entities (encouraged by AI tech leaders and AGI/Singularity folks) and another to oversimplify them and view them as literal Markov chains that just got lots of training data. Maybe those articles could help guarding against both extremes. [1] https://www.verysane.ai/p/do-we-understand-how-neural-networks https://www.verysane.ai/p/do-we-understand-how-neural-networ...

mattgreenrocks 1y ago
Previously when someone called out the tendency to over-anthropomorphize LLMs, a lot of the answers amounted to, “but I like doing it, therefore we should!” I’ll be the first to say one should pick their battles. But hearing that over and over from a crowd like this that can be quite pedantic is very telling.
- tempodox 1y ago
  This very comment thread demonstrates how utterly hopeless it is trying to educate the believers. It has developed into a full-blown religion by now.

nojs 1y ago

Even stronger than our need to anthropomorphize seems to be our innate desire to believe our species is special, and that “real intelligence” couldn’t ever be replicated. If you keep redefining real intelligence as the set of things machines can’t do, then it’s always going to be true.

safetytrick 1y ago
Yes, I agree, we seem to need to feel "special". Language is really powerful, I think it's a huge part of our intelligence. The interesting part of the article to me is the focus on fluency. I have not seen anything that LLMs do well that isn't related to powerful utilization of fluency.

ArnavAgrawal03 1y ago

> They had known him for only 15 seconds, yet they still perceived the act of snapping him in half as violent. This is right out of Community

WorkerBee28474 1y ago
Clip from s01e01: https://www.youtube.com/watch?v=z906aLyP5fg https://www.youtube.com/watch?v=z906aLyP5fg

vcarrico 1y ago

I might be mixing the concepts of intelligence and conscience etc, but the human mind is more than language and data; it's also experience. LLMs have all the data and can express anything around that context, but will never experience anything, which is singular for each of us, and it's part of what makes what we call intelligence (?). So they will never replicate the human mind; they can just mimic it. I heard from Miguel Nicolelis that language is a filter for the human mind, so you can never build a mind from language. I interpreted this like trying to build an orange from its juice.

hackyhacky 1y ago
> LLMs have all the data and can express anything around that context, but will never experience anything, On the contrary, all their training data is their "experience".

nataliste 1y ago

The author's argument is built on fallacies that always pop up in these kinds of critiques. The "summary vs shortening" distinction is moving the goalposts. They makes the empirical claim that LLMs fail at summarizing novel PDFs without any actual evidence. For a model trained on a huge chunk of the internet, the line between "reworking existing text" and "drawing on external context" is so blurry it's practically meaningless. Similarly, can we please retire the ELIZA and Deep Blue analogies? Comparing a modern transformer to a 1960s if-then script or a brute-force chess engine is a category error. It's a rhetorical trick to make LLMs seem less novel than they actually are. And blaming everything on anthropomorphism is an easy out. It lets you dismiss the model's genuinely surprising capabilities by framing it as a simple flaw in human psychology. The interesting question isn't that we anthropomorphize, but why this specific technology is so effective at triggering that response from humans. The whole piece basically boils down to: "If we define intelligence in a way that is exclusively social and human, then this non-social, non-human thing isn't intelligent." It's a circular argument.

ticulatedspline 1y ago

- LLMs don't need to be intelligent to take jobs, bash scripts have replaced people. - Even if CEOs are completely out of touch and the tool can't do the job you can still get laid off in an ill informed attempt to replace you. Then when the company doesn't fall over because the leftover people, desperate to keep covering rent fill the gaps it just looks like efficiency to the top. - I don't think our tendency anthropomorphize LLMs is really the problem here.

andoando 1y ago

This, along with a ton of commentary on LLMs, seems like its written by someone who has no technical understanding of LLMs.

Joeri 1y ago

LLMs mimic intelligence, but they aren’t intelligent. They aren’t just intelligence mimics, they are people mimics, and they’re getting better at it with every generation. Whether they are intelligent or not, whether they are people or not, it ultimately does not matter when it comes to what they can actually do, what they can actually automate. If they mimic a particular scenario or human task well enough that the job gets done, they can replace intelligence even if they are “not intelligent”. If by now someone still isn’t convinced that LLMs can indeed automate some of those intelligence tasks, then I would argue they are not open to being convinced.

shafoshaf 1y ago
They can mimic well documented behavior. Applying an LLM to a novel task is where the model breaks down. This obviously has huge implications for automation. For example, most business do not have unique ways of handling accounting transactions, yet each company has a litany of AR and AP specialists who create semmingly unique SOPs. LLMs can easily automate those workers since they are simply doing a slight variation at best of a very well documented system. Asking an LLM to take all this knowledge and apply it to a new domain? That will take a whole new paradigm.
- quesera 1y ago
  Absolutely agreed, but I suspect that a whole lot of what humans do every day can be reduced to pattern-following. If/when LLMs or other AIs can create novel work / discover new knowledge, they will be "genius" in the literal sense of the word. More genius would be great! (probably) . But genius is not required for the vast majority of tasks.
- andrewla 1y ago
  > Applying an LLM to a novel task is where the model breaks down I mean, don't most people break down in this case too? I think this needs to be more precise. What is the specific task that you think can reliably distinguish between an LLM's capability in this sense vs. what a human can typically manage? That is, in the sense of [1], what is the result that we're looking to use to differentiate. [1] https://news.ycombinator.com/item?id=44913498 https://news.ycombinator.com/item?id=44913498

0x457 1y ago

> A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. How is that a summary? It reads as a one-liner review I would leave on Letterboxed or something I would say, trying to be pretentious and treating the movie as a work of art. It is a work of art, because all movies are art, but that's an awful summary.

pwdisswordfishz 1y ago
Review? Where does it say anything about the quality of the film?
- 0x457 1y ago
  * * * * (4/5) A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. ----- Do I need to explain more?

deleted 1y ago

[deleted]

intalentive 1y ago

Good point about the Turing Test: >The original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. By the original meaning of the test it's easy to tell an LLM from a human.

jhanschoo 1y ago

I disagree with the author in a big way. '25's LLMs are designed to echo material already out there on the Internet if it exists because we value that more. If I want a summary of The Matrix, I prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary that requires a justification as to its deviation. In fact, the example provided by the author is a great illustration of this: > A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control. The words here refer back to the notions "free will" that is prominent in Western discourse from St. Augustine through Descartes and thereafter and similarly of "sci-fi". These are notions an uneducated East-Asian with limited Internet use and pop culture fluency will simply not understand. They would in fact prefer the latter description. The author and this hypothetical East-Asian live in very different zeitgeists, and correspondingly experience the movie differently and value different summaries of the film. They each prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary (relative to their zeitgeist) that requires a justification as to its deviation. On the other hand, if you asked LLMs to explain material and concepts, one modality in which it does is use formulaic, novel, and unorthodox analogies to explain it to you. By formulaic and novel, I mean that the objects and scenarios in the analogies are frequently of a certain signature kind that it has been trained with, but it is novel in that the analogies are not found in the wild on the internet. If you have frequently used LLMs for the purpose of explaining concepts, you will have encountered these analogies and know what I mean by this. The analogies are frequently not too accurate, but they round out the response by giving an ELI5-style answer. Ironically, the author may have succumbed to LLM sycophancy.

justsomedumbass 1y ago

Intelligence is a social construct? Interesting thought, but denies the innovator credit for their efforts. Does technology naturally evolve in an 'intelligent' society? I don't think so. Now without somebody going deep into the weeds to find gold. Without individual genius, we would be far behind in our collective progress.

jstgunderscore 1y ago

Firstly, I think a lot of commenters here should ask themselves "Do you believe that ANY machine could EVER be intelligent?" Unless you believe in the magic sauce of a supernatural soul/mind/etc, our brains function as deterministic biological machines. There's essentially nothing that separates the processing and memory potential of silicon from neurons. And if the building blocks can be made analogous, then all the same emergent properties are possible. There's no reason to believe a circuit couldn't be made to behave in exactly the same way as a human brain. I'm not saying that's where LLMs are; only that it is theoretically possible. So if you imagine such a machine, and you deem that it is not intelligent, you have reserved intelligence as an exclusive human trait and this entire discussion is meaningless. Secondly, although I'm not in either pro or anti LLM-intelligence camp, I find a lot of the arguments against machine intelligence disingenuous and/or unbalanced. For instance the "Can't process information it's not familiar with" argument. Another commenter stated the case of scientific papers that it doesn't have any reference for, that it may hallucinate a garbage interpretation of the paper. Not surprising, but guess what, a human would do the same thing if they were forced! Imagine holding a gun to someone's head and telling them to explain a concept or system they've never heard of. That's essentially what we're doing with LLMs; obviously we don't need to threaten, because we haven't given them agency to say no. Another example is the "Can't be novel, unique, or create something completely new." First of all, difficult to prove, but okay let's take it as given that an LLM can't be novel. Can you prove that a human can? We make all these assumptions on how intelligent and creative we are as humans, and how original our thoughts can be... but how original are they, and can we prove it? How do you know your original thought, or Beethoven's 5th, or the fast inverse square root trick was completely separate from any prior influences? Or... was that "original thought" the conglomeration of a thousand smaller inputs and data points that you trained on, that became part of your brains subconscious processing system, and came together in a synthesis that looks like brilliance. Finally, whenever this discussion comes up with friends I ask them to think of the least intelligent person they know. Then imagine how many of those there are in the world (likely millions). Could you imagine any conceivable test of any length or depth which would designate all the those humans as intelligent, and all the LLMs as not? I certainly can't. I highly doubt there could be anything approaching 100% accuracy at this point. Ultimately I think we should ditch both the intelligence and consciousness questions. We can't define them in ourselves, we certainly can't define them in another entity. Let's just come to terms with the fact we're highly functioning biological machines who are both scared and excited to have created something so similar to us.

jjaksic 1y ago

There's a lot of moving the goal posts here. What would it mean for an AI to be considered "intelligent"? At various points it meant different things (chess, jeopardy, go, human language, math, etc). The most well-established benchmark was the Turing test, which LLMs now pass spectacularly -- in fact the only way to reliably tell that you're talking to an AI is usually because it responds too fast and too perfectly. Sure, AI often makes mistakes, but so do humans. Ask a random person a random question and I guarantee that at least 90% of the times an LLM will produce a better answer. So if knowledge or reasoning mistakes are a benchmark, then humans are not intelligent either. What is the difference then? I feel that whoever claims that AI is not intelligent simply feels uncomfortable thinking that they are, and is going out of their way to rationalize that, by using various double standards. Like, AI making occasional mistakes counts as evidence against it, but the fact that the vast majority of people are incapable of answering some of the most basic questions somehow doesn't count against human intelligence. And then there are various contrived "definitions of intelligence", such as "living in the physical world". These have nothing to do with cognitive intelligence and are just made up to make humans fit the definition and AI to not fit. Doing "truly novel things"? How many people have ever come up with anything truly novel? What about the other 99.99%, are they intelligent? I think there's one thing that's tripping people up. Humans do have a very high COLLECTIVE intelligence. Collectively (and given enough time) we're capable of accomplishing great things, much more than current AI. But each individual human on their own is no more intelligent than a caveman (we have essentially the same brain).