6 ms·
This is the 2023 take on LLMs. It still gets repeated a lot. But it doesn’t really hold up anymore - it’s more complicated than that. Don’t let some factoid abo
by libraryofbabel 10mo ago
This is the 2023 take on LLMs. It still gets repeated a lot. But it doesn’t really hold up anymore - it’s more complicated than that. Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network.
Sure, LLMs do not think like humans and they may not have human-level creativity. Sometimes they hallucinate. But they can absolutely solve new problems that aren’t in their training set, e.g. some rather difficult problems on the last Mathematical Olympiad. They don’t just regurgitate remixes of their training data. If you don’t believe this, you really need to spend more time with the latest SotA models like Opus 4.5 or Gemini 3.
Nontrivial emergent behavior is a thing. It will only get more impressive. That doesn’t make LLMs like humans (and we shouldn’t anthropomorphize them) but they are not “autocomplete on steroids” anymore either.
- deadbolt 10mo agoAs someone who still might have a '2023 take on LLMs', even though I use them often at work, where would you recommend I look to learn more about what a '2025 LLM' is, and how they operate differently?
- krackers 10mo agoPapers on mechanistic interpratability and representation engineering, e.g. from Anthropic would be a good start.
- otabdeveloper4 10mo agoDon't bother. This bubble will pop in two years, you don't want to look back on your old comments in shame in three.
- dingnuts 10mo ago[dead]
- root_axis 10mo ago> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By refining training data quality, increasing parameter size, and increasing context length we can squeeze more utility out of LLMs than ever before, but ultimately, Opus 4.5 is the same thing as GPT2, it's only that coherence lasts a few pages rather than a few sentences.
- A4ET8a8uTh0_v2 10mo agoBut.. and I am not asking it for giggles, does it mean humans are giant autocomplete machines?
- root_axis 10mo agoNot at all. Why would it?
- A4ET8a8uTh0_v2 10mo agoCall it a.. thought experiment about the question of scale.
- root_axis 10mo agoI'm not exactly sure what you mean. Could you please elaborate further?
- a1j9o94 10mo agoNot the person you're responding to, but I think there's a non trivial argument to make that our thoughts are just auto complete. What is the next most likely word based on what you're seeing. Ever watched a movie and guessed the plot? Or read a comment and know where it was going to go by the end? And I know not everyone thinks in a literal stream of words all the time (I do) but I would argue that those people's brains are just using a different "token"
- otabdeveloper4 10mo ago> it’s more complicated than that. No it isn't. > ...fool you into thinking you understand what is going on in that trillion parameter neural network. It's just matrix multiplication and logistic regression, nothing more.
- hackinthebochs 10mo agoLLMs are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Roughly the same architecture can generate passable images, music, or even video. The sequence of matrix multiplications are the high level constraint on the space of programs discoverable. But the specific parameters discovered are what determines the specifics of information flow through the network and hence what program is defined. The complexity of the trained network is emergent, meaning the internal complexity far surpasses that of the course-grained description of the high level matmul sequences. LLMs are not just matmuls and logits. [1] https://x.com/karpathy/status/1582807367988654081 https://x.com/karpathy/status/1582807367988654081
- otabdeveloper4 10mo ago> LLMs are a general purpose computing paradigm. Yes, so is logistic regression.
- hackinthebochs 10mo agoNo, not at all.
- otabdeveloper4 10mo agoYes at all. I think you misunderstand the significance of "general computing". The binary string 01101110 is a general-purpose computer, for example.
- vachina 10mo agoI use enterprise LLM provided by work, working on very proprietary codebase on a semi esoteric language. My impression is it is still a very big autocompletion machine. You still need to hand hold it all the way as it is only capable of regurgitating the tiny amount of code patterns it saw in the public. As opposed to say a Python project.
- libraryofbabel 10mo agoWhat model is your “enterprise LLM”? But regardless, I don’t think anyone is claiming that LLMs can magically do things that aren’t in their training data or context window. Obviously not: they can’t learn on the job and the permanent knowledge they have is frozen in during training.
- beernet 10mo ago>> Sometimes they hallucinate. For someone speaking as you knew everything, you appear to know very little. Every LLM completion is a "hallucination", some of them just happen to be factually correct.
- Am4TIfIsER0ppos 10mo agoI can say "I don't know" in response to a question. Can an LLM?
- nl 10mo agoYes, frequently. Most modern post training setups encourage this. It isn't 2023 anymore.
- Smaug123 10mo agoThis is one of the easiest questions in the world to answer. My first try on the smallest and fastest model it was convenient to access, GPT-5.2 Instant: https://chatgpt.com/share/69468764-01cc-8008-b734-0fb55fd7efe0 https://chatgpt.com/share/69468764-01cc-8008-b734-0fb55fd7ef... > What did I have for breakfast this morning? > I don’t know what you had for breakfast this morning…