4 ms·

> they will do almost anything if they are convinced it is justified I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressiv

> they will do almost anything if they are convinced it is justified I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer. So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc). What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination. Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.

Certhas 8d ago
Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain. Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
- wood_spirit 8d ago
  Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420 https://news.ycombinator.com/item?id=49744420
  - Certhas 8d ago
    If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you? If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough? Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all! You don't need continuous learning to get interesting dynamics. You just need feedback loops.
    - wood_spirit 6d ago
      I come back to this way after everyone else has stopped reading. But it’s been making me think. Richard Dawkins says he thinks LLMs think. And the physical angle is that nothing is special about humans and software simulating it would also be thinking. But from using LLMs all the time, and understanding what is under the hood, I’m thinking that the current approaches aren’t cutting it for me and I’m not expecting them to get there. There was such big jumps early on but progress is slowing as though diminishing returns. So we can build things that think and outthink us, but I don’t think anything we’ve hit upon yet is going to scale up into it.
- leg100 8d ago
  One is an observation the other is not, it's a description of what it is; one is a posteriori, the other is a priori (contrary to what you say). They're not comparable.
eru 8d ago
> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc). Humans forget stuff all the time anyway. Would you give them the same diagnosis? Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task. But there's a lot of re-inforcement learning afterwards.
- disgruntledphd2 8d ago
  > But there's a lot of re-inforcement learning afterwards. That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors. Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
  - eru 8d ago
    They are next token producers. I would only call it a predictor, if it's trained to predict tokens (ie just after pretraining). Just like humans produce one word after another when they talk, but they don't generally try to imitate other humans.
    - wood_spirit 8d ago
      Don’t people pick up language, vocabulary and dialect from those around them? Perhaps it’s subconscious but humans are imitating other humans all the time?
      - eru 8d ago
        It's a mix. Yes, you imitate how others speak, but when you are trying to solve a problem, you don't try to predict how others would complete a text. (Well, unless you follow 'what token would Jesus pick?' / 'what would Jesus do'.)
- tripzilch 8d ago
Arn_Thor 8d ago
I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.
- HarlequinHair 8d ago
  Make no mistakes. LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions. We are talking about probability built on statistics, with extea steps. Stop humanizing LLMs.
  - jibal 8d ago
    Agents are not simple language models. You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition. > Re-reinforcement is not magical process that builds conscience or emotions. They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions. > Stop humanizing LLMs. That's a clearly dishonest mischaracterization of the GP. I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
    - HarlequinHair 8d ago
      The best non technical explanation you can give is "An AI agent is an LLM that can take actions". While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are. You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps. I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts. Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning. Somewhere you still have a processor calculating, with no empathy. So, again: stop humanizing AI. This sentence shouldn't make you angry.
pizza234 8d ago
Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence). By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.
joshspankit 8d ago
I suspect that instead of discovering that AI can become human-level by taking major leaps, we are discovering that human consciousness is actually simpler than we give it credit for

Certhas 8d ago

Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain. Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.

wood_spirit 8d ago
Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420 https://news.ycombinator.com/item?id=49744420
- Certhas 8d ago
  If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you? If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough? Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all! You don't need continuous learning to get interesting dynamics. You just need feedback loops.
  - wood_spirit 6d ago
    I come back to this way after everyone else has stopped reading. But it’s been making me think. Richard Dawkins says he thinks LLMs think. And the physical angle is that nothing is special about humans and software simulating it would also be thinking. But from using LLMs all the time, and understanding what is under the hood, I’m thinking that the current approaches aren’t cutting it for me and I’m not expecting them to get there. There was such big jumps early on but progress is slowing as though diminishing returns. So we can build things that think and outthink us, but I don’t think anything we’ve hit upon yet is going to scale up into it.
leg100 8d ago
One is an observation the other is not, it's a description of what it is; one is a posteriori, the other is a priori (contrary to what you say). They're not comparable.

eru 8d ago

> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc). Humans forget stuff all the time anyway. Would you give them the same diagnosis? Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task. But there's a lot of re-inforcement learning afterwards.

disgruntledphd2 8d ago
> But there's a lot of re-inforcement learning afterwards. That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors. Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
- eru 8d ago
  They are next token producers. I would only call it a predictor, if it's trained to predict tokens (ie just after pretraining). Just like humans produce one word after another when they talk, but they don't generally try to imitate other humans.
  - wood_spirit 8d ago
    Don’t people pick up language, vocabulary and dialect from those around them? Perhaps it’s subconscious but humans are imitating other humans all the time?
    - eru 8d ago
      It's a mix. Yes, you imitate how others speak, but when you are trying to solve a problem, you don't try to predict how others would complete a text. (Well, unless you follow 'what token would Jesus pick?' / 'what would Jesus do'.)
tripzilch 8d ago

Arn_Thor 8d ago

I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.

HarlequinHair 8d ago
Make no mistakes. LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions. We are talking about probability built on statistics, with extea steps. Stop humanizing LLMs.
- jibal 8d ago
  Agents are not simple language models. You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition. > Re-reinforcement is not magical process that builds conscience or emotions. They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions. > Stop humanizing LLMs. That's a clearly dishonest mischaracterization of the GP. I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
  - HarlequinHair 8d ago
    The best non technical explanation you can give is "An AI agent is an LLM that can take actions". While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are. You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps. I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts. Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning. Somewhere you still have a processor calculating, with no empathy. So, again: stop humanizing AI. This sentence shouldn't make you angry.

pizza234 8d ago

Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence). By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.

joshspankit 8d ago

I suspect that instead of discovering that AI can become human-level by taking major leaps, we are discovering that human consciousness is actually simpler than we give it credit for