3 ms·

You don't think it's possible that an LLM's internal machinery could decide that an underused-by-humans word should be used more frequently in output than it se

You don't think it's possible that an LLM's internal machinery could decide that an underused-by-humans word should be used more frequently in output than it sees in input because it maps cleanly onto a frequently needed semantic? I think that's possible

bunderbunder 3mo ago
It sounds like you are trying to understand LLM behavior using a mental model that inaccurately personifies the stochastic parrot. A more parsimonious explanation is that this term got more-or-less randomly boosted by the reinforcement learning loop because there was nothing in the training data to discourage its use.
- Myrmornis 3mo ago
  Ah right, you don't like AI and don't care to understand how it works.
  - bunderbunder 3mo ago
    I’ve been working in AI - and specifically NLP - since 2003. I am no stranger to how weird quirks can sneak into overparametrized models, nor am I a stranger to how good humans can be at inferring meaning where there is none in specific language model behaviors. So, yeah, I am inclined to assume non-teleological causes are more parsimonious than inferring the presence of a strange loop, because that continues to be the winning bet. Even for generative LLMs.
  - bigfishrunning 3mo ago
    Ah right, so you like AI and don't care to understand how it works. It doesn't "decide" anything or "need" any semantic. It derives the likelihood of the token, and "bearing" is likely to come after "load".
    - Myrmornis 3mo ago
      Sure but the question is why "load" after X?
      - bunderbunder 3mo ago
        More-or-less the same principle, but scaled up massively, and with context-dependent probability conditioning maps.
      - bigfishrunning 3mo ago
        Because, for some high number of contexts, its likelihood comes out high in the big tree of multiplies that is claude's model. For some sets of 500 words (or whatever), the next word is "load". The classifier that decides which sets of 500 (or whatever) words is a prefix for "load" is returning "true" too often.
- Myrmornis 2mo ago
  > the stochastic parrot. https://news.ycombinator.com/item?id=49010345 https://news.ycombinator.com/item?id=49010345

bunderbunder 3mo ago

It sounds like you are trying to understand LLM behavior using a mental model that inaccurately personifies the stochastic parrot. A more parsimonious explanation is that this term got more-or-less randomly boosted by the reinforcement learning loop because there was nothing in the training data to discourage its use.

Myrmornis 3mo ago
Ah right, you don't like AI and don't care to understand how it works.
- bunderbunder 3mo ago
  I’ve been working in AI - and specifically NLP - since 2003. I am no stranger to how weird quirks can sneak into overparametrized models, nor am I a stranger to how good humans can be at inferring meaning where there is none in specific language model behaviors. So, yeah, I am inclined to assume non-teleological causes are more parsimonious than inferring the presence of a strange loop, because that continues to be the winning bet. Even for generative LLMs.
- bigfishrunning 3mo ago
  Ah right, so you like AI and don't care to understand how it works. It doesn't "decide" anything or "need" any semantic. It derives the likelihood of the token, and "bearing" is likely to come after "load".
  - Myrmornis 3mo ago
    Sure but the question is why "load" after X?
    - bunderbunder 3mo ago
      More-or-less the same principle, but scaled up massively, and with context-dependent probability conditioning maps.
    - bigfishrunning 3mo ago
      Because, for some high number of contexts, its likelihood comes out high in the big tree of multiplies that is claude's model. For some sets of 500 words (or whatever), the next word is "load". The classifier that decides which sets of 500 (or whatever) words is a prefix for "load" is returning "true" too often.
Myrmornis 2mo ago
> the stochastic parrot. https://news.ycombinator.com/item?id=49010345 https://news.ycombinator.com/item?id=49010345