4 ms·

You'd sit for eternity if you suffered a lesion in your reticular activating system - a relatively small cluster of neurons that generates a kind of clock signa

You'd sit for eternity if you suffered a lesion in your reticular activating system - a relatively small cluster of neurons that generates a kind of clock signal in animal brains. Coma patients with RAS lesions seem to visualize scenes, given prompts, despite not really being conscious. Conversely, ChatGPT does decently well on multi-armed bandit tasks, demonstrating (rudimentary) reinforcement learning capability during inference. It's known that LLMs evolve their own optimizers in the process of acquiring few-shot learning, so I assume it picked up these RL abilities similarly. That kind of on-line RL is foundational to autonomous agents. The prompt isn't part of the LLM, it's part of how the LLM is wired into a chat window. You can make them stream tokens forever, or prompt themselves, or ditch causality entirely. The foundational abilities for autonomy, I think, are in there, for the simple reason that they've learned to model autonomous agents - human beings.

Retric 3y ago
The prompts or at least being fed a sequence of tokens including output from prior passes is integral to how language models function. Rather than being “hooked up to one” the neural networks only function is to pick a single token based on a set of inputs. So without being feed it’s own output you get a single token and then nothing. There’s some randomness injected into the process and whatnot but that’s ultimately just window dressing to make them seem less mechanical. There’s all kinds of ways to disrupt human or animal consciousness such as reducing oxygen supply, but saying the human brain is vulnerable doesn’t change anything about how it operates normally. Plenty of ways to break an LLM’s, but then you’re talking about a different system. Similarly the reticular activation system’s purpose is to regulate wakefulness, which aspects are directly useful or not isn’t particularly relevant because it’s part of the brain.
- sterlind 3y ago
  No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why techniques like beam search and Viterbi work so well - you don't have to commit to the next token at each step. And temperature (what I assume you mean by "randomness injected") isn't "window dressing," it fundamentally gives better results because LMs model probability distributions. You'll get crappy results with any probability model if you run them purely greedily. And you're also neglecting non-causal LMs (like BERT, and encoders in general), which don't predict the next token in a series, but instead predict previous masked tokens. You're conflating how LLMs are used for generation with what LLMs are, and that's just plain wrong. They're not trained autoregressively at all! To repeat, the generation mechanism is simply not part of the LLM. The LLM is a probability model; the generator just uses that model. It's not "breaking it" to use a different generation strategy than greedy autoregression, since they're not even trained a token at a time.
  - Retric 3y ago
    There’s plenty of different ways you can use the output of those functions to feed a new token sequences back to the model, but you can only feed a specific token not the full probability distribution from a prior run. As to randomness that’s simply one approach, there’s deterministic approaches that have their own advantages. What randomness provides over them is avoiding always responding to the same opening in the same way as that’s quite off-putting.
svnt 3y ago
The is the equivalent of saying “you’d be unable to see if someone turned off the lights” and then implying that in order to sight the genetically blind you’d just need to give them a light switch.

Retric 3y ago

The prompts or at least being fed a sequence of tokens including output from prior passes is integral to how language models function. Rather than being “hooked up to one” the neural networks only function is to pick a single token based on a set of inputs. So without being feed it’s own output you get a single token and then nothing. There’s some randomness injected into the process and whatnot but that’s ultimately just window dressing to make them seem less mechanical. There’s all kinds of ways to disrupt human or animal consciousness such as reducing oxygen supply, but saying the human brain is vulnerable doesn’t change anything about how it operates normally. Plenty of ways to break an LLM’s, but then you’re talking about a different system. Similarly the reticular activation system’s purpose is to regulate wakefulness, which aspects are directly useful or not isn’t particularly relevant because it’s part of the brain.

sterlind 3y ago
No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why techniques like beam search and Viterbi work so well - you don't have to commit to the next token at each step. And temperature (what I assume you mean by "randomness injected") isn't "window dressing," it fundamentally gives better results because LMs model probability distributions. You'll get crappy results with any probability model if you run them purely greedily. And you're also neglecting non-causal LMs (like BERT, and encoders in general), which don't predict the next token in a series, but instead predict previous masked tokens. You're conflating how LLMs are used for generation with what LLMs are, and that's just plain wrong. They're not trained autoregressively at all! To repeat, the generation mechanism is simply not part of the LLM. The LLM is a probability model; the generator just uses that model. It's not "breaking it" to use a different generation strategy than greedy autoregression, since they're not even trained a token at a time.
- Retric 3y ago
  There’s plenty of different ways you can use the output of those functions to feed a new token sequences back to the model, but you can only feed a specific token not the full probability distribution from a prior run. As to randomness that’s simply one approach, there’s deterministic approaches that have their own advantages. What randomness provides over them is avoiding always responding to the same opening in the same way as that’s quite off-putting.

svnt 3y ago

The is the equivalent of saying “you’d be unable to see if someone turned off the lights” and then implying that in order to sight the genetically blind you’d just need to give them a light switch.