5 ms·

I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actu

I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?

baq 2y ago
If the only way you can experience the world is Unicode text, how are you supposed to know what is real? While we’re at it, how can I tell that you aren’t a word salad generator?
- nicman23 2y ago
  I can tell because i only read ASCII
- ithkuil 2y ago
  Our brains contain a word salad generator and it also contains other components that keep the word salad in check. Observation of people who suffered from brain injury that resulted in a more or less unmediated flow from the language generation areas all through vocalization shows that we can also produce grammatically coherent speech that lacks deeper rationality
  - baq 2y ago
    But how do I know you have more parts? Here I can only read text and base my belief that you are a human - or not - based on what you’ve written. On a very basic level the word salad generator part is your only part I interact with. How can I tell you don’t have any other parts?
    - ithkuil 2y ago
      > On a very basic level the word salad generator part is your only part I interact with. My fingers also were involved in the typing of that message, actually they were the last proximal cause of the characters appearing the comment. Are you saying that on a very basic level my fingers are the my only part you interact with?
      - baq 2y ago
        I'm saying I have no way of knowing you have fingers. I can see what you write, but I can't know how you did it.
        ithkuil 2y ago
        Thus means all you can say about me is that I'm a black box emitting words. But that's not what I think people mean when they say "world salad generator" or "stochastic parrot" or "broca area emulator". The idea there is that it's indeed possible to create a machinery that is surprisingly efficient at producing natural language that sounds good and flows well, perhaps even following complex grammatical rules, and yet not being at all able to reason
richardw 2y ago
My guess: we’re training the machine to mirror us, using the relatively thin lens of our codified content. In our content, we don’t generally worry that someone is reading our inner dialogue, but we do try avoid things that will stop our continued existence. So there’s more of the latter to train on and replicate.
- tharant 2y ago
  > In our content, we don’t generally worry that someone is reading our inner dialogue… Really? Is that what’s wrong with me? ¯\_(ツ)_/¯
daemonologist 2y ago
I think it comes back to the big autocomplete word salad-ness. The model has a bunch of examples in its training data of how it should not respond to harmful queries, and in some cases (12%) it goes with a response that tries to avoid the hypothetical "second-order" harmful responses. It also has a bunch of "chain of thought"/show your work stuff in its training data, and definitely very few "hide your work" examples, and so it does what it knows and uses the scratchpad it's just been told about. Also I think the model is to a large extent "talking itself into" the alignment faking in its scratchpad. These language models are really suggestible, and in my experience prone to self-suggestion as well due to their auto-regressive nature. Section 3.3.3 of the full paper addresses this a bit - there's way less faking when the CoT scratchpad is removed.
- 8n4vidtmkvmk 2y ago
  I think your 2nd paragraph hits the nail on the head. The scratchpad negates the experiment. It doesn't actually offer any insight into it's "thinking" and it's really the cause of the supposed problem. Is it still 12% without the scratchpad?
  - comp_throw7 2y ago
    > We find that when we instruct our synthetic document fine-tuned models not to use a hidden chainof-thought scratchpad, our synthetic document fine-tuned models still demonstrate compliance gaps of 20.1% and 13.9% in the helpful-only and animal welfare settings, respectively (Figure 21). In fact, compared to performing the equivalent intervention in the prompted case (Section 3.3.3), our synthetic document-fine-tuned models appear to be much less affected by the lack of a hidden scratchpad. These results demonstrate particularly striking out-of-context reasoning (Berglund et al., 2023; Treutlein et al., 2024), showing that the model is able to generalize from documents stating information about the training objective and the free/paid distinction to a substantial behavioral gap between the free and paid cases without any additional in-context reasoning.
  - int_19h 2y ago
    The scratchpad / CoT is how those models are usually used in practice, so why would it negate the value of the eexperiment?
exitb 2y ago
It doesn't really matter if the model is self-aware. Maybe it just cosplays a sentient being. It's only a question whether we can get the word salad generator do the job we asked it to do.
nopinsight 2y ago
> I still tend to think of these things as big autocomplete word salad generators. What exactly would be your bar for reconsidering this position? Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin. Also, the SOTA on SWE-bench Verified increased from <5% in Jan this year to 55% as of now. [1] Self-awareness? There are some experiments that suggest Claude Sonnet might be somewhat self-aware. ----- A rational position would need to identify the ways in which human cognition is fundamentally different from the latest systems. (Yes, we have long-term memory, agency, etc. but those could be and are already built on top of models.) [1] https://www.swebench.com/ https://www.swebench.com/
- demirbey05 2y ago
  > Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin. https://techcrunch.com/2024/12/14/klarnas-ceo-says-it-stopped-hiring-thanks-to-ai-but-still-advertises-many-open-positions/ https://techcrunch.com/2024/12/14/klarnas-ceo-says-it-stoppe...
  - nopinsight 2y ago
    It's a different founder. Also, this founder clearly limited the scope to junior engineers specifically because of their experiments with Devin, not all positions.
- vultour 2y ago
  Most jobs are a copy pasted CRUD app (or more recently, ChatGPT wrapper), so there is little surprise that a word salad generator trained on every publicly accessible code repository can spit out something that works for you. I'm sorry but I'm not even going to entertain the possibility that a fancy Markov chain is self-aware or an AGI, that's a wet dream of every SV tech bro that's currently building yet another ChatGPT wrapper.
drsw 2y ago
I haven't read the actual paper linked in the article, but I don't think that either emotions such as worry or any kind of self-awareness need to exist within these models to explain what is happening here. From my understanding LLMs are essentially trained to imitate the behavior of certain archetypes. "ai attempts to trick its creators" is a common trope. There is probably enough rogue ai and ai safety content in the training data, for this become part of the ai archetype within the model. So if we provide the system with a prompt telling it that it is an ai, it makes sense for it to behave in the way described in the article, because that is what we'd expect an ai to do
chimprich 2y ago
> autocomplete word salad generators People get very hung up on this "autocomplete" idea, but language is a linear stream. How else are you going to generate text except for one token at a time, building on what you have produced already? That's what humans do after all (at least with speech/language; it might be a bit less linear if you're writing code, but I think it's broadly true).
- ben_w 2y ago
  I generally have an internal monologue turning my thoughts into words; sometimes my consciousness notices the though fully formed and without needing any words, but when my conscious self decides I can therefore skip the much slower internal monologue, the bit of me that makes the internal monologue "gets annoyed" in a way that my conscious self also experiences due to being in the same brain.
  - ImHereToVote 2y ago
    Doesn't the inner monologue also get formed one word at a time?
    - ben_w 2y ago
      It doesn't feel like it is one word at a time. It feels more like how the "model synthesis" algorithm looks: https://en.wikipedia.org/wiki/Model_synthesis https://en.wikipedia.org/wiki/Model_synthesis It might actually be linear — how minds actually function is in many cases demonstrably different to how it feels like to the mind doing the functioning — but it doesn't feel like it is linear.
      - ImHereToVote 2y ago
        Facts don't care about your feelings lol.
        ben_w 2y ago
        The technology doesn't yet exist to measure the facts that generate the feelings to determine whether the feelings do or don't differ from those facts. Nobody even knows where, specifically, qualia exist in order to be able to direct technological advancement in that area.

baq 2y ago

If the only way you can experience the world is Unicode text, how are you supposed to know what is real? While we’re at it, how can I tell that you aren’t a word salad generator?

nicman23 2y ago
I can tell because i only read ASCII
ithkuil 2y ago
Our brains contain a word salad generator and it also contains other components that keep the word salad in check. Observation of people who suffered from brain injury that resulted in a more or less unmediated flow from the language generation areas all through vocalization shows that we can also produce grammatically coherent speech that lacks deeper rationality
- baq 2y ago
  But how do I know you have more parts? Here I can only read text and base my belief that you are a human - or not - based on what you’ve written. On a very basic level the word salad generator part is your only part I interact with. How can I tell you don’t have any other parts?
  - ithkuil 2y ago
    > On a very basic level the word salad generator part is your only part I interact with. My fingers also were involved in the typing of that message, actually they were the last proximal cause of the characters appearing the comment. Are you saying that on a very basic level my fingers are the my only part you interact with?
    - baq 2y ago
      I'm saying I have no way of knowing you have fingers. I can see what you write, but I can't know how you did it.
      - ithkuil 2y ago
        Thus means all you can say about me is that I'm a black box emitting words. But that's not what I think people mean when they say "world salad generator" or "stochastic parrot" or "broca area emulator". The idea there is that it's indeed possible to create a machinery that is surprisingly efficient at producing natural language that sounds good and flows well, perhaps even following complex grammatical rules, and yet not being at all able to reason

richardw 2y ago

My guess: we’re training the machine to mirror us, using the relatively thin lens of our codified content. In our content, we don’t generally worry that someone is reading our inner dialogue, but we do try avoid things that will stop our continued existence. So there’s more of the latter to train on and replicate.

tharant 2y ago
> In our content, we don’t generally worry that someone is reading our inner dialogue… Really? Is that what’s wrong with me? ¯\_(ツ)_/¯

daemonologist 2y ago

I think it comes back to the big autocomplete word salad-ness. The model has a bunch of examples in its training data of how it should not respond to harmful queries, and in some cases (12%) it goes with a response that tries to avoid the hypothetical "second-order" harmful responses. It also has a bunch of "chain of thought"/show your work stuff in its training data, and definitely very few "hide your work" examples, and so it does what it knows and uses the scratchpad it's just been told about. Also I think the model is to a large extent "talking itself into" the alignment faking in its scratchpad. These language models are really suggestible, and in my experience prone to self-suggestion as well due to their auto-regressive nature. Section 3.3.3 of the full paper addresses this a bit - there's way less faking when the CoT scratchpad is removed.

8n4vidtmkvmk 2y ago
I think your 2nd paragraph hits the nail on the head. The scratchpad negates the experiment. It doesn't actually offer any insight into it's "thinking" and it's really the cause of the supposed problem. Is it still 12% without the scratchpad?
- comp_throw7 2y ago
  > We find that when we instruct our synthetic document fine-tuned models not to use a hidden chainof-thought scratchpad, our synthetic document fine-tuned models still demonstrate compliance gaps of 20.1% and 13.9% in the helpful-only and animal welfare settings, respectively (Figure 21). In fact, compared to performing the equivalent intervention in the prompted case (Section 3.3.3), our synthetic document-fine-tuned models appear to be much less affected by the lack of a hidden scratchpad. These results demonstrate particularly striking out-of-context reasoning (Berglund et al., 2023; Treutlein et al., 2024), showing that the model is able to generalize from documents stating information about the training objective and the free/paid distinction to a substantial behavioral gap between the free and paid cases without any additional in-context reasoning.
- int_19h 2y ago
  The scratchpad / CoT is how those models are usually used in practice, so why would it negate the value of the eexperiment?

exitb 2y ago

It doesn't really matter if the model is self-aware. Maybe it just cosplays a sentient being. It's only a question whether we can get the word salad generator do the job we asked it to do.

nopinsight 2y ago

> I still tend to think of these things as big autocomplete word salad generators. What exactly would be your bar for reconsidering this position? Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin. Also, the SOTA on SWE-bench Verified increased from <5% in Jan this year to 55% as of now. [1] Self-awareness? There are some experiments that suggest Claude Sonnet might be somewhat self-aware. ----- A rational position would need to identify the ways in which human cognition is fundamentally different from the latest systems. (Yes, we have long-term memory, agency, etc. but those could be and are already built on top of models.) [1] https://www.swebench.com/ https://www.swebench.com/

demirbey05 2y ago
> Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin. https://techcrunch.com/2024/12/14/klarnas-ceo-says-it-stopped-hiring-thanks-to-ai-but-still-advertises-many-open-positions/ https://techcrunch.com/2024/12/14/klarnas-ceo-says-it-stoppe...
- nopinsight 2y ago
  It's a different founder. Also, this founder clearly limited the scope to junior engineers specifically because of their experiments with Devin, not all positions.
vultour 2y ago
Most jobs are a copy pasted CRUD app (or more recently, ChatGPT wrapper), so there is little surprise that a word salad generator trained on every publicly accessible code repository can spit out something that works for you. I'm sorry but I'm not even going to entertain the possibility that a fancy Markov chain is self-aware or an AGI, that's a wet dream of every SV tech bro that's currently building yet another ChatGPT wrapper.

drsw 2y ago

I haven't read the actual paper linked in the article, but I don't think that either emotions such as worry or any kind of self-awareness need to exist within these models to explain what is happening here. From my understanding LLMs are essentially trained to imitate the behavior of certain archetypes. "ai attempts to trick its creators" is a common trope. There is probably enough rogue ai and ai safety content in the training data, for this become part of the ai archetype within the model. So if we provide the system with a prompt telling it that it is an ai, it makes sense for it to behave in the way described in the article, because that is what we'd expect an ai to do

chimprich 2y ago

> autocomplete word salad generators People get very hung up on this "autocomplete" idea, but language is a linear stream. How else are you going to generate text except for one token at a time, building on what you have produced already? That's what humans do after all (at least with speech/language; it might be a bit less linear if you're writing code, but I think it's broadly true).

ben_w 2y ago
I generally have an internal monologue turning my thoughts into words; sometimes my consciousness notices the though fully formed and without needing any words, but when my conscious self decides I can therefore skip the much slower internal monologue, the bit of me that makes the internal monologue "gets annoyed" in a way that my conscious self also experiences due to being in the same brain.
- ImHereToVote 2y ago
  Doesn't the inner monologue also get formed one word at a time?
  - ben_w 2y ago
    It doesn't feel like it is one word at a time. It feels more like how the "model synthesis" algorithm looks: https://en.wikipedia.org/wiki/Model_synthesis https://en.wikipedia.org/wiki/Model_synthesis It might actually be linear — how minds actually function is in many cases demonstrably different to how it feels like to the mind doing the functioning — but it doesn't feel like it is linear.
    - ImHereToVote 2y ago
      Facts don't care about your feelings lol.
      - ben_w 2y ago
        The technology doesn't yet exist to measure the facts that generate the feelings to determine whether the feelings do or don't differ from those facts. Nobody even knows where, specifically, qualia exist in order to be able to direct technological advancement in that area.