4 ms·

At this rate, we will have a paper about every single psychological aspect discovered in LLMs. This could have been just a reddit post. Every phenomenon found

At this rate, we will have a paper about every single psychological aspect discovered in LLMs. This could have been just a reddit post. Every phenomenon found massively in training set will eventually pop up in LLMs. I just don't find the discoveries made in these papers very meaningful. Edit: May be I am being too short sighted. The researchers probably start from "Humans are good at X and the training data had many examples of X. How good is LLM at X?" and X happens to be deception this time.

SkyMarshal 2y ago
Even so they should probably get documented in the scientific literature anyway, to encourage review and replication, reduce unintentionally duplicated work, and provide references for further experimentation.
- picometer 2y ago
  Yes, but in a major journal like PNAS? (Proceedings of the National Academy of Sciences)
picometer 2y ago
I do think it’s an interesting line of inquiry… but not robust enough. E.g. this paper would be much more interesting if it measured the threshold at which the LLM starts to become good at X, and linked that threshold to the number and character of training examples of X. Then, maybe, we can begin to think about comparing the LLM to a human. Alas, it requires access to the training data to do that study, and it requires a vast amount of compute to do it robustly.
wg0 2y ago
At this point - is this even real science? I mean for every single claim made in such papers, these claims are hard to reproduce even with the same model for a set of different inputs. LLMs have no idea they're adopting some behavior. Just numbers being multiplied and token probabilities being calculated based on the distributions extracted from huge sample corpus.
- llamaimperative 2y ago
  > LLMs have no idea they're adopting some behavior I suspect you're right, but we absolutely do not know this. We have no idea when, why, or how consciousness emerges. We have no reason to think there's something inherently special about the atoms in an animal brain that imbues them with the capability to "produce" consciousness and that the atoms on a silicon wafer are inherently incapable of doing the same thing. > Just numbers being multiplied and token probabilities being calculated based on the distributions extracted from huge sample corpus. This is not wildly different from how brains appear to work. Sure we get a gigantic, constant, highly heterogeneous multimodal stream of data inputs from our sensory systems and we get to test theories about our world model via our own actions, but of course, AI companies are all trying to produce analogs for each of these for their models too.
  - simple_quest_9 2y ago
    LLMs don't have self-perception, egos, etc. right?
    - deleted 2y ago
      [deleted]
    - llamaimperative 2y ago
      1. They definitely have self-perception to the extent necessary to "know they're adopting a behavior" if there is something there to "do" the "knowing." They ingest their own outputs as a normal matter of operation. 2. We don't know if there's something conscious there to do the knowing 3. Ego doesn't seem necessary either to being conscious or for self-perception I suspect you're right, but we don't understand nearly enough about consciousness to be confident about any of these questions.
      - wg0 2y ago
        We don't know is the main argument. So... Can I worship an LLM and start my own religion? Because we really don't know if consciousness requires X, Y or Z let alone what a a superior all knowing almighty's consciousness consists of. To me, LLMs look so superior in some ways and maybe they are the gods now that they have manifested themselves as large blobs of floating point numbers when humanity is mature enough to receive them. We don't know.
gwern 2y ago
> Every phenomenon found massively in training set will eventually pop up in LLMs. That's a nontrivial and controversial claim that many on HN would hotly disagree with - that LLMs are guaranteed to do everything humans do, if you just feed them enough ordinary data, and so finding behaviors like deception or manipulation is not even worth writing up. And I think even you would abandon that nonchalant position if the "phenomenon" were sufficiently extreme... ("Sure, the LLM autonomously stalked and hunted down a person that a malicious user badmouthed, and called in a swatting using the Twilio API to get the person killed by the police; but you can find plenty of stuff like that in the Internet training corpus, so I don't understand why anyone is interested. This could have just been a reddit post.")

SkyMarshal 2y ago

Even so they should probably get documented in the scientific literature anyway, to encourage review and replication, reduce unintentionally duplicated work, and provide references for further experimentation.

picometer 2y ago
Yes, but in a major journal like PNAS? (Proceedings of the National Academy of Sciences)

picometer 2y ago

I do think it’s an interesting line of inquiry… but not robust enough. E.g. this paper would be much more interesting if it measured the threshold at which the LLM starts to become good at X, and linked that threshold to the number and character of training examples of X. Then, maybe, we can begin to think about comparing the LLM to a human. Alas, it requires access to the training data to do that study, and it requires a vast amount of compute to do it robustly.

wg0 2y ago

At this point - is this even real science? I mean for every single claim made in such papers, these claims are hard to reproduce even with the same model for a set of different inputs. LLMs have no idea they're adopting some behavior. Just numbers being multiplied and token probabilities being calculated based on the distributions extracted from huge sample corpus.

llamaimperative 2y ago
> LLMs have no idea they're adopting some behavior I suspect you're right, but we absolutely do not know this. We have no idea when, why, or how consciousness emerges. We have no reason to think there's something inherently special about the atoms in an animal brain that imbues them with the capability to "produce" consciousness and that the atoms on a silicon wafer are inherently incapable of doing the same thing. > Just numbers being multiplied and token probabilities being calculated based on the distributions extracted from huge sample corpus. This is not wildly different from how brains appear to work. Sure we get a gigantic, constant, highly heterogeneous multimodal stream of data inputs from our sensory systems and we get to test theories about our world model via our own actions, but of course, AI companies are all trying to produce analogs for each of these for their models too.
- simple_quest_9 2y ago
  LLMs don't have self-perception, egos, etc. right?
  - deleted 2y ago
    [deleted]
  - llamaimperative 2y ago
    1. They definitely have self-perception to the extent necessary to "know they're adopting a behavior" if there is something there to "do" the "knowing." They ingest their own outputs as a normal matter of operation. 2. We don't know if there's something conscious there to do the knowing 3. Ego doesn't seem necessary either to being conscious or for self-perception I suspect you're right, but we don't understand nearly enough about consciousness to be confident about any of these questions.
    - wg0 2y ago
      We don't know is the main argument. So... Can I worship an LLM and start my own religion? Because we really don't know if consciousness requires X, Y or Z let alone what a a superior all knowing almighty's consciousness consists of. To me, LLMs look so superior in some ways and maybe they are the gods now that they have manifested themselves as large blobs of floating point numbers when humanity is mature enough to receive them. We don't know.

gwern 2y ago

> Every phenomenon found massively in training set will eventually pop up in LLMs. That's a nontrivial and controversial claim that many on HN would hotly disagree with - that LLMs are guaranteed to do everything humans do, if you just feed them enough ordinary data, and so finding behaviors like deception or manipulation is not even worth writing up. And I think even you would abandon that nonchalant position if the "phenomenon" were sufficiently extreme... ("Sure, the LLM autonomously stalked and hunted down a person that a malicious user badmouthed, and called in a swatting using the Twilio API to get the person killed by the police; but you can find plenty of stuff like that in the Internet training corpus, so I don't understand why anyone is interested. This could have just been a reddit post.")