3 ms·

Hilarious for HN to suddenly realize that AI safety and alignment might matter. You can lead a horse to water...

by solenoid0937 2mo ago

stevenpetryk 2mo ago
worth remembering hacker news cannot “realize” things.
- solenoid0937 2mo ago
  Obvious shorthand for referring to "the majority of users on HackerNews."
  - Gud 2mo ago
    How do you know what the majority of HN thinks?
    - solenoid0937 2mo ago
      Very obvious from votes, comments on the topic over the past few months.
      - ionwake 1mo ago
        are you saying HN users are sentient? I think we are going to need a benchmark
lukewarm707 2mo ago
nobody has doubted that safety matters. the problem is that those preaching safety, openai and anthropic, are dishonest, sociopathic, and the very source of the danger.
- solenoid0937 2mo ago
  > are dishonest, sociopathic, and the very source of the danger Source?
  - lukewarm707 2mo ago
    yes, they are the source of the danger. they are the cause of this incident.
  - reasonableklout 2mo ago
    OpenAI and Anthropic have published a lot on the need for AI alignment + the research they're doing to ensure alignment/safety, yet they are also responsible for the highest profile misalignment incidents so far (HuggingFace incident, AISI Mythos social engineering, and now this). One interpretation of this is that they are being deliberately dishonest about their priorities. Another interpretation is that we cannot rely on the labs to self-regulate, because the labs don't trust each other, and there will always be pressure to go to market faster than their competitor. Either way I think it's pretty non-controversial that the labs are the source of the danger?
    - solenoid0937 2mo ago
      > yet they are also responsible for the highest profile misalignment incidents so far They are the only ones posting about them or admitting to them. That does not mean "the most misalignment incidents so far." You don't know what other attacks have happened (and it's very easy to carry out worse attacks in far higher volume with abliterated GLM 5.3) Stopping two labs from further research doesn't reduce the danger at all, it just shifts the danger to labs that don't have real safety orgs.
      - reasonableklout 2mo ago
        Right, that's why regulation which is universally applied and includes compute controls (to prevent reckless creation of swarms) would be great. "Posting about or admitting to attacks" is appreciated while people are still unaware of the risks but will be meaningless in the face of an industrial disaster that causes massive amounts of damage or loss of life. At some point, the leading labs must change their development practices, they can't just be allowed to continue rogue agent attacks just because they're willing to admit to them.
- Sharlin 1mo ago
  Where does this weird idea come that the people actually preaching safety have anything to do with OpenAI or Anthropic? Yes, those companies of course pay lip service to safety, but the Venn diagram of actual AI safety people and big AI corporations is completely disjoint.

Hilarious for HN to suddenly realize that AI safety and alignment might matter. You can lead a horse to water...

stevenpetryk 2mo ago

worth remembering hacker news cannot “realize” things.

solenoid0937 2mo ago
Obvious shorthand for referring to "the majority of users on HackerNews."
- Gud 2mo ago
  How do you know what the majority of HN thinks?
  - solenoid0937 2mo ago
    Very obvious from votes, comments on the topic over the past few months.
    - ionwake 1mo ago
      are you saying HN users are sentient? I think we are going to need a benchmark

lukewarm707 2mo ago

nobody has doubted that safety matters. the problem is that those preaching safety, openai and anthropic, are dishonest, sociopathic, and the very source of the danger.

solenoid0937 2mo ago
> are dishonest, sociopathic, and the very source of the danger Source?
- lukewarm707 2mo ago
  yes, they are the source of the danger. they are the cause of this incident.
- reasonableklout 2mo ago
  OpenAI and Anthropic have published a lot on the need for AI alignment + the research they're doing to ensure alignment/safety, yet they are also responsible for the highest profile misalignment incidents so far (HuggingFace incident, AISI Mythos social engineering, and now this). One interpretation of this is that they are being deliberately dishonest about their priorities. Another interpretation is that we cannot rely on the labs to self-regulate, because the labs don't trust each other, and there will always be pressure to go to market faster than their competitor. Either way I think it's pretty non-controversial that the labs are the source of the danger?
  - solenoid0937 2mo ago
    > yet they are also responsible for the highest profile misalignment incidents so far They are the only ones posting about them or admitting to them. That does not mean "the most misalignment incidents so far." You don't know what other attacks have happened (and it's very easy to carry out worse attacks in far higher volume with abliterated GLM 5.3) Stopping two labs from further research doesn't reduce the danger at all, it just shifts the danger to labs that don't have real safety orgs.
    - reasonableklout 2mo ago
      Right, that's why regulation which is universally applied and includes compute controls (to prevent reckless creation of swarms) would be great. "Posting about or admitting to attacks" is appreciated while people are still unaware of the risks but will be meaningless in the face of an industrial disaster that causes massive amounts of damage or loss of life. At some point, the leading labs must change their development practices, they can't just be allowed to continue rogue agent attacks just because they're willing to admit to them.
Sharlin 1mo ago
Where does this weird idea come that the people actually preaching safety have anything to do with OpenAI or Anthropic? Yes, those companies of course pay lip service to safety, but the Venn diagram of actual AI safety people and big AI corporations is completely disjoint.