3 ms·

Anthropic regularly publishes research papers on the subject and details different methods they use to prevent misalignment/jailbreaks/etc. And it's not even ab

Anthropic regularly publishes research papers on the subject and details different methods they use to prevent misalignment/jailbreaks/etc. And it's not even about fear of being sued, but needing to deliver some level of resilience and stability for real enterprise use cases. I think there's a pretty clear profit incentive for safer models. https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...

tovej 8mo ago
Alternative take: this is all marketing. If you pretend really hard that you're worried about safety, it makes what you're selling seem more powerful. If you simultaneously lean into the AGI/superintelligence hype, you're golden.
gessha 8mo ago
Not to be cynical about it BUT a few safety papers a year with proper support is totally within the capabilities of a single PhD student and it costs about 100-150k to fund them through a university. Not saying that’s what Anthropocene does, I’m just saying chump change for those companies.
- rrr_oh_man 8mo ago
  You are very off (unfortunately) about how little PhD students are being paid
  - pja 8mo ago
    > You are very off (unfortunately) about how little PhD students are being paid All in costs for a PhD student include university overheads & tuition fees. The total probably doesn't hit $150k but is 2-3x the stipend that the student is receiving. Someone currently working in academia might have current figures to hand.
    - latexr 8mo ago
      Worth mentioning that numbers for the US are unlikely to be representative when discussing it as a whole, though might be relevant to this specific case.
      - pja 8mo ago
        In the UK the all in cost of a PhD student starts somewhere around £45k once you include overheads I believe. If you need expensive lab support then it probably goes up from there. So about $75k for the bottom end? The quoted numbers sound about right in PPP terms in that case.
  - gessha 8mo ago
    Figure cited is what the company gets charged, not what the student gets. I’m fairly familiar with what gets thrown at students :(
- pixl97 8mo ago
  Sometimes I think people misunderstand how hard of problem AI safety actually is. It's politics and mathematics wrapped up in a black box of interactions we barely understand. More so we train them on human behavior and humans have a lot of rather unstable behaviors.
542354234235 8mo ago
Anthropic is investing, conservatively, $100+ billion in AI infrastructure and development. A 20-person research team could put out several papers a year. That would cost them what, $5 million a year, or one half of one percent? They don't have to spend much to get that kind of output.

tovej 8mo ago

Alternative take: this is all marketing. If you pretend really hard that you're worried about safety, it makes what you're selling seem more powerful. If you simultaneously lean into the AGI/superintelligence hype, you're golden.

gessha 8mo ago

Not to be cynical about it BUT a few safety papers a year with proper support is totally within the capabilities of a single PhD student and it costs about 100-150k to fund them through a university. Not saying that’s what Anthropocene does, I’m just saying chump change for those companies.

rrr_oh_man 8mo ago
You are very off (unfortunately) about how little PhD students are being paid
- pja 8mo ago
  > You are very off (unfortunately) about how little PhD students are being paid All in costs for a PhD student include university overheads & tuition fees. The total probably doesn't hit $150k but is 2-3x the stipend that the student is receiving. Someone currently working in academia might have current figures to hand.
  - latexr 8mo ago
    Worth mentioning that numbers for the US are unlikely to be representative when discussing it as a whole, though might be relevant to this specific case.
    - pja 8mo ago
      In the UK the all in cost of a PhD student starts somewhere around £45k once you include overheads I believe. If you need expensive lab support then it probably goes up from there. So about $75k for the bottom end? The quoted numbers sound about right in PPP terms in that case.
- gessha 8mo ago
  Figure cited is what the company gets charged, not what the student gets. I’m fairly familiar with what gets thrown at students :(
pixl97 8mo ago
Sometimes I think people misunderstand how hard of problem AI safety actually is. It's politics and mathematics wrapped up in a black box of interactions we barely understand. More so we train them on human behavior and humans have a lot of rather unstable behaviors.

542354234235 8mo ago

Anthropic is investing, conservatively, $100+ billion in AI infrastructure and development. A 20-person research team could put out several papers a year. That would cost them what, $5 million a year, or one half of one percent? They don't have to spend much to get that kind of output.