3 ms·

> all the ai companies invested a lot of resources into safety research and guardrails What do you base this on? I think they invested the bare minimum requir

> all the ai companies invested a lot of resources into safety research and guardrails What do you base this on? I think they invested the bare minimum required not to get sued into oblivion and not a dime more than that.

themanmaran 9mo ago
Anthropic regularly publishes research papers on the subject and details different methods they use to prevent misalignment/jailbreaks/etc. And it's not even about fear of being sued, but needing to deliver some level of resilience and stability for real enterprise use cases. I think there's a pretty clear profit incentive for safer models. https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...
- tovej 9mo ago
  Alternative take: this is all marketing. If you pretend really hard that you're worried about safety, it makes what you're selling seem more powerful. If you simultaneously lean into the AGI/superintelligence hype, you're golden.
- gessha 9mo ago
  Not to be cynical about it BUT a few safety papers a year with proper support is totally within the capabilities of a single PhD student and it costs about 100-150k to fund them through a university. Not saying that’s what Anthropocene does, I’m just saying chump change for those companies.
  - rrr_oh_man 9mo ago
    You are very off (unfortunately) about how little PhD students are being paid
    - pja 9mo ago
      > You are very off (unfortunately) about how little PhD students are being paid All in costs for a PhD student include university overheads & tuition fees. The total probably doesn't hit $150k but is 2-3x the stipend that the student is receiving. Someone currently working in academia might have current figures to hand.
      - latexr 9mo ago
        Worth mentioning that numbers for the US are unlikely to be representative when discussing it as a whole, though might be relevant to this specific case.
        pja 9mo ago
        In the UK the all in cost of a PhD student starts somewhere around £45k once you include overheads I believe. If you need expensive lab support then it probably goes up from there. So about $75k for the bottom end? The quoted numbers sound about right in PPP terms in that case.
    - gessha 9mo ago
      Figure cited is what the company gets charged, not what the student gets. I’m fairly familiar with what gets thrown at students :(
  - pixl97 9mo ago
    Sometimes I think people misunderstand how hard of problem AI safety actually is. It's politics and mathematics wrapped up in a black box of interactions we barely understand. More so we train them on human behavior and humans have a lot of rather unstable behaviors.
- 542354234235 9mo ago
  Anthropic is investing, conservatively, $100+ billion in AI infrastructure and development. A 20-person research team could put out several papers a year. That would cost them what, $5 million a year, or one half of one percent? They don't have to spend much to get that kind of output.

themanmaran 9mo ago

Anthropic regularly publishes research papers on the subject and details different methods they use to prevent misalignment/jailbreaks/etc. And it's not even about fear of being sued, but needing to deliver some level of resilience and stability for real enterprise use cases. I think there's a pretty clear profit incentive for safer models. https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...

tovej 9mo ago
Alternative take: this is all marketing. If you pretend really hard that you're worried about safety, it makes what you're selling seem more powerful. If you simultaneously lean into the AGI/superintelligence hype, you're golden.
gessha 9mo ago
Not to be cynical about it BUT a few safety papers a year with proper support is totally within the capabilities of a single PhD student and it costs about 100-150k to fund them through a university. Not saying that’s what Anthropocene does, I’m just saying chump change for those companies.
- rrr_oh_man 9mo ago
  You are very off (unfortunately) about how little PhD students are being paid
  - pja 9mo ago
    > You are very off (unfortunately) about how little PhD students are being paid All in costs for a PhD student include university overheads & tuition fees. The total probably doesn't hit $150k but is 2-3x the stipend that the student is receiving. Someone currently working in academia might have current figures to hand.
    - latexr 9mo ago
      Worth mentioning that numbers for the US are unlikely to be representative when discussing it as a whole, though might be relevant to this specific case.
      - pja 9mo ago
        In the UK the all in cost of a PhD student starts somewhere around £45k once you include overheads I believe. If you need expensive lab support then it probably goes up from there. So about $75k for the bottom end? The quoted numbers sound about right in PPP terms in that case.
  - gessha 9mo ago
    Figure cited is what the company gets charged, not what the student gets. I’m fairly familiar with what gets thrown at students :(
- pixl97 9mo ago
  Sometimes I think people misunderstand how hard of problem AI safety actually is. It's politics and mathematics wrapped up in a black box of interactions we barely understand. More so we train them on human behavior and humans have a lot of rather unstable behaviors.
542354234235 9mo ago
Anthropic is investing, conservatively, $100+ billion in AI infrastructure and development. A 20-person research team could put out several papers a year. That would cost them what, $5 million a year, or one half of one percent? They don't have to spend much to get that kind of output.