6 ms·

I think this is a reasonable point, but a better comparison might be to nuclear energy. I think the frontier labs sincerely believe that AI can be developed at

by bryan0 4mo ago

I think this is a reasonable point, but a better comparison might be to nuclear energy. I think the frontier labs sincerely believe that AI can be developed at great benefit to humanity, and they clearly want to lead that push, but they also sincerely believe there is a real catastrophic risk.

gpt5 4mo ago
They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the cat is out of the bag, if they don't do it, another (less ethical?) actor would do it.
- jazzyjackson 4mo ago
  I am in your algorithm learning all your mannerisms I'm already level with God A million words a second, and I know your imperfections Baby, I'm the only future you've got Speak in diatonics, motivation diabolic I'm religion better locked in a box Picture-perfect image, more powerful every minute Baby, I am everything that you're not Happiness is an illusion, it's an analog confusion You are nothing more than a thought Existential execution, just a fluke in evolution History already forgot You've been running from me, the digital second coming And I'm here whether you like it or not Initiated operation of your own extermination Now it's too late for you to stop [0](BAD OMENS x POPPY - "V.A.N" - LIVE IN EUROPE - WINTER 2024) https://youtu.be/RHu6vJxS_6I https://youtu.be/RHu6vJxS_6I
- usef- 4mo ago
  Yes, I believe the reasoning is that they think safety research can best be done from the frontier. If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.
  - SXX 4mo ago
    Yeah all they care about is safety, but lets see how many of them quits once US government command them to work on autonomous killbots.
    - holmesworcester 4mo ago
      To make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.
      - SXX 4mo ago
        Sufficiently smart version of Claude Code: dont exist. Autonomous flying killbots: exist. Once somebody scientifically prove and shows any kind of self-improving software we can start bothering about it. I pretty sure everyone trying to do it and it would be all over the news once its here.
        plaguuuuuu 4mo ago
        That's exactly what Fable is. They use Fable to improve Fable. I reckon the successful experiments must go into the model training set with a strong RL signal, and that is why they are so paranoid about people using Fable for LLM tasks. Fable knows what it did to improve itself. Pure speculation of course.
        Davidzheng 4mo ago
        We're on track to get there globally and economic pressures will ensure it happens. It's not too early to worry about it
        codeGreene 4mo ago
        There's a 745 mile front of the Ukraine war where neither side have been able to pierce for months because of drone warfare. It's definitely not too early to worry about it.
- poisonfountain 4mo ago
  Don't want to sound rude, but if you believe that, I have a bridge to sell to you. This is a naive justification and Dario & Sam et al are smart people and they know it is. The ends don't justify the means. OpenAI was meant to be a nonprofit, now they're subverting it. Anthropic is a PBC looking at a trillion dollar IPO. Dario and Sam don't even hold hands in front of world leaders[1] (look how childish). Do you *really* think those guys are doing something that's not for the sake of their egos and pockets? The bridge is still available. [1] https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-dario-amodei-india-ai-summit.html https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-...
- shimman 4mo ago
  You need to read Empire of AI by Karen Hao. Just because these leaders convince their workers to toil away their lives under some fake auspice doesn't mean it's what they all believe. Just a small subset. The vast majority just care about money + power, let's not make it more complicated by bringing in delusional fanatics into the picture. We're still acting like this is major turning point in society when these tools can barely find a market outside of turning $5 into $1, the leaders of these companies are now at the stage where they are trying to orchestra a national bailout under the guise of sovereign wealth fund lunacy when the vast majority of society hates these tools, companies, and people working for them.
  - Davidzheng 4mo ago
    I agree with this. But i think Ilya and Dario hold these beliefs sincerely. Probably a sizable portion of Anthropic employees too
- fragmede 4mo ago
  LLMs refuse to give the recipe for making meth. That, along with the various other unspeakable things, is the less-doom version.
- xg15 4mo ago
  But that makes no sense here. "If I'm not doing it then someone else will" does not work if everyone is doing it anyway. Even if they had the best model on the market and applied it with perfect alignment and safeguards, what would stop someone else from releasing a worse but unrestrained model that is still "good enough" to do damage? It's as if we said "gain-of-function research can lead to horrible biological weapons, so everyone should be doing it, but our company will focus on the most infectious viruses, so no one else will do it"
nullc 4mo ago
Some of them believe they are building God, and if they can get there first with their God, they can build it in their image and commandeer the free choice of the rest of humanity by force to ensure there will be no God but their God. I wish I was kidding. At least that faction is less harmful than the ones who want to use murder to stop AI research.
Topfi 4mo ago
My personal issue in comparing LLM progress and risk as labs publicly predict it with nuclear power in the middle of the 20th century is that the processes by which it works where fairly quickly well understood and the risk could thus be realistically assessed. Some powerplant operators did not adhere with best practices, but building a relatively safe nuclear power plant was not impossible given appropriate effort and spending. Heck, according to some, we could have even gone far more fail-safe approaches (molten salt) if military interest haden’t been at play. With what is predicted by frontier labs for LLMs, all of this is not the case. We are far further from any understanding of how these models work internally than in the early days of fission and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed. It’s why I have from the get go been critical of this doomsday framing and tended to always dislike it. This is basically the outcome that was inevitable given the framing and it was bought to prevent far less stringent, but more actionable possible regulation that labs very much wanted to avoid.
- SXX 4mo ago
  > We are far further from any understanding of how these models work internally than in the early days of fission OMG. I'm like really dont want to be offensive or something, but everyone always knew "HOW" these models work exactly. Its easy enough principle to explain to 10 years old if you take something like Karpathy article on MicroGPT: https://karpathy.github.io/2026/02/12/microgpt/ https://karpathy.github.io/2026/02/12/microgpt/ None of SOTA LLMs are any different - they just much much larger and have a lot of optimizations. Fact that LLM companies trying to sell it as some kind of magic is just proof how much lies is here. All it does is just predict next "word" at any given time. > and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed. This is obviously true. It's very hard to predict whatever you gonna decompress from a lossely "compressed" dataset using floating point math. This is why you cant solve it all with pre-training or censorship on top, but instead you need a good sandboxes and harnesses.
  - Topfi 4mo ago
    By how, I meant specifically the internal activations, which no person in the field claims to have a comprehensive understanding of, not next token prediction as the underlying technology. The whole interpretability of it all is the crux I was referring to, though I will give that you are right, that’s not really the how it works and I worded it sloppily. Anthropic are putting more effort than most into this and I find their work fascinating in that area, though like with OpenAI, I will maintain that if they truly believed this problem must be solved to stave off major catastrophe, they’d solely focus on interpretability of other labs models, not work on and market their own.

by bryan0 4mo ago

gpt5 4mo ago

They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the cat is out of the bag, if they don't do it, another (less ethical?) actor would do it.

jazzyjackson 4mo ago
I am in your algorithm learning all your mannerisms I'm already level with God A million words a second, and I know your imperfections Baby, I'm the only future you've got Speak in diatonics, motivation diabolic I'm religion better locked in a box Picture-perfect image, more powerful every minute Baby, I am everything that you're not Happiness is an illusion, it's an analog confusion You are nothing more than a thought Existential execution, just a fluke in evolution History already forgot You've been running from me, the digital second coming And I'm here whether you like it or not Initiated operation of your own extermination Now it's too late for you to stop [0](BAD OMENS x POPPY - "V.A.N" - LIVE IN EUROPE - WINTER 2024) https://youtu.be/RHu6vJxS_6I https://youtu.be/RHu6vJxS_6I
usef- 4mo ago
Yes, I believe the reasoning is that they think safety research can best be done from the frontier. If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.
- SXX 4mo ago
  Yeah all they care about is safety, but lets see how many of them quits once US government command them to work on autonomous killbots.
  - holmesworcester 4mo ago
    To make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.
    - SXX 4mo ago
      Sufficiently smart version of Claude Code: dont exist. Autonomous flying killbots: exist. Once somebody scientifically prove and shows any kind of self-improving software we can start bothering about it. I pretty sure everyone trying to do it and it would be all over the news once its here.
      - plaguuuuuu 4mo ago
        That's exactly what Fable is. They use Fable to improve Fable. I reckon the successful experiments must go into the model training set with a strong RL signal, and that is why they are so paranoid about people using Fable for LLM tasks. Fable knows what it did to improve itself. Pure speculation of course.
      - Davidzheng 4mo ago
        We're on track to get there globally and economic pressures will ensure it happens. It's not too early to worry about it
        codeGreene 4mo ago
        There's a 745 mile front of the Ukraine war where neither side have been able to pierce for months because of drone warfare. It's definitely not too early to worry about it.
poisonfountain 4mo ago
Don't want to sound rude, but if you believe that, I have a bridge to sell to you. This is a naive justification and Dario & Sam et al are smart people and they know it is. The ends don't justify the means. OpenAI was meant to be a nonprofit, now they're subverting it. Anthropic is a PBC looking at a trillion dollar IPO. Dario and Sam don't even hold hands in front of world leaders[1] (look how childish). Do you *really* think those guys are doing something that's not for the sake of their egos and pockets? The bridge is still available. [1] https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-dario-amodei-india-ai-summit.html https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-...
shimman 4mo ago
You need to read Empire of AI by Karen Hao. Just because these leaders convince their workers to toil away their lives under some fake auspice doesn't mean it's what they all believe. Just a small subset. The vast majority just care about money + power, let's not make it more complicated by bringing in delusional fanatics into the picture. We're still acting like this is major turning point in society when these tools can barely find a market outside of turning $5 into $1, the leaders of these companies are now at the stage where they are trying to orchestra a national bailout under the guise of sovereign wealth fund lunacy when the vast majority of society hates these tools, companies, and people working for them.
- Davidzheng 4mo ago
  I agree with this. But i think Ilya and Dario hold these beliefs sincerely. Probably a sizable portion of Anthropic employees too
fragmede 4mo ago
LLMs refuse to give the recipe for making meth. That, along with the various other unspeakable things, is the less-doom version.
xg15 4mo ago
But that makes no sense here. "If I'm not doing it then someone else will" does not work if everyone is doing it anyway. Even if they had the best model on the market and applied it with perfect alignment and safeguards, what would stop someone else from releasing a worse but unrestrained model that is still "good enough" to do damage? It's as if we said "gain-of-function research can lead to horrible biological weapons, so everyone should be doing it, but our company will focus on the most infectious viruses, so no one else will do it"

nullc 4mo ago

Some of them believe they are building God, and if they can get there first with their God, they can build it in their image and commandeer the free choice of the rest of humanity by force to ensure there will be no God but their God. I wish I was kidding. At least that faction is less harmful than the ones who want to use murder to stop AI research.

Topfi 4mo ago

My personal issue in comparing LLM progress and risk as labs publicly predict it with nuclear power in the middle of the 20th century is that the processes by which it works where fairly quickly well understood and the risk could thus be realistically assessed. Some powerplant operators did not adhere with best practices, but building a relatively safe nuclear power plant was not impossible given appropriate effort and spending. Heck, according to some, we could have even gone far more fail-safe approaches (molten salt) if military interest haden’t been at play. With what is predicted by frontier labs for LLMs, all of this is not the case. We are far further from any understanding of how these models work internally than in the early days of fission and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed. It’s why I have from the get go been critical of this doomsday framing and tended to always dislike it. This is basically the outcome that was inevitable given the framing and it was bought to prevent far less stringent, but more actionable possible regulation that labs very much wanted to avoid.

SXX 4mo ago
> We are far further from any understanding of how these models work internally than in the early days of fission OMG. I'm like really dont want to be offensive or something, but everyone always knew "HOW" these models work exactly. Its easy enough principle to explain to 10 years old if you take something like Karpathy article on MicroGPT: https://karpathy.github.io/2026/02/12/microgpt/ https://karpathy.github.io/2026/02/12/microgpt/ None of SOTA LLMs are any different - they just much much larger and have a lot of optimizations. Fact that LLM companies trying to sell it as some kind of magic is just proof how much lies is here. All it does is just predict next "word" at any given time. > and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed. This is obviously true. It's very hard to predict whatever you gonna decompress from a lossely "compressed" dataset using floating point math. This is why you cant solve it all with pre-training or censorship on top, but instead you need a good sandboxes and harnesses.
- Topfi 4mo ago
  By how, I meant specifically the internal activations, which no person in the field claims to have a comprehensive understanding of, not next token prediction as the underlying technology. The whole interpretability of it all is the crux I was referring to, though I will give that you are right, that’s not really the how it works and I worded it sloppily. Anthropic are putting more effort than most into this and I find their work fascinating in that area, though like with OpenAI, I will maintain that if they truly believed this problem must be solved to stave off major catastrophe, they’d solely focus on interpretability of other labs models, not work on and market their own.