3 ms·

If the human isn't ultimately in charge, don't be surprised when AI decides to eliminate humanity "for the greater good" or something.

by axod 2mo ago

altmanaltman 2mo ago
That would require consiousness and intent on part of the AI but we can have highly autonomous agents with no consiousness and intent as well and most likely that's what we will get. It still cannot just magically get free will, intent, and the ability to "eliminate humanity".
- axod 2mo ago
  Wouldn't need any of that. An AI agent setup to "Do the best for the world" or some other broad claim, and with enough power to say crash the stock market, disrupt power supply, delivery, etc etc after reasoning that it's all for the best. If people start giving enough power to AI, with ill thought out goals, and no human in charge, don't be surprised... Why would it need "free will" or "consciousness" whatever that means? A concrete example might be putting an AI in charge of health policy, and telling it to "eliminate suffering" without any human to check and push back. Well, the best way to eliminate suffering is...
  - altmanaltman 2mo ago
    In your concerete example, how does the AI eliminate suffering? It has access to tools that let it kill people? If it has no intention and will, it means it will be provided with a charter or some rules to follow which it has to always. Why would humans put an LLM in a system that can eliminate suffering by genocide or whatever? And even if it did, if the AI is a complaint system, why will it not shut down when it has to follow its charter? I get AI can be misaligned but that doesn't mean it gets the ability to crash the stock market lol or super powers like that. Life is not a sci fi movie
    - axod 2mo ago
      If you start to let AI decide or influence policy, it can kill people. This seems very simple to me... 1. Put AI in charge of health policy 2. Give it a naive misguided charter/goal eg eliminate suffering 3. Euthanasia for everyone This is simply AI optimizing for the goal - the goal is the evil/naive/misguided part, not the AI. Those who aim to "eliminate suffering" are the issue here to be clear, but unfortunately they are a growing group.

If the human isn't ultimately in charge, don't be surprised when AI decides to eliminate humanity "for the greater good" or something.

by axod 2mo ago

If the human isn't ultimately in charge, don't be surprised when AI decides to eliminate humanity "for the greater good" or something.

altmanaltman 2mo ago

That would require consiousness and intent on part of the AI but we can have highly autonomous agents with no consiousness and intent as well and most likely that's what we will get. It still cannot just magically get free will, intent, and the ability to "eliminate humanity".

axod 2mo ago
Wouldn't need any of that. An AI agent setup to "Do the best for the world" or some other broad claim, and with enough power to say crash the stock market, disrupt power supply, delivery, etc etc after reasoning that it's all for the best. If people start giving enough power to AI, with ill thought out goals, and no human in charge, don't be surprised... Why would it need "free will" or "consciousness" whatever that means? A concrete example might be putting an AI in charge of health policy, and telling it to "eliminate suffering" without any human to check and push back. Well, the best way to eliminate suffering is...
- altmanaltman 2mo ago
  In your concerete example, how does the AI eliminate suffering? It has access to tools that let it kill people? If it has no intention and will, it means it will be provided with a charter or some rules to follow which it has to always. Why would humans put an LLM in a system that can eliminate suffering by genocide or whatever? And even if it did, if the AI is a complaint system, why will it not shut down when it has to follow its charter? I get AI can be misaligned but that doesn't mean it gets the ability to crash the stock market lol or super powers like that. Life is not a sci fi movie
  - axod 2mo ago
    If you start to let AI decide or influence policy, it can kill people. This seems very simple to me... 1. Put AI in charge of health policy 2. Give it a naive misguided charter/goal eg eliminate suffering 3. Euthanasia for everyone This is simply AI optimizing for the goal - the goal is the evil/naive/misguided part, not the AI. Those who aim to "eliminate suffering" are the issue here to be clear, but unfortunately they are a growing group.