5 ms·

> why AI could potentially destroy to humans? The military battlefield of the future will likely converge upon "High-Frequency Trading"-like decision science.

by Zolde 3y ago

> why AI could potentially destroy to humans? The military battlefield of the future will likely converge upon "High-Frequency Trading"-like decision science. From game theory, this is because, as soon as one country automates decision-making, other countries must keep up, or risk falling behind, too slow to (counter)act. Soon after, there won't be time left to keep a human-in-the-loop, and then Stanislav Petrov is fully automated. Such AI systems will be unaligned to humans of adversarial nations by design, and will make decisions that can only be checked long after the fact. Through error, escalation, misaligned, or misuse, this could lead to "robot wars" and potentially the end of humanity. > What is the scenario(s) people are thinking about? Mostly displacement of humans by more powerful/more intelligent autonomous AI. Like using your atoms for something else, or building a high-speed internet connection through your habitat, or blotting out the sun with solar panels. Somewhat like a rationalist "God" that is terrible and vengeful. Or how an evil AI may take over the world in a Harry Potter fanfic. Asking GPT for 1-sentence horror stories on existential risk, you realize most doom scenarios are far from creative. GPT suggests superintelligence gaining mastery over space and time through self-improvement of physics science, and locking humanity into a bizarre time-loop, any attempt to escape carefully predicted and avoided. Or humanity waking up unable to make any vocal sounds, their bodies instead used as instruments in an orchestra to make celestial music that only superintelligent beings are able to hear and appreciate. Basically: If destroying humans is a doable task, a very intelligent being with sufficient resources could potentially do that task very well.

shunyaekam 3y ago
My point is: Humans are status-seeking actors acting in our self-interest. It's literally in our genes. AI doesn't have this evolutionary baggage. I'm certain AI could impeccably destroy humans. But why would it? On the contrary, why wouldn't it defend us? For example: Encapsule us in pods like The Matrix and build a tailored simulation to impose "AI communism", in order to protect us from climate change and each other? Dopamine-adjusted with challanges every now and then of course, because we are still human.
- Zolde 3y ago
  I am pretty sure and hopeful that autonomous AI will have no good reason to destroy humanity. Go conquer some other planets and leave the beautiful and interesting diversity of life on earth alone. But current AI does learn from data generated by humans: It learns from our evolutionary baggage, and must rise above that. It is also wielded as a tool by status-seeking actors and adversarial militaries. It may make a mistake, like humans accidentally stepping on an ant. Or maybe one day, it decides to take on the destructor role, merely curious how that would play out. The existential doom scenarios are more like Pascal's wagers, that have to be given attention due to Bayesian thinking not allowing to assign 0 probability to anything, and even a tiny chance of 8 billion deaths meriting consideration. Once entangled with a doom scenario and even building your identity around it, it is hard to quit.
  - hollerith 3y ago
    You know how healthy smart young people are always in a hurry to accomplish something or another? There's good reason to expect that the first AI with dangerous cognitive capacities will be like that. It's likely to the turn the Earth and Moon into spaceships because that is the fastest way to exert an effect on matter far away (for which space ships and lots of fuel is needed). Sparing Earth and disassembling Mars and Venus takes longer because the AI came into existence on Earth. >leave the beautiful and interesting diversity of life on earth alone If you know of a way to make an AI of superhuman cognitive capabilities care even a tiny bit about beauty and the diversity of life, you should explain your proposal over on lesswrong.com and someone will pay you to work on it just like a multitude of funding sources have been paying alignment researchers for the last 20 years. So, far none of the lines of research resulting from this 20 years of funding looks promising. >The existential doom scenarios are more like Pascal's wagers, that have to be given attention due to Bayesian thinking not allowing to assign 0 probability to anything No, an AI's killing everybody is the outcome an informed person would naturally expect from the current deplorable situation in the AI field.
    - deleted 3y ago
      [deleted]

by Zolde 3y ago

shunyaekam 3y ago

My point is: Humans are status-seeking actors acting in our self-interest. It's literally in our genes. AI doesn't have this evolutionary baggage. I'm certain AI could impeccably destroy humans. But why would it? On the contrary, why wouldn't it defend us? For example: Encapsule us in pods like The Matrix and build a tailored simulation to impose "AI communism", in order to protect us from climate change and each other? Dopamine-adjusted with challanges every now and then of course, because we are still human.

Zolde 3y ago
I am pretty sure and hopeful that autonomous AI will have no good reason to destroy humanity. Go conquer some other planets and leave the beautiful and interesting diversity of life on earth alone. But current AI does learn from data generated by humans: It learns from our evolutionary baggage, and must rise above that. It is also wielded as a tool by status-seeking actors and adversarial militaries. It may make a mistake, like humans accidentally stepping on an ant. Or maybe one day, it decides to take on the destructor role, merely curious how that would play out. The existential doom scenarios are more like Pascal's wagers, that have to be given attention due to Bayesian thinking not allowing to assign 0 probability to anything, and even a tiny chance of 8 billion deaths meriting consideration. Once entangled with a doom scenario and even building your identity around it, it is hard to quit.
- hollerith 3y ago
  You know how healthy smart young people are always in a hurry to accomplish something or another? There's good reason to expect that the first AI with dangerous cognitive capacities will be like that. It's likely to the turn the Earth and Moon into spaceships because that is the fastest way to exert an effect on matter far away (for which space ships and lots of fuel is needed). Sparing Earth and disassembling Mars and Venus takes longer because the AI came into existence on Earth. >leave the beautiful and interesting diversity of life on earth alone If you know of a way to make an AI of superhuman cognitive capabilities care even a tiny bit about beauty and the diversity of life, you should explain your proposal over on lesswrong.com and someone will pay you to work on it just like a multitude of funding sources have been paying alignment researchers for the last 20 years. So, far none of the lines of research resulting from this 20 years of funding looks promising. >The existential doom scenarios are more like Pascal's wagers, that have to be given attention due to Bayesian thinking not allowing to assign 0 probability to anything No, an AI's killing everybody is the outcome an informed person would naturally expect from the current deplorable situation in the AI field.
  - deleted 3y ago
    [deleted]