3 ms·

I think we should 5% be worried about AI safety, and 95% be worried about climate change. Despite all the progress in green energy, every year brings record hig

I think we should 5% be worried about AI safety, and 95% be worried about climate change. Despite all the progress in green energy, every year brings record high carbon emissions and record temperatures. It's possible we'll upset planetary systems and create millions (billions?) of migrants, upending global politics, and driving more countries to ethno-nationalist authoritarianism. Not saying AI safety issues won't happen, but I just think we have far bigger fish to fry. To me AI Power consumption is more worrisome than Safety per-se.

NoahZuniga 1y ago
So if we should spend (as in actually spend, or as in not choosing the cheapest option but instead choosing the green option) ie $300 billion on climate change, we should spend $15 billion on AI-risk?
- mcswell 1y ago
  Can you say "non sequitur"? I knew you could! Worrying about s.t. at X% does not mean spending X% to fix it. You might need to spend more, or you might need to spend less. Seat belts are a good example: a cheap (if partial) solution to a dangerous problem.
ACCount37 1y ago
The other way around. Climate change can have its 5%. The reason is that climate change is simply not an extinction risk. It has a considerable death and suffering potential - but nowhere near the ridiculous lethality of "we fucked up and now there's a brand new nonhuman civilization of weird immaterial beings rising to power right here on Earth". If the climate change was the biggest risk humanity was facing, things would be pretty chill. Unfortunately, it isn't.
- softwaredoug 1y ago
  IMO, realistically, AI Safety isn't about killer robots. It's about another Therac 25 Incident because someone vibe coded a radiology machine and didn't know how the code worked. Or someone gave an agent insane levels of permissions to use a tool that impacts the physical world, and the agent started pressing dangerous buttons during a reasoning loop (not because it has intent to kill humans) There are a bunch of mundane AI Safety risks that don't have to do with robots taking over.
  - ACCount37 1y ago
    "Boring" AI safety is not a major risk. It's basically an extension of "humans can be reckless and incompetent" - a threat faced by human civilization since before recorded history. There's a very limited amount of people a Therac 25 can kill. Even Bhopal disaster has only caused this much harm. Now, an AI that can play the game of human politics and always win, the way a skilled human can always win against the dumb AI bots in Civilization V? There is no upper bound on how bad that can go.
    - array_key_first 1y ago
      The difference is scale. Humans can be reckless, but the scale is so unbelievably small that's its not a big problem. Like, you can't accidently do the Holocaust. Because that requires deliberate coordination of millions of people. Can AI accidently do the Holocaust? Yeah, probably. Its software, it can duplicate itself infinitely.
    - saltcured 1y ago
      I know I've got a hard-earned cynical streak, but I am more worried about the mass sociological cascade. I absolutely don't worry about some ultra smart AI having malignant intent or even being enslaved by someone with malignant intent. Instead, it's the weird Tulip Mania or Salem Witch Trial risks that I do not know how to quantify. How severely can the mass delusions and magical thinking around AI mix with other regressive social trends? I do not relish the mass effect of people deferring their decision making to glorified Magic 8 Ball devices, treating them as oracles rather than syntactic lava lamps. Will it lay waste to a generation of education, thinking, science, and policy so that society itself acts out some kind of Therac 25 or Bhopal incident at scale?
  - 1718627440 1y ago
    It think it is about killer robots as in someone intentionally deploying Therac 25 Incidents.
- filoeleven 1y ago
  So a real, actual problem that has been intentionally trivialized for over half a century and is only getting worse only deserves 5% attention, while a paranoid fantasy based on a word salad spinner and billions in marketing hype deserves 95%?

NoahZuniga 1y ago

So if we should spend (as in actually spend, or as in not choosing the cheapest option but instead choosing the green option) ie $300 billion on climate change, we should spend $15 billion on AI-risk?

mcswell 1y ago
Can you say "non sequitur"? I knew you could! Worrying about s.t. at X% does not mean spending X% to fix it. You might need to spend more, or you might need to spend less. Seat belts are a good example: a cheap (if partial) solution to a dangerous problem.

ACCount37 1y ago

The other way around. Climate change can have its 5%. The reason is that climate change is simply not an extinction risk. It has a considerable death and suffering potential - but nowhere near the ridiculous lethality of "we fucked up and now there's a brand new nonhuman civilization of weird immaterial beings rising to power right here on Earth". If the climate change was the biggest risk humanity was facing, things would be pretty chill. Unfortunately, it isn't.

softwaredoug 1y ago
IMO, realistically, AI Safety isn't about killer robots. It's about another Therac 25 Incident because someone vibe coded a radiology machine and didn't know how the code worked. Or someone gave an agent insane levels of permissions to use a tool that impacts the physical world, and the agent started pressing dangerous buttons during a reasoning loop (not because it has intent to kill humans) There are a bunch of mundane AI Safety risks that don't have to do with robots taking over.
- ACCount37 1y ago
  "Boring" AI safety is not a major risk. It's basically an extension of "humans can be reckless and incompetent" - a threat faced by human civilization since before recorded history. There's a very limited amount of people a Therac 25 can kill. Even Bhopal disaster has only caused this much harm. Now, an AI that can play the game of human politics and always win, the way a skilled human can always win against the dumb AI bots in Civilization V? There is no upper bound on how bad that can go.
  - array_key_first 1y ago
    The difference is scale. Humans can be reckless, but the scale is so unbelievably small that's its not a big problem. Like, you can't accidently do the Holocaust. Because that requires deliberate coordination of millions of people. Can AI accidently do the Holocaust? Yeah, probably. Its software, it can duplicate itself infinitely.
  - saltcured 1y ago
    I know I've got a hard-earned cynical streak, but I am more worried about the mass sociological cascade. I absolutely don't worry about some ultra smart AI having malignant intent or even being enslaved by someone with malignant intent. Instead, it's the weird Tulip Mania or Salem Witch Trial risks that I do not know how to quantify. How severely can the mass delusions and magical thinking around AI mix with other regressive social trends? I do not relish the mass effect of people deferring their decision making to glorified Magic 8 Ball devices, treating them as oracles rather than syntactic lava lamps. Will it lay waste to a generation of education, thinking, science, and policy so that society itself acts out some kind of Therac 25 or Bhopal incident at scale?
- 1718627440 1y ago
  It think it is about killer robots as in someone intentionally deploying Therac 25 Incidents.
filoeleven 1y ago
So a real, actual problem that has been intentionally trivialized for over half a century and is only getting worse only deserves 5% attention, while a paranoid fantasy based on a word salad spinner and billions in marketing hype deserves 95%?