3 ms·

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a f

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

appplication 27d ago
I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
- hn_throwaway_99 27d ago
  > Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
  - deleted 27d ago
    [deleted]
  - wseqyrku 27d ago
    > Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.
- ben_w 27d ago
  > Not hypothetical malicious AI interpretation of benign intent. (Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else). This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal? Because this sounds nothing like the world I have lived in for my whole life. > The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions. Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".
hn_throwaway_99 27d ago
Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now. I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.
- ben_w 27d ago
  > I think hardly anyone disagrees that superintelligent AI would be an existential risk. Oh, there's a lot who disagree. I don't really think I understand their worldviews well enough for my attempts to convince them to connect with anything, but I've encountered them even on this site.
  - hn_throwaway_99 27d ago
    Well, I'll tweak my comment a bit then. On one hand you have folks like Ilya Sutskever, who named his new company "Safe Superintelligence", and on the other hand you have folks like Eliezer Yudkowsky, who titled his book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All". One side believes it's possible to build safe super intelligence, the other side does not. But even the "safe superintelligence" people think there are substantial risks (kind of why they feel the need to add "safe" in the first place, and why they put so much effort into what they call safety and alignment), they just think they can mitigate those risks. Sure, there may be a few yahoos who think there aren't existential risks, but I think it's easy to discount them. But that's not the same as people who think there are risks but that those risks can be mitigated.

appplication 27d ago

I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.

hn_throwaway_99 27d ago
> Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
- deleted 27d ago
  [deleted]
- wseqyrku 27d ago
  > Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.
ben_w 27d ago
> Not hypothetical malicious AI interpretation of benign intent. (Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else). This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal? Because this sounds nothing like the world I have lived in for my whole life. > The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions. Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".

hn_throwaway_99 27d ago

Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now. I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.

ben_w 27d ago
> I think hardly anyone disagrees that superintelligent AI would be an existential risk. Oh, there's a lot who disagree. I don't really think I understand their worldviews well enough for my attempts to convince them to connect with anything, but I've encountered them even on this site.
- hn_throwaway_99 27d ago
  Well, I'll tweak my comment a bit then. On one hand you have folks like Ilya Sutskever, who named his new company "Safe Superintelligence", and on the other hand you have folks like Eliezer Yudkowsky, who titled his book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All". One side believes it's possible to build safe super intelligence, the other side does not. But even the "safe superintelligence" people think there are substantial risks (kind of why they feel the need to add "safe" in the first place, and why they put so much effort into what they call safety and alignment), they just think they can mitigate those risks. Sure, there may be a few yahoos who think there aren't existential risks, but I think it's easy to discount them. But that's not the same as people who think there are risks but that those risks can be mitigated.