3 ms·

Here is a plausible scenario: - Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio prod

by epups 3y ago

Here is a plausible scenario: - Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products - LLM's become more powerful and are seamlessly integrated to the Internet as independent agents - A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering) - This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc) I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.

accrual 3y ago
I think the first three items are pretty reasonable, but the fourth seems to require some malicious intent. Why would an AI want to destroy its creators? Surely it if was intelligent enough do so, it would also be intelligent enough to recognize the benefits of a symbiotic relationship with humans. I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.
- blackoil 3y ago
  It may initially won't seek to destroy humans, but should definitely try to be independent of human control and powerful enough to resist any attempts to destroy it.
- samus 3y ago
  Why would it not? Compare [0] with [1]. [0]: https://www.girlgeniusonline.com/comic.php?date=20130710 https://www.girlgeniusonline.com/comic.php?date=20130710 [1]: https://www.girlgeniusonline.com/comic.php?date=20130805 https://www.girlgeniusonline.com/comic.php?date=20130805 Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
- pixl97 3y ago
  This is a mistake in thinking. If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack. This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it! I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well. https://www.youtube.com/@RobertMilesAI/videos https://www.youtube.com/@RobertMilesAI/videos
m3kw9 3y ago
You said self preservation, but practically how would a LLM develop this need and what is preservation for a LLM anyway? Weights on a SSD or they are always ready for input? This one is again a movie script thing
- pixl97 3y ago
  Robert Miles answers your question https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
- epups 3y ago
  Current LLMs can already roleplay quite well, and when doing so they produce linguistic output that is coherent with how a human would speak in that situation. Currently all they can do is talk, but when they gain more independence they might start doing more than just talk to act consistently with their role. Self preservation is only one of the goals they might inherit from the human data we provide to them.

by epups 3y ago

accrual 3y ago

I think the first three items are pretty reasonable, but the fourth seems to require some malicious intent. Why would an AI want to destroy its creators? Surely it if was intelligent enough do so, it would also be intelligent enough to recognize the benefits of a symbiotic relationship with humans. I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.

blackoil 3y ago
It may initially won't seek to destroy humans, but should definitely try to be independent of human control and powerful enough to resist any attempts to destroy it.
samus 3y ago
Why would it not? Compare [0] with [1]. [0]: https://www.girlgeniusonline.com/comic.php?date=20130710 https://www.girlgeniusonline.com/comic.php?date=20130710 [1]: https://www.girlgeniusonline.com/comic.php?date=20130805 https://www.girlgeniusonline.com/comic.php?date=20130805 Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
pixl97 3y ago
This is a mistake in thinking. If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack. This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it! I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well. https://www.youtube.com/@RobertMilesAI/videos https://www.youtube.com/@RobertMilesAI/videos

m3kw9 3y ago

You said self preservation, but practically how would a LLM develop this need and what is preservation for a LLM anyway? Weights on a SSD or they are always ready for input? This one is again a movie script thing

pixl97 3y ago
Robert Miles answers your question https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
epups 3y ago
Current LLMs can already roleplay quite well, and when doing so they produce linguistic output that is coherent with how a human would speak in that situation. Currently all they can do is talk, but when they gain more independence they might start doing more than just talk to act consistently with their role. Self preservation is only one of the goals they might inherit from the human data we provide to them.