3 ms·
I mean, I think it's scarier that once these models get smart enough, they will be able to act in the world in the same way that a shell script can execute on y
by NumberWangMan 3y ago
I mean, I think it's scarier that once these models get smart enough, they will be able to act in the world in the same way that a shell script can execute on your machine. A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copies of itself and give them instructions... That's what keeps me up at night. They aren't smart enough to do long-term planning and reasoning yet, but it's a problem we absolutely need to solve before they can.
Imagine what a world-class programmer could accomplish in this world if they thought 100 times faster than a human, and had no fear of going to jail. Our world is an insecure machine, and we're preparing to run untrusted code with root access.
And sure, maybe we can try to use less-intelligent AIs to secure things before then, but the weak point is still humans. Social engineering is typically way easier than straight up hacking. We've seen these lesser AIs threaten people, and while we can keep bonking them on the nose when they do that, we can't prove or tell that they won't ever do it in a different situation, when they judge that it's likely to be the most effective course of action.
I hope every day that this is all just hype and that another AI winter is coming, because we need time (who knows how long) for a way to align these things. But I really fear that it isn't.
- bawolff 3y agoAgency (short or long term) is still in the realm of scifi. The scary part is bad humans manipulating these systems. The AI waking up and becoming skynet is not a realistic threat in the here and now. 4chan screwing with your ai for the lulz, is.
- ChatGTP 3y agoIf we achieve agency, then what ?
- pnt12 3y agoThat's very speculative. We are not that worried about other Sci Fi concepts such as teleportation and aliens. I respect the rationalist argument if there's a 1% chance to achieve AGI, we should spend 1% of our resources to think about it. Makes sense! But having it come up in every discussion with just speculative opinions doesn't really advance the cause, and gets tiring soon (to me, at least).
- aoeusnth1 3y agoI mean, the top repo on GitHub is explicitly trying to create agents out of these LLMs. In 10 years what kind of progress do you expect to see in their reasoning, planning and memory capabilities? How about in 20 years? This is not something that might someday affect our descendants. It’s developing now. People right now are trying to build skynet.
- nowahe 3y agoIf you haven't seen it already, you might enjoy this story from gwern : https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
- dotancohen 3y ago> A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copies of itself and give them instructions... That's what keeps me up at night. You're not far off from one of my stories. Spoiler: it ends with the nations still using analogue pots telephone service becoming the new first world, while former technologically-advanced nations crumble. I now don't know if I'm happy or disappointed that the idea is not so original!