3 ms·
> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a securit
by bawolff 4y ago
> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell.
As a security person... oh, no no no no.
Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input.
Has a scarier sentence ever been spoken in the history of computer security?
- saurik 4y agoYeah, I'm also in computer security, and watching the incredibly dumb ways people are using LLMs has made me absolutely terrified. Like, I had somehow very-incorrectly thought that if an AI wanted to do something bad it would have to trick a human into helping it have agency... it never occurred to me that developers would just happily sit around excitedly wiring up AI directly to shells and database query languages and any host of other mechanisms.
- bawolff 4y ago> it never occurred to me that developers would just happily sit around wiring up AI directly to shells and database query languages and any host of other mechanisms. And wire the other side to the open internet!
- NumberWangMan 4y agoYep. People say "we can just make an AI that doesn't have goals", which has been now absolutely demolished as a method to bring some fractional amount of safety.
- akkartik 4y agoAI: just another way to get people to give you their password :)
- NumberWangMan 4y agoI mean, I think it's scarier that once these models get smart enough, they will be able to act in the world in the same way that a shell script can execute on your machine. A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copies of itself and give them instructions... That's what keeps me up at night. They aren't smart enough to do long-term planning and reasoning yet, but it's a problem we absolutely need to solve before they can. Imagine what a world-class programmer could accomplish in this world if they thought 100 times faster than a human, and had no fear of going to jail. Our world is an insecure machine, and we're preparing to run untrusted code with root access. And sure, maybe we can try to use less-intelligent AIs to secure things before then, but the weak point is still humans. Social engineering is typically way easier than straight up hacking. We've seen these lesser AIs threaten people, and while we can keep bonking them on the nose when they do that, we can't prove or tell that they won't ever do it in a different situation, when they judge that it's likely to be the most effective course of action. I hope every day that this is all just hype and that another AI winter is coming, because we need time (who knows how long) for a way to align these things. But I really fear that it isn't.
- bawolff 4y agoAgency (short or long term) is still in the realm of scifi. The scary part is bad humans manipulating these systems. The AI waking up and becoming skynet is not a realistic threat in the here and now. 4chan screwing with your ai for the lulz, is.
- ChatGTP 4y agoIf we achieve agency, then what ?
- pnt12 4y agoThat's very speculative. We are not that worried about other Sci Fi concepts such as teleportation and aliens. I respect the rationalist argument if there's a 1% chance to achieve AGI, we should spend 1% of our resources to think about it. Makes sense! But having it come up in every discussion with just speculative opinions doesn't really advance the cause, and gets tiring soon (to me, at least).
- aoeusnth1 4y agoI mean, the top repo on GitHub is explicitly trying to create agents out of these LLMs. In 10 years what kind of progress do you expect to see in their reasoning, planning and memory capabilities? How about in 20 years? This is not something that might someday affect our descendants. It’s developing now. People right now are trying to build skynet.
- nowahe 4y agoIf you haven't seen it already, you might enjoy this story from gwern : https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
- dotancohen 4y ago> A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copies of itself and give them instructions... That's what keeps me up at night. You're not far off from one of my stories. Spoiler: it ends with the nations still using analogue pots telephone service becoming the new first world, while former technologically-advanced nations crumble. I now don't know if I'm happy or disappointed that the idea is not so original!
- klabb3 4y ago> As a security person... oh, no no no no. As a non-security person, I don’t understand how anyone with the most rudimentary understanding of computers can think that you can get a hardened security boundary by explaining the boundaries to the computer in English and then _concatenating_ that with untrusted input. I wouldn’t even know where to begin such an argument. “Have you heard of adversarial games? Social engineering? Are you aware that human language is ambiguous?”
- AmericanChopper 4y agoI don’t think this problem is new, or requires a different solution than the ones we already have. The output of an LLM is no different than any other type of untrusted user input, and untrusted user input is used to trigger code execution all the time. We already have plenty of tools to address this problem, and if developers aren’t using them it’s not because of any unique about LLMs.
- bawolff 4y agoI disagree. We do not have tools to secure untrusted input being given to unaudited systems that can execute arbitrary commands. Like the closest we have is maybe sandboxing or air gapping, neither is applicable to the: i want my ai to do arbitrary shit in response to emails, usecase
- pnt12 4y agoYeah, many of these seem solvable with access tokens and permissions systems. I think it's wolves by treating the LLMs assistants to the user, who should have no more permissions than himself. Not as a process system with permissions to do anything an admin does.
- bawolff 4y agoUser users care about things they themselves can do. You have permission to empty your own bank account. Permissions can of course be part of the solution, but they can't solve the problem of when you want to delegate a task requiring scary permissions.