3 ms·

Omnipresent reminder that nobody knows how to do any of this securely and every 3rd-party text you add to an LLM's prompt (email, company slack, calendar, code)

Omnipresent reminder that nobody knows how to do any of this securely and every 3rd-party text you add to an LLM's prompt (email, company slack, calendar, code) has the capability to reprogram your LLM to both execute malicious instructions and to exfiltrate data. And again, no one in the industry has any really demonstrable solutions The next decade is going to be a security disaster, and nobody in the industry knows how to fix it, and the AI-agent proponents are basically sticking their heads in the sand over this because "there might be a fundamental security flaw in LLMs that makes this impossible to do" is just not a "fun" conversation to have. Downstream of the parent comment we immediately have someone saying, "human oversight of this will of course be temporary until the AI gets smart enough." There's such a lack of perspective in the current computing industry of the risks involved here. I'm starting to feel like the only way this narrative is going to change is for a company to actually get bitten by it, badly, and to have a bunch of internal secrets exfiltrated by a malicious email in a very public way. And even then, maybe the hype is too much, I don't know.

411111111111111 3y ago
I think you're overstating the new danger level. The same applies today, you just have to utilize any other internal attack vector, such as code injection in CI, on the deployment server, through a personal dependency etc.
- danShumway 3y ago
  > The same applies today, you just have to utilize any other internal attack vector, such as code injection in CI, on the deployment server, through a personal dependency etc. None of the mitigation techniques for code injection work for LLMs. Very literally, nobody in the industry has come up with a solution for malicious input into LLMs. It's an entirely different class of vulnerability. There isn't any dependency you can import or sanitization you can do or technique for getting rid of this. In fact, I've seen research papers where security researchers are starting to hint that hardening prompts might actually be impossible for the current batch of LLMs.

411111111111111 3y ago

I think you're overstating the new danger level. The same applies today, you just have to utilize any other internal attack vector, such as code injection in CI, on the deployment server, through a personal dependency etc.

danShumway 3y ago
> The same applies today, you just have to utilize any other internal attack vector, such as code injection in CI, on the deployment server, through a personal dependency etc. None of the mitigation techniques for code injection work for LLMs. Very literally, nobody in the industry has come up with a solution for malicious input into LLMs. It's an entirely different class of vulnerability. There isn't any dependency you can import or sanitization you can do or technique for getting rid of this. In fact, I've seen research papers where security researchers are starting to hint that hardening prompts might actually be impossible for the current batch of LLMs.