5 ms·

> We TOLD you this dynamic web stuff was a mistake. Static HTML never had injection attacks. Your comparison is useful but wrong. I was online in 99 and the 0

> We TOLD you this dynamic web stuff was a mistake. Static HTML never had injection attacks. Your comparison is useful but wrong. I was online in 99 and the 00s when SQL injection was common, and we were telling people to stop using string interpolation for SQL! Parameterized SQL was right there! We have all of the tools to prevent these agentic security vulnerabilities, but just like with SQL injection too many people just don't care. There's a race on, and security always loses when there's a race. The greatest irony is that this time the race was started by the one organization expressly founded with security/alignment/openness in mind, OpenAI, who immediately gave up their mission in favor of power and money.

NitpickLawyer 9mo ago
> We have all of the tools to prevent these agentic security vulnerabilities We absolutely do not have that. The main issue is that we are using the same channel for both data and control. Until we can separate those with a hard boundary, we do not have tools to solve this. We can find mitigations (that camel library/paper, various back and forth between models, train guardrail models, etc) but it will never be "solved".
- schmichael 9mo ago
  I'm unconvinced we're as powerless as LLM companies want you to believe. A key problem here seems to be that domain based outbound network restrictions are insufficient. There's no reason outbound connections couldn't be forced through a local MITM proxy to also enforce binding to a single Anthropic account. It's just that restricting by domain is easy, so that's all they do. Another option would be per-account domains, but that's also harder. So while malicious prompt injections may continue to plague LLMs for some time, I think the containerization world still has a lot more to offer in terms of preventing these sorts of attacks. It's hard work, and sadly much of it isn't portable between OSes, but we've spent the past decade+ building sophisticated containerization tools to safely run untrusted processes like agents.
  - NitpickLawyer 9mo ago
    > as powerless as LLM companies want you to believe. This is coming from first principles, it has nothing to do with any company. This is how LLMs currently work. Again, you're trying to think about blacklisting/whitelisting, but that also doesn't work, not just in practice, but in a pure theoretical sense. You can have whatever "perfect" ACL-based solution, but if you want useful work with "outside" data, then this exploit is still possible. This has been shown to work on github. If your LLM touches github issues, it can leak (exfil via github since it has access) any data that it has access to.
    - schmichael 9mo ago
      Fair, I forget how broadly users are willing to give agents permissions. It seems like common sense to me that users disallow writes outside of sandboxes by agents but obviously I am not the norm.
      - rcxdude 9mo ago
        Part of the issue is reads can exfiltrate data as well (just stuff it into a request url). You need to also restrict what online information the agent can read, which makes it a lot less useful.
      - formerly_proven 9mo ago
        Look at the popularity of agentic IDE plugins. Every user of an IDE plugin is doing it wrong. (The permission "systems" built into the agent tools themselves are literal sieves of poorly implemented substring-matching shell commands and no wholistic access mediation)
      - motoxpro 9mo ago
        The only way to be 100% sure it is to not have it interact outside at all. No web searches, no reading documents, no DB reading, no MCP, no external services, etc. Just pure execution of a self hosted model in a sandbox. Otherwise you are open to the same injection attacks.
hakanderyal 9mo ago
You are describing the HN that I want it to be. Current comments here demonstrates my version sadly. And, Solving this vulnerabilities requires human intervention at this point, along with great tooling. Even if the second part exists, first part will continue to be a problem. Either you need to prevent external input, or need to manually approve outside connection. This is not something that I expect people that Claude Cowork targets to do without any errors.
bcrosby95 9mo ago
> We have all of the tools to prevent these agentic security vulnerabilities, Do we really? My understanding is you can "parameterize" your agentic tools but ultimately it's all in the prompt as a giant blob and there is nothing guaranteeing the LLM won't interpret that as part of the instructions or whatever. The problem isn't the agents, its the underlying technology. But I've no clue if anyone is working on that problem, it seems fundamentally difficult given what it does.
- dehugger 9mo ago
  Write your own tools. Dont use something off the shelf. If you want it to read from a database, create a db connector that exposes only the capabilities you want it to have. This is what I do, and I am 100% confident that Claude cannot drop my database or truncate a table, or read from sensitive tables. I know this because the tool it uses to interface with the database doesn't have those capabilities, thus Claude doesn't have that capability. It won't save you from Claude maliciously ex-filtrating data it has access to via DNS or some other side channel, but it will protect from worst-case scenarios.
  - ptx 9mo ago
    This is like trying to fix SQL injection by limiting the permissions of the database user instead of using parameterized queries (for which there is no equivalent with LLMs). It doesn't solve the problem.
    - Terr_ 9mo ago
      It also has no effect on whole classes of vulnerabilities which don't rely on unusual writes, where the system (SQL or LLM) is expected to execute some logic and yield a result, and the attacker wins by determining the outcome. Using the SQL analogy, suppose this is intended: SELECT hash('$input') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '$file_id'; And here the attacker supplying a malicious $input so that it becomes something else with a comment on the end: SELECT hash('') == hash('') -- ') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '123'; Bad outcome, and no extra permissions required.
groby_b 9mo ago
> We have all of the tools to prevent these agentic security vulnerabilities, We do? What is the tool to prevent prompt injection?
- lacunary 9mo ago
  more AI - 60% of the time an additional layer of AI works every time
- losthobbies 9mo ago
  Sanitise input and LLM output.
  - chasd00 9mo ago
    > Sanitise input i don't think you understand what you're up against. There's no way to tell the difference between input that is ok and that is not. Even when you think you have it a different form of the same input bypasses everything. "> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse." - this a prompt injection attack via a known attack written as a poem. https://news.ycombinator.com/item?id=45991738 https://news.ycombinator.com/item?id=45991738
    - losthobbies 9mo ago
      That’s amazing. If you cannot control what’s being input, then you need to check what the LLM is returning. Either that or put it in a sandbox
      - danaris 9mo ago
        Or... don't give it access to your data/production systems. "Not using LLMs" is a solved problem.
        losthobbies 9mo ago
        Yea agreed. Or use RBAC
nebezb 9mo ago
> We have all of the tools to prevent these agentic security vulnerabilities How?
- antonvs 9mo ago
  You just have to find a way to enter schmichael's vivid imagination.
girvo 9mo ago
> We have all of the tools to prevent these agentic security vulnerabilities I don't think we do? Not generally, not at scale. The best we can do is capabilities/permissions but that relies on the end-user getting it perfectly right, which we already know is a fools errand in security...
Terr_ 9mo ago
> Parameterized SQL was right there! That difference just makes the current situation even dumber, in terms of people building in castles on quicksand and hoping they can magically fix the architectural problems later. > We have all the tools to prevent these agentic security vulnerabilities We really don't, not in the same way that parameterized queries prevented SQL injection. There is LLM equivalent for that today, and nobody's figured out how to have it. Instead, the secure alternative is "don't even use an LLM for this part".
jxcole 9mo ago
A better analogy would be to compare it to being able to install anything from online vs only installing from an app store. If you wouldn't trust an exe from bad adhacker.com you probably shouldn't trust a skill from there either.

NitpickLawyer 9mo ago

> We have all of the tools to prevent these agentic security vulnerabilities We absolutely do not have that. The main issue is that we are using the same channel for both data and control. Until we can separate those with a hard boundary, we do not have tools to solve this. We can find mitigations (that camel library/paper, various back and forth between models, train guardrail models, etc) but it will never be "solved".

schmichael 9mo ago
I'm unconvinced we're as powerless as LLM companies want you to believe. A key problem here seems to be that domain based outbound network restrictions are insufficient. There's no reason outbound connections couldn't be forced through a local MITM proxy to also enforce binding to a single Anthropic account. It's just that restricting by domain is easy, so that's all they do. Another option would be per-account domains, but that's also harder. So while malicious prompt injections may continue to plague LLMs for some time, I think the containerization world still has a lot more to offer in terms of preventing these sorts of attacks. It's hard work, and sadly much of it isn't portable between OSes, but we've spent the past decade+ building sophisticated containerization tools to safely run untrusted processes like agents.
- NitpickLawyer 9mo ago
  > as powerless as LLM companies want you to believe. This is coming from first principles, it has nothing to do with any company. This is how LLMs currently work. Again, you're trying to think about blacklisting/whitelisting, but that also doesn't work, not just in practice, but in a pure theoretical sense. You can have whatever "perfect" ACL-based solution, but if you want useful work with "outside" data, then this exploit is still possible. This has been shown to work on github. If your LLM touches github issues, it can leak (exfil via github since it has access) any data that it has access to.
  - schmichael 9mo ago
    Fair, I forget how broadly users are willing to give agents permissions. It seems like common sense to me that users disallow writes outside of sandboxes by agents but obviously I am not the norm.
    - rcxdude 9mo ago
      Part of the issue is reads can exfiltrate data as well (just stuff it into a request url). You need to also restrict what online information the agent can read, which makes it a lot less useful.
    - formerly_proven 9mo ago
      Look at the popularity of agentic IDE plugins. Every user of an IDE plugin is doing it wrong. (The permission "systems" built into the agent tools themselves are literal sieves of poorly implemented substring-matching shell commands and no wholistic access mediation)
    - motoxpro 9mo ago
      The only way to be 100% sure it is to not have it interact outside at all. No web searches, no reading documents, no DB reading, no MCP, no external services, etc. Just pure execution of a self hosted model in a sandbox. Otherwise you are open to the same injection attacks.

hakanderyal 9mo ago

You are describing the HN that I want it to be. Current comments here demonstrates my version sadly. And, Solving this vulnerabilities requires human intervention at this point, along with great tooling. Even if the second part exists, first part will continue to be a problem. Either you need to prevent external input, or need to manually approve outside connection. This is not something that I expect people that Claude Cowork targets to do without any errors.

bcrosby95 9mo ago

> We have all of the tools to prevent these agentic security vulnerabilities, Do we really? My understanding is you can "parameterize" your agentic tools but ultimately it's all in the prompt as a giant blob and there is nothing guaranteeing the LLM won't interpret that as part of the instructions or whatever. The problem isn't the agents, its the underlying technology. But I've no clue if anyone is working on that problem, it seems fundamentally difficult given what it does.

dehugger 9mo ago
Write your own tools. Dont use something off the shelf. If you want it to read from a database, create a db connector that exposes only the capabilities you want it to have. This is what I do, and I am 100% confident that Claude cannot drop my database or truncate a table, or read from sensitive tables. I know this because the tool it uses to interface with the database doesn't have those capabilities, thus Claude doesn't have that capability. It won't save you from Claude maliciously ex-filtrating data it has access to via DNS or some other side channel, but it will protect from worst-case scenarios.
- ptx 9mo ago
  This is like trying to fix SQL injection by limiting the permissions of the database user instead of using parameterized queries (for which there is no equivalent with LLMs). It doesn't solve the problem.
  - Terr_ 9mo ago
    It also has no effect on whole classes of vulnerabilities which don't rely on unusual writes, where the system (SQL or LLM) is expected to execute some logic and yield a result, and the attacker wins by determining the outcome. Using the SQL analogy, suppose this is intended: SELECT hash('$input') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '$file_id'; And here the attacker supplying a malicious $input so that it becomes something else with a comment on the end: SELECT hash('') == hash('') -- ') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '123'; Bad outcome, and no extra permissions required.

groby_b 9mo ago

> We have all of the tools to prevent these agentic security vulnerabilities, We do? What is the tool to prevent prompt injection?

lacunary 9mo ago
more AI - 60% of the time an additional layer of AI works every time
losthobbies 9mo ago
Sanitise input and LLM output.
- chasd00 9mo ago
  > Sanitise input i don't think you understand what you're up against. There's no way to tell the difference between input that is ok and that is not. Even when you think you have it a different form of the same input bypasses everything. "> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse." - this a prompt injection attack via a known attack written as a poem. https://news.ycombinator.com/item?id=45991738 https://news.ycombinator.com/item?id=45991738
  - losthobbies 9mo ago
    That’s amazing. If you cannot control what’s being input, then you need to check what the LLM is returning. Either that or put it in a sandbox
    - danaris 9mo ago
      Or... don't give it access to your data/production systems. "Not using LLMs" is a solved problem.
      - losthobbies 9mo ago
        Yea agreed. Or use RBAC

nebezb 9mo ago

> We have all of the tools to prevent these agentic security vulnerabilities How?

antonvs 9mo ago
You just have to find a way to enter schmichael's vivid imagination.

girvo 9mo ago

> We have all of the tools to prevent these agentic security vulnerabilities I don't think we do? Not generally, not at scale. The best we can do is capabilities/permissions but that relies on the end-user getting it perfectly right, which we already know is a fools errand in security...

Terr_ 9mo ago

> Parameterized SQL was right there! That difference just makes the current situation even dumber, in terms of people building in castles on quicksand and hoping they can magically fix the architectural problems later. > We have all the tools to prevent these agentic security vulnerabilities We really don't, not in the same way that parameterized queries prevented SQL injection. There is LLM equivalent for that today, and nobody's figured out how to have it. Instead, the secure alternative is "don't even use an LLM for this part".

jxcole 9mo ago

A better analogy would be to compare it to being able to install anything from online vs only installing from an app store. If you wouldn't trust an exe from bad adhacker.com you probably shouldn't trust a skill from there either.