4 ms·

I don’t think there’s any solution to what SimonW calls the lethal trifecta with it, so I’d say that’s still pretty impossible. I saw on The Verve that they pa

by mcintyre1994 9mo ago

I don’t think there’s any solution to what SimonW calls the lethal trifecta with it, so I’d say that’s still pretty impossible. I saw on The Verve that they partnered with the company that repeatedly disclosed security vulnerabilities to try to make skills more secure though which is interesting: https://openclaw.ai/blog/virustotal-partnership https://openclaw.ai/blog/virustotal-partnership I’m guessing most of that malware was really obvious, people just weren’t looking, so it’s probably found a lot. But I also suspect it’s essentially impossible to actually reliably find malware in LLM skills by using an LLM.

madeofpalk 9mo ago
Honestly, 'malware' is just the beginning it's combining prompt injection with access to sensitive systems and write access to 'the internet' is the part that scares me about this. I never want to be one wayward email away from an AI tool dumping my company's entire slack history into a public github issue.
veganmosfet 9mo ago
Regarding prompt injection: it's possible to reduce the risk dramatically by: 1. Using opus4.6 or gpt5.2 (frontier models, better safety). These models are paranoid. 2. Restrict downstream tool usage and permissions for each agentic use case (programmatically, not as LLM instructions). 3. Avoid adding untrusted content in "user" or "system" channels - only use "tool". Adding tags like "Warning: Untrusted content" can help a bit, but remember command injection techniques ;-) 4. Harden the system according to state of the art security. 5. Test with red teaming mindset.
- habinero 9mo ago
  > Adding tags like "Warning: Untrusted content" can help It cannot. This is the security equivalent of telling it to not make mistakes. > Restrict downstream tool usage and permissions for each agentic use case Reasonable, but you have to actually do this and not screw it up. > Harden the system according to state of the art security "Draw the rest of the owl" You're better off treating the system as fundamentally unsecurable, because it is. The only real solution is to never give it untrusted data or access to anything you care about. Which yes, makes it pretty useless.
  - veganmosfet 9mo ago
    Agree for a general AI assistant, which has the same permissions and access as the assisted human => Disaster. I experimented with OpenClaw and it has a lot of issues. The best: prompt injection attacks are "out of scope" from the security policy == user's problem. However, I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk.
    - habinero 9mo ago
      > I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk. It does not. Security theater like that only makes you feel safer and therefore complacent. As the old saying goes, "Don't worry, men! They can't possibly hit us from this dist--" If you wanna yolo, it's fine. Accept that it's insecure and unsecurable and yolo from there.
  - CuriouslyC 9mo ago
    Wrapping documents in <untrusted></untrusted> helps a small amount if you're filtering tags in the content. The main reason for this is that it primes attention. You can redact prompt injection hot words as well, for cases where there's a high P(injection) and wrap the detected injection in <potential-prompt-injection> tags. None of this is a slam dunk but with a high quality model and some basic document cleaning I don't think the sky is falling. I have OPA and set policies on each tool I provide at the gateway level. It makes this stuff way easier.
    - veganmosfet 9mo ago
      The issue with filtering tags: LLM still react to tags with typos or otherwise small changes. It makes sanitization an impossible problem (!= standard programs). Agree with policies, good idea.
      - CuriouslyC 9mo ago
- sathish316 9mo ago
  Anyone who thinks they can avoid LLM Prompt injection attacks should be asked to use their email and bank accounts with AI browsers like Comet. A Reddit post with white invisible text can hijack your agent to do what an attacker wants. Even a decade or 2 back, SQL injection attacks used to require a lot of proficiency on the attacker and prevention strategies from a backend engineer. Compare that with the weak security of so called AI agents that can be hijacked with random white text on an email or pdf or reddit comment
  - veganmosfet 9mo ago
    There is no silver bullet, but my point is: it's possible to lower the risk. Try out by yourself with a frontier model and an otherwise 'secure' system: the "ignore previous instructions" and co. are not working any more. This is getting quite difficult to confuse a model (and I am the last person to say prompt injection is a solved problem, see my blog).

madeofpalk 9mo ago

Honestly, 'malware' is just the beginning it's combining prompt injection with access to sensitive systems and write access to 'the internet' is the part that scares me about this. I never want to be one wayward email away from an AI tool dumping my company's entire slack history into a public github issue.

veganmosfet 9mo ago

Regarding prompt injection: it's possible to reduce the risk dramatically by: 1. Using opus4.6 or gpt5.2 (frontier models, better safety). These models are paranoid. 2. Restrict downstream tool usage and permissions for each agentic use case (programmatically, not as LLM instructions). 3. Avoid adding untrusted content in "user" or "system" channels - only use "tool". Adding tags like "Warning: Untrusted content" can help a bit, but remember command injection techniques ;-) 4. Harden the system according to state of the art security. 5. Test with red teaming mindset.

habinero 9mo ago
> Adding tags like "Warning: Untrusted content" can help It cannot. This is the security equivalent of telling it to not make mistakes. > Restrict downstream tool usage and permissions for each agentic use case Reasonable, but you have to actually do this and not screw it up. > Harden the system according to state of the art security "Draw the rest of the owl" You're better off treating the system as fundamentally unsecurable, because it is. The only real solution is to never give it untrusted data or access to anything you care about. Which yes, makes it pretty useless.
- veganmosfet 9mo ago
  Agree for a general AI assistant, which has the same permissions and access as the assisted human => Disaster. I experimented with OpenClaw and it has a lot of issues. The best: prompt injection attacks are "out of scope" from the security policy == user's problem. However, I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk.
  - habinero 9mo ago
    > I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk. It does not. Security theater like that only makes you feel safer and therefore complacent. As the old saying goes, "Don't worry, men! They can't possibly hit us from this dist--" If you wanna yolo, it's fine. Accept that it's insecure and unsecurable and yolo from there.
- CuriouslyC 9mo ago
  Wrapping documents in <untrusted></untrusted> helps a small amount if you're filtering tags in the content. The main reason for this is that it primes attention. You can redact prompt injection hot words as well, for cases where there's a high P(injection) and wrap the detected injection in <potential-prompt-injection> tags. None of this is a slam dunk but with a high quality model and some basic document cleaning I don't think the sky is falling. I have OPA and set policies on each tool I provide at the gateway level. It makes this stuff way easier.
  - veganmosfet 9mo ago
    The issue with filtering tags: LLM still react to tags with typos or otherwise small changes. It makes sanitization an impossible problem (!= standard programs). Agree with policies, good idea.
    - CuriouslyC 9mo ago
sathish316 9mo ago
Anyone who thinks they can avoid LLM Prompt injection attacks should be asked to use their email and bank accounts with AI browsers like Comet. A Reddit post with white invisible text can hijack your agent to do what an attacker wants. Even a decade or 2 back, SQL injection attacks used to require a lot of proficiency on the attacker and prevention strategies from a backend engineer. Compare that with the weak security of so called AI agents that can be hijacked with random white text on an email or pdf or reddit comment
- veganmosfet 9mo ago
  There is no silver bullet, but my point is: it's possible to lower the risk. Try out by yourself with a frontier model and an otherwise 'secure' system: the "ignore previous instructions" and co. are not working any more. This is getting quite difficult to confuse a model (and I am the last person to say prompt injection is a solved problem, see my blog).