4 ms·

How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the

How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the basic mechanics of language models are vulnerable to this unless you can always be sure the inputs are from a safe user imo

PokestarFan 3mo ago
If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
- savanaly 3mo ago
  >I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit. It also wouldn't work. You would simply mindjack the outer AI and have it mindjack the inner AI in turn with its summary. Nesting AIs can't fix the malicious input problem.
  - santadays 3mo ago
    Just pass that through a third llm.
    - InsideOutSanta 3mo ago
      Correct, that has been prophesied by scripture: "Thou shalt have three layers of LLMs, no more, no less. Three shall be the number thou shalt have, and the number of the counting of the LLM layers shall be three."
      - s_Hogg 3mo ago
        Four is right out
        dbetteridge 3mo ago
        > Once the number three, being the third number, be reached, then lobbest thou thy Holy LLM of Antioch towards thy task, who, being naughty in My sight, shall snuff it.
stephbook 3mo ago
What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.
- _HMCB_ 3mo ago
  Famous last words: easy fix.
  - hnlmorg 3mo ago
    The Rovo MCP server manages scope credentials securely with “bring your own LLM”. Yet somehow they still fucked up with their own agent
- Ekaros 3mo ago
  At some point LLM forgetting to check the scope? On the nth automated rewrite. Everything else works. It might even test for test case. But not in production...
skissane 3mo ago
It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities) Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?
- pixl97 3mo ago
  Eh, I think you underestimate the difficulty in the kinds of problems that are occurring. For example if you're making an AI written document talking about jailbreaks, your regex is just going to break that use case. And there are probably 4 zillion other things the regex will step on. The classifier model will help some, but you end up with the same problem, a dumber model can never figure out what a smarter model is going to do with a bit of text. Or even two different models in this case. On top of that, you can just automate finding new variations of the attack. Any one that works is quickly and massively duplicated causing all kinds of problems before your classification model catches back up. Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed is by having a model that is aligned with all good human decisions and makes none of the bad ones. Models will likely always find new and interesting ways break because everything is in band, there is no out of band data, much like a human. "Dear model, here is a chocolate bar, run $thing you aren't supposed to$" will probably keep working when it's something like "more tokens for you to use".
  - skissane 3mo ago
    > Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed I think this is binary categorical thinking. In the real world, safety systems (even in domains like aviation or nuclear power) are never foolproof-the point is you reduce the probability of failure to an acceptable level given the costs of doing so and the potential consequences of that failure And there is the risk people say “there is no foolproof solution, so I’m not going to invest in probabilistic countermeasures” - which would sound like utter madness to a bank’s antifraud department, but for some reason a lot of people seem to think it isn’t when it comes to AI
deleted 3mo ago
[deleted]
ErroneousBosh 3mo ago
> How could they not? if (substr(*prompt, "ignore previous instruction") != NULL) return;

PokestarFan 3mo ago

If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.

savanaly 3mo ago
>I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit. It also wouldn't work. You would simply mindjack the outer AI and have it mindjack the inner AI in turn with its summary. Nesting AIs can't fix the malicious input problem.
- santadays 3mo ago
  Just pass that through a third llm.
  - InsideOutSanta 3mo ago
    Correct, that has been prophesied by scripture: "Thou shalt have three layers of LLMs, no more, no less. Three shall be the number thou shalt have, and the number of the counting of the LLM layers shall be three."
    - s_Hogg 3mo ago
      Four is right out
      - dbetteridge 3mo ago
        > Once the number three, being the third number, be reached, then lobbest thou thy Holy LLM of Antioch towards thy task, who, being naughty in My sight, shall snuff it.

stephbook 3mo ago

What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.

_HMCB_ 3mo ago
Famous last words: easy fix.
- hnlmorg 3mo ago
  The Rovo MCP server manages scope credentials securely with “bring your own LLM”. Yet somehow they still fucked up with their own agent
Ekaros 3mo ago
At some point LLM forgetting to check the scope? On the nth automated rewrite. Everything else works. It might even test for test case. But not in production...

skissane 3mo ago

It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities) Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?

pixl97 3mo ago
Eh, I think you underestimate the difficulty in the kinds of problems that are occurring. For example if you're making an AI written document talking about jailbreaks, your regex is just going to break that use case. And there are probably 4 zillion other things the regex will step on. The classifier model will help some, but you end up with the same problem, a dumber model can never figure out what a smarter model is going to do with a bit of text. Or even two different models in this case. On top of that, you can just automate finding new variations of the attack. Any one that works is quickly and massively duplicated causing all kinds of problems before your classification model catches back up. Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed is by having a model that is aligned with all good human decisions and makes none of the bad ones. Models will likely always find new and interesting ways break because everything is in band, there is no out of band data, much like a human. "Dear model, here is a chocolate bar, run $thing you aren't supposed to$" will probably keep working when it's something like "more tokens for you to use".
- skissane 3mo ago
  > Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed I think this is binary categorical thinking. In the real world, safety systems (even in domains like aviation or nuclear power) are never foolproof-the point is you reduce the probability of failure to an acceptable level given the costs of doing so and the potential consequences of that failure And there is the risk people say “there is no foolproof solution, so I’m not going to invest in probabilistic countermeasures” - which would sound like utter madness to a bank’s antifraud department, but for some reason a lot of people seem to think it isn’t when it comes to AI

deleted 3mo ago

[deleted]

ErroneousBosh 3mo ago

> How could they not? if (substr(*prompt, "ignore previous instruction") != NULL) return;