4 ms·

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given L

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.

Angostura 4mo ago
Jokes on them. My bank will just truncate it to 10 characters.
- TacticalCoder 4mo ago
  > Jokes on them. My bank will just truncate it to 10 characters. You do understand that this is just an example out of a bazillion and that planning to solve every place where data is fed to LLMs at 10 characters so that it's not mistaken for instructions ain't a viable solution?
  - Angostura 4mo ago
    Yes. I was being humorous. Apologies
jmount 4mo ago
I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.
- crooked-v 4mo ago
  I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the decision-making process is reading everything. The only "fix" is to make an AI smart enough that it can understand context for each item, which is a tall order.
  - wat10000 4mo ago
    Humans are vulnerable to prompt injection as well. We usually call it something like "social engineering."
    - danlitt 4mo ago
      Yes, it's a serious problem. It's why we remove humans from these systems whenever possible!
      - wat10000 4mo ago
        Right, and add controls to limit the damage they can do where possible. Avoiding prompt injection looks to require superhuman intelligence.
  - acdha 4mo ago
    This is especially true because so much of that data comes from outside of your organization. I receive Google Calendar invites from scammers a couple of times a week and those show up in my invitation list just like anything else. If LLMs start screening things, that kind of thing will become even more popular but most of us can’t just ignore everyone outside of our employer’s directory.
    - brokenmachine 4mo ago
      Interestingly, if you look at the posted link, in the top-right there's a "talk to Blue41" link that allows you to do exactly that. I wonder if they have a "risk control platform" for their calendar? It's LLMs all the way down!!
  - gopher_space 4mo ago
    > The only "fix" is to make an AI smart enough that it can understand context for each item, which is a tall order. Impossible as you said. Context isn’t static, it’s continuous, analog, and a conglomeration of viewpoints. AI cannot create useful context for itself because it is a machine with no desires. It doesn’t have a point of view, it has historical records. It moves forward in time by walking backwards (if that makes sense?)
- dejj 4mo ago
  RFC 3514 “evil bit” header flag to the rescue: https://www.rfc-editor.org/info/rfc3514/ https://www.rfc-editor.org/info/rfc3514/
  - BobbyTables2 4mo ago
    At least the evil bit was a dedicated field at a known place… AI guardrails can’t even dream of that! Imagine if it was just the absence of “I’m evil” in the payload.
- airstrike 4mo ago
  It's not possible with today's LLM models, but we are not wedded to the current architecture.
  - SlinkyOnStairs 4mo ago
    Realistically, we are. This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all.
    - airstrike 4mo ago
      Define "realistically". You're basically saying attention is all we need indefinitely into the future and all other gains come from more compute or scaffolding around current architectures. Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently. Who's to say we won't find a way to encode provenance or privilege natively into models such that the tradeoff changes? It's hard to say what the solution will be. If I knew it, I'd build it. But it's even harder to sustain that the current architecture is a crystalized global optimum.
      - gbalduzzi 4mo ago
        Aside from LLM architecture, that already is a complex issue, an issue is that training data is unstructured text. An LLM able to structurally separate context and instructions, should logically need separated data to train, and we don't have it. Moreover, while an equally powerful LLM architecture solving this may exists, there are no guarantees at all that we are able to come up with it in a reasonable timeframe. Without some signals moving in that direction, the most pragmatic and realistic way of looking at the problem is that it will not be solved in the near future
        airstrike 4mo ago
        Thanks, I appreciate the thoughtful reply. I agree this doesn't mean we shouldn't try to address limitations with the current architecture. I just mean that I expect the root cause to be solved eventually if we ever really want to take steps towards AGI. Regarding signals moving in that direction, here's a paper you might enjoy https://arxiv.org/abs/2503.21937 https://arxiv.org/abs/2503.21937
embedding-shape 4mo ago
> It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide. Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones. The difficulty comes when you accept completely unreviewed/unchanged user-input as user messages, as your system/developer prompts needs to take this into account. You're better off to kind of whitelist what's possible rather than trying to prevent specific things, but seems that hasn't fully caught on yet. It feels like people and organizations are still trying to discover what works or not, and there are huge gaps being being left open because there simply isn't enough understanding of the limitations and impact of what they make available to users. We're already seeing it in lots of places, feels like it won't get better before it gets worse.
- sillysaurusx 4mo ago
  > Today's SOTA LLMs have pretty excellent following of these divisions Unfortunately "pretty excellent" is different from "perfect." I haven't kept track, but are you certain that given all possible inputs, the user prompt will never override the system prompt? Those are strong claims, and unless there's been an advancement in the tech, it doesn't seem possible. Reinforcement learning might make it much less likely, but that's different from impossible.
- Muromec 4mo ago
  If it was solved, the bug like this would not happen. It is also not always clear who is the user and how much they should be obeyed
  - embedding-shape 4mo ago
    > If it was solved, the bug like this would not happen. Only if you only read the first line in my comment, there is more under that one too. It is clear, if you make it clear. These bugs happen because they don't clearly understand what should go where.
- 4mo ago
madamelic 4mo ago
> Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). I have been working on something like that: https://clawband.io https://clawband.io It's not quite ready for 'showtime' but feel free to take a look and give your impressions if you'd like. I feel the exact same way: I want to allow my agent to perform actions on all services but also limit what they can do. Basically my idea is wrapping individual service's APIs and then the middleware (Clawband in this case) enforces granular permissioning such as "can make credit cards but only up to $50" or "can send emails but only to specific domains". The agent never gets a raw API key to a service, it uses an intermediate API key that gets exchanged in the backend for calling the service after permissioning has been enforced.
KaiShips 4mo ago
[flagged]
mike_hock 4mo ago
I can't believe that fucking Terminator was prophetic.
hannahnine 4mo ago
[flagged]

Angostura 4mo ago

Jokes on them. My bank will just truncate it to 10 characters.

TacticalCoder 4mo ago
> Jokes on them. My bank will just truncate it to 10 characters. You do understand that this is just an example out of a bazillion and that planning to solve every place where data is fed to LLMs at 10 characters so that it's not mistaken for instructions ain't a viable solution?
- Angostura 4mo ago
  Yes. I was being humorous. Apologies

jmount 4mo ago

I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.

crooked-v 4mo ago
I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the decision-making process is reading everything. The only "fix" is to make an AI smart enough that it can understand context for each item, which is a tall order.
- wat10000 4mo ago
  Humans are vulnerable to prompt injection as well. We usually call it something like "social engineering."
  - danlitt 4mo ago
    Yes, it's a serious problem. It's why we remove humans from these systems whenever possible!
    - wat10000 4mo ago
      Right, and add controls to limit the damage they can do where possible. Avoiding prompt injection looks to require superhuman intelligence.
- acdha 4mo ago
  This is especially true because so much of that data comes from outside of your organization. I receive Google Calendar invites from scammers a couple of times a week and those show up in my invitation list just like anything else. If LLMs start screening things, that kind of thing will become even more popular but most of us can’t just ignore everyone outside of our employer’s directory.
  - brokenmachine 4mo ago
    Interestingly, if you look at the posted link, in the top-right there's a "talk to Blue41" link that allows you to do exactly that. I wonder if they have a "risk control platform" for their calendar? It's LLMs all the way down!!
- gopher_space 4mo ago
  > The only "fix" is to make an AI smart enough that it can understand context for each item, which is a tall order. Impossible as you said. Context isn’t static, it’s continuous, analog, and a conglomeration of viewpoints. AI cannot create useful context for itself because it is a machine with no desires. It doesn’t have a point of view, it has historical records. It moves forward in time by walking backwards (if that makes sense?)
dejj 4mo ago
RFC 3514 “evil bit” header flag to the rescue: https://www.rfc-editor.org/info/rfc3514/ https://www.rfc-editor.org/info/rfc3514/
- BobbyTables2 4mo ago
  At least the evil bit was a dedicated field at a known place… AI guardrails can’t even dream of that! Imagine if it was just the absence of “I’m evil” in the payload.
airstrike 4mo ago
It's not possible with today's LLM models, but we are not wedded to the current architecture.
- SlinkyOnStairs 4mo ago
  Realistically, we are. This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all.
  - airstrike 4mo ago
    Define "realistically". You're basically saying attention is all we need indefinitely into the future and all other gains come from more compute or scaffolding around current architectures. Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently. Who's to say we won't find a way to encode provenance or privilege natively into models such that the tradeoff changes? It's hard to say what the solution will be. If I knew it, I'd build it. But it's even harder to sustain that the current architecture is a crystalized global optimum.
    - gbalduzzi 4mo ago
      Aside from LLM architecture, that already is a complex issue, an issue is that training data is unstructured text. An LLM able to structurally separate context and instructions, should logically need separated data to train, and we don't have it. Moreover, while an equally powerful LLM architecture solving this may exists, there are no guarantees at all that we are able to come up with it in a reasonable timeframe. Without some signals moving in that direction, the most pragmatic and realistic way of looking at the problem is that it will not be solved in the near future
      - airstrike 4mo ago
        Thanks, I appreciate the thoughtful reply. I agree this doesn't mean we shouldn't try to address limitations with the current architecture. I just mean that I expect the root cause to be solved eventually if we ever really want to take steps towards AGI. Regarding signals moving in that direction, here's a paper you might enjoy https://arxiv.org/abs/2503.21937 https://arxiv.org/abs/2503.21937

embedding-shape 4mo ago

> It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide. Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones. The difficulty comes when you accept completely unreviewed/unchanged user-input as user messages, as your system/developer prompts needs to take this into account. You're better off to kind of whitelist what's possible rather than trying to prevent specific things, but seems that hasn't fully caught on yet. It feels like people and organizations are still trying to discover what works or not, and there are huge gaps being being left open because there simply isn't enough understanding of the limitations and impact of what they make available to users. We're already seeing it in lots of places, feels like it won't get better before it gets worse.

sillysaurusx 4mo ago
> Today's SOTA LLMs have pretty excellent following of these divisions Unfortunately "pretty excellent" is different from "perfect." I haven't kept track, but are you certain that given all possible inputs, the user prompt will never override the system prompt? Those are strong claims, and unless there's been an advancement in the tech, it doesn't seem possible. Reinforcement learning might make it much less likely, but that's different from impossible.
Muromec 4mo ago
If it was solved, the bug like this would not happen. It is also not always clear who is the user and how much they should be obeyed
- embedding-shape 4mo ago
  > If it was solved, the bug like this would not happen. Only if you only read the first line in my comment, there is more under that one too. It is clear, if you make it clear. These bugs happen because they don't clearly understand what should go where.
4mo ago

madamelic 4mo ago

> Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). I have been working on something like that: https://clawband.io https://clawband.io It's not quite ready for 'showtime' but feel free to take a look and give your impressions if you'd like. I feel the exact same way: I want to allow my agent to perform actions on all services but also limit what they can do. Basically my idea is wrapping individual service's APIs and then the middleware (Clawband in this case) enforces granular permissioning such as "can make credit cards but only up to $50" or "can send emails but only to specific domains". The agent never gets a raw API key to a service, it uses an intermediate API key that gets exchanged in the backend for calling the service after permissioning has been enforced.

KaiShips 4mo ago

[flagged]

mike_hock 4mo ago

I can't believe that fucking Terminator was prophetic.

hannahnine 4mo ago

[flagged]