3 ms·

> I get the impression that every AI lab is desperately trying... Of course. I wonder how we managed way back in the day to produce systems that can handle un

> I get the impression that every AI lab is desperately trying... Of course. I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.

simonw 2mo ago
If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.
- skydhash 2mo ago
  It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions. We have even observed that the most effective LLM usage is when paired with an expert in charge of the goals. Dark factory and other automated harnesses (specs engineering and what not) seem to be a dead end. The most impactful approach to this date is an interactive conversation as a succession of small and verifiable tasks.
  - deleted 2mo ago
    [deleted]
- simoncion 2mo ago
  Yeah, this matches what I've learned over the past couple of years from reading some of your blog posts and reading your interactions in comment threads here and elsewhere. You're a politician, rather than a truthseeker. The absolute most I've seen from you in response to an extensive teardown of your argument, supporting evidence, and subsequent conversational judo was a «Wow. That was well phrased.» and no subsequent change in your publicly-expressed opinions. I'd do more than gesture at the relevant lesson taught to us by Google Fiber, Tesla, SpaceX, etc., but you'd not be publicly moved, so it's a waste of time.
  - simonw 2mo ago
    > You're a politician, rather than a truthseeker. Justify that. Also, which "extensive teardown" are you talking about there?
    - simoncion 2mo ago
      > Justify that. ... Also, which "extensive teardown" are you talking about there? I haven't bothered to keep a detailed dossier, [0] and I CBA to go back through a year or two's worth of conversation between you and other folks to synthesize one. You're someone who's well-regarded, which is why I did a year or two of careful, open-minded observation before I came to my conclusion. [0] Don't you think it'd be weird if I had kept one? Who the hell does that?
- 0xDEAFBEAD 2mo ago
  Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?
  - simonw 2mo ago
    Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#special-tokens https://github.com/openai/harmony/blob/main/docs/format.md#s... Then you train your model to take those tokens into account. Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 https://arxiv.org/abs/2603.12277 > We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label
famouswaffles 2mo ago
>reliably instruct a dumb-as-bricks CPU Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Like, you're not making any sense here. None of the things that make this possible with CPUs is remotely relevant here, and the fact that you don't seem to understand this but act so smug is strange.
- simoncion 2mo ago
  > Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Just as the immense amount of scaffolding around the dumb-as-bricks CPU enables extremely sophisticated and useful things to be done with that pile of fused sand and copper, the immense amount of scaffolding around the dumb-as-bricks LLM enables very sophisticated and useful things to be done with that pile of linear algebra. Don't confuse the infrastructure that makes the stupid bit in the middle actually useful with the stupid bit in the middle.
  - famouswaffles 2mo ago
    LLMs are not the "stupid bit in the middle." They're almost the entire value. LLMs were wildly useful before any sort of scaffolding. They are not "dumb as bricks". They are highly capable, flexible, intelligent prediction machines. The only one confused here is you, and you've still not managed to tell us in an actionable way how exactly CPU scaffolding is relevant here. Tell us, if it's so easy, or make your millions selling it. We're all waiting. I'll give you a hint. CPUs never had to interpret the meaning of arbitrary content in order to do their job, and LLMs do.
    - simoncion 2mo ago
      > [LLMs are] almost the entire value. I agree that without the LLM, the system around it is useless. I also agree that without the CPU, the system around that is useless. > LLMs are not the "stupid bit in the middle." Both a model and a CPU without their scaffolding are incredibly useless. You don't even get I/O! If you can't get data out of or into the model or CPU, what use is it?

simonw 2mo ago

If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.

skydhash 2mo ago
It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions. We have even observed that the most effective LLM usage is when paired with an expert in charge of the goals. Dark factory and other automated harnesses (specs engineering and what not) seem to be a dead end. The most impactful approach to this date is an interactive conversation as a succession of small and verifiable tasks.
- deleted 2mo ago
  [deleted]
simoncion 2mo ago
Yeah, this matches what I've learned over the past couple of years from reading some of your blog posts and reading your interactions in comment threads here and elsewhere. You're a politician, rather than a truthseeker. The absolute most I've seen from you in response to an extensive teardown of your argument, supporting evidence, and subsequent conversational judo was a «Wow. That was well phrased.» and no subsequent change in your publicly-expressed opinions. I'd do more than gesture at the relevant lesson taught to us by Google Fiber, Tesla, SpaceX, etc., but you'd not be publicly moved, so it's a waste of time.
- simonw 2mo ago
  > You're a politician, rather than a truthseeker. Justify that. Also, which "extensive teardown" are you talking about there?
  - simoncion 2mo ago
    > Justify that. ... Also, which "extensive teardown" are you talking about there? I haven't bothered to keep a detailed dossier, [0] and I CBA to go back through a year or two's worth of conversation between you and other folks to synthesize one. You're someone who's well-regarded, which is why I did a year or two of careful, open-minded observation before I came to my conclusion. [0] Don't you think it'd be weird if I had kept one? Who the hell does that?
0xDEAFBEAD 2mo ago
Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?
- simonw 2mo ago
  Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#special-tokens https://github.com/openai/harmony/blob/main/docs/format.md#s... Then you train your model to take those tokens into account. Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 https://arxiv.org/abs/2603.12277 > We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label

famouswaffles 2mo ago

>reliably instruct a dumb-as-bricks CPU Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Like, you're not making any sense here. None of the things that make this possible with CPUs is remotely relevant here, and the fact that you don't seem to understand this but act so smug is strange.

simoncion 2mo ago
> Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Just as the immense amount of scaffolding around the dumb-as-bricks CPU enables extremely sophisticated and useful things to be done with that pile of fused sand and copper, the immense amount of scaffolding around the dumb-as-bricks LLM enables very sophisticated and useful things to be done with that pile of linear algebra. Don't confuse the infrastructure that makes the stupid bit in the middle actually useful with the stupid bit in the middle.
- famouswaffles 2mo ago
  LLMs are not the "stupid bit in the middle." They're almost the entire value. LLMs were wildly useful before any sort of scaffolding. They are not "dumb as bricks". They are highly capable, flexible, intelligent prediction machines. The only one confused here is you, and you've still not managed to tell us in an actionable way how exactly CPU scaffolding is relevant here. Tell us, if it's so easy, or make your millions selling it. We're all waiting. I'll give you a hint. CPUs never had to interpret the meaning of arbitrary content in order to do their job, and LLMs do.
  - simoncion 2mo ago
    > [LLMs are] almost the entire value. I agree that without the LLM, the system around it is useless. I also agree that without the CPU, the system around that is useless. > LLMs are not the "stupid bit in the middle." Both a model and a CPU without their scaffolding are incredibly useless. You don't even get I/O! If you can't get data out of or into the model or CPU, what use is it?