What makes LLMs somewhat unique as a software product is that there is little-to-no separation between input and instruction. In most cases, the user's input ca

What makes LLMs somewhat unique as a software product is that there is little-to-no separation between input and instruction. In most cases, the user's input can also be considered part of the "prompt". This leads to the well-known "prompt injection" "vulnerability" which is really just a byproduct of the fundamental inability (and indeed undesirability) for the model to distinguish instruction from input (undesirable because the value and flexibility comes from allowing the user, rather than a programmer, to specify an action). On top of it though, it introduces a sort of disciplinary sloppiness around whether the program can be reasoned about. It's assumed that prompt `P` works for whatever input `I`, but the concatenated `P+I` is really the full input to the program that produces a desired output. But the only way to be confident about the program's behavior is to exhaust the input space, as no `P+A` tells you anything about how `P+B` will behave. This makes it difficult to leverage an LLM in any process where the desired result is 1. unknown and 2. matters. If the result is unknown it's not clear how to determine mistakes or correct them if they're made. And if the correctness of the result matters it's courting disaster to connect it to a process which is not able to be reasoned about. I think that's why LLMs are primarily being used to assist ideation (which is cool!) or "spammy" use cases like third-tier customer service or listicle generation, and haven't yet broken into any use case where they need to be reliable for complex tasks.