Language models as I crudely understand it, predict the probability of the next token being N for every possible token N, and then pick the most likely one (if

Language models as I crudely understand it, predict the probability of the next token being N for every possible token N, and then pick the most likely one (if running "cold"), or from a distribution based on those probabilities for a bit more "heat". Hot = more imaginative, cold = more predictable. In any case, hot or cold, no part of the input has any natural authority. It is kind of a "shouting match" between the prompt and the injected part. Hard probably to solve, maybe impossible. Maybe you need a language model that is trained on "Prompt" and "Input" as 2 separate pieces of data in the first place, so the model is trained specifically on "given this example XYZ and this separate data ABC, what is the next lightly token for XYZ" This might be possible with a special separation token, which is not part of the gamut of regular tokens (so you can't make one yourself from the prompts), and have that there in training. Then you don't need delimiters at all. Whether such a thing is trainable or can be as good as GPT is another question I don't know!