3 ms·
> The problem is that these instructions are not applied after the model has finished doing the work, it becomes part of the same work - If you tell an agent to
by wren6991 2mo ago
> The problem is that these instructions are not applied after the model has finished doing the work, it becomes part of the same work - If you tell an agent to use short sentences, avoid jargon, never overwhelm you and only include the most important details, you are asking it to continuously compress its output into a lower-bandwidth format.
> That compression is lossy.
> You probably never notice what got dropped because the output still reads nicely.
> ASD-STE is a great example because it sounds so reasonable. It was designed to make documentation unambiguous for humans. But an agent isn’t a human technical writer, and the raw state is often the most information-dense representation available. Meanwhile the style rules sit on the same instruction list as: solve the task, use tools correctly, preserve abstractions, don’t break anything.
Author seems to have some misconceptions about LLMs. They already code-switch for us: the way they speak in chain-of-thought is completely different from the relatively normal language generated as human-facing output. You can observe this in any open-weight LLM, or in leaked CoT content from GPT5.x series etc: it's terse, barely follows sentence structure, lots of repeated checks and second-guessing.
On the next turn the model usually still has access to its previous turn's chain-of-thought, and I imagine that's what it'll use as reference, rather than the softer human-facing prose.
This being the case, asking the LLM to code-switch to an easier dialect for us doesn't seem that harmful.
For a more extreme example: if I talk to an LLM in Japanese then its response will be in Japanese, but its CoT will still be in either English or Chinese (depending on the model). These are two completely separate languages, but the LLM just kinda deals with it.
- pindab0ter 2mo agoThe CoT might "in English", but is it really, or is the CoT just another presentation layer over the actual weights? What I think the author (and I) are wondering about, is whether instructions like these might influence not _only_ the final output, but also the way it got there.
- wren6991 2mo ago> The CoT might "in English", but is it really Yes, it really is. It's emitting literal English text in patterns that have been coaxed through RL into doing some kind of useful fuzzy computation. > is the CoT just another presentation layer over the actual weights? What does this mean? Which part of the forward pass are you talking about here? > What I think the author (and I) are wondering about, is whether instructions like these might influence not _only_ the final output, but also the way it got there. Sure, LLMs are chaotic. If I mention as a casual aside that the sky is blue, I'll get a different answer, even when my query has nothing to do with the colour of the sky. The fact that style instructions compete for attention with more concrete instructions is probably the one part of the author's post that I agree with. Different doesn't mean worse; if I run with greedy sampling (so deterministic) then minor punctuation differences in my query still produce completely different answers. My reading is that they are actually making a more concrete point, which is: requesting a simpler style compresses the output, and this causes loss of fidelity in future turns. I think this is something you would have to demonstrate instead of hand-waving. I'm happy to be corrected on specifics.