4 ms·
No, the thinking trace is generated tokens but demarcated by control tokens to suppress them from API output. To inject things into that you'd just add words, w
by mike_hearn 1y ago
No, the thinking trace is generated tokens but demarcated by control tokens to suppress them from API output. To inject things into that you'd just add words, which is what their prefill experiment did. That experiment is where they distinguish between just tampering with the context window to inject thoughts vs injecting activations.
- andy99 1y agoWhat I was wondering is, do the injections cause the thinking trace to change (not whether they actually typed text into the thinking trace) and then the model “reflects” on the fact that it’s thinking trace has some weird stuff in it, or do these reflections occur absent any prior mention of the injected thought.
- mike_hearn 1y agoWell, the paper makes no mention of any separate hidden traces. These seem to be just direct answers without any hidden thinking tokens. But as the thinking part is just a regular part of the generated answer I'm not sure it makes much difference either way.