9 ms·
But an LLM provider could very easily add a "Jev mode" to any existing model, right? LLMs already produce a probability distribution over arbitrary classes. Jus
by ryeights 11d ago
But an LLM provider could very easily add a "Jev mode" to any existing model, right? LLMs already produce a probability distribution over arbitrary classes. Just tell e.g. 5.6 Luna “here is the user's question, you must respond ONLY with the words 'foo', 'bar', or 'baz',” run a single forward pass of the model, and report the normalized probabilities of 'foo' 'bar' and 'baz' tokens before the first output.
With such an approach you could even retain full reasoning capability
- xrisk 11d agoMy hunch is that you would need some post training. On top of that; I don’t think the llm itself can read inside the transformer state although I can see how that could be enabled. (I feel that would open up yet another class of exfiltration opportunities)