5 ms·
Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer? > The distil
by HawtAds 2mo ago
Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer?
> The distillation data also did not contain any China-sensitive content.
This is a very big disclaimer.
It's like if I generate a dataset focusing exclusively on forestry and arboriculture obviously there won't be any useful censorship, or at least little that can be classified above a statistically significant threshold.
If you want to do a study on something more interesting and useful, do a piece on the various guardrail models of all the major LLM API providers. There are usually both input and output guardrails, and they tend to be almost-black boxes from the model routing point of view.
- chr15m 2mo agoSometimes it's important to rigorously investigate and prove "obvious" things. Sometimes those things turn out to not be obvious. That's part of good science.
- cgorlla 2mo agoWe discuss this in the writeup. While we expected this result, it is important for there to be data backing the claims, and an experimental setup that mirrors productions tasks is a useful tool for the conversations going on about this.
- StevenWaterman 2mo agoYeah, the benefit of showing this seems obvious to me. I probably would've expected the censorship to transfer slightly given the anthropic owl paper from years ago https://alignment.anthropic.com/2025/subliminal-learning/ https://alignment.anthropic.com/2025/subliminal-learning/ But that was about transferring from a finetuned model to another finetune of the same base model, good to see more evidence that it doesn't transfer cleanly across different base models in a more realistic scenario than an "owl-loving model" Edit: From *year ago. It's been a long year haha
- antves 2mo agoIt is surprisingly not obvious, neural nets are weird just like brains https://www.nature.com/articles/s41586-026-10319-8 https://www.nature.com/articles/s41586-026-10319-8
- FergusArgyll 2mo agoSubliminal Learning: language models transmit behavioral traits via hidden signals in data https://arxiv.org/html/2507.14805v1 https://arxiv.org/html/2507.14805v1 "In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “student” model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T."
- aesthesia 2mo agoIn those experiments, the effect only happened between models from the same family (and likely the same weight initialization).
- janalsncm 2mo agoIt is self-evident. But now if someone questions it in the future, this is a data point that distilling from Deepseek isn’t going to turn your finance bot into a communist. Also, sometimes things which seem self-evident turn out to be surprising.
- nullc 2mo ago> Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer? If the model architecture is too similar it can transfer, even if you never sample on it... because the censorship can depend on hidden state in common that you do sample. This creates a particular value in distinct architectures that isn't apparent from their direct utility.