3 ms·
> “By definition it must make text worse … because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a wor
by ucha 2mo ago
> “By definition it must make text worse … because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.”
Gruber made an effort to but doesn't fully understand how SynthID works. LLMs select the next word randomly from a set probability distribution, so there is no "best choice" unless you run the LLM with a temperature of 0 which would give out terrible results. Anthropic runs a non-distorting version of SynthID that doesn't change the probabilities of the underlying distribution of tokens. It makes the watermark less likely to work over smaller samples but preserves text quality. I encourage the mathematically inclined to read the paper:
https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4
- tarvaina 2mo agoI came here to quote the same sentence. Here's another way to look at it: Suppose there actually is a best word choice. The LLM doesn't know what it is but makes a guess. Maybe it's the best one, maybe it isn't. The probability that SynthID changes the best choice to a worse one is equal to the probability that it changes a worse choice to the best one.
- abustamam 2mo agoI think that depends on the distribution of good choices and bad ones. There may be 10 choices and maybe 8 of them could be appropriate given a context, and 2 are absolutely nonsensical. Or it could be vice versa. And its a spectrum as well.
- tempestn 2mo agoBut the choices are weighted based on those probabilities. This doesn't affect the weightings, only how the final weighted pseudo-random selection is made.
- abustamam 2mo agoRight, but that's what makes me doubt this statement in the comment I was replying to > The probability that SynthID changes the best choice to a worse one is equal to the probability that it changes a worse choice to the best one.
- Marsymars 2mo ago> LLMs select the next word randomly from a set probability distribution, so there is no "best choice" unless you run the LLM with a temperature of 0 which would give out terrible results. I'm not understanding how the word with the highest probability isn't the "best choice"?
- baobabKoodaa 2mo agoYou will understand if you run some LLM models with a greedy sampler that does that. The text quality begins to deteriorate rapidly. This is a very counter intuitive result so I don't blame you for not understanding until you actually tried it and experienced it for yourself.
- Marsymars 2mo ago> You will understand if you run some LLM models with a greedy sampler that does that. The text quality begins to deteriorate rapidly. Right, I've done this, and this makes sense to me, but I'm not following how that falsifies the top probability word being the best choice in any particular instance. "Picking only the best word at each decision point results in a worse final result" seems like an imminently reasonable hypothesis.
- StevenWaterman 2mo agoI think you're using different definitions of best. If best = leads to a correct answer overall then by definition anything that leads to a bad outcome can't be best
- Max-Limelihood 1mo agoBecause it's pure exploit on the explore/exploit tradeoff. The best outputs come when the LLM comes up with lots of different ideas, considers them, and selects the best one. If you sample at low temperature it tends to regenerate the same ideas over and over.