4 ms·
> All those methods are applied with the specific goal of improving output quality and are applied to the extent that they do this Yes, that's the goal that wa
by simiones 2mo ago
> All those methods are applied with the specific goal of improving output quality and are applied to the extent that they do this
Yes, that's the goal that was used, but they are quite simplistic and crude methods, not some specifically designed function, with carefully fine tuned parameters or something. So, if a basic function like top_k can improve model utility, it's not impossible to imagine that watermarking could also happen to do so, or at least not have a significant negative effect. So whether the effect is deleterious or not is an empirical question, not something we can assume ahead of time.
> You're basically saying that if we train LLMs on watermarked text they'll be really good at producing text that looks watermarked
No, you're misunderstanding how the training works. If we train the model's output so that it minimizes the error function after the watermark is applied on it, the model will learn how to produce the best output it can given the watermark. It will produce better text that happens to be watermarked, not "more watermarked text". Same as if you train the model on minimizing `top_k_error(input) = |top_k_sampling(model_output(input)) - desired_output(input)|`, the model will learn to produce better output under top_k sampling, not learn to produce output that's "looks more top_k".
- simonh 2mo agoBut as I explained, the watermark is functionally random PRNG noise overlaid on the token probabilities. It’s not something that can be compensated for because it’s not predictable if you don’t have the seed and PRNG function.
- tsimionescu 2mo agoIf it's functionally random PRNG, then how does it differ from any other random sampling? If it's biased PRNG, then the LLM can adapt to the bias, and coincidentally might even benefit from this bias.
- simonh 2mo ago>If it's functionally random PRNG, then how does it differ from any other random sampling? For practical purposes it isn't.