4 ms·

Utterly imperceptible, even when studied under the microscope in a way that LLM text very rarely is in practice. It will be interesting to see whose concerns a

• red_admiral 2mo ago
  If anyone notices degraded quality, that would imply they could break the crypto behind the watermark. For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
  • rcxdude 2mo ago
    I'm not sure the watermark has been demonstrated to have that kind of property. Even if you use a CSRNG as the 'key' once you're feeding it through the token selection process it's going to risk re-introducing certain correlations.
    • kalkin 2mo ago
      How would you re-introduce correlations on top of a CSPRNG without a cryptographic break?
      • rcxdude 2mo ago
        Because in order to not break the token selection process you are making decisions biased by the LLM's output, and in order to make the watermark detectable without the entire context, you are making those decisions based on a relatively small number of preceding tokens. To give an example in the extreme: if you make the bias total and only make that decision based on the preceding token, there are certain token pairs that your model will never output, and this will be pretty obvious even to people just reading the text (because of any given common two-token phrase, there's a 50% chance you would just disappear in watermarked text). You can make this less extreme and more hidden by increasing the window and reducing the bias, but at the cost of reducing the signal. I don't know exactly what the tradeoff curve looks like, so it might be that you can reach set of parameters where the bias is in principle undetectable without the key but still reliably detectable for realistic lengths of text segments, but I would not assume that this is definitely the case.
        • kalkin 2mo ago
          This doesn't change the temperature of the model. It's not going to make something that's a 70% chance suddenly a 100% chance. It also doesn't change the context length. At most, it seems like it may reduce some of the variation between different requests sampled from the same prompt with nonzero temperature, if I'm reading this right from the Google paper that Anthropic says they're implementing: > For our experiments, we configure SynthID-Text to be single-sequence non-distortionary; this preserves text quality and provides good detectability, while having some reduction to inter-response diversity. We call this configuration ‘non-distortionary SynthID-Text’ (and where not otherwise specified, ‘SynthID-Text’ also refers to this). https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4
          • rcxdude 2mo ago
            That approach is pretty much what I was describing. The example in the paper uses the previous 4 tokens to derive the bias for the next token, so some 5 token sequences become more likely and others becomes less likely compared to what the LLM would normally produce. The context and temperature of the LLM stays the same, but the sampling process is biased by something that depends on far shorter runs of text.
  • kmeisthax 2mo ago
    Great, but SynthID watermarking isn't a cryptographically-secure scheme, nor could one be made, because all the things necessary for the scheme to be crypto-secure are also things that go against being human-comprehensible. The SynthID watermark scheme relies on the LLM's own logits being a map into acceptable substitutions for specific words. For example, if we have the sentence "The quick brown fox jumps over the lazy dog", the LLM might give the last word[0] as 98.5% "dog", 1% "hound", 0.5% "canine". In normal decoding we sample the logits after scaling them against a temperature parameter; in SynthID we additionally bias the logits to encode output from the watermark CSPRNG - e.g. the CSPRNG might decide to boost the third logit of this particular token to 65%. But wait! The LLM might have selected valid synonyms for the token, but those synonyms will never be correct because this is a very well-known sentence. To humans, it's one sentence-long token, in the sense that most English-speaking humans who encounter this sentence will see the first few words and already know the ending. It thus has one bit of entropy[1]. Adding SynthID watermarks to this sentence necessarily has to change the words and thus adds more information - namely, that some process outside of the LLM itself has modified the text. We don't need to cryptographically distinguish AES from random if both the AES (SynthID) and random (normal decoding) happens to also be constrained to form a valid English sentence. [0] I am too lazy to account for multi-token words. Let's just assume the LLM was trained with a pretokenizer that splits exclusively on whitespace and every word in the training set got assigned a token. [1] Compressible information
    • usef- 2mo ago
      Your example is not realistic: it won't arbitrarily change a 0.5% token up to 65%. If it did it would absolutely degrade performance in a very measurable way. It's only biasing the randomness for tokens where it does have multiple plausible choices. For constrained quotes like "quick brown..." the synthid watermark would be weak, but that's by design.
    • red_admiral 2mo ago
      That doesn't match my understanding of how the watermark works. Let's say we have your distribution of (0.985, 0.01, 0.005). The AI somehow has to sample that distribution, let's say the pre-watermark model picks as many uniform (i.i.d.) bits as it needs to do the sampling, then does a Smirnov transform or something to get the correct distribution. The AI with watermark picks its bits from a CSPRNG, then does the same transform to get the distribution. The result I think is that you can reduce distinguishing the distribution to breaking the CSPRNG. In this example the entropy after the transform is around 0.13 bits, so the watermarking scheme cannot add any useful information. Once you have hundreds or prehaps thousands of bits of entropy, it starts to become feasible to add information into that 'spare' space.
• Yizahi 2mo ago
  My personal concern after the initial news broke about the topic, was not that generated content would become worse, but that to verify the presence of the watermark, one need to upload the complete and unedited source back to these megacorpos, and likely separately to each of them in a few years. Basically this is a truly novel mechanism to legally siphon any new and not yet stolen intellectual property and do that in perpetuity. And there is an absolutely disastrous loss of responsibility here, because soon any organization which requires content integrity (say science, education, literature etc.) will add a new requirement to accept that the submitted new work will be legally and "voluntarily" fed into the LLM megacorpos (with implied loss of property right). Basically we are on the verge of moment when intellectual property will cease to exist outside of a handful of max security closed sites for military or similar work. Everything else will be automatically and now legally fed into the maw of LLMs. The fact that watermarking work so spectacularly transparent and undetectable only allows this moment to happen sooner.
  • Ygg2 2mo ago
    If you suspect that, make sure your text mentions at random gremlins, groblings and jack bunnies. I.e. pre poison your input.
    • Yizahi 2mo ago
      If you get a choice to do anything like that, then the whole issue in my top comment is irrelevant. My concern was about not having such choice at a global scale.