4 ms·

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only i

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.

ddalex 1mo ago
Probably they bias the RNG for selecting the next token. This can be done practically in a lot of ways, including during training. I suspect the signal will be significantly under the noise floor, so it's not detectable if you don't know exactly what to look for, but certainly you can submit more information then the textual contents.
- vintermann 1mo ago
  But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.
  - Majromax 1mo ago
    A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226 https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation. After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts. The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).
    - tmp10423288442 1mo ago
      Anthropic actually tells you the approach they use, and it's not that. From their Claude Text Watermark page[0]: "Claude’s text watermark is a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024." The Nature paper is "Scalable watermarking for identifying large language model outputs"[1]. This method does not separate out tokens into separate classes, but merely uses a seed for the PRNG that selects which among the most likely tokens generated by the LLM will actually be output. This has the advantage that there's no green and red token sets, so no token is systematically favored or disfavored. If a particular token is overwhelmingly predicted to be the most likely candidate, it will almost certainly be selected, so the watermark doesn't affect that. Even if there are several choices of output token at a point that have similar probability of selection, the watermark doesn't systematically bias in favor of one token or the other. This is actually a quite elegant method of watermarking that, contrary to people's fears, won't adversely affect the model output. The main concern I have with it is that it appears that you can't actually test the watermark locally, without uploading it to Anthropic. I'm not sure why that's the case, since there's no particular reason the watermarking key has to be private, except if you want to prevent others from generating text with their own LLMs that is watermarked to look like it's generated by Anthropic - but everybody wants their text to not have the watermark. [0]: https://www.anthropic.com/news/claude-text-watermark#:~:text=to%20do%20so.-,Which%20specific%20method%20of%20watermarking%20do%20you%20use%3F,changes%20the%20source%20of%20the%20randomness%20used%20to%20pick%20among%20words.,-There%20are%20limitations https://www.anthropic.com/news/claude-text-watermark#:~:text... [1]: https://doi.org/10.1038/s41586-024-08025-4 https://doi.org/10.1038/s41586-024-08025-4
unbelievr 1mo ago
The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking. The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.
- bsenftner 1mo ago
  In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.
  - pavlov 1mo ago
    Their AI model tends to write a lot of lengthy comment blocks. That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.
    - kfse 1mo ago
      The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy
nullbio 1mo ago
I'm sure they could use it to fingerprint people, at the very least.
unclebucknasty 1mo ago
Simple version: In instances wherein the otherwise statistically chosen next word is a "toss-up", watermarking removes the randomness by imposing specific choices, determined by a key. This then becomes a detectable pattern when scanned with the key (stastically—detection itself is probabilistic). >use it to store arbitrary information No additional data is embedded. The range of available data is constrained by the text being generated (i.e. the sets of "next words" per text).
- mbreese 1mo ago
  From what I’ve read, they won’t be imposing specific choices, but using a different (biased) RNG for those “toss-up” choices. With enough sampling, you could detect if the RNG was biased or not.
  - unclebucknasty 1mo ago
    This is what I meant by "imposing specific choices, determined by a key". Maybe "impose" or "specific" were too strong in my attempt to simplify? I attempted to clarify that the impositions themselves are not deterministic, by indicating that the entire process is still probabilistic. Maybe Anthropic's explanation is simple enough [0]: >When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude. [0] https://www.anthropic.com/news/claude-text-watermark https://www.anthropic.com/news/claude-text-watermark
cratermoon 1mo ago
I realize that short attention spans are pervasive now, but the link to the explanation is only eight paragraphs in https://declaude.org/watermarking/ https://declaude.org/watermarking/
- ozozozd 1mo ago
  Yeah kind of surprised that people didn’t have enough patience for Gruber’s writing. It’s quite fun and engaging.
- notfed 1mo ago
  In addition to holding the key, wouldn't you additionally need to know exactly which model to check against? So for passive detection to happen, I think each company would need to check every message against every model version? Also would need to spend resources re-invoking the each model version against each message.
cocoa19 1mo ago
Removing the watermark usefulness depends on your use case. If you care to avoid detection, yes, it is useful. If you care about the best possible sequence of words, then the damage is already done once watermarked.
- jeansilga 1mo ago
  [dead]
j45 1mo ago
It’s possible to ask each model and build it.
mickdarling 1mo ago
If it stores unique information, by definition it can store arbitrary information because it can point to arbitrary information. So they can have it relate to anything they want. Even a full breakdown of the original text if they choose.

ddalex 1mo ago

Probably they bias the RNG for selecting the next token. This can be done practically in a lot of ways, including during training. I suspect the signal will be significantly under the noise floor, so it's not detectable if you don't know exactly what to look for, but certainly you can submit more information then the textual contents.

vintermann 1mo ago
But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.
- Majromax 1mo ago
  A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226 https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation. After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts. The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).
  - tmp10423288442 1mo ago
    Anthropic actually tells you the approach they use, and it's not that. From their Claude Text Watermark page[0]: "Claude’s text watermark is a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024." The Nature paper is "Scalable watermarking for identifying large language model outputs"[1]. This method does not separate out tokens into separate classes, but merely uses a seed for the PRNG that selects which among the most likely tokens generated by the LLM will actually be output. This has the advantage that there's no green and red token sets, so no token is systematically favored or disfavored. If a particular token is overwhelmingly predicted to be the most likely candidate, it will almost certainly be selected, so the watermark doesn't affect that. Even if there are several choices of output token at a point that have similar probability of selection, the watermark doesn't systematically bias in favor of one token or the other. This is actually a quite elegant method of watermarking that, contrary to people's fears, won't adversely affect the model output. The main concern I have with it is that it appears that you can't actually test the watermark locally, without uploading it to Anthropic. I'm not sure why that's the case, since there's no particular reason the watermarking key has to be private, except if you want to prevent others from generating text with their own LLMs that is watermarked to look like it's generated by Anthropic - but everybody wants their text to not have the watermark. [0]: https://www.anthropic.com/news/claude-text-watermark#:~:text=to%20do%20so.-,Which%20specific%20method%20of%20watermarking%20do%20you%20use%3F,changes%20the%20source%20of%20the%20randomness%20used%20to%20pick%20among%20words.,-There%20are%20limitations https://www.anthropic.com/news/claude-text-watermark#:~:text... [1]: https://doi.org/10.1038/s41586-024-08025-4 https://doi.org/10.1038/s41586-024-08025-4

unbelievr 1mo ago

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking. The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.

bsenftner 1mo ago
In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.
- pavlov 1mo ago
  Their AI model tends to write a lot of lengthy comment blocks. That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.
  - kfse 1mo ago
    The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy

nullbio 1mo ago

I'm sure they could use it to fingerprint people, at the very least.

unclebucknasty 1mo ago

Simple version: In instances wherein the otherwise statistically chosen next word is a "toss-up", watermarking removes the randomness by imposing specific choices, determined by a key. This then becomes a detectable pattern when scanned with the key (stastically—detection itself is probabilistic). >use it to store arbitrary information No additional data is embedded. The range of available data is constrained by the text being generated (i.e. the sets of "next words" per text).

mbreese 1mo ago
From what I’ve read, they won’t be imposing specific choices, but using a different (biased) RNG for those “toss-up” choices. With enough sampling, you could detect if the RNG was biased or not.
- unclebucknasty 1mo ago
  This is what I meant by "imposing specific choices, determined by a key". Maybe "impose" or "specific" were too strong in my attempt to simplify? I attempted to clarify that the impositions themselves are not deterministic, by indicating that the entire process is still probabilistic. Maybe Anthropic's explanation is simple enough [0]: >When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude. [0] https://www.anthropic.com/news/claude-text-watermark https://www.anthropic.com/news/claude-text-watermark

cratermoon 1mo ago

I realize that short attention spans are pervasive now, but the link to the explanation is only eight paragraphs in https://declaude.org/watermarking/ https://declaude.org/watermarking/

ozozozd 1mo ago
Yeah kind of surprised that people didn’t have enough patience for Gruber’s writing. It’s quite fun and engaging.
notfed 1mo ago
In addition to holding the key, wouldn't you additionally need to know exactly which model to check against? So for passive detection to happen, I think each company would need to check every message against every model version? Also would need to spend resources re-invoking the each model version against each message.

cocoa19 1mo ago

Removing the watermark usefulness depends on your use case. If you care to avoid detection, yes, it is useful. If you care about the best possible sequence of words, then the damage is already done once watermarked.

jeansilga 1mo ago
[dead]

j45 1mo ago

It’s possible to ask each model and build it.

mickdarling 1mo ago

If it stores unique information, by definition it can store arbitrary information because it can point to arbitrary information. So they can have it relate to anything they want. Even a full breakdown of the original text if they choose.