6 ms·

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs

The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking. The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.

bsenftner 2mo ago
In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.
- pavlov 2mo ago
  Their AI model tends to write a lot of lengthy comment blocks. That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.
  - kfse 2mo ago
    The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy
    - alasdair_ 2mo ago
      It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.
- applicative 2mo ago
  EU regulations are going to force it, friend. The tsunami is coming and cannot be stopped and Anthropic has jack to do with it.
  - HarHarVeryFunny 2mo ago
    I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI"). In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.
    - kergonath 2mo ago
      > It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.
      - 2_foos_in_a_bar 2mo ago
        The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese. So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.
- alasdair_ 2mo ago
  They say in the paper that they will not do this for code but it may happen for comments.
vintermann 2mo ago
A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.
- Majromax 2mo ago
  > But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.
  - ttoinou 2mo ago
    Whereas if you fully avoid the letter e, everyone will know you are George Perec
    - r2_pilot 2mo ago
      Who is Gorg Prc?
    - cratermoon 2mo ago
      And if you lowercase everything, you're e e cummings.
      - jambalaya8 2mo ago
        most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.
demibabs 2mo ago
> The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens. And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.
estebarb 2mo ago
What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?
- ludicrousdispla 2mo ago
  no, and in fact they should be extremely easy to detect if they are substituting a token that is a least-probable synonym
slickytail 2mo ago
[dead]
out_of_protocol 2mo ago
Does that also mean owner must re-run very model ever released by them to detect anything?

bsenftner 2mo ago

In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

pavlov 2mo ago
Their AI model tends to write a lot of lengthy comment blocks. That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.
- kfse 2mo ago
  The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy
  - alasdair_ 2mo ago
    It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.
applicative 2mo ago
EU regulations are going to force it, friend. The tsunami is coming and cannot be stopped and Anthropic has jack to do with it.
- HarHarVeryFunny 2mo ago
  I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI"). In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.
  - kergonath 2mo ago
    > It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.
    - 2_foos_in_a_bar 2mo ago
      The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese. So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.
alasdair_ 2mo ago
They say in the paper that they will not do this for code but it may happen for comments.

vintermann 2mo ago

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits. But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.

Majromax 2mo ago
> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.
- ttoinou 2mo ago
  Whereas if you fully avoid the letter e, everyone will know you are George Perec
  - r2_pilot 2mo ago
    Who is Gorg Prc?
  - cratermoon 2mo ago
    And if you lowercase everything, you're e e cummings.
    - jambalaya8 2mo ago
      most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.

demibabs 2mo ago

> The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens. And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.

estebarb 2mo ago

What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?

ludicrousdispla 2mo ago
no, and in fact they should be extremely easy to detect if they are substituting a token that is a least-probable synonym

slickytail 2mo ago

[dead]

out_of_protocol 2mo ago

Does that also mean owner must re-run very model ever released by them to detect anything?