4 ms·

Maybe it has no false positive rate

by sureMan6 2mo ago

suddenlybananas 2mo ago
That's essentially impossible, unless you mean they didn't measure a false positive rate.
- Filligree 2mo ago
  For watermarked long-form text, it is actually possible. Makes the watermark more fragile, but the math is considerably more forgiving than usual.
  - embedding-shape 2mo ago
    > For watermarked long-form text What about "watermarked long-form code"? I'm having a hard time understanding how a model could watermark not prose, but functional/semantic text like code, that actually has meaning. You can't switch our the characters, you can't use various types of whitespace, you can't add arbitrary code comments, and a lot of other restrictions. Is there any state of the art methods for watermarking code without affecting the quality/correctness?
    - wpietri 2mo ago
      As anybody who has put together a coding standard knows, there are a lot of options for individual expression, meaning a lot of room for things like watermarking. And of course you can add arbitrary comments; my Claude-generated code is very verbose.
      - embedding-shape 2mo ago
        > there are a lot of options for individual expression, meaning a lot of room for things like watermarking The way I use LLMs (and I'd advice everyone to do the same) there really isn't, the agent implements things exactly how I want them, or I use the agent to massage it into the exact bit-by-bit version I imagined when I first sent the prompt afterwards. I honestly don't know what the point would be to let the agents compose worse code than what I'd do manually, although I know it's a popular approach taken by many. > And of course you can add arbitrary comments; my Claude-generated code is very verbose. So watermarking for all users who allow code comments from agents, no watermarking for us who force the agents to never write a single code comment? Alright, I'd be fine with that.
        wpietri 2mo ago
        From what I've seen, your approach to LLMs is exceedingly rare, so I suspect it's one the people who care about watermarking aren't very concerned with. And the reason to let Claude make worse code than a professional would by hand is basically suppressed demand. Since programmers are expensive, previously code mostly got written when a large number of dollars were on the line, or when an individual programmer did something not economically optimum (e.g., hobby project). That left a whole lot of somewhat less valuable software unwritten. It's the economic space that no-code tools have been nibbling on for years. One way to think of things like Claude Code is as effectively no-code tools. Pre-LLM no-code tools would produce data structures that got executed by special environments without ever being seen or tuned by a human. Claude Code can be used just like that, with text as the input and python as the intermediate representation that nobody ever looks at. That approach probably isn't sustainable for what we professional programmers would call a serious project. Claude can easily get in over its head and I expect that its code decays over time, in a fashion similar to how many human teams get in a state where they just have to rewrite everything. But faster, I'd expect. But there are a lot of unserious projects that previously would have never been created. E.g., a quick app to manage your little league team, or a bit of in-house business stuff in the "a little hard to do with a spreadsheet" range.
        olmo23 2mo ago
        > I honestly don't know what the point would be to let the agents compose worse code than what I'd do manually You never generate throwaway code used to test an external service? or try out an interface idea? There's a lot of code that's only meant to be ran once. I often dont even care what language it's written in.
        embedding-shape 2mo ago
        > You never generate throwaway code used to test an external service? or try out an interface idea? And save/persist it? No, most of any experimental stuff goes into /tmp which gets cleared out on reboot, nothing I care to save in any repository. Or just "show me how this would look like" and then it's only in the session itself (and the logs/state I suppose, technically...).
    - junofan 2mo ago
      [dead]
    - SkyBelow 2mo ago
      For straight generated code it'll likely need more text, but it'll still show up. In cases where one token is extremely likely, it'll randomly be red or green and still be picked in either case as it is simply the best (or only) option. So you'll have more tokens that don't show a pattern either way (half of these cases will match and half won't, just the same as if a human wrote it). Meaning you'll need more instances where multiple tokens were all likely to see if there is a pattern. Given the check algorithm can't identify these cases, it can only judge on the overall text, so the more strict a language, the more the length requirement scales. Where I wonder if this keeps working is in tool calls. Often, you don't take code straight from the llm, you take the results of a tool call to edit already existing code. It might be that the result of this leads to far too few signals to pick up, meaning that this only works when one does significant generation with a single model (even swapping between different models, at least by different companies, breaks this just as much as having a human write parts of the code). Think of it like finding a loaded dice. A dice that has a slight bias in a few dozen roles is just random chance. If that bias continues after hundreds of thousands of roles, the dice is loaded. But will a code base have enough samples, especially when edits made from tool calls? I could see this being unable to detect things at the size of a reasonable PR and only being useful for massive sets of changes and only if the person behind them didn't structure their AI usage to avoid detection.
  - OGWhales 2mo ago
    And yet, it remains possible that a human could write the same sequence of characters.
    - no_multitudes 2mo ago
      How often do you add seemingly-random zero-width unicode characters to the text you write?
      - OrangeMusic 2mo ago
        That's not how it works (that would be trivial to erase).
embedding-shape 2mo ago
Read said section yourself perhaps.
mysterydip 2mo ago
If LLM training data is human-written, and LLM output mimics that input, how could you not have false positives?
- SkyBelow 2mo ago
  Because it won't be in the training directly. It is applied after a model generates its distribution of likely tokens, biasing each token randomly based on a random key and unrelated to any meaning of the words. So half the time, the most likely token becomes more likely and half the time it becomes less likely, and the same for every other token (when temperature is above 0). You then look at the tokens actually picked to see how closely they follow this pattern that isn't connected to the meaning of the tokens. With enough text, you can then analyze the chance of it happening by chance verses being because the generation of the tokens was done using the algorithm, and you can save a positive result until you are arbitrarily sure. There is a chance of a false positive, but the chance of a false positive approaches the chance that the murderer happened to have fingerprints that matched your and both forensics labs happened to have mixed up the dna tests and the eye witness happened to misremember the face and your phone gps happened to glitch out and put you at the murder scene at the time of the crime all happening. It is theoretically possible only in the same sense that quantum teleporting a cat is theoretically possible. The real question is how much text do they need for a given level of certainty and what do they check for. If they flag a positive at a p value <.01, that's a problem. If they can reasonably get a p value of < 1e-12 in only a few paragraphs of text, that is effectively no false positives (but a lot of 'too short to analyze' outcomes).
  - ricericerice 2mo ago
    How do you verify in practice then? Wouldn't you need the original prompt so you can reobtain the likely token distribution to validate again the random key(s)?
    - Eisenstein 2mo ago
      A token is hashed and used to seed a random number generator, which produces the red list for the token after it. Paper: * https://arxiv.org/abs/2301.10226 https://arxiv.org/abs/2301.10226
wrsh07 2mo ago
Somewhat trivially, if I ask Claude to transcribe an image and then check if that transcription is ai generated it will likely say yes. Many users are not smart enough to realize that the transcription step is where the ai (watermarks) were necessarily injected.
- basch 2mo ago
  How is a perfect transcription of an image watermarked?
  - FeteCommuniste 2mo ago
    "Perfect as far as human perception can tell" is a weaker standard than "bit-to-bit copy." Maybe it's that?
  - wrsh07 2mo ago
    It depends on how it does watermarking!! Note, there are many ways to represent words visually on computers that look identical
    - basch 2mo ago
      If they were substituting glyphs for identical ones people would be able to reverse engineer it. Theres no way that’s what they are doing.
wtfwhateven 2mo ago
Why would you say something so ridiculous?
- jobigoud 2mo ago
  I think they mean it like this: imagine you ask me a random number sequence. I give you a random number sequence. Little did you know, I used a very specific PRNG to generate it, so later I can prove with certainty that your number was generated by me, and you can't say you came up with it yourself. There is no room for false positive here in the same way you can't randomly find a collision in a hash function if it's strong enough. Like the rate is so infinitesimal that it is effectively zero. Now replace random number sequence with prompted string of words. And instead of using the PRNG on every word I use it every n words. If the generated text is sufficiently long I can tell by matching the expected deterministic pattern. You can defeat it by changing the words yourself and triggering a false negative but there isn't really any room for a false positive if the text is long enough and the pattern matches perfectly. If the pattern doesn't match then I can compute a probability.
bufbupa 2mo ago
Sorry you're getting downvoted, this interpretation doesn't seem that far fetched to me. Here's the strawman: The text-based watermarking is going to be done procedurally instead of generatively. Maybe they add some sequence of zero-width Unicode characters to all generated text at certain intervals. Then, there is effectively no false positive possible (because humans would [effectively] never type such sequences of unicode naturally). It may survive some editing (depending on how you select/edit the characters), and it's possible to be stripped (false negatives).

Maybe it has no false positive rate

suddenlybananas 2mo ago

That's essentially impossible, unless you mean they didn't measure a false positive rate.

Filligree 2mo ago
For watermarked long-form text, it is actually possible. Makes the watermark more fragile, but the math is considerably more forgiving than usual.
- embedding-shape 2mo ago
  > For watermarked long-form text What about "watermarked long-form code"? I'm having a hard time understanding how a model could watermark not prose, but functional/semantic text like code, that actually has meaning. You can't switch our the characters, you can't use various types of whitespace, you can't add arbitrary code comments, and a lot of other restrictions. Is there any state of the art methods for watermarking code without affecting the quality/correctness?
  - wpietri 2mo ago
    As anybody who has put together a coding standard knows, there are a lot of options for individual expression, meaning a lot of room for things like watermarking. And of course you can add arbitrary comments; my Claude-generated code is very verbose.
    - embedding-shape 2mo ago
      > there are a lot of options for individual expression, meaning a lot of room for things like watermarking The way I use LLMs (and I'd advice everyone to do the same) there really isn't, the agent implements things exactly how I want them, or I use the agent to massage it into the exact bit-by-bit version I imagined when I first sent the prompt afterwards. I honestly don't know what the point would be to let the agents compose worse code than what I'd do manually, although I know it's a popular approach taken by many. > And of course you can add arbitrary comments; my Claude-generated code is very verbose. So watermarking for all users who allow code comments from agents, no watermarking for us who force the agents to never write a single code comment? Alright, I'd be fine with that.
      - wpietri 2mo ago
        From what I've seen, your approach to LLMs is exceedingly rare, so I suspect it's one the people who care about watermarking aren't very concerned with. And the reason to let Claude make worse code than a professional would by hand is basically suppressed demand. Since programmers are expensive, previously code mostly got written when a large number of dollars were on the line, or when an individual programmer did something not economically optimum (e.g., hobby project). That left a whole lot of somewhat less valuable software unwritten. It's the economic space that no-code tools have been nibbling on for years. One way to think of things like Claude Code is as effectively no-code tools. Pre-LLM no-code tools would produce data structures that got executed by special environments without ever being seen or tuned by a human. Claude Code can be used just like that, with text as the input and python as the intermediate representation that nobody ever looks at. That approach probably isn't sustainable for what we professional programmers would call a serious project. Claude can easily get in over its head and I expect that its code decays over time, in a fashion similar to how many human teams get in a state where they just have to rewrite everything. But faster, I'd expect. But there are a lot of unserious projects that previously would have never been created. E.g., a quick app to manage your little league team, or a bit of in-house business stuff in the "a little hard to do with a spreadsheet" range.
      - olmo23 2mo ago
        > I honestly don't know what the point would be to let the agents compose worse code than what I'd do manually You never generate throwaway code used to test an external service? or try out an interface idea? There's a lot of code that's only meant to be ran once. I often dont even care what language it's written in.
        embedding-shape 2mo ago
        > You never generate throwaway code used to test an external service? or try out an interface idea? And save/persist it? No, most of any experimental stuff goes into /tmp which gets cleared out on reboot, nothing I care to save in any repository. Or just "show me how this would look like" and then it's only in the session itself (and the logs/state I suppose, technically...).
  - junofan 2mo ago
    [dead]
  - SkyBelow 2mo ago
    For straight generated code it'll likely need more text, but it'll still show up. In cases where one token is extremely likely, it'll randomly be red or green and still be picked in either case as it is simply the best (or only) option. So you'll have more tokens that don't show a pattern either way (half of these cases will match and half won't, just the same as if a human wrote it). Meaning you'll need more instances where multiple tokens were all likely to see if there is a pattern. Given the check algorithm can't identify these cases, it can only judge on the overall text, so the more strict a language, the more the length requirement scales. Where I wonder if this keeps working is in tool calls. Often, you don't take code straight from the llm, you take the results of a tool call to edit already existing code. It might be that the result of this leads to far too few signals to pick up, meaning that this only works when one does significant generation with a single model (even swapping between different models, at least by different companies, breaks this just as much as having a human write parts of the code). Think of it like finding a loaded dice. A dice that has a slight bias in a few dozen roles is just random chance. If that bias continues after hundreds of thousands of roles, the dice is loaded. But will a code base have enough samples, especially when edits made from tool calls? I could see this being unable to detect things at the size of a reasonable PR and only being useful for massive sets of changes and only if the person behind them didn't structure their AI usage to avoid detection.
- OGWhales 2mo ago
  And yet, it remains possible that a human could write the same sequence of characters.
  - no_multitudes 2mo ago
    How often do you add seemingly-random zero-width unicode characters to the text you write?
    - OrangeMusic 2mo ago
      That's not how it works (that would be trivial to erase).

embedding-shape 2mo ago

Read said section yourself perhaps.

mysterydip 2mo ago

If LLM training data is human-written, and LLM output mimics that input, how could you not have false positives?

SkyBelow 2mo ago
Because it won't be in the training directly. It is applied after a model generates its distribution of likely tokens, biasing each token randomly based on a random key and unrelated to any meaning of the words. So half the time, the most likely token becomes more likely and half the time it becomes less likely, and the same for every other token (when temperature is above 0). You then look at the tokens actually picked to see how closely they follow this pattern that isn't connected to the meaning of the tokens. With enough text, you can then analyze the chance of it happening by chance verses being because the generation of the tokens was done using the algorithm, and you can save a positive result until you are arbitrarily sure. There is a chance of a false positive, but the chance of a false positive approaches the chance that the murderer happened to have fingerprints that matched your and both forensics labs happened to have mixed up the dna tests and the eye witness happened to misremember the face and your phone gps happened to glitch out and put you at the murder scene at the time of the crime all happening. It is theoretically possible only in the same sense that quantum teleporting a cat is theoretically possible. The real question is how much text do they need for a given level of certainty and what do they check for. If they flag a positive at a p value <.01, that's a problem. If they can reasonably get a p value of < 1e-12 in only a few paragraphs of text, that is effectively no false positives (but a lot of 'too short to analyze' outcomes).
- ricericerice 2mo ago
  How do you verify in practice then? Wouldn't you need the original prompt so you can reobtain the likely token distribution to validate again the random key(s)?
  - Eisenstein 2mo ago
    A token is hashed and used to seed a random number generator, which produces the red list for the token after it. Paper: * https://arxiv.org/abs/2301.10226 https://arxiv.org/abs/2301.10226

wrsh07 2mo ago

Somewhat trivially, if I ask Claude to transcribe an image and then check if that transcription is ai generated it will likely say yes. Many users are not smart enough to realize that the transcription step is where the ai (watermarks) were necessarily injected.

basch 2mo ago
How is a perfect transcription of an image watermarked?
- FeteCommuniste 2mo ago
  "Perfect as far as human perception can tell" is a weaker standard than "bit-to-bit copy." Maybe it's that?
- wrsh07 2mo ago
  It depends on how it does watermarking!! Note, there are many ways to represent words visually on computers that look identical
  - basch 2mo ago
    If they were substituting glyphs for identical ones people would be able to reverse engineer it. Theres no way that’s what they are doing.

wtfwhateven 2mo ago

Why would you say something so ridiculous?

jobigoud 2mo ago
I think they mean it like this: imagine you ask me a random number sequence. I give you a random number sequence. Little did you know, I used a very specific PRNG to generate it, so later I can prove with certainty that your number was generated by me, and you can't say you came up with it yourself. There is no room for false positive here in the same way you can't randomly find a collision in a hash function if it's strong enough. Like the rate is so infinitesimal that it is effectively zero. Now replace random number sequence with prompted string of words. And instead of using the PRNG on every word I use it every n words. If the generated text is sufficiently long I can tell by matching the expected deterministic pattern. You can defeat it by changing the words yourself and triggering a false negative but there isn't really any room for a false positive if the text is long enough and the pattern matches perfectly. If the pattern doesn't match then I can compute a probability.

bufbupa 2mo ago

Sorry you're getting downvoted, this interpretation doesn't seem that far fetched to me. Here's the strawman: The text-based watermarking is going to be done procedurally instead of generatively. Maybe they add some sequence of zero-width Unicode characters to all generated text at certain intervals. Then, there is effectively no false positive possible (because humans would [effectively] never type such sequences of unicode naturally). It may survive some editing (depending on how you select/edit the characters), and it's possible to be stripped (false negatives).