4 ms·

As far as I can tell, what they're proposing is: Today, for each output token the LLM produces a probability for each possible output, then a 'sampler' makes a

As far as I can tell, what they're proposing is: Today, for each output token the LLM produces a probability for each possible output, then a 'sampler' makes a probability-weighted random choice. If the next-token probabilities are 90% for foo, 9% for bar and 1% for baz then the sampler chooses a random number between 0 and 1, if it's <0.9 it outputs foo, 0.9-0.99 it outputs bar, 0.99-1 it outputs baz. But what if instead of using random numbers, you had a source of evenly distributed random numbers that was deterministic, based on some secret key? Each candidate token would remain just as likely as it was before - there would still be a 90% chance of foo being chosen. So the output shouldn't degrade in quality. And sure, some tokens will have 99.999% probability and their selection doesn't tell you much. But in most real-world use multiple wordings are possible and so on. So across a large enough sample of the output, you could detect whether the sampler was following your secret deterministic pattern. Of course the downside is you've got to check on exactly the same LLM, and only people with the secret key can perform the check. And it's only applicable to closed-source LLMs. I'm also not quite sure if it works when you don't know the exact prompt - so maybe my understanding of the paper is all wrong?

genrilz 2y ago
My understanding from the "Watermark Detection" section is that it only requires the key and the output text in order to do the detection. In particular, it seems like the random seed used for each token is only based off the previous 4 tokens and the LLM specific key, so for any output larger than 4 tokens, you can start to get a signal. I don't think the key actually needs to be secret, as it's not trying to cryptographicly secure anything. So all closed weights LLM providers could just publicly share the keys they use for water marking, and then anybody could use them to check if a particular piece of text was generated by a particular LLM. That being said, I think you are right about this only really being useful for closed weights models. If you have the weights, you can just run an LLM through a standard sampler and it won't be watermarked.
- danielmarkbruce 2y ago
  Why would anyone ever use such a model? And then, given the significant reduction in users, why would any closed model service do this? Seems like a cool theoretical trick that has little practical implication.
  - genrilz 2y ago
    There are situations where the model output being watermarked doesn't matter. For instance, I hear people on HN asking LLMs to explain things to them all the time, (which I think is a bad idea, but YMMV) and people use LLMs to write code quickly. (which I think is at least possibly a good idea) There are also some content farms which churn out low quality books on Amazon on various topics, and I don't think they care if they get caught using LLM outputs. Thus it might reduce usage some, but it certainly wouldn't block all usage. Additionally, there are only a few providers of truly massive LLMs on the market right now. If they decided that doing this would be a social good, or more likely that it would bring bad PR to not do this when their competitors do, then they would at least be able to watermark all of the massive LLM outputs.
    - danielmarkbruce 2y ago
      You say that as though there isn't a choice though. There will always be a good offering who doesn't water mark. And there is no good reason for a provider to watermark - they aren't helping the customer. They'd be helping some other party who isn't paying them. This will never be a thing.
      - alach11 2y ago
        > There will always be a good offering who doesn't water mark There's a possible future where this gets legislated, right? Of course, there are lots of implementation challenges to this and it's probably a bad idea...
        danielmarkbruce 2y ago
        Sure, it's possible. I'd lay 100-1 against though.
- iflp 2y ago
  The seed is computed using the last 4 tokens, but to get the predictive distribution they should still need the full history?
zmgsabst 2y ago
What happens if I tell the LLM to reword every other sentence? — or every 5th word? I must be missing something, because this seems to assume a contiguous output.
- genrilz 2y ago
  It's possible that this might break the method, but what seems most likely to me is that the LLM will simply reword every 5th word with some other word that it is more likely to use due to the watermark sampling. Thus the resulting output would display roughly the same level of "watermarkedness". You might be able to have one LLM output the original, and then another to do a partial rewording though. The resulting text would likely have higher than chance "watermarkedness" for both LLMs, but less than you would expect from a plain output. Perhaps this would be sufficient for short enough outputs?
  - namrog84 2y ago
    What happens when we all reading llm output all the time. Simply start to adapt more to llm writing styles, word choice, and possibly without realizing it watermark our own original writing?
    - genrilz 2y ago
      You might be right, but my first instinct is that this probably wouldn't happen enough to throw off the water marking to badly. The most likely used word is based off the previous four, and only works if there is enough entropy present that one of multiple word would work. Thus its not a simple matter of humans picking up particular word choices. There might be some cases where there are 3 tokens in a row that occur with low entropy after the first token, and then one token generation with high entropy at the end. That would cause a particular 5 word phrase to occur. Otherwise, the word choice would appear pretty random. I don't think humans pick up on stuff like that even subconsciously, but I could be wrong. I would be interested to see if LLMs pick up the watermarks when fed watermarked training data though. Evidently ChatGPT can decode base64, [0] so it seems like these things can pick up on some pretty subtle patterns. [0] https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_chatgpt_can_decode_base64_so_i_asked_it/ https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_...
Buttons840 2y ago
I haven't been in school since LLMs became useful, but if I were to "cheat", I'd ask the LLM for a very fine grained outline, and then just translate the outline into my own words. Then ask the LLM to fill in my citations in AMA format. And honestly, this still retains like 95% of the value of writing a paper, because I did write it, the words did flow through my brain. I just used the LLM to avoid facing a blank page. I've also thought about asking LLMs to simulate a forum conversation about the Civil War (or whatever the topic may be), and include a wrong comment that can be countered by writing exactly what the assignment requires, because I seem to have no trouble writing an essay when duty calls and someone is wrong in the internet.
- qeternity 2y ago
  These things don’t have to be foolproof to be effective deterrents.

genrilz 2y ago

My understanding from the "Watermark Detection" section is that it only requires the key and the output text in order to do the detection. In particular, it seems like the random seed used for each token is only based off the previous 4 tokens and the LLM specific key, so for any output larger than 4 tokens, you can start to get a signal. I don't think the key actually needs to be secret, as it's not trying to cryptographicly secure anything. So all closed weights LLM providers could just publicly share the keys they use for water marking, and then anybody could use them to check if a particular piece of text was generated by a particular LLM. That being said, I think you are right about this only really being useful for closed weights models. If you have the weights, you can just run an LLM through a standard sampler and it won't be watermarked.

danielmarkbruce 2y ago
Why would anyone ever use such a model? And then, given the significant reduction in users, why would any closed model service do this? Seems like a cool theoretical trick that has little practical implication.
- genrilz 2y ago
  There are situations where the model output being watermarked doesn't matter. For instance, I hear people on HN asking LLMs to explain things to them all the time, (which I think is a bad idea, but YMMV) and people use LLMs to write code quickly. (which I think is at least possibly a good idea) There are also some content farms which churn out low quality books on Amazon on various topics, and I don't think they care if they get caught using LLM outputs. Thus it might reduce usage some, but it certainly wouldn't block all usage. Additionally, there are only a few providers of truly massive LLMs on the market right now. If they decided that doing this would be a social good, or more likely that it would bring bad PR to not do this when their competitors do, then they would at least be able to watermark all of the massive LLM outputs.
  - danielmarkbruce 2y ago
    You say that as though there isn't a choice though. There will always be a good offering who doesn't water mark. And there is no good reason for a provider to watermark - they aren't helping the customer. They'd be helping some other party who isn't paying them. This will never be a thing.
    - alach11 2y ago
      > There will always be a good offering who doesn't water mark There's a possible future where this gets legislated, right? Of course, there are lots of implementation challenges to this and it's probably a bad idea...
      - danielmarkbruce 2y ago
        Sure, it's possible. I'd lay 100-1 against though.
iflp 2y ago
The seed is computed using the last 4 tokens, but to get the predictive distribution they should still need the full history?

zmgsabst 2y ago

What happens if I tell the LLM to reword every other sentence? — or every 5th word? I must be missing something, because this seems to assume a contiguous output.

genrilz 2y ago
It's possible that this might break the method, but what seems most likely to me is that the LLM will simply reword every 5th word with some other word that it is more likely to use due to the watermark sampling. Thus the resulting output would display roughly the same level of "watermarkedness". You might be able to have one LLM output the original, and then another to do a partial rewording though. The resulting text would likely have higher than chance "watermarkedness" for both LLMs, but less than you would expect from a plain output. Perhaps this would be sufficient for short enough outputs?
- namrog84 2y ago
  What happens when we all reading llm output all the time. Simply start to adapt more to llm writing styles, word choice, and possibly without realizing it watermark our own original writing?
  - genrilz 2y ago
    You might be right, but my first instinct is that this probably wouldn't happen enough to throw off the water marking to badly. The most likely used word is based off the previous four, and only works if there is enough entropy present that one of multiple word would work. Thus its not a simple matter of humans picking up particular word choices. There might be some cases where there are 3 tokens in a row that occur with low entropy after the first token, and then one token generation with high entropy at the end. That would cause a particular 5 word phrase to occur. Otherwise, the word choice would appear pretty random. I don't think humans pick up on stuff like that even subconsciously, but I could be wrong. I would be interested to see if LLMs pick up the watermarks when fed watermarked training data though. Evidently ChatGPT can decode base64, [0] so it seems like these things can pick up on some pretty subtle patterns. [0] https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_chatgpt_can_decode_base64_so_i_asked_it/ https://www.reddit.com/r/ChatGPT/comments/1645n6i/i_noticed_...

Buttons840 2y ago

I haven't been in school since LLMs became useful, but if I were to "cheat", I'd ask the LLM for a very fine grained outline, and then just translate the outline into my own words. Then ask the LLM to fill in my citations in AMA format. And honestly, this still retains like 95% of the value of writing a paper, because I did write it, the words did flow through my brain. I just used the LLM to avoid facing a blank page. I've also thought about asking LLMs to simulate a forum conversation about the Civil War (or whatever the topic may be), and include a wrong comment that can be countered by writing exactly what the assignment requires, because I seem to have no trouble writing an essay when duty calls and someone is wrong in the internet.

qeternity 2y ago
These things don’t have to be foolproof to be effective deterrents.