7 ms·

DeepMind debuts watermarks for AI-generated text

131 points by ambigious7777 2y ago

FilipSivak 2y ago
How is this supposed to work? By inserting special unicode characters? How can you watermark text?
- hiatus 2y ago
  You can insert known spelling errors, choose certain phrasings, and more. It doesn't have to be new characters added to the text. Government security services have done stuff like this for decades to weed out moles.
  - luigibosco 2y ago
    moles should know better than to utilize mountweazels! https://en.wikipedia.org/wiki/Fictitious_entry https://en.wikipedia.org/wiki/Fictitious_entry
- zorked 2y ago
  We've been studying unintentional watermarks for years. https://en.wikipedia.org/wiki/Stylometry https://en.wikipedia.org/wiki/Stylometry
- das_keyboard 2y ago
  > SynthID-Text works by discreetly interfering in the generation process: It alters some of the words that a chatbot outputs to the user in a way that’s invisible to humans but clear to a SynthID detector. “Such modifications introduce a statistical signature into the generated text,” [...] “During the watermark detection phase, the signature can be measured to determine whether the text was indeed generated by the watermarked LLM.”
- a2128 2y ago
  I haven't read how Google is doing it, but one way it could be done is to nudge which tokens get sampled. For example, every other token could have an odd numbered id (where each token is assigned an id from 0 to 32000 or however many it has). Then in order to detect the watermark you just tokenize the text and see if the pattern is there. A problem with this approach is that it harms the accuracy and coherency, for example if you ask "What is 2+2", and the token "4" is token #102, and it has to pick an odd-numbered token, then it may respond with a wrong answer or yap on strangely due to its limited selection of tokens (like "The accurate answer to your mathematical query is the number Four")
- sebstefan 2y ago
  There's an article from ieee that explains it: https://spectrum.ieee.org/watermark#:~:text=How%20Google%E2%80%99s%20Text%20Watermarks%20Work https://spectrum.ieee.org/watermark#:~:text=How%20Google%E2%...
- sumtechguy 2y ago
  You do not even need extra characters (although they help). You can use spaces, missing punctuation, upper/lower case in particular cases, conjunction usage and not using it, word substitution, common misspellings, transposed letters, etc. How many extra spaces/tabs can you add to the end of a paragraph? At the beginning? Between sentences? Inside them? Then you have an AI agent design it and then train another one to detect it.
- voidUpdate 2y ago
  As stated in the article, it alters the probabilities that the network produces in a predictable way so that a different (but still correct-sounding) word is picked. It subtly alters the wording from what it would have output normally in such a way that you can detect it, while still sounding correct to the user
ksaj 2y ago
Some of the watermarking is really obvious. If you write song lyrics in ChatGPT, watch for phrases like "come what may" and "I stand tall." It's not just that they are (somewhat) unusual phrases, it's that ChatGPT comes up with those phrases so very often. It's quite like how earlier versions always had a "However" in between explanations.
- GaggiX 2y ago
  ChatGPT does not have a watermark.
  - jgalt212 2y ago
    I suggest we "delve" deeper int this problem.
  - aleph_minus_one 2y ago
    What makes you sure about that?
  - sunaookami 2y ago
    It has a rich tapestry of watermarks.
    - becquerel 2y ago
      A delicate dance of them, perhaps
      - mondobe 2y ago
        It's like a symphony of watermarks, all playing in harmony.
- fkyoureadthedoc 2y ago
  Coheed and Cambria were using ChatGPT this whole damn time, smh
- ksaj 2y ago
tokioyoyo 2y ago
Correct me if I’m wrong, but wouldn’t it simply drive people to use LLMs that are not watermarking their content?
- ndr 2y ago
  Agreed, that's the obvious prediction. They're also going to perform worse on 3p benchmarks right?
- aleph_minus_one 2y ago
  I think your idea is basically right, but there are two points to consider: - Your hypothesis only holds if the alternative LLM is also "sufficiently good". If Gemini does not stay competitive with other LLMs, Google's AI plans have a much more serious problem. - Your hypothesis assumes that many people will be capable of detecting the watermarks (both of Gemini and other LLMs) so that they can make a conscious choice for another LLM. But the idea behind good watermarking is that it is not that easy to detect.
- kranner 2y ago
  According to the article, you can just have another LLM summarise Gemini's watermarked output and that will "likely" defeat the watermark detection.
  - scarmig 2y ago
    But, if all the good models can only be trained by large mega corps with close connections to the government, it's only a matter of time until that other LLM will just add its own watermark.
    - throwaway313373 2y ago
      You could probably use much smaller model for such rewriting
- onion2k 2y ago
  People use Google Search despite it being littered with adverts and tracking. Maybe Google are counting on either being better than the competition despite watermarking, or simply accepting that people who don't care are enough of a market that it's still worth adding.
mateus1 2y ago
Google is branding this in a positive light but this is just AI text DRM.
- fastball 2y ago
  I for one am glad we might have a path forward to filtering out LLM-generated sludge.
  - pyrale 2y ago
    > we If by "we" you mean anyone else than Google and the select few other LLM provider they choose to associate with, I'm afraid you're going to be disappointed.
    - fastball 2y ago
      If there is a detectable fingerprint, we can detect it too. Probably don't even need a Bletchley Park.
- sebstefan 2y ago
  It's likely more about preventing model incest than digital rights management
- gwbas1c 2y ago
  Like all things a computer can / can't do; DRM isn't inherently bad: It's how its used that's a problem. IE, DRM can't change peoples' motivations. It's useful for things like national security secrets and trade secrets, where the people who have access to the information have very clear motivations to protect that information, and very clear consequences for violating the rules that DRM is in place to protect. In this case, the big question of if AI watermarking will work / fail has more to do with peoples' motivations: Will the general public accept AI watermarking because it fits our motivations and the consequences we set up for AI masquerading as a real person, or AI being used for misinformation? That's a big question that I can't answer.
  - 2y ago
playingalong 2y ago
> It has also open-sourced the tool and made it available to developers and businesses, allowing them to use the tool to determine whether text outputs have come from their own large language models (LLMs), the AI systems that power chatbots. However, only Google and those developers currently have access to the detector that checks for the watermark. These two sentences next to each other don't make much sense. Or are misleading. Yeah. I know. Only the client is open source and it calls home.
- falcor84 2y ago
  Is there significant throttling to prevent us from training a classification model against it?
namanyayg 2y ago
"An LLM generates text one token at a time. These tokens can represent a single character, word or part of a phrase. To create a sequence of coherent text, the model predicts the next most likely token to generate. These predictions are based on the preceding words and the probability scores assigned to each potential token. For example, with the phrase “My favorite tropical fruits are __.” The LLM might start completing the sentence with the tokens “mango,” “lychee,” “papaya,” or “durian,” and each token is given a probability score. When there’s a range of different tokens to choose from, SynthID can adjust the probability score of each predicted token, in cases where it won’t compromise the quality, accuracy and creativity of the output. This process is repeated throughout the generated text, so a single sentence might contain ten or more adjusted probability scores, and a page could contain hundreds. The final pattern of scores for both the model’s word choices combined with the adjusted probability scores are considered the watermark. This technique can be used for as few as three sentences. And as the text increases in length, SynthID’s robustness and accuracy increases." Better link: https://deepmind.google/technologies/synthid/ https://deepmind.google/technologies/synthid/
- bgro 2y ago
  Couldn’t this be easily disrupted as a watermark system by simply changing the words to interfere with the relative checksum? I suspect sentence structure is also being used or, more likely, the primary “watermark”. Similar to how you can easily identify if something is at least NOT a Yoda quote based on it having incorrect structure. Combine that with other negative patterns like the quote containing Harry Potter references instead of Star Wars, and you can start to build up a profile of trends like this statement. By rewriting the sentence structure and altering usual wording instead of directly copying the raw output, it seems like you could defeat any current raw watermarking. Though this hasn’t stopped Google and others in the past using bad science and stats to make unhinged entitled claims like when they added captcha problems everybody said would be “literally impossible“ for bots to solve. What a surprise how trivial they were to automate and the data they produce can be sold for profit at the expense of mass consumer time.
  - scarmig 2y ago
rany_ 2y ago
I really want to be able to try Gemini without the AI watermark. IIRC they've used SynthID from the start and it makes me wonder if it's the source of all of Gemini's issues. Obviously Google claims that it doesn't cause any issues but I'd think that OpenAI and other competitors would have something similar to SynthID if it didn't impact performance.
- throwaway314155 2y ago
  > IIRC they've used SynthID from the start Is that not at odds with what's presented in the article here?
  - rany_ 2y ago
    It's not. They've mentioned that they had SynthID integrated before (I'm almost certain it was from the very start). What changed is that tools to detect that something is from Google's LLM is public now. > Google has already integrated this new watermarking system into its Gemini chatbot, the company announced today. Key word: already > It has also open-sourced the tool and made it available to developers and businesses, allowing them to use the tool to determine whether text outputs have come from their own large language models (LLMs), the AI systems that power chatbots. That's basically the change.
js8 2y ago
I think people are already doing that. I frequently hear people watermarking their speeches with phrases like "are we aligned on this?", or "let's circle back" and similar.
- lcnPylGDnU4H9OF 2y ago
  I can’t tell if this is satire but that’s just corp-speak. I imagine those people also occasionally suggest “touching base” and “taking this offline”. The phrases usually mean something useful, if one knows the meaning, but it is amusing how much people seem to stick with the same ones, even across companies.
  - js8 2y ago
    I am not sure whether it was satire. I personally don't like corp speak - it feels like people talking like that are not humans. I am not sure I would welcome our AI overlords speaking like this, either. But I find the idea that people will subconsciously start copying AI speech patterns (perhaps as a signal of submission) amusing. I think it's gonna throw a wrench into the idea. IMHO LLMs either should help us communicate more clearly and succinctly, or we can use them as tools for creativity ("rephrase this in 18th century English"). Watermarking speech sabotages both of these use cases.
playingalong 2y ago
> the team tested it on 20 million prompts given to Gemini. Half of those prompts were routed to the SynthID-Text system and got a watermarked response, while the other half got the standard Gemini response. Judging by the “thumbs up” and “thumbs down” feedback from users, the watermarked responses were just as satisfactory to users as the standard ones. Three comments here: 1. I wonder how many of the 20M prompts got a thumbs up or down. I don't think people click that a lot. Unless the UI enforces it. I haven't used Gemini, so I might be unaware. 2. Judging a single response might be not enough to tell if watermarking is acceptable or not. For instance, imagine the watermarking is adding "However," to the start of each paragraph. In a single GPT interaction you might not notice it. Once you get 3 or 4 responses it might stand out. 3. Since when Google is happy with measuring by self declared satisfaction? Aren't they the kings of A/B testing and high volume analysis of usage behavior?
- varispeed 2y ago
  > I don't think people click that a lot. I sometimes do, but I almost always give wrong answer or opposite answer where possible.
  - froh 2y ago
    but why? what for?
    - thebruce87m 2y ago
      My timesheet SAAS constantly asks for feedback, which I give 0/10 as constantly asking for feedback really annoys me. They then contact me and ask me why, so I tell them then they say there is nothing they can do. A week later I’ll get a pop up asking for feedback and we go round the same loop again.
    - varispeed 2y ago
      Because companies like Google are a cancer and I don't want to give them data they didn't pay for.
bko 2y ago
This article goes into it a little bit, but an interview with Scott Aaronson goes into some detail about how watermarking works[0]. He's a theoretical computer scientist but he was recruited by OpenAI to work on AI safety. He has a very practical view on the matter and is focusing his efforts on leveraging the probabilistic nature of LLMs to provide a digital undetectable watermark. So it nudges certain words to be paired together slightly more than random and you can mathematically derive with some level of certainty whether an output or even a section of an output was generated by the LLM. It's really clever and apparently he has a working prototype in development. Some work arounds he hasn't figured out yet is asking for an output in language X and then translating it into language Y. But those may still be eventually figured out. I think watermarking would be a big step forward to practical AI safety and ideally this method would be adopted by all major LLMs. That part starts around 1 hour 25 min in. > Scott Aaronson: Exactly. In fact, we have a pseudorandom function that maps the N-gram to, let’s say, a real number from zero to one. Let’s say we call that real number ri for each possible choice i of the next token. And then let’s say that GPT has told us that the ith token should be chosen with probability pi. https://axrp.net/episode/2023/04/11/episode-20-reform-ai-alignment-scott-aaronson.html https://axrp.net/episode/2023/04/11/episode-20-reform-ai-ali...
- littlestymaar 2y ago
  Sounds interesting, but it also sounds like something that could very well be circumvented by using a technique similar to speculative decoding: you use the censored model like you'd use the fast llm in speculative decoding, and you check whether the other model agrees with it or not. But instead of correcting the token every time both models disagree like you'd do with speculative decoding, you just need to change it often enough to mess with the watermark detection function (maybe you'd change every other mismatched token, or maybe one every 5 tokens would be enough to reduce the signal-to-noise ratio below the detection threshold). You wouldn't even need to have access to an unwatermarked model, the “correcting model” could even be watermaked itself as long as it's not the same watermarking function applied to both. Or am I misunderstanding something?
  - jkhdigital 2y ago
tiffanyh 2y ago
OT: The publication (Spectrum by IEEE) has some really good content. It's starting to become a common destination for when I want to read about interesting things.
samatman 2y ago
This is information-theoretically guaranteed to make LLM output worse. My reasoning is simple: the only way to watermark text is to inject some relatively low-entropy signal into it, which can be detected later. This has to a) work for "all" output for some values of all, and b) have a low false positive rate on the detection side. The amount of signal involved cannot be subtle, for this reason. That signal has a subtractive effect on the predictive-output signal. The entropy of the output is fixed by the entropy of natural language, so this is a zero-sum game: the watermark signal will remove fidelity from the predictive output. This is impossible to avoid or fix.
- thornewolf 2y ago
  you are correct of we suppose we are at a global optimum. however, consider this example: i have two hands i have 2 hands these sentences communicate the same thing but one could be a watermarked result. we can apply this equivalent meaning word/phrase change many times over and be confident something is watermark while having avoided any semantic shifts.
- jkhdigital 2y ago
  You're not wrong, but natural language has a lot of stylistic "noise" which can be utilized as a subliminal channel without noticeably degrading the semantic signal.
espadrine 2y ago
The academic paper: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 They use the last N prefix tokens, hash them (with a keyed hash), and use the random value to sample the next token by doing an 8-wise tournament, by assigning random bits to each of the top 8 preferred tokens, making pairwise comparisons, and keeping the token with a larger bit. (Yes, it seems complicated, but apparently it increases the watermarking accuracy compared to a straightforward nucleus9 sampling.) The negative of this approach is that you need to rerun the LLM, so you must keep all versions of all LLMs that you trained, forever.
- mmoskal 2y ago
  They actually run 2^30-way tournament (they derive an equivalent form that doesn't requires 2B operations). You do not need to run the LLM, it only depends on the tokenizer.
  - espadrine 2y ago
    You’re right. I understood it to require taking the top 2^30 tokens, but instead they sample 2^30 times with replacement. Too bad they only formulate the detection positive rate empirically. I am curious what the exact probability would be mathematically.
- jkhdigital 2y ago
  Why do you need to rerun the LLM? Watermark detection only requires the hash functions (equation (1) from the paper).
blintz 2y ago
These watermarks are not robust to paraphrasing attacks: AUC ROC falls from 0.95 to 0.55 (barely better than guessing) for a 100 token passage. The existing impossibility results imply that these attacks are essentially unavoidable (https://arxiv.org/abs/2311.04378 https://arxiv.org/abs/2311.04378) and not very costly, so this line of inquiry into LLM watermarking seems like a dead end.
- TibbityFlanders 2y ago
  [dead]
- jkhdigital 2y ago
  I spent the last five years doing PhD research into steganography, with a particular focus on how to embed messages into LLM outputs. Watermarking is basically one-bit steganography. The first serious investigations into "secure" steganography were about 30 years ago and it was clearly a dead end even back then. Sure, watermarking might be effective against lazy adversaries--college students, job applicants, etc.--but can be trivially defeated otherwise. All this time I'd been lamenting my research area as unpopular and boring when I should've been submitting to Nature!
  - impossiblefork 2y ago
    Though, surely secure steganography with LLMs should be quite easy? Presumably there are things like key exchanges that look like randomness, and then you could choose LLM output using that randomness in such a way that you can send messages that look like an LLM conversation? Someone starts the conversation with a real message 'Hello!' and then you do some kind of key exchange where what is exchanged is hard to distinguish from randomness, and use those keys to select the probabilities of the coming tokens from the LLM. Then once they is established you use some kind of cipher to generate random-looking ciphertext and use that as the randomness used to select words in the final bit? Surely that would work? If there is guaranteed insecurity, it's for things like watermarking, not for steganography?
- sbszllr 2y ago
  I’ve been working in the space since 2018. Watermarking and fingerprinting (of models themselves and outputs) are useful tools but they have a weak adversary model. Yet, it doesn’t stop companies from making claims like these, and what’s worse, people buying into them.
cowmix 2y ago
"I hope this message finds you well." --- busted!
ajwin 2y ago
Do LLM's always pick the most probable next word? I would have thought this would lead to having the same output for every input? How does this deal with the randomness that you get from prompting the same thing over and over?
- 8note 2y ago
  There is at least a parameter called Temperature which decides how much randomness to include in the output. It doesn't get you perfectly deterministic output to set it to 0 though, per https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... as you don't have perfect control over what approximations are being made on your floating point operations
  - mmoskal 2y ago
    The most typical reason argmax (temp 0) is non-deterministic is that your request is running batched with other people requests. The number and size of these affects the matrix sizes and thus tiling decisions. Then you get different floating point order and thus different results. Nvidia gives some guarantees about deterministic results of their kernels but that only applies when you have exact same input data and this is not the case when in-flight batching.
- janalsncm 2y ago
  It depends. If we use beam search we pick the most likely sequence of tokens rather than the most likely token at each point in time. This process is deterministic though. We can also sample from the distribution, which introduces randomness. Basically, if word1 should be chosen 75% of the time and word2 25% of the time, it will do that. The randomness you’re seeing can also be due to implementation details. https://community.openai.com/t/a-question-on-determinism/8185/9 https://community.openai.com/t/a-question-on-determinism/818...
lowbloodsugar 2y ago
I want AI to use just the right word when it’s writing for me. If it’s going to nerf itself to not choose the perfect word so it can be watermarked, then why would I use that product? I’ll go somewhere else. And if it does use just the right word, then how is that different from a great human writer?
- Nasrudith 2y ago
  There is the 'loser's litigation' method of getting all of your non-watermarked competitors banned. Usually involving some combination of magical rights removing brain-hacks like national security or 'the children'.
fny 2y ago
I think we just need to give up on this. What’s the harm? It’s not like some ground truth is fabricated. I’m far, far more concerned about photo, video, and audio verification. We need a camera that can guarantee a recording is real.
- ziofill 2y ago
  I've been thinking about this for a while. Digital signatures can guarantee that a piece of data is authentic, if the author wishes to sign it.
- foxglacier 2y ago
  Why do we need that for photo, video and audio? If it's about the general public believing something false, they're not going to check the watermarks of random internet content or trust anyone who says they checked it. If they really want to know, they can go to the source and if they trust that person or organization, they can also trust the content they published. If it's about use in court, we already have a system for that - the person who recorded it appears in court as a witness and promises that they didn't alter it then if it turns out they did, they can go to prison.
harimau777 2y ago
This strikes me as potentially a bad thing for regular people. For example, corporations call still use AI filtering to force job seekers to jump through hoops but job seekers won't be able to use AI to generate the cover letters and resumes that those hoops demand.
matthewmorgan 2y ago
Who is going to pay for watermarked output?
nprateem 2y ago
Google are obviously pushing this as a way to root out AI blog spam. If only they can get other providers to use it because of 'safety' or something they won't have to change their indexer much. Otherwise page rank is dead due to the ease of creating content farms.
- riffraff 2y ago
  Not just them, openai is doing the same for the same reason: they need to avoid an Habsburg ai issue when the next half of their training material will be generated by themselves
ruuda 2y ago
Some comments here point at impossibility results, but after screening hundreds of job applications at work, it's not hard to pick out the LLM writing, even without watermark. My internal LLM detector is now so sensitive that I can tell when my confirmed-human colleagues used an LLM to rephrase something when it's longer than just one sentence. The writing style is just so different. Maybe if you prompt it right, it can do a better job of masking itself, but people don't seem to do that.
- auggierose 2y ago
  So, how many times did you actually get the confirmation that an LLM has/has not been used? My guess is zero times. So, you are not describing an experiment here, you are just describing how you built up your internal bias.
  - tessierashpool9 2y ago
    this is an important realization!
  - kuhewa 2y ago
    Probably not entirely fair. e.g. After enough sentences it is trivially easy to identify LLM output. So you repeatedly get the opportunity to test a sentence or two, guess the provenance and then realise it is the first sentence in several paragraphs of generated output.
    - exe34 2y ago
      without external validation, you still just have a gut feeling. it doesn't matter if it's the first or thousandth. I can tell if somebody ate cornflakes or oatmeal for breakfast just by looking at how they walk. I'm always right. you better believe me. I've seen thousands of people walk.
      - kuhewa 2y ago
        If the cornflakes had been sitting in old milk for a week, then yes, I would believe you could tell by how they walk, either hunched over with a cramping stomach or with shit pants. LLM text is quite similar.
villmann 2y ago
To what degree will AI-generated text and watermarking influence how human language evolves... I bet "delve" will become more frequent in the spoken language :)
tomxor 2y ago
> Such modifications introduce a statistical signature into the generated text, Great so now people have to be worried about being too statistically similar to an arbitrary "watermark".
sharpshadow 2y ago
To archive the watermark they store every output which they create and let partners check against it. That’s how I understand the article. Then they also store everything which the partners upload to check if it’s created by them. If other AI players also would store everything they create and make it available in a similar way there could be indeed some working watermark. If one would use a private run AI to change the public run AI generated content to alter it there still would be a percentage similarity recognisable to hint that it might come from one of the public AIs. Timestamps would become quite relevant since much content would start to repeat itself at some point and the answers generated might be similar.
matteoraso 2y ago
By design, a watermark would make it easy to create a discriminator that distinguishes between LLM content and human content. In that case, just make a discriminator yourself and use regex to find and remove any of the watermarks.

131 points by ambigious7777 2y ago

FilipSivak 2y ago

How is this supposed to work? By inserting special unicode characters? How can you watermark text?

hiatus 2y ago
You can insert known spelling errors, choose certain phrasings, and more. It doesn't have to be new characters added to the text. Government security services have done stuff like this for decades to weed out moles.
- luigibosco 2y ago
  moles should know better than to utilize mountweazels! https://en.wikipedia.org/wiki/Fictitious_entry https://en.wikipedia.org/wiki/Fictitious_entry
zorked 2y ago
We've been studying unintentional watermarks for years. https://en.wikipedia.org/wiki/Stylometry https://en.wikipedia.org/wiki/Stylometry
das_keyboard 2y ago
> SynthID-Text works by discreetly interfering in the generation process: It alters some of the words that a chatbot outputs to the user in a way that’s invisible to humans but clear to a SynthID detector. “Such modifications introduce a statistical signature into the generated text,” [...] “During the watermark detection phase, the signature can be measured to determine whether the text was indeed generated by the watermarked LLM.”
a2128 2y ago
I haven't read how Google is doing it, but one way it could be done is to nudge which tokens get sampled. For example, every other token could have an odd numbered id (where each token is assigned an id from 0 to 32000 or however many it has). Then in order to detect the watermark you just tokenize the text and see if the pattern is there. A problem with this approach is that it harms the accuracy and coherency, for example if you ask "What is 2+2", and the token "4" is token #102, and it has to pick an odd-numbered token, then it may respond with a wrong answer or yap on strangely due to its limited selection of tokens (like "The accurate answer to your mathematical query is the number Four")
sebstefan 2y ago
There's an article from ieee that explains it: https://spectrum.ieee.org/watermark#:~:text=How%20Google%E2%80%99s%20Text%20Watermarks%20Work https://spectrum.ieee.org/watermark#:~:text=How%20Google%E2%...
sumtechguy 2y ago
You do not even need extra characters (although they help). You can use spaces, missing punctuation, upper/lower case in particular cases, conjunction usage and not using it, word substitution, common misspellings, transposed letters, etc. How many extra spaces/tabs can you add to the end of a paragraph? At the beginning? Between sentences? Inside them? Then you have an AI agent design it and then train another one to detect it.
voidUpdate 2y ago
As stated in the article, it alters the probabilities that the network produces in a predictable way so that a different (but still correct-sounding) word is picked. It subtly alters the wording from what it would have output normally in such a way that you can detect it, while still sounding correct to the user

ksaj 2y ago

Some of the watermarking is really obvious. If you write song lyrics in ChatGPT, watch for phrases like "come what may" and "I stand tall." It's not just that they are (somewhat) unusual phrases, it's that ChatGPT comes up with those phrases so very often. It's quite like how earlier versions always had a "However" in between explanations.

GaggiX 2y ago
ChatGPT does not have a watermark.
- jgalt212 2y ago
  I suggest we "delve" deeper int this problem.
- aleph_minus_one 2y ago
  What makes you sure about that?
- sunaookami 2y ago
  It has a rich tapestry of watermarks.
  - becquerel 2y ago
    A delicate dance of them, perhaps
    - mondobe 2y ago
      It's like a symphony of watermarks, all playing in harmony.
fkyoureadthedoc 2y ago
Coheed and Cambria were using ChatGPT this whole damn time, smh
ksaj 2y ago

tokioyoyo 2y ago

Correct me if I’m wrong, but wouldn’t it simply drive people to use LLMs that are not watermarking their content?

ndr 2y ago
Agreed, that's the obvious prediction. They're also going to perform worse on 3p benchmarks right?
aleph_minus_one 2y ago
I think your idea is basically right, but there are two points to consider: - Your hypothesis only holds if the alternative LLM is also "sufficiently good". If Gemini does not stay competitive with other LLMs, Google's AI plans have a much more serious problem. - Your hypothesis assumes that many people will be capable of detecting the watermarks (both of Gemini and other LLMs) so that they can make a conscious choice for another LLM. But the idea behind good watermarking is that it is not that easy to detect.
kranner 2y ago
According to the article, you can just have another LLM summarise Gemini's watermarked output and that will "likely" defeat the watermark detection.
- scarmig 2y ago
  But, if all the good models can only be trained by large mega corps with close connections to the government, it's only a matter of time until that other LLM will just add its own watermark.
  - throwaway313373 2y ago
    You could probably use much smaller model for such rewriting
onion2k 2y ago
People use Google Search despite it being littered with adverts and tracking. Maybe Google are counting on either being better than the competition despite watermarking, or simply accepting that people who don't care are enough of a market that it's still worth adding.

mateus1 2y ago

Google is branding this in a positive light but this is just AI text DRM.

fastball 2y ago
I for one am glad we might have a path forward to filtering out LLM-generated sludge.
- pyrale 2y ago
  > we If by "we" you mean anyone else than Google and the select few other LLM provider they choose to associate with, I'm afraid you're going to be disappointed.
  - fastball 2y ago
    If there is a detectable fingerprint, we can detect it too. Probably don't even need a Bletchley Park.
sebstefan 2y ago
It's likely more about preventing model incest than digital rights management
gwbas1c 2y ago
Like all things a computer can / can't do; DRM isn't inherently bad: It's how its used that's a problem. IE, DRM can't change peoples' motivations. It's useful for things like national security secrets and trade secrets, where the people who have access to the information have very clear motivations to protect that information, and very clear consequences for violating the rules that DRM is in place to protect. In this case, the big question of if AI watermarking will work / fail has more to do with peoples' motivations: Will the general public accept AI watermarking because it fits our motivations and the consequences we set up for AI masquerading as a real person, or AI being used for misinformation? That's a big question that I can't answer.
- 2y ago

playingalong 2y ago

> It has also open-sourced the tool and made it available to developers and businesses, allowing them to use the tool to determine whether text outputs have come from their own large language models (LLMs), the AI systems that power chatbots. However, only Google and those developers currently have access to the detector that checks for the watermark. These two sentences next to each other don't make much sense. Or are misleading. Yeah. I know. Only the client is open source and it calls home.

falcor84 2y ago
Is there significant throttling to prevent us from training a classification model against it?

namanyayg 2y ago

"An LLM generates text one token at a time. These tokens can represent a single character, word or part of a phrase. To create a sequence of coherent text, the model predicts the next most likely token to generate. These predictions are based on the preceding words and the probability scores assigned to each potential token. For example, with the phrase “My favorite tropical fruits are __.” The LLM might start completing the sentence with the tokens “mango,” “lychee,” “papaya,” or “durian,” and each token is given a probability score. When there’s a range of different tokens to choose from, SynthID can adjust the probability score of each predicted token, in cases where it won’t compromise the quality, accuracy and creativity of the output. This process is repeated throughout the generated text, so a single sentence might contain ten or more adjusted probability scores, and a page could contain hundreds. The final pattern of scores for both the model’s word choices combined with the adjusted probability scores are considered the watermark. This technique can be used for as few as three sentences. And as the text increases in length, SynthID’s robustness and accuracy increases." Better link: https://deepmind.google/technologies/synthid/ https://deepmind.google/technologies/synthid/

bgro 2y ago
Couldn’t this be easily disrupted as a watermark system by simply changing the words to interfere with the relative checksum? I suspect sentence structure is also being used or, more likely, the primary “watermark”. Similar to how you can easily identify if something is at least NOT a Yoda quote based on it having incorrect structure. Combine that with other negative patterns like the quote containing Harry Potter references instead of Star Wars, and you can start to build up a profile of trends like this statement. By rewriting the sentence structure and altering usual wording instead of directly copying the raw output, it seems like you could defeat any current raw watermarking. Though this hasn’t stopped Google and others in the past using bad science and stats to make unhinged entitled claims like when they added captcha problems everybody said would be “literally impossible“ for bots to solve. What a surprise how trivial they were to automate and the data they produce can be sold for profit at the expense of mass consumer time.
- scarmig 2y ago

rany_ 2y ago

I really want to be able to try Gemini without the AI watermark. IIRC they've used SynthID from the start and it makes me wonder if it's the source of all of Gemini's issues. Obviously Google claims that it doesn't cause any issues but I'd think that OpenAI and other competitors would have something similar to SynthID if it didn't impact performance.

throwaway314155 2y ago
> IIRC they've used SynthID from the start Is that not at odds with what's presented in the article here?
- rany_ 2y ago
  It's not. They've mentioned that they had SynthID integrated before (I'm almost certain it was from the very start). What changed is that tools to detect that something is from Google's LLM is public now. > Google has already integrated this new watermarking system into its Gemini chatbot, the company announced today. Key word: already > It has also open-sourced the tool and made it available to developers and businesses, allowing them to use the tool to determine whether text outputs have come from their own large language models (LLMs), the AI systems that power chatbots. That's basically the change.

js8 2y ago

I think people are already doing that. I frequently hear people watermarking their speeches with phrases like "are we aligned on this?", or "let's circle back" and similar.

lcnPylGDnU4H9OF 2y ago
I can’t tell if this is satire but that’s just corp-speak. I imagine those people also occasionally suggest “touching base” and “taking this offline”. The phrases usually mean something useful, if one knows the meaning, but it is amusing how much people seem to stick with the same ones, even across companies.
- js8 2y ago
  I am not sure whether it was satire. I personally don't like corp speak - it feels like people talking like that are not humans. I am not sure I would welcome our AI overlords speaking like this, either. But I find the idea that people will subconsciously start copying AI speech patterns (perhaps as a signal of submission) amusing. I think it's gonna throw a wrench into the idea. IMHO LLMs either should help us communicate more clearly and succinctly, or we can use them as tools for creativity ("rephrase this in 18th century English"). Watermarking speech sabotages both of these use cases.

playingalong 2y ago

> the team tested it on 20 million prompts given to Gemini. Half of those prompts were routed to the SynthID-Text system and got a watermarked response, while the other half got the standard Gemini response. Judging by the “thumbs up” and “thumbs down” feedback from users, the watermarked responses were just as satisfactory to users as the standard ones. Three comments here: 1. I wonder how many of the 20M prompts got a thumbs up or down. I don't think people click that a lot. Unless the UI enforces it. I haven't used Gemini, so I might be unaware. 2. Judging a single response might be not enough to tell if watermarking is acceptable or not. For instance, imagine the watermarking is adding "However," to the start of each paragraph. In a single GPT interaction you might not notice it. Once you get 3 or 4 responses it might stand out. 3. Since when Google is happy with measuring by self declared satisfaction? Aren't they the kings of A/B testing and high volume analysis of usage behavior?

varispeed 2y ago
> I don't think people click that a lot. I sometimes do, but I almost always give wrong answer or opposite answer where possible.
- froh 2y ago
  but why? what for?
  - thebruce87m 2y ago
    My timesheet SAAS constantly asks for feedback, which I give 0/10 as constantly asking for feedback really annoys me. They then contact me and ask me why, so I tell them then they say there is nothing they can do. A week later I’ll get a pop up asking for feedback and we go round the same loop again.
  - varispeed 2y ago
    Because companies like Google are a cancer and I don't want to give them data they didn't pay for.

bko 2y ago

This article goes into it a little bit, but an interview with Scott Aaronson goes into some detail about how watermarking works[0]. He's a theoretical computer scientist but he was recruited by OpenAI to work on AI safety. He has a very practical view on the matter and is focusing his efforts on leveraging the probabilistic nature of LLMs to provide a digital undetectable watermark. So it nudges certain words to be paired together slightly more than random and you can mathematically derive with some level of certainty whether an output or even a section of an output was generated by the LLM. It's really clever and apparently he has a working prototype in development. Some work arounds he hasn't figured out yet is asking for an output in language X and then translating it into language Y. But those may still be eventually figured out. I think watermarking would be a big step forward to practical AI safety and ideally this method would be adopted by all major LLMs. That part starts around 1 hour 25 min in. > Scott Aaronson: Exactly. In fact, we have a pseudorandom function that maps the N-gram to, let’s say, a real number from zero to one. Let’s say we call that real number ri for each possible choice i of the next token. And then let’s say that GPT has told us that the ith token should be chosen with probability pi. https://axrp.net/episode/2023/04/11/episode-20-reform-ai-alignment-scott-aaronson.html https://axrp.net/episode/2023/04/11/episode-20-reform-ai-ali...

littlestymaar 2y ago
Sounds interesting, but it also sounds like something that could very well be circumvented by using a technique similar to speculative decoding: you use the censored model like you'd use the fast llm in speculative decoding, and you check whether the other model agrees with it or not. But instead of correcting the token every time both models disagree like you'd do with speculative decoding, you just need to change it often enough to mess with the watermark detection function (maybe you'd change every other mismatched token, or maybe one every 5 tokens would be enough to reduce the signal-to-noise ratio below the detection threshold). You wouldn't even need to have access to an unwatermarked model, the “correcting model” could even be watermaked itself as long as it's not the same watermarking function applied to both. Or am I misunderstanding something?
- jkhdigital 2y ago

tiffanyh 2y ago

OT: The publication (Spectrum by IEEE) has some really good content. It's starting to become a common destination for when I want to read about interesting things.

samatman 2y ago

This is information-theoretically guaranteed to make LLM output worse. My reasoning is simple: the only way to watermark text is to inject some relatively low-entropy signal into it, which can be detected later. This has to a) work for "all" output for some values of all, and b) have a low false positive rate on the detection side. The amount of signal involved cannot be subtle, for this reason. That signal has a subtractive effect on the predictive-output signal. The entropy of the output is fixed by the entropy of natural language, so this is a zero-sum game: the watermark signal will remove fidelity from the predictive output. This is impossible to avoid or fix.

thornewolf 2y ago
you are correct of we suppose we are at a global optimum. however, consider this example: i have two hands i have 2 hands these sentences communicate the same thing but one could be a watermarked result. we can apply this equivalent meaning word/phrase change many times over and be confident something is watermark while having avoided any semantic shifts.
jkhdigital 2y ago
You're not wrong, but natural language has a lot of stylistic "noise" which can be utilized as a subliminal channel without noticeably degrading the semantic signal.

espadrine 2y ago

The academic paper: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 They use the last N prefix tokens, hash them (with a keyed hash), and use the random value to sample the next token by doing an 8-wise tournament, by assigning random bits to each of the top 8 preferred tokens, making pairwise comparisons, and keeping the token with a larger bit. (Yes, it seems complicated, but apparently it increases the watermarking accuracy compared to a straightforward nucleus9 sampling.) The negative of this approach is that you need to rerun the LLM, so you must keep all versions of all LLMs that you trained, forever.

mmoskal 2y ago
They actually run 2^30-way tournament (they derive an equivalent form that doesn't requires 2B operations). You do not need to run the LLM, it only depends on the tokenizer.
- espadrine 2y ago
  You’re right. I understood it to require taking the top 2^30 tokens, but instead they sample 2^30 times with replacement. Too bad they only formulate the detection positive rate empirically. I am curious what the exact probability would be mathematically.
jkhdigital 2y ago
Why do you need to rerun the LLM? Watermark detection only requires the hash functions (equation (1) from the paper).

blintz 2y ago

These watermarks are not robust to paraphrasing attacks: AUC ROC falls from 0.95 to 0.55 (barely better than guessing) for a 100 token passage. The existing impossibility results imply that these attacks are essentially unavoidable (https://arxiv.org/abs/2311.04378 https://arxiv.org/abs/2311.04378) and not very costly, so this line of inquiry into LLM watermarking seems like a dead end.

TibbityFlanders 2y ago
[dead]
jkhdigital 2y ago
I spent the last five years doing PhD research into steganography, with a particular focus on how to embed messages into LLM outputs. Watermarking is basically one-bit steganography. The first serious investigations into "secure" steganography were about 30 years ago and it was clearly a dead end even back then. Sure, watermarking might be effective against lazy adversaries--college students, job applicants, etc.--but can be trivially defeated otherwise. All this time I'd been lamenting my research area as unpopular and boring when I should've been submitting to Nature!
- impossiblefork 2y ago
  Though, surely secure steganography with LLMs should be quite easy? Presumably there are things like key exchanges that look like randomness, and then you could choose LLM output using that randomness in such a way that you can send messages that look like an LLM conversation? Someone starts the conversation with a real message 'Hello!' and then you do some kind of key exchange where what is exchanged is hard to distinguish from randomness, and use those keys to select the probabilities of the coming tokens from the LLM. Then once they is established you use some kind of cipher to generate random-looking ciphertext and use that as the randomness used to select words in the final bit? Surely that would work? If there is guaranteed insecurity, it's for things like watermarking, not for steganography?
sbszllr 2y ago
I’ve been working in the space since 2018. Watermarking and fingerprinting (of models themselves and outputs) are useful tools but they have a weak adversary model. Yet, it doesn’t stop companies from making claims like these, and what’s worse, people buying into them.

cowmix 2y ago

"I hope this message finds you well." --- busted!

ajwin 2y ago

Do LLM's always pick the most probable next word? I would have thought this would lead to having the same output for every input? How does this deal with the randomness that you get from prompting the same thing over and over?

8note 2y ago
There is at least a parameter called Temperature which decides how much randomness to include in the output. It doesn't get you perfectly deterministic output to set it to 0 though, per https://medium.com/google-cloud/is-a-zero-temperature-deterministic-c4a7faef4d20 https://medium.com/google-cloud/is-a-zero-temperature-determ... as you don't have perfect control over what approximations are being made on your floating point operations
- mmoskal 2y ago
  The most typical reason argmax (temp 0) is non-deterministic is that your request is running batched with other people requests. The number and size of these affects the matrix sizes and thus tiling decisions. Then you get different floating point order and thus different results. Nvidia gives some guarantees about deterministic results of their kernels but that only applies when you have exact same input data and this is not the case when in-flight batching.
janalsncm 2y ago
It depends. If we use beam search we pick the most likely sequence of tokens rather than the most likely token at each point in time. This process is deterministic though. We can also sample from the distribution, which introduces randomness. Basically, if word1 should be chosen 75% of the time and word2 25% of the time, it will do that. The randomness you’re seeing can also be due to implementation details. https://community.openai.com/t/a-question-on-determinism/8185/9 https://community.openai.com/t/a-question-on-determinism/818...

lowbloodsugar 2y ago

I want AI to use just the right word when it’s writing for me. If it’s going to nerf itself to not choose the perfect word so it can be watermarked, then why would I use that product? I’ll go somewhere else. And if it does use just the right word, then how is that different from a great human writer?

Nasrudith 2y ago
There is the 'loser's litigation' method of getting all of your non-watermarked competitors banned. Usually involving some combination of magical rights removing brain-hacks like national security or 'the children'.

fny 2y ago

I think we just need to give up on this. What’s the harm? It’s not like some ground truth is fabricated. I’m far, far more concerned about photo, video, and audio verification. We need a camera that can guarantee a recording is real.

ziofill 2y ago
I've been thinking about this for a while. Digital signatures can guarantee that a piece of data is authentic, if the author wishes to sign it.
foxglacier 2y ago
Why do we need that for photo, video and audio? If it's about the general public believing something false, they're not going to check the watermarks of random internet content or trust anyone who says they checked it. If they really want to know, they can go to the source and if they trust that person or organization, they can also trust the content they published. If it's about use in court, we already have a system for that - the person who recorded it appears in court as a witness and promises that they didn't alter it then if it turns out they did, they can go to prison.

harimau777 2y ago

This strikes me as potentially a bad thing for regular people. For example, corporations call still use AI filtering to force job seekers to jump through hoops but job seekers won't be able to use AI to generate the cover letters and resumes that those hoops demand.

matthewmorgan 2y ago

Who is going to pay for watermarked output?

nprateem 2y ago

Google are obviously pushing this as a way to root out AI blog spam. If only they can get other providers to use it because of 'safety' or something they won't have to change their indexer much. Otherwise page rank is dead due to the ease of creating content farms.

riffraff 2y ago
Not just them, openai is doing the same for the same reason: they need to avoid an Habsburg ai issue when the next half of their training material will be generated by themselves

ruuda 2y ago

Some comments here point at impossibility results, but after screening hundreds of job applications at work, it's not hard to pick out the LLM writing, even without watermark. My internal LLM detector is now so sensitive that I can tell when my confirmed-human colleagues used an LLM to rephrase something when it's longer than just one sentence. The writing style is just so different. Maybe if you prompt it right, it can do a better job of masking itself, but people don't seem to do that.

auggierose 2y ago
So, how many times did you actually get the confirmation that an LLM has/has not been used? My guess is zero times. So, you are not describing an experiment here, you are just describing how you built up your internal bias.
- tessierashpool9 2y ago
  this is an important realization!
- kuhewa 2y ago
  Probably not entirely fair. e.g. After enough sentences it is trivially easy to identify LLM output. So you repeatedly get the opportunity to test a sentence or two, guess the provenance and then realise it is the first sentence in several paragraphs of generated output.
  - exe34 2y ago
    without external validation, you still just have a gut feeling. it doesn't matter if it's the first or thousandth. I can tell if somebody ate cornflakes or oatmeal for breakfast just by looking at how they walk. I'm always right. you better believe me. I've seen thousands of people walk.
    - kuhewa 2y ago
      If the cornflakes had been sitting in old milk for a week, then yes, I would believe you could tell by how they walk, either hunched over with a cramping stomach or with shit pants. LLM text is quite similar.

villmann 2y ago

To what degree will AI-generated text and watermarking influence how human language evolves... I bet "delve" will become more frequent in the spoken language :)

tomxor 2y ago

> Such modifications introduce a statistical signature into the generated text, Great so now people have to be worried about being too statistically similar to an arbitrary "watermark".

sharpshadow 2y ago

To archive the watermark they store every output which they create and let partners check against it. That’s how I understand the article. Then they also store everything which the partners upload to check if it’s created by them. If other AI players also would store everything they create and make it available in a similar way there could be indeed some working watermark. If one would use a private run AI to change the public run AI generated content to alter it there still would be a percentage similarity recognisable to hint that it might come from one of the public AIs. Timestamps would become quite relevant since much content would start to repeat itself at some point and the answers generated might be similar.

matteoraso 2y ago

By design, a watermark would make it easy to create a discriminator that distinguishes between LLM content and human content. In that case, just make a discriminator yourself and use regex to find and remove any of the watermarks.