17 ms·

Anthropic's ‘watermark’ text adulteration in Claude is a perversion of writing

• gilrain 2mo ago
  Best writer I’ve ever seen not understand writing. Proof positive that you needn’t understand something to produce lots of it!
• fluxem 2mo ago
  Why does everyone love AI and is impressed by AI quality but when the time comes to admitting that they use the magical AI, no one wants the watermark?
• andy99 2mo ago
  I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?
  • chrisjj 2mo ago
    > In any code or structured output, there just isn’t the flexibility Variable name perversion incoming...
  • smallerize 2mo ago
    Yes, and it says that in https://www.anthropic.com/news/claude-text-watermark https://www.anthropic.com/news/claude-text-watermark
    • andy99 2mo ago
      I should have read that, it’s actually quite reasonable and I don’t really understand the objections in TFA having read it. > One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. Doesn’t make sense at all in light of the actual approach, they’re just choosing a different RNG. It’s not like they’re corrupting it by flipping words. Should add I don’t support the watermarking and requiring it is idiotic.
      • krackers 2mo ago
        I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been altered irreversibly during RLHF which is how you get the obvious Claudism like "load bearing" and "seams"? Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.
        • troupo 2mo ago
          > I don't understand Gruber's points either, Gruber's point is to bash the EU. He couldn't care much about anything else.
      • smallerize 2mo ago
        I think Anthropic should have put all the info into one blog post. Splitting it up is really confusing people.
  • Cakez0r 2mo ago
    This could be why claude code has recently started to write reams of inane comments alongside the code it generates.
• nomel 2mo ago
  > The provider must mandate in their terms-of-service that users not remove the watermarking. So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark? Security theater, and vague enough to be used as a weapon against who the government wishes. I hope it's left off for non-EU customers.
  • inigyou 2mo ago
    You know you can just ignore EU laws outside of the EU
    • knollimar 2mo ago
      They are literally asking for Anthropic to do that rather than roll out 1 system for everyone.
• chrisjj 2mo ago
  > the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text. It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.
• smallerize 2mo ago
  Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude. I think that was intended, yes.
  • ButlerianJihad 2mo ago
    It is quite just, if you think about it. Human works are copyrighted and protected at the moment of creation. All rights reserved. Yet, LLM outputs are uncopyrightable. Therefore, if Claude or any AI has processed my copyrighted work, the end result is uncopyrightable and in the Public Domain. The public has a right to know: is this a human copyrighted work, an LLM PD work, or is the human falsely claiming authorship in order to retain copyright? A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful. If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.
    • fwipsy 2mo ago
      Perhaps LLM outputs are uncopyrightable, but derivative works of copyrighted works are not automatically in the public domain.
      • ButlerianJihad 2mo ago
        That's an intriguing twist, isn't it? It could lead to a tug-of-war. Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now? If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works? This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.
• syrrim 2mo ago
  > I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.
  • dragonwriter 2mo ago
    That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.
    • reliablereason 2mo ago
      (1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.
      • case540 2mo ago
        Citation needed
        • inigyou 2mo ago
          You know you can just try it and see on any inference system thst has this knob, right? Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
• walrus01 2mo ago
  > I want any LLM I use to choose the very best, most precise words at every single decision point. Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.
  • LoganDark 2mo ago
    That is not what that means. Generally, precise word choice requires more than autocomplete. Larger models simulate this with hidden layers.
    • walrus01 2mo ago
      Excessively precise word choice does not result in something that looks like content written by, or palatable to humans. It looks like you gave a high school 12 grade student a science paper and told them to apply a thesaurus to at least one word in every sentence and replace it with something else.
      • LoganDark 2mo ago
        There is a difference between precise word choice and concise word choice. You can be precisely accessible the same as you can be concisely terse.
• aselimov3 2mo ago
  This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed
  • beering 2mo ago
    Exactly. The watermark is proportional to how much text is AI generated. Either the AI really just “fixed some typos” (not enough AI content to hide a watermark) or the AI did most of the writing (enough AI content to hide a watermark).
  • breezybottom 2mo ago
    This feels like the inevitable outcome of a STEM-only education system. Now people think there's a mathematical formula for picking the "best" words, instead of having to be thoughtful and creative.
• bushido 2mo ago
  This is not meant to be snarky, But almost any writing done by Claude is a perversion of writing. I honestly can't stand the way Claude writes. This watermark change just makes it scarier.
  • _kulang 2mo ago
    I moved to Sol for my writing and it is so so much better. But it makes more mistakes. I think they have different ideas of product but it seems OpenAI is going to follow Anthropic’s lead over the next year. I think I am going to put more effort into my writing skills to remove myself from this awful situation
    • einpoklum 2mo ago
      > I moved to Sol for my writing and it is so so much better. No it's not. The bad part about it is that some machine is writing instead of you, not the specific stylistic idiosyncracies.
      • _kulang 2mo ago
        Sure – but I’m entitled to make a judgement on what I consider to be “good” and “bad” output from an LLM, where “good” just means “helpful to my process”.
  • gitaarik 2mo ago
    Sorry, what's scary about it exactly?
• nian2326076 2mo ago
  The objection isn’t that normal sampling is somehow pure. It’s that watermarking uses token choice to carry an additional, covert signal. The quality cost may be small on average, but it is still an optimization constraint unrelated to meaning or style—and it turns ordinary prose into provenance metadata. The more serious questions are detection reliability, false positives, and what happens after human editing. https://prachub.com/ https://prachub.com/
• arjie 2mo ago
  It seems fine. I use an LLM to argue with me prior to posting blog posts so that I don't post obvious incorrectness, but the UX element to it is that it constructs notes about various sections of the text and we talk about those. There's no way for the generated text to enter the blog unless I copy-paste it and I'm not going to do that because the entire point is for me to write it. At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
• stabbles 2mo ago
  Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.
• Finnucane 2mo ago
  "Anthropic's . . . Claude is a Perversion of Writing." FITFY. I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.
• levocardia 2mo ago
  Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
  • reader9274 2mo ago
    [flagged]
    • brookst 2mo ago
      I think he’s still generally good on business, UX, and hardware design. That’s all subjective and taste I suppose, but his taste works for me. On deeper tech stuff, like this utterly nonsensical misunderstanding of watermarks… yeah, classic case of a guy who is smart, and has lost the ability to realize when they’re not knowledgeable in a domain.
  • tapland 2mo ago
    Making blog posts about AI that make it apparent that the tech is going whoosh is a choice.
  • selectively 2mo ago
    [dead]
  • docjay 2mo ago
    [dead]
  • Art9681 2mo ago
    If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem. Cognitive surrender.
• lemarchr 2mo ago
  Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided. The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition. I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens. Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion. I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.
  • capitalsigma 2mo ago
    If the author has preferences on their "own writing" that conflict with Anthropic's, then they should actually write it themselves rather than paying Anthropic to do it. Private companies don't owe you anything, even less so when they're beholden to laws in foreign jurisdictions.
  • Barrin92 2mo ago
    >Assume that there is a sequence of tokens, such that it is considered the absolute best by the author You can't assume that because if that was the case he'd already know what sentence to write, because that's what that means. The notion of a best sentence requires a final cause, an end to write to. By their very nature that's not how LLMs work, so you can't 'degrade' them on that front. They can't lose a property they didn't have.
• capitalsigma 2mo ago
  > I chose to depend on a private company to express my own thoughts and now I'm mad that I'm not in control of the output Who could have seen this coming???
• 4d4m 2mo ago
  Reminder: your favorite distilled model does not treat you, the customer, as an adversary and mess with your output.... May the free market win.
  • deleted 2mo ago
    [deleted]
• herf 2mo ago
  Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.
• DarkmSparks 2mo ago
  I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy. Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless. Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.
• jeffgreco 2mo ago
  Gruber has a ridiculous knee-jerk response to anything the EU does, so hardly a surprise he didn't come to the table with a sober facts-based response.
• LoganDark 2mo ago
  I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.
• Imnimo 2mo ago
  >I want any LLM I use to choose the very best, most precise words at every single decision point. Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing? This entire article just seems so detached from the basics of how LLMs work.
  • Gigachad 2mo ago
    I think the author is just mad people will be able to detect and filter out their AI slop writing in the future.
    • dofm 2mo ago
      This is not it, no. He is not using AI and it is not I think remotely in his nature to surrender that control. He is engaging with this on principle. Again I am not sure I agree with him, but then it’s a hypothetical because I am not going to get an LLM to write for me either.
      • beering 2mo ago
        Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI. Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.
        • dofm 2mo ago
          His complaint about the ChatGPT app is that it’s a shitty non-Mac-ish Mac app. Complaining about shitty non-Mac-ish Mac apps to people who hate shitty non-Mac-ish Mac apps is more or less how he became a full time writer.
      • NitpickLawyer 2mo ago
        > He is not using AI That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.
• RegardDetector 2mo ago
  [flagged]
  • ghomst 2mo ago
    [flagged]
• Planktonne 2mo ago
  There is no coherent position in which the watermarking is a perversion of writing but AI writing as a whole is not a worse one.
• codedokode 2mo ago
  Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.
  • alienbaby 2mo ago
    This is the first post I've seen mention it. How traceable are the embedded codes?
    • addandsubtract 2mo ago
      There was an earlier instance of this here: https://news.ycombinator.com/item?id=48734373 https://news.ycombinator.com/item?id=48734373
  • dofm 2mo ago
    I don’t disagree about open weights (though the enabling aspect there is actually open source inference, right?) But it feels to me like you would need a hell of a lot of text to bury even a simple account ID. The nudges they are talking about are of the order of a handful of bits over several hundred words, I think?
• pibaker 2mo ago
  I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this. > The difference between watermarked and un-watermarked text will not be distinguishable to readers https://www.anthropic.com/news/claude-text-watermark https://www.anthropic.com/news/claude-text-watermark Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here. If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.
  • cubefox 2mo ago
    > The EU regulation, from my understanding, requires AI content to be labeled for human viewers. How would that work? Claude appending " written by AI" to each of its messages? That would both be impractical and useless.
    • inigyou 2mo ago
      I think there are two separate requirements? One that if you post something like an AI video on the internet or anywhere else, you must label it as AI. And another one that AI providers must watermark their outputs. If you get caught uploading watermarked media without the clear label, you're in big trouble, mister.
      • cubefox 2mo ago
        We were talking about text.
        • inigyou 2mo ago
          Which must also be marked as AI, I hope. (But I doubt that is the law, because the EU is cucked to big businesses interests)
• etchalon 2mo ago
  The objection seems to be that Claude will always write worse prose than a human writer, even if the writing Claude generates is understandable. Yeah, John. We're all OK with that.
• jacobgold 2mo ago
  Watermarking will be one more nail in the coffin of proprietary models if the world is so fortunate. Reminds me of printer tracking dots. https://en.wikipedia.org/wiki/Printer_tracking_dots https://en.wikipedia.org/wiki/Printer_tracking_dots
  • inigyou 2mo ago
    And yet we still use printers and 90% of our color documents have the tracking dots.
• ghomst 2mo ago
  I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?
• roywiggins 2mo ago
  it serves to show just how little regard the people behind these generated-text fingerprinting schemes have for the actual craft of writing. LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.
• egypturnash 2mo ago
  LLMs are already perversions of writing, so what else is new. Oh no, the over-long circumlocution generated by three autocorrects in a trenchcoat might be slightly longer because of this and maybe people will start noticing the subtle rhythms of vaguely peculiar word choices as yet another cue that you are wasting their time with machine-generated wordslop, what a terrible fate. Your long rambling walls of machine-waffling might be 37.05% longer than they need to be instead of the mere 36.58% longer they are now.
• tacker2000 2mo ago
  Lots of faux outrage, rambling and hyperbole here from Gruber. “Absurdly and insultingly”? Come on…
• wewewedxfgdf 2mo ago
  It's good to be the King. And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.
• amanzi 2mo ago
  I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.
  • inigyou 2mo ago
    Can we install random unapproved apps on our iPhones yet, or is Apple aiming to just be fined a trillion dollars because they make more than that from the 30% cut?
    • rsynnott 2mo ago
      Yes: https://support.apple.com/en-mk/117767 https://support.apple.com/en-mk/117767
      • inigyou 2mo ago
        This says we can only install Apple-approved apps.
        • rsynnott 2mo ago
          ... Maybe it's localised or something? I see a section "How to install apps from a developer’s website in the European Union". Do you see that? (They do require notarisation, but not review/approval).
          • inigyou 2mo ago
            Notarisation is review/approval
  • rimliu 2mo ago
    I am from EU. Alas it has a tendency to produce some idiotic regulations. Cookie banner, new packaging fee, etc. I genuinely think some Apple related ones hurt customers more than help them.
• micromacrofoot 2mo ago
  gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp
• inigyou 2mo ago
  > One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best. This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.
  • FeteCommuniste 2mo ago
    Yeah, I snorted at the sentence "The exact words we choose when writing matter." Well, then why the heck are you using an LLM to "write," man?
• 0x_rs 2mo ago
  I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156 https://arxiv.org/abs/2303.11156 Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices: "1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text" A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services: "Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves." Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now: "Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."
  • troupo 2mo ago
• avazhi 2mo ago
  [flagged]
  • breezybottom 2mo ago
    Clearly Anthropic thinks its more profitable to comply and have access to the European market, but I'm sure you know better than the people who brought it to a $2 trillion valuation.
    • avazhi 2mo ago
      I didn't say anything about profitability. Since Anthropic is all about the good of mankind etc, ostensibly profitability isn't their number one priority.
  • carlosrg 2mo ago
    Hi Gruber.
• dmix 2mo ago
  I will be happy to move off Anthropic given the chance. They are burning all of their good will.
• snickerbockers 2mo ago
  What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated. I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.
• robomc 2mo ago
  This is moronic. This is like being mad that the slot machine you think is lucky is occupied.
• pizzly 2mo ago
  Now for the human generated watermark. Timez to addd the speelling mistakes, decreaze the usegage of big words and proper gramicaly usuage. Wish I was joking.
• veidr 2mo ago
  This (Anthropic's "watermark" stance, I mean) is so fundamentally ludicrous that I have assumed it is a (wholly insincere, but arguably pragmatic, at least from their perspective) attempt to deal with the EU and their latest misguided, ham-fisted attempt to solve a real-world problem by drenching the entire world with more regulatory slop[1]. The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule. [1]: https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content https://digital-strategy.ec.europa.eu/en/policies/code-pract... (FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)
• bagacrap 2mo ago
  > But only Anthropic will be able to determine if text was seemingly generated by Claude, and Anthropic will only be able to detect the watermarks that are applied by Claude. Claude can’t detect the hidden watermark signals generated by, say, Gemini, and Gemini can’t detect the hidden watermark signals created by Claude, because each implementation is predicated on secret keys held only by the LLM provider Well, akshwally... > Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
  • deleted 2mo ago
    [deleted]
• brcmthrowaway 2mo ago
  Wow, never has a single article revealed the incompetency of a tech writer.
• nojs 2mo ago
  There are many reasons to hate this watermarking but affecting the output quality isn’t one of them. The central argument he’s making is wrong. Switching out one RNG for another doesn’t make the results worse.
• akersten 2mo ago
  Respectfully, you are all missing the point. Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly. As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over. It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.
  • wasabi991011 2mo ago
    You should submit an article about this instead of having your point buried in a comment section of an article making an unrelated argument.
  • otterley 2mo ago
    I suspect there’s a niche market for software that records your writing process to help you avoid being falsely accused of using AI to write.
• wasabi991011 2mo ago
  > because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice. ... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s
• ChrisArchitect 2mo ago
  Related: How Claude's text watermarking works https://news.ycombinator.com/item?id=49303350 https://news.ycombinator.com/item?id=49303350
• beej71 2mo ago
  I guess I understand the complaint, but LLMs are already crap at writing, IMNSHO. And, yes, maybe this will make them marginally more crap, but in my mind we're talking the difference between a 30% grade and a 29%.
• vancekai 2mo ago
  [dead]
• otterley 2mo ago
  I often agree with John Gruber, but I think he’s lost the plot with this one. The thing I don’t understand is why he seems to care so damned much about this subject--enough to write over 4,500 words on it! John writes for a living. That’s his profession. He’s been writing for over 25 years now. When you’re that good at writing, and you care this much about your writing, you don’t allow an LLM to take over your job. I just can’t imagine that he’s in the market for LLMs and that literary excellence is his number one selection criterion. So why is he so livid about it? It’s like being angry that wine is going to start coming in smaller bottles even though you don’t drink wine. Even if he’s angry on behalf of other people, I don’t get it either. In my view, having LLMs write publishable content on your behalf is not a socially-acceptable use case, nor a professionally-acceptable one in most professions, even though people are abusing it for this purpose anyway. And besides, the models aren’t even all that good at it today. If you agree with that, then you certainly should not care if it’s using different phrasing than you otherwise might prefer if the meaning is similar enough. I can't help but wonder if perhaps his hatred of EU technology regulation (which, admittedly, is mostly pretty dumb and is mainly just making life worse for users) is getting the better of him.
  • knollimar 2mo ago
    This guarantees he can never use it for the one purpose he might care about though
• andOlga 2mo ago
  What a truly bizarre article. Arguments about pre-existing randomness, temperature and whatnot aside, I simply cannot comprehend what the author here really thinks the "best word" is. There's no such thing. We humans fall on familiar patterns of writing ourselves, so we may forego something with a flourish in favor of a more commonly-used word unless we put in effort to be "special", which should be used sparingly. That is to say, human writers are likely to choose a "worse" word in far more than the supposed 51% of cases, and that has no effect on the actual quality of writing in the end. But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.
• rsynnott 2mo ago
  ... If watermarking is a perversion of writing, what does that make 'writing' with an LLM?
• dexterlagan 2mo ago
  I needed another reason to cancel my Claude sub. Thanks Anthropic! This is akin to adding a giant watermark on things one would made with a free product "Made with XXX". Except you're paying $200/month for it, and there's no way to disable that watermark. I don't disagree with EU regulations, but I strongly believe the onus should be on the content publisher, not the toolmaker. If the toolmaker watermarks whatever his tool produces, it opens a giant can of worms that cannot be closed. That means anything and everything you make with this tool is no longer fully yours, it contaminates everything and makes your work traceable. Who wants that? I was already annoyed by the fact that Claude marked everything it did on my repos under its own account (I didn't ask for any of this), but now everything is invisibly marked, even the code. Not that I care that my writing would be watermarked, since I'd rather write my stuff myself, but code? No thanks. Meanwhile I'm running a DeepSeek V4 Flash or Pro, or a Qwen3.8, and it writes my code without a peep. Resulting repos are clean, just the way I want them. No 'Claude' account, no watermarking, nothing. I won't be looking back after having tried these new models. Whoever makes good models that don't broadcast their maker will get my business. This watermarking will simply push people more towards Chinese models. Keep pushing in the wrong direction Anthropic. Doing this right before an IPO is a great idea.
• slhck 2mo ago
  The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output. If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much. NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.
  • ilogik 2mo ago
    Gruber really doesn't like any EU regulation
• aenis 2mo ago
  The same absolute morons who gave us cookie consent strike again. I swear, one of those days I will get into politics just to fight those two things, and the cottage industry of batshit crazy lawyers that gave birth to those things.
  • vrganj 2mo ago
    The cookie consent banner is not the EU's fault. It's either don't track or ask for consent. The fact that the industry chooses to track is not on the EU.
    • aenis 2mo ago
      Its more nuanced than that. Even companies that do not track, and use only essential cookies, ask for consent, as the consensus among compliance teams and external lawyers is "its safer this way". Thats the reality which the regulators failed to anticipate. Of course this misses a bigger point that tracking in the web moved in a direction that requires no cookies whatsoever, and if anything, feels more pervasive than it ever was. And it misses the even bigger point, that the morons who legislated cookie consent did not notice either of those two realities. And the same thing is already true with the AI act; the text watermarking is trivially defeated and everyone knows it. And I'd bet it will remain a requirement for the next decade or three.
      • vrganj 2mo ago
        What company compliance teams do is once again, the responsibility of companies, not the EU. And the move away from cookies was addressed. That's why we have the GDPR now. We can always write new regulation, but its important to handle things as they come up and not use technological change as an excuse to do nothing.
• runtime_lens 2mo ago
  [flagged]
• voidUpdate 2mo ago
  > "The exact words we choose when writing matter." Then write your own damn text if you care about the exact wording so much
  • allisdust 2mo ago
    LLMs are no more than pen and paper at this point. Especially for those who aren't trying to create slop. We all would want our pens to accurately reflect the strokes (well in this case thoughts) rather than adding tiny watermarks to identify that it is generated by a particular pen or a user. Watermarking per model is just the start. The method is cheap enough to distinguish individual users.
    • voidUpdate 2mo ago
      LLMs do a whole lot more than writing your thoughts down. They write extra text. If you just want a pen and paper, use Notepad. Or better, a pen and paper
    • Arodex 2mo ago
      >LLMs are no more than pen and paper at this point. Then use pen and paper. It is the same, you say, right?
      • deleted 2mo ago
        [deleted]
    • ehnto 2mo ago
      That is an insane statement, LLMs generate swaths of text from almost nothing. If they are adding so little value as to be as transparent as a pen and paper then why use one at all? Transcription doesn't need an LLM so that's not what you're taking about I assume.
    • a2ff6eeb0 2mo ago
      What? The entire reason I use an LLM is to be able to avoid thinking about a topic. That's their whole damn value prop: outsourcing thinking and producing without understanding. I don't need to read emails in detail to respond any more.
• armchairhacker 2mo ago
  Are there some narrow cases (like in outputting the input almost verbatim) where the probability for every best token is always much higher than second best, and in those cases would there be no watermark?
• zzril 2mo ago
  > Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot. How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? And then it takes the EU another 20 years to understand what a silly idea this was?
  • ghrl 2mo ago
    Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API. I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.
• blfr 2mo ago
  The main reason we don't see much quality degradation in LLM writing output is because they're already poor writers. This is the load bearing reason. I was bulding a small interpreter and writing an article in ~markdown yesterday with Fable. And while it codes like a pro, it writes like a sixth grader. Let's see how these watermarking stats hold up if/when llms start writing well.
• tempestn 2mo ago
  LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just generating pseudo-random results differently. Quality shouldn't be affected.
  • simonh 2mo ago
    That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output. If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.
    • phiresky 2mo ago
      An encrypted hard drive is EXACTLY uniformly distributed random bytes if you do not know the encryption key. No one would be able to tell the difference between a drive that is just purely random numbers or is actually filled with content. (Of course excluding the usually intentionally added readable header) If this was not the case, the encryption would be broken, and most everyone agrees that good encryption does exist. The "quality of the probability calculations" as you put it is 100% in this case and any less would be a huge deal (as in - breaks all of the internet). So, now you just take those same random bytes and use them as the seed for your LLM token choices. The output has the _cryptographically_ proven exact same quality as if you were using a true RNG (which you likely weren't using anyways). You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner. > choosing different words that it otherwise would The "that is otherwise would" is carrying all the weight here. "Otherwise" is sampling from a distribution. You just sample from the same distribution but with a cryptographically secure, seeded RNG. https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator https://en.wikipedia.org/wiki/Cryptographically_secure_pseud... "Knowing the LLM distribution" seems to me like the only hard part because you don't know the context of any random snippet.
• mrweasel 2mo ago
  This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking? Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?
  • Laurel1234 2mo ago
    [dead]
• TheOtherHobbes 2mo ago
  The issue here isn't (just) adulteration, it's that watermarking in general is unworkable. If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text. Which is clearly ridiculous. And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial. That's not even getting into the legal complexities of businesses running open source models without watermarking locally.
  • vrganj 2mo ago
    > Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
    • frm88 2mo ago
      No idea why this was flagged to death. I vouched for it because it is a direct quote from the AI transparency act: https://theaicounsel.net/wp-content/uploads/2026/07/07_26_codeOfPractice.pdf https://theaicounsel.net/wp-content/uploads/2026/07/07_26_co... It clarifies interoperability requirements.
• woadwarrior01 2mo ago
  I think writing is the killer use case for local LLMs. We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point. One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.
• jimnotgym 2mo ago
  > My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text. Or just em dashes? /s
• Bluestein 2mo ago
  Heck. The entire process itself of LLM text generation is a perversion of writing.- Further heck: It can be said it ain't even writing.-
• jimnotgym 2mo ago
  I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map). A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore. This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.
• ghrl 2mo ago
  My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ... So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means. Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
  • josephg 2mo ago
    I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated. I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!
    • piker 2mo ago
      Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.
      • josephg 2mo ago
        > Won't we just be able to fine tune OSS models to detect these patterns across providers? A good fingerprint should make use of cryptographic signatures. Without knowing the keys, the fingerprint should be indistinguishable from noise (or just random token selection)
        • piker 1mo ago
          Wouldn’t those hashes be trivially defeated by tweaking the language?
• thinkingemote 2mo ago
  I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value. Everyone who invests in AI companies wants to see the value of their investment increase. I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.
  • Cakez0r 2mo ago
    I think you're right that it's actually to do with attribution (i.e. They're not just watermarking your output as claude generated, but watermarking it as claude generated _by claude user id 73684_). I think it's more to do with the growing militarization of the internet. Just another brick in the wall of enter your phone number to create an account, send your id to prove your age, smile for the flock cameras, etc. Allowing the plebs to have privacy and anonymity is not allowed anymore.
  • tjpnz 2mo ago
    As someone who doesn't utilize LLMs for writing production code I'm looking forward to starting a consultancy and reaping the rewards.
• LIMEVINCE 2mo ago
  The author makes a lot of great points. I find it surprising that somebody who has such a nuanced appreciation for the subtleties of language would be in the crowd complaining about the watermarking policy. I expected this kind of complaints from mostly students interested in academic dishonesty, who generally don't have enough command over written language to notice the slight decrease in output quality.
• nomilk 2mo ago
  tl;dr: > It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of (watermarking) And an example of the impact of watermarking on word choice [0]: > The results of the study were quite [important || significant || substantial || notable] The meaning of the sentence to changes slightly even in just this tiny example. Imagine the degradation when applied across an entire response! [0] https://declaude.org/watermarking/ https://declaude.org/watermarking/
• fbrncci 2mo ago
  Then then don’t use Claude ? What’s wrong with all these people getting vendor locked in.
• fwlr 2mo ago
  If we wish to use “the very best, most precise words at every single decision point” then I will note that “writing” is not the best word for what LLMs are doing. A more precise synonym might be “generating”, in which case this essay becomes “Watermarking is a perversion of text generation” - which is true, albeit somewhat trivial.
• Twey 2mo ago
  People are very upset, especially in the arts, that Anthropic is changing the text to watermark it, but isn't that missing the point a little bit? They're not changing _your_ text whose every word you've carefully chosen for the exact effect, they're changing text that they're generating, i.e. text you've already chosen to give up control over. LLMs can't understand emotional nuance anyway. The phrasing of the announcement implying that phrasing and diction don't change the meaning of text is insultingly dismissive of the whole field of literature, and I can see why people might take it as an afront, but the actual technology shouldn't have a negative impact as far as I can see. It seems to me that this one is more of a PR problem than something with real-world impact.
• iammjm 2mo ago
  Great, now Claude will sound even more generic. It’s not only annoying - it’s also dumb. We are all correct to push back on this load-bearing issue
• carlosrg 2mo ago
  Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles? Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News the next day calling for an invasion. As a former reader of Daring Fireball, it’s just sad to see.
  • redfloatplane 2mo ago
    Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality. I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog underneath the bar for me.
    • jorisw 2mo ago
      As an EU citizen I've found myself in agreement with everything he's written regarding EU policy
      • redfloatplane 2mo ago
        Good for you! As an EU citizen I have found myself in almost total disagreement with everything he's written regarding EU policy. Horses for courses.
    • solid_fuel 2mo ago
      Freaking out about basic EU privacy and safety measures is always one of the biggest red flags that someone is becoming a politically motivated hack, especially when that someone lives in the US where there is an active fascist movement tearing apart the government.
      • 2mo ago
• floki165 2mo ago
  [flagged]
• redsocksfan45 2mo ago
  [dead]
• sbszllr 2mo ago
  I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again: "I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026." Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous). And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs. Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.
• deleted 2mo ago
  [deleted]
• Chrisszz 2mo ago
  This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs
• mdavid626 2mo ago
  Just copy Claude’s output and shove it into Gemini and ask to rephrase.
• GaryBluto 2mo ago
  I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms. https://arxiv.org/abs/2107.06751 https://arxiv.org/abs/2107.06751
  • Alpha3031 2mo ago
    Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.
    • GaryBluto 2mo ago
      I see how that was confusingly written. I'm not suggesting Anthropic are somehow retroactively causing it, just wondering if it could create a similar effect.
      • Alpha3031 2mo ago
        Less unlikely, but I would still suggest that it is somewhat unlikely for any recent LLM to place any significant probably on, e.g. "disappointment" instead of "failure" as the appropriate token (or series thereof) after "kidney" or any similar example.
• bejd 2mo ago
  >in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code. I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)
  • croemer 2mo ago
    Opus 5 does not have the watermark yet. Only models released from now on.
• deleted 2mo ago
  [deleted]
• ppeetteerr 2mo ago
  Gruber was a good voice in the industry but this article misses the mark in a lot of ways. A company the size of Anthropic would not voluntarily jeopardize their massive valuation if they didn’t feel the resulting output would maintain a similar level of quality as before. Is there a similar worry that their system prompt, which is injected at the start of every conversation also influences token generation in an artificial way? If regulation will ruin Claude as a product, market forces will fill the void. There are also a ton of open weight models to choose from. It’s going to be okay.
  • tmgldn 2mo ago
    Gruber is not much of a details man - he helped invent Markdown (to be lauded) but ghosted its standardisation. I would be fascinated to hear Prod John MacFarlane of UC Berkeley's opinion on it all given he was heavily involved in the push to get Markdown standardised.
• Voultapher 2mo ago
  What a bunch of entitled whining. How is the system to know that it's just a private conversation that won't be used in some fraudulent way? Abuse is currently rampant, yes please let's find a way to mark LLM output. The thing I'm worried about is giving the providers the power to claim provenance. Even ignoring the privacy issues, the operational hassle of having to check N providers makes these approaches at best limited. I want to see research into providing a shared public or ideally self-hostable oracle that uses some standardized method for watermark detection. Similar to asymmetric crypto where users can't reasonably find out the secret part but can do something useful with it nonetheless.
• PufPufPuf 2mo ago
  LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.
• Create 2mo ago
  "possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity," The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.
• flufluflufluffy 2mo ago
  There is no very best token to choose at each decision point. It is context dependent and subjective.
• its-summertime 2mo ago
  > My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate. Has that ever been the case? Are they not actively tweaking their models, their fine tuning, the system prompts, the tool definitions and implementations, the guard rails, tool calls, instant responses. There are hundreds of knobs that they can change daily, or between each prompt, or even half way through a generation.
• tosh 2mo ago
  any watermarking ai researchers who can explain this? what if the llm should - repeat something verbatim (important in a compaction prompt) - there is just one correct order of tokens for a somewhat long chain (a certain sequence of control signals) - provide a diff of 2 inputs without punctuation or whitespace wiggle room? how does the drifting work? does it postpone the drifting and drift stronger later? what if max_tokens is set to a low number? in what way does this not affect output quality?
  • talljeff68 2mo ago
    The Anthropic annoucement addresses lots of this. They give an example of generating code where there is clearly a requirement to follow a specific syntax and therefore the watermarking will be much less effective and likely require much larger sample of generated content to build statistical confidence in the validity/existance of a watermark. For code, it is the generated comments that will be more likely/able to contribute to the watermark confidence. The key to understanding the watermark technology is to realize that the model is/was already using randomness to select among the top most probable tokens, often randomly picking between choices of [nearly] equal weight. The watermarking does NOT change the distribution of the random number generation nor does it affect the range of probabilities for which tokens are being considered. Instead, it only drives the sequence of the random numbers such that they form a cryptographically generated known ordering pattern that is determined by the secret key generating the pseudo-random sequence. As a result of the approach, assuming inference is being done with all other parameterization of token selection being the same, there should be NO impact on the output quality....the amount of variation of output is within sample of the variation in output that already exists run-to-run of the same prompt. FWIW too: Google has confirmed this experimentally as well through full scale tests and evaluation of online Gemini output in search result pages.
• Retr0id 2mo ago
  What a strange take. LLMs themselves are a perversion of writing. I couldn't care less about the implementation details of the PRNG they use for next-token sampling (well, as long as they're not stuffing a user ID in there).
• _joel 2mo ago
  So the watermark can be removed by rearranging words and choice of words. This seems trivial to bypass with a local model. If I understand this correctly.
  • elpakal 2mo ago
    my understanding is that it's a probability so it can never really be removed? only made smaller.
• bonoboTP 2mo ago
  Author read the posts but didn't understand them. > At each decision point, they’re a little more likely to pick a word from the green list than the red list. Wrong. There is no global red and green list. It's dependent on context and balances out on average. It won't change the result when one token is predicted overwhelmingly likely.
• avadodin 2mo ago
  This is the obvious solution to watermarking text in a way that isn't easily removed by a dumb tool. The output quality will likely suffer as stated in the article although this can be mitigated to an extent by only enabling it on more irrelevant filler text while leaving the more functional sections untouched. The solution using Unicode tricks amounts to malicious compliance as only the most unsophisticated users are going to fail to remove the AI watermarks when trying to pass off AI slop as their own prose. The real solution here is not having stupid EU–tier laws in the first place.
• epsteingpt 2mo ago
  The idea that language models can write anything remotely useful yet is - mostly - a fallacy. It's a good idea for many human endeavors to be able to identify AI writing. Communication, after all, is our main way of building the social fabric. However - and crucially - good writing is still beyond the frontier of any model I've seen so far. Watermarks for the things that truly matter may not be important at all. Finally, as X commentators have shown, simply removing punctuation or changing a word here or adding an adverb there manually will screw up the whole process enormously. The best will be the clever folks who retroactively apply the model distribution to fraud or other crimes to try to implicate the companies via watermark. Gotta feel for their product, policy and legal team.
• fenestella 2mo ago
  [flagged]
• balherian 2mo ago
  A) without water marking, llm will poison the open internet and poison their own source of fresh new material. B) without watermarking, it potentially exposes an infinite deluge of garbage getting to people , fostering hate, eventually leading to dis-engagement, ergo destroying their training pipeline. C) ...Sadly, them water marking the claude chats with absolute dementia levels of output text is beyond me, i seriously can't tolerate this bullshit "randomly replace words with "similar" words", bs anymore, the output that claude makes right now is outright *corrosive* to my eyes, eventually leading to their training pipeline dying if devs dis-engaging ( of the 3 i think this one is the worst ) I think they need to get their shit-together and realize this is a death warrant for the tech ( in my opinion ).
• analog31 2mo ago
  The first thing that came to my mind was "security theater." Making people think that AI is detectable could have the same effect as actually making it detectable.
• lluisantoni 2mo ago
  Text watermarking is another EU rule made without real world input. The Union is stuck on major economic crises (electricity prices for instance) because nobody can agree on anything. However, the bureaucracy forces tech into a privacy nightmare. Brussels cannot bring together its own members but it loves pretending it can govern the internet.
  • bramhaag 2mo ago
    > Text watermarking is another EU rule made without real world input. Except for the input of the hundreds of stakeholders they consulted, Anthropic included [1]? > The Union is stuck on major economic crises (electricity prices for instance) because nobody can agree on anything. That sure seems relevant for the implementation of AI watermarking... > However, the bureaucracy forces tech into a privacy nightmare. No, this transparency allows consumers to more easily detect AI generated content. [1] https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content https://digital-strategy.ec.europa.eu/en/policies/code-pract...
  • einpoklum 2mo ago
    > (electricity prices for instance) because nobody can agree on anything. I would say that's more like because the US has arranged for Europe's fossil fuel energy sources to be disrupted or cut off: * Libya - NATO made a pig's breakfast of that, it's a failed state now. * Iran - transitive sanctions, because why not prevent non-US states from trading with each other. * Russia (& Kazahkhstan) - The US (with or without Ukranian involvement) bombed the NordStream pipeline(s), led the EU into the proxy war in Ukraine and a sanctions regime against Russia. Kazakh oil goes to Europe through Russia. * Gulf states - until recently, possible but not very convenient ; since Feburary of this year, the war on Iran messed that up badly too. the US is the winner here not just geo-politically, but also as an oil exporter, with the EU now depending on purchasing US-exported oil.
    • Lumich 2mo ago
      Spot on. Also, in 2003, the war on Iraq, still occupied. And the proxy war on Syria (stifling an unwelcome pipeline project). European “leaders” pretend to not comprehend how they're being screwed. Stockholm syndrome. Populations don't understand, propaganda (“free press”) working correctly.
• holoduke 2mo ago
  Whole watermark thing is just bullshit. What if I add a watermark text and another AI as well. How many watermarks and who is the real creator. It just doesn't make sense. I ll eat my shoes if this concept is still a thing in 6 months
• einpoklum 2mo ago
  Are we now going to live in a world in which people bitch and moan about large private corporations' LLM text generation service and whether it's good or bad etc.? As though they're supposed to be benevolent and serve the public interest? They're not and they don't. Also, write your own damn text. > "My error was believing Anthropic" The error is rearranging part of one's life around Anthropic.
• DoneWithAllThat 2mo ago
  I know (and understand why) a lot of people cheer the EU’s increasingly vast regulatory environment as being “pro consumer” but I’m really tired of said regulations being inflicted on the rest of the world. If this is what Europeans want for themselves that’s fine. But I no more want their regulations to be the de facto world’s any more than I want China’s.
• exabrial 2mo ago
  Such a stupid regulation. Don’t make the rest of the world suffer because of one dumb law passed for a fractional share of users. This also, just another precedent of anti-user, pro Authoritarian, from LLM companies.
• dev1ycan 2mo ago
  While I like that LLMs won't be able to produce the entire internet anymore, I am worried that the real reason for this move is for Anthropic to claim everything is theirs even though they STOLE humanity's collective knowledge including billions in private property worth of knowledge (or maybe even trillions), and all they do is regurgitate it, but now with a watermark on top as if it was theirs.
• a2ff6eeb0 2mo ago
  I assume this means Gruber has fully embraced generating his blog, but is too embarrassed to admit it?
• herrkanin 2mo ago
  Is this 'best word' with us in the room right now?
• masswerk 2mo ago
  Mind that even in their first example, "The results of the study were quite (important | significant | substantial | notable)", the meaning is by no means interchangeable. "Important" refers to impact, "significant" to the statistical qualities of the underlying hypothesis, "substantial" to the work involved, and "notable" is a referential judgement by the speaker. The implied normalization of words and their respective meaning also marks one of the mechanisms how "slop" is typically creeping into the productions of "broad verbose interchange replicas" (it's all interchangeable, and a choice isn't really that, a choice, isn't it?).
• LogicFailsMe 2mo ago
  I have no doubt they can benchmax their way into believing the differences caused by watermarking are imperceptible. But I'm skeptical there won't be a drop in quality that puts them at a disadvantage relative to competitors that don't do this. This adds a new constraint changing predicting the next token to predicting the next token that carries the watermark. It's hard to see how this can avoid making the watermarked output <= the desired output in quality.
• tantalor 2mo ago
  Fix it for you: Generative AI is a perversion of writing
• 13639366668 2mo ago
  [flagged]
• mangoman 2mo ago
  I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those percentages were learned from the training data and RL. To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? Or more likely they dont have the data that shows the failure modes? also, don’t these choices compound the changes to the distribution in later sampling choices? It is a bit of a mystery to say that “its okay to choose different tokens that we would have for watermarking bc people don’t notice” as though word choice doesn’t matter. If it doesn’t matter, doesn’t that mean that intelligence is more of a commodity than they would want it to be?
  • matheusmoreira 2mo ago
    > But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. This. I want the model I'm paying for to be "pure". I don't Anthropic or anyone else messing around with it, especially not for idiotic reasons like facillitating AI stigmatization. The "safety" nonsense is obnoxious enough. They should train the best possible model and let the weights speak for themselves, not degrade it into some perverted form to appease people who hate AI anyway.
    • applicative 2mo ago
      You can’t be serious. The weights for composition are massively degraded by RLVR training for coding
      • mangoman 2mo ago
        I agree, but thats also learned, and it is done to encourage specific responses for a task, different than applying a mask to the distribution based on a key.
    • 2mo ago
• root_axis 2mo ago
  It's no more a perversion of writing than the act of using an LLM to write in the first place.
• andy_ppp 2mo ago
  As the algorithm tries to keep the stenography in place this will mean if you specifically ask for a different phrasing of one paragraph, other parts of the document will need to change to keep the supposedly impossible to detect AI watermark in place? Won't people also just quickly do analysis on this to figure it out and remove the watermark - it's hard to do this for individual messages but when you can create an infinite number of messaging to train on, I'd think analysis of how the signal works will be quite trivial.
• cbondurant 2mo ago
  > I want any LLM I use to choose the very best, most precise words at every single decision point. Oh! If you want that, you should run your own model and set the generation temperature to 0 :) Because that's not what any commercial LLM is doing. Never has been. This is just making up a universe that doesn't exist so you can get mad about no longer being in the universe that doesn't exist. The masking technique of using a subset of the statistical distribution for each next token isn't going to be meaningfully distinguishable from a natural language perspective. I honestly think its a very elegant way to implement watermarking. I've got no real opinions on how effective it will be to people actively trying to defeat it, but I suspect that the people who are trying to pretend that LLM text was something they wrote themselves are probably too lazy to put in the work to try and defeat it anyway.
• matheusmoreira 2mo ago
  > It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of embedding hidden clues within the text to suggest its provenance. Yeah, that about sums it up!
• VikRubenfeld 2mo ago
  I was using Claude yesterday and the "advice" it was giving me quickly became confused and irrelevant to the prompt, even though there was not much text in the context window. Speculation: since Claude re-reads the entire chat at every turn, the "minimal" text revisions required by watermarking quickly compound such that even Claude can't follow the discussion.
• tancop 2mo ago
  This scheme has a fatal flaw, the same secret key is used for watermarking and detection. This means you need to trust that Anthropic: * Shows the real result from their detector instead of manipulating them, you have no way to verify * Has good enough security to prevent a key leak * Rotates keys to reduce the impact of a leak (once a key is leaked anyone can rewrite text to look more/less claude generated and it becomes useless) * Will not secretly give watermark-less access to governments or high profile corporate users * Will not use multiple secret keys to track individual users. This one might be less realistic because embedding ~32 bits of signal would probably affect quality a lot more than 1 bit. And don't forget that the detection API will work as an oracle. If it detects your content you can send it to a different model and try again until it comes back clean.
• jihadjihad 2mo ago
  > The idea that anything other than my needs should factor into the generation of text for me is patently offensive. And this attitude is incompatible with any models produced by frontier labs. Your needs will always be subordinate to and in service of the needs of the corporation that produced the model. And we haven't even gotten to ads yet.
• Marazan 2mo ago
  Imagine using a random word generator and getting upset that the random words it generates are not truly artisanal random words.
• nbulka 2mo ago
  They chose the wrong word "watermarking." Perhaps they find the statistics _after_ a certain amount of text is generated at scale, so that it doesn't affect the fidelity or integrity of the LLMs output. There is evidence this exists already, and it's why "I have to be honest..." and "This is the right lens, ..." keep popping up. Probably would be cheaper too. Maybe I am missing something?
• darkstarsys 2mo ago
  Sorry, this article's argument just doesn't hold water. Yes, we want Claude to write "the best text" and having its word choice even very slightly varied could arguably be construed as "not the best." But "best" is highly subjective, always has been. Claude has never, and will never, write what _you_ consider the best version of a piece of text. It has many choices, influenced by all kinds of random, context-dependent weights and environmental settings. Slightly tweaking weights to prefer certain phrasings might even tilt it toward your idea of "best."
• m3kw9 2mo ago
  His argument for needing it to write every word at its best is weak. The black box within can change quality at every moment based on many factors they he hasn’t known about, say system prompt, or other harness adjustments. Models get better every 2 months, and they write better, but he is comparing to what? The real argument should be watermarking itself. I don’t want my shit water marked if I ask you to just rephrase a certain part.
• jackk03 2mo ago
  [dead]
• bccdee 2mo ago
  > “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter. I want any LLM I use to choose the very best, most precise words at every single decision point. Neither of these is "better" or "more precise"; in fact, LLMs will generally choose randomly between these candidates based on temperature, and SynthID should not distort the output of an LLM any more than the default temperature settings do already. I agree that those are not the same sentences, but if the difference matters to you, you shouldn't be using an LLM. This difference exists at the level of what sounds better and is more evocative; to an LLM, nothing sounds like or evokes anything. They simply do not write good prose.
  • slowin 2mo ago
    I’m more concerned that this will negatively impact code generation. An additional constraint completely unrelated to code quality is unacceptable as far as I’m concerned. I was an Anthropic user but now I’m looking at OpenAI or even better, open models.
    • skort 2mo ago
      If you're concerned about code generation, then learn how to actually write the damn code yourself!
    • jweber123 2mo ago
      According to their article, these kinds of arbitrary choices don’t come up as often with code, so it’s less likely to have watermarks: https://www.anthropic.com/news/claude-text-watermark#:~:text=for%20a%20watermark.-,What%20about%20code?%C2%A0,-As%20we%20noted https://www.anthropic.com/news/claude-text-watermark#:~:text...
    • bccdee 2mo ago
      Unless you're already manually setting temperature to zero on the models you work with, the deviation from the "optimal" path imposed by the watermark should be no greater than what you already get from the randomization.
• whack 2mo ago
  "The difference between the almost right word and the right word is really a large matter. ’tis the difference between the lightning bug and the lightning." - Mark Twain "Just flip a coin to pick a random synonym. Who cares?" - AI Labs
• adamretter 2mo ago
  I almost never post a comment here, but everything about the author's position is offensive and self entitled. I am so enraged that I can't even beging to formulate a response without resorting to very bad language. It's sad, because until now I respected the author. But clearly, and sadly, he has been afflicted with AI brain rot, and is likely in some stage of withdrawal. I wish him a speedy and safe recovery.
• andrethegiant 2mo ago
  Great question buried at the bottom: > Also, what happens if another major global market makes it unlawful for AI to secretly watermark generated text?
  • elpakal 2mo ago
    That is a good question > We’re applying watermarking globally at launch because we don't yet have a durable way to scope it by region. However, we will continue to evaluate different approaches, and will share updates when we have them. So unless they figure it out, would that 'major global market' essentially need to be the US?
• ltbarcly3 2mo ago
  Wait are we supposed to be mad because clankers are displacing human creative workers, or mad because clankers don't do their level best when producing creative works because they are forced to watermark text? Or is it that they use all the water (I know they don't but are we supposed to be mad about it still)? I can't keep up with the current Chinese psycop. There should be some kind of status page like whywewantamericatofailataitoday.ai so we can keep up with it.
• kmeisthax 2mo ago
  No, machines being used to replace human writing is a perversion of writing. Some might call it worse than cannibalism[0]. If you're only noticing now because Anthropic is changing things behind your back, well... I've got some bad news for you, but the entire cloud-hosted subset of the AI space, especially Anthropic, is premised on the fact that doing things behind your back to their models is socially preferable, or worse, should be outright mandated. While I generally hate legal mandates to stab your customers in the back, in my opinion there is no harmless way to use AI and mandatory text watermarking is a good bare minimum. The EU probably made the right call. The entire AI space - open models included - is predicated upon worker exploitation, replacement, and deskilling; we should at least be able to know how much of our media diet has Anthropic's fingerprints on it. A lot of hay is made over the pretraining process in which copious amounts of stolen data are trained on; but parallel to this is a huge data labeling and human feedback operation staffed almost entirely by people in third-world countries with robust English as a Second Language (ESL) programs. The thing is, AI models already watermark their text, they just happen to do so with the textual watermarks of the Indians and Nigerians that the AI companies hired to do RLHF because they were cheap. That's why certain AI models love the word "delve" so damned much. It's neocolonialism, designed specifically to do the kind of replacement the anti-immigrant idiots keep screaming their heads off about[1]. Furthermore, as we've seen with Hank Green, even non-cannibalism-adjacent AI usage is a recipe for worker deskilling and AI psychosis. The other half of the RLHF pipeline is to turn a pile of compressed text into a chatbot that feeds you a steady drip of unsourced information while praising you every time you spot one of its lies and never saying no[2]. This is a recipe for addicting your customers. Also, this might just be because this is on daringfireball.net, but I can't help but think the author has an axe to grind against the EU because the EU mandated Apple sign third-party app stores. The fact that he's balking at Anthropic not going along with gating the watermarks to just the EU seems downstream of this - "why aren't you maximally attempting to resist the EU?" [0] https://www.youtube.com/watch?v=YCPAIg7RUq8 https://www.youtube.com/watch?v=YCPAIg7RUq8 [1] To be clear, upwards of none of the far-right have actually clued into the fact that AI is trained by underpaid immigrants, mainly because it doesn't fit the narratives the people running the far-right want to push. There are some AI robotics companies that are even very explicit that their robots are there primarily to launder foreign labor into rich companies and make permanent labor arbitrage. [2] Continuing on from [1], the far-right actually really loves AI specifically because it rarely says no to even stupid ideas, even if it's also a manifestation of everything they claim to hate.
• summarybot 2mo ago
  What's gonna be a real trip, is when you can tell which LLM produced it by the subtle pattern recognition you'll have developed to catch the watermarks
• sleepybrett 2mo ago
  How does this work for code?
• kergonath 2mo ago
  A lot of that rant is nonsense. Is Gruber also outraged that a RNG is already involved in everything LLM produce with a non-zero temperature? Surely it already leads to widespread use of “non-optimal words”. Simply the idea that there is a single optimal choice, down to every single word, to convey a meaning and anything straying from that is adulteration is laughable. It’s very difficult to take seriously an argument based on this kind of foundation. He had a very similar rant a couple of days ago when he somehow thought that they would use invisible characters. It’s just as useless.
• luckydata 2mo ago
  is that why everything Claude writes lately sounds like a riddle?
• heap0x20rot 2mo ago
  [flagged]
• solid_fuel 2mo ago
  What a stupid take. Generating text with an LLM is already a ‘perversion’ of writing. Tweaking the last random-choice step doesn’t meaningfully change that at all.
• laurentlb 2mo ago
  Watermarking seems feasible for creative tasks (that admit many valid results). But if I craft a prompt that doesn't leave space for creativity, how can they include a watermark? e.g. "Rewrite the following text, replacing 'foo' with 'bar'." I'm curious to see where they draw the line, and whether the watermarking really affects the (perceived) quality of results.
  • wisemanwillhear 2mo ago
    Although that feels like an overly simplistic example to the point of not being not being a helpful example, I agree that the amount of tokens to encode the watermark is so low in text, that it wouldn't take much to erase or distort it.
  • rreichman 2mo ago
    They can't include a watermark in that case, that's not the relevant use case.
• Hahna11 2mo ago
  Gruber is a smart, thoughtful man. This is a bizarre take from him. It exhibits an undeveloped understanding of LLMs, and a righteous view that generated prose should assimilate... which should be offensive to organic intelligence. Issues with the proprietary nature of Anthropic's watermarking aside, we will look back on this as a 'thank god' moment in the history of LLMs.
• declan_roberts 2mo ago
  I really don't think Anthropic has the leverage to pull this kind of thing off in the face of competition. OpenAI doesn't do it (yet), Grok doesn't do it, neither does Deepseek. I'm sure the latter two have plans to intentionally never do it. Who is even asking for this? Sounds like something some obsessive internal employees would push on the world.
• cratermoon 2mo ago
  He's complaining that a coin-flipping synthetic text extruder is using an unfair coin. The text generated by an LLM is already adulterated. His objections are so much titling at windmills.
• Duanemclemore 2mo ago
  As a university professor, back in the early days of this when students were just copy / pasting output directly I was really hoping the model builders would implement steganographic fuzzy hashes. Not that it's the ideal solution. But it would be really neat.
• pikuseru 2mo ago
  If you don’t like it don’t use it
• Jeff_Brown 2mo ago
  A question not addressed here is whether there exists any solution to the EU requirement that doesn't suck. If the same red/green algorithm described in the article is applied to generated code, I cannot imagine how that does not degrade code quality (probabilistically, not at every point).
• keito 2mo ago
  I've been seeing a lot of the same questions about watermarking over the last week, so I made this playground that lets you try out 3 different watermarking schemes (including a version of SynthID-Text) on your own text: https://watermark.keito.me/ https://watermark.keito.me/ to understand it experientially myself. I found it illustrative to try different examples like code, text rewriting, etc. to see how it affected the tokens. Playing around with the detector is interesting too to see how much of the watermark can remain in edited text. I hope this is helpful for others too.
• mrtesthah 2mo ago
  Claude is already a perversion of writing.
• mickdarling 2mo ago
  Watermarks are context poisoning.
• zebomon 2mo ago
  Posting for visibility a reminder that this type of watermarking is comically easy to beat. See Scott Aaronson's 2023 research in which he coined the term "pineapple attack": one must simply prompt the model to add the word "pineapple" after every other word and then do a search + replace to remove the word, to obliterate the watermark altogether. https://www.youtube.com/live/2Kx9jbSMZqA?si=0QgCPBX2_KPZ0QTU&t=3073 https://www.youtube.com/live/2Kx9jbSMZqA?si=0QgCPBX2_KPZ0QTU...
• asmnzxklopqw 2mo ago
  Awwww, little blogger can’t write his slop anymore without rest of the world knowing that he’s not the author?
• Havoc 2mo ago
  Not sure about the strength of that particular argument, but can't say I have any faith in this being used to benefit end consumers. Somehow they'll find a way to use this for regulatory capture
• bluegatty 2mo ago
  This was nice but without hard evidence. I'll bet if nobody were to have said anything, nobody would nave noticed. That doesn't make it right, but it might be less intelligible and intrusive than the claims here. Also this: "> I want any LLM I use to choose the very best, most precise words at every single decision point." I don't think the author quite appreciates the level of randomness here. This is more subtle then Claude changing prose. Again - I suggest that the author would have to run a test on themselves to determine if they can actually find a difference.
• sfink 2mo ago
  I agree with the substance of this article, and disagree with the author's reaction to it. The part I agree with: It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the output, and using those bits of entropy for a purpose that does not improve the quality. If you maximize your tradeoff of bits for quality, those extra bits lower the ceiling of what's possible. It's a very simple information theoretic argument, and the only plausible argument against it (that we're using those bits so inefficiently now that the new PRNG is no worse than the old) only holds in the short term. I also agree that having TOS that forbid removing the watermarking is deeply, deeply problematic. Hell, the whole essay is well-written and persuasive, and gives good reasons why this is a poor approach. What I disagree with, and the reason for this comment, is the entitlement. > The idea that anything other than my needs should factor into the generation of text for me is patently offensive. This attitude is what is patently offensive for me. This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral. It's another form of certain people being above the law, shareholder profit maximization über alles, might makes right, we have to do it or someone else will, "we just help people connect", {code,a gun,roofie} is just a tool. So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.
  • Spooky23 2mo ago
    > So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible. Computers are tools that exist to serve. Creating some bizarro process where we are compromising the technology in service of it's owner to achieve some nebulous goal is gross. Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop. There's some vague notion of preventing harm without any backing, but a very real cost for startups to develop a compliant watermarked AI model.
• renecito 2mo ago
  They just want to prevent their own slop being fed back into their AIs. I think this would be a good guard on websites to use it as another content protection layer.
• Akranazon 2mo ago
  > I want any LLM I use to choose the very best, most precise words at every single decision point. No, you don't. If you wanted that, you would set the temperature parameter to 0. But that would lead to less desirable results, not better. LLMs do not set the temperature to 0; they typically set it 0.4-0.7.
• DrBazza 2mo ago
  So Anthropic goes on the shit list with Sony Blu-ray and its cinavia watermarking.
• khelavastr 2mo ago
  It's because the individuals who write those laws are literally trying to shove neo-Nazi policies into EU practice. I wonder how many people would make policies like this if they and their families were publicly identified and criticized as neo-Nazi elements in society. You think someone will write Nazi-promoting AI policy like this when society is encouraged to look at their families as examples of neo-Nazi corruption? When their wives' and kids' friends spurn them while their families engage in obvious criminal activity to harm world productivity? Critics need to be more precise.
• potlee 2mo ago
  Doesn't this require the whole message history including the system prompt for them to check for the watermark?
• Magicrafter13 2mo ago
  Tl;dr the author is upset that the slop generator will generate slightly different slop, and calls this perversion of the text, ironically missing the fact that the entire technology takes existing human work and perverts it in order to give him the output that he seemingly enjoys. Perhaps it's not actually irony, perhaps it is hypocrisy. This isn't to say there aren't correct statements in the article, but its framed very strangely.
• deleted 2mo ago
  [deleted]
• ucha 2mo ago
  > “By definition it must make text worse … because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.” Gruber made an effort to but doesn't fully understand how SynthID works. LLMs select the next word randomly from a set probability distribution, so there is no "best choice" unless you run the LLM with a temperature of 0 which would give out terrible results. Anthropic runs a non-distorting version of SynthID that doesn't change the probabilities of the underlying distribution of tokens. It makes the watermark less likely to work over smaller samples but preserves text quality. I encourage the mathematically inclined to read the paper: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4
  • tarvaina 2mo ago
    I came here to quote the same sentence. Here's another way to look at it: Suppose there actually is a best word choice. The LLM doesn't know what it is but makes a guess. Maybe it's the best one, maybe it isn't. The probability that SynthID changes the best choice to a worse one is equal to the probability that it changes a worse choice to the best one.
    • abustamam 2mo ago
      I think that depends on the distribution of good choices and bad ones. There may be 10 choices and maybe 8 of them could be appropriate given a context, and 2 are absolutely nonsensical. Or it could be vice versa. And its a spectrum as well.
      • tempestn 2mo ago
        But the choices are weighted based on those probabilities. This doesn't affect the weightings, only how the final weighted pseudo-random selection is made.
        • abustamam 2mo ago
• marcfrommelious 2mo ago
  [flagged]
• mbonnet 2mo ago
  Using LLMs for prose that matters is already a perversion of writing.
• kristianbrigman 2mo ago
  The watermark is based on stylometry - and we all know Claude has a style. But it doesn’t have to - everyone knows caveman - so not sure there is enough room in all cases for it to work.
• 1vuio0pswjnm7 2mo ago
  "My writing is my work, and Anthropic's current strategy is aggressively writer-hostile."
• markstos 2mo ago
  I wonder how much they tested on non-English languages with fewer synonyms.
• alphazard 2mo ago
  Humans have been inserting benign noise into their writing without affecting the signal for thousands of years. They called it "style". Apparently superfluous descriptions, unnecessary words, and paragraph footnotes are all fine, but gumbel softmax (or whatever's going on here) isn't.
• aa_is_op 2mo ago
  Plagiators are having a fit these days
• niemandhier 2mo ago
  Ai is a perversion of writing. A useful one, but still. I don’t see how this makes it worse.
• fencer12 1mo ago
  Man this watermark nonsense is all over the place
• zerotolerance 2mo ago
  I just don't like the idea of an AI company saying that they've added a secret signature that isn't verifiable by any third party to all responses. And we're supposed to both take them at their word, and feed them all the content we want to check so they can continue gobbling up a bunch of fresh works. What is to stop them from saying "Oh yeah, that is ours. We signed it. Trust us." The obvious conflicts here are wild.
• mustafamoiz 2mo ago
  This post makes no sense. You can either care about writing and write things yourself, or you can not care and farm it out to a model. There is no third option where you care about writing and farm it out to a model.
• mikewarot 2mo ago
  Doc Searls, the long time Editor-in-Chief of the Linux Journal, brings an editors viewpoint to this, and mixes in a number of other voices.[1] Many writers use AI to edit their own words, and this watermarking poisons the well for that use. I think the backlash from this could be the seed that undoes their attempted 2 Trillion IPO this fall. [1] https://doc.searls.com/2026/08/17/you-can-hear-the-squeak-of-sphincters-closing/ https://doc.searls.com/2026/08/17/you-can-hear-the-squeak-of...
• bitcurious 2mo ago
  It’s super ironic to see this many bots on this thread. Sorry dang.
• slowmovintarget 2mo ago
  "The more words in the text, the more accurate the analysis will be that the text was generated by a specific AI model or not." And this is why the latest Claude models blather so much more. All for the sake of this "watermark."
• boesboes 2mo ago
  Using AI is the only perversion here And what a self-reporting from all the AI bros!
• _m_p 2mo ago
  https://en.wikipedia.org/wiki/Oulipo https://en.wikipedia.org/wiki/Oulipo
• guido26 2mo ago
  I'm doing a fiction project using AI to write the framework, then fixing it. AI writes terrible fiction. Every single sentence in the project is approved and/or changed by me. And I like em dashes for this project. I've been using em dashes since TeX came out. The question is: does this make my project "AI" or does it make it mine? People use professional editors all the time. But they agree to give up any copyright in their edits. If I approve a change made by an editor, it becomes mine. But there is this idea that approving an AI sentence still make it AI.
• DivingForGold 1mo ago
  There will be services online that copy/paste (or you provide) the text from ai, they OCR it to get rid of watermarking, then feed it back to you for a fee. The premium upgrade will be "obfusicators" that further modify the text sporadically to make it look more like a human wrote it. Students for one, will gladly pay.
• midas89 1mo ago
  You would think that if the EU is requiring this, then why not just implement this for the EU users.
  • amai 1mo ago
    Because it is not about the EU. Google has implemented watermarking for a long time. And soon we will see that all AI providers will do it, because it allows them to filter out AI generated data from their training sets.
• amai 1mo ago
  I predict we will soon see more AI providers implement watermarking, because it allows them to filter out AI generated data from their training sets. That is probably the real reason why they implement it. Gruber just doesn't get it.
• toaste_ 1mo ago
  Get fucked? Really? As if the quality of machine-generated prose were somehow sacred? Is Gruber only now waking up to the idea that LLM companies do not give a shit about the quality of the writing they generate? That they're disdainful of the entire concept of writing as a profession? What rock has he been under? If you care one iota for the quality and craft of your writing, you would already recognize that any degree of AI "processing" obliterates stylistic choices. You're left with readable text, but it's fluffy, meandering, and has a cadence to the writing that screams to the reader "every second you spent on this was wasted."
• jannw 1mo ago
  you can remove AI company watermarks at www.demarkify.com
• jannw 1mo ago
  you can remove AI company invisible text watermarks at www.demarkify.com - and also perform other tonal text changes like removing AI lexical tells
• kjimlau 1mo ago
  [flagged]
• impalallama 1mo ago
  Fundamental misunderstanding of what AI is and what it does if this watermark is preversion of what AI is already doing.