5 ms·

> I want any LLM I use to choose the very best, most precise words at every single decision point. Then bad news: LLMs already use randomness in a fundamental

• dragonwriter 2mo ago
  That's inaccurate in two ways: (1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and (2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.
  • reliablereason 2mo ago
    (1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.
    • case540 2mo ago
      Citation needed
      • inigyou 2mo ago
        You know you can just try it and see on any inference system thst has this knob, right? Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
        • cadamsdotcom 2mo ago
          That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.
          • inigyou 2mo ago
            No, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?
            • cadamsdotcom 2mo ago
              It's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.
    • dragonwriter 2mo ago
      > LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.
  • beering 2mo ago
    > which inherently compromises quality. I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.
    • deleted 2mo ago
      [deleted]
    • TheOtherHobbes 2mo ago
      Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it. Never mind metaphors and similes, which are even more tightly constrained. LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class journalism. LLMs are still somewhere between the first two, and nowhere close to approaching the third.
      • tempestn 2mo ago
        It's not just swapping synonyms. The way llms work is by predicting the likelihood of the next token. It's inherently probabilistic. Choices are made based on weighted random number generation, based on those probabilities. Changing how you generate the random numbers doesn't degrade the output.
      • Closi 2mo ago
        > Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it. We already know that a non-zero temperature improves quality though with current models (particularly with creative writing). The assumption that always picking the 'best' token results in the 'best' output is not the current reality. And if you are already intentionally putting in randomness, I can imagine that it would be possible to seed the randomness in a way that is detectable but results in the same quality. This is obviously not true for queries where temp = 0, but at temp = 0 then it becomes easier to identify anyway. I assume this technique implies some level of temperature.
  • thatjoeoverthr 2mo ago
    1) we’re not discussing those systems. We’re discussing a chat AI product called Claude, which does not offer those knobs. 2) Claude’s PRNG having a P is immaterial
    • daishi55 2mo ago
      Claude has those knobs, they are just not exposed to the user. They could make Claude nearly completely deterministic if they wanted to (of course it would be a far inferior product then. But they could).
      • thatjoeoverthr 2mo ago
        “Not exposed” = has no knobs. Of course all autoregressive LLMs can be operated this way but Claude, the product, employs LLMs but isn’t one.
        • daishi55 2mo ago
          Your original statement “LLMs use randomness in a fundamental way” is incorrect. LLMs have these knobs and randomness is not an inherent property of LLMs.
  • tempestn 2mo ago
    You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?
    • TeMPOraL 2mo ago
      Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality. ( https://xkcd.com/221/ https://xkcd.com/221/ ) In case of LLMs, you can look at it from high and low level. At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph. At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before? That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.
      • Nnnes 2mo ago
        You're missing the same point that the blog post is missing. What they're doing is much less like replacing rand() with 4 and much more like setting seed(4) before generating any numbers. There is no "loss of quality" unless you're already using a temperature of 0.
    • Phemist 2mo ago
      1. As watermarked text is added to the training data, watermark-related tokens will be associated more with AI outputs and thus lower quality outputs which will hasten model collapse. Especially because every provider has its own secret key and they are all training on eachother's outputs anyway. I guess they can at scale filter the watermarked documents (by necessarily allowing eachother to at scale checked for watermarks, but banning the labs not part of the watermarking-cabal). Makes me wonder how useful the human quality filter is on AI output - if a human judges a given output as genuinely good and posts it somewhere for the scrapers to find and take into the training sets, will these types of outputs also be filtered out? 2. (raw, pre-watermarked) Output token probability situations where 1 output token has the majority of the probability mass associated with it, but it is not in the watermarked set, will force the model with much higher probability to walk a non-optimal latent space. E.g., if the next OBVIOUS token for a given sentence would be a point, but the model is in this way not allowed to output it, it might put a comma and start off on a whole different tangent just to make the initial non-optimal comma grammatically make sense.
• npilk 2mo ago
  I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want. On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense. Even good human writers would probably struggle with this constraint. It would be like someone interrupting your writing to tell you the next word MUST be such-and-such, and then you have to try and make it work as best you can first try, without going back to edit. The result would probably be a little clunky. (Maybe it’s impressive LLMs write as well as they do.)
  • scuppernong 2mo ago
    auto-oulipo
    • Alive-in-2025 2mo ago
      Today I learned a new word, "Oulipo". Interesting. But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something? I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.
  • hyusap 2mo ago
    autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.
    • npilk 2mo ago
      Sure, but mightn’t there be several plausible long horizon plans? Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was: — *Long journey tracks* Clinic — “The Return of Evil Bill” Guided by Voices — not really, wrong band Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic. — So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact. Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time. Human writing is like 90% editing.
• colmmacc 2mo ago
  I think the article is wrong on this but it's more subtle than that. Probability distributions have a peak; there is still a token with a peak probability. What's interesting about these techniques is that token by token it can actually make the peak token even more probable. A distribution doesn't have to be "flattened" to leave a watermark - it can be "amplified" and made "more peaky".
• avaer 2mo ago
  That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad. Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that bad, and if it is they won't admit it. There's no way for anyone to check. And your argument will still be technically correct but missing the point.
  • beering 2mo ago
    The watermark doesn’t change the distribution, only per-token selection. I think not understanding that is the source of most people’s FUD.
    • cubefox 2mo ago
      This comment disagrees with you: https://news.ycombinator.com/item?id=49324387 https://news.ycombinator.com/item?id=49324387
      • beering 2mo ago
        That comment merely says quality must be compromised. It doesn’t make it clear why that must be true. Empirical study seems to say that quality is not compromised, and looking at various proposed schemes, it seems intuitively true.
    • inigyou 2mo ago
      There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.
      • beering 2mo ago
        On average the distribution of selected watermarked tokens is the same as the original distribution. You can test this experimentally. I think there is a possible weakness in the context of the watermarker but that is not your claim iiuc.
• demibabs 2mo ago
  Yeah this is my main issue with the argument. He acknowledges in the article that LLMs are already non-deterministic, but he doesn’t seem to actually understand that.
• red_admiral 2mo ago
  The best discussion I've seen so far is from Scott Aaronson: https://scottaaronson.blog/?p=6823 https://scottaaronson.blog/?p=6823 > To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all n-grams and see that it was anomalously large. The general case, where the token probabilities can all be different, is a little more technical, but the basic idea is similar.
  • tmp10423288442 2mo ago
    Scott Aaronson basically invented this form of watermarking, so it's not surprising he has a good description of it.
  • montyanne 1mo ago
    > instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI. Seems like - given enough text to encode information into - it would be possible for OA to uniquely identify the user account (and maybe even the specific request) that generated some content, even if the chat text itself isn’t stored. Interesting argument in favor of local AI as a mechanism for privacy-preserving generated content. Though, I wonder if there’s a way to “bake” a hardware fingerprint into local models as well…
• simonh 2mo ago
  The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output. Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.
  • simiones 2mo ago
    No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7". Also, as long as the same sampling strategy is used during training as the one used during inference, then the LLM will actually do much better with the biased sampling strategy than it would with a fair one - because that is what it was trained to optimize.
    • simonh 2mo ago
      >No one uses a pure random function over the whole probability distribution described by the LLM's output. So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality. I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.
      • simiones 2mo ago
        My main point is that sampling with a modified distribution compared to the one produced by the model is already being done, and it is generally found to increase quality, not decrease it. So there is no reason a priori to assume that the watermarked distribution would be lower quality than other schemes for altering the "raw" output distribution (such as top P, top K, temperature, etc). My second point is that the training of a model by definition maximizes the fitness between the final output function and the training metrics. So, if the model is trained with the watermark applied, the training process will minimize the function `model_error(input) = |watermarked_sampling(model_output(input)) - expected_output(input)|`, by definition. This means that a model trained in this way will perform better when sampled using the watermaked_sampling method than if using, say, top_k sampling.
• neuroticnews25 2mo ago
  This is true and the author seems to not understand the problems with greedy (top 1) decoding or the fact that watermarking affects only high entropy tokens. But the published watermarking methods still have a slight negative effect on perplexity, so there is something more to it.
• exabrial 2mo ago
  This is not the same thing
• nathan_compton 2mo ago
  I cannot believe the markdown guy is this deranged.
• georgelyon 2mo ago
  Another good way to think about this is that it does change the output, but in a way that is equally likely to make it "better" as it is to make it "worse".
  • kylebebak 2mo ago
    That is not a good way to think about this. I don't have deep knowledge of how LLM's work, but the following is accurate enough to illustrate the point. Let's say the LLM is in the middle of text generation and "decides" that the next token is "dog" with p=0.55, or "cat" with p=0.45. With a temperature of 0, the model always picks dog, because it's the most likely next token. With a temperature of 1 the model picks dog 55% of the time and pick cat 45% of the time. With this watermarking scheme, the model might alter these probabilities s.t. p_dog for this particular generated token goes up or down. Let's say it does down, s.t. p_dog is now 0.45 and p_cat=0.55. Now, with T=1 the model picks cat 55% of the time and dog 45% of the time. Regardless of whether the "watermarking function" raises or lowers p_dog, the probability distribution for this token has changed, and whatever math this trillion dollar company and its brainiacs came up with to decide that p_dog ought to be 0.55 has been "adulterated". As others have mentioned there is no way around this. --- Regarding the watermarking scheme, it works because it doesn't just alter p_dog for this single output token. It alters probabilities for many of the generated tokens (it could do this to all of the output tokens; it's an implementation detail). E.g. at token N, it favors "cat", at token N+1 it favors "house", etc. This way, if you have the secret key that lets you generate the watermarking function for any output token, you can analyze a run of tokens and check whether it's likely they were generated according to your watermarking scheme. The longer the run of tokens, the more certain this check becomes (it becomes extremely certain quite fast).