4 ms·

Usually I would very much agree with you, but those things are not deterministic so if that's an issue for you you're probably not making the right choices.

• tomrod 25d ago
  Nope, strong disagree. The model is one small part of the process harness; behaviors are usually routable with expected propensities. Unexpected model changes avoiding change management messes with monitoring and observability thresholds. Stochastic controls are a real thing when you have your distributions defined; your workflows on a new model will throw that expected prior out the window.
  • nolok 25d ago
    I think it depends on the workload, which put you in the right (in some case, it's dangerous to do that), but for me even for very clean always correct path I always assume a model can, at time, have a vector brain fart (because it keeps happening). Also in principle it's similar to Anthropic downgrading. Personally I use the basis that if I don't self host (I include remote host, but that I pay per hosting nor per model or api), it can change behavior without me asking. But they shouldn't, but it doesn't matter that's what they do.
• gcanyon 25d ago
  They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper. Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.
  • packetlost 25d ago
    > They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper. This isn't true. Even Sol messes up JSON formatting for me on occasion. Do not delude yourself into thinking these things are reliable. They are not.
    • kamranjon 25d ago
      Is nobody using structured outputs? They use constrained decoding at the generation stage to ensure the probability of tokens that would break the format are set to 0. I kinda figured everyone was doing this at this point.
      • nolok 25d ago
        If you use a large enough volume, you will know this isn't fully reliable. You might get json, and it might not match what the model actually sent because the last layer cut it up to match what you want. At the end, not json, or json but not really matching what the model wanted, it's sort of the same issue: when you use them you HAVE to assume they can have a brain fart. That's fine, just code around it.
      • wongarsu 25d ago
        Most serious providers are now supporting structured outputs in a reasonable way for all model configs. But for example on ollama structured outputs are still incompatible with tool calling and with reasoning
        • Zopieux 25d ago
          Friends Don't Let Friends Use Ollama. https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
      • gcanyon 25d ago
        We use structured output and To my knowledge it has never failed (millions of data points). There seem to be two classes of people: those doing productive work with LLMs, and those who only get replies insulting their mothers…
• idiotsecant 25d ago
  Models are not deterministic, but they do have a flavor. When that flavor changes it can change the nature of output in a way that is undesirable. Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.
  • tomrod 25d ago
    This is a great way to think of it, idiotsecant. That flavor profile is known -- it's typical behavioral distribution is somewhat understood (and, often, common failure modes addressed). If JSON breaks about 20% of the time, and that drops for 2% or blows up to 90%, it can drive all sorts of issues (not the least, costs for retries).
  • nolok 25d ago
    Yeah but then it doesn't matter if it's "the model changed to another one" or "the model changed but it's the same name". Point is, you're not hosting it, as far as you know it can change at any moment, build around that idea. Is that great no, is that ideal no, that's why I self host (I include actually renting online the capacity and hosting the model myself on it).
    • switchbak 25d ago
      You've added a similar response like 5 times, but I don't see you adding more information. Yes - model hosts can do nasty things to you aside from changing the underlying model. That doesn't mean it's cool to have them change the model automatically. Yes, it would be preferable to have complete control over your model serving, and no - not everyone is in a position to do that themselves.
      • nolok 25d ago
        [dead]
    • vikramkr 25d ago
• lkois 25d ago
  That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree. I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.
  • nolok 25d ago
    I work in the same field for one of my company, in europe, and if you're not self hosting sorry but your worries are not something I can accept because models are very much not reliable on that front, let alone when you let the host decide HOW to serve a model (ressources allocated, different version of the same model, etc ...). I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.
    • frde_me 25d ago
      It's not that you're a d*, it's just that you lack any kind of nuance There's a whole spectrum between self-hosting open weight models and having a cloud provider swap models from under you Should you self host a model if want to maximize predictability to the limit? Yes. Does that mean it's wrong for someone hitting a model on API to expect that it won't switch to a completely different model under the hood from one day to another? Probably not.
      • nolok 25d ago
        Sure, but if you're working in a field where that limitation is not "because you like it" but "because you have to" it doesn't matter. If you cannot assume it to be true, then you have to assume it isn't.
    • lkois 24d ago
• WhyNotHugo 25d ago
  A dice roll is non-deterministic. Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic. That doesn't mean it's fine to just replace the dice mid-game.
  • disiplus 25d ago
    Sure, but they are replacing same generation model to another not just switching opus to gpt.
    • neodymiumphish 25d ago
      But it’s more like them swapping Opus 5 for Sonnet 5.5 or something. IMO OP’s right to argue against it.
• hyperpape 25d ago
  Crossing the street and Russian roulette both have non-deterministic risks of injury. And yet I would be bothered to find out that that on my way to work, I was playing Russian roulette by surprise.
  • nolok 25d ago
    Yes and if you decide to play not using a game rulebook but a website that call it "game A" you can't be shocked if "game A" switched from one to the other, even though the doc said opposite yesterday.
    • hyperpape 25d ago
      That would absolutely be a dick move on the part of the website and would confuse users. Also, in this case, the game name is not “Game A” but something like “Deep Seek v4 Pro”, which they have previously chosen to use to describe Deep Seek v4 Pro, not Deep Seek v4.1 Flash.
    • genxy 25d ago
      You should look at why you are protecting your earlier comment rather than changing your mind or understanding why so many people disagree with you.
• gpugreg 25d ago
  > those things are not deterministic Determinism was an explicit goal of DeepSeek-V4. From their paper: https://arxiv.org/html/2606.19348v1#S3.SS3 https://arxiv.org/html/2606.19348v1#S3.SS3 > we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead Of course, providers may not implement deterministic inference for various reasons, but it is possible.
  • qeternity 25d ago
    Most people are not even referring to CUDA batching nuances. They think that sampling is an inherent part of Transformers. Even on this site, it is regurgitated with confidence.
• kristjansson 25d ago
  The user cares about the distribution of outputs. That distribution is structurally determined by the distribution of inputs (i.e. prompts), the weights of the model, and (these days) the dynamics of the harness guiding successive generations. The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution. But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.
• vikramkr 25d ago
  These models have specific behavioral characteristics trained into them from reinforcement learning and prompts optimized for one aren't guaranteed to transfer to the new generation. Think if the difference between gpt 5.4 and 5.5 and then 5.5 to 5.6 for example. 5.5 was "better" than 5.4 for struggled more across compaction boundaries and needed much more precise instructions before 5.6 sol recovered some of 5.4's ergonomics. All from the same lab but each model was trained with specific behavioral patterns that were basically product decisions. I would be quite annoyed to find that a model provider was routing a promt optimized for one model to a different one, especially for a dumber/cheaper non frontier model that's not going to be as good at just figuring out what you meant
• allenxu 24d ago
  It’s like being told one day that one of your teammates will be replaced tomorrow by another teammate who’s more capable (has a higher test score), regardless of how long you’ve already worked together and gotten used to each other.
• allenxu 24d ago
  In deterministic scenarios, this kind of switch can sometimes actually be safe, because we’ve used various engineering techniques to converge from non-deterministic behavior to deterministic decisions. On the other hand, in scenarios that are inherently non-deterministic, the impact of such a change is much harder to predict, so we need comprehensive evaluations to assess the extent of its impact.