7 ms·
but degradation from servers being overloaded would be the type of degradation this SHOULD measure no? Unless it's only intended for measuring their quietly dis
by Davidzheng 9mo ago
but degradation from servers being overloaded would be the type of degradation this SHOULD measure no? Unless it's only intended for measuring their quietly distilling models (which they claim not to do? idk for certain)
- cmrdporcupine 9mo agoI've personally witnessed large variability in behaviour even within a given session -- which makes sense as there's nothing stopping Anthropic from shuttling your context/session around load balanced through many different servers, some of which might be quantized heavily to manage load and others not at all. I don't know if they do this or not, but the nature of the API is such you could absolutely load balance this way. The context sent at each point is not I believe "sticky" to any server. TLDR you could get a "stupid" response and then a "smart" response within a single session because of heterogeneous quantization / model behaviour in the cluster.
- epolanski 9mo agoI've defended opus in the last weeks but the degradation is tangible. It feels like it degraded by a generation tbh.
- cmrdporcupine 9mo agoit's just extremely variable
- megabless123 9mo agonoob question: why would increased demand result in decreased intelligence?
- vidarh 9mo agoIt would happen if they quietly decide to serve up more aggressively distilled / quantised / smaller models when under load.
- chrisjj 9mo agoThey advertise the Opus 4.5 model. Secretly substituting a cheaper one to save costs would be fraud.
- kingstnap 9mo agoOld school Gemini used to do this. It was super obvious because mid day the model would go from stupid to completely brain dead. I have a screenshot of Google's FAQ on my PC from 2024-09-13 that says this (I took it to post to discord): > How do I know which model Gemini is using in its responses? > We believe in using the right model for the right task. We use various models at hand for specific tasks based on what we think will provide the best experience.
- chrisjj 9mo ago> We use various models at hand for specific tasks based on what we think will provide the best experience ... for Google :)
- vidarh 9mo agoIf you use the API, you pay for a specific model, yes, but even then there are "workarounds" for them, such as someone else pointed out by reducing the amount of time they let it "think". If you use the subscriptions, the terms specifically says that beyond the caps they can limit your "model and feature usage, at our discretion".
- chrisjj 9mo agoSure. I was separating the model - which Anthropic promises not to downgrade - and the "thinking time" - which Anthropic doesn't promise not to downgrade. It seems the latter is very likely the culprit in this case.
- seunosewa 9mo agoOr just reducing the reasoning tokens.
- Wheaties466 9mo agofrom what I understand this can come from the batching of requests.
- chrisjj 9mo agoSo, a known bug?
- embedding-shape 9mo agoNo, basically, the requests are processed in batches, together, and the order they're listed in matters for the results, as the grid (tiles) that the GPU is ultimately processing, are different depending on what order they entered at. So if you want batching + determinism, you need the same batch with the same order which obviously don't work when there are N+1 clients instead of just one.
- chrisjj 9mo agoSure, but how can that lead to increased demand resulting in decreased intelligence? That is the effect we are discussing.
- embedding-shape 9mo agoSmall subtle errors that are only exposed at certain execution parts could be one. You might place things differently onto the GPU depending on how large the batch is, if you've found one way to be faster batch_size<1024, but another when batch_size>1024. As number of concurrent incoming requests goes up, you increase batch_size. Just one possibility, guess there could be a multitude of reasons, as it's really hard to reason about until you sit with the data in front of you. vLLM has had bugs with these sort of thing too, so wouldn't surprise me.
- chrisjj 9mo agoWouldn't you think that was as likely to increase as decrease intelligence, so average to nil in the benchmarks?
- awestroke 9mo agoI've seen some issues with garbage tokens (seemed to come from a completely different session, mentioned code I've never seen before, repeated lines over and over) during high load, suspect anthropic have some threading bugs or race conditions in their caching/inference code that only happen during very high load
- exitb 9mo agoAn operator at load capacity can either refuse requests, or move the knobs (quantization, thinking time) so requests process faster. Both of those things make customers unhappy, but only one is obvious.
- codeflo 9mo agoThis is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.
- chrisjj 9mo agoThere is no level of quality advertised, as far as I can see.
- pseidemann 9mo agoWhat is "level of quality"? Doesn't this apply to any product?
- chrisjj 9mo agoIn this case, it is benchmark performance. See the root post.
- denysvitali 9mo agoIf there's no way to check, then how can you claim it's fraud? :)
- mcny 9mo agoPersonally, I'd rather get queued up on a long wait time I mean not ridiculously long but I am ok waiting five minutes to get correct it at least more correct responses. Sure, I'll take a cup of coffee while I wait (:
- lurking_swe 9mo ago
- botacode 9mo agoLoad just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.
- bgirard 9mo ago> malicious It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me. I care about -expected- performance when picking which model to use, not optimal benchmark performance.
- Aurornis 9mo agoNon-determinism isn’t the same as degradation. The non-determinism means that even with a temperature of 0.0, you can’t expect the outputs to be the same across API calls. In practice people tend to index to the best results they’ve experienced and view anything else as degradation. In practice it may just be randomness in either direction from the prompts. When you’re getting good results you assume it’s normal. When things feel off you think something abnormal is happening. Rerun the exact same prompts and context with temperature 0 and you might get a different result.
- dingnuts 9mo ago[dead]
- bonoboTP 9mo agoThis has nothing to do with overloading. The suspicion is that when there is too much demand (or they just want to save costs), Anthropic sometimes uses a less capable (quantized, distilled, etc) version of the model. People want to measure this so there is concrete evidence instead of hunches and feelings. To say that this measurement is bad because the server might just be overloaded completely misses the point. The point is to see if the model sometimes silently performs worse. If I get a response from "Opus", I want a response from Opus. Or at least want to be told that I'm getting slightly-dumber-Opus this hour because the server load is too much.
- samusiam 9mo agoI'd argue that it depends how that degradation manifests whether you want to include it or not. Consider two scenarios: (1) degradation leads to the model being routed behind the scenes to a different server, with subtly different performance characteristics, all unbeknownst to the user; (2) degradation leads to the model refusing a request and returning an "overloaded" message. In the first case, absolutely you want to include that because that's the kind of lack of transparency about performance that you'd want signal on. In the second case, an automated test harness might fail, but in the real world the user will just wait and retry when the server is under less load. Maybe you don't include that because it's actually misleading to say that performance (in terms of the model's intelligence, which is how the benchmark will be interpreted) is worse.