3 ms·
Ask HN: Agents get dumber before release of new model version?
I've noticed an effect with openai where my codex agents seem to perform worse in the week(s) leading up to a new release. I'm wondering if the vendors tweak effort params at all to free up hardware to host the new version. It's a double win as the new model will look night and day better to their regular users when the new model is released presumably with effort back at normal levels.
Is this a known phenomenon? Are any folks trying to measure any of this objectively?
- gauravvij137 4mo ago[dead]
- Fizzadar 4mo agoReminds me of a similar effect around iPhone releases (https://www.statista.com/chart/2514/iphone-releases/ https://www.statista.com/chart/2514/iphone-releases/, many others). Can’t say I’ve noticed any change.
- cyanydeez 3mo agomore likely there's some pressure on GPU vs CPU compute resources while the AI company tries to scale out a new model, leading to some type of degradation. Assuming it's not apple like fraud.
- suprjami 4mo agoYes, this is already a widely circulated unproven LLM conspiracy theory.
- sporkland 4mo agoWas hoping to understand if anyone has done any research on it. I could only really find one research paper on the topic [1], but it seems less focused on this specific issue. [1] https://arxiv.org/abs/2307.09009 https://arxiv.org/abs/2307.09009
- sama004 4mo agotbh doesn't it depend on the service provider entirely, its their choice if they want to make it dumber or not open source models winning is the only deliberate solution to this
- faangguyindia 4mo agoyes, when they come with 5.5 5.3 becomes 5.4 and optimization and improvement to 5.4 are provided as new 5.5 this gives boost effect via anchor/decoy.
- PreownedPlaid 3mo agoI figure it's because limited compute, must redirect some for latest product
- aurenvale 3mo ago[flagged]