5 ms·

This comment thread is a good learner for founders; look at how much anguish can be put to bed with just a little honest communication. 1. Oops, we're oversubs

This comment thread is a good learner for founders; look at how much anguish can be put to bed with just a little honest communication. 1. Oops, we're oversubscribed. 2. Oops, adaptive reasoning landed poorly / we have to do it for capacity reasons. 3. Here's how subscriptions work. Am I really writing this bullet point? As someone with a production application pinned on Opus 4.5, it is extremely difficult to tell apart what is code harness drama and what is a problem with the underlying model. It's all just meshed together now without any further details on what's affected.

drewnick 6mo ago
Hasn't Opus 4.5 been famously consistent while 4.6 was floating all over the place?
- JohnMakin 6mo ago
  I'm still on 4.5. My coworkers are describing a lot of problems I just don't have. I suspect it was some combination of the larger context window, the model itself, and various bugs like the cache miss thing reported a little while ago.
- YZF 6mo ago
  For me 4.6 has been a noticeable leap in performance from 4.5. I'm not missing 4.5 at all.
kulikalov 6mo ago
Or it could be a selection bias. The ground truth is not what HN herd mentality complains about, but the usage stats.
- lanyard-textile 6mo ago
  I suppose I come forward with my own usage stats, but it is anecdata :) And the andecdata matches other anecdata. Maybe I'm missing why that's selection bias.
zarzavat 6mo ago
These threads are always full of superstitious nonsense. Had a bad week at the AIs? Someone at Anthropic must have nerfed the model! The roulette wheel isn't rigged, sometimes you're just unlucky. Try another spin, maybe you'll do better. Or just write your own code.
- delbronski 6mo ago
  Nah dude, that roulette wheel is 100% rigged. From top to bottom. No doubt about that. If you think they are playing fair you are either brand new to this industry, or a masochist.
- unshavedyak 6mo ago
  Part of me wonders if there's some subtle behavioral change with it too. Early on we're distrusting of a model and so we're blown away, we were giving it more details to compensate for assumed inability, but the model outperformed our expectations. Weeks later we're more aligned with its capabilities and so we become lazy. The model is very good, why do we have to put in as much work to provide specifics, specs, ACs, etc. So then of course the quality slides because we assumed it's capabilities somehow absolved the need for the same detailed guardrails (spec, ACs, etc) for the LLM. This scenario obviously does not apply to folks who run their own benches with the same inputs between models. I'm just discussing a possible and unintentional human behavioral bias. Even if this isn't the root cause, humans are really bad at perceiving reality. Like, really really bad. LLMs are also really difficult to objectively measure. I'm sure the coupling of these two facts play a part, possibly significant, in our perception of LLM quality over time.
  - mewpmewp2 6mo ago
    Still I don't previously remember Claude constantly trying to stop conversations or work, as in "something is too much to do", "that's enough for this session, let's leave rest to tomorrow", "goodbye", etc. It's almost impossible to get it do refactoring or anything like that, it's always "too massive", etc.
    - 6mo ago
teling 6mo ago
Good shout. Wish they were more transparent about these 3 things.
stasomatic 6mo ago
I am a neophyte regarding pros and cons of each model. I am learning the ropes, writing shell scripts, a tiny Mac app, things like that. Reading about all the “rage switching”, isn’t it prudent to use a model broker like GH Copilot with your own harness or something like oh-my-pi? The frontier guys one up each other monthly, it’s really tiring. I get that large corps may have contracts in place, but for an in indie?
- smw 6mo ago
  Unfortunately, the subscription pricing is so much cheaper than usage pricing that it's probably worth using one of the official harnesses.
sobellian 6mo ago
This, plus the alchemical nature of these tools, seems to have made users pretty paranoid (I admit I am also guilty of paranoia). Maybe there's room for a Standard AI - we may change the prices based on market conditions, but we always give you exactly the model you ask for.
preommr 6mo ago
> This comment thread is a good learner for founders; lmao, no they shouldn't. Public sentiment, especially on reactionary mediums like social media should be taken with a huge grain of salt. I've seen overwhelming negativity for products/companies, only for it it completely dissapear, or be entirely wrong. It's like that meme showing members of a steam group that are boycotting some CoD game, and you can see that a bunch of them were playing in-game of the very thing they forsook. People are fickle, and their words cheap.
- lanyard-textile 6mo ago
  The internet is a stupid place with people who can't make up their mind, I don't disagree :) But this isn't like a minor debacle about a brand. The flagship product had a severe degradation, and the parent company won't be forthcoming about it. It's short term thinking. Congratulations, everyone still uses your product for now, but it diluted your brand. Why take the risk when the alternative is so incredibly easily? Build engagement with your users and enjoy your loyal army.
Barbing 6mo ago
This is why we took business ethics & I know Dario had to too How will your project/decision look on the front page of the Wall Street Journal? Well when a whistleblower reveals what everyone knows ($9b->$30b rev jump w/o servers growing on trees simultaneously = tough decisions), it's gonna be public anyway.
SkyPuncher 6mo ago
I agree. I have flexibility to shift my core working hours (and what I do during N/A business hours). Knowing they're explicitly making it dumb because of load is important. It allows me to shuffle my work around and run heavy workloads late at night (plan during working hours then come click "yes" a few times in the evening).

drewnick 6mo ago

Hasn't Opus 4.5 been famously consistent while 4.6 was floating all over the place?

JohnMakin 6mo ago
I'm still on 4.5. My coworkers are describing a lot of problems I just don't have. I suspect it was some combination of the larger context window, the model itself, and various bugs like the cache miss thing reported a little while ago.
YZF 6mo ago
For me 4.6 has been a noticeable leap in performance from 4.5. I'm not missing 4.5 at all.

kulikalov 6mo ago

Or it could be a selection bias. The ground truth is not what HN herd mentality complains about, but the usage stats.

lanyard-textile 6mo ago
I suppose I come forward with my own usage stats, but it is anecdata :) And the andecdata matches other anecdata. Maybe I'm missing why that's selection bias.

zarzavat 6mo ago

These threads are always full of superstitious nonsense. Had a bad week at the AIs? Someone at Anthropic must have nerfed the model! The roulette wheel isn't rigged, sometimes you're just unlucky. Try another spin, maybe you'll do better. Or just write your own code.

delbronski 6mo ago
Nah dude, that roulette wheel is 100% rigged. From top to bottom. No doubt about that. If you think they are playing fair you are either brand new to this industry, or a masochist.
unshavedyak 6mo ago
Part of me wonders if there's some subtle behavioral change with it too. Early on we're distrusting of a model and so we're blown away, we were giving it more details to compensate for assumed inability, but the model outperformed our expectations. Weeks later we're more aligned with its capabilities and so we become lazy. The model is very good, why do we have to put in as much work to provide specifics, specs, ACs, etc. So then of course the quality slides because we assumed it's capabilities somehow absolved the need for the same detailed guardrails (spec, ACs, etc) for the LLM. This scenario obviously does not apply to folks who run their own benches with the same inputs between models. I'm just discussing a possible and unintentional human behavioral bias. Even if this isn't the root cause, humans are really bad at perceiving reality. Like, really really bad. LLMs are also really difficult to objectively measure. I'm sure the coupling of these two facts play a part, possibly significant, in our perception of LLM quality over time.
- mewpmewp2 6mo ago
  Still I don't previously remember Claude constantly trying to stop conversations or work, as in "something is too much to do", "that's enough for this session, let's leave rest to tomorrow", "goodbye", etc. It's almost impossible to get it do refactoring or anything like that, it's always "too massive", etc.
  - 6mo ago

teling 6mo ago

Good shout. Wish they were more transparent about these 3 things.

stasomatic 6mo ago

I am a neophyte regarding pros and cons of each model. I am learning the ropes, writing shell scripts, a tiny Mac app, things like that. Reading about all the “rage switching”, isn’t it prudent to use a model broker like GH Copilot with your own harness or something like oh-my-pi? The frontier guys one up each other monthly, it’s really tiring. I get that large corps may have contracts in place, but for an in indie?

smw 6mo ago
Unfortunately, the subscription pricing is so much cheaper than usage pricing that it's probably worth using one of the official harnesses.

sobellian 6mo ago

This, plus the alchemical nature of these tools, seems to have made users pretty paranoid (I admit I am also guilty of paranoia). Maybe there's room for a Standard AI - we may change the prices based on market conditions, but we always give you exactly the model you ask for.

preommr 6mo ago

> This comment thread is a good learner for founders; lmao, no they shouldn't. Public sentiment, especially on reactionary mediums like social media should be taken with a huge grain of salt. I've seen overwhelming negativity for products/companies, only for it it completely dissapear, or be entirely wrong. It's like that meme showing members of a steam group that are boycotting some CoD game, and you can see that a bunch of them were playing in-game of the very thing they forsook. People are fickle, and their words cheap.

lanyard-textile 6mo ago
The internet is a stupid place with people who can't make up their mind, I don't disagree :) But this isn't like a minor debacle about a brand. The flagship product had a severe degradation, and the parent company won't be forthcoming about it. It's short term thinking. Congratulations, everyone still uses your product for now, but it diluted your brand. Why take the risk when the alternative is so incredibly easily? Build engagement with your users and enjoy your loyal army.

Barbing 6mo ago

This is why we took business ethics & I know Dario had to too How will your project/decision look on the front page of the Wall Street Journal? Well when a whistleblower reveals what everyone knows ($9b->$30b rev jump w/o servers growing on trees simultaneously = tough decisions), it's gonna be public anyway.

SkyPuncher 6mo ago

I agree. I have flexibility to shift my core working hours (and what I do during N/A business hours). Knowing they're explicitly making it dumb because of load is important. It allows me to shuffle my work around and run heavy workloads late at night (plan during working hours then come click "yes" a few times in the evening).