3 ms·

> to create a perceived improvement when in reality there isn’t really one? This wouldn't explain progress on benchmarks (including closed sets), or the fact t

> to create a perceived improvement when in reality there isn’t really one? This wouldn't explain progress on benchmarks (including closed sets), or the fact that newer models are providing solutions to major math problems that older models cannot.

pixl97 11d ago
Far more likely it's about reducing costs.
well_ackshually 11d ago
* Release new model that scores an arbitrary 100 on a benchmark * Get everyone to talk about you as the first model to ever score 100 on the 100benchmark. * Tune it down over time so that you end up only scoring 75 on the benchmark and people get used to it, gaslight them into thinking it never changed or that it's just a harness problem, they can't run the old version locally anyways to verify. This also cuts your costs in half. Your gross margin on API calls goes from 70% to 150%. * Release new model that scores 120 on the benchmark and advertise it as 50% better than the current model, while it's only in practice a minor increment. Everyone praises it as the second coming of Jesus Christ. * Get everyone to talk about you as the first model to ever score 120 on the 100benchmark. Bis repetitae.
- scrollop 11d ago
  Why can't the models be benchmarked again after a few weeks/months to confirm this (likely true) theory? I imagine some people have their own personal in depth benchmarks they could do this for.
- Aurornis 11d ago
  > * Tune it down over time so that you end up only scoring 75 on the benchmark Where? I see so many accusations of this happening and it's so easy to check, but nobody ever proves it.
mobelkh 11d ago
but there is a gap between benchmarks and user feel. Opus 5 came out with better benchmark results than Fable, but it really did not feel better to use at all.
talon8635 11d ago
This is a good point I hadn’t considered, thank you. Is there any training variable here? For example, can a model released in October perform better on the same benchmarks vs its predecessor released in July just by virtue of training on newer data that was made available on those 3 months? Sorry if it’s a dumb question, I don’t really know much about the topic.

pixl97 11d ago

Far more likely it's about reducing costs.

well_ackshually 11d ago

* Release new model that scores an arbitrary 100 on a benchmark * Get everyone to talk about you as the first model to ever score 100 on the 100benchmark. * Tune it down over time so that you end up only scoring 75 on the benchmark and people get used to it, gaslight them into thinking it never changed or that it's just a harness problem, they can't run the old version locally anyways to verify. This also cuts your costs in half. Your gross margin on API calls goes from 70% to 150%. * Release new model that scores 120 on the benchmark and advertise it as 50% better than the current model, while it's only in practice a minor increment. Everyone praises it as the second coming of Jesus Christ. * Get everyone to talk about you as the first model to ever score 120 on the 100benchmark. Bis repetitae.

scrollop 11d ago
Why can't the models be benchmarked again after a few weeks/months to confirm this (likely true) theory? I imagine some people have their own personal in depth benchmarks they could do this for.
Aurornis 11d ago
> * Tune it down over time so that you end up only scoring 75 on the benchmark Where? I see so many accusations of this happening and it's so easy to check, but nobody ever proves it.

mobelkh 11d ago

but there is a gap between benchmarks and user feel. Opus 5 came out with better benchmark results than Fable, but it really did not feel better to use at all.

talon8635 11d ago

This is a good point I hadn’t considered, thank you. Is there any training variable here? For example, can a model released in October perform better on the same benchmarks vs its predecessor released in July just by virtue of training on newer data that was made available on those 3 months? Sorry if it’s a dumb question, I don’t really know much about the topic.