2 ms·
Did Anthropic give you third-party benchmarks? Is that what you said to them? Yes, they're important, but the attitude is wrong.
by OutOfHere 4mo ago
Did Anthropic give you third-party benchmarks? Is that what you said to them? Yes, they're important, but the attitude is wrong.
- bloppe 4mo agoAnthropic always publishes 3p benchmarks every time they announce a new model
- MostlyStable 4mo agoAnd even if they didn't, they have a track record. Even if we did have benchmarks in this case I would still wait until people got there hands on it and formed a more holistic opinion.
- OutOfHere 4mo agoNo, stop right there. Anything published by Anthropic implicitly is not third party. For it to be third party, the third party has to be the one publishing it.
- bloppe 4mo agoWhen you're announcing a new model, typically, nobody else has benchmarked it yet, because it hasn't been released yet. You can still run 3p benchmarks on it and publish those results. If other parties later run the same benchmarks independently, and find major discrepancies, that would be a scandal.
- fwipsy 4mo agoFudging benchmarks is a cheap way to get attention. If the model is really that good, it will have plenty of attention soon enough.
- greenavocado 4mo agoYeah, what happened to that scam startup that alleged to have made a model context window breakthrough a few weeks ago?