4 ms·
To what degree can you really trust these data? They obviously have a financial interest in playing with the numbers.
by suddenlybananas 2y ago
To what degree can you really trust these data? They obviously have a financial interest in playing with the numbers.
- chairhairair 2y agoI would say it is trust worthy because if it were found to be gamed then Anthropic’s reputation would crater. But, we found out that OpenAI is/was gaming benchmarks (https://news.ycombinator.com/item?id=42761648 https://news.ycombinator.com/item?id=42761648) and that seems to be forgotten history now - so I don’t know.
- diggan 2y ago> I would say it is trust worthy because if it were found to be gamed then Anthropic’s reputation would crater. But on the other hand, how would we found out that they've gamed the numbers, if they were gamed? Unless you work at Anthropic and have abnormally high ethics/morals, or otherwise private insight into their business, sounds like we wouldn't be able to find out regardless.
- marcosdumay 2y agoHave you noticed that the numbers are all relative? Unless they are trying to mislead competitors (who don't look at their own numbers...), they have no reason at all to game those numbers there.