3 ms·
But how would you test a closed model independent of their API? For example, the speed score (tokens/s) is also variable and changes over time.
by XCSme 6mo ago
But how would you test a closed model independent of their API? For example, the speed score (tokens/s) is also variable and changes over time.
- pertymcpert 6mo agoI really don't understand how you don't understand how your site is completely misleading. Everyone here is telling you that including API reliability in with actual model performance is nonsense.
- XCSme 6mo agoI agree that it's confusing, I have already implemented a reliability score, but it will only apply for new tests from now on. I have already re-tested DeepSeek v4, so it doesn't have any API error issues. API errors are quite rare, most models tested have usually max 1 API Error failure reason, so fixing them won't change rankings much: https://aibenchy.com/fail/api-error/ https://aibenchy.com/fail/api-error/ I will try to retest all with API errors, so the score is only given by correct/wrong answers, and the reliability score will be an extra metric just as an indication of how the API performs. That being said, the reliability of the API is still a huge factor for production use-cases.
- XCSme 6mo agoSome API errors are actually not about reliability, are because that specific API doesn't support some common features (e.g. specific structured_output formats).