2 ms·
"We wished to also evaluate Claude 3.7 Sonnet, but were unable to complete the experiments given rate limits with the Anthropic API."
by attentive 2y ago
"We wished to also evaluate Claude 3.7 Sonnet, but were unable to complete the experiments given rate limits with the Anthropic API."
- swyx 2y agooverall i REALLY like this paper and effort, but this part sounds like a bit of bullshit. they dont have the ability to implement retries and backoffs to deal with rate limits?
- eightysixfour 2y agoBecause they used wall clock time, not compute time, flops, or watts, to standardize. 24 hours and 36 hours of compute. They could build a system which gives them equal compute time by ignoring time spent rate limiting and such, but they chose not to.
- swyx 2y agoah. fair answer.
- moralestapia 2y ago"Why don't they just break the TOS" Damned if you do, damned if you don't.