3 ms·
Author is clearly confused about the Anthropic case. The request rate at these generation endpoints is so high that the current batching delay is effectively ne
by christina97 8mo ago
Author is clearly confused about the Anthropic case. The request rate at these generation endpoints is so high that the current batching delay is effectively negligible.