3 ms·
Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larg
by jw1224 3y ago
Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models.
It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.
- pbhjpbhj 3y agoSeems like a means to avoid DoS?
- jw1224 3y agoI think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).
- charcircuit 3y ago>that's what API key rate limits are for You don't need an API key to make a connection to the API.
- yossi_peti 3y agoSince it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.