4 ms·
I had to chuckle at this, its true of course that http as a standard is greatly abused and it is one of the big reasons that writing a web crawler is not just a
by ChuckMcM 12y ago
I had to chuckle at this, its true of course that http as a standard is greatly abused and it is one of the big reasons that writing a web crawler is not just a matter of calling GET again and again.
In providing APIs for third parties that are rate limited, our APIs return a '509' error to tell you that you've exceeded your query rate. Which if you just sleep for a bit and then send a request you'll get an answer, but no, as a 'soft' error too many developers just retry it immediately which is useless. Some folks want only 3 errors, 'ok', 'dead', or 'try again' and can't deal with any more subtlety than that.