4 ms·
In addition to crawl-delay there's also HTTP 429 and the retry-after header. https://tools.ietf.org/html/rfc6585#page-3 https://tools.ietf.org/html/rfc6585#pag
by the8472 7y ago
In addition to crawl-delay there's also HTTP 429 and the retry-after header.
https://tools.ietf.org/html/rfc6585#page-3 https://tools.ietf.org/html/rfc6585#page-3
- greglindahl 7y agoSites also use 403 and 503 to send rate-limit signals, despite what the RFCs say.