3 ms·
When any crawler spams you, but most bots do not spam you. They retrieve something at some reasonable rate. Btw cloudflare considers any browser other than the
by immibis 2y ago
When any crawler spams you, but most bots do not spam you. They retrieve something at some reasonable rate.
Btw cloudflare considers any browser other than the last few versions of Firefox, Safari and Chrome to be a bot. Such as Pale Moon. It blew up on Hacker News when CF demanded the PM devs to sign an insane NDA to get unblocked.