4 ms·
That is a good hypothesis but then you’d imagine they could combine it with registered clients to keep crawlers accountable.
by ksajadi 5mo ago
That is a good hypothesis but then you’d imagine they could combine it with registered clients to keep crawlers accountable.
- brudgers 5mo ago‘Keeping crawlers accountable’ sounds like whack-a-mole. At least to me because benign crawling is likely to be a tiny subset of all crawling activity that is not done by google, etc.