4 ms·
Here's a tricky one. Common Crawl runs its bots from AWS. AWS has like a jillion IP addresses. How do you tell which traffic are legit Common Crawl bots and w
by jgalt212 5y ago
Here's a tricky one. Common Crawl runs its bots from AWS. AWS has like a jillion IP addresses. How do you tell which traffic are legit Common Crawl bots and which are imposters?
- ComputerGuru 5y agoI mean, they’re all bots one way or the other. The only exception would be a personal VPN running off of AWS, but that’s a bad idea given how many sites block that range.