3 ms·
Perhaps Fedbot crawls in a less deterministic manner, uses a lot of different ips, and sets user agent to IE?
by wisty 15y ago
Perhaps Fedbot crawls in a less deterministic manner, uses a lot of different ips, and sets user agent to IE?
- bigiain 15y agoI suspect "fedbot" works by calling up google and saying "Hi, it's us again, we've got another white van on the way to the googleplex, have a petabyte or two of the Internet ready for us to collect in 20 minutes. thanks"