2 ms·
The big fish (experts exchange etc...) are big enough to be caught manually. Programatically they can crawl with a typical browser user agent, from a netblock n
by cbs 14y ago
The big fish (experts exchange etc...) are big enough to be caught manually. Programatically they can crawl with a typical browser user agent, from a netblock not registered to google nor announced by their ASes. They'd need to account for the dynamic nature of sites to determine if the difference between different crawls is a sign of trouble, but as luck have it they're already in the business of algorithms and archiving websites.
- Steko 14y agoI'm sure they'd do it algorithmically both for elegance and legal liability reasons. It'd be done in an identical fashion to the Panda update. Use the big fish to identify techniques you can build rules around and then make sure your arbitrary penalty doesn't affect too many innocents too much before you push it live.