3 ms·
Have you contacted them and asked them to stop? If they continue, then you start to have legal rights. Figure out how to uniquely identify them, use mod_rewri
by cd34 16y ago
Have you contacted them and asked them to stop? If they continue, then you start to have legal rights.
Figure out how to uniquely identify them, use mod_rewrite or an equivalent if you can determine a UserAgent or something unique. mod_evasive perhaps.
Find a way to automate your pattern searches, i.e. x hits to site without fetching a single image or static asset, must be bot, deny.
80legs was doing this to a client, numerous requests to get them to stop hammering a site went unanswered. Same with cuil - though, cuil's bots finally got fixed.
- jdrock 16y agoShion from 80legs here. Please try contacting us again and I'll make sure that we adjust our crawl rate file accordingly for your domain(s) or IP(s).