3 ms·
I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see
by dpweb 12y ago
I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see your ads or sign up for your deal, and just inflate your traffic numbers (unless you find that desirable).
There are a number of ways to try to detect and trick them out. I've done some unusual things to attempt to detect them, finding people coming thru proxies as well.
Sure, these simple methods can be defeated by an adversary, but at least judging from my experience, even simple methods you're going to catch some of them. Some hackers ain't that smart. A little surpised Cloudflare for instance wouldn't offer this (PhantomJS blocking), if they don't already. I'm all for hacking, but a site owner you should have the right to block bots if you so choose.
- jalfresi 12y agoIf you don't want it crawled, don't put it on the public internet. You're just participating in an arms race.