4 ms·
How would you know if the robot is autonomous or not? They could just use a typical browser user-agent and you wouldn't know...
by catch23 15y ago
How would you know if the robot is autonomous or not? They could just use a typical browser user-agent and you wouldn't know...
- tlrobinson 15y agoHide the link from users?
- pyre 15y agoI seem to remember there being a thread on HN about spam bot scraping for email addresses, and some of them are sophisticated enough that they can even determine if an element has been hidden through rules applied via an external CSS file.
- Aloisius 15y agoLink a 1 pixel image and stick it in the bottle left hand corner? While someone could click on it, it is extremely unlikely. IIRC, technically you can have a 0 pixel image in some formats (gif) though I imagine some browsers won't like that.
- deleted 15y ago[deleted]
- autoreverse 15y agoUsing a 1 pixel transparent GIF works - also worth putting in a warning for screen readers not to follow the link. FWIW I've been using robots.txt honeypots on two sites for the past few years to deter scrapers. Approx 1,600 IP addresses have been banned. Most of these are for data centers.
- derpy 15y agoFuck you guy, that's pretty pricky. Is bandwidth that expensive for you?
- migrantgeek 15y agoThat's uncalled for. If he owns the data and pays for the connectivity, he should be free to do as he pleases just like I can exclude you from my home or business without justification.
- autoreverse 15y agoI like humans browsing my sites but I certainly don't want scrapers copying my work wholesale, claiming it as their own and ranking ahead of me in SERPs. I'm not alone :-) see http://incredibill.blogspot.com/ http://incredibill.blogspot.com/
- eli 15y agoThis is what Project Honeypot does http://www.projecthoneypot.org/faq.php#f http://www.projecthoneypot.org/faq.php#f