4 ms·

Why would your users post the addresses of your honeypots and tarpits to a 3rd party website?

• fnulp 14y ago
  What? I simply meant that if some brainiacs think robots.txt can just be disregarded, it's time to make it a minimum requirement of every self-respecting webmaster to make a tarpit (disallowed in robots.txt) and ban any and all bots going there. You would exactly NOT want a human visitor to post, or ever see, such a link. So yeah, it wouldn't even apply to this github thing, but don't tell that other guy about it. These are supposedly good guys. So my reaction was "You gotta be fucking kidding?! You didn't just say that it's inconvient how some sites use robots.txt, so you just throw it out altogether for your precious little bot and epically important link checking quest. No wait, you did. Oh well then, BYE." Oh well. I guess this is hack news, not hacker news, my bad :P
  • rsofaer 14y ago
    The sort of tarpit you're talking about wouldn't even affect this link validator. You really think Stack Exchange should have given up on validating links because Github's robots.txt has: User-agent: * Disallow: / in it?
    • lazugod 14y ago
      They could ask for Github's permission.
      • JasonPunyon 14y ago
        Yeah, we did. http://meta.stackoverflow.com/a/132677/6212 http://meta.stackoverflow.com/a/132677/6212