3 ms·

I set up a robots.txt with one entry: a disallow for a specific html file. The html file is not listed elsewhere and thus can only be discovered by the robots.t

I set up a robots.txt with one entry: a disallow for a specific html file. The html file is not listed elsewhere and thus can only be discovered by the robots.txt. The file features random stuff and an <a href=""> element with the nofollow attribute and a link target set. That link has no text contents, no size, and thus doesn't occupy pixels that a user could click on. When clicked, that link would report the http request to me. At 2025-02-22 02:14:52, less than a day after i set up this honeypot, 66.249.68.37 (AS15169 GOOGLE) made a request on that invisible link, using user agent string "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Mobile Safari/537.36 (compatible; GoogleOther)" So now i have hard evidence that google runs bots that ignore the rules from the robots.txt, but still parses urls from there, queries these resources anyways and follows links marked as "nofollow".

bayindirh 2y ago
It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore). It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.
- pimlottc 2y ago
  It’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
  - johnea 2y ago
    Yes, it is, and now we know both of those things don't we. What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets. And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.
extraduder_ire 2y ago
Did any other crawlers hit it? Would be nice to have someone keeping track of this over time and publishing the data online somewhere. Probably need to collect logs across multiple sites though, as some crawlers would exclude your domain once they knew.
- blueflow 2y ago
  Only one other hit, 2025-03-05 13:32:09 45.89.148.57 (AS46844 SHARKTECH) "Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-BR; rv:1.9.2.18) Gecko/20110614 Firefox/3.6.18 (.NET CLR 3.5.30729)"
acdw 2y ago
could you share the code for that? thinking I could set up a fail2ban rule for these ips
- blueflow 2y ago
  its a pathetic piece of php code: $db=... $sth=$db->prepare("INSERT INTO beacon (time, addr, query, agent, referer) VALUES (FROM_UNIXTIME(?), ?, ?, ?, ?)"); $sth->execute([ $_SERVER["REQUEST_TIME"], $_SERVER["REMOTE_ADDR"], $_SERVER["QUERY_STRING"], isset($_SERVER["HTTP_USER_AGENT"])?$_SERVER["HTTP_USER_AGENT"]:NULL, isset($_SERVER["HTTP_REFERER"])?$_SERVER["HTTP_REFERER"]:NULL ]);
  - bbkane 2y ago
    Pathetic? Looks nice and straightforward to me!

bayindirh 2y ago

It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore). It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.

pimlottc 2y ago
It’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
- johnea 2y ago
  Yes, it is, and now we know both of those things don't we. What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets. And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.

extraduder_ire 2y ago

Did any other crawlers hit it? Would be nice to have someone keeping track of this over time and publishing the data online somewhere. Probably need to collect logs across multiple sites though, as some crawlers would exclude your domain once they knew.

blueflow 2y ago
Only one other hit, 2025-03-05 13:32:09 45.89.148.57 (AS46844 SHARKTECH) "Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-BR; rv:1.9.2.18) Gecko/20110614 Firefox/3.6.18 (.NET CLR 3.5.30729)"

acdw 2y ago

could you share the code for that? thinking I could set up a fail2ban rule for these ips

blueflow 2y ago
its a pathetic piece of php code: $db=... $sth=$db->prepare("INSERT INTO beacon (time, addr, query, agent, referer) VALUES (FROM_UNIXTIME(?), ?, ?, ?, ?)"); $sth->execute([ $_SERVER["REQUEST_TIME"], $_SERVER["REMOTE_ADDR"], $_SERVER["QUERY_STRING"], isset($_SERVER["HTTP_USER_AGENT"])?$_SERVER["HTTP_USER_AGENT"]:NULL, isset($_SERVER["HTTP_REFERER"])?$_SERVER["HTTP_REFERER"]:NULL ]);
- bbkane 2y ago
  Pathetic? Looks nice and straightforward to me!