3 ms·
For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour.
by Ndymium 7mo ago
For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.
- antonyh 7mo agoAs facebookexternalhit is listed in the robots.txt, it does look like it's optimistically rechecking in the hope it's no longer disallowed. That rate of request is obscene though, and falls firmly into the category of Bad Bot.
- mghackerlady 7mo agoThat is probably the dumbest yet most genius solution to getting your scraper blocked I've ever seen
- RobotToaster 7mo agoMy guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.