4 ms·

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404, which then meant I continued to crawl that site. I had to change the code so a robots.txt timeout is treated like a Disallow /. It feels odd because I find I'm writing code to detect anti-bot tools even though I'm trying my best to follow conventions.

navane 1y ago
That's like deterring burglars by hiding your doorbell
brianwawok 1y ago
I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it. Never assume malice what can be attributed to incompetence.
- cyanydeez 1y ago
  Its likely just a shitty attempt to rate limit bots
- aequitas 1y ago
  Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.
NegativeK 1y ago
I really appreciate you giving a shit. Not sarcastically -- it seems like you're actually doing everything right, and it makes a difference. Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and slow it down. It's crap that it's affecting your work, but a website owner isn't likely to care about the distinction when they're pissed off at having to deal with bad actors that they should never have to care about.
- gabeio 1y ago
  > It's also a page that's never visited by humans. Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random reasons. - remembering the format - seeing what they might have tried to "hide" - using it like a site's directory - testing if the website is working if their main dashboard/index is offline
  - sdenton4 1y ago
    Are you sure you are human?
    - gspencley 1y ago
      Yes. I have checked many checkboxes that say "Verify You Are a Human" and they have always confirmed that I am. In fairness, however, my daughters ask me that question all the time and it is possible that the verification checkboxes are lying to me as part of some grand conspiracy to make me think I am a human when I am not.
      - nullc 1y ago
        https://www.youtube.com/watch?v=4VrLQXR7mKU https://www.youtube.com/watch?v=4VrLQXR7mKU --- though I think passing them is more a sign that you're a robot than anything else.
- ghxst 1y ago
  I usually hit robots.txt when I want to make fetch requests to a domain from the console without running into CORS or CSP issues. Since it's just a static file, there's no client-side code interfering, which makes it nice for testing. If you're hunting for vulnerabilities it's also worth probing (especially with crawler UAs), since it can leak hidden endpoints or framework-specific paths that devs didn't expect anyone to notice.
ronsor 1y ago
> The latest was a slow loris approach where it takes forever for robots.txt to download. I'd treat this in a client the same way as I do in a server application. If the peer is behaving maliciously or improperly, I silently drop the TCP connection without notifying the other party. They can waste their resources by continuing to send bytes for the next few minutes until their own TCP stack realizes what happens.
- conradludgate 1y ago
  How do you silently drop a TCP connection? Closing the socket fd usually results in a FIN packet being sent whether I want it to or not. Additionally, it's not going to be using that many resources before your kernel sends it a RST next time a data packet is sent
  - ronsor 1y ago
    TCP_REPAIR: https://tinselcity.github.io/TCP_Repair/ https://tinselcity.github.io/TCP_Repair/
fsckboy 1y ago
>a slow loris approach does this refer to the word loris recently and only after several years being added to Wordle™?
- bananananananan 1y ago
  No. A slowloris is an existing attack predating Wordle.
- jeltz 1y ago
  No, why would it? The attack was named after the animal, slow loris, many years ago.
jandrese 1y ago
> The latest was a slow loris approach where it takes forever for robots.txt to download Applying penalties that exclusively hurt people who are trying to be respectful seems counterproductive.
Snacklive 1y ago
would you be interested in writing an article about it ? sounds really interesting
- 8organicbits 1y ago
  Yeah, seems like a good fit. It will end up here, if I get to it. https://alexsci.com/blog/ https://alexsci.com/blog/

navane 1y ago

That's like deterring burglars by hiding your doorbell

brianwawok 1y ago

I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it. Never assume malice what can be attributed to incompetence.

cyanydeez 1y ago
Its likely just a shitty attempt to rate limit bots
aequitas 1y ago
Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.

NegativeK 1y ago

I really appreciate you giving a shit. Not sarcastically -- it seems like you're actually doing everything right, and it makes a difference. Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and slow it down. It's crap that it's affecting your work, but a website owner isn't likely to care about the distinction when they're pissed off at having to deal with bad actors that they should never have to care about.

gabeio 1y ago
> It's also a page that's never visited by humans. Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random reasons. - remembering the format - seeing what they might have tried to "hide" - using it like a site's directory - testing if the website is working if their main dashboard/index is offline
- sdenton4 1y ago
  Are you sure you are human?
  - gspencley 1y ago
    Yes. I have checked many checkboxes that say "Verify You Are a Human" and they have always confirmed that I am. In fairness, however, my daughters ask me that question all the time and it is possible that the verification checkboxes are lying to me as part of some grand conspiracy to make me think I am a human when I am not.
    - nullc 1y ago
      https://www.youtube.com/watch?v=4VrLQXR7mKU https://www.youtube.com/watch?v=4VrLQXR7mKU --- though I think passing them is more a sign that you're a robot than anything else.
ghxst 1y ago
I usually hit robots.txt when I want to make fetch requests to a domain from the console without running into CORS or CSP issues. Since it's just a static file, there's no client-side code interfering, which makes it nice for testing. If you're hunting for vulnerabilities it's also worth probing (especially with crawler UAs), since it can leak hidden endpoints or framework-specific paths that devs didn't expect anyone to notice.

ronsor 1y ago

> The latest was a slow loris approach where it takes forever for robots.txt to download. I'd treat this in a client the same way as I do in a server application. If the peer is behaving maliciously or improperly, I silently drop the TCP connection without notifying the other party. They can waste their resources by continuing to send bytes for the next few minutes until their own TCP stack realizes what happens.

conradludgate 1y ago
How do you silently drop a TCP connection? Closing the socket fd usually results in a FIN packet being sent whether I want it to or not. Additionally, it's not going to be using that many resources before your kernel sends it a RST next time a data packet is sent
- ronsor 1y ago
  TCP_REPAIR: https://tinselcity.github.io/TCP_Repair/ https://tinselcity.github.io/TCP_Repair/

fsckboy 1y ago

>a slow loris approach does this refer to the word loris recently and only after several years being added to Wordle™?

bananananananan 1y ago
No. A slowloris is an existing attack predating Wordle.
jeltz 1y ago
No, why would it? The attack was named after the animal, slow loris, many years ago.

jandrese 1y ago

> The latest was a slow loris approach where it takes forever for robots.txt to download Applying penalties that exclusively hurt people who are trying to be respectful seems counterproductive.

Snacklive 1y ago

would you be interested in writing an article about it ? sounds really interesting

8organicbits 1y ago
Yeah, seems like a good fit. It will end up here, if I get to it. https://alexsci.com/blog/ https://alexsci.com/blog/