5 ms·

Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers

Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers anonymizing themselves / coming from different IP addresses?

sc68cal 1y ago
They are spreading themselves across lots of different IP blocks
thaumaturgy 1y ago
Yeah, that's where IP intelligence comes in. They're using pretty big IP pools, so, either you're manually adding individual IPs to a list all day (and updating that list as ASNs get continuously shuffled around), or you've got a process in the background that essentially does whois lookups (and caches them, so you aren't also being abusive), parses the metadata returned, and decides whether that request is "okay" or not. The classic 80/20 rule applies. You can catch about 80% of lazy crawler activity pretty easily with something like this, but the remaining 20% will require a lot more effort. You start encountering edge cases, like crawlers that use AWS for their crawling activity, but also one of your customers somewhere is syncing their WooCommerce orders to their in-house ERP system via a process that also runs on AWS.
- asddubs 1y ago
  I've had crawlers get stuck in a loop before on a search page where you basically could just keep adding things, even if there are no results. I filtered requests that are bots for sure (requests which are specified long past the point of any results). It was over a million unique IPs, most of which only doing 1 or 2 requests on their own (from many different ip blocks)
loloquwowndueo 1y ago
Its called Anubis.
- dylan604 1y ago
  Isn't that the one that shows anime characters? Or is Anubis the "professional" version that doesn't show anime chars?
  - greazy 1y ago
    Yes that's Anubis. And yes you pay to not show anime cat girl.
    - tempaccount420 1y ago
      That's genius.
      - krapp 1y ago
        Honestly the more Anubis' anime mascot annoys people the more I like it.
        dylan604 1y ago
        The point of this is to make things difficult for bots, not to annoy visitors of the site. I respect it is the dev's choice to do what they want with the software they create and make available for free. Anime is a polarizing format for reasons beyond the scope of this discussion. It definitely says a lot about the dev
        krapp 1y ago
        Anime is only "polarizing" for an extreme subset of people. Most people won't care. No one should care, it's just a cute mascot image.
        linotype 1y ago
        It says a lot more about the pearl clutching of the people complaining about it than it does the dev.
- 1y ago

sc68cal 1y ago

They are spreading themselves across lots of different IP blocks

thaumaturgy 1y ago

Yeah, that's where IP intelligence comes in. They're using pretty big IP pools, so, either you're manually adding individual IPs to a list all day (and updating that list as ASNs get continuously shuffled around), or you've got a process in the background that essentially does whois lookups (and caches them, so you aren't also being abusive), parses the metadata returned, and decides whether that request is "okay" or not. The classic 80/20 rule applies. You can catch about 80% of lazy crawler activity pretty easily with something like this, but the remaining 20% will require a lot more effort. You start encountering edge cases, like crawlers that use AWS for their crawling activity, but also one of your customers somewhere is syncing their WooCommerce orders to their in-house ERP system via a process that also runs on AWS.

asddubs 1y ago
I've had crawlers get stuck in a loop before on a search page where you basically could just keep adding things, even if there are no results. I filtered requests that are bots for sure (requests which are specified long past the point of any results). It was over a million unique IPs, most of which only doing 1 or 2 requests on their own (from many different ip blocks)

loloquwowndueo 1y ago

Its called Anubis.

dylan604 1y ago
Isn't that the one that shows anime characters? Or is Anubis the "professional" version that doesn't show anime chars?
- greazy 1y ago
  Yes that's Anubis. And yes you pay to not show anime cat girl.
  - tempaccount420 1y ago
    That's genius.
    - krapp 1y ago
      Honestly the more Anubis' anime mascot annoys people the more I like it.
      - dylan604 1y ago
        The point of this is to make things difficult for bots, not to annoy visitors of the site. I respect it is the dev's choice to do what they want with the software they create and make available for free. Anime is a polarizing format for reasons beyond the scope of this discussion. It definitely says a lot about the dev
        krapp 1y ago
        Anime is only "polarizing" for an extreme subset of people. Most people won't care. No one should care, it's just a cute mascot image.
        linotype 1y ago
        It says a lot more about the pearl clutching of the people complaining about it than it does the dev.
1y ago