3 ms·

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

nromiun 1y ago
CDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well. Anubis's creator says the same thing: > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. Source: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
- Nextgrid 1y ago
  Alternatively, go-away is also an option: https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away, especially if you don't want cringy branding.
bakugo 1y ago
robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barely noticeable. To stop malicious bots like this, Cloudflare is a great solution if you don't mind using it (you can enable a basic browser check for all users and all pages, or write custom rules to only serve a check to certain users or on certain pages). If you're not a fan of Cloudflare, Anubis works well enough for now if you don't mind the branding. Here's the cloudflare rule I currently use (vast majority of bot traffic originates from these countries): ip.src.continent in {"AF" "SA"} or ip.src.country in {"CN" "HK" "SG"} or ip.src.country in {"AE" "AO" "AR" "AZ" "BD" "BR" "CL" "CO" "DZ" "EC" "EG" "ET" "ID" "IL" "IN" "IQ" "JM" "JO" "KE" "KZ" "LB" "MA" "MX" "NP" "OM" "PE" "PK" "PS" "PY" "SA" "TN" "TR" "TT" "UA" "UY" "UZ" "VE" "VN" "ZA"} or ip.src.asnum in {28573 45899 55836}
- GuB-42 1y ago
  Surely there are solutions more subtle than blocking 80% of the world population...
  - sumtechguy 1y ago
    is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'. It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).
    - GuB-42 1y ago
      > is there an http code for 'hey I gave you this already 10 times. 429 Too Many Requests > This is a you problem not a me problem That's the "4" in "429"
  - bakugo 1y ago
    I should've made it clear that it's not a block rule, just a challenge rule. Those people can still access the website, they just have to go through the "checking your browser" page that you're probably familiar with. As I said, you can just enable that for everyone and be done with it, but with a custom rule, you can avoid showing it to people that are unlikely to be bots.
hombre_fatal 1y ago
CloudFlare's Super Bot Fight Mode completely killed the surge in bot traffic for my large forum.
- ajsnigrutin 1y ago
  And added captchas to every user with an adblock or sensible privacy settings.
  - pjc50 1y ago
    How would you suggest that such users prove they're not a crawler?
    - ajsnigrutin 1y ago
      Why would they have to? What's wrong with crawlers? That's how google finds you, and people find you on google. Just put some sensible request limits per hour per ip, and be done.
      - skydhash 1y ago
        Or use CDN caching. That's one of the reasons they here for.
      - nucleardog 1y ago
        > Just put some sensible request limits per hour per ip, and be done. I have no personal experience, but probably worth reading like... any of the comments where people are complaining about these crawlers. Claims are that they're: ignoring robots.txt; sending fake User-Agent headers; they're crawling from multiple IPs; when blocked they will use residential proxies. People who have deployed Anubis to try and address this include: Linux Kernel Mailing List, FreeBSD, Arch Linux, NixOS, Proxmox, Gnome, Wine, FFMPEG, FreeDesktop, Gitea, Marginalia, FreeCAD, ReactOS, Duke University, The United Nations (UNESCO)... I'm relatively certain if this were as simple as "just set a sensible rate limit and the crawlers will stop DDOS'ing your site" one person at one of these organizations would have figured that out by now. I don't think they're all doing it because they really love anime catgirls.
davidfischer 1y ago
My employer, Read the Docs, has a blog on the subject (https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill. We've done a few things since then: - We already had very generous rate limiting rules by IP (~4 hits/second sustained) but some of the crawlers used thousands of IPs. Cloudflare has a list that they update of AI crawler bots (https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ https://developers.cloudflare.com/bots/additional-configurat...). We're using this list to block these bots and any new bots that get added to the list. - We have more aggressive rate limiting rules by ASN on common hosting providers (eg. AWS, GCP, Azure) which also hits a lot of these bots. - We are considering using the AI crawler list to rate limit by user agent in addition to rate limiting by IP. This will allow well behaved AI crawlers while blocking the badly behaved ones. We aren't against the crawlers generally. - We now have alert rules that alert us when we get a certain amount of traffic (~50k uncached reqs/min sustained). This is basically always some new bot cranked to the max and usually an AI crawler. We get this ~monthly or so and we just ban them. Auto-scaling made our infra good enough where we don't even notice big traffic spikes. However, the downside of that is that the AI crawlers were hammering us without causing anything noticeable. Being smart with rate limiting helps a lot.

CDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well. Anubis's creator says the same thing: > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. Source: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis

Nextgrid 1y ago
Alternatively, go-away is also an option: https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away, especially if you don't want cringy branding.

bakugo 1y ago

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barely noticeable. To stop malicious bots like this, Cloudflare is a great solution if you don't mind using it (you can enable a basic browser check for all users and all pages, or write custom rules to only serve a check to certain users or on certain pages). If you're not a fan of Cloudflare, Anubis works well enough for now if you don't mind the branding. Here's the cloudflare rule I currently use (vast majority of bot traffic originates from these countries): ip.src.continent in {"AF" "SA"} or ip.src.country in {"CN" "HK" "SG"} or ip.src.country in {"AE" "AO" "AR" "AZ" "BD" "BR" "CL" "CO" "DZ" "EC" "EG" "ET" "ID" "IL" "IN" "IQ" "JM" "JO" "KE" "KZ" "LB" "MA" "MX" "NP" "OM" "PE" "PK" "PS" "PY" "SA" "TN" "TR" "TT" "UA" "UY" "UZ" "VE" "VN" "ZA"} or ip.src.asnum in {28573 45899 55836}

GuB-42 1y ago
Surely there are solutions more subtle than blocking 80% of the world population...
- sumtechguy 1y ago
  is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'. It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).
  - GuB-42 1y ago
    > is there an http code for 'hey I gave you this already 10 times. 429 Too Many Requests > This is a you problem not a me problem That's the "4" in "429"
- bakugo 1y ago
  I should've made it clear that it's not a block rule, just a challenge rule. Those people can still access the website, they just have to go through the "checking your browser" page that you're probably familiar with. As I said, you can just enable that for everyone and be done with it, but with a custom rule, you can avoid showing it to people that are unlikely to be bots.

hombre_fatal 1y ago

CloudFlare's Super Bot Fight Mode completely killed the surge in bot traffic for my large forum.

ajsnigrutin 1y ago
And added captchas to every user with an adblock or sensible privacy settings.
- pjc50 1y ago
  How would you suggest that such users prove they're not a crawler?
  - ajsnigrutin 1y ago
    Why would they have to? What's wrong with crawlers? That's how google finds you, and people find you on google. Just put some sensible request limits per hour per ip, and be done.
    - skydhash 1y ago
      Or use CDN caching. That's one of the reasons they here for.
    - nucleardog 1y ago
      > Just put some sensible request limits per hour per ip, and be done. I have no personal experience, but probably worth reading like... any of the comments where people are complaining about these crawlers. Claims are that they're: ignoring robots.txt; sending fake User-Agent headers; they're crawling from multiple IPs; when blocked they will use residential proxies. People who have deployed Anubis to try and address this include: Linux Kernel Mailing List, FreeBSD, Arch Linux, NixOS, Proxmox, Gnome, Wine, FFMPEG, FreeDesktop, Gitea, Marginalia, FreeCAD, ReactOS, Duke University, The United Nations (UNESCO)... I'm relatively certain if this were as simple as "just set a sensible rate limit and the crawlers will stop DDOS'ing your site" one person at one of these organizations would have figured that out by now. I don't think they're all doing it because they really love anime catgirls.

davidfischer 1y ago

My employer, Read the Docs, has a blog on the subject (https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill. We've done a few things since then: - We already had very generous rate limiting rules by IP (~4 hits/second sustained) but some of the crawlers used thousands of IPs. Cloudflare has a list that they update of AI crawler bots (https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ https://developers.cloudflare.com/bots/additional-configurat...). We're using this list to block these bots and any new bots that get added to the list. - We have more aggressive rate limiting rules by ASN on common hosting providers (eg. AWS, GCP, Azure) which also hits a lot of these bots. - We are considering using the AI crawler list to rate limit by user agent in addition to rate limiting by IP. This will allow well behaved AI crawlers while blocking the badly behaved ones. We aren't against the crawlers generally. - We now have alert rules that alert us when we get a certain amount of traffic (~50k uncached reqs/min sustained). This is basically always some new bot cranked to the max and usually an AI crawler. We get this ~monthly or so and we just ban them. Auto-scaling made our infra good enough where we don't even notice big traffic spikes. However, the downside of that is that the AI crawlers were hammering us without causing anything noticeable. Being smart with rate limiting helps a lot.