2 ms·

And added captchas to every user with an adblock or sensible privacy settings.

• pjc50 1y ago
  How would you suggest that such users prove they're not a crawler?
  • ajsnigrutin 1y ago
    Why would they have to? What's wrong with crawlers? That's how google finds you, and people find you on google. Just put some sensible request limits per hour per ip, and be done.
    • skydhash 1y ago
      Or use CDN caching. That's one of the reasons they here for.
    • nucleardog 1y ago
      > Just put some sensible request limits per hour per ip, and be done. I have no personal experience, but probably worth reading like... any of the comments where people are complaining about these crawlers. Claims are that they're: ignoring robots.txt; sending fake User-Agent headers; they're crawling from multiple IPs; when blocked they will use residential proxies. People who have deployed Anubis to try and address this include: Linux Kernel Mailing List, FreeBSD, Arch Linux, NixOS, Proxmox, Gnome, Wine, FFMPEG, FreeDesktop, Gitea, Marginalia, FreeCAD, ReactOS, Duke University, The United Nations (UNESCO)... I'm relatively certain if this were as simple as "just set a sensible rate limit and the crawlers will stop DDOS'ing your site" one person at one of these organizations would have figured that out by now. I don't think they're all doing it because they really love anime catgirls.