> Just put some sensible request limits per hour per ip, and be done. I have no personal experience, but probably worth reading like... any of the comments whe

> Just put some sensible request limits per hour per ip, and be done. I have no personal experience, but probably worth reading like... any of the comments where people are complaining about these crawlers. Claims are that they're: ignoring robots.txt; sending fake User-Agent headers; they're crawling from multiple IPs; when blocked they will use residential proxies. People who have deployed Anubis to try and address this include: Linux Kernel Mailing List, FreeBSD, Arch Linux, NixOS, Proxmox, Gnome, Wine, FFMPEG, FreeDesktop, Gitea, Marginalia, FreeCAD, ReactOS, Duke University, The United Nations (UNESCO)... I'm relatively certain if this were as simple as "just set a sensible rate limit and the crawlers will stop DDOS'ing your site" one person at one of these organizations would have figured that out by now. I don't think they're all doing it because they really love anime catgirls.