5 ms·
Well, it's possible to also return a 403 (forbidden) to any request based off the user agent. Of course, this can be relatively easily circumvented, but then it
by bartread 4y ago
Well, it's possible to also return a 403 (forbidden) to any request based off the user agent. Of course, this can be relatively easily circumvented, but then it's also possible to block IP ranges and suchlike. You can return a 403 off of any detectable aspect of the client that you don't like if you so wish.
I don't know how well this would work with a CDN, but presumably if you pay for the right tier of Cloudflare (or whatever) you can perform similar operations to prevent content being hoovered from their by clients you'd prefer not to serve.
- superkuh 4y agoYep. I 403 turnitin and similar companies via nginx configuration, if ($http_referer ~* (TurnitinBot|PaperLiBot|idmarch|FairShare|Lightspeedsystems|ZmEu|BPImageWalker|semrushBot|ias_crawler|360spider|copyrightinfringementportal|PetalBot|Adsbot|SlySearch|NPBot)) { return 403; } But my favorite robots.txt is, User-agent: Zombies Disallow: /brains
- rcarmo 4y agoShouldn’t that be… User-agent: Zombies Disallow: /braaains ?
- amazing_stories 4y ago
- easrng 4y agoWhy are you blocking PetalBot? It's an actual search engine.
- superkuh 4y agoLegit Huawei IP ranges identifying as Huawei PetalBot were being abusive, definitely not obeying robots.txt, and searching for subsets of content that indicated they were looking to identify political dissidents with no worries about actually indexing the full site. I don't consider it a real search engine. But yeah, maybe not a good fit for this list of educational and copyright parasites.
- easrng 4y agoOh yikes, that sounds bad. It is a real search engine though, https://petalsearch.com/ https://petalsearch.com/