3 ms·
Yep. I 403 turnitin and similar companies via nginx configuration, if ($http_referer ~* (TurnitinBot|PaperLiBot|idmarch|FairShare|Lightspeedsystems|ZmEu|BPImage
by superkuh 5y ago
Yep. I 403 turnitin and similar companies via nginx configuration, if ($http_referer ~* (TurnitinBot|PaperLiBot|idmarch|FairShare|Lightspeedsystems|ZmEu|BPImageWalker|semrushBot|ias_crawler|360spider|copyrightinfringementportal|PetalBot|Adsbot|SlySearch|NPBot)) { return 403; }
But my favorite robots.txt is,
User-agent: Zombies
Disallow: /brains
- rcarmo 5y agoShouldn’t that be… User-agent: Zombies Disallow: /braaains ?
- amazing_stories 5y ago
- easrng 5y agoWhy are you blocking PetalBot? It's an actual search engine.
- superkuh 5y agoLegit Huawei IP ranges identifying as Huawei PetalBot were being abusive, definitely not obeying robots.txt, and searching for subsets of content that indicated they were looking to identify political dissidents with no worries about actually indexing the full site. I don't consider it a real search engine. But yeah, maybe not a good fit for this list of educational and copyright parasites.
- easrng 5y agoOh yikes, that sounds bad. It is a real search engine though, https://petalsearch.com/ https://petalsearch.com/