4 ms·

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping

by bakugo 1y ago

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barely noticeable. To stop malicious bots like this, Cloudflare is a great solution if you don't mind using it (you can enable a basic browser check for all users and all pages, or write custom rules to only serve a check to certain users or on certain pages). If you're not a fan of Cloudflare, Anubis works well enough for now if you don't mind the branding. Here's the cloudflare rule I currently use (vast majority of bot traffic originates from these countries): ip.src.continent in {"AF" "SA"} or ip.src.country in {"CN" "HK" "SG"} or ip.src.country in {"AE" "AO" "AR" "AZ" "BD" "BR" "CL" "CO" "DZ" "EC" "EG" "ET" "ID" "IL" "IN" "IQ" "JM" "JO" "KE" "KZ" "LB" "MA" "MX" "NP" "OM" "PE" "PK" "PS" "PY" "SA" "TN" "TR" "TT" "UA" "UY" "UZ" "VE" "VN" "ZA"} or ip.src.asnum in {28573 45899 55836}

GuB-42 1y ago
Surely there are solutions more subtle than blocking 80% of the world population...
- sumtechguy 1y ago
  is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'. It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).
  - GuB-42 1y ago
    > is there an http code for 'hey I gave you this already 10 times. 429 Too Many Requests > This is a you problem not a me problem That's the "4" in "429"
- bakugo 1y ago
  I should've made it clear that it's not a block rule, just a challenge rule. Those people can still access the website, they just have to go through the "checking your browser" page that you're probably familiar with. As I said, you can just enable that for everyone and be done with it, but with a custom rule, you can avoid showing it to people that are unlikely to be bots.

by bakugo 1y ago

GuB-42 1y ago

Surely there are solutions more subtle than blocking 80% of the world population...

sumtechguy 1y ago
is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'. It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).
- GuB-42 1y ago
  > is there an http code for 'hey I gave you this already 10 times. 429 Too Many Requests > This is a you problem not a me problem That's the "4" in "429"
bakugo 1y ago
I should've made it clear that it's not a block rule, just a challenge rule. Those people can still access the website, they just have to go through the "checking your browser" page that you're probably familiar with. As I said, you can just enable that for everyone and be done with it, but with a custom rule, you can avoid showing it to people that are unlikely to be bots.