3 ms·

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "p

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to help anything either.

otherme123 2y ago
In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_" and "limit_rate_" of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how much they request.
- dns_snek 2y ago
  > a crawl requesting about 2 pages per second. It was so intense [...] Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.
  - BrutalCoding 2y ago
    I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.
    - gtirloni 2y ago
      If ypu open DevTools and visit any website these days, you'll be surprised.
      - otherme123 2y ago
        In my scenario you request one single page from the proxy endpoint, and all other requests go straight to the static files and have no limits. I know than no human needs to request more than 1/s from the proxy, unless you are opening tabs frantically. So far, I only get praises about how responsive and quick the sites are: being harsh with the abusers means more resources for the regulars.
  - dns_snek 2y ago
    Downvoted for asking a completely reasonable question? Where am I?
- efilife 2y ago
  Rookie question, how do you ban an ip range?
  - aeonik 2y ago
    You can do it in a few places, but I use my network firewall for this I use PFSense at home, but there are many enterprise grade brands). It's common to use the host's firewall as well (nftables, firewalld, or iptables). You can do it at the webserver too, with access.conf in nginx. Apache uses mod_authz. I usually do it at the network though so it uses the least amount of resources (no connection ever gets to the webserver). Though if you only have access to your webserver it's faster to ban it there than to send a request to the network team (depending on your org, some orgs might have this automated).
  - otherme123 2y ago
    In your nginx, server section: deny 1.2.3.0/24; And all 256 ips from 1.2.3.0 to 1.2.3.255 get banned. You can have multiple "deny" lines, or a file with "deny" and then include it. It's better to do it at the firewall.

otherme123 2y ago

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_" and "limit_rate_" of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how much they request.

dns_snek 2y ago
> a crawl requesting about 2 pages per second. It was so intense [...] Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.
- BrutalCoding 2y ago
  I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.
  - gtirloni 2y ago
    If ypu open DevTools and visit any website these days, you'll be surprised.
    - otherme123 2y ago
      In my scenario you request one single page from the proxy endpoint, and all other requests go straight to the static files and have no limits. I know than no human needs to request more than 1/s from the proxy, unless you are opening tabs frantically. So far, I only get praises about how responsive and quick the sites are: being harsh with the abusers means more resources for the regulars.
- dns_snek 2y ago
  Downvoted for asking a completely reasonable question? Where am I?
efilife 2y ago
Rookie question, how do you ban an ip range?
- aeonik 2y ago
  You can do it in a few places, but I use my network firewall for this I use PFSense at home, but there are many enterprise grade brands). It's common to use the host's firewall as well (nftables, firewalld, or iptables). You can do it at the webserver too, with access.conf in nginx. Apache uses mod_authz. I usually do it at the network though so it uses the least amount of resources (no connection ever gets to the webserver). Though if you only have access to your webserver it's faster to ban it there than to send a request to the network team (depending on your org, some orgs might have this automated).
- otherme123 2y ago
  In your nginx, server section: deny 1.2.3.0/24; And all 256 ips from 1.2.3.0 to 1.2.3.255 get banned. You can have multiple "deny" lines, or a file with "deny" and then include it. It's better to do it at the firewall.