3 ms·

Most websites don’t need DDOS protection. Many websites which use Cloudflare to block basic bot vulnerability scanning. You could block this type of traffic wi

Most websites don’t need DDOS protection. Many websites which use Cloudflare to block basic bot vulnerability scanning. You could block this type of traffic with other methods; ja3/ja4, Ip to ASN & ASN filtering, etc.

esseph 1y ago
Your first line is wrong. While it may not impact your site, it does impact your hosting provider. As their costs go up, your costs go up. Anything on the Internet at this point needs DDoS / scraping protection. If may not drop your service, but your ISP or upstreams may blackhole your route. The "old web" (current web) was largely based on an open exchange of information. The "new web", post AI bot scraping, is taking its place. Websites are getting paywalls. Advertising revenue is plummeting. Hosting providers are getting decimated by the massive shift in bandwidth demand and impact to systems scraped by the bots.
- VladVladikoff 1y ago
  I guess my products fall into a niche that doesn’t seem to attract AI crawlers. I’ve seen only a few and they haven’t been too aggressive. I mean they ignore typical crawl rate limits defined in robots.txt but account for maybe only 1-2% of my overall traffic.
  - esseph 1y ago
    That's crazy! I've been watching the scraping go up exponentially for months now. https://www.eff.org/deeplinks/2025/06/keeping-web-under-weight-ai-crawlers https://www.eff.org/deeplinks/2025/06/keeping-web-under-weig...
- immibis 1y ago
  DDoS and AI are mostly unrelated. Sure, AI companies are running low-quality scrapers, but they don't cause nearly as much traffic as a DDoS. They might cause as much CPU load as a DDoS, which is an application-level problem.
  - esseph 1y ago
    https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/ https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-domi... They are creating a network and application load that in effect, is a DDoS. Tens or hundreds of thousands of hosts hitting the same domain at once.
    - immibis 1y ago
      That's a bug in Gitea. I suffered the same bug and worked around it in Gitea. Non-malicious traffic can also break a Gitea server.
      - esseph 1y ago
        Alright, let me go get a dozen posts on the internet by network and system engineers working at the biggest companies at the world to tell you the same thing that you apparently haven't heard yet - they're all running into the same thing. We have about 400 domains at $dayjob and a decent sized network. All these domains, even silly ones with a handful of tabs and no dynamic content, are getting absolutely brutalized with traffic non-stop. Geoblocking did not help. So much of the "AI scraping" is coming from compromised customer internet connections. Compromised CPE equipment, desktops, phones, etc. It is a fucking nightmare.
        immibis 1y ago
        What is the evidence that has anything to do with AI? Seems to be very circumstantial at best: we're getting AI scrapers, and at the same time DDoS scrapers, so the DDoS scrapers must also be AI.
- immibis 1y ago
  "As their costs go up, your costs go up" - great, let them charge for bandwidth then. Actually, incoming bandwidth to a hosting provider is free, because basically every interconnection on the Internet is billed based on the dominant direction of traffic. It's really important that you do not preemptively make assumptions about other people's business models in a way that impacts your own. If they have a problem with it, let them tell you. Capitalism is a symmetric game (played asymmetrically). If your hosting provider thought something they did would increase your costs, do you think they'd refrain from doing it?
  - esseph 1y ago
    Let's play this out bubba. Let's talk economics. How many monetized webpages are out there, vs free websites? How many blogs, forums, linktrees, etc? With what you're suggesting, all of that goes away. You may want that internet. I sure as fuck do not.
    - immibis 1y ago
      You're still making no sense here?

esseph 1y ago

Your first line is wrong. While it may not impact your site, it does impact your hosting provider. As their costs go up, your costs go up. Anything on the Internet at this point needs DDoS / scraping protection. If may not drop your service, but your ISP or upstreams may blackhole your route. The "old web" (current web) was largely based on an open exchange of information. The "new web", post AI bot scraping, is taking its place. Websites are getting paywalls. Advertising revenue is plummeting. Hosting providers are getting decimated by the massive shift in bandwidth demand and impact to systems scraped by the bots.

VladVladikoff 1y ago
I guess my products fall into a niche that doesn’t seem to attract AI crawlers. I’ve seen only a few and they haven’t been too aggressive. I mean they ignore typical crawl rate limits defined in robots.txt but account for maybe only 1-2% of my overall traffic.
- esseph 1y ago
  That's crazy! I've been watching the scraping go up exponentially for months now. https://www.eff.org/deeplinks/2025/06/keeping-web-under-weight-ai-crawlers https://www.eff.org/deeplinks/2025/06/keeping-web-under-weig...
immibis 1y ago
DDoS and AI are mostly unrelated. Sure, AI companies are running low-quality scrapers, but they don't cause nearly as much traffic as a DDoS. They might cause as much CPU load as a DDoS, which is an application-level problem.
- esseph 1y ago
  https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/ https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-domi... They are creating a network and application load that in effect, is a DDoS. Tens or hundreds of thousands of hosts hitting the same domain at once.
  - immibis 1y ago
    That's a bug in Gitea. I suffered the same bug and worked around it in Gitea. Non-malicious traffic can also break a Gitea server.
    - esseph 1y ago
      Alright, let me go get a dozen posts on the internet by network and system engineers working at the biggest companies at the world to tell you the same thing that you apparently haven't heard yet - they're all running into the same thing. We have about 400 domains at $dayjob and a decent sized network. All these domains, even silly ones with a handful of tabs and no dynamic content, are getting absolutely brutalized with traffic non-stop. Geoblocking did not help. So much of the "AI scraping" is coming from compromised customer internet connections. Compromised CPE equipment, desktops, phones, etc. It is a fucking nightmare.
      - immibis 1y ago
        What is the evidence that has anything to do with AI? Seems to be very circumstantial at best: we're getting AI scrapers, and at the same time DDoS scrapers, so the DDoS scrapers must also be AI.
immibis 1y ago
"As their costs go up, your costs go up" - great, let them charge for bandwidth then. Actually, incoming bandwidth to a hosting provider is free, because basically every interconnection on the Internet is billed based on the dominant direction of traffic. It's really important that you do not preemptively make assumptions about other people's business models in a way that impacts your own. If they have a problem with it, let them tell you. Capitalism is a symmetric game (played asymmetrically). If your hosting provider thought something they did would increase your costs, do you think they'd refrain from doing it?
- esseph 1y ago
  Let's play this out bubba. Let's talk economics. How many monetized webpages are out there, vs free websites? How many blogs, forums, linktrees, etc? With what you're suggesting, all of that goes away. You may want that internet. I sure as fuck do not.
  - immibis 1y ago
    You're still making no sense here?