4 ms·

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet.

I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI extortion: https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/

NooneAtAll3 11mo ago
it can't even spy on us silently, damn
james2doyle 11mo ago
You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?
- cpncrunch 11mo ago
  >You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots https://habeasdata.neocities.org/ai-bots
  - james2doyle 11mo ago
    This is just using robots.txt and asking "pretty please, don’t scrape me". Here is an article (from TODAY) about the case where Perplexity is being accused of ignoring robots.txt: https://www.theverge.com/news/839006/new-york-times-perplexity-lawsuit-copyright https://www.theverge.com/news/839006/new-york-times-perplexi... If you think a robots.txt is the answer to stopping the billion-dollar AI machine from scraping you, I don’t know what to say.
    - cpncrunch 11mo ago
      Yes, I was referring to legitimate companies, and Perplexity doesn't seem to be one of those.
      - albedoa 11mo ago
        Oh for sure. When he wrote of the AI companies that are "stealing/crawling/hammering", you thought he meant the legitimate ones that do honor robots.txt. That makes sense.
        cpncrunch 11mo ago
        Actually, it looks like all the major ones do honour robots.txt including perplexity. They seemingly get around it using google serps, so theyre not actually crawling or hammering the site servers (or even cloudflare). https://www.ailawandpolicy.com/2025/10/anti-circumvention-reddits-case-against-perplexity/ https://www.ailawandpolicy.com/2025/10/anti-circumvention-re...
    - Aeolun 11mo ago
      If someone has a robots.txt, and I want to request their page, but I want to do that in an automated way, should I open the browser to do it instead of issue a curl request? How about if I am going to ask claude to fetch the page for me?
      - kentm 11mo ago
        Respect the robots.txt and don’t do it?
  - jacobgkau 11mo ago
    I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.
    - dingnuts 11mo ago
      [dead]
    - cpncrunch 11mo ago
      I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem that affects me multiple times daily (as a user of sites that use it), and those sites don't seem to need protection against scraping.
      - filleduchaos 11mo ago
        It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.
        cpncrunch 11mo ago
        I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.
        oasisbob 11mo ago
        No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.
  - mplewis 11mo ago
    No you cannot! I blocked all of the user agents on a community wiki I run, and the traffic came back hours later masquerading as Firefox and Chrome. They just fucking lie to you and continue vacuuming your CPU.
    - cpncrunch 11mo ago
      There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS? Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).
      - literalAardvark 11mo ago
        There is a noticeable hit, there's also a noticeable cost, and it's not a ddos. Not all sites can have full caching, we've tried.
        cpncrunch 11mo ago
        I was referring to the community wiki.
      - deleted 11mo ago
        [deleted]
  - chrneu 11mo ago
    this is the equivalent of asking people not to speed on your street.
  - literalAardvark 11mo ago
    Tell me you don't run a site without telling me you don't run a site
    - cpncrunch 11mo ago
      Tell me you make incorrect assumptions without specifically saying so. (Yes, you're incorrect).
  - Sohcahtoa82 11mo ago
    How are you this naive? Do you really think scrapers give a damn about your robots.txt?
    - cpncrunch 11mo ago
      The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.
gblargg 11mo ago
More and more sites I can't even visit because of this "prove you're human" because it's not compatible with older web browsers, even though the website it's blocking is.
_kidlike 11mo ago
the two things are unrelated... The pay-per-crawl thing, is about them thinking ahead about post-AI business/revenue models. The way AI happened, it removed a big chunk of revenue from news companies, blogs, etc. Because lots of people go to AI instead of reaching the actual 3rd party website. AI currently gets the content for free from the 3rd party websites, but they have revenue from their users. So Cloudflare is proposing that AI companies should be paying for their crawling. Cloudflare's solution would give the lost revenue back where it belongs, just through a different mechanism. The ugly side of the story is that this was already an existing solution, and open source, called L402.org. Cloudflare wants to be the first to take a piece of the pie, but also instead of using the open source version, they forked it internally and published it as their own service, which is cloudflare specific. To be completely fair, the l402 requires you to solve the payment mechanism itself, which for Cloudflare is easy because they already deal with payments.
pmdr 11mo ago
In my experience it's been in recent years, not months.
stef25 11mo ago
> I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. Good to know I'm not the only one
bobbob1921 11mo ago
Ive been seeing more of those prove your human pages as well, but I generally assume they are there to combat a DDOS or other type of attack (or maybe ai/bot). I remember how annoying it was combating DDOS attacks, or hacked sites before Cloudflare existed. I also remember how annoying capcha s were, everywhere. Cloudflare is not perfect but net, I think it’s been a great improvement.
- cpncrunch 11mo ago
  Ovh provides ddos protection without that nonsense, for free. Aws does it too, for a fee.
chamomeal 11mo ago
Feel like that’s the fault of LLMs, not cloudflare
- cpncrunch 11mo ago
  Looking into this more, it does indeed seem to be a cloudflare problem. It looks like cloudflare made a significant error in their bot fingerprinting, and Perplexity wasn't actually bypassing robots.txt. https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web https://www.perplexity.ai/hub/blog/agents-or-bots-making-sen... To be honest I find cloudflare a much more scammy company than Perplexity. I had a DDoS attack a few years ago which originated from their network, and they had zero interest in it.

NooneAtAll3 11mo ago

it can't even spy on us silently, damn

james2doyle 11mo ago

You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?

cpncrunch 11mo ago
>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots https://habeasdata.neocities.org/ai-bots
- james2doyle 11mo ago
  This is just using robots.txt and asking "pretty please, don’t scrape me". Here is an article (from TODAY) about the case where Perplexity is being accused of ignoring robots.txt: https://www.theverge.com/news/839006/new-york-times-perplexity-lawsuit-copyright https://www.theverge.com/news/839006/new-york-times-perplexi... If you think a robots.txt is the answer to stopping the billion-dollar AI machine from scraping you, I don’t know what to say.
  - cpncrunch 11mo ago
    Yes, I was referring to legitimate companies, and Perplexity doesn't seem to be one of those.
    - albedoa 11mo ago
      Oh for sure. When he wrote of the AI companies that are "stealing/crawling/hammering", you thought he meant the legitimate ones that do honor robots.txt. That makes sense.
      - cpncrunch 11mo ago
        Actually, it looks like all the major ones do honour robots.txt including perplexity. They seemingly get around it using google serps, so theyre not actually crawling or hammering the site servers (or even cloudflare). https://www.ailawandpolicy.com/2025/10/anti-circumvention-reddits-case-against-perplexity/ https://www.ailawandpolicy.com/2025/10/anti-circumvention-re...
  - Aeolun 11mo ago
    If someone has a robots.txt, and I want to request their page, but I want to do that in an automated way, should I open the browser to do it instead of issue a curl request? How about if I am going to ask claude to fetch the page for me?
    - kentm 11mo ago
      Respect the robots.txt and don’t do it?
- jacobgkau 11mo ago
  I'm guessing you don't manage any production web servers? robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.
  - dingnuts 11mo ago
    [dead]
  - cpncrunch 11mo ago
    I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem that affects me multiple times daily (as a user of sites that use it), and those sites don't seem to need protection against scraping.
    - filleduchaos 11mo ago
      It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.
      - cpncrunch 11mo ago
        I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.
        oasisbob 11mo ago
        No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.
- mplewis 11mo ago
  No you cannot! I blocked all of the user agents on a community wiki I run, and the traffic came back hours later masquerading as Firefox and Chrome. They just fucking lie to you and continue vacuuming your CPU.
  - cpncrunch 11mo ago
    There shouldn't be any noticeable hit on your cpu from bots from a site like that. Are you sure it's not a DDoS? Obviously it depends on the bot, and you can't block the scammy ones. I was really just referring to the major legitimate companies (which might not include Perplexity).
    - literalAardvark 11mo ago
      There is a noticeable hit, there's also a noticeable cost, and it's not a ddos. Not all sites can have full caching, we've tried.
      - cpncrunch 11mo ago
        I was referring to the community wiki.
    - deleted 11mo ago
      [deleted]
- chrneu 11mo ago
  this is the equivalent of asking people not to speed on your street.
- literalAardvark 11mo ago
  Tell me you don't run a site without telling me you don't run a site
  - cpncrunch 11mo ago
    Tell me you make incorrect assumptions without specifically saying so. (Yes, you're incorrect).
- Sohcahtoa82 11mo ago
  How are you this naive? Do you really think scrapers give a damn about your robots.txt?
  - cpncrunch 11mo ago
    The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.

gblargg 11mo ago

More and more sites I can't even visit because of this "prove you're human" because it's not compatible with older web browsers, even though the website it's blocking is.

_kidlike 11mo ago

the two things are unrelated... The pay-per-crawl thing, is about them thinking ahead about post-AI business/revenue models. The way AI happened, it removed a big chunk of revenue from news companies, blogs, etc. Because lots of people go to AI instead of reaching the actual 3rd party website. AI currently gets the content for free from the 3rd party websites, but they have revenue from their users. So Cloudflare is proposing that AI companies should be paying for their crawling. Cloudflare's solution would give the lost revenue back where it belongs, just through a different mechanism. The ugly side of the story is that this was already an existing solution, and open source, called L402.org. Cloudflare wants to be the first to take a piece of the pie, but also instead of using the open source version, they forked it internally and published it as their own service, which is cloudflare specific. To be completely fair, the l402 requires you to solve the payment mechanism itself, which for Cloudflare is easy because they already deal with payments.

pmdr 11mo ago

In my experience it's been in recent years, not months.

stef25 11mo ago

> I've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. Good to know I'm not the only one

bobbob1921 11mo ago

Ive been seeing more of those prove your human pages as well, but I generally assume they are there to combat a DDOS or other type of attack (or maybe ai/bot). I remember how annoying it was combating DDOS attacks, or hacked sites before Cloudflare existed. I also remember how annoying capcha s were, everywhere. Cloudflare is not perfect but net, I think it’s been a great improvement.

cpncrunch 11mo ago
Ovh provides ddos protection without that nonsense, for free. Aws does it too, for a fee.

chamomeal 11mo ago

Feel like that’s the fault of LLMs, not cloudflare

cpncrunch 11mo ago
Looking into this more, it does indeed seem to be a cloudflare problem. It looks like cloudflare made a significant error in their bot fingerprinting, and Perplexity wasn't actually bypassing robots.txt. https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web https://www.perplexity.ai/hub/blog/agents-or-bots-making-sen... To be honest I find cloudflare a much more scammy company than Perplexity. I had a DDoS attack a few years ago which originated from their network, and they had zero interest in it.