3 ms·

I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do inde

I have been managing production commercial web servers for 28 years. Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt. Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not). Cloudflare's "solution" is a much bigger problem that affects me multiple times daily (as a user of sites that use it), and those sites don't seem to need protection against scraping.

filleduchaos 11mo ago
It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.
- cpncrunch 11mo ago
  I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.
  - oasisbob 11mo ago
    No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.
    - cpncrunch 11mo ago
      But is there any actual evidence that any major AI bots are bypassing robots.txt? It looked as if Perplexity was doing this, but after looking into it further it seems that likely isn't the case. Quite often people believe single source news stories without doing any due diligence or fact checking.
      - literalAardvark 11mo ago
        I haven't been in the weeds in a few months, but last time I was there we did have a lot of traffic from bots that didn't care about robots. Bytedance is one that comes to mind.
kvirani 11mo ago
Security almost always brings inconvenience (to everyone involved, including end users). That is part of its cost.
- cpncrunch 11mo ago
  What security issue is actually being solved here though?

filleduchaos 11mo ago

It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.

cpncrunch 11mo ago
I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.
- oasisbob 11mo ago
  No true scotsman needs Cloudflare, as any true scotsman can block AI bots themselves is not a strong argument.
  - cpncrunch 11mo ago
    But is there any actual evidence that any major AI bots are bypassing robots.txt? It looked as if Perplexity was doing this, but after looking into it further it seems that likely isn't the case. Quite often people believe single source news stories without doing any due diligence or fact checking.
    - literalAardvark 11mo ago
      I haven't been in the weeds in a few months, but last time I was there we did have a lot of traffic from bots that didn't care about robots. Bytedance is one that comes to mind.

kvirani 11mo ago

Security almost always brings inconvenience (to everyone involved, including end users). That is part of its cost.

cpncrunch 11mo ago
What security issue is actually being solved here though?