4 ms·

I am not trying to "make an exception", I'm asking for information external to Cloudflare so I can look at what people are experiencing and compare with what ou

I am not trying to "make an exception", I'm asking for information external to Cloudflare so I can look at what people are experiencing and compare with what our systems are doing and figure out what needs to improve.

robertlagrant 2y ago
This is useful info: https://news.ycombinator.com/item?id=33675847 https://news.ycombinator.com/item?id=33675847
PaulRobinson 2y ago
Some "bots" are legitimate. RSS is intended for machine consumption. You should not be blocking content intended for machine consumption because a machine is attempting to consume it. You should not expect a machine, consuming content intended for a machine, to do some sort of step to show they aren't a machine, because they are in fact a machine. There is a lot of content on the internet that is not used by humans, and so checking that humans are using it is an aggressive anti-pattern that ruins experiences for millions of people. It's not that hard. If the content being requested is RSS (or Atom, or some other syndication format intended for consumption by software), just don't do bot checks, use other mechanisms like rate limiting if you must stop abuse. As an example: would you put a captcha on robots.txt as well? As other stories here can attest to, Cloudflare is slowly killing off independent publishing on the web through poor product management decisions and technology implementations, and the fix seems pretty simple.
- jamespo 2y ago
  From another post, if the content-type is correct it gets through. If this is the case I don't see the problem.
  - Scramblejams 2y ago
    It's a very common misconfiguration, though, because it happens by default when setting up CF. If your customers are, by default, configuring things incorrectly, then it's reasonable to ask if the service should surface the issue more proactively in an attempt to help customers get it right. As another commenter noted, not even CF's own RSS feed seems to get the content type right. This issue could clearly use some work.

robertlagrant 2y ago

This is useful info: https://news.ycombinator.com/item?id=33675847 https://news.ycombinator.com/item?id=33675847

PaulRobinson 2y ago

Some "bots" are legitimate. RSS is intended for machine consumption. You should not be blocking content intended for machine consumption because a machine is attempting to consume it. You should not expect a machine, consuming content intended for a machine, to do some sort of step to show they aren't a machine, because they are in fact a machine. There is a lot of content on the internet that is not used by humans, and so checking that humans are using it is an aggressive anti-pattern that ruins experiences for millions of people. It's not that hard. If the content being requested is RSS (or Atom, or some other syndication format intended for consumption by software), just don't do bot checks, use other mechanisms like rate limiting if you must stop abuse. As an example: would you put a captcha on robots.txt as well? As other stories here can attest to, Cloudflare is slowly killing off independent publishing on the web through poor product management decisions and technology implementations, and the fix seems pretty simple.

jamespo 2y ago
From another post, if the content-type is correct it gets through. If this is the case I don't see the problem.
- Scramblejams 2y ago
  It's a very common misconfiguration, though, because it happens by default when setting up CF. If your customers are, by default, configuring things incorrectly, then it's reasonable to ask if the service should surface the issue more proactively in an attempt to help customers get it right. As another commenter noted, not even CF's own RSS feed seems to get the content type right. This issue could clearly use some work.