15 ms·

Using Cloudflare on your website could be blocking RSS users

559 points by campuscodi 2y ago

kevincox 2y ago
I dislike advice of whitelisting specific readers by user-agent. Not only is this endless manual work that will only solve the problem for a subset of users but it also is easy to bypass by malicious actors. My recommendation would be to create a page rule that disables bot blocking for your feeds. This will fix the problem for all readers with no ongoing maintenance. If you are worried about DoS attacks that may hammer on your feeds then you can use the same configuration rule to ignore the query string for cache keys (if your feed doesn't use query strings) and overriding the caching settings if your server doesn't set the proper headers. This way Cloudflare will cache your feed and you can serve any number of visitors without putting load onto your origin. As for Cloudflare fixing the defaults, it seems unlikely to happen. It has been broken for years, Cloudflare's own blog is affected. They have been "actively working" on fixing it for at least 2 years according to their VP of product: https://news.ycombinator.com/item?id=33675847 https://news.ycombinator.com/item?id=33675847
- vaylian 2y ago
  I don't know if cloudflare offers it, but whitelisting the URL of the RSS feed would be much more effective than filtering user agents.
  - derkades 2y ago
    Yes it supports it, and I think that's what the parent comment was all about
    - BiteCode_dev 2y ago
      Specifically, whitelisting the URL for the bot protection, but not the cache, so that you are still somewhat protected against adversarial use.
      - londons_explore 2y ago
        An adversary can easily send no-cache headers to bust the cache.
        acdha 2y ago
        The CDN can choose whether to honor those. That hasn’t been an effective adversarial technique since the turn of the century.
        londons_explore 2y ago
        does cloudflare give such an option? Even for non-paid accounts?
        acdha 2y ago
        They ignore request cache control headers, I believe unconditionally so you’d have to disable caching for the endpoints which clients are allowed to request uncached.
  - deleted 2y ago
    [deleted]
  - jks 2y ago
    Yes, you can do it with a "page rule", which the parent comment mentioned. The CloudFlare free tier has a budget of three page rules, which might mean that you have to bundle all your rss feeds in one folder so they share a path prefix.
- a-french-anon 2y ago
  And for those of us using sfeed, the default UA is Curl's.
- benregenspan 2y ago
  AI crawlers have changed the picture significantly and in my opinion are a much bigger threat to the open web than Cloudflare. The training arms race has drastically increased bot traffic, and the value proposition behind that bot traffic has inverted. Previously many site operators could rely on the average automated request being net-beneficial to the site and its users (outside of scattered, time-limited DDoS attacks) but now most of these requests represent value extraction. Combine this with a seemingly related increase in high-volume bots that don't respect robots.txt and don't set a useful User-Agent, and using a heavy-handed firewall becomes a much easier business decision, even if it may target some desirable traffic (like valid RSS requests).
rcarmo 2y ago
Ironically, the site seems to currently be hugged to death, so maybe they should consider using Cloudflare to deal with HN traffic?
- timeon 2y ago
  If it is unintentional DDoS, we can wait. Not everything needs to be on demand.
  - dewey 2y ago
    The website is built to get attention, the attention is here right now. Nobody will remember to go back tomorrow and read the site again when it’s available.
    - BlueTemplar 2y ago
      I'm not sure an open web can exist under this kind of assumption... Once you start chasing views, it's going to come at the detriment of everything else.
      - dewey 2y ago
        This happened at least 15 years ago and we are doing okay.
- sofixa 2y ago
  Doesn't have to be using CloudFlare, just a static web host that will be able to scale to infinity (of which CloudFlare is one with Pages, but there's also Google with Firebase Hosting, AWS with Amplify, Microsoft with something in Azure with a verbose name, Netlify, Vercel, GitHub Pages, etc etc etc).
  - kawsper 2y ago
    Or just add Varnish or Nginx configured with a cache in front.
veeti 2y ago
I believe that disabling "Bot Fight Mode" is not enough, you may also need to create a rule to disable "Browser Integrity Check".
belkinpower 2y ago
I maintain an RSS reader for work and Cloudflare is the bane of my existence. Tons of feeds will stop working at random and there’s nothing we can do about it except for individually contacting website owners and asking them to add an exception for their feed URL.
- sammy2255 2y ago
  Unfortunately its not really Cloudflare but webadmins who have configured it to block everything thats not a browser, whether unknowingly or not
  - afandian 2y ago
    If Cloudflare offer a product, for a particular purpose, that breaks existing conventions of that purpose, then it’s Cloudflare.
    - sammy2255 2y ago
      Not really. You wouldn’t complain to a fence company for blocking a path if there were hired to do exactly that
      - gsich 2y ago
        They are enablers. They get part of the blame.
      - shakna 2y ago
        Yes, I would. Experts are expected to relay back to their client with their thoughts on a matter, not just blindly do as they're told. Your builder is meant to do their due diligence, which includes making recommendations.
    - echoangle 2y ago
      Well it doesn’t break the conventions of the purpose they offer it for. Cloudflare attempts to block non-human users, and this is supposed to be used for human-readable websites. If someone puts cloudflare in front of a RSS feed, that’s user error. It’s like someone putting a captcha in front of an API and then complaining that the Captcha provider is breaking conventions.
amatecha 2y ago
I get blocked from websites with some regularity, running Firefox with strict privacy settings, "resist fingerprinting" etc. on OpenBSD. They just give a 403 Forbidden with no explanation, but it's only ever on sites fronted by CloudFlare. Good times. Seems legit.
- BiteCode_dev 2y ago
  Cloudflare is a fantastic service with an unmatched value proposition, but it's unfortunately slowly killing web privacy, with 1000s paper cuts. Another problem is "resist fingerprinting" prevents some canvas processing, and many websites like bluesky, linked in or substack uses canvas to handle image upload, so your images appear to be stripes of pixel. Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. I understand why people give up, trying to fight for your privacy is an uphill battle with no end in sight.
  - madeofpalk 2y ago
    > Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. Is that true? At least on iOS you can log into the ChatGPT with same email/password as the website. I never use Google login for stuff and ChatGPT works fine for me.
    - BiteCode_dev 2y ago
      See other comment.
  - KomoD 2y ago
    > Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. That's not true, I use ChatGPT's app on my phone without logging into a Google account. You don't even need any kind of account at all to use it.
    - BiteCode_dev 2y ago
erikrothoff 2y ago
As the owner of an RSS reader I love that they are making this more public. 30% of our support requests are ”my feed doesn’t” work. It sucks that the only thing we can say is ”contact the site owner, it’s their firewall”. And to be fair it’s not only Cloudflare, so many different firewall setups cause issues. It’s ironic that a public API endpoint meant for bots is blocked for being a bot.
ricardo81 2y ago
iirc even if you're listed as a "good bot" with Cloudflare, high security settings by the CF user can still result in 403s. No idea if CF already does this, but allowing users to generate access tokens for 3rd party services would be another way of easing access alongside their apparent URL and IP whitelisting.
mbo 2y ago
This is an active issue with Rate Your Music right now: https://rateyourmusic.com/rymzilla/view?id=6108 https://rateyourmusic.com/rymzilla/view?id=6108 Unfixed for 4 months.
jgrahamc 2y ago
My email is jgc@cloudflare.com. I'd like to hear from the owners of RSS readers directly on what they are experiencing. Going to ask team to take a closer look.
- viraptor 2y ago
  It's cool and all that you're making an exception here, but how about including a "no, really, I'm actually a human" link on the block page rather than giving the visitor a puzzle: how to report the issue to the page owner (hard on its own for normies) if you can't even load the page. This is just externalising issues that belong to the Cloudflare service.
  - methou 2y ago
    Some clients are more like a bot/service, imagine google reader that fetches and caches content for you. The client I’m currently using is miniflux, it also works in this way. I understand that there are some more interactive rss readers, but from personal experience it’s more like “hey I’m a good bot, let me in”
    - _Algernon_ 2y ago
      An rss reader is a user agent (ie. a software acting on behalf of its users). If you define rss readers as a bot (even if it is a good bot), you may as well call Firefox a bot (it also sends off web requests without explicit approval of each request by the browser).
      - sofixa 2y ago
        Their point was that the RSS reader does the scraping on its own in the background, without user input. If it can't read the page, it can't; it's not initiated by the user where the user can click on a "I'm not a bot, I promise" button.
    - viraptor 2y ago
      It was a mental skip, but the same idea. It would awesome if CF just allowed reporting issues at the point something gets blocked - regardless if it's a human or a bot. They're missing an "I'm misclassified" button for people actually affected without the third-party runaround.
dewey 2y ago
I’m using Miniflix and I always run into that on a few blogs which now I just stopped reading.
soraminazuki 2y ago
This is an issue with techdirt.com. I contacted them about this through their feedback form a long time ago, but the issue still remains unfortunately.
MarvinYork 2y ago
In any case, it blocks German Telekom users. There is an ongoing dispute between Cloudflare and Telekom as to who pays for the traffic costs. Telekom is therefore throttling connections to Cloudflare. This is the reason why we can no longer use Cloudflare.
- SSLy 2y ago
  as much as I am not a fan of cloudflare's practices, in this particular case DTAG seems to be the party at fault.
- nisa 2y ago
  There is no dispute. Telekom is not peering on public exchanges and wants ransom as in expensive private ip-transit contracts from everyone. Their customers are used as a bargain for this. Recently Meta stopped playing that game and Cloudflare never did afaik. Telekom could solve a part of this problem with a few 100k€ and a few weeks time if they would peer at the bigger German exchanges. If every big ISP would act like them the Internet would be dead.
hwj 2y ago
I had problems accessing Cloudflare-hosted websites via the Tor browser also. Don't know it that is still true.
whs 2y ago
My company runs a tech news website. We offer RSS feed as any Drupal website would, which content farm just scrape our RSS feed to rehost our content in full. This is usually fine for us - the content is CC-licensed and they do post the correct source. But they run thousands of different WordPress instances on the same IP and they individually fetch the feed. In the end we had to use Cloudflare to rate limit the RSS endpoint.
- kevincox 2y ago
  > In the end we had to use Cloudflare to rate limit the RSS endpoint. I think this is fine. You are solving a specific problem and still allowing some traffic. The problem with the Cloudflare default settings is that they block all requests leading to users failing to get any updates even when fetching the feed at a reasonable rate. BTW in this case another solution may just be to configure proper caching headers. Even if you only cache for 5min at a time that will be at most 1 request every 5min per Cloudflare caching location (I don't know the exact configuration but typically use ~5 locations per origin, so that would be only 1req/min which is trivial load and will handle both these inconsiderate scrapers and regular users. You can also configure all fetches to come from a single location and then you would only need to actually serve the feed once per 5min)
- yjftsjthsd-h 2y ago
  > In the end we had to use Cloudflare to rate limit the RSS endpoint. Isn't the correct solution to use CF to cache RSS endpoints aggressively?
  - whs 2y ago
    We do both, but the enterprise plan isn't as unlimited as the self service plans so we need to limit them as well. (It's not a large site but the Cloudflare contract is for every affiliated companies - it is funny when we serve news of ongoing Cloudflare outages on Cloudflare Enterprise)
prmoustache 2y ago
I believe this also pose issues to people running adblockers. I get tons of repetitive captchas on some websites. Also other companies offering similar services like imperva seems to be straight banning my ip after one visit to a website with uBlock Origin I first get a captcha, then a page saying I am not allowed, and whatever I do, even using an extensionless chrome browser with a new profile I can't visit it anymore because my ip is banned.
- acdha 2y ago
  One thing to keep in mind is that the modern web sees a lot of spam and scraping, and ad revenue has been sliding for years. If you make your activity look like a not, most operators will assume you’re not generating revenue and block you. It sucks but thank a spammer for the situation.
  - immibis 2y ago
    They should provide an API if they don't like scraping, but also, any sane scraper isn't really a problem, unless you are trying to enshittify your site by forcing people to use your app. I heard some AI scrapers are insane, and should be individually blocked.
    - Klonoar 2y ago
      Some AI scrapers have been proven to not report themselves as AI scrapers and mimic true users. This is part of what’s leading to the bludgeoning approach you see with blocking. They are not an individual thjng that can be blocked.
    - acdha 2y ago
      “Sane scraper” doesn’t have a definition or anyone to enforce it. Similarly, APIs aren’t magic - if you make things publicly available, people will harvest it whether that’s with a 90s-style bot making individual requests or a headless browser which runs the JavaScript you use to make API calls. The other thing to think about is the lack of enforcement: you can’t complain to the bot police when some dude in China decides to harvest your data, and if you try blocking by user-agent or IP you’ll play whack-a-mole trying to stay ahead of the bot operators who will spoof the former and churn the latter. After developing an appreciation for why security people talk about validating correctness rather than trying to enumerate badness, you’ll end up with a combination of rate-limiting and broader blocking for the same reasons. Yes, it’s no fun but the problem isn’t the sites but the people abusing the free services we’ve been given.
est 2y ago
Hmmm, that's why "feedburner" is^H^Hwas a thing, right? We have come to full circle.
- kevincox 2y ago
  Yeah, this is the recommendation that I usually give people who reach out to support. Feedburner tends to be on the whitelists to avoids this problem.
pointlessone 2y ago
I see this on a regular basis. My self-hosted RSS reader is blocked by Cloudflare even after my IP address was explicitly allowlisted by a few feed owners.
account42 2y ago
Or just normal human users with a niche browser like Firefox.
glub103011 2y ago
[dead]
wraptile 2y ago
Cloudflare has been the bane of my web existance on Thai IP and a Linux Firefox fingerprint. I wonder how much traffic is lost because of Cloudflare and of course none of that is reported to the web admins so everyone continues with their jolly ignorance. I wrote my own RSS bridge that scrapes websites using Scrapfly web scraping API that bypasses all that because it's so annoying that I can't even scrape some company's /blog that they are literally buying ads for but somehow have an anti-bot enabled that blocks all RSS readers. Modern web is so anti social that the web 2.0 guys should be rolling in their "everything will be connected with APIs" graves by now.
- vundercind 2y ago
  The late '90s-'00s solution was to blackhole address blocks associated with entire countries or continents. It was easily worth it for many US sites that weren't super-huge to lose the the 0.1% of legitimate requests they'd get from, say, China or Thailand or Russia, to cut the speed their logs scrolled at by 99%. The state of the art isn't much better today, it seems. Similar outcome with more steps.
hkt 2y ago
It also manages to break IRC bots that do things like show the contents of the title tag when someone posts a link. Another cloudy annoyance, albeit a minor one.
shaunpud 2y ago
Namesilo are the same, their csv/rss behind Cloudflare so don't even bother anymore with their auctions and their own interface is meh
anilakar 2y ago
...and there is a good number of people who see this as a feature, not a bug.
timnetworks 2y ago
RSS is the future that is being kept from us for twenty years already, fusion can kick bricks.
nfriedly 2y ago
Liliputing.com had this problem a couple of years ago. I emailed the author and he got it sorted out after a bit of back and forth.
015a 2y ago
Suggesting that website operators should allowlist RSS clients through the Cloudflare bot detection system via their user-agent is a rather concerning recommendation.
artooro 2y ago
This is a truly problematic issue that I've experienced as well. The best solution is probably for Cloudflare to figure out what normal RSS usage looks like and have a provision for that in their bot detection.
idunnoman1222 2y ago
Yes, the way to retain your privacy is to not use the Internet if you don’t like it, make your own Internet: assumedly one not funded by ads
hugoromano 2y ago
"could be blocking RSS users" it says it all "could". I use RSS on my websites, which are serviced by Cloudflare, and my users are not blocked. For that, fine-tuning and setting Configuration Rules at Cloudflare Dashboard are required. Anyone on a free has access to 10 Configuration Rules. I prefer using Cloudflare Workers to tune better, but there is a cost. My suggestion for RSS these days is to reduce the info on RSS feed to teasers, AI bots are using RSS to circumvent bans, and continue to scrape.
srmarm 2y ago
I'd have thought the website owner whitelisting their RSS feed URI (or pattern matching *.xml/*.rss) might be better than doing it based on the users agent string. For one you'd expect bot traffic on these end points and you're also not leaving a door open to anyone who fakes their user agent. Looks like it should be possible under the WAF
wenbin 2y ago
At Listen Notes, we rely heavily on Cloudflare to manage and protect our services, which cater to both human users and scripts/bots. One particularly effective strategy we've implemented is using separate subdomains for services designed for different types of traffic, allowing us to apply customized firewall and page rules to each subdomain. For example: - www. listennotes.com is dedicated to human users. E.g., https://www.listennotes.com/podcast-realtime/ https://www.listennotes.com/podcast-realtime/ - feeds. listennotes.com is tailored for bots, providing access to RSS feeds. Eg., https://feeds.listennotes.com/listen/wenbin-fangs-podcast-playlist-kr3-ta28cJu/rss/ https://feeds.listennotes.com/listen/wenbin-fangs-podcast-pl... - audio. listennotes.com serves both humans and bots, handling audio URL proxies. E.g., https://audio.listennotes.com/e/p/1a0b2d081cae4d6d9889c496513646e2/ https://audio.listennotes.com/e/p/1a0b2d081cae4d6d9889c49651... This subdomain-based approach enables us to fine-tune security and performance settings for each type of traffic, ensuring optimal service delivery.
- kevindamm 2y ago
  Where do you put your sitemap (or its equivalent)? Looking at the site, I don't notice one in the metadata but I do see a "site index" on the www subdomain, though possibly that's intended for humans not bots? I think the usual recommendation is to have a sitemap per subdomain and not mix them, but clearly they're meant for bots not humans...
  - wenbin 2y ago
    Great question. We only need to provide the sitemap (with custom paths, not publicly available) in a few specific places, like Google Search Console. This means the rules for managing sitemaps are quite manageable. It’s not a perfect setup, but once we configure it, we can usually leave it untouched for a long time.
butz 2y ago
Not "could" but it is actually blocking. Very annoying when government website does that, as usually it is next to impossible to explain the issue and ask for a fix. And even if the fix is made, it is reverted several weeks later. Other websites does that too, it was funny when one website was asking RSS reader to resolve captcha and prove they are human.
elwebmaster 2y ago
Using Cloudflare on your website could be blocking Safari users, Chrome users, or just any users. It’s totally broken. They have no way of measuring the false positives. Website owners are paying for it in lost revenue. And poor users who lose access for no fault of their own. Until some C-level exec at a BigTech randomly gets blocked and makes noise. But even then, Cloudflare will probably just whitelist that specific domain/IP. It is very interesting how I have never been blocked when trying to access Cloudflare itself, only blocked on their customer’s sites.
pentagrama 2y ago
Can you whitelists urls to be readead by bot on Cloudflare? Maybe this is a good solution, and there you can put your RSS feeds, sitemaps, and other content for bots. Also Cloudflare can make a dedicated fields to whitelists RSS and Sitemaps on the admin panel so users can discover more easily that they may don't want block those bots. Can you whitelist URLs to be read by bots on Cloudflare? Maybe this is a good solution, where you as a site mantainer can include your RSS feeds, sitemaps, and other content for bots. Also, Cloudflare could ship a feature by creating a dedicated section in the admin panel to let the user add and whitelist RSS feeds and sitemaps, making it easier (and educate) users to avoid blocking those bots who aren't a threat to your site, of course sill considering rules to avoid DDOS on this urls, like massive requests or stuff that common bots from RSS readers don't do.
ectospheno 2y ago
I love that I get a cloudflare human check on almost every page they serve for customers except for when I login to my cloudflare account. Good times.
- deleted 2y ago
  [deleted]
conesus 2y ago
I run NewsBlur[0] and I’ve been battling this issue of NewsBlur fetching 403s across the web for months now. My users are revolting and asking for refunds. I’ve tried emailing dozens of site owners and publishers and only two of them have done the work of whitelisting their RSS feed. It’s maddening and is having a real negative effect on NewsBlur. NewsBlur is an open-source RSS news reader (full source available at [1]), something we should all agree is necessary to support the open web! But Cloudflare blocking all of my feed fetchers is bizarre behavior. And we’re on the verified bots list for years, but it hasn’t made a difference. Let me know what I can do. NewsBlur publishes a list of IPs that it uses for feed fetching that I've shared with Cloudflare but it hasn't made a difference. I'm hoping Cloudflare uses the IP address list that I publish and adds them to their allowlist so NewsBlur can keep fetching (and archiving) millions of feeds. [0]: https://newsblur.com https://newsblur.com [1]: https://github.com/samuelclay/NewsBlur https://github.com/samuelclay/NewsBlur
- AyyEye 2y ago
  Three consenting parties trying to use their internet blocked by a single intermediary that's too big to care is just gross. It's the web we deserve.
  - deleted 2y ago
    [deleted]
  - eddythompson80 2y ago
    > Three consenting parties Clearly they are not 100% consenting, or at best one of them (the content publisher) is misconfiguring/misunderstanding their setup. They enabled RSS on their service, then setup a rule to require human verification for accessing that RSS feed. It's like a business advertising a singles only area, then hiring a security company and telling them to only allow couples in the building.
    - AyyEye 2y ago
      If Cloudflare was honest and upfront about the tradeoffs being made and the fact that it's still going to require configuration and maintenance work they'd have significantly less customers.
tandav 2y ago
As an admin of my personal website, I completely disable all Cloudflare features and use it only for DNS and domain registration. I also stop following websites that use Cloudflare checks or cookie popups (cookies are fine, but the popups are annoying).
renewiltord 2y ago
Ah, the Cloudflare free plan does not automatically turn these on. I know since I use it for some small things and don't have these on. I wouldn't use User-Agent filtering because those are spoofable. But putting feeds on a separate URL is probably a good idea. Right now the feed is actually generated on request for these sites, so caching it is probably a good idea anyway. I can just rudimentarily do that by periodically generating and copying it over.
drudru 2y ago
I noticed this a while back when I was trying to read cloudflare's own blog. Periodically they would block my newsreader. I ended up just dropping their feed. I am glad to see other people calling out the problem. Hopefully, a solution will emerge.
davidfischer 2y ago
My employer, Read the Docs is a heavy user of Cloudflare. It's actually hard to imagine serving as much traffic as we do as cheaply as we can without them. That said, for publicly hosted open source documentation, we turn down the security settings almost all the way. Security level is set to "essentially off" (that's the actual setting name), no browser integrity check, TOR friendly (onion routing on), etc. We still have rate limits in place but they're pretty generous (~4 req/s sustained). For sites that don't require a login and don't accept inbound leads or something like that, that's probably around the right level. Our domains where doc authors manage their docs have higher security settings. That said, being too generous can get you into trouble so I understand why people crank up the settings and just block some legitimate traffic. See our past post where AI scrapers scraped almost 100TB (https://news.ycombinator.com/item?id=41072549 https://news.ycombinator.com/item?id=41072549).
imartin2k 2y ago
I’m happy to see that a post regarding the use of RSS gets so much attention on HN. It’s a good sign. As I basically live in my feed reader since 2007 or so, one of my greatest fears is the slow demise of RSS by way of reduced support of RSS feeds by websites owners.
3np 2y ago
Also: Sign in on gitlab.com is broken for me on Tor Browser because of an infinite "Verify you are human" refresh/redirect loop...
deleted 2y ago
[deleted]
PeterStuer 2y ago
I was bitten by this as well. My product retrieves RSS feeds from public government sites, and suddely I'm blocked by cloudflair's antibotting for tryng to access a page that was specifically created for machine consumption. It is not that the website owner or publisher intend to block this. They are unaware that turng on Cloudflare will block everything, even stuff allowed to be consumed according to robots.txt . P.S. when I mentioned this here on HN a few weeks back, it was implied that I probably did not respect robots.txt ( I do, Cloudflair does not) or that I should get in touch with the site administrators (impossible to do in any reasonably effective way at scale).
samplifier 2y ago
I've noticed that Old Reddit still supports RSS feeds without returning a 403 error. This is in contrast to the main site, which often blocks RSS requests. Here are some DNS details: The main Reddit site (www.reddit.com) uses Fastly. Old Reddit (old.reddit.com) also uses Fastly. However, the "vomit" address (which often returns 403s for RSS requests) uses AWS DNS. Is Old Reddit not behind Cloudflare, or is there another reason why it handles RSS requests differently?
- samplifier 2y ago
  Ignore above. Brainfarted. It doesn't work.
qwertyuiop_ 2y ago
I have always suspected cloudflare being a classic intelligence community op. Just like Google was funded by qinetq

559 points by campuscodi 2y ago

I dislike advice of whitelisting specific readers by user-agent. Not only is this endless manual work that will only solve the problem for a subset of users but it also is easy to bypass by malicious actors. My recommendation would be to create a page rule that disables bot blocking for your feeds. This will fix the problem for all readers with no ongoing maintenance. If you are worried about DoS attacks that may hammer on your feeds then you can use the same configuration rule to ignore the query string for cache keys (if your feed doesn't use query strings) and overriding the caching settings if your server doesn't set the proper headers. This way Cloudflare will cache your feed and you can serve any number of visitors without putting load onto your origin. As for Cloudflare fixing the defaults, it seems unlikely to happen. It has been broken for years, Cloudflare's own blog is affected. They have been "actively working" on fixing it for at least 2 years according to their VP of product: https://news.ycombinator.com/item?id=33675847 https://news.ycombinator.com/item?id=33675847

vaylian 2y ago
I don't know if cloudflare offers it, but whitelisting the URL of the RSS feed would be much more effective than filtering user agents.
- derkades 2y ago
  Yes it supports it, and I think that's what the parent comment was all about
  - BiteCode_dev 2y ago
    Specifically, whitelisting the URL for the bot protection, but not the cache, so that you are still somewhat protected against adversarial use.
    - londons_explore 2y ago
      An adversary can easily send no-cache headers to bust the cache.
      - acdha 2y ago
        The CDN can choose whether to honor those. That hasn’t been an effective adversarial technique since the turn of the century.
        londons_explore 2y ago
        does cloudflare give such an option? Even for non-paid accounts?
        acdha 2y ago
        They ignore request cache control headers, I believe unconditionally so you’d have to disable caching for the endpoints which clients are allowed to request uncached.
- deleted 2y ago
  [deleted]
- jks 2y ago
  Yes, you can do it with a "page rule", which the parent comment mentioned. The CloudFlare free tier has a budget of three page rules, which might mean that you have to bundle all your rss feeds in one folder so they share a path prefix.
a-french-anon 2y ago
And for those of us using sfeed, the default UA is Curl's.
benregenspan 2y ago
AI crawlers have changed the picture significantly and in my opinion are a much bigger threat to the open web than Cloudflare. The training arms race has drastically increased bot traffic, and the value proposition behind that bot traffic has inverted. Previously many site operators could rely on the average automated request being net-beneficial to the site and its users (outside of scattered, time-limited DDoS attacks) but now most of these requests represent value extraction. Combine this with a seemingly related increase in high-volume bots that don't respect robots.txt and don't set a useful User-Agent, and using a heavy-handed firewall becomes a much easier business decision, even if it may target some desirable traffic (like valid RSS requests).

rcarmo 2y ago

Ironically, the site seems to currently be hugged to death, so maybe they should consider using Cloudflare to deal with HN traffic?

timeon 2y ago
If it is unintentional DDoS, we can wait. Not everything needs to be on demand.
- dewey 2y ago
  The website is built to get attention, the attention is here right now. Nobody will remember to go back tomorrow and read the site again when it’s available.
  - BlueTemplar 2y ago
    I'm not sure an open web can exist under this kind of assumption... Once you start chasing views, it's going to come at the detriment of everything else.
    - dewey 2y ago
      This happened at least 15 years ago and we are doing okay.
sofixa 2y ago
Doesn't have to be using CloudFlare, just a static web host that will be able to scale to infinity (of which CloudFlare is one with Pages, but there's also Google with Firebase Hosting, AWS with Amplify, Microsoft with something in Azure with a verbose name, Netlify, Vercel, GitHub Pages, etc etc etc).
- kawsper 2y ago
  Or just add Varnish or Nginx configured with a cache in front.

veeti 2y ago

I believe that disabling "Bot Fight Mode" is not enough, you may also need to create a rule to disable "Browser Integrity Check".

belkinpower 2y ago

I maintain an RSS reader for work and Cloudflare is the bane of my existence. Tons of feeds will stop working at random and there’s nothing we can do about it except for individually contacting website owners and asking them to add an exception for their feed URL.

sammy2255 2y ago
Unfortunately its not really Cloudflare but webadmins who have configured it to block everything thats not a browser, whether unknowingly or not
- afandian 2y ago
  If Cloudflare offer a product, for a particular purpose, that breaks existing conventions of that purpose, then it’s Cloudflare.
  - sammy2255 2y ago
    Not really. You wouldn’t complain to a fence company for blocking a path if there were hired to do exactly that
    - gsich 2y ago
      They are enablers. They get part of the blame.
    - shakna 2y ago
      Yes, I would. Experts are expected to relay back to their client with their thoughts on a matter, not just blindly do as they're told. Your builder is meant to do their due diligence, which includes making recommendations.
  - echoangle 2y ago
    Well it doesn’t break the conventions of the purpose they offer it for. Cloudflare attempts to block non-human users, and this is supposed to be used for human-readable websites. If someone puts cloudflare in front of a RSS feed, that’s user error. It’s like someone putting a captcha in front of an API and then complaining that the Captcha provider is breaking conventions.

amatecha 2y ago

I get blocked from websites with some regularity, running Firefox with strict privacy settings, "resist fingerprinting" etc. on OpenBSD. They just give a 403 Forbidden with no explanation, but it's only ever on sites fronted by CloudFlare. Good times. Seems legit.

BiteCode_dev 2y ago
Cloudflare is a fantastic service with an unmatched value proposition, but it's unfortunately slowly killing web privacy, with 1000s paper cuts. Another problem is "resist fingerprinting" prevents some canvas processing, and many websites like bluesky, linked in or substack uses canvas to handle image upload, so your images appear to be stripes of pixel. Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. I understand why people give up, trying to fight for your privacy is an uphill battle with no end in sight.
- madeofpalk 2y ago
  > Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. Is that true? At least on iOS you can log into the ChatGPT with same email/password as the website. I never use Google login for stuff and ChatGPT works fine for me.
  - BiteCode_dev 2y ago
    See other comment.
- KomoD 2y ago
  > Then you have mobile apps that just don't run if you don't have a google account, like chatgpt's native app. That's not true, I use ChatGPT's app on my phone without logging into a Google account. You don't even need any kind of account at all to use it.
  - BiteCode_dev 2y ago

erikrothoff 2y ago

As the owner of an RSS reader I love that they are making this more public. 30% of our support requests are ”my feed doesn’t” work. It sucks that the only thing we can say is ”contact the site owner, it’s their firewall”. And to be fair it’s not only Cloudflare, so many different firewall setups cause issues. It’s ironic that a public API endpoint meant for bots is blocked for being a bot.

ricardo81 2y ago

iirc even if you're listed as a "good bot" with Cloudflare, high security settings by the CF user can still result in 403s. No idea if CF already does this, but allowing users to generate access tokens for 3rd party services would be another way of easing access alongside their apparent URL and IP whitelisting.

mbo 2y ago

This is an active issue with Rate Your Music right now: https://rateyourmusic.com/rymzilla/view?id=6108 https://rateyourmusic.com/rymzilla/view?id=6108 Unfixed for 4 months.

jgrahamc 2y ago

My email is jgc@cloudflare.com. I'd like to hear from the owners of RSS readers directly on what they are experiencing. Going to ask team to take a closer look.

viraptor 2y ago
It's cool and all that you're making an exception here, but how about including a "no, really, I'm actually a human" link on the block page rather than giving the visitor a puzzle: how to report the issue to the page owner (hard on its own for normies) if you can't even load the page. This is just externalising issues that belong to the Cloudflare service.
- methou 2y ago
  Some clients are more like a bot/service, imagine google reader that fetches and caches content for you. The client I’m currently using is miniflux, it also works in this way. I understand that there are some more interactive rss readers, but from personal experience it’s more like “hey I’m a good bot, let me in”
  - _Algernon_ 2y ago
    An rss reader is a user agent (ie. a software acting on behalf of its users). If you define rss readers as a bot (even if it is a good bot), you may as well call Firefox a bot (it also sends off web requests without explicit approval of each request by the browser).
    - sofixa 2y ago
      Their point was that the RSS reader does the scraping on its own in the background, without user input. If it can't read the page, it can't; it's not initiated by the user where the user can click on a "I'm not a bot, I promise" button.
  - viraptor 2y ago
    It was a mental skip, but the same idea. It would awesome if CF just allowed reporting issues at the point something gets blocked - regardless if it's a human or a bot. They're missing an "I'm misclassified" button for people actually affected without the third-party runaround.

dewey 2y ago

I’m using Miniflix and I always run into that on a few blogs which now I just stopped reading.

soraminazuki 2y ago

This is an issue with techdirt.com. I contacted them about this through their feedback form a long time ago, but the issue still remains unfortunately.

MarvinYork 2y ago

In any case, it blocks German Telekom users. There is an ongoing dispute between Cloudflare and Telekom as to who pays for the traffic costs. Telekom is therefore throttling connections to Cloudflare. This is the reason why we can no longer use Cloudflare.

SSLy 2y ago
as much as I am not a fan of cloudflare's practices, in this particular case DTAG seems to be the party at fault.
nisa 2y ago
There is no dispute. Telekom is not peering on public exchanges and wants ransom as in expensive private ip-transit contracts from everyone. Their customers are used as a bargain for this. Recently Meta stopped playing that game and Cloudflare never did afaik. Telekom could solve a part of this problem with a few 100k€ and a few weeks time if they would peer at the bigger German exchanges. If every big ISP would act like them the Internet would be dead.

hwj 2y ago

I had problems accessing Cloudflare-hosted websites via the Tor browser also. Don't know it that is still true.

whs 2y ago

My company runs a tech news website. We offer RSS feed as any Drupal website would, which content farm just scrape our RSS feed to rehost our content in full. This is usually fine for us - the content is CC-licensed and they do post the correct source. But they run thousands of different WordPress instances on the same IP and they individually fetch the feed. In the end we had to use Cloudflare to rate limit the RSS endpoint.

kevincox 2y ago
> In the end we had to use Cloudflare to rate limit the RSS endpoint. I think this is fine. You are solving a specific problem and still allowing some traffic. The problem with the Cloudflare default settings is that they block all requests leading to users failing to get any updates even when fetching the feed at a reasonable rate. BTW in this case another solution may just be to configure proper caching headers. Even if you only cache for 5min at a time that will be at most 1 request every 5min per Cloudflare caching location (I don't know the exact configuration but typically use ~5 locations per origin, so that would be only 1req/min which is trivial load and will handle both these inconsiderate scrapers and regular users. You can also configure all fetches to come from a single location and then you would only need to actually serve the feed once per 5min)
yjftsjthsd-h 2y ago
> In the end we had to use Cloudflare to rate limit the RSS endpoint. Isn't the correct solution to use CF to cache RSS endpoints aggressively?
- whs 2y ago
  We do both, but the enterprise plan isn't as unlimited as the self service plans so we need to limit them as well. (It's not a large site but the Cloudflare contract is for every affiliated companies - it is funny when we serve news of ongoing Cloudflare outages on Cloudflare Enterprise)

prmoustache 2y ago

I believe this also pose issues to people running adblockers. I get tons of repetitive captchas on some websites. Also other companies offering similar services like imperva seems to be straight banning my ip after one visit to a website with uBlock Origin I first get a captcha, then a page saying I am not allowed, and whatever I do, even using an extensionless chrome browser with a new profile I can't visit it anymore because my ip is banned.

acdha 2y ago
One thing to keep in mind is that the modern web sees a lot of spam and scraping, and ad revenue has been sliding for years. If you make your activity look like a not, most operators will assume you’re not generating revenue and block you. It sucks but thank a spammer for the situation.
- immibis 2y ago
  They should provide an API if they don't like scraping, but also, any sane scraper isn't really a problem, unless you are trying to enshittify your site by forcing people to use your app. I heard some AI scrapers are insane, and should be individually blocked.
  - Klonoar 2y ago
    Some AI scrapers have been proven to not report themselves as AI scrapers and mimic true users. This is part of what’s leading to the bludgeoning approach you see with blocking. They are not an individual thjng that can be blocked.
  - acdha 2y ago
    “Sane scraper” doesn’t have a definition or anyone to enforce it. Similarly, APIs aren’t magic - if you make things publicly available, people will harvest it whether that’s with a 90s-style bot making individual requests or a headless browser which runs the JavaScript you use to make API calls. The other thing to think about is the lack of enforcement: you can’t complain to the bot police when some dude in China decides to harvest your data, and if you try blocking by user-agent or IP you’ll play whack-a-mole trying to stay ahead of the bot operators who will spoof the former and churn the latter. After developing an appreciation for why security people talk about validating correctness rather than trying to enumerate badness, you’ll end up with a combination of rate-limiting and broader blocking for the same reasons. Yes, it’s no fun but the problem isn’t the sites but the people abusing the free services we’ve been given.

est 2y ago

Hmmm, that's why "feedburner" is^H^Hwas a thing, right? We have come to full circle.

kevincox 2y ago
Yeah, this is the recommendation that I usually give people who reach out to support. Feedburner tends to be on the whitelists to avoids this problem.

pointlessone 2y ago

I see this on a regular basis. My self-hosted RSS reader is blocked by Cloudflare even after my IP address was explicitly allowlisted by a few feed owners.

account42 2y ago

Or just normal human users with a niche browser like Firefox.

glub103011 2y ago

[dead]

wraptile 2y ago

Cloudflare has been the bane of my web existance on Thai IP and a Linux Firefox fingerprint. I wonder how much traffic is lost because of Cloudflare and of course none of that is reported to the web admins so everyone continues with their jolly ignorance. I wrote my own RSS bridge that scrapes websites using Scrapfly web scraping API that bypasses all that because it's so annoying that I can't even scrape some company's /blog that they are literally buying ads for but somehow have an anti-bot enabled that blocks all RSS readers. Modern web is so anti social that the web 2.0 guys should be rolling in their "everything will be connected with APIs" graves by now.

vundercind 2y ago
The late '90s-'00s solution was to blackhole address blocks associated with entire countries or continents. It was easily worth it for many US sites that weren't super-huge to lose the the 0.1% of legitimate requests they'd get from, say, China or Thailand or Russia, to cut the speed their logs scrolled at by 99%. The state of the art isn't much better today, it seems. Similar outcome with more steps.

hkt 2y ago

It also manages to break IRC bots that do things like show the contents of the title tag when someone posts a link. Another cloudy annoyance, albeit a minor one.

shaunpud 2y ago

Namesilo are the same, their csv/rss behind Cloudflare so don't even bother anymore with their auctions and their own interface is meh

anilakar 2y ago

...and there is a good number of people who see this as a feature, not a bug.

timnetworks 2y ago

RSS is the future that is being kept from us for twenty years already, fusion can kick bricks.

nfriedly 2y ago

Liliputing.com had this problem a couple of years ago. I emailed the author and he got it sorted out after a bit of back and forth.

015a 2y ago

Suggesting that website operators should allowlist RSS clients through the Cloudflare bot detection system via their user-agent is a rather concerning recommendation.

artooro 2y ago

This is a truly problematic issue that I've experienced as well. The best solution is probably for Cloudflare to figure out what normal RSS usage looks like and have a provision for that in their bot detection.

idunnoman1222 2y ago

Yes, the way to retain your privacy is to not use the Internet if you don’t like it, make your own Internet: assumedly one not funded by ads

hugoromano 2y ago

"could be blocking RSS users" it says it all "could". I use RSS on my websites, which are serviced by Cloudflare, and my users are not blocked. For that, fine-tuning and setting Configuration Rules at Cloudflare Dashboard are required. Anyone on a free has access to 10 Configuration Rules. I prefer using Cloudflare Workers to tune better, but there is a cost. My suggestion for RSS these days is to reduce the info on RSS feed to teasers, AI bots are using RSS to circumvent bans, and continue to scrape.

srmarm 2y ago

I'd have thought the website owner whitelisting their RSS feed URI (or pattern matching *.xml/*.rss) might be better than doing it based on the users agent string. For one you'd expect bot traffic on these end points and you're also not leaving a door open to anyone who fakes their user agent. Looks like it should be possible under the WAF

wenbin 2y ago

At Listen Notes, we rely heavily on Cloudflare to manage and protect our services, which cater to both human users and scripts/bots. One particularly effective strategy we've implemented is using separate subdomains for services designed for different types of traffic, allowing us to apply customized firewall and page rules to each subdomain. For example: - www. listennotes.com is dedicated to human users. E.g., https://www.listennotes.com/podcast-realtime/ https://www.listennotes.com/podcast-realtime/ - feeds. listennotes.com is tailored for bots, providing access to RSS feeds. Eg., https://feeds.listennotes.com/listen/wenbin-fangs-podcast-playlist-kr3-ta28cJu/rss/ https://feeds.listennotes.com/listen/wenbin-fangs-podcast-pl... - audio. listennotes.com serves both humans and bots, handling audio URL proxies. E.g., https://audio.listennotes.com/e/p/1a0b2d081cae4d6d9889c496513646e2/ https://audio.listennotes.com/e/p/1a0b2d081cae4d6d9889c49651... This subdomain-based approach enables us to fine-tune security and performance settings for each type of traffic, ensuring optimal service delivery.

kevindamm 2y ago
Where do you put your sitemap (or its equivalent)? Looking at the site, I don't notice one in the metadata but I do see a "site index" on the www subdomain, though possibly that's intended for humans not bots? I think the usual recommendation is to have a sitemap per subdomain and not mix them, but clearly they're meant for bots not humans...
- wenbin 2y ago
  Great question. We only need to provide the sitemap (with custom paths, not publicly available) in a few specific places, like Google Search Console. This means the rules for managing sitemaps are quite manageable. It’s not a perfect setup, but once we configure it, we can usually leave it untouched for a long time.

butz 2y ago

Not "could" but it is actually blocking. Very annoying when government website does that, as usually it is next to impossible to explain the issue and ask for a fix. And even if the fix is made, it is reverted several weeks later. Other websites does that too, it was funny when one website was asking RSS reader to resolve captcha and prove they are human.

elwebmaster 2y ago

Using Cloudflare on your website could be blocking Safari users, Chrome users, or just any users. It’s totally broken. They have no way of measuring the false positives. Website owners are paying for it in lost revenue. And poor users who lose access for no fault of their own. Until some C-level exec at a BigTech randomly gets blocked and makes noise. But even then, Cloudflare will probably just whitelist that specific domain/IP. It is very interesting how I have never been blocked when trying to access Cloudflare itself, only blocked on their customer’s sites.

pentagrama 2y ago

Can you whitelists urls to be readead by bot on Cloudflare? Maybe this is a good solution, and there you can put your RSS feeds, sitemaps, and other content for bots. Also Cloudflare can make a dedicated fields to whitelists RSS and Sitemaps on the admin panel so users can discover more easily that they may don't want block those bots. Can you whitelist URLs to be read by bots on Cloudflare? Maybe this is a good solution, where you as a site mantainer can include your RSS feeds, sitemaps, and other content for bots. Also, Cloudflare could ship a feature by creating a dedicated section in the admin panel to let the user add and whitelist RSS feeds and sitemaps, making it easier (and educate) users to avoid blocking those bots who aren't a threat to your site, of course sill considering rules to avoid DDOS on this urls, like massive requests or stuff that common bots from RSS readers don't do.

ectospheno 2y ago

I love that I get a cloudflare human check on almost every page they serve for customers except for when I login to my cloudflare account. Good times.

deleted 2y ago
[deleted]

conesus 2y ago

I run NewsBlur[0] and I’ve been battling this issue of NewsBlur fetching 403s across the web for months now. My users are revolting and asking for refunds. I’ve tried emailing dozens of site owners and publishers and only two of them have done the work of whitelisting their RSS feed. It’s maddening and is having a real negative effect on NewsBlur. NewsBlur is an open-source RSS news reader (full source available at [1]), something we should all agree is necessary to support the open web! But Cloudflare blocking all of my feed fetchers is bizarre behavior. And we’re on the verified bots list for years, but it hasn’t made a difference. Let me know what I can do. NewsBlur publishes a list of IPs that it uses for feed fetching that I've shared with Cloudflare but it hasn't made a difference. I'm hoping Cloudflare uses the IP address list that I publish and adds them to their allowlist so NewsBlur can keep fetching (and archiving) millions of feeds. [0]: https://newsblur.com https://newsblur.com [1]: https://github.com/samuelclay/NewsBlur https://github.com/samuelclay/NewsBlur

AyyEye 2y ago
Three consenting parties trying to use their internet blocked by a single intermediary that's too big to care is just gross. It's the web we deserve.
- deleted 2y ago
  [deleted]
- eddythompson80 2y ago
  > Three consenting parties Clearly they are not 100% consenting, or at best one of them (the content publisher) is misconfiguring/misunderstanding their setup. They enabled RSS on their service, then setup a rule to require human verification for accessing that RSS feed. It's like a business advertising a singles only area, then hiring a security company and telling them to only allow couples in the building.
  - AyyEye 2y ago
    If Cloudflare was honest and upfront about the tradeoffs being made and the fact that it's still going to require configuration and maintenance work they'd have significantly less customers.

tandav 2y ago

As an admin of my personal website, I completely disable all Cloudflare features and use it only for DNS and domain registration. I also stop following websites that use Cloudflare checks or cookie popups (cookies are fine, but the popups are annoying).

renewiltord 2y ago

Ah, the Cloudflare free plan does not automatically turn these on. I know since I use it for some small things and don't have these on. I wouldn't use User-Agent filtering because those are spoofable. But putting feeds on a separate URL is probably a good idea. Right now the feed is actually generated on request for these sites, so caching it is probably a good idea anyway. I can just rudimentarily do that by periodically generating and copying it over.

drudru 2y ago

I noticed this a while back when I was trying to read cloudflare's own blog. Periodically they would block my newsreader. I ended up just dropping their feed. I am glad to see other people calling out the problem. Hopefully, a solution will emerge.

davidfischer 2y ago

My employer, Read the Docs is a heavy user of Cloudflare. It's actually hard to imagine serving as much traffic as we do as cheaply as we can without them. That said, for publicly hosted open source documentation, we turn down the security settings almost all the way. Security level is set to "essentially off" (that's the actual setting name), no browser integrity check, TOR friendly (onion routing on), etc. We still have rate limits in place but they're pretty generous (~4 req/s sustained). For sites that don't require a login and don't accept inbound leads or something like that, that's probably around the right level. Our domains where doc authors manage their docs have higher security settings. That said, being too generous can get you into trouble so I understand why people crank up the settings and just block some legitimate traffic. See our past post where AI scrapers scraped almost 100TB (https://news.ycombinator.com/item?id=41072549 https://news.ycombinator.com/item?id=41072549).

imartin2k 2y ago

I’m happy to see that a post regarding the use of RSS gets so much attention on HN. It’s a good sign. As I basically live in my feed reader since 2007 or so, one of my greatest fears is the slow demise of RSS by way of reduced support of RSS feeds by websites owners.

3np 2y ago

Also: Sign in on gitlab.com is broken for me on Tor Browser because of an infinite "Verify you are human" refresh/redirect loop...

deleted 2y ago

[deleted]

PeterStuer 2y ago

I was bitten by this as well. My product retrieves RSS feeds from public government sites, and suddely I'm blocked by cloudflair's antibotting for tryng to access a page that was specifically created for machine consumption. It is not that the website owner or publisher intend to block this. They are unaware that turng on Cloudflare will block everything, even stuff allowed to be consumed according to robots.txt . P.S. when I mentioned this here on HN a few weeks back, it was implied that I probably did not respect robots.txt ( I do, Cloudflair does not) or that I should get in touch with the site administrators (impossible to do in any reasonably effective way at scale).

samplifier 2y ago

I've noticed that Old Reddit still supports RSS feeds without returning a 403 error. This is in contrast to the main site, which often blocks RSS requests. Here are some DNS details: The main Reddit site (www.reddit.com) uses Fastly. Old Reddit (old.reddit.com) also uses Fastly. However, the "vomit" address (which often returns 403s for RSS requests) uses AWS DNS. Is Old Reddit not behind Cloudflare, or is there another reason why it handles RSS requests differently?

samplifier 2y ago
Ignore above. Brainfarted. It doesn't work.

qwertyuiop_ 2y ago

I have always suspected cloudflare being a classic intelligence community op. Just like Google was funded by qinetq