21 ms·
Feed readers which don't take "no" for an answer
- RA2lover 2y agoRelated: https://news.ycombinator.com/item?id=42470035 https://news.ycombinator.com/item?id=42470035
- jannes 2y agoThe HTTP protocol is a lost art. These days people don't even look at the status code and expect some mumbo jumbo JSON payload explaining the error.
- klntsky 2y agoI would argue that HTTP statuses are a bad design decision, because they are intended to be consumed by apps, but are not app-specific. They are effectively a part of every API automatically without considerations whether they are needed. People often implement error handling using constructs like regexp matching on status codes, while with domain-specified errors it would be obvious what exactly is the range of possible errors. Moreover, when people do implement domain errors, they just have to write more code to handle two nested levels of branching.
- marcosdumay 2y ago> because they are intended to be consumed by apps, but are not app-specific Well, good luck designing any standard app-independent protocol that works and doesn't do that. And yes, you must handle two nested levels of branching. That's how it works. The only improvement possible to make it clearer is having codes for API specific errors... what 400 and 500 aren't exactly. But then, that doesn't gain you much.
- throw0101b 2y ago> I would argue that HTTP statuses are a bad design decision, because they are intended to be consumed by apps, but are not app-specific. Perhaps put the app-specific part in the body of the reply. In the RFC they give a human specific reply to (presumably) be displayed in the browser: HTTP/1.1 429 Too Many Requests Content-Type: text/html Retry-After: 3600 <html> <head> <title>Too Many Requests</title> </head> <body> <h1>Too Many Requests</h1> <p>I only allow 50 requests per hour to this Web site per logged in user. Try again soon.</p> </body> </html> * https://datatracker.ietf.org/doc/html/rfc6585#section-4 https://datatracker.ietf.org/doc/html/rfc6585#section-4 * https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 But if the URL is specific to an API, you can document that you will/may give further debugging details (in text, JSON, XML, whatever).
- est 2y ago> error handling using constructs like regexp matching on status codes Oh the horror. I would assume the practice is encourage by "RESTful" people?
- AznHisoka 2y agoI dont look at the code because its wrong sometimes. Some pages return a 200 yet display an error in the page
- DaSHacka 2y agoNothing more annoying than a 200 response when the server 'successfully' serves a 404 page
- CodesInChaos 2y agoReturning a 3xx redirect to an generic error page is even worse than 200.
- KomoD 2y agoThat's because a lot of people refuse to use status codes properly, like just using 200 everywhere.
- kstrauser 2y agoA colleague who should’ve known better argued that a 404 response to an API call was confusing because we were, in fact, successfully returning a response to the client. We had a long talk about that afterward.
- Joker_vD 2y agoNo, it is pretty confusing: the difference between 404 from hitting an endpoint that the server doesn't serve (because you forgot to expose this endpoint, oops!) and a 404 that means "we've successfully performed the search in our DB for the business entity you've requested and guarantee you that it does not exist" is rather difficult to tell programmatically.
- yjftsjthsd-h 2y agoI'm open to arguing about which error to return in each case, but surely we can agree that neither of those warrant a 200?
- echoangle 2y agoWhy not? I wouldn’t say „I performed the search and there’s 0 results“ is an error condition. It’s just the result of a search, and everything went fine.
- yjftsjthsd-h 2y agoHm, maybe? I guess it depends on what we mean by search; if myapp.com/search?someproduct finds that there are 0 matches then yeah that's probably a 200, but if myapp.com/products/123456 fails because no product has id 123456 then that's a textbook 404.
- deleted 2y ago[deleted]
- kelsey98765431 2y agoif you have to 429 people for an rss feed the problem is you
- ramses0 2y agoIf you don't stop at red lights, the problem is other people. /s
- dxdm 2y agoNobody owes these people and their feed readers a 200 whenever they want one.
- deleted 2y ago[deleted]
- quectophoton 2y agoI think it's an acceptable response. Not only there's no SLA, but people are free to not provide a service to misbehaving user agents. It's like rejecting connections from Tor. If anything, a 429 is a nice heads up. It could have been worse; she could have redirected those requests to a separate URL with an... unpleasant content, like a certain domain that redirects to I-don't-know-what whenever they detect the Referer header is from HN.
- redleader55 2y agoAs interesting as that site is, and as much as I sympathise with the author's plight, that site's behavior is so anti-me that I'm going to ignore it whenever/wherever it pops up. I'm not trolling the author, I'm not calling them names or anything, I was just interested in the technical stuff. I wish them good luck.
- sangnoir 2y agoI think that's the whole point - that author won't begrudge you for not visiting their site. They detest HN ideologically, so losing HNer traffic won't ruin their day.
- generationP 2y agoRejecting every unconditional GET after the first? That sounds a bit excessive. What if the reader crashed after the first and lost the data?
- brookst 2y agoIt’s a RSS feed. In that case, wait until the specified time and try again and any missed article will appear then. If it is constantly crashing so articles never get loaded, fix that.
- aleph_minus_one 2y ago> If it is constantly crashing so articles never get loaded, fix that. This often requires to do lots of tests against the endpoint, which the server prohibits.
- im3w1l 2y agoIf you are an rss-reader dev then you can set up a caching layer of your own.
- aleph_minus_one 2y ago> If you are an rss-reader dev then you can set up a caching layer of your own. But are RSS reader devs willing to jump through such hoops? I would claim that writing a (simple) RSS reader (using a programming language that provides suitable libraries) is something that would be rather easy for me, but setting up a caching layer would (because I have less knowledge about the latter topic) take a lot more research from my side concerning how to do it.
- im3w1l 2y agoSure, I have done such a thing myself and it was very simple. Let's say you do http_get(rss_address). Create a function http_cached_get, that looks for a recent cached response, and if none exists delegates to http_get and saves the response. In python this is like 10 lines.
- PittleyDunkin 2y ago[flagged]
- Tyr42 2y agoYou do pay for your link, right?
- rzzzt 2y agoWhat am I actually paying for? Is it "the entire speedometer" ie. 24/7 100% utilization of the advertised upload/download capability of the link? Why not?
- masklinn 2y ago> Why not? Because that costs way more than you're paying for your connection. The business model is predicated upon oversuscription of the ISP's network because near enough nobody does that.
- Dilettante_ 2y agoIt's wasted operations on the user's end, the server's and the network infrastructure. None of those are free.
- mdp2021 2y ago> to waste data Useless traffic. RSS has a structural problem: you download a "window" of data (the feeds between some timestamp in day A and some other in day B), which may or may not contain new data. You will easily lose some and receive a lot of duplicates. It would have been better to be able to first check if new data is available. (And even better to only download the new ones, and all of them - "everything after YY-MM-DD hh:mm:ss")
- paulryanrogers 2y agoSuch as via HEAD and Etag?
- Apreche 2y agoFeed readers should be sending the If-Modified-Since header and web sites should properly recognize it and send the 304 Unmodified response. This isn’t new tech.
- dartos 2y agoIf only people know of the standards
- graemep 2y agoThat is exactly what the article says.
- smallerize 2y agoThe article implies this but doesn't actually say it. It's nice to have the extra detail.
- avg_dev 2y agoWhile it might be nice if the article spelled out the header, I do believe that there is more than implication present. > 00:04:51 GET /w/atom.xml, unconditional. > Fulfilled with 200, 502 KB. > [...] > A 20 minute retry rate with unconditional requests is wasteful. [...] And If-Modified-Since makes a request conditional. https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#conditional_headers https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditiona...
- JadeNB 2y agoYou left out a further explicit mention of conditional requests: > Advised (via Retry-After header) to come back in one day since they are unwilling or unable to do conditional requests. But I think it's still unarguable that the post doesn't explicitly mention If-Modified-Since, which it's not obliged to do, but the mention of it here could be helpful to someone. So why fuss?
- strogonoff 2y agoA friend of mine co-runs a semi-popular semi-niche news site (for now more than a decade), and complains that recently traffic rose with bots masquerading as humans. How would they know? Well, because Google, in its omniscience, started to downrank them for faking views with bots (which they do not do): it shows bot percentage in traffic stats, and it skyrocketed relative to non-bot traffic (which is now less than 50%) as they started to fall from the front page (feeding the vicious circle). Presumably, Google does not know or care it is a bot when it serves ads, but correlates it later with the metrics it has from other sites that use GA or ads. Or, perhaps, Google spots the same anomalies that my friend (an old school sysadmin who pays attention to logs) did, such as the increase of traffic along with never seen before popularity among iPhone users (who are so tech savvy that they apparently do not require CSS), or users from Dallas who famously love their QQBrowser. I’m not going to list all telltale signs as the crowd here is too hype on LLMs (which is our going theory so far, it is very timely), but my friend hopes Google learns them quickly. These newcomers usually fake UA, use inconspicuous Western IPs (requests from Baidu/Tencent data center ranges do sign themselves as bots in UA), ignore robots.txt and load many pages very quickly. I would assume bot traffic increase would apply to feeds, since they are of as much use for LLM training purposes. My friend does not actually engage in stringent filtering like Rachel does, but I wonder how soon it becomes actually infeasible to operate a website with actual original content (which my friend co-writes) without either that or resorting to Cloudflare or the like for protection because of the domination of these creepy-crawlies. Edit: Google already downranked them, not threatened to downrank. Also, traffic rose but did not skyrocket, but relative amount of bot traffic skyrocketed. (Presumably without downranking the traffic would actually skyrocket.)
- blfr 2y agoQQBrowser users from Dallas are more likely to be Chinese using a VPN than bots, I would guess.
- strogonoff 2y agoThat much is clear, yeah. The VPN they use may not be a service advertised to public and featured in lists, however. Some of the new traffic did come directly from Tencent data center IP ranges and reportedly those bots signed themselves in UA. I can’t say whether they respect robots.txt because I am told their ranges were banned along with robots.txt tightening. However, US IP bots that remain unblocked and fake UA naturally ignore robot rules.
- 6510 2y agoI ban the feed for 24 hours if it doesnt work. I also design 2 new formats that no one (including myself) has ever implemented. https://go-here.nl/ess-and-nno https://go-here.nl/ess-and-nno enjoy
- Forge36 2y agoI couldn't find the tester. Thankfully the client i was tested... And it behaves poorly. Thankfully emacs has a client I can switch to!
- bombcar 2y agoAt some point instead of 429 it should return a feed with this post as always newest.
- 0xDEAFBEAD 2y agoOr a feed with only this post
- cpeterso 2y agoThat’s a great point: the client software isn’t listening to the server, so the server software should break the loop by escalating to the human reader. The message response should probably be even more direct with a call to action about their feed reader (naming it, if possible) causing server problems.
- internet2000 2y agoDoes anyone know if FreshRSS behaves properly here?
- reocha 2y agoEarlier article with some info on freshrss: https://rachelbythebay.com/w/2024/10/25/fs/ https://rachelbythebay.com/w/2024/10/25/fs/
- Havoc 2y agoBlocked for 2 hits in 20 minutes on a light protocol like rss? That seems hilariously aggressive to me, but her server her rules I guess.
- garfij 2y agoI believe if you read carefully, it's not blocked, it's rate limited to once daily, with very clear remediation steps included in the response.
- that_guy_iain 2y agoIf you understand what rate limiting is, you block them for a period of time. Let's stop being pedantic here. 72 requests per day is nothing and acting like it's mayhem is a bit silly. And for a lot of people would result in them getting possible news slower. Sure OP won't publish that often but their rate limiting is an edge case and should be treated as such. If they're blocked until the next day and nothing gets updated then the only person harmed is OP for being overly bothered by their HTTP logs. Sure it's their server and they can do whatever they want. But all this does is hurts the people trying to reach their blog.
- HomeDeLaPot 2y ago72 requests per day _per user with a naive feed reader_. This is a small personal blog with no ads that OP is self-hosting on her own hardware, so blocking all this junk traffic is probably saving her money. Plus she's calling attention to how feed readers can be improved!
- that_guy_iain 2y agoEven if they had 1000 feed readers which would be a massive amount for a blog, if you can't scale that cheaply, that's on you. As I pointed out, her blog and rate limiting are an extreme edge case, it would be silly for anyone to put effort into changing their feed reader for a single small blog. It's bad product management.
- nilslindemann 2y agoI am stupid, why not just return an HTML document explaining the issue, when there is such an incorrect second request in 20 minutes, then blocking that IP for 24 hours? The feed reader software author has to react, otherwise its users will complain to him, no?
- ImPostingOnHN 2y agoIt might be clever to return an rss feed containing 1 item: the html document you mention.
- ruszki 2y agoThat’s what 429 return is for, which is mentioned in the article.
- Too 2y agoMost readers presumably keep showing the last valid response when they get an error, so that the user doesn’t notice. Returning a fake ok response, explaining that the reader is dumb, will lift attention to the user. Not that I would advocate for this solution, except for desperate moments.
- deleted 2y ago[deleted]
- PaulHoule 2y agoThis is why RSS for the birds. My RSS reader YOShInOn subscribes to 110 RSS feeds through Superfeedr which absolves me of the responsibility of being on the other side of Rachel's problem. With RSS you are always polling too fast or too slow; if you are polling too slow you might even miss items. When a blog gets posted Superfeedr hits an AWS lambda function that stores the entry in SQS so my RSS reader can update itself at its own pace. The only trouble is Superfeedr costs 10 cents a feed per month which is a good deal for an active feed such as comments from Hacker News or article from The Guardian but is not affordable for subscribing to 2000+ indy blogs which YOShInOn could handle just fine. I might yet write my own RSS head end, but there is something to say for protocols like ActivityPub and AT Protocol.
- rakoo 2y agoThat's why websub (formerly pubsubhubbub) was created and should be the proper solution, not a proprietary middleware
- PaulHoule 2y agoSuperfeedr is pubsubhubhub.
- rakoo 2y agoAws lambdas and sqs are not
- PaulHoule 2y agoBut it is just a way to answer a webhook. Sure I could get DNS to point to my ADSL connection and set something up in my router so that my home computer can answer the webhook but then I can never turn my computer off. On top of that I have about one power outage a month. It would also be non-proprietary to spend $5k on a server and $300 a month on colo costs but with AWS I can use what would be 1 cent of resources if I was optimizing that colo (with $10k of labor) and pay 10 cents for it (could really be spending up to $50 on a non-optimized colo, which is what I might have if I don't need to handle a billion webhooks a month) If I want to switch to Azure or some other service that could answer a webhook the labor involved is minuscule.
- donatj 2y agoOn the flip side, what percent of RSS feed generators actually support conditional requests? I've written many over the last twenty years and I can tell you plainly, none of the ones I wrote have. I never even considered the option or necessity. It's easy and cheap just to send everything. I guess static generators with a apache style web server probably do, but I can't imagine any dynamic generators bother to try to save the small handful of bytes.
- aendruk 2y agoFor another perspective, I can offer the data point that the one dynamic feed generator I’ve written supports both If-Modified-Since and If-None-Match, and that I considered that to be an obvious requirement from the beginning.
- wheybags 2y agoRss is pretty light. Even if you say it's too much to be re-sending, you could remove the content from the rss feed (so they need to click through to read it), which would shrink the feed size massively. Alternatively, remove old posts. Or do both. Hopefully you don't have some expensive code generating the feed on the fly, so processing overhead is negligible. But if it's not, cache the result and reset the cache every time you post. Surely this is easier than spending the effort and emotional bandwidth to care about this issue? I might be wrong here, but this feels more emotionally driven ("someone is wrong on the internet") than practical.
- gavinsyancey 2y agoAs a user of the RSS feed, please don't remove content from it so I have to click through. This makes it much less useful and more annoying to use.
- wheybags 2y agoI always click through regardless, because the rss text is probably missing formatting and images. I'll never be sure I'm getting a proper copy of the article unless I click through anyway.
- ruuda 2y agoI have a blog where I post a few posts per year. [1] /feed.xml is served with an Expires header of 24 hours. I wrote a tool that allows me to query the webserver logs using SQLite [2]. Over the past 90 days, these are the top 10 requesters grouped by ip address (remote_addr column redacted here): requests_per_day user_agent 283 Reeder/5050001 CFNetwork/1568.300.101 Darwin/24.2.0 274 CommaFeed/4.4.0 (https://github.com/Athou/commafeed) 127 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 52 NetNewsWire (RSS Reader; https://netnewswire.com/) 47 Tiny Tiny RSS/23.04-0578bf80 (https://tt-rss.org/) 47 Refeed Reader/v1 (+https://www.refeed.dev/) 46 Selfoss/2.18 (SimplePie/1.5.1; +https://selfoss.aditu.de) 41 Reeder/5040601 CFNetwork/1568.100.1.1.1 Darwin/24.0.0 39 Tiny Tiny RSS/23.04 (Unsupported) (https://tt-rss.org/) 34 FreshRSS/1.24.3 (Linux; https://freshrss.org) Reeder is loading the feed every 5 minutes, and in the vast majority of cases it’s getting a 301 response because it tries to access the http version that redirects to https. At least it has state and it gets 304 Not Modified in the remaining cases. If I order by body bytes served rather than number of requests (and group by remote_addr again), these are the worst consumers: body_megabytes_per_year user_agent 149.75943975 Refeed Reader/v1 (+https://www.refeed.dev/) 95.90771025 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 75.00080025 rss-parser 73.023702 Tiny Tiny RSS/24.09-0163884ef (Unsupported) (https://tt-rss.org/) 38.402385 Tiny Tiny RSS/24.11-42ebdb02 (https://tt-rss.org/) 37.984539 Selfoss/2.20-cf74581 (+https://selfoss.aditu.de) 30.3982965 NetNewsWire (RSS Reader; https://netnewswire.com/) 28.18013325 Tiny Tiny RSS/23.04-0578bf80 (https://tt-rss.org/) 26.330142 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 24.838461 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 The top consumer, Refeed, is responsible for about 2.25% of all egress of my webserver. (Counting only body bytes, not http overhead.) [1]: https://ruudvanasseldonk.com/writing https://ruudvanasseldonk.com/writing [2]: https://github.com/ruuda/sqlog/blob/d129db35da9bbf95d8c2e97d575b4d5beb3bb40c/queries/feed_readers.sql https://github.com/ruuda/sqlog/blob/d129db35da9bbf95d8c2e97d...
- mixmastamyk 2y agoI have a few feeds configured into Thunderbird but wasn’t reading them very often, so I “disabled” them to load manually. Despite this it tries to contact the sites often and, when not able to (firewall) goes into a frenzy of trying to contact them. All this despite being disabled. Disappointing combined with the various update sites it tries to contact every startup, which is completely unnecessary as well. Couple of times a week should be the maximum rate.
- euroderf 2y agowhich => that
- thaumasiotes 2y agoThey're exactly equivalent. What are you hoping to correct?
- euroderf 2y agoThey're obviously not. It's a Britticism (AFAICT) making inroads.
- wetpaws 2y ago[dead]
- sangnoir 2y agoOh no, not the British influencing the English language! I can't be arsed* about which vs. that when "on accident" has become semi-accepted (as the opposite of "on purpose"). Yuck.
- thaumasiotes 2y agoWhat do you hope to accomplish by making random false statements?
- euroderf 2y agoWot
- EdwardDiego 2y agoBritticisms? In English? The hell you say!
- euroderf 2y agoWell, we know they cannot spell.
- shepherdjerred 2y agoI like Rachel's writing, but I don't understand this recent crusade against RSS readers. Sure, they should work properly and optimizations can be made to reduce bandwidth and processing power. But... why not throw a CDN in front of your site and focus your energy somewhere else? I guess every problem has to be solved by someone, but this just seems like a very strange hill to die on.
- EdwardDiego 2y agoBecause she's old school sysadmin mate, likes running her own stuff her own way, fair enough. And she posts on it lots because she has a bunch of RSS clients pointed at her writing, because she's rather popular. And she'd rather people writing this stuff just learn HTTP properly, at least out of professionalism, if not courtesy. Hey, you might not, I might not, but we all choose our hills to die on. My personal hill is "It's lollies and biscuits, not candy and cookies".
- est 2y ago> But... why not throw a CDN in front of your site and focus your energy somewhere else? Yes it's been invented before, known as Feedburner, which was acquired & abandoned by Google.
- rollcat 2y ago> why not throw a CDN in front of your site [...] Because this is how the open web dies - one website at a time. It's already near-dead on the client side - web browsers are not really "user" agents, but agents of oligopolist corporations, that have a stake in abusing you[1]. It's been attempted before with WAP[2], then AMP. But effectively, we're almost there. [1]: https://www.5snb.club/posts/2023/do-not-stab/ https://www.5snb.club/posts/2023/do-not-stab/ [2]: https://news.ycombinator.com/item?id=42479172 https://news.ycombinator.com/item?id=42479172
- aaron695 2y ago[dead]