4 ms·

I like RSS for serving users trying to get updates from the site. But I've heard that it requires creating a massive feed file containing many articles, includi

I like RSS for serving users trying to get updates from the site. But I've heard that it requires creating a massive feed file containing many articles, including text and images, which can produce a large bandwidth load from clients that pull it frequently. As a result, some blogs (like christine.website) stopped serving article contents in the RSS feeds. Is it a problem in practice (cannot be solved by removing images or similar)? Could it be improved through smarter polling or some "last updated since" header? Or by using an "active" RPC-like protocol where the reader specifically queries for "list of all articles since this timestamp" and can fetch them together or individually, with or without images?

CarelessExpert 5y ago
> But I've heard that it requires creating a massive feed file containing many articles, including text and images So a) no, it doesn't contain images, just URLs to images where necessary; an RSS or ATOM feed is just a text XML document adhering to a certain schema, and b) the feed file is only as big as the site decides it needs to be. For example, a typical blog or news organization could generate a feed containing the last 24 hours worth of new content. Feed readers will poll that and pull in new content while ignoring existing content. In short: this should be a non-issue unless you're turning out massive amounts of new material on a regular basis, in which case you're probably a major news organization and can afford it. > As a result, some blogs (like christine.website) stopped serving article contents in the RSS feeds. This isn't because of bandwidth concerns. This is because they can't advertise in an RSS feed so they want to drive eyeballs to the site. Personally, I run ttrss and use a plugin that scrapes the content from the source site and embeds it right in my feed. As a result I only have to leave my feed reader if I really want to for some reason (e.g. to read comments on the originating site... like, say, HN!)
- kixiQu 5y ago
  To be clear, I don't think it's the case that christine.website has any advertising whatsoever (or even meaningful JS?) but I think your diagnosis is correct more broadly.
  - moehm 5y ago
    It also doesn't seem to be the case that christine.website strips their content from the feed. https://christine.website/blog.rss https://christine.website/blog.rss
mattowen_uk 5y ago
A LOT of sites only serve the article headline and maybe just the intro sentence for exactly this reason. Also by forcing you to click through, you land on the site proper and feature in their visitor counts, AND get to look at ads on the page too.
- pantulis 5y ago
  Unless you use a "reader mode" capable RSS reader, that is.
  - k1m 5y ago
    Or use something like this to convert partial feeds into full text before subscribing: http://ftr.fivefilters.org http://ftr.fivefilters.org (a project I work on).
jcrawfordor 5y ago
I don't think performance/load has ever been a real reason to only serve partial content. The actual reason has always been getting users onto the site where you can deliver advertising more flexibly along with analytics. For my extremely minimal site, the RSS feed is still smaller than normal pageloads because it only has graphics by reference. RSS readers scrape surprisingly often, but the file is small and highly cacheable, and most of them do a HEAD request only initially and never ask for the file if it hasn't changed. Despite accounting for a large percentage of requests, RSS clients account for only a small portion of traffic. The cool thing is that a lot of these RSS readers are shared (commercial feed readers, ttrss instances, etc) and so they are caching the feed on their end and actually saving me bandwidth vs an equivalent number of users accessing the site directly - a lot of these put the subscriber account in their UA string so I can see that e.g. Feedly is making regular requests during the day but has 30-some users behind those requests. It's a better deal from a cost-perspective than those 30-some people checking each day.
acdha 5y ago
The load issues were a concern with very early implementations around the turn of the century but they're really not now. Feeds are usually extremely cacheable and readers have supported conditional fetches using If-Modified-Since since the early days. These days the average site is serving considerably more data in JavaScript, not to mention the images and video which you're going to see in any case (they're just links in RSS). The real reason why sites started switching to partial content is advertising. This is why personal or corporate blogs usually provide full content and some sites like ArsTechnica.com offer it to paid subscribers.
onli 5y ago
In addition to the answers you got refuting load as a problem: "Last updated since" headers are indeed a thing. Look for example at how RSS gets cached in wordpress [0] or the classical blog engine serendipity [1]. Also, push instead of pull is also common. Made popular for feeds by pubsubhubbub, since enterprise-ready renamed to WebSub [2]. Readers like bazqux do support that. [0]: https://github.com/WordPress/WordPress/blob/270f2011f8ec7265c3f4ddce39c77ef5b496ed1c/wp-includes/rss.php#L471 https://github.com/WordPress/WordPress/blob/270f2011f8ec7265... [1]: https://github.com/s9y/Serendipity/blob/e2044472c202a8368774282264b9d55286cb636c/rss.php#L69 https://github.com/s9y/Serendipity/blob/e2044472c202a8368774... [2]: https://en.wikipedia.org/wiki/WebSub https://en.wikipedia.org/wiki/WebSub
pantulis 5y ago
RSS was almost killed because it's more difficult to monetize the contents for content publishers/creators, not due to performance problems. Obviously Google has something in mind.
xena 5y ago
Owner of christine.website here. I did stop doing it in the atom feed (though i don't remember why) however the RSS and JSONFeeds do have article text in them. If you want to do more advanced queries against the data I could whip something up I guess. It's a really dumb webapp written in Rust and I'm more than happy to put some extra fun into it.
- tomcam 5y ago
  Well if you’re taking requests I would love that, and would also like a writeup on the implementation in your blog.

CarelessExpert 5y ago

> But I've heard that it requires creating a massive feed file containing many articles, including text and images So a) no, it doesn't contain images, just URLs to images where necessary; an RSS or ATOM feed is just a text XML document adhering to a certain schema, and b) the feed file is only as big as the site decides it needs to be. For example, a typical blog or news organization could generate a feed containing the last 24 hours worth of new content. Feed readers will poll that and pull in new content while ignoring existing content. In short: this should be a non-issue unless you're turning out massive amounts of new material on a regular basis, in which case you're probably a major news organization and can afford it. > As a result, some blogs (like christine.website) stopped serving article contents in the RSS feeds. This isn't because of bandwidth concerns. This is because they can't advertise in an RSS feed so they want to drive eyeballs to the site. Personally, I run ttrss and use a plugin that scrapes the content from the source site and embeds it right in my feed. As a result I only have to leave my feed reader if I really want to for some reason (e.g. to read comments on the originating site... like, say, HN!)

kixiQu 5y ago
To be clear, I don't think it's the case that christine.website has any advertising whatsoever (or even meaningful JS?) but I think your diagnosis is correct more broadly.
- moehm 5y ago
  It also doesn't seem to be the case that christine.website strips their content from the feed. https://christine.website/blog.rss https://christine.website/blog.rss

mattowen_uk 5y ago

A LOT of sites only serve the article headline and maybe just the intro sentence for exactly this reason. Also by forcing you to click through, you land on the site proper and feature in their visitor counts, AND get to look at ads on the page too.

pantulis 5y ago
Unless you use a "reader mode" capable RSS reader, that is.
- k1m 5y ago
  Or use something like this to convert partial feeds into full text before subscribing: http://ftr.fivefilters.org http://ftr.fivefilters.org (a project I work on).

jcrawfordor 5y ago

I don't think performance/load has ever been a real reason to only serve partial content. The actual reason has always been getting users onto the site where you can deliver advertising more flexibly along with analytics. For my extremely minimal site, the RSS feed is still smaller than normal pageloads because it only has graphics by reference. RSS readers scrape surprisingly often, but the file is small and highly cacheable, and most of them do a HEAD request only initially and never ask for the file if it hasn't changed. Despite accounting for a large percentage of requests, RSS clients account for only a small portion of traffic. The cool thing is that a lot of these RSS readers are shared (commercial feed readers, ttrss instances, etc) and so they are caching the feed on their end and actually saving me bandwidth vs an equivalent number of users accessing the site directly - a lot of these put the subscriber account in their UA string so I can see that e.g. Feedly is making regular requests during the day but has 30-some users behind those requests. It's a better deal from a cost-perspective than those 30-some people checking each day.

acdha 5y ago

The load issues were a concern with very early implementations around the turn of the century but they're really not now. Feeds are usually extremely cacheable and readers have supported conditional fetches using If-Modified-Since since the early days. These days the average site is serving considerably more data in JavaScript, not to mention the images and video which you're going to see in any case (they're just links in RSS). The real reason why sites started switching to partial content is advertising. This is why personal or corporate blogs usually provide full content and some sites like ArsTechnica.com offer it to paid subscribers.

onli 5y ago

In addition to the answers you got refuting load as a problem: "Last updated since" headers are indeed a thing. Look for example at how RSS gets cached in wordpress [0] or the classical blog engine serendipity [1]. Also, push instead of pull is also common. Made popular for feeds by pubsubhubbub, since enterprise-ready renamed to WebSub [2]. Readers like bazqux do support that. [0]: https://github.com/WordPress/WordPress/blob/270f2011f8ec7265c3f4ddce39c77ef5b496ed1c/wp-includes/rss.php#L471 https://github.com/WordPress/WordPress/blob/270f2011f8ec7265... [1]: https://github.com/s9y/Serendipity/blob/e2044472c202a8368774282264b9d55286cb636c/rss.php#L69 https://github.com/s9y/Serendipity/blob/e2044472c202a8368774... [2]: https://en.wikipedia.org/wiki/WebSub https://en.wikipedia.org/wiki/WebSub

pantulis 5y ago

RSS was almost killed because it's more difficult to monetize the contents for content publishers/creators, not due to performance problems. Obviously Google has something in mind.

xena 5y ago

Owner of christine.website here. I did stop doing it in the atom feed (though i don't remember why) however the RSS and JSONFeeds do have article text in them. If you want to do more advanced queries against the data I could whip something up I guess. It's a really dumb webapp written in Rust and I'm more than happy to put some extra fun into it.

tomcam 5y ago
Well if you’re taking requests I would love that, and would also like a writeup on the implementation in your blog.