6 ms·

As much as I like RSS, one of the repeated pain points is that I'd like to actually read the content in my Feed Reader, but on many of the sites I pull from, th

As much as I like RSS, one of the repeated pain points is that I'd like to actually read the content in my Feed Reader, but on many of the sites I pull from, the feeds only contain the article title and maybe half of the introductory sentence before dropping a link to the full article, which I then have to click and open in my browser. To my mind, it kinda defeats the purpose of having the feed, since I could just as easily scroll the front page of the site. Was it the intention of RSS from the outset that your feed would only provide a "preview" of the article, or was the hope that you would get the full body of the text?

bpye 8y ago
There definitely exist scrapers that work to get the full body. The reason for this is reading in a feed reader will never get your ad revenue or analytics, forcing your reader to the full page will.
- AdmiralAsshat 8y ago
  Oh I definitely understand why sites do it this way. I simply wanted clarification on whether it was always the intention to do it that way, or if the content sites took the standard and crippled it in order to bring more ad revenue.
  - blihp 8y ago
    It could probably be argued that the standard is ambiguous as what content goes where is up to the creator of it. However, I think as a user it's fairly obvious that using it for clickbait and teaser headlines/articles rather than at least a complete thought is abusive. It's probably a non-trivial part of the reason its popularity with end users has waned in the much the same way spam polluted peoples email inboxes.
    - TeMPOraL 8y ago
      Yup. Standards from that age were written with friendly, cooperative actors in mind. The Internet is no longer a friendly place. Unfortunately, you can't just force people to not be greedy. (Clickbait itself is an example of that. You absolutely can get to the point in the title. But most media sources don't, because it's not their goal to inform you - their goal is to just exploit you for all they can.
      - matt4077 8y ago
        It's also unfriendly, and not very cooperative, to dismiss the stated wishes and motivations of content creators to save clicking on a link. Similarly unfriendly is assuming bad faith, i. e. "most media sources don't [seek to] inform you - their goal is to just exploit you". Yeah, yeah, I know. You'd gladly spoil the writers you read by deigning to visit, if only they didn't use javascript/have big images that load slowly/used your favorite font/...
        Dylan16807 8y ago
        Nobody's wishes about link clicks should matter one bit except the user's.
- scarface74 8y ago
  99.999% of the time you’re right. But Daring Fireball offers the full article as part of the RSS feed and it has one sponsored RSS article at the beginning of the week and a thank you post in the RSS feed at the end of the week. He charges around $6500 per week the last time I checked. It’s amazing what you can do if you stay small and have “1000 true fans”.
- burtonator 8y ago
  Yes. The analytics is another key issue. AMP for example ads analytics.
  - r3bl 8y ago
    In the server logs I see that RSS services tend to send the number of subscribers when they check for new content. It's kind of a useful data point, and I could imagine the controversy if RSS services made an attempt to share more.
- aaaaaaaaaab 8y ago
  I remember when I could sync my feed reader at home, then read the articles while commuting on public transport with no internet connection. Then sites started offering only the first paragraph in their feeds, so RSS became useless for this use case.
  - akho 8y ago
    You could set up a “reader” to download full pages, using RSS as basically a notification service.
- hiccuphippo 8y ago
  Yahoo!Pipes was awesome at this. You could give it a feed, pull the page for each item and generate a new feed with the full content. There was also ways to specify what parts of the page to get by using xpath or regex. I miss it dearly.
icebraining 8y ago
I believe the full text was the goal, as IIRC it includes a separate tag for a summary.
- nicolaslem 8y ago
  RSS only has a single "description" tag while Atom has both "content" and "summary".
tedunangst 8y ago
> since I could just as easily scroll the front page of the site. For how many sites? I much prefer having a consolidated list of new content than clicking on a hundred bookmarks, waiting for the page to load, then scanning to see if anything is different.
delecti 8y ago
Amusingly your complaint is actually my preferred use-case. I want my RSS reader to essentially work like an automated link gatherer, because the reader's UI is inherently not going to be the intended presentation medium for most content. I exclusively open the collected links in separate tabs to read and I get annoyed when I accidentally expand them.
- Dylan16807 8y ago
  And including the article body preserves both options as a choice. > because the reader's UI is inherently not going to be the intended presentation medium for most content Huh, you must use RSS for different things than I do. The blogs I follow via RSS work fine, and I only go to the site for comments on some of them. The podcasts and tumblr posts work much better in their respective readers than visiting any website.
  - anyfoo 8y ago
    Apparently we do use it for different things. One example is Raymond Chen’s excellent blog The Old New Thing, which has formatting that often breaks completely on RSS, to the point of being unreadable (he does not have any influence on the blog software as far as I know, though). Generally, I often noticed that articles look (and sometimes convey their content) much better on their actual websites, since that’s what they were formatted for first most. Much more infuriating is that the various “Reader Modes” in browsers and e.g. Pocket sometimes not only tends to omit images, but even whole paragraphs of the text. It’s hard to notice that you missed a paragraph that you don’t know existed, so I don’t trust them anymore. I believe that should be less of an issue in RSS, since it of course includes the text to be displayed in the feed. Still, since I cannot be sure that authors take full attention to their RSS feeds, I prefer reading them on the website just in case some other formatting quirk messes things up. Stuff like info boxes or illustrations might not even be part of the main body. In that way, I actually prefer it when the RSS feed just contains a meaningful preview. Though I guess the correct solution would be to include both a preview and the full article, designated as such in the structure.
  - delecti 8y ago
burtonator 8y ago
The original intention was the full content. The problem is that sites don't want to put the full content in because they lose ads...
- netghost 8y ago
  It's also just duplicated work for the producer. I'd rather someone just visit my little blog and read my article on the context where I know the syntax highlighted coffee looks good, my dynamic examples render, etc. There probably are done folks with great intents, but there's legitimate reasons to channel people to your site as well.
kbenson 8y ago
Some of the web based RSS readers have modes where it will automatically load the article in an iframe for you, so you don't have to actually open a new window. I know NewsBlur[1] has a mode for this, because I just went and fixed a feed that has this problem since you reminded me I could. 1: I highly recommend NewsBlur. I went through a few readers over a couple years after Google Reader shut down, and settled on NewBlur, and actually pay for it.
CM30 8y ago
Wonder how much of this is because some sites use it as a quick way to get content from another system rather than only for the user? For instance, I know quite a few forums have RSS import functions, and quite a few CMS systems are set up to auto import content from an RSS feed (like say, an associated forum). In those cases, the system sometimes doesn't let the user customise the output of the RSS importer, but does let them set what content is in the RSS file. So admins will provide a preview so their forum doesn't end up with the entire article or vice versa.
chmars 8y ago
http://fivefilters.org/content-only http://fivefilters.org/content-only is your friend. I use the self-hosted version.
the_clarence 8y ago
Inoreader can pull the whole content
crunchiebones 8y ago
it would feel more natural to just have the feeds in the sidebar instead of a dedicated feed reader
chriswarbo 8y ago
> As much as I like RSS, one of the repeated pain points is that I'd like to actually read the content in my Feed Reader, but on many of the sites I pull from, the feeds only contain the article title and maybe half of the introductory sentence before dropping a link to the full article, which I then have to click and open in my browser. This is definitely annoying, but the nice thing about RSS (and ATOM) is that it's machine readable, so post-processors can follow the link (e.g. using wget) and insert the page content into the feed. Here's a script I use to postprocess BBC news RSS feeds: https://github.com/Warbo/warbo-utilities/blob/master/raw/getBBCContent.py https://github.com/Warbo/warbo-utilities/blob/master/raw/get... (Looks complicated, but mostly deals with stipping side bars, navigation menus, etc. from the page, rendering the HTML to plain text, and also includes test cases; the RSS code is mostly copy/pasted from tutorials)
- revicon 8y ago
  Check out newspaper too. https://newspaper.readthedocs.io/en/latest/ https://newspaper.readthedocs.io/en/latest/
CUViper 8y ago
While I prefer full articles, preview feeds are still useful, because I can follow many sites this way. I don't want to manually poll each of those sites on my own to see if they've posted anything new.
tootie 8y ago
I know it's been shit on for being a power grab by Google, but an RSS reader today could pull in AMP versions of pages very easily.
- aaaaaaaaaab 8y ago
  Hmm... not a bad idea actually! Subverting their own SEO crap and turning it into something useful.
beagle3 8y ago
NewsBlur, as a feed reader does both (and more) - for each feed you can decide whether it shows the rss snippet, extracted story, original site. It’s awesome, open source, self hostable, freemium if you don’t self host, has mobile and desktop apps if you care. Not affiliated, just a very satisfied paying customer.
jacobbudin 8y ago
Many RSS reader services (e.g., Feedbin) and apps (e.g., Reeder for macOS) integrate Diffbot or Postlight's Mercury or similar to scrape the article's full text. I use and recommend both.
netghost 8y ago
Try a browser plugin for reading feeds. The plugin aggregates your new links and you read it like anything else on the internet. It's a great way to play to RSS' strength while avoiding the myriad of ways folks bungle feeds. I wrote an open source one for Firefox here: http://github.com/adamsanderson/brook http://github.com/adamsanderson/brook It's certainly not the only option though, so try a few things.

bpye 8y ago

There definitely exist scrapers that work to get the full body. The reason for this is reading in a feed reader will never get your ad revenue or analytics, forcing your reader to the full page will.

AdmiralAsshat 8y ago
Oh I definitely understand why sites do it this way. I simply wanted clarification on whether it was always the intention to do it that way, or if the content sites took the standard and crippled it in order to bring more ad revenue.
- blihp 8y ago
  It could probably be argued that the standard is ambiguous as what content goes where is up to the creator of it. However, I think as a user it's fairly obvious that using it for clickbait and teaser headlines/articles rather than at least a complete thought is abusive. It's probably a non-trivial part of the reason its popularity with end users has waned in the much the same way spam polluted peoples email inboxes.
  - TeMPOraL 8y ago
    Yup. Standards from that age were written with friendly, cooperative actors in mind. The Internet is no longer a friendly place. Unfortunately, you can't just force people to not be greedy. (Clickbait itself is an example of that. You absolutely can get to the point in the title. But most media sources don't, because it's not their goal to inform you - their goal is to just exploit you for all they can.
    - matt4077 8y ago
      It's also unfriendly, and not very cooperative, to dismiss the stated wishes and motivations of content creators to save clicking on a link. Similarly unfriendly is assuming bad faith, i. e. "most media sources don't [seek to] inform you - their goal is to just exploit you". Yeah, yeah, I know. You'd gladly spoil the writers you read by deigning to visit, if only they didn't use javascript/have big images that load slowly/used your favorite font/...
      - Dylan16807 8y ago
        Nobody's wishes about link clicks should matter one bit except the user's.
scarface74 8y ago
99.999% of the time you’re right. But Daring Fireball offers the full article as part of the RSS feed and it has one sponsored RSS article at the beginning of the week and a thank you post in the RSS feed at the end of the week. He charges around $6500 per week the last time I checked. It’s amazing what you can do if you stay small and have “1000 true fans”.
burtonator 8y ago
Yes. The analytics is another key issue. AMP for example ads analytics.
- r3bl 8y ago
  In the server logs I see that RSS services tend to send the number of subscribers when they check for new content. It's kind of a useful data point, and I could imagine the controversy if RSS services made an attempt to share more.
aaaaaaaaaab 8y ago
I remember when I could sync my feed reader at home, then read the articles while commuting on public transport with no internet connection. Then sites started offering only the first paragraph in their feeds, so RSS became useless for this use case.
- akho 8y ago
  You could set up a “reader” to download full pages, using RSS as basically a notification service.
hiccuphippo 8y ago
Yahoo!Pipes was awesome at this. You could give it a feed, pull the page for each item and generate a new feed with the full content. There was also ways to specify what parts of the page to get by using xpath or regex. I miss it dearly.

icebraining 8y ago

I believe the full text was the goal, as IIRC it includes a separate tag for a summary.

nicolaslem 8y ago
RSS only has a single "description" tag while Atom has both "content" and "summary".

tedunangst 8y ago

> since I could just as easily scroll the front page of the site. For how many sites? I much prefer having a consolidated list of new content than clicking on a hundred bookmarks, waiting for the page to load, then scanning to see if anything is different.

delecti 8y ago

Amusingly your complaint is actually my preferred use-case. I want my RSS reader to essentially work like an automated link gatherer, because the reader's UI is inherently not going to be the intended presentation medium for most content. I exclusively open the collected links in separate tabs to read and I get annoyed when I accidentally expand them.

Dylan16807 8y ago
And including the article body preserves both options as a choice. > because the reader's UI is inherently not going to be the intended presentation medium for most content Huh, you must use RSS for different things than I do. The blogs I follow via RSS work fine, and I only go to the site for comments on some of them. The podcasts and tumblr posts work much better in their respective readers than visiting any website.
- anyfoo 8y ago
  Apparently we do use it for different things. One example is Raymond Chen’s excellent blog The Old New Thing, which has formatting that often breaks completely on RSS, to the point of being unreadable (he does not have any influence on the blog software as far as I know, though). Generally, I often noticed that articles look (and sometimes convey their content) much better on their actual websites, since that’s what they were formatted for first most. Much more infuriating is that the various “Reader Modes” in browsers and e.g. Pocket sometimes not only tends to omit images, but even whole paragraphs of the text. It’s hard to notice that you missed a paragraph that you don’t know existed, so I don’t trust them anymore. I believe that should be less of an issue in RSS, since it of course includes the text to be displayed in the feed. Still, since I cannot be sure that authors take full attention to their RSS feeds, I prefer reading them on the website just in case some other formatting quirk messes things up. Stuff like info boxes or illustrations might not even be part of the main body. In that way, I actually prefer it when the RSS feed just contains a meaningful preview. Though I guess the correct solution would be to include both a preview and the full article, designated as such in the structure.
- delecti 8y ago

burtonator 8y ago

The original intention was the full content. The problem is that sites don't want to put the full content in because they lose ads...

netghost 8y ago
It's also just duplicated work for the producer. I'd rather someone just visit my little blog and read my article on the context where I know the syntax highlighted coffee looks good, my dynamic examples render, etc. There probably are done folks with great intents, but there's legitimate reasons to channel people to your site as well.

kbenson 8y ago

Some of the web based RSS readers have modes where it will automatically load the article in an iframe for you, so you don't have to actually open a new window. I know NewsBlur[1] has a mode for this, because I just went and fixed a feed that has this problem since you reminded me I could. 1: I highly recommend NewsBlur. I went through a few readers over a couple years after Google Reader shut down, and settled on NewBlur, and actually pay for it.

CM30 8y ago

Wonder how much of this is because some sites use it as a quick way to get content from another system rather than only for the user? For instance, I know quite a few forums have RSS import functions, and quite a few CMS systems are set up to auto import content from an RSS feed (like say, an associated forum). In those cases, the system sometimes doesn't let the user customise the output of the RSS importer, but does let them set what content is in the RSS file. So admins will provide a preview so their forum doesn't end up with the entire article or vice versa.

chmars 8y ago

http://fivefilters.org/content-only http://fivefilters.org/content-only is your friend. I use the self-hosted version.

the_clarence 8y ago

Inoreader can pull the whole content

crunchiebones 8y ago

it would feel more natural to just have the feeds in the sidebar instead of a dedicated feed reader

chriswarbo 8y ago

> As much as I like RSS, one of the repeated pain points is that I'd like to actually read the content in my Feed Reader, but on many of the sites I pull from, the feeds only contain the article title and maybe half of the introductory sentence before dropping a link to the full article, which I then have to click and open in my browser. This is definitely annoying, but the nice thing about RSS (and ATOM) is that it's machine readable, so post-processors can follow the link (e.g. using wget) and insert the page content into the feed. Here's a script I use to postprocess BBC news RSS feeds: https://github.com/Warbo/warbo-utilities/blob/master/raw/getBBCContent.py https://github.com/Warbo/warbo-utilities/blob/master/raw/get... (Looks complicated, but mostly deals with stipping side bars, navigation menus, etc. from the page, rendering the HTML to plain text, and also includes test cases; the RSS code is mostly copy/pasted from tutorials)

revicon 8y ago
Check out newspaper too. https://newspaper.readthedocs.io/en/latest/ https://newspaper.readthedocs.io/en/latest/

CUViper 8y ago

While I prefer full articles, preview feeds are still useful, because I can follow many sites this way. I don't want to manually poll each of those sites on my own to see if they've posted anything new.

tootie 8y ago

I know it's been shit on for being a power grab by Google, but an RSS reader today could pull in AMP versions of pages very easily.

aaaaaaaaaab 8y ago
Hmm... not a bad idea actually! Subverting their own SEO crap and turning it into something useful.

beagle3 8y ago

NewsBlur, as a feed reader does both (and more) - for each feed you can decide whether it shows the rss snippet, extracted story, original site. It’s awesome, open source, self hostable, freemium if you don’t self host, has mobile and desktop apps if you care. Not affiliated, just a very satisfied paying customer.

jacobbudin 8y ago

Many RSS reader services (e.g., Feedbin) and apps (e.g., Reeder for macOS) integrate Diffbot or Postlight's Mercury or similar to scrape the article's full text. I use and recommend both.

netghost 8y ago

Try a browser plugin for reading feeds. The plugin aggregates your new links and you read it like anything else on the internet. It's a great way to play to RSS' strength while avoiding the myriad of ways folks bungle feeds. I wrote an open source one for Firefox here: http://github.com/adamsanderson/brook http://github.com/adamsanderson/brook It's certainly not the only option though, so try a few things.