6 ms·

For reasons I don't understand, it remains incredibly difficult to scrape a well-organized blog and turn it into something I can consume like RSS or a kindle bo

For reasons I don't understand, it remains incredibly difficult to scrape a well-organized blog and turn it into something I can consume like RSS or a kindle book. Note that many/most WordPress RSS feeds aren't that useful because they only show a snippet rather than the entire post. This dreary state of affairs is due in part to the fact that "checking" an RSS just means downloading the entire file containing all posts the author would like to make public, regardless of how many are new to the reader. This insanely unnecessary bandwidth usage penalizes sites that have long (>10) feeds with complete posts. (The single RSS file on my blog takes up the majority of my bandwidth costs.) RSS needs to be replaced, not revived.

Tehdasi 8y ago
Sortof, if you are reading via a web based reader, the service itself only reads the feed once, and the users view the cached version.
- jessriedel 8y ago
  You mean the fact that aggregators like The Old Reader just download the RSS file from the website periodically, and then serve up cached copies to their many user? True, although this is another way of saying that some of the value added by aggregators only exists by virtue of the terrible design of RSS.
  - JetSpiegel 8y ago
    It might be construed as a feature, since a blog can consist of simply static files, no need for running any code relating to RSS.
    - jessriedel 8y ago
      It doesn't need code. Just a static text file with a list of publication dates and URLs, each of which points to static HTML file for each publicly available posts. Then each user could check the list and download only what's new, without needing an aggregator. Users would never need to download the same content twice.
      - jerf 8y ago
        You can pretty much do that with RSS, if you want. Slightly decorated with some XML stuff but that'll pretty much gzip right away. Statically, if you use the right web servers and magic invocations the web server may specify. You'll find your feed readers aren't particularly impressed, though.
        jessriedel 8y ago
        This won't work because my users don't have software to automatically download the new posts, curate them, and save them for offline reading (in contrast to having the posts directly in the RSS file, for which all that does happen). That would require...a new standard, i.e., a replacement for RSS.
        jerf 8y ago
        No, it would require a new type of consumer. RSS can carry a list of URLs just fine, as a degenerate case of RSS. RSS does not itself mandate that the content of the RSS must be displayed in "some sort of feed reader". The good news is that means you don't have to wait. You can write that now. It will work on existing RSS feeds, just not quite as optimally for your proposed use case as you might personally like, but it will still work. It will work even better on yours, which will also work in conventional RSS readers. Now, you might have problems getting "the real page content" from your URLs, but that's a separate problem. (History strongly suggests the large-scale content producers would actively fight you if you try, because you'll probably be trying to strip their ad revenue either deliberately or accidentally as part of what you'd be doing. Which is, after all, the reason why RSS is already not terribly favored by that crowd and why they want you in closed gardens of their own devising... unfortunately getting around this problem is a great deal more difficult than hypothesizing that some sort of new standard could somehow deal with it....)
        jessriedel 8y ago
        I'm using "RSS" to refer to the standard practice people actually do. It's not very important that some organization somewhere defined an official RSS standard which in principle is flexible but which in practice is never used other than in a very specific way. (Without an agreement on how to use it more generally, no one can build an offline blog archive reader, and the fact that RSS could in principle be the base is irrelevant.) If you want to play semantics, I'm fine with rephrasing my complaint as: "We need to build on the flexible super official RSS standard -- which is little more than an XML file -- and actually agree on a way of delivering blog archives for offline reading. RSS in practice does not currently achieve this very simple goal." This is just different words to describe the same thing. > You can write that now. It will work on existing RSS feeds, just not quite as optimally for your proposed use case as you might personally like, but it will still work. Huh? Other website owners who would like me to be able to read their archives can modify their RSS file, but we have no agreement on how to do that in a standard way. Likewise, I could modify my RSS file, but since there isn't a standard my readers won't have software to take advantage of it. > (History strongly suggests the large-scale content producers would actively fight you if you try,...unfortunately getting around this problem is a great deal more difficult than hypothesizing that some sort of new standard could somehow deal with it....) The blogs I want to read offline do not have ads and do not care about this. I just want a solution that works for this simple problem, not a way to take content from people who don't want to give it to me without attaching ads.
23443463455 8y ago
Is the problem really that RSS needs replacement, or can it simply be improved. I think there is quite a bit of room for extension to address some of the concerns that are expressed here too, including accurate meta tagging for educational resource discovery purposes.
- klez 8y ago
  I'm under the impression that pulling extensions on RSS will land us in the XMPP situation, where you have servers and clients speaking different languages. On the other hand, if you want to say that your client supports this supposed RSS-ng standard they need to support those new features because they're part of the protocol. This is to say, I don't like protocol extensions, but that's just me :)
bArray 8y ago
>This dreary state of affairs is due in part to the fact that "checking" an RSS just means downloading the entire file containing all posts the author would like to make public, regardless of how many are new to the reader. This insanely unnecessary bandwidth usage penalizes sites that have long (>10) feeds with complete posts. (The single RSS file on my blog takes up the majority of my bandwidth costs.) Still, exceptionally low cost compared to running pretty much any website's massive CSS and JS files. A single image in most cases will take more than the entire RSS feed before compression. That said, I would like to see a new standard (a new one would be needed [1]) that only gets the difference from what you last read - I think that would really take RSS feeds to new places of usefulness. There's no reason why you couldn't send the server an ID (not timestamp to avoid issues with timezones, clock stretching, forward/backward time setting, etc) of the last request and have it send back everything since (within reason). [1] https://www.w3schools.com/XML/xml_rss.asp https://www.w3schools.com/XML/xml_rss.asp
- abecedarius 8y ago
  That’d impose a cost on the server, remembering an ID map. Instead make the client send back a value given by the server last time.
  - Izkata 8y ago
    I think that's what they meant by "an ID [...] of the last request"
  - jerf 8y ago
    That's something both RSS consumers and produces should already support, but don't always, called an E-Tag, a standard part of the HTTP standard. However, the E-Tag is all-or-nothing; either it matches, and the entire request is essentially aborted, or it doesn't match, and the entire file is served up. Passing "the ID of the last piece of content I saw" would allow the server to return just the updated stuff, or abort early like an E-Tag. However, counterintuitively, as is often the way in computer science, I'm not sure it would be that big a win to be able to return partial content. The vast bulk of the win on most blogs will just be the ability to abort at all, provided just fine by E-Tags. I would say that if your site is getting hammered by HTTP requests for your RSS, do double-check that you've got E-Tags set up and working correctly. It is in the best interests of the big scrapers to support that properly, as they are paying for that bandwidth too. RSS aggregators don't have to get too large before this becomes a top-priority feature request. Unless the feed is literally changing on roughly the same frequency as it is scanned, it shouldn't be the dominant factor in your bandwidth bill.
gkya 8y ago
For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. WRT scraping, it is getting harder because funny websites w/ no static content where everything is generated via Angular or Perpendicular or sth. are really hard to deal with. Recently my uni switched from an army of Wordpress websites to an homegrown AJAX-MVC-Reactive abomination where the links are reimplemented via some funny black magic where the actual link items (which are not anchors btw) don't have encoded in them the link targets, but only an onclick event that knows somehow where to go. And because they just killed all the RSS feeds, I wrote up sth. to revive them for me via phantomjs, but could not figure how to find the link targets, I cannot link the RSS items to anywhere but the main announcements page, and I can't add any description from the link target. RSS should be kept, those who don't know the job they are doing should be replaced.
- jessriedel 8y ago
  > For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. But people who don't use the right aggregators will not be able to read past 10 posts. And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. > WRT scraping, it is getting harder because funny websites w/ no static content. But sites that can have an RSS feed necessarily need to deliver their content in a static form. What surprises me is that we don't have a tool for even those cases.
  - gkya 8y ago
    > But people who don't use the right aggregators will not be able to read past 10 posts. I do not think such aggregators exist, and you can just ignore people using software that do not comply to widespread conventions. > And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. I don't understand this sentence. All RSS client software is called aggregator. > But sites that can have an RSS feed necessarily need to deliver their content in a static form. Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps. My uni's websites included. Some do it for the $$$, some for reasons that I can not know (jumping the bandwagons with minds toggled off). You can just set your blog software to truncate your feeds to a reasonable number w/o any worries. And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds.
prepend 8y ago
“This insanely unnecessary bandwidth usage penalizes sites that have long (>10) feeds with complete posts.” Since rss is text only the sizes are very small and compress well. Considering the average web page is 3M[0], pulling down 10-100k of every post ever doesn’t matter. And http takes care of not pulling the same file over and over. It’s certainly a downside, but completely useable as is, and better than any viable alternative. As far as standards go, I prefer simple, static file, over something requiring dynamic response. There’s also nothing stopping the site from limiting the RSS feed to only 5 posts with a link for full. [0] https://speedcurve.com/blog/web-performance-page-bloat/ https://speedcurve.com/blog/web-performance-page-bloat/
- viraptor 8y ago
  Even better: instead of serving the generated for, point your RSS subdomain to your favourite CDN and only update your files when the site changes. That's likely to save you some money.
- jessriedel 8y ago
  > Since rss is text only the sizes are very small and compress well. Considering the average web page is 3M[0], pulling down 10-100k of every post ever doesn’t matter. My web pages are 100k and my RSS feed is 500k. > It’s certainly a downside, but completely useable as is, and better than any viable alternative. It doesn't fulfill the need I originally mentioned: making blog archives readable offline. > As far as standards go, I prefer simple, static file, over something requiring dynamic response. I agree static is better, but a dynamic response isn't necessary. You could just have a static file that listed all the blog posts, with a link to another static file for each post. This avoids having the user download 10 blog posts each time they want to poll if something new has happened.
iamdeedubs 8y ago
They only show a snippet because they can't sell your eyeballs via the RSS feed. They need just need to drsw you to click onto their full site to blast you with ads.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]

Tehdasi 8y ago

Sortof, if you are reading via a web based reader, the service itself only reads the feed once, and the users view the cached version.

jessriedel 8y ago
You mean the fact that aggregators like The Old Reader just download the RSS file from the website periodically, and then serve up cached copies to their many user? True, although this is another way of saying that some of the value added by aggregators only exists by virtue of the terrible design of RSS.
- JetSpiegel 8y ago
  It might be construed as a feature, since a blog can consist of simply static files, no need for running any code relating to RSS.
  - jessriedel 8y ago
    It doesn't need code. Just a static text file with a list of publication dates and URLs, each of which points to static HTML file for each publicly available posts. Then each user could check the list and download only what's new, without needing an aggregator. Users would never need to download the same content twice.
    - jerf 8y ago
      You can pretty much do that with RSS, if you want. Slightly decorated with some XML stuff but that'll pretty much gzip right away. Statically, if you use the right web servers and magic invocations the web server may specify. You'll find your feed readers aren't particularly impressed, though.
      - jessriedel 8y ago
        This won't work because my users don't have software to automatically download the new posts, curate them, and save them for offline reading (in contrast to having the posts directly in the RSS file, for which all that does happen). That would require...a new standard, i.e., a replacement for RSS.
        jerf 8y ago
        No, it would require a new type of consumer. RSS can carry a list of URLs just fine, as a degenerate case of RSS. RSS does not itself mandate that the content of the RSS must be displayed in "some sort of feed reader". The good news is that means you don't have to wait. You can write that now. It will work on existing RSS feeds, just not quite as optimally for your proposed use case as you might personally like, but it will still work. It will work even better on yours, which will also work in conventional RSS readers. Now, you might have problems getting "the real page content" from your URLs, but that's a separate problem. (History strongly suggests the large-scale content producers would actively fight you if you try, because you'll probably be trying to strip their ad revenue either deliberately or accidentally as part of what you'd be doing. Which is, after all, the reason why RSS is already not terribly favored by that crowd and why they want you in closed gardens of their own devising... unfortunately getting around this problem is a great deal more difficult than hypothesizing that some sort of new standard could somehow deal with it....)
        jessriedel 8y ago
        I'm using "RSS" to refer to the standard practice people actually do. It's not very important that some organization somewhere defined an official RSS standard which in principle is flexible but which in practice is never used other than in a very specific way. (Without an agreement on how to use it more generally, no one can build an offline blog archive reader, and the fact that RSS could in principle be the base is irrelevant.) If you want to play semantics, I'm fine with rephrasing my complaint as: "We need to build on the flexible super official RSS standard -- which is little more than an XML file -- and actually agree on a way of delivering blog archives for offline reading. RSS in practice does not currently achieve this very simple goal." This is just different words to describe the same thing. > You can write that now. It will work on existing RSS feeds, just not quite as optimally for your proposed use case as you might personally like, but it will still work. Huh? Other website owners who would like me to be able to read their archives can modify their RSS file, but we have no agreement on how to do that in a standard way. Likewise, I could modify my RSS file, but since there isn't a standard my readers won't have software to take advantage of it. > (History strongly suggests the large-scale content producers would actively fight you if you try,...unfortunately getting around this problem is a great deal more difficult than hypothesizing that some sort of new standard could somehow deal with it....) The blogs I want to read offline do not have ads and do not care about this. I just want a solution that works for this simple problem, not a way to take content from people who don't want to give it to me without attaching ads.

23443463455 8y ago

Is the problem really that RSS needs replacement, or can it simply be improved. I think there is quite a bit of room for extension to address some of the concerns that are expressed here too, including accurate meta tagging for educational resource discovery purposes.

klez 8y ago
I'm under the impression that pulling extensions on RSS will land us in the XMPP situation, where you have servers and clients speaking different languages. On the other hand, if you want to say that your client supports this supposed RSS-ng standard they need to support those new features because they're part of the protocol. This is to say, I don't like protocol extensions, but that's just me :)

bArray 8y ago

>This dreary state of affairs is due in part to the fact that "checking" an RSS just means downloading the entire file containing all posts the author would like to make public, regardless of how many are new to the reader. This insanely unnecessary bandwidth usage penalizes sites that have long (>10) feeds with complete posts. (The single RSS file on my blog takes up the majority of my bandwidth costs.) Still, exceptionally low cost compared to running pretty much any website's massive CSS and JS files. A single image in most cases will take more than the entire RSS feed before compression. That said, I would like to see a new standard (a new one would be needed [1]) that only gets the difference from what you last read - I think that would really take RSS feeds to new places of usefulness. There's no reason why you couldn't send the server an ID (not timestamp to avoid issues with timezones, clock stretching, forward/backward time setting, etc) of the last request and have it send back everything since (within reason). [1] https://www.w3schools.com/XML/xml_rss.asp https://www.w3schools.com/XML/xml_rss.asp

abecedarius 8y ago
That’d impose a cost on the server, remembering an ID map. Instead make the client send back a value given by the server last time.
- Izkata 8y ago
  I think that's what they meant by "an ID [...] of the last request"
- jerf 8y ago
  That's something both RSS consumers and produces should already support, but don't always, called an E-Tag, a standard part of the HTTP standard. However, the E-Tag is all-or-nothing; either it matches, and the entire request is essentially aborted, or it doesn't match, and the entire file is served up. Passing "the ID of the last piece of content I saw" would allow the server to return just the updated stuff, or abort early like an E-Tag. However, counterintuitively, as is often the way in computer science, I'm not sure it would be that big a win to be able to return partial content. The vast bulk of the win on most blogs will just be the ability to abort at all, provided just fine by E-Tags. I would say that if your site is getting hammered by HTTP requests for your RSS, do double-check that you've got E-Tags set up and working correctly. It is in the best interests of the big scrapers to support that properly, as they are paying for that bandwidth too. RSS aggregators don't have to get too large before this becomes a top-priority feature request. Unless the feed is literally changing on roughly the same frequency as it is scanned, it shouldn't be the dominant factor in your bandwidth bill.

gkya 8y ago

For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. WRT scraping, it is getting harder because funny websites w/ no static content where everything is generated via Angular or Perpendicular or sth. are really hard to deal with. Recently my uni switched from an army of Wordpress websites to an homegrown AJAX-MVC-Reactive abomination where the links are reimplemented via some funny black magic where the actual link items (which are not anchors btw) don't have encoded in them the link targets, but only an onclick event that knows somehow where to go. And because they just killed all the RSS feeds, I wrote up sth. to revive them for me via phantomjs, but could not figure how to find the link targets, I cannot link the RSS items to anywhere but the main announcements page, and I can't add any description from the link target. RSS should be kept, those who don't know the job they are doing should be replaced.

jessriedel 8y ago
> For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. But people who don't use the right aggregators will not be able to read past 10 posts. And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. > WRT scraping, it is getting harder because funny websites w/ no static content. But sites that can have an RSS feed necessarily need to deliver their content in a static form. What surprises me is that we don't have a tool for even those cases.
- gkya 8y ago
  > But people who don't use the right aggregators will not be able to read past 10 posts. I do not think such aggregators exist, and you can just ignore people using software that do not comply to widespread conventions. > And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. I don't understand this sentence. All RSS client software is called aggregator. > But sites that can have an RSS feed necessarily need to deliver their content in a static form. Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps. My uni's websites included. Some do it for the $$$, some for reasons that I can not know (jumping the bandwagons with minds toggled off). You can just set your blog software to truncate your feeds to a reasonable number w/o any worries. And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds.

prepend 8y ago

“This insanely unnecessary bandwidth usage penalizes sites that have long (>10) feeds with complete posts.” Since rss is text only the sizes are very small and compress well. Considering the average web page is 3M[0], pulling down 10-100k of every post ever doesn’t matter. And http takes care of not pulling the same file over and over. It’s certainly a downside, but completely useable as is, and better than any viable alternative. As far as standards go, I prefer simple, static file, over something requiring dynamic response. There’s also nothing stopping the site from limiting the RSS feed to only 5 posts with a link for full. [0] https://speedcurve.com/blog/web-performance-page-bloat/ https://speedcurve.com/blog/web-performance-page-bloat/

viraptor 8y ago
Even better: instead of serving the generated for, point your RSS subdomain to your favourite CDN and only update your files when the site changes. That's likely to save you some money.
jessriedel 8y ago
> Since rss is text only the sizes are very small and compress well. Considering the average web page is 3M[0], pulling down 10-100k of every post ever doesn’t matter. My web pages are 100k and my RSS feed is 500k. > It’s certainly a downside, but completely useable as is, and better than any viable alternative. It doesn't fulfill the need I originally mentioned: making blog archives readable offline. > As far as standards go, I prefer simple, static file, over something requiring dynamic response. I agree static is better, but a dynamic response isn't necessary. You could just have a static file that listed all the blog posts, with a link to another static file for each post. This avoids having the user download 10 blog posts each time they want to poll if something new has happened.

iamdeedubs 8y ago

They only show a snippet because they can't sell your eyeballs via the RSS feed. They need just need to drsw you to click onto their full site to blast you with ads.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]