4 ms·

For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's s

by gkya 8y ago

For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. WRT scraping, it is getting harder because funny websites w/ no static content where everything is generated via Angular or Perpendicular or sth. are really hard to deal with. Recently my uni switched from an army of Wordpress websites to an homegrown AJAX-MVC-Reactive abomination where the links are reimplemented via some funny black magic where the actual link items (which are not anchors btw) don't have encoded in them the link targets, but only an onclick event that knows somehow where to go. And because they just killed all the RSS feeds, I wrote up sth. to revive them for me via phantomjs, but could not figure how to find the link targets, I cannot link the RSS items to anywhere but the main announcements page, and I can't add any description from the link target. RSS should be kept, those who don't know the job they are doing should be replaced.

jessriedel 8y ago
> For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. But people who don't use the right aggregators will not be able to read past 10 posts. And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. > WRT scraping, it is getting harder because funny websites w/ no static content. But sites that can have an RSS feed necessarily need to deliver their content in a static form. What surprises me is that we don't have a tool for even those cases.
- gkya 8y ago
  > But people who don't use the right aggregators will not be able to read past 10 posts. I do not think such aggregators exist, and you can just ignore people using software that do not comply to widespread conventions. > And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. I don't understand this sentence. All RSS client software is called aggregator. > But sites that can have an RSS feed necessarily need to deliver their content in a static form. Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps. My uni's websites included. Some do it for the $$$, some for reasons that I can not know (jumping the bandwagons with minds toggled off). You can just set your blog software to truncate your feeds to a reasonable number w/o any worries. And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds.
  - jessriedel 8y ago
    > I don't understand this sentence. All RSS client software is called aggregator I'm trying to distinguish between (1) people using software the directly downloads the RSS feed from my website to their device and (2) people who use services that download the RSS feed to a server which can then serve a cached copy of the posts to many users. If everyone used (2), then I would only have my RSS file downloaded as many times as there are separate services, which is not very many. So apparently many people are doing (1), and if my feed is short then they are limited in the blog history they can read to how long they've personally been subscribed (or less, if they need to clear their device and can't re-download my old posts). > Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps That's why I specified "But sites that can have an RSS feed...". If you have an RSS feed with useful posts in the feed, then you must be delivering static pages. (If you're just delivering snippets with links to a dynamic page, then there is no way for any service to cache the page either.) So my question is: why isn't there software to scrape webpages of blogs that offer an RSS feed with complete posts? This would enable me to comveneiently read post history going back more than 10 posts. > And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds. All the bots put together make up 27% of my bandwidth. It's a lot, but it's not the root cause.
    - gkya 8y ago
      > So apparently many people are doing (1), and if my feed is short then they are limited in the blog history they can read to how long they've personally been subscribed (or less, if they need to clear their device and can't re-download my old posts). RSS is a means for people to follow new posts from you, in order to read new posts they are supposed to come to your blog and use your archives.
      - jessriedel 8y ago
        The whole point of this subthread is about reading archives offline, and how RSS can't manage this. From my first comment: > it remains incredibly difficult to scrape a well-organized blog and turn it into something I can consume like RSS or a kindle book. > Note that many/most WordPress RSS feeds aren't that useful because...
ChristianBundy 8y ago
> ... where N is an arbitrary number you choose (12 is good, because https://dozenal.org https://dozenal.org). FTFY ;)
- gkya 8y ago
  Take a nice little joke please, downvoters, will ya?
- progval 8y ago
  > because http://dozenal.org http://dozenal.org FTFY
mtremsal 8y ago
FWIW, I had a similar issue recently: if you cannot find the link targets, you can still simulate a click on the element using phantomjs.
- gkya 8y ago
  Yeah, but for my case it means waiting 10 seconds per link, and another 10 seconds when returning to the initial page, which has some 20 entries, and there are more than ten such pages that I need to scrape, so I passed on that...

by gkya 8y ago

jessriedel 8y ago

> For your RSS issue, you can safely truncate your RSS feed to the latest N articles, where N is a random reasonable number you choose (10 is good, because it's so decimal). RSS reader software will know how to deal with it. But people who don't use the right aggregators will not be able to read past 10 posts. And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. > WRT scraping, it is getting harder because funny websites w/ no static content. But sites that can have an RSS feed necessarily need to deliver their content in a static form. What surprises me is that we don't have a tool for even those cases.

gkya 8y ago
> But people who don't use the right aggregators will not be able to read past 10 posts. I do not think such aggregators exist, and you can just ignore people using software that do not comply to widespread conventions. > And if everyone was using aggregators, I wouldn't have such a high bandwidth bill. I don't understand this sentence. All RSS client software is called aggregator. > But sites that can have an RSS feed necessarily need to deliver their content in a static form. Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps. My uni's websites included. Some do it for the $$$, some for reasons that I can not know (jumping the bandwagons with minds toggled off). You can just set your blog software to truncate your feeds to a reasonable number w/o any worries. And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds.
- jessriedel 8y ago
  > I don't understand this sentence. All RSS client software is called aggregator I'm trying to distinguish between (1) people using software the directly downloads the RSS feed from my website to their device and (2) people who use services that download the RSS feed to a server which can then serve a cached copy of the posts to many users. If everyone used (2), then I would only have my RSS file downloaded as many times as there are separate services, which is not very many. So apparently many people are doing (1), and if my feed is short then they are limited in the blog history they can read to how long they've personally been subscribed (or less, if they need to clear their device and can't re-download my old posts). > Not really. Many websites which are essentially blogs are transforming themselves into single-page web apps That's why I specified "But sites that can have an RSS feed...". If you have an RSS feed with useful posts in the feed, then you must be delivering static pages. (If you're just delivering snippets with links to a dynamic page, then there is no way for any service to cache the page either.) So my question is: why isn't there software to scrape webpages of blogs that offer an RSS feed with complete posts? This would enable me to comveneiently read post history going back more than 10 posts. > And I suggest you look at your logs, because some silly bots might be consuming your bandwith along with your normal traffic; there are some that like RSS feeds. All the bots put together make up 27% of my bandwidth. It's a lot, but it's not the root cause.
  - gkya 8y ago
    > So apparently many people are doing (1), and if my feed is short then they are limited in the blog history they can read to how long they've personally been subscribed (or less, if they need to clear their device and can't re-download my old posts). RSS is a means for people to follow new posts from you, in order to read new posts they are supposed to come to your blog and use your archives.
    - jessriedel 8y ago
      The whole point of this subthread is about reading archives offline, and how RSS can't manage this. From my first comment: > it remains incredibly difficult to scrape a well-organized blog and turn it into something I can consume like RSS or a kindle book. > Note that many/most WordPress RSS feeds aren't that useful because...

ChristianBundy 8y ago

> ... where N is an arbitrary number you choose (12 is good, because https://dozenal.org https://dozenal.org). FTFY ;)

gkya 8y ago
Take a nice little joke please, downvoters, will ya?
progval 8y ago
> because http://dozenal.org http://dozenal.org FTFY

mtremsal 8y ago

FWIW, I had a similar issue recently: if you cannot find the link targets, you can still simulate a click on the element using phantomjs.

gkya 8y ago
Yeah, but for my case it means waiting 10 seconds per link, and another 10 seconds when returning to the initial page, which has some 20 entries, and there are more than ten such pages that I need to scrape, so I passed on that...