6 ms·

My perfect aggregator would also create a WARC archive of the webpage of each post, including all external references, maybe the referenced external websites an

by aw3c2 11y ago

My perfect aggregator would also create a WARC archive of the webpage of each post, including all external references, maybe the referenced external websites and their references (with that single depth of recursion). The internet is friggen fragile and I would love to archive what I consume.

vidarh 11y ago
I agree with that. I've recently been going through my archived blog posts while revamping my blog design, and so much has just flat out disappeared. And so much of what has disappeared have been things I would have guessed would stay...
rakoo 11y ago
I know it's not a feed aggregator, but you could intrumentalize pinboard (pinboard.in), they have the option of retrieving and storing an archive of all your links if you so desire. They even resolve first level dependencies, so external images are also stored (see https://blog.pinboard.in/2010/11/bookmark_archives_that_don_t/ https://blog.pinboard.in/2010/11/bookmark_archives_that_don_...). See some numbers on link rot here: https://blog.pinboard.in/2011/05/remembrance_of_links_past/ https://blog.pinboard.in/2011/05/remembrance_of_links_past/ Pinboard is built as a bookmark manager, but if you say that all entries in a feed is a bookmark then it should work for you. Oh and there's full-text search as well.
- Gmo 11y ago
  I think you can replace "they" by "he", since, as far as I know, it's a one-man shop !
- epaulson 11y ago
  Pinboard is fantastic, but it doesn't look like it exposes the archived pages through the API. If you ever wanted to leave Pinboard and do a data export, you might not be able to take everything with you.
zrail 11y ago
That's a really great idea. Here's a one-line wget that will grab the provided URL and all of the data necessary to render, to one level of recursion, and dump it to a WARC: wget -e robots=off \ --user-agent="Mozilla" \ -r 1 -p -E -H -k -K \ --warc-file=/path/to/your/warc/file/without/warc/extension \ 'http://www.example.com' I think I might start capturing these. Shouldn't take up too much additional disk space. edit: previously it was `-r 2` which is two levels of recursion.
- scrollaway 11y ago
  You can shorten that to `-pEHkKr 1`, so you know.
- aw3c2 11y ago
  You might have more fun with wpull, it has more sensible options to grab resources on referenced urls. https://github.com/chfoo/wpull https://github.com/chfoo/wpull
sheraz 11y ago
What if, instead of a WARC file, just a high-res screenshot available in desktop, tablet, and mobile mode?
- jimmyswimmy 11y ago
  Why would a binary file be better than its source? Seems like a lot more work (you'd need to format and store for each display size) in exchange for less capability (can't search, probably a larger overall file).
  - sheraz 11y ago
    ..was just thinking out loud :-) But one advantage is that this is immediately viewable on any device, because otherwise you would need a WARC viewer, which are not so easy to setup.
derefr 11y ago
To go further: there's basically no point in the "description" part of an RSS item. RSS is broken in that authors need to lure people onto their sites, so they make the RSS item itself enclose just enough of a preview to make you "click through"—whereupon their site can show you ads and they can make money. How RSS should work, in an ideal technical sense, is to eschew enclosing any content-body in feed items themselves, and instead just encourage RSS consumers (feed-reader clients; feed-muxer daemons) to scrape the permalinks of the feed items, and then heuristically extract the body-content from the scrape-result, and cache both the resulting page-archive and the resulting cleaned-up text, making both representations available offline. This, obviously, kills blog ad revenue. But it's better to kill it and replace it with something better (402 micropayment-required errors at point-of-caching, handled automatically by the RSS content-spidering daemon as an HTTP client, with costs passed on to its subscribers?) than to continue on with this semi-braindamaged "I have an offline cache but that doesn't actually mean I can read anything offline" world.
- okasaki 11y ago
  All the blogs I read include the full content in the feed. Some even include top comments. And if you're parsing sites then you have no use for RSS - you can just parse the index to see what's new. Sounds like an nightmare to me though. Who's going to maintain these parsers for hundreds of sites?
  - derefr 11y ago
    RSS tells you what's actually a new page. Usually, if a site wants to be Google-friendly, it'll have a page for every new content item—so, if a site publishes an RSS feed, that's usually enough to "chunk" their content by time. Heuristics to decide what's new on a site are much, much harder to code than heuristics to extract content when you're explicitly told what's new. There was a service called Dapper, who tried the "extract content from a CSS-selector-specified zone of a page when content changes" approach... and it didn't go well. Yahoo Pipes had similar aspirations; again, got shuttered. There are always sites that are really bad internet citizens. Some sites just change their front page to update, without creating a canonical archival URL—so then, if they do publish an RSS feed, every item is just a link back to said front page. There's not much you can do in these cases beyond just crowdsourcing "parsers for hundreds of sites" (which does happen—webcomics being a frequent and horrifying example.) But there is a pretty good alternative, I think: if you can scrape the site itself as a whole at regular intervals, with fine enough granularity that any two contiguous scrapes will detect either 0 or 1 change, then you can probably convert that into a useful RSS feed without trying to figure out from "what changed" from DOM deltas. Basically, shove each scrape into the working index of a git repo, commit, and generate an RSS feed of the diffs. (Wikipedia can give you this in some form; I think gwern.net has an RSS feed that also basically works this way.) And, if the site has even the most minimal source RSS feed, you don't need the "regular intervals" approach: you can use the source RSS feed to provide "cueing" information (i.e. timestamps of when the site has changed) for your scrapes. Which is all to say, RSS is a very "utilitarian" format. You don't need to rely on all of this processing happening on the server side, but rather can just take one RSS feed, and then use whatever bits of it you want to generate another RSS feed, and then someone can consume that RSS feed and write a heuristic to generate cluster and combine the feed items from it into higher-level summaries, etc. This sort of thing really shows the "theme" of RSS, to me. RSS wasn't really designed or intended for direct consumption by readers, but rather to make it easy to have something on your site (even your statically-generated site, or your crap one-off PHP blog) that's easy for other services to consume and turn into stuff. It's "Really Simple Syndication": it's not the whole process of getting stuff to clients, it's just the lowest barrier-to-entry to get the supply-side of the equation involved, so that well-made delivery infrastructure can take over from there. There were never meant to be "RSS reader" software, really, or even e.g. consume-side RSS-to-email gateways. Instead, RSS was meant to be lowest-common-denominator format for other supplier-side services to consume. Instead of writing "lifecycle emails", for example, you were meant to just have your web app generate a lifecycle notification stream RSS feed for each user—and then subscribe an emailing service to it. We've replaced this behavior, for the most part, with webhooks—having our web-apps actually reach out using a REST client and prod other services' APIs. But RSS (with one consumer—the hooked service) is much, much cheaper than a REST client, from all perspectives: your web-app can likely already generate XML, it likely already has a list of changes and "view rendering" capabilities, etc. Webhooks are for the 1% of apps that can both 1. run server-side, 2. reach-out from their sandboxes to poke something elsewhere on the web, and 3. have a public URL where they can be poked back at. An RSS-based pub-sub event system, meanwhile, will work even if the event-generator is on your local machine: as long as you proxy over the RSS feed URL itself, you don't need to worry about machines trying to figure out whether the site that prodded them with a webhook request is coming from the right IP for that client or not, etc. They just get to blindly consume a URL, like any other piece of idiomatic web software.

by aw3c2 11y ago

vidarh 11y ago

I agree with that. I've recently been going through my archived blog posts while revamping my blog design, and so much has just flat out disappeared. And so much of what has disappeared have been things I would have guessed would stay...

rakoo 11y ago

I know it's not a feed aggregator, but you could intrumentalize pinboard (pinboard.in), they have the option of retrieving and storing an archive of all your links if you so desire. They even resolve first level dependencies, so external images are also stored (see https://blog.pinboard.in/2010/11/bookmark_archives_that_don_t/ https://blog.pinboard.in/2010/11/bookmark_archives_that_don_...). See some numbers on link rot here: https://blog.pinboard.in/2011/05/remembrance_of_links_past/ https://blog.pinboard.in/2011/05/remembrance_of_links_past/ Pinboard is built as a bookmark manager, but if you say that all entries in a feed is a bookmark then it should work for you. Oh and there's full-text search as well.

Gmo 11y ago
I think you can replace "they" by "he", since, as far as I know, it's a one-man shop !
epaulson 11y ago
Pinboard is fantastic, but it doesn't look like it exposes the archived pages through the API. If you ever wanted to leave Pinboard and do a data export, you might not be able to take everything with you.

zrail 11y ago

That's a really great idea. Here's a one-line wget that will grab the provided URL and all of the data necessary to render, to one level of recursion, and dump it to a WARC: wget -e robots=off \ --user-agent="Mozilla" \ -r 1 -p -E -H -k -K \ --warc-file=/path/to/your/warc/file/without/warc/extension \ 'http://www.example.com' I think I might start capturing these. Shouldn't take up too much additional disk space. edit: previously it was `-r 2` which is two levels of recursion.

scrollaway 11y ago
You can shorten that to `-pEHkKr 1`, so you know.
aw3c2 11y ago
You might have more fun with wpull, it has more sensible options to grab resources on referenced urls. https://github.com/chfoo/wpull https://github.com/chfoo/wpull

sheraz 11y ago

What if, instead of a WARC file, just a high-res screenshot available in desktop, tablet, and mobile mode?

jimmyswimmy 11y ago
Why would a binary file be better than its source? Seems like a lot more work (you'd need to format and store for each display size) in exchange for less capability (can't search, probably a larger overall file).
- sheraz 11y ago
  ..was just thinking out loud :-) But one advantage is that this is immediately viewable on any device, because otherwise you would need a WARC viewer, which are not so easy to setup.

derefr 11y ago

To go further: there's basically no point in the "description" part of an RSS item. RSS is broken in that authors need to lure people onto their sites, so they make the RSS item itself enclose just enough of a preview to make you "click through"—whereupon their site can show you ads and they can make money. How RSS should work, in an ideal technical sense, is to eschew enclosing any content-body in feed items themselves, and instead just encourage RSS consumers (feed-reader clients; feed-muxer daemons) to scrape the permalinks of the feed items, and then heuristically extract the body-content from the scrape-result, and cache both the resulting page-archive and the resulting cleaned-up text, making both representations available offline. This, obviously, kills blog ad revenue. But it's better to kill it and replace it with something better (402 micropayment-required errors at point-of-caching, handled automatically by the RSS content-spidering daemon as an HTTP client, with costs passed on to its subscribers?) than to continue on with this semi-braindamaged "I have an offline cache but that doesn't actually mean I can read anything offline" world.

okasaki 11y ago
All the blogs I read include the full content in the feed. Some even include top comments. And if you're parsing sites then you have no use for RSS - you can just parse the index to see what's new. Sounds like an nightmare to me though. Who's going to maintain these parsers for hundreds of sites?
- derefr 11y ago
  RSS tells you what's actually a new page. Usually, if a site wants to be Google-friendly, it'll have a page for every new content item—so, if a site publishes an RSS feed, that's usually enough to "chunk" their content by time. Heuristics to decide what's new on a site are much, much harder to code than heuristics to extract content when you're explicitly told what's new. There was a service called Dapper, who tried the "extract content from a CSS-selector-specified zone of a page when content changes" approach... and it didn't go well. Yahoo Pipes had similar aspirations; again, got shuttered. There are always sites that are really bad internet citizens. Some sites just change their front page to update, without creating a canonical archival URL—so then, if they do publish an RSS feed, every item is just a link back to said front page. There's not much you can do in these cases beyond just crowdsourcing "parsers for hundreds of sites" (which does happen—webcomics being a frequent and horrifying example.) But there is a pretty good alternative, I think: if you can scrape the site itself as a whole at regular intervals, with fine enough granularity that any two contiguous scrapes will detect either 0 or 1 change, then you can probably convert that into a useful RSS feed without trying to figure out from "what changed" from DOM deltas. Basically, shove each scrape into the working index of a git repo, commit, and generate an RSS feed of the diffs. (Wikipedia can give you this in some form; I think gwern.net has an RSS feed that also basically works this way.) And, if the site has even the most minimal source RSS feed, you don't need the "regular intervals" approach: you can use the source RSS feed to provide "cueing" information (i.e. timestamps of when the site has changed) for your scrapes. Which is all to say, RSS is a very "utilitarian" format. You don't need to rely on all of this processing happening on the server side, but rather can just take one RSS feed, and then use whatever bits of it you want to generate another RSS feed, and then someone can consume that RSS feed and write a heuristic to generate cluster and combine the feed items from it into higher-level summaries, etc. This sort of thing really shows the "theme" of RSS, to me. RSS wasn't really designed or intended for direct consumption by readers, but rather to make it easy to have something on your site (even your statically-generated site, or your crap one-off PHP blog) that's easy for other services to consume and turn into stuff. It's "Really Simple Syndication": it's not the whole process of getting stuff to clients, it's just the lowest barrier-to-entry to get the supply-side of the equation involved, so that well-made delivery infrastructure can take over from there. There were never meant to be "RSS reader" software, really, or even e.g. consume-side RSS-to-email gateways. Instead, RSS was meant to be lowest-common-denominator format for other supplier-side services to consume. Instead of writing "lifecycle emails", for example, you were meant to just have your web app generate a lifecycle notification stream RSS feed for each user—and then subscribe an emailing service to it. We've replaced this behavior, for the most part, with webhooks—having our web-apps actually reach out using a REST client and prod other services' APIs. But RSS (with one consumer—the hooked service) is much, much cheaper than a REST client, from all perspectives: your web-app can likely already generate XML, it likely already has a list of changes and "view rendering" capabilities, etc. Webhooks are for the 1% of apps that can both 1. run server-side, 2. reach-out from their sandboxes to poke something elsewhere on the web, and 3. have a public URL where they can be poked back at. An RSS-based pub-sub event system, meanwhile, will work even if the event-generator is on your local machine: as long as you proxy over the RSS feed URL itself, you don't need to worry about machines trying to figure out whether the site that prodded them with a webhook request is coming from the right IP for that client or not, etc. They just get to blindly consume a URL, like any other piece of idiomatic web software.