3 ms·

All the blogs I read include the full content in the feed. Some even include top comments. And if you're parsing sites then you have no use for RSS - you can j

by okasaki 11y ago

All the blogs I read include the full content in the feed. Some even include top comments. And if you're parsing sites then you have no use for RSS - you can just parse the index to see what's new. Sounds like an nightmare to me though. Who's going to maintain these parsers for hundreds of sites?

derefr 11y ago
RSS tells you what's actually a new page. Usually, if a site wants to be Google-friendly, it'll have a page for every new content item—so, if a site publishes an RSS feed, that's usually enough to "chunk" their content by time. Heuristics to decide what's new on a site are much, much harder to code than heuristics to extract content when you're explicitly told what's new. There was a service called Dapper, who tried the "extract content from a CSS-selector-specified zone of a page when content changes" approach... and it didn't go well. Yahoo Pipes had similar aspirations; again, got shuttered. There are always sites that are really bad internet citizens. Some sites just change their front page to update, without creating a canonical archival URL—so then, if they do publish an RSS feed, every item is just a link back to said front page. There's not much you can do in these cases beyond just crowdsourcing "parsers for hundreds of sites" (which does happen—webcomics being a frequent and horrifying example.) But there is a pretty good alternative, I think: if you can scrape the site itself as a whole at regular intervals, with fine enough granularity that any two contiguous scrapes will detect either 0 or 1 change, then you can probably convert that into a useful RSS feed without trying to figure out from "what changed" from DOM deltas. Basically, shove each scrape into the working index of a git repo, commit, and generate an RSS feed of the diffs. (Wikipedia can give you this in some form; I think gwern.net has an RSS feed that also basically works this way.) And, if the site has even the most minimal source RSS feed, you don't need the "regular intervals" approach: you can use the source RSS feed to provide "cueing" information (i.e. timestamps of when the site has changed) for your scrapes. Which is all to say, RSS is a very "utilitarian" format. You don't need to rely on all of this processing happening on the server side, but rather can just take one RSS feed, and then use whatever bits of it you want to generate another RSS feed, and then someone can consume that RSS feed and write a heuristic to generate cluster and combine the feed items from it into higher-level summaries, etc. This sort of thing really shows the "theme" of RSS, to me. RSS wasn't really designed or intended for direct consumption by readers, but rather to make it easy to have something on your site (even your statically-generated site, or your crap one-off PHP blog) that's easy for other services to consume and turn into stuff. It's "Really Simple Syndication": it's not the whole process of getting stuff to clients, it's just the lowest barrier-to-entry to get the supply-side of the equation involved, so that well-made delivery infrastructure can take over from there. There were never meant to be "RSS reader" software, really, or even e.g. consume-side RSS-to-email gateways. Instead, RSS was meant to be lowest-common-denominator format for other supplier-side services to consume. Instead of writing "lifecycle emails", for example, you were meant to just have your web app generate a lifecycle notification stream RSS feed for each user—and then subscribe an emailing service to it. We've replaced this behavior, for the most part, with webhooks—having our web-apps actually reach out using a REST client and prod other services' APIs. But RSS (with one consumer—the hooked service) is much, much cheaper than a REST client, from all perspectives: your web-app can likely already generate XML, it likely already has a list of changes and "view rendering" capabilities, etc. Webhooks are for the 1% of apps that can both 1. run server-side, 2. reach-out from their sandboxes to poke something elsewhere on the web, and 3. have a public URL where they can be poked back at. An RSS-based pub-sub event system, meanwhile, will work even if the event-generator is on your local machine: as long as you proxy over the RSS feed URL itself, you don't need to worry about machines trying to figure out whether the site that prodded them with a webhook request is coming from the right IP for that client or not, etc. They just get to blindly consume a URL, like any other piece of idiomatic web software.

by okasaki 11y ago

derefr 11y ago

RSS tells you what's actually a new page. Usually, if a site wants to be Google-friendly, it'll have a page for every new content item—so, if a site publishes an RSS feed, that's usually enough to "chunk" their content by time. Heuristics to decide what's new on a site are much, much harder to code than heuristics to extract content when you're explicitly told what's new. There was a service called Dapper, who tried the "extract content from a CSS-selector-specified zone of a page when content changes" approach... and it didn't go well. Yahoo Pipes had similar aspirations; again, got shuttered. There are always sites that are really bad internet citizens. Some sites just change their front page to update, without creating a canonical archival URL—so then, if they do publish an RSS feed, every item is just a link back to said front page. There's not much you can do in these cases beyond just crowdsourcing "parsers for hundreds of sites" (which does happen—webcomics being a frequent and horrifying example.) But there is a pretty good alternative, I think: if you can scrape the site itself as a whole at regular intervals, with fine enough granularity that any two contiguous scrapes will detect either 0 or 1 change, then you can probably convert that into a useful RSS feed without trying to figure out from "what changed" from DOM deltas. Basically, shove each scrape into the working index of a git repo, commit, and generate an RSS feed of the diffs. (Wikipedia can give you this in some form; I think gwern.net has an RSS feed that also basically works this way.) And, if the site has even the most minimal source RSS feed, you don't need the "regular intervals" approach: you can use the source RSS feed to provide "cueing" information (i.e. timestamps of when the site has changed) for your scrapes. Which is all to say, RSS is a very "utilitarian" format. You don't need to rely on all of this processing happening on the server side, but rather can just take one RSS feed, and then use whatever bits of it you want to generate another RSS feed, and then someone can consume that RSS feed and write a heuristic to generate cluster and combine the feed items from it into higher-level summaries, etc. This sort of thing really shows the "theme" of RSS, to me. RSS wasn't really designed or intended for direct consumption by readers, but rather to make it easy to have something on your site (even your statically-generated site, or your crap one-off PHP blog) that's easy for other services to consume and turn into stuff. It's "Really Simple Syndication": it's not the whole process of getting stuff to clients, it's just the lowest barrier-to-entry to get the supply-side of the equation involved, so that well-made delivery infrastructure can take over from there. There were never meant to be "RSS reader" software, really, or even e.g. consume-side RSS-to-email gateways. Instead, RSS was meant to be lowest-common-denominator format for other supplier-side services to consume. Instead of writing "lifecycle emails", for example, you were meant to just have your web app generate a lifecycle notification stream RSS feed for each user—and then subscribe an emailing service to it. We've replaced this behavior, for the most part, with webhooks—having our web-apps actually reach out using a REST client and prod other services' APIs. But RSS (with one consumer—the hooked service) is much, much cheaper than a REST client, from all perspectives: your web-app can likely already generate XML, it likely already has a list of changes and "view rendering" capabilities, etc. Webhooks are for the 1% of apps that can both 1. run server-side, 2. reach-out from their sandboxes to poke something elsewhere on the web, and 3. have a public URL where they can be poked back at. An RSS-based pub-sub event system, meanwhile, will work even if the event-generator is on your local machine: as long as you proxy over the RSS feed URL itself, you don't need to worry about machines trying to figure out whether the site that prodded them with a webhook request is coming from the right IP for that client or not, etc. They just get to blindly consume a URL, like any other piece of idiomatic web software.