4 ms·
I used to work on Bloglines. It's sad to see it finally die, but the writing was on the wall for years. There are two questions I'd like Ask.com to answer: Wha
by AngryParsley 16y ago
I used to work on Bloglines. It's sad to see it finally die, but the writing was on the wall for years.
There are two questions I'd like Ask.com to answer: What will happen to the source code? and What will happen to the archive of RSS items?
Ideally, Ask.com would open source Bloglines and get the item data over to archive.org. We never deleted anything, so the item archive is around 11TB and has blog posts going back to 2003. I bet quite a bit of that data doesn't exist anywhere else. Although Sturgeon's Law applies, there are some gems in that data. I took a look once and found some interesting blog posts from the 2004 election cycle and Hurricane Katrina.
I seriously doubt that will happen though. Building an archival tool for the podserver/itemdbs would require more dev work, and there are the hardware costs of reliably storing 11TB.
- mikemol 16y agoWhat might it cost to buy off the data to a reliable datastore? Could the format of the data be opened to let outsiders write the processing tools?
- AngryParsley 16y agoNo idea and probably not. The Bloglines podserver/itemdb was built internally. You can get some idea of the setup by reading the slides from pquerna's GeekSession's talk: http://journal.paul.querna.org/wp-content/uploads/2007/10/bl-geeksessions.pdf http://journal.paul.querna.org/wp-content/uploads/2007/10/bl...
- tomjen3 16y agoI hope not - I had private RSS feed from things I don't want to share with the world, and I can't be the only one.
- AngryParsley 16y agoBloglines already has that logic. It would be built into any archival tool as well. If the feed url scheme is https or if it's using http auth, it's private.
- jemfinch 16y ago> I bet quite a bit of that data doesn't exist anywhere else. You'd really put money on that?
- gwern 16y agoI'd put everything I own on it (for reasonable interpretations of 'quite a bit'). I know from personal experience that web pages and blogs die like flies, and that the Internet Archive - the only live possibility for pages in general - often simply doesn't have copies or the copies are unavailable. (Lost a number of _Mainichi Shimbun_ articles, IIRC, to a robots.txt that went up years after the articles did; the IA respects it anyway. Sometimes the domain changes hands and the new owners put up a restrictive robots.txt; I hate that too.) 11TB of HTML & images goes a very long way. To put that in perspective, that's in the same area as a full copy of all Wikimedia projects, including the English Wikipedia and Commons.
- jemfinch 16y agoIt seems to have firsthand knowledge that this data has been stored permanently by Bloglines; what I'm curious about is why you think it wasn't stored by any other companies that crawl the web for a living.
- gwern 16y agoIf a tree falls in Google's archives and no non-Googler can access it, does it really exist? Anyway. As AngryParsley says, the specific Google services in question inherently have a gap of a few years. I have already given examples from my personal trifling experiences looking up a few hundred dead sites where the sole public archive, the IA, fails. Scale that up to hundreds of thousands and millions of sites...
- AngryParsley 16y agoIt's just HTML/text and it's 11TB gzipped as of 2008. I bet the archive has grown since then, but the valuable data is the oldest. Uncompressed it's probably in the 20TB range. I'd also bet everything I own that a significant fraction of the data is unavailable anywhere else. Google Reader started in 2005. Ditto for Google Blog Search. Even if they crawled the same feeds as Bloglines and kept the data indefinitely, they'll still be missing those first two years.
- patrickgzill 16y ago11TB can be reliably stored on a 2U server with 8x2TB drives. Figure $150 per drive = $1200 plus cost of the server.