8 ms·

Notes on bookmarks from 1997

220 points by spindritf 12y ago

thrownaway2424 12y ago
"... a VRML tutorial is now a video about birth control." VRML tutorials are all fundamentally about birth control.
wodenokoto 12y ago
This is the main reason I adopted zotero for my citation manager - it saves a local version of the resources I want to cite, preventing link rot and paywall from obstructing returning to an old research topic. If your bookmarking habit is article, rather than web-services, your bookmark manager needs a way of saving bookmarked content.
- lingben 12y ago
  thanks, I hadn't heard of zotero before, so it is like evernote except the data is stored locally on your own computer?
  - deong 12y ago
    It's intended as a reference manager for academics, but in addition to PDFs, you can also save other resources like web pages. The interface isn't really designed for using it as an Evernote replacement, but for some use cases at least, it would be workable.
- hawktron 12y ago
  I use Stache it stores screenshots and web archives perfect for this kinda stuff http://getstache.com http://getstache.com
  - vitovito 12y ago
    Hi! You could ask Stache, too, to do full WARC captures, which you could then donate to the Internet Archive and have them be included in the Wayback Machine, which would benefit everyone.
  - sepeth 12y ago
    Is Stache mac only? and is there a way to change iCloud? e.g. I would prefer dropbox.
- vitovito 12y ago
  That helps you, but that doesn't help everyone else. Consider asking Zotero to do full WARC captures, which you could then donate to the Internet Archive and have them be included in the Wayback Machine, which would benefit everyone.
  - dredmorbius 12y ago
    Oooh! I like this.
  - dredmorbius 12y ago
    Thinking about this further, this might also be a really useful tie-in for Wikipedia. It also increasingly relies on Web citations, many of which fail. If cited material were automatically archived and submitted to TIA, this could be further useful. The fact that this is an inherent archival of information which is deemed relevant and citable is also worth noting.
    - vitovito 12y ago
      They're starting to do this: http://blog.archive.org/2013/10/25/fixing-broken-links/ http://blog.archive.org/2013/10/25/fixing-broken-links/ We have started crawling the outlinks for every new article and update as they are made – about 5 million new URLs are archived every day. Now we have to figure out how to get archived pages back in to Wikipedia to fix some of those dead links. Kunal Mehta, a Wikipedian from San Jose, recently wrote a protoype bot that can add archived versions to any link in Wikipedia so that when those links are determined to be dead the links can be switched over automatically and continue to work. It will take a while to work this through the process the Wikipedia community of editors uses to approve bots, but that conversation is under way.
      - dredmorbius 12y ago
        That's great news.
  - whyenot 12y ago
    Hi, the Zotero project is open source and non-profit, currently getting the majority of its funding from the Roy Rosenzweig Center for History and New Media at George Mason University. Following the open source model, it depends on people like you to help out with coding new features. If doing full WARC captures is something that you feel is important, you might want to help out and contribute the necessary code. Zotero's code base is pretty approachable. I've personally submitted fixes to the project. Go here for more information on how to get involved: https://www.zotero.org/getinvolved/ https://www.zotero.org/getinvolved/
chimeracoder 12y ago
> First, donate to the Internet Archive: http://archive.org/donate/ http://archive.org/donate/ This is a good step, but there's one massive problem with the Internet Archive: If the domain registration lapses (or is sold), the new owner can direct the Internet Archive to remove archived copies of the site previously hosted on that domain[0]. In other words, even if the archives are there today, they may not be there when you want them a year (or more) from now). I've been bitten by this in the past. This means that the Internet Archive wouldn't have been able to prevent or mitigate some of the cases of link rot described here, as the content could have been scrubbed from the archive. EDIT In case it wasn't clear, I still support the Internet Archive - I just want to make sure people know this issue exists and that it's not a "silver bullet" for this problem. [0] I'm fairly certain this is still the case; if they have changed their policy recently I would be pleasantly surprised.
- idlewords 12y ago
  It would shock me if there's a working copy of 'rm' allowed anywhere near the Internet Archive. They take this stuff down for compliance, but my dream is that the data lives on, waiting for a saner day when the legal climate for archivists gets a little warmer.
  - TazeTSchnitzel 12y ago
    Unless it's child porn, say.
  - ghaff 12y ago
    It's tough though--supportive as I am of the Internet Archive's goals. How is an "archivist" different from a random individual who scrapes stuff of the Internet and rehosts it? In the aggregate, the Internet Archive looks different from the typical person who is copying articles and blog posts, wrapping them up in ads, and displaying them. The IA is non-profit, doesn't run ads, etc. It also respects robots.txt. But it's not that clear to me what the legal regime would be that allows the Internet Archive to function free and clear and doesn't hit cases that most would agree are shady.
DanBC 12y ago
Is there an update to the (now ancient) W3C "Cool URIs Don't Change"? http://www.w3.org/Provider/Style/URI.html http://www.w3.org/Provider/Style/URI.html
- dredmorbius 12y ago
  From the article, the clear winner would be "cool URIs enable Wayback Machine archival".
pronoiac 12y ago
I hadn't realized I could write notes on Pinboard! I've written long descriptions for my bookmarks on occasion, though.
- RexRollman 12y ago
  I didn't either. What a nice feature. (BTW: Pinboard is one of my favorite one-person companies, the other being Tarsnap. I love that the both seem to be run be people interested in building something and are not just looking for an exit strategy.)
- joshu 12y ago
  Notes was on the todo list very early on for delicious, but never allowed to proceed. Glad to see it working.
rospaya 12y ago
I have around 4000 bookmarks from ~2003 to 2009 when I stopped hoarding them. Afraid to check how many of them will work.
- vitovito 12y ago
  It's a terrible problem. Maciej Ceglowski is doing his own study on link rot: https://blog.pinboard.in/2014/08/researching_link_rot/ https://blog.pinboard.in/2014/08/researching_link_rot/ He says my numbers match his own so far, about 5% per year. So I'd guess ~2200 bookmarks are dead, and maybe you could get ~1000 of them back from the Wayback Machine. I have 28MB of personal bookmark files I'll be post-processing, as well as ~47,000 links from a shared, private bookmarking service dating from 2005 through 2011. I'm not looking forward to it.
- dredmorbius 12y ago
  NB: wget's '--spider' argument can be used to test link integrity (though it only checks that the link returns 200, not that the content is intact).
purplerails 12y ago
Here's my take on a solution to the problem of linkrot: https://www.purplerails.com/ https://www.purplerails.com/ Major points: * Automatically saves an exact copy of pages (no need to explicitly bookmark) in the background. * Data is encrypted with your password before being sync'd to cloud. * Search through your pages. * Works as a Chrome browser extension. No need for a native app.
- pain 12y ago
  If I need to rely on your private domain to access my own research, how is it different or less risky than diigo etc? I'm waiting for an extension that lets me keep my own full data activity (and use your cloud too, optionally in addition).
  - purplerails 12y ago
    Understood. Thx for the comment. The ability to download your data in a well-documented format (possibly WARC) is coming soon. I hope you will try out PurpleRails in the meanwhile. Thx again!
- vitovito 12y ago
  Hi! Same thing here as with the other examples mentioned in this thread: this only helps you. If you save a page but someone else needs it, they're out of luck. But, if, in addition to making you a private, encrypted archive, they also tested to see if the URL was publicly visible and, if so, made a WARC of it, then they could package up all those WARCs for donation to the Internet Archive, and everyone could benefit.
  - purplerails 12y ago
    > If you save a page but someone else needs it, they're out of luck. There is a sharing feature to solve this problem. :) But I agree with your point. I actually looked into WARC earlier but didn't have the bandwidth to do it my first version. When I implement the ability to download your data, I'll try hard to use WARC. Unless there's some brain damage in the format: I hope not! :)
sugarfactory 12y ago
> Every URL saved in more than one place increases the likelihood that their content will survive as domains change owners. Surely it's obvious that the more copies the better for the backup purpose. Then, why is he advocating cloud services including the Wayback Machine? There's no doubt that it's important for everyone to save web pages locally to create as many copies as possible on the earth to prevent the disaster of the Library of Alexandria from happening again.
- vitovito 12y ago
  Because it does someone no good if you have a copy of an old page they're looking for, and they have no way to find you, and you have no way of providing it to them. Centralized services like libraries and archives provide that. You absolutely should save everything yourself. But you should also give copies to as many centralized services as possible. Lots of copies keeps stuff safe.
  - sugarfactory 12y ago
    "Centralized" was not the word you were looking for. Decentralized systems can serve the searching functionality as proved by many (pure) P2P file-sharing protocols.
bigethan 12y ago
Funny, I used to work for DataRealm (impressed the OP capitalized it right). Wasn't there when they sold the serve.com domain, but it was a big deal for them. Everything internal used serve.com (including email addresses) and switching to Datarealm.com was a lot of changes. As a small business it's hard to say "cool URLs don't change" when you're just trying to get by.
slackpad 12y ago
I worry about this a lot. There's archive.org doing an amazing job but it seems like so much of our culture is getting lost because it's online. My box of Popular Science magazines from the 1950s is amazing - even the ads are interesting. The difficulty in archiving this stuff is worrying, as is the discovery experience for someone 60 or 70 years down the road. I can pick up a crusty magazine and flip through it, cutting across a whole bunch of topics and their surrounding context. There are lots of silos trying to save bits of it, but if this guy put a bunch of papers in a box in 1997 and it didn't get eaten by mold, then all of it would have been around to browse today.
- vitovito 12y ago
  Absolutely. We know how to archive paper pretty well. Your box of magazines will last another 20 years in your garage. Maybe twice that in a climate-controlled vault. But they're still made of cheap paper, and will eventually decompose, or your garage will flood, or bugs will eat them, or mold will be particularly bad one year. And, like anything physical, unless someone makes a pilgrimage to your garage (or that vault), no-one else will ever (re)discover them, or share them. That's the problem with physical things: they can be controlled, hoarded, kept secret, or destroyed. Born-digital artifacts, and digitized physical ones, can be much more discoverable than physical ones. They often can be repurposed and remixed much more easily. And they absolutely can be copied and distributed widely in ways that physical artifacts can't. Sticking to paper isn't the answer. Figuring out better ways to index, preserve and distribute digital artifacts is.
  - _delirium 12y ago
    I agree that paper per se isn't the advantage, but the fact that something comes in a relatively unencumbered and self-contained unit does help a lot with digital archiving. That is so far still more common with paper on average. A library can build a more or less complete digital archive of a paper newspaper by simply subscribing to it and setting up a routine digitization process. Building a complete archive of an online news site is much harder: you have to figure out how to scrape it in full, and many sites actively try to stop you from doing so. As publications have discovered that selling access to their own archives (or "vault" as some call it) is profitable, they are even less likely to want free third-party archives to be available. The best-case for "born digital" archiving at the moment are probably publications that still produce a full PDF version of each "issue". For example a library could archive The New Inquiry by simply subscribing and downloading each month's subscribers-only PDF, rather than trying to scrape the website. A publication that put the full content in an RSS feed would also be fairly easy to "subscribe" to and archive. But most don't have an easy way of subscribing to full updates that can be archived and stored/viewed independently of the original site, and many are even kind of hostile to the idea. Journals are another problem. Paper journal subscriptions go into a library's permanent holdings, while digital journal subscriptions often give them cheaper per-issue prices, but in return it's more like renting access to the journal-controlled archive, which goes away if the library cancels the subscription. Most don't allow a library to mass-download the archive into the library's own digital holdings, though some do have some self-archival arrangements (e.g. a library might be allowed to self-archive issues that came out during their period of subscription, but not the whole back catalogue).
ibisum 12y ago
Bookmarks? I just print-to-PDF. Offline, permanent, safe. Bookmarks have never been useful.
- onion2k 12y ago
  A print of a page is not the same as a bookmark. Rather than a link to a point in time (which is what your print out is), a bookmark is a pointer to the latest version of a page. That has many benefits if the page is actively updated. The cost of that advantage is that it can fail completely if the page is taken down. Ideally a browser should cache a page when you bookmark it and optionally refresh the cache when you revisit. That way, if you get an error you could look back at the cached version instead. In fact, that'd be a damn useful browser plugin.
- vitovito 12y ago
  But that only helps you. That doesn't help anyone else who needs that old reference. Saving things in a way that those backups can be shared and distributed, like WARC files donated to the Internet Archive, can help ensure that when your hard disk crashes and your backups fail and you lose those PDFs, the things you wanted to save will still be out there.

220 points by spindritf 12y ago

"... a VRML tutorial is now a video about birth control." VRML tutorials are all fundamentally about birth control.

This is the main reason I adopted zotero for my citation manager - it saves a local version of the resources I want to cite, preventing link rot and paywall from obstructing returning to an old research topic. If your bookmarking habit is article, rather than web-services, your bookmark manager needs a way of saving bookmarked content.

lingben 12y ago
thanks, I hadn't heard of zotero before, so it is like evernote except the data is stored locally on your own computer?
- deong 12y ago
  It's intended as a reference manager for academics, but in addition to PDFs, you can also save other resources like web pages. The interface isn't really designed for using it as an Evernote replacement, but for some use cases at least, it would be workable.
hawktron 12y ago
I use Stache it stores screenshots and web archives perfect for this kinda stuff http://getstache.com http://getstache.com
- vitovito 12y ago
  Hi! You could ask Stache, too, to do full WARC captures, which you could then donate to the Internet Archive and have them be included in the Wayback Machine, which would benefit everyone.
- sepeth 12y ago
  Is Stache mac only? and is there a way to change iCloud? e.g. I would prefer dropbox.
vitovito 12y ago
That helps you, but that doesn't help everyone else. Consider asking Zotero to do full WARC captures, which you could then donate to the Internet Archive and have them be included in the Wayback Machine, which would benefit everyone.
- dredmorbius 12y ago
  Oooh! I like this.
- dredmorbius 12y ago
  Thinking about this further, this might also be a really useful tie-in for Wikipedia. It also increasingly relies on Web citations, many of which fail. If cited material were automatically archived and submitted to TIA, this could be further useful. The fact that this is an inherent archival of information which is deemed relevant and citable is also worth noting.
  - vitovito 12y ago
    They're starting to do this: http://blog.archive.org/2013/10/25/fixing-broken-links/ http://blog.archive.org/2013/10/25/fixing-broken-links/ We have started crawling the outlinks for every new article and update as they are made – about 5 million new URLs are archived every day. Now we have to figure out how to get archived pages back in to Wikipedia to fix some of those dead links. Kunal Mehta, a Wikipedian from San Jose, recently wrote a protoype bot that can add archived versions to any link in Wikipedia so that when those links are determined to be dead the links can be switched over automatically and continue to work. It will take a while to work this through the process the Wikipedia community of editors uses to approve bots, but that conversation is under way.
    - dredmorbius 12y ago
      That's great news.
- whyenot 12y ago
  Hi, the Zotero project is open source and non-profit, currently getting the majority of its funding from the Roy Rosenzweig Center for History and New Media at George Mason University. Following the open source model, it depends on people like you to help out with coding new features. If doing full WARC captures is something that you feel is important, you might want to help out and contribute the necessary code. Zotero's code base is pretty approachable. I've personally submitted fixes to the project. Go here for more information on how to get involved: https://www.zotero.org/getinvolved/ https://www.zotero.org/getinvolved/

chimeracoder 12y ago

> First, donate to the Internet Archive: http://archive.org/donate/ http://archive.org/donate/ This is a good step, but there's one massive problem with the Internet Archive: If the domain registration lapses (or is sold), the new owner can direct the Internet Archive to remove archived copies of the site previously hosted on that domain[0]. In other words, even if the archives are there today, they may not be there when you want them a year (or more) from now). I've been bitten by this in the past. This means that the Internet Archive wouldn't have been able to prevent or mitigate some of the cases of link rot described here, as the content could have been scrubbed from the archive. EDIT In case it wasn't clear, I still support the Internet Archive - I just want to make sure people know this issue exists and that it's not a "silver bullet" for this problem. [0] I'm fairly certain this is still the case; if they have changed their policy recently I would be pleasantly surprised.

idlewords 12y ago
It would shock me if there's a working copy of 'rm' allowed anywhere near the Internet Archive. They take this stuff down for compliance, but my dream is that the data lives on, waiting for a saner day when the legal climate for archivists gets a little warmer.
- TazeTSchnitzel 12y ago
  Unless it's child porn, say.
- ghaff 12y ago
  It's tough though--supportive as I am of the Internet Archive's goals. How is an "archivist" different from a random individual who scrapes stuff of the Internet and rehosts it? In the aggregate, the Internet Archive looks different from the typical person who is copying articles and blog posts, wrapping them up in ads, and displaying them. The IA is non-profit, doesn't run ads, etc. It also respects robots.txt. But it's not that clear to me what the legal regime would be that allows the Internet Archive to function free and clear and doesn't hit cases that most would agree are shady.

DanBC 12y ago

Is there an update to the (now ancient) W3C "Cool URIs Don't Change"? http://www.w3.org/Provider/Style/URI.html http://www.w3.org/Provider/Style/URI.html

dredmorbius 12y ago
From the article, the clear winner would be "cool URIs enable Wayback Machine archival".

pronoiac 12y ago

I hadn't realized I could write notes on Pinboard! I've written long descriptions for my bookmarks on occasion, though.

RexRollman 12y ago
I didn't either. What a nice feature. (BTW: Pinboard is one of my favorite one-person companies, the other being Tarsnap. I love that the both seem to be run be people interested in building something and are not just looking for an exit strategy.)
joshu 12y ago
Notes was on the todo list very early on for delicious, but never allowed to proceed. Glad to see it working.

rospaya 12y ago

I have around 4000 bookmarks from ~2003 to 2009 when I stopped hoarding them. Afraid to check how many of them will work.

vitovito 12y ago
It's a terrible problem. Maciej Ceglowski is doing his own study on link rot: https://blog.pinboard.in/2014/08/researching_link_rot/ https://blog.pinboard.in/2014/08/researching_link_rot/ He says my numbers match his own so far, about 5% per year. So I'd guess ~2200 bookmarks are dead, and maybe you could get ~1000 of them back from the Wayback Machine. I have 28MB of personal bookmark files I'll be post-processing, as well as ~47,000 links from a shared, private bookmarking service dating from 2005 through 2011. I'm not looking forward to it.
dredmorbius 12y ago
NB: wget's '--spider' argument can be used to test link integrity (though it only checks that the link returns 200, not that the content is intact).

purplerails 12y ago

Here's my take on a solution to the problem of linkrot: https://www.purplerails.com/ https://www.purplerails.com/ Major points: * Automatically saves an exact copy of pages (no need to explicitly bookmark) in the background. * Data is encrypted with your password before being sync'd to cloud. * Search through your pages. * Works as a Chrome browser extension. No need for a native app.

pain 12y ago
If I need to rely on your private domain to access my own research, how is it different or less risky than diigo etc? I'm waiting for an extension that lets me keep my own full data activity (and use your cloud too, optionally in addition).
- purplerails 12y ago
  Understood. Thx for the comment. The ability to download your data in a well-documented format (possibly WARC) is coming soon. I hope you will try out PurpleRails in the meanwhile. Thx again!
vitovito 12y ago
Hi! Same thing here as with the other examples mentioned in this thread: this only helps you. If you save a page but someone else needs it, they're out of luck. But, if, in addition to making you a private, encrypted archive, they also tested to see if the URL was publicly visible and, if so, made a WARC of it, then they could package up all those WARCs for donation to the Internet Archive, and everyone could benefit.
- purplerails 12y ago
  > If you save a page but someone else needs it, they're out of luck. There is a sharing feature to solve this problem. :) But I agree with your point. I actually looked into WARC earlier but didn't have the bandwidth to do it my first version. When I implement the ability to download your data, I'll try hard to use WARC. Unless there's some brain damage in the format: I hope not! :)

sugarfactory 12y ago

> Every URL saved in more than one place increases the likelihood that their content will survive as domains change owners. Surely it's obvious that the more copies the better for the backup purpose. Then, why is he advocating cloud services including the Wayback Machine? There's no doubt that it's important for everyone to save web pages locally to create as many copies as possible on the earth to prevent the disaster of the Library of Alexandria from happening again.

vitovito 12y ago
Because it does someone no good if you have a copy of an old page they're looking for, and they have no way to find you, and you have no way of providing it to them. Centralized services like libraries and archives provide that. You absolutely should save everything yourself. But you should also give copies to as many centralized services as possible. Lots of copies keeps stuff safe.
- sugarfactory 12y ago
  "Centralized" was not the word you were looking for. Decentralized systems can serve the searching functionality as proved by many (pure) P2P file-sharing protocols.

bigethan 12y ago

Funny, I used to work for DataRealm (impressed the OP capitalized it right). Wasn't there when they sold the serve.com domain, but it was a big deal for them. Everything internal used serve.com (including email addresses) and switching to Datarealm.com was a lot of changes. As a small business it's hard to say "cool URLs don't change" when you're just trying to get by.

slackpad 12y ago

I worry about this a lot. There's archive.org doing an amazing job but it seems like so much of our culture is getting lost because it's online. My box of Popular Science magazines from the 1950s is amazing - even the ads are interesting. The difficulty in archiving this stuff is worrying, as is the discovery experience for someone 60 or 70 years down the road. I can pick up a crusty magazine and flip through it, cutting across a whole bunch of topics and their surrounding context. There are lots of silos trying to save bits of it, but if this guy put a bunch of papers in a box in 1997 and it didn't get eaten by mold, then all of it would have been around to browse today.

vitovito 12y ago
Absolutely. We know how to archive paper pretty well. Your box of magazines will last another 20 years in your garage. Maybe twice that in a climate-controlled vault. But they're still made of cheap paper, and will eventually decompose, or your garage will flood, or bugs will eat them, or mold will be particularly bad one year. And, like anything physical, unless someone makes a pilgrimage to your garage (or that vault), no-one else will ever (re)discover them, or share them. That's the problem with physical things: they can be controlled, hoarded, kept secret, or destroyed. Born-digital artifacts, and digitized physical ones, can be much more discoverable than physical ones. They often can be repurposed and remixed much more easily. And they absolutely can be copied and distributed widely in ways that physical artifacts can't. Sticking to paper isn't the answer. Figuring out better ways to index, preserve and distribute digital artifacts is.
- _delirium 12y ago
  I agree that paper per se isn't the advantage, but the fact that something comes in a relatively unencumbered and self-contained unit does help a lot with digital archiving. That is so far still more common with paper on average. A library can build a more or less complete digital archive of a paper newspaper by simply subscribing to it and setting up a routine digitization process. Building a complete archive of an online news site is much harder: you have to figure out how to scrape it in full, and many sites actively try to stop you from doing so. As publications have discovered that selling access to their own archives (or "vault" as some call it) is profitable, they are even less likely to want free third-party archives to be available. The best-case for "born digital" archiving at the moment are probably publications that still produce a full PDF version of each "issue". For example a library could archive The New Inquiry by simply subscribing and downloading each month's subscribers-only PDF, rather than trying to scrape the website. A publication that put the full content in an RSS feed would also be fairly easy to "subscribe" to and archive. But most don't have an easy way of subscribing to full updates that can be archived and stored/viewed independently of the original site, and many are even kind of hostile to the idea. Journals are another problem. Paper journal subscriptions go into a library's permanent holdings, while digital journal subscriptions often give them cheaper per-issue prices, but in return it's more like renting access to the journal-controlled archive, which goes away if the library cancels the subscription. Most don't allow a library to mass-download the archive into the library's own digital holdings, though some do have some self-archival arrangements (e.g. a library might be allowed to self-archive issues that came out during their period of subscription, but not the whole back catalogue).

ibisum 12y ago

Bookmarks? I just print-to-PDF. Offline, permanent, safe. Bookmarks have never been useful.

onion2k 12y ago
A print of a page is not the same as a bookmark. Rather than a link to a point in time (which is what your print out is), a bookmark is a pointer to the latest version of a page. That has many benefits if the page is actively updated. The cost of that advantage is that it can fail completely if the page is taken down. Ideally a browser should cache a page when you bookmark it and optionally refresh the cache when you revisit. That way, if you get an error you could look back at the cached version instead. In fact, that'd be a damn useful browser plugin.
vitovito 12y ago
But that only helps you. That doesn't help anyone else who needs that old reference. Saving things in a way that those backups can be shared and distributed, like WARC files donated to the Internet Archive, can help ensure that when your hard disk crashes and your backups fail and you lose those PDFs, the things you wanted to save will still be out there.