8 ms·

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. Of course it's terrible when large sites

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. Of course it's terrible when large sites go offline and take vast swaths of the Internet with them, and we should continue to shame the ones that do it. At the same time, if something is really important to you, you shouldn't store it in the form of links to random third-party servers. One problem we need to solve as coders is giving people better tools for saving stuff. It's really hard right now to save a webpage (or worse, series of connected pages) with any confidence that you've captured everything you need to see it again if the original server disappears. A project that I think has struck a really good balance between permanence and retaining authors' control over their writing is the Archive of Our Own (AO3). A bunch of fanfic authors got tired of sites falling out from under them, and decided to implement their own system, along with sensible governance and a way to fund its ongoing operations. The only broken links I've ever seen to AO3 are ones where the author consciously decided to take the material offline.

rando289 12y ago
It always seemed like such a regression to me when browsers disabled entire page caches and only ever show the freshest content. Before, if a page served 404, I could easily see my local cached version from my previous visit. Now, I'm shit out of luck.
- walterbell 12y ago
  What is amazing is that: 1) Never in the history of civilization has local knowledge storage (disk) and local compute been so cheap 2) Never have we had a larger free software ecosystem or more hackers to deploy free software locally 3) Never have we had more evidence about the differences in civilizational freedom between local and central storage/compute What if backup/restore skills were taught alongside home economics? Is a home a place of both shelter & storage? Are we abandoning a set of possible futures because we want the "convenience" of "someone else" backing up our digital selves/souls? "For want of a nail the shoe was lost. For want of a shoe the horse was lost. For want of a horse the rider was lost. For want of a rider the battle was lost. For want of a battle the kingdom was lost. And all for the want of a horseshoe nail."
MetaCosm 12y ago
> Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. I recently converted all my bookmarks to saved copies of the pages in Evernote. This means I have full text search over everything I "bookmark" -- still have the links -- and never lose it regardless of what happens. In this process, I was shocked at the number of 404s I encountered from relatively recent (last 18 months) bookmarks. Evernote is a bit of a wreck, but so far the combination of decent copy of page, browser plugins, tags, search, accessibility (phone clients, web client, some native clients) have made it my best option.
- Springtime 12y ago
  Since 2008 or so I've been saving all pages that I would otherwise be bookmarking. Only in MHT format, but it's convenient. After reading a few quotes on linkrot years ago, and noticing many pages I'd come back to 404'ing I decided bookmarks are useless long term (or even at times short term).
- walterbell 12y ago
  Is this a feature built into Evernote? If not, what tools did you use for the conversion?
  - MetaCosm 12y ago
    Built in. "Clip This Page" or "Clip This Selection". It is a single keypress in FireFox for me.
oridecon 12y ago
Luckly if Stack Overflow goes down the one billion shady scrapper bots will have (at least) the most viewed questions.
jaimebuelta 12y ago
This is the only real solution (even if it's through an intermediate agent, which may have the same problem. Remember Google Reader). Of course, then the problem to keep that info in a good way (indexable, accessible, backed up, etc...) is another interesting problem. I'm afraid that most of the people are simply not aware of all of this (both on the content supplier or the content consumer) This is also somehow of a problem on printed books. There are tons of books that are out of print and cannot be easily found.
DanBC 12y ago
Many people just don't have the clue to make usable local archives. You're also proposing mass copyright infringement which - stupidly in this example - is not legal.
- sillysaurus3 12y ago
  Is saving a webpage to your local harddrive copyright infringement? The data is already on your local system when you view a webpage.
  - tedks 12y ago
    Copyright is a legal construct, not a technical construct. It doesn't matter where the data is. If a judge decides it's copyright infringement to save webpages, it will be. I can't imagine that the WSJ or any other paywalled institution wouldn't consider saving pages locally to be copyright infringement; how would they enforce a limit on article views?
    - idlewords 12y ago
      It doesn't matter what the WSJ thinks, it matters how the law sees this. Let's wait for one of the nice real HN lawyers to answer!
    - grkvlt 12y ago
      The same way they do now? I can't save articles if I can't view them.
      - c_c_c 12y ago
        You can view all wsj articles if you use google as referer
- anon4 12y ago
  If the system itself is federated the same way as the rest of the data, then it doesn't matter if it's legal or not. You can't make an omelette without breaking a few eggs and you can't fix the world without breaking a few laws.
sroerick 12y ago
Slightly related to this, the other day I tried to download every youtube video in my watch history. Turns out it's pretty much impossible. The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. I tried manually loading the watch history page, and was only able to get about 1000 out of ~8000 results. When I selected those 1000 results and tried to add them to a playlist, the interface crashed. Does anyone have a solution for this, or is my watch history just at the mercy of Google?
- lutusp 12y ago
  > The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. That may not be a bug. It might be a way to limit the traffic created by scrapers who submit requests and then download all the videos listed on the result page.
  - sroerick 12y ago
    Hrm, then why offer the feature? I'm kind of still a beginner when it comes to APIs, but it seems disingenuous to offer an API feature and then intentionally break it so that it can't be used.
    - lutusp 12y ago
      > Hrm, then why offer the feature? Because people can still get 20 results. I mean, 20 results is way better than no results. > I'm kind of still a beginner when it comes to APIs, but it seems disingenuous to offer an API feature and then intentionally break it so that it can't be used. It's not broken. It returns 20 results. Maybe Google decided that was enough of a hit on their database. One could also argue that most people wouldn't want more than 20 results on their small-screen Android device served by a slow connection.
      - sroerick 12y ago
        https://code.google.com/p/gdata-issues/issues/detail?can=2&start=0&num=100&q=&colspec=API%20ID%20Type%20Status%20Priority%20Stars%20Summary&groupby=&sort=&id=4642 https://code.google.com/p/gdata-issues/issues/detail?can=2&s... The number of results returned is random. Some people report seeing videos watched on computers but not locally. One person says it works. That's not a limit. That's broken.
sroerick 12y ago
My second reply, but, I think that this is really important. We only need to look at early film history to know how easy it is to lose massive parts of our history. Going back to old pages, I frequently get 404 results. For politically sensitive documents, the problem is much more widespread. I would like something that not only archives pages I visit, but also versions them and tracks changes. If there was a bookmarking tool that did this, you could easily have an opt-in feature that shared content. This type of system would be a huge boost to something like the wayback machine.
- lutusp 12y ago
  It clearly hasn't occurred to you that private collections evaporate over time, just like public ones. > I would like something that not only archives pages I visit, but also versions them and tracks changes. That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate several terabytes per week. > This type of system would be a huge boost to something like the wayback machine. Here's the road to madness. Someone, aware of the rapidly declining cost of storage, rebuilds the Wayback machine based on your scheme, with the intent of archiving every Web page in existence, including all required resources so the pages look just as they originally did. Then, as the project approaches completion, this genius says, "For the next phase, I need to archive the Wayback machine itself." At that point, as the implications of what he's said occur to him, a strange look crosses his face and his imagination begins writing checks his intellect can't cash.
  - sroerick 12y ago
    I never said I was trying to save my entire web cache locally. Just bookmarks. I don't have Flash Player installed. Instead, I use youtube-dl for flash videos across many websites. I'm well aware of the storage implications of this kind of activity.
    - idlewords 12y ago
      The storage implications are really not so bad. You would have trouble ever breaking 100 GB unless you are some kind of bookmarking mutant.
Terretta 12y ago
> I really think people should take steps to archive stuff that is important to them I understand why you didn't post a "self promoting" link, but I want others to know about this option: https://pinboard.in/tour/#archive https://pinboard.in/tour/#archive Currently $10 for a lifetime bookmarking account, plus $25/year to archive every bookmarked page: "Pinboard offers a bookmark archiving service for an annual fee of $25. The site will crawl and store a copy of every bookmark in your account, and display a special icon you can click to see the cached copy. If the page you bookmarked goes offline, you'll still be able to see the archived copy indefinitely."

rando289 12y ago

It always seemed like such a regression to me when browsers disabled entire page caches and only ever show the freshest content. Before, if a page served 404, I could easily see my local cached version from my previous visit. Now, I'm shit out of luck.

walterbell 12y ago
What is amazing is that: 1) Never in the history of civilization has local knowledge storage (disk) and local compute been so cheap 2) Never have we had a larger free software ecosystem or more hackers to deploy free software locally 3) Never have we had more evidence about the differences in civilizational freedom between local and central storage/compute What if backup/restore skills were taught alongside home economics? Is a home a place of both shelter & storage? Are we abandoning a set of possible futures because we want the "convenience" of "someone else" backing up our digital selves/souls? "For want of a nail the shoe was lost. For want of a shoe the horse was lost. For want of a horse the rider was lost. For want of a rider the battle was lost. For want of a battle the kingdom was lost. And all for the want of a horseshoe nail."

MetaCosm 12y ago

> Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. I recently converted all my bookmarks to saved copies of the pages in Evernote. This means I have full text search over everything I "bookmark" -- still have the links -- and never lose it regardless of what happens. In this process, I was shocked at the number of 404s I encountered from relatively recent (last 18 months) bookmarks. Evernote is a bit of a wreck, but so far the combination of decent copy of page, browser plugins, tags, search, accessibility (phone clients, web client, some native clients) have made it my best option.

Springtime 12y ago
Since 2008 or so I've been saving all pages that I would otherwise be bookmarking. Only in MHT format, but it's convenient. After reading a few quotes on linkrot years ago, and noticing many pages I'd come back to 404'ing I decided bookmarks are useless long term (or even at times short term).
walterbell 12y ago
Is this a feature built into Evernote? If not, what tools did you use for the conversion?
- MetaCosm 12y ago
  Built in. "Clip This Page" or "Clip This Selection". It is a single keypress in FireFox for me.

oridecon 12y ago

Luckly if Stack Overflow goes down the one billion shady scrapper bots will have (at least) the most viewed questions.

jaimebuelta 12y ago

This is the only real solution (even if it's through an intermediate agent, which may have the same problem. Remember Google Reader). Of course, then the problem to keep that info in a good way (indexable, accessible, backed up, etc...) is another interesting problem. I'm afraid that most of the people are simply not aware of all of this (both on the content supplier or the content consumer) This is also somehow of a problem on printed books. There are tons of books that are out of print and cannot be easily found.

DanBC 12y ago

Many people just don't have the clue to make usable local archives. You're also proposing mass copyright infringement which - stupidly in this example - is not legal.

sillysaurus3 12y ago
Is saving a webpage to your local harddrive copyright infringement? The data is already on your local system when you view a webpage.
- tedks 12y ago
  Copyright is a legal construct, not a technical construct. It doesn't matter where the data is. If a judge decides it's copyright infringement to save webpages, it will be. I can't imagine that the WSJ or any other paywalled institution wouldn't consider saving pages locally to be copyright infringement; how would they enforce a limit on article views?
  - idlewords 12y ago
    It doesn't matter what the WSJ thinks, it matters how the law sees this. Let's wait for one of the nice real HN lawyers to answer!
  - grkvlt 12y ago
    The same way they do now? I can't save articles if I can't view them.
    - c_c_c 12y ago
      You can view all wsj articles if you use google as referer
anon4 12y ago
If the system itself is federated the same way as the rest of the data, then it doesn't matter if it's legal or not. You can't make an omelette without breaking a few eggs and you can't fix the world without breaking a few laws.

sroerick 12y ago

Slightly related to this, the other day I tried to download every youtube video in my watch history. Turns out it's pretty much impossible. The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. I tried manually loading the watch history page, and was only able to get about 1000 out of ~8000 results. When I selected those 1000 results and tried to add them to a playlist, the interface crashed. Does anyone have a solution for this, or is my watch history just at the mercy of Google?

lutusp 12y ago
> The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. That may not be a bug. It might be a way to limit the traffic created by scrapers who submit requests and then download all the videos listed on the result page.
- sroerick 12y ago
  Hrm, then why offer the feature? I'm kind of still a beginner when it comes to APIs, but it seems disingenuous to offer an API feature and then intentionally break it so that it can't be used.
  - lutusp 12y ago
    > Hrm, then why offer the feature? Because people can still get 20 results. I mean, 20 results is way better than no results. > I'm kind of still a beginner when it comes to APIs, but it seems disingenuous to offer an API feature and then intentionally break it so that it can't be used. It's not broken. It returns 20 results. Maybe Google decided that was enough of a hit on their database. One could also argue that most people wouldn't want more than 20 results on their small-screen Android device served by a slow connection.
    - sroerick 12y ago
      https://code.google.com/p/gdata-issues/issues/detail?can=2&start=0&num=100&q=&colspec=API%20ID%20Type%20Status%20Priority%20Stars%20Summary&groupby=&sort=&id=4642 https://code.google.com/p/gdata-issues/issues/detail?can=2&s... The number of results returned is random. Some people report seeing videos watched on computers but not locally. One person says it works. That's not a limit. That's broken.

sroerick 12y ago

My second reply, but, I think that this is really important. We only need to look at early film history to know how easy it is to lose massive parts of our history. Going back to old pages, I frequently get 404 results. For politically sensitive documents, the problem is much more widespread. I would like something that not only archives pages I visit, but also versions them and tracks changes. If there was a bookmarking tool that did this, you could easily have an opt-in feature that shared content. This type of system would be a huge boost to something like the wayback machine.

lutusp 12y ago
It clearly hasn't occurred to you that private collections evaporate over time, just like public ones. > I would like something that not only archives pages I visit, but also versions them and tracks changes. That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate several terabytes per week. > This type of system would be a huge boost to something like the wayback machine. Here's the road to madness. Someone, aware of the rapidly declining cost of storage, rebuilds the Wayback machine based on your scheme, with the intent of archiving every Web page in existence, including all required resources so the pages look just as they originally did. Then, as the project approaches completion, this genius says, "For the next phase, I need to archive the Wayback machine itself." At that point, as the implications of what he's said occur to him, a strange look crosses his face and his imagination begins writing checks his intellect can't cash.
- sroerick 12y ago
  I never said I was trying to save my entire web cache locally. Just bookmarks. I don't have Flash Player installed. Instead, I use youtube-dl for flash videos across many websites. I'm well aware of the storage implications of this kind of activity.
  - idlewords 12y ago
    The storage implications are really not so bad. You would have trouble ever breaking 100 GB unless you are some kind of bookmarking mutant.

Terretta 12y ago

> I really think people should take steps to archive stuff that is important to them I understand why you didn't post a "self promoting" link, but I want others to know about this option: https://pinboard.in/tour/#archive https://pinboard.in/tour/#archive Currently $10 for a lifetime bookmarking account, plus $25/year to archive every bookmarked page: "Pinboard offers a bookmark archiving service for an annual fee of $25. The site will crawl and store a copy of every bookmark in your account, and display a special icon you can click to see the cached copy. If the page you bookmarked goes offline, you'll still be able to see the archived copy indefinitely."