3 ms·

I'm "hoarding" web sites with technical information. I've got TBs of disk where I mirror things. Because I started to notice that things just.. disappear. And t

by Tor3 8y ago

I'm "hoarding" web sites with technical information. I've got TBs of disk where I mirror things. Because I started to notice that things just.. disappear. And they're not on archive.org, or their mirror is very incomplete. I'll never need 99.99% of the stuff that I copy. But just yesterday somebody on a forum looked for information and mentioned a web site that apparently had that info at one point.. but had disappeared. Completely. Except that I had a copy and could hand it over.

jbverschoor 8y ago
How do you easily mirror? I tend to use pocket for articles, but even that will not last. I'm also saving some youtubes these days.
- Tor3 8y ago
  wget will do the job most of the time. At least for the sites I copy - it's mostly directories with files (documents etc).
drilldrive 8y ago
You should set up your own archive website with plans for long term (post-mortum) servers.
adrianN 8y ago
Why don't you contribute to archive.org instead?
- Tor3 8y ago
  It's not something I could do instead - it's something I would have to do in addition. And there's some problems with that - at the time I'm copying/mirroring, the site is out there on the net. Uploading (as in direct uploading) to archive.org of sites that exists would look a bit strange, I think. But if you mean to get archive.org to mirror instead - yes, I do that too, but the thing is, you can't get archive.org to recursively mirror anything. You can point save.archive.org (IIRC - they change the URL sometimes) to a not-yet archived site (or one which needs an update), and archive.org will copy that page. You'll have to click-and-walk through every page if you want archive.org to get a full mirror. And I have done that too, with a few sites. But that's a lot of time-consuming work. Just quickly mirroring a site that pops up during a technical discussion takes all of 30 seconds to set up.
- jodrellblank 8y ago
  I know this is dated, but if a website changes the current robots.txt policy, Archive.org will delete all the archives in response to it and not crawl it again https://www.webmasterworld.com/robots_txt/4898080.htm https://www.webmasterworld.com/robots_txt/4898080.htm

by Tor3 8y ago

jbverschoor 8y ago

How do you easily mirror? I tend to use pocket for articles, but even that will not last. I'm also saving some youtubes these days.

Tor3 8y ago
wget will do the job most of the time. At least for the sites I copy - it's mostly directories with files (documents etc).

drilldrive 8y ago

You should set up your own archive website with plans for long term (post-mortum) servers.

adrianN 8y ago

Why don't you contribute to archive.org instead?

Tor3 8y ago
It's not something I could do instead - it's something I would have to do in addition. And there's some problems with that - at the time I'm copying/mirroring, the site is out there on the net. Uploading (as in direct uploading) to archive.org of sites that exists would look a bit strange, I think. But if you mean to get archive.org to mirror instead - yes, I do that too, but the thing is, you can't get archive.org to recursively mirror anything. You can point save.archive.org (IIRC - they change the URL sometimes) to a not-yet archived site (or one which needs an update), and archive.org will copy that page. You'll have to click-and-walk through every page if you want archive.org to get a full mirror. And I have done that too, with a few sites. But that's a lot of time-consuming work. Just quickly mirroring a site that pops up during a technical discussion takes all of 30 seconds to set up.
jodrellblank 8y ago
I know this is dated, but if a website changes the current robots.txt policy, Archive.org will delete all the archives in response to it and not crawl it again https://www.webmasterworld.com/robots_txt/4898080.htm https://www.webmasterworld.com/robots_txt/4898080.htm