3 ms·
It's an invaluable resource. In the past week I realized it is quite likely that a site I used to work at which hosts a lot of my portfolio might disappear or b
by Hard_Space 8y ago
It's an invaluable resource. In the past week I realized it is quite likely that a site I used to work at which hosts a lot of my portfolio might disappear or be heavily amended. So I located and indexed a complete list of my articles there -- and I was even able to click a button and create an archive for the few pages that IA hadn't bothered to index (I wish I had known about this before, since it was a shock to find that IA can be quite selective, and to find a page you were hoping was there simply isn't, and is now irretrievable).
But as one other commenter here has mentioned, you're only a robots.txt amend away from the oblivion that the entire IMDB comments section fell into [1], so a good archiving system is essential. I use (no affiliation)Save Page WE on Waterfox:
https://addons.mozilla.org/en-US/firefox/addon/save-page-we/ https://addons.mozilla.org/en-US/firefox/addon/save-page-we/
[1] https://news.ycombinator.com/item?id=13571893 https://news.ycombinator.com/item?id=13571893
- zmw 8y agoWayback Machine isn’t Googlebot, it doesn’t crawl the web, so there’s no such thing as “hadn’t bother to index”... Someone, be it a human or a bot, needs to submit a page for archival. Programmatically submitting to Wayback Machine is trivial enough, so I have cron jobs backing up most of my static sites (in their entirety) periodically.
- duckington 8y ago"bothered to index" and "IA can be quite selective" implies that there's a human overseer to the Wayback operation, not true! Pages are archived automatically, and sites are crawled by robots, not humans.