2 ms·
FYI, I work on web archiving at archive.org. Our tools for crawling and creating your own Wayback machine, including search at smaller scales, are open source.
by gojomo 18y ago
FYI, I work on web archiving at archive.org.
Our tools for crawling and creating your own Wayback machine, including search at smaller scales, are open source. See the projects 'Heritrix', 'Wayback', and 'NutchWAX'. (Though, the bulk of our public archive still comes via Alexa's closed-source crawling.)
There's a company called Iterasi now offering personal archiving. A company called HanzoWeb at one point offered a del.icio.us-like bookmarking-plus-archiving service, and that might return.
CS professor Frank McCown, while a PhD student at Old Dominion, built a tool called 'Warrick' for reconstructing recently-disappeared websites from a combination of public sources.
So many options for collection and access are out there -- it's a matter of organization: building redundant stores and collecting the right materials at the right time.
- aristus 18y agoThis is good to hear! The more the merrier, and thank you for your work on archive.org. I agree that the organization and redundant stores need work, but the collection and access need help, too. If something is not in Google Cache or the Wayback, it's effectively gone. There may be a copy in a LOCKSS server or in a nearby browser cache, but if I can't know that or am not allowed to read it, it's useless.