3 ms·
The Archive Team have something called the Archive Team Warrior (http://archiveteam.org/index.php?title=ArchiveTeam_Warrior http://archiveteam.org/index.php?tit
by ris 12y ago
The Archive Team have something called the Archive Team Warrior (http://archiveteam.org/index.php?title=ArchiveTeam_Warrior http://archiveteam.org/index.php?title=ArchiveTeam_Warrior) which seems quite similar to this except it's a virtual rather than physical appliance. Worth a look.
(edit: although looking further this is more for distributed crawling, while the results will be uploaded to their archive)
- sp332 12y agoThe warriors download items assigned by a central tracker, then upload each item to a holding area. The warriors don't keep items around when they're finished. Also, the current tracker keeps everything in RAM, and can't keep up with more than ~2 million items at once. It would take a major rewrite just to keep track of all the pieces. P.S. Please run a warrior instance. It comes in a small VM image that you can run in the background all the time.