2 ms·
Thanks for the previous positive comments about Heritrix, which is my project at the Internet Archive. If anyone has questions, please send them my way. Heritr
by gojomo 19y ago
Thanks for the previous positive comments about Heritrix, which is my project at the Internet Archive. If anyone has questions, please send them my way.
Heritrix was designed for archival projects, which has meant an emphasis on having a "true record" (including non-text resources) and high configurability for inclusion/exclusion. Any text indexing or link-graph-analysis is completely external; we've used Nutch (without their crawler) for that.
Whole-web multi-billion-page crawls have not been the focus yet, though we've tried one and have heard of outside groups successfully using Heritrix for 2+ and 4+ billion page crawls.
Our distribution story is spotty; we provide some options that help you split the URL-space you want to crawl across crawlers, and remote-control crawlers from other programs, but syncing their launch and other steps is left to an expert operator's own devices. We've run coordinated Heritrix crawls on groups of 4-8 machines (dual-opteron, 4G+ RAM, 4x500GB+ HDs) and understand others have used up to 12.