3 ms·

Could you outline how to configure it to collect only textual content?

• gojomo 18y ago
  Very roughly: (1) Add a scope rule that throws out discovered URIs with popular non-textual extensions (.gif, .jpe?g, .mp3, etc.) before they are even queued. (2) Add a 'mid-fetch' rule to FetchHTTP module that early-cancels any fetches with unwanted MIME types. (These rules run after HTTP headers are available.) (3) add a processor rule to whatever is writing your content to disk (usually ARCWriterProcessor) that skips writing results (such as the early-cancelled non-textual results above) of unwanted MIME types. Followup questions should go to the Heritrix project discussion list, http://groups.yahoo.com/group/archive-crawler/ http://groups.yahoo.com/group/archive-crawler/ .