3 ms·

gojomo, I have looked at the file formats. Could you propose some off-the-shelf web spiders? I would like to accumulate a lot of text for NLP research.

by bravura 18y ago

cnu 18y ago
Try out Nutch - http://lucene.apache.org/nutch/ http://lucene.apache.org/nutch/ or the one archive.org uses Heretix - http://crawler.archive.org/ http://crawler.archive.org/
gojomo 18y ago
At the Internet Archive we've created Heritrix for 'archival quality' crawling -- especially when you want to get every media type, and sites to complete/arbitrary depth, in large but polite crawls. (It's possible, but not usual for us, to configure it to only collect textual content.) The Nutch crawler is also reasonable for broad survey crawls. HTTrack is also reasonable for 'mirroring' large groups of sites to a filesystem directory tree.
- bravura 18y ago
  Could you outline how to configure it to collect only textual content?
  - gojomo 18y ago
    Very roughly: (1) Add a scope rule that throws out discovered URIs with popular non-textual extensions (.gif, .jpe?g, .mp3, etc.) before they are even queued. (2) Add a 'mid-fetch' rule to FetchHTTP module that early-cancels any fetches with unwanted MIME types. (These rules run after HTTP headers are available.) (3) add a processor rule to whatever is writing your content to disk (usually ARCWriterProcessor) that skips writing results (such as the early-cancelled non-textual results above) of unwanted MIME types. Followup questions should go to the Heritrix project discussion list, http://groups.yahoo.com/group/archive-crawler/ http://groups.yahoo.com/group/archive-crawler/ .
- bravura 18y ago
  Also, what do you mean by "broad survey crawls"?
  - gojomo 18y ago
    'Broad' crawl means roughly, "I'm interested in the whole web, let the crawler follow links everywhere." Even starting with a small seed set, such crawls quickly expand to touch hundreds of thousands or millions of sites. Even as a fairly large operation, you're might to be happy with a representative/well-linked set of 10 million, 100 million, 1 billion, etc. URLs -- which is only a subset of the whole web, hence a 'survey'. A constrasting kind of crawl would be to focus on some smaller set of sites/domains you want to crawl as deeply and completely as possible. You might invest weeks or many months to get these deeply in a gradual, polite manner.
brendano 18y ago
I don't work at Internet Archive, but I've found Heritrix is great. Crawling is all about crazy wonky edge cases and it deals with lots of them. Use the version 1.x series, not the new version 2. 1.x is less convoluted and easier to use. There might be some http://en.wikipedia.org/wiki/Second-system_effect http://en.wikipedia.org/wiki/Second-system_effect I think.

gojomo, I have looked at the file formats. Could you propose some off-the-shelf web spiders? I would like to accumulate a lot of text for NLP research.

by bravura 18y ago

gojomo, I have looked at the file formats. Could you propose some off-the-shelf web spiders? I would like to accumulate a lot of text for NLP research.

cnu 18y ago

Try out Nutch - http://lucene.apache.org/nutch/ http://lucene.apache.org/nutch/ or the one archive.org uses Heretix - http://crawler.archive.org/ http://crawler.archive.org/

gojomo 18y ago

At the Internet Archive we've created Heritrix for 'archival quality' crawling -- especially when you want to get every media type, and sites to complete/arbitrary depth, in large but polite crawls. (It's possible, but not usual for us, to configure it to only collect textual content.) The Nutch crawler is also reasonable for broad survey crawls. HTTrack is also reasonable for 'mirroring' large groups of sites to a filesystem directory tree.

bravura 18y ago
Could you outline how to configure it to collect only textual content?
- gojomo 18y ago
  Very roughly: (1) Add a scope rule that throws out discovered URIs with popular non-textual extensions (.gif, .jpe?g, .mp3, etc.) before they are even queued. (2) Add a 'mid-fetch' rule to FetchHTTP module that early-cancels any fetches with unwanted MIME types. (These rules run after HTTP headers are available.) (3) add a processor rule to whatever is writing your content to disk (usually ARCWriterProcessor) that skips writing results (such as the early-cancelled non-textual results above) of unwanted MIME types. Followup questions should go to the Heritrix project discussion list, http://groups.yahoo.com/group/archive-crawler/ http://groups.yahoo.com/group/archive-crawler/ .
bravura 18y ago
Also, what do you mean by "broad survey crawls"?
- gojomo 18y ago
  'Broad' crawl means roughly, "I'm interested in the whole web, let the crawler follow links everywhere." Even starting with a small seed set, such crawls quickly expand to touch hundreds of thousands or millions of sites. Even as a fairly large operation, you're might to be happy with a representative/well-linked set of 10 million, 100 million, 1 billion, etc. URLs -- which is only a subset of the whole web, hence a 'survey'. A constrasting kind of crawl would be to focus on some smaller set of sites/domains you want to crawl as deeply and completely as possible. You might invest weeks or many months to get these deeply in a gradual, polite manner.

brendano 18y ago

I don't work at Internet Archive, but I've found Heritrix is great. Crawling is all about crazy wonky edge cases and it deals with lots of them. Use the version 1.x series, not the new version 2. 1.x is less convoluted and easier to use. There might be some http://en.wikipedia.org/wiki/Second-system_effect http://en.wikipedia.org/wiki/Second-system_effect I think.