4 ms·
Don't write your own crawler. Use nutch. It is designed to scale and do mapreduce kind of parallel processing. I would strongly recommend you to take a look be
by mig 19y ago
Don't write your own crawler. Use nutch.
It is designed to scale and do mapreduce kind of parallel processing. I would strongly recommend you to take a look before writing your own.
http://lucene.apache.org/nutch/ http://lucene.apache.org/nutch/
- imsteve 19y agomapreduce? Just how many requests will you be making to third-party sites at once? Sounds like a good way to get blocked fast.