crawler4j is a nice open source (Apache 2) Java web crawler. Multi-threaded. Built-in page delay (200ms default). Does HTTPS, headers, POST, cookies, follows r

crawler4j is a nice open source (Apache 2) Java web crawler. Multi-threaded. Built-in page delay (200ms default). Does HTTPS, headers, POST, cookies, follows robots.txt . I prefer it to nutch for small to medium sized jobs. https://code.google.com/p/crawler4j/ https://code.google.com/p/crawler4j/