That's interesting. I'd love to hear more about your problems with it. My email is in my profile, feel free to drop me a line. I deployed a small but moderatel

That's interesting. I'd love to hear more about your problems with it. My email is in my profile, feel free to drop me a line. I deployed a small but moderately complex crawler backed by a database with Scrapy. It had some custom pipelining as well that deduplicated information (based upon a hash) and exponentially backed-off if the page hadn't changed in a while.