4 ms·
Very timely and interesting. I am currently looking for a crawler that tightly integrated with Drupal and that can be easily managed through Drupal nodes. Any s
by arkitaip 16y ago
Very timely and interesting. I am currently looking for a crawler that tightly integrated with Drupal and that can be easily managed through Drupal nodes. Any suggestions on a solution for a small site that only needs to handle thousands of pages/urls?
- rb2k_ 16y agoI don't really know what the "managed through Drupal nodes" means in this context. For a simple drupal fulltext search I can recommend apache solr ( http://drupal.org/project/apachesolr http://drupal.org/project/apachesolr ). For regular crawling: I found anemone ( http://anemone.rubyforge.org/ http://anemone.rubyforge.org/ ) to be a lovely framework for single page crawls. Other interesting candidates: https://github.com/hasmanydevelopers/RDaneel https://github.com/hasmanydevelopers/RDaneel http://www.redaelli.org/matteo-blog/projects/ebot/ http://www.redaelli.org/matteo-blog/projects/ebot/ http://nutch.apache.org/ http://nutch.apache.org/ (meh, java)
- toumhi 16y agoscrapy (http://scrapy.org/ http://scrapy.org/) is a well-documented and open source python scraping framework that I've used in a couple of projects.
- rb2k_ 16y agoIndeed, seems like a great framework. Considering the timespan of the project, I had to rely on something I'm pretty ok at (Ruby), but I remember hitting a lot of posts about scrapy on the way