I don't really know what the "managed through Drupal nodes" means in this context. For a simple drupal fulltext search I can recommend apache solr ( http://drup

I don't really know what the "managed through Drupal nodes" means in this context. For a simple drupal fulltext search I can recommend apache solr ( http://drupal.org/project/apachesolr http://drupal.org/project/apachesolr ). For regular crawling: I found anemone ( http://anemone.rubyforge.org/ http://anemone.rubyforge.org/ ) to be a lovely framework for single page crawls. Other interesting candidates: https://github.com/hasmanydevelopers/RDaneel https://github.com/hasmanydevelopers/RDaneel http://www.redaelli.org/matteo-blog/projects/ebot/ http://www.redaelli.org/matteo-blog/projects/ebot/ http://nutch.apache.org/ http://nutch.apache.org/ (meh, java)