Solr can serve great for your purposes. As far as domain specific crawl is concerned, maybe you could use our semantic classification API which takes in a URL a

Solr can serve great for your purposes. As far as domain specific crawl is concerned, maybe you could use our semantic classification API which takes in a URL and tells the DMOZ category it belongs to - http://www.wingify.com/contextsense/ http://www.wingify.com/contextsense/