I've had good results combining results of SOLR and sphinx, using each with and without different numbers stopwords (50-200), i.e. 4 queries through 2 indexes;

I've had good results combining results of SOLR and sphinx, using each with and without different numbers stopwords (50-200), i.e. 4 queries through 2 indexes; might want to look into Xapian as well You end up spending a lot of time evaluating combinations of stopwords, stemmers, token separators, UTF-8 to ISO latin conversions, etc. __________________________________________ Here's some things about evaluating quality of your search hits, e.g. precision vs. recall: http://www.alistapart.com/articles/testing-search-for-relevancy-and-precision/ http://www.alistapart.com/articles/testing-search-for-releva... http://www.alistapart.com/articles/internal-site-search-analysis-simple-effective-life-altering/ http://www.alistapart.com/articles/internal-site-search-anal... mean reciprocal rank (MRR), average precision (MAP) , precision at 10 http://www.computationalmedicine.org/challenge/cmcChallengeDetails.pdf http://www.computationalmedicine.org/challenge/cmcChallengeD... ______________________________ The social media (reddit, digg, Y-C, delicious, tweets, facebook, stumbles, stackoverflow) can bootstrap you pretty far. Also look at Mixx, Y Buzz, etc. For setting domain bounds on your crawl: you can assemble a list of most-frequently tagged domains from e.g. delicious or subreddits, This works well for generating google custom search engines (email w/ questions/comments)