>search ranking is a complicated and subtle problem Absolutely ! I think never before the saying "the devil is in the details"... is more appropriate than here

>search ranking is a complicated and subtle problem Absolutely ! I think never before the saying "the devil is in the details"... is more appropriate than here. Sure most users agree on the extreme's the REALLY bad search result (putting in 'apple' getting out LCD TV's [maybe the tv's are in parent-category 'Electronics' and you have category and popularity boost to high' ?]) and the REALLY good (results that you expected) Search Results are commonly evaluated with Recall(did ALL the documents that are relevant got returned) vs Precision (how many of the results are 'correct') But that is "one" of many-many metrics. The biggest issues are non-tech ppl (like your boss or manager) walking in and "discussing" his/her pet-peeve-search query, cause in his mind when he put in APPLE iPhone we should be returning ONLY apple-iphones and NOT apple-iPhone accessories) or maybe we should be returning ONLY the "latest iphone" not the model from 2 generations back I've commented this before, you can usually only shoot for an "average amount of happiness" (sounds like Arthur Schopenhauer ? :P) for most users. Never "ok ppl, search works perfect" for everyone one now As to the "practical matters", we found that building a "search-test-suite" where you put in the "manager's pet-peeve" as well as any angry-emails about search queries, and whenever do you search-tuning it's easy to see any-query-regression oh and of course this needs to be an automated search-test-suite.