Yea agreed, as to my tokenization example. Back when I was doing search for a price comparison service. The model numbers of the products were sometimes entered

Yea agreed, as to my tokenization example. Back when I was doing search for a price comparison service. The model numbers of the products were sometimes entered as LGA775 or LGA-775 or LGA 77-5 or LGA—775 (unicode dash) or "LGA 775" And that is just one example :) I ended up by developing a suite of "search tests" so whenever we were doing tuning (daily basis) we had a set of automatic search tests of about 80 weird/prev-wrong search cases we could check in a blink of an eye. One could specify the top X result needed to include at least Y number of products in XYZ category or have "brand" as ABC etc.. One might say it was "test-driven-search-tuning"