3 ms·
As you get into this, you’ll want to improve the tokenization to something less naive. (The naive approach is fine for the educational purpose of the article.)
by mwsherman 6y ago
As you get into this, you’ll want to improve the tokenization to something less naive. (The naive approach is fine for the educational purpose of the article.)
I’ve made the mistake of simply splitting on spaces or punctuation, and eventually finding that will do the wrong thing.
Eventually I learned about Unicode text segmentation[0], which solves this pretty well for many languages. I believe the default Lucene tokenizer uses it. I implemented it for Go[1].
[0] https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
[1] https://github.com/clipperhouse/uax29 https://github.com/clipperhouse/uax29
- rawoke083600 6y agoYea agreed, as to my tokenization example. Back when I was doing search for a price comparison service. The model numbers of the products were sometimes entered as LGA775 or LGA-775 or LGA 77-5 or LGA—775 (unicode dash) or "LGA 775" And that is just one example :) I ended up by developing a suite of "search tests" so whenever we were doing tuning (daily basis) we had a set of automatic search tests of about 80 weird/prev-wrong search cases we could check in a blink of an eye. One could specify the top X result needed to include at least Y number of products in XYZ category or have "brand" as ABC etc.. One might say it was "test-driven-search-tuning"