4 ms·
Notably tokenization for traditional search. LLMs use very different tokenization with very different goals
by wongarsu 11mo ago
Notably tokenization for traditional search. LLMs use very different tokenization with very different goals
- tgv 11mo agoIt's a rather old-fashioned style of tokenization. In the 1980s this was common, I think. But, as noted in another comment, it doesn't work that well for languages with a richer morphology, or compounding. It's a very "English" approach.
- jamesgresql 11mo agoChinese, Japanese, Korean etc.. don’t work like this either. However, even though the approach is “old fashioned” it’s still widely used for English. I’m not sure there is a universal approach that semantic search could use that would be both fast and accurate? At the end of the day people choose a tokenizer that matches their language. I will update the article to make all this clearer though!
- matusp 11mo agoThis was common even in 2015. You can still see people removing stop words from text, even when they feed it to LLMs. It's of course terrible for performance, but old habits die hard I guess.
- jamesgresql 11mo ago100%, maybe we should do a follow up on other types of tokenization.