3 ms·
Yes. Thank you for pointing this out. I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"... I'm n
by hnfong 1mo ago
Yes. Thank you for pointing this out.
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
- usernametaken29 1mo agoI am a linguist and can say your view is too simplistic. This works only for things that are common enough to be mapped onto the same vector from the training set. Try searching for my company XY and it won’t work, because your embedding doesn’t have an adequate embedding, so you will end up having to build a translation table anyways..