4 ms·
I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems?
by JacobiX 3y ago
I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems?
For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?
- jxmorris12 3y agoGood embeddings are supposed to handle this natively. For example the embedding for API should be close to the embedding for Application Programming Interface. Purportedly.
- vl 3y agoAFAIK, modern tokenizers/vocabs don’t have OOV anymore because they use sub-word tokens if there is no entry for the given word. This also works for languages with composite words. In your example sum of embeddings for non-typo and typo version should have close cosine distance if it trained correctly.