4 ms·

How you do RAG with embeddings NOT in English? I mean there are a few thousand more languages.

• dbish 3y ago
  Just use your embeddings model of choice that works with your language, I believe Ada from openai is multilingual but I don’t know what languages it works well on, there are many embedding models out there, huggingface is your friend in this search. The output is just a vector and the rest of the system can basically stay the same. The only other thing that may need to change depending on language is any text preprocessing you need to do like word or sentence breaking for languages with compound words (German), agglutination (Turkish), etc.
  • noiv 3y ago
    Well, thanks, how do they deal with technical terms never seen during training?
    • dbish 3y ago
      Unknown words can’t really be handled other than mapping to a placeholder “unknown” vector in most cases. This can be dealt with in a variety of ways, text preprocessing being one of them
• htrp 3y ago
  Embeddings are not just english based ....