3 ms·

After playing around with it for a few minutes, all of the results scored between 0.5 and 0.8 even when using nonsense queries like "interdimensional cable" and

After playing around with it for a few minutes, all of the results scored between 0.5 and 0.8 even when using nonsense queries like "interdimensional cable" and "eat my plumbus" which is a sign that the model you're using for embeddings is very poorly tuned for cosine similarity for your use case. A little fine tuning would probably go a long way since the embeddings are likely trained mostly on a nonreligious corpus in the modern tongue. It might also be overfitted so trying smaller models might also help.

j-b 3y ago
Thanks for the feedback - this particular model is BAAI/bge-large-en-v1.5. What alternative embeddings would you recommend?
- civilitty 3y ago
  Sadly I can't help much here. In my experience you have to create test cases and experiment with different models (a LOT). Some general thoughts in no specific order: - InstructorXL is my favorite embedding model because it has a two part input, sort of like a system prompt, that you can use to qualify the user input without modifying it yourself. You can experiment with using different system prompts on the initial embeddings and the user prompt. You can also use a bunch of different system prompts and weigh their scores, average them, add them, etc. - You can start with qualitative test cases like the obvious Leviticus prohibitions and see what the range of scores are like before you create automated test cases and evaluations. Find one of those bibles where one side is the original King James translation and the other side is in modern English to use for more complex pairings. - If that doesn't lead to an obvious winner, you may need to create a dataset for fine tuning. Make sure the dataset includes lots of negative examples too - cosine similarity scores should have a range of -1 to 1 to be most useful. Maybe take some important verses and change them to the opposite meaning ("thou shall not" => "thou shall") to create those negatives. Split your fine tuning data set into different categories so you can experiment with different combinations (i.e. the aforementioned autogenerated opposite pairs might really hurt the fine tuning because they're too similar). - You can probably fine tune it using a completely synthetic dataset using GPT-3.5/4 to do all the work. It's "aware" of the concept of vector embeddings and the training data format so it can create positive and negative pairs for you based on your instructions. You can probably find some sort of ranking of the most important passages (say most quoted or something) and feed those to a prompt to generate tons of pairs quickly.
rolisz 3y ago
That's a known issue with the BGE embeddings, the authors warn about that in the model card. Their recommendation is to choose more carefully the thresholds for similarity (which will be much higher than for other embeddings)
- civilitty 3y ago
  I've experienced the same issue with OpenAI and InstructorXL embeddings in certain cases so I think it's a rather common failure mode for embedding models.

j-b 3y ago

Thanks for the feedback - this particular model is BAAI/bge-large-en-v1.5. What alternative embeddings would you recommend?

civilitty 3y ago
Sadly I can't help much here. In my experience you have to create test cases and experiment with different models (a LOT). Some general thoughts in no specific order: - InstructorXL is my favorite embedding model because it has a two part input, sort of like a system prompt, that you can use to qualify the user input without modifying it yourself. You can experiment with using different system prompts on the initial embeddings and the user prompt. You can also use a bunch of different system prompts and weigh their scores, average them, add them, etc. - You can start with qualitative test cases like the obvious Leviticus prohibitions and see what the range of scores are like before you create automated test cases and evaluations. Find one of those bibles where one side is the original King James translation and the other side is in modern English to use for more complex pairings. - If that doesn't lead to an obvious winner, you may need to create a dataset for fine tuning. Make sure the dataset includes lots of negative examples too - cosine similarity scores should have a range of -1 to 1 to be most useful. Maybe take some important verses and change them to the opposite meaning ("thou shall not" => "thou shall") to create those negatives. Split your fine tuning data set into different categories so you can experiment with different combinations (i.e. the aforementioned autogenerated opposite pairs might really hurt the fine tuning because they're too similar). - You can probably fine tune it using a completely synthetic dataset using GPT-3.5/4 to do all the work. It's "aware" of the concept of vector embeddings and the training data format so it can create positive and negative pairs for you based on your instructions. You can probably find some sort of ranking of the most important passages (say most quoted or something) and feed those to a prompt to generate tons of pairs quickly.

rolisz 3y ago

That's a known issue with the BGE embeddings, the authors warn about that in the model card. Their recommendation is to choose more carefully the thresholds for similarity (which will be much higher than for other embeddings)

civilitty 3y ago
I've experienced the same issue with OpenAI and InstructorXL embeddings in certain cases so I think it's a rather common failure mode for embedding models.