I would think you could improve your embedding space to address that issue, partially. Similarity search (as a result of some contrastive loss) definitely suff

I would think you could improve your embedding space to address that issue, partially. Similarity search (as a result of some contrastive loss) definitely suffers at the tails and the OOD is pretty bad. That being said, you're more likely to have higher recall than a more classical technique.