3 ms·
Calculating the embeddings is probably going to be an application-specific thing. Either your application has reasonable pre-trained encoders or you train one o
by mattnewton 3y ago
Calculating the embeddings is probably going to be an application-specific thing. Either your application has reasonable pre-trained encoders or you train one off a mountain of matching pairs of data.
Once you have the embeddings in some space, for PoC I’ve mostly seen people shove them into faiss, which handles most of the rest very well for small/medium datasets:
https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss
- NhanH 3y agoCould you please point to some materials to understand the data needed to train the embedding model for a specific domain?
- anon7725 3y agoYou don’t need to train anything if you just need embeddings. The data is text. You apply the pretrained model to your text and it returns the embedding. You save it in a vector database if you’re fancy, or a big numpy array if you’re like me. Then run your similarity search (cosine, Euclidean, etc).
- famouswaffles 3y agoa lot of embedding models have poor performance on domain specific data that is only mitigated with finetuning. alternately the instructor series mitigates this by fine-tuning the model on instructions and giving specific instructions to targeted domains.