6 ms·
SentenceTransformers: Python framework for sentence, text and image embeddings
- nmstoker 3y agoShould probably have "(2019)" appended to the title as per HN conventions, given the date in the citation paper links (1x 2019 and 2x 2020) and the fact that this site has been around for quite a few years...
- zwaps 3y agoIt should be mentioned that the original author Nils Reimers has moved on and the repo has been stale since 2021/2022. It has recently (e.g. end of 2023) gotten a new team and has since seen updates. This is obviously quite significant given how important sentence embedding models are.
- PaulHoule 3y agoI use these all the time. For many of the classification tasks I do it works very well to pass an image or text through an embedding and then apply some kind of classical machine learning like the SVM. Model training is super reliable and takes maybe 3 minutes to train multiple models and cross-validate. In maybe 45 minutes I can train a single fine-tuned model but the results are really hit and miss.
- spxneo 3y agowould love to try this out myself. seems like there are lighter solutions than relying a giant LLM
- gillesjacobs 3y agoUsed this library quite a bit. I still have no idea if there is a good reason this API is not packaged within huggingface/transformers. Probably historic reasons, anyway solid 9/10 API.
- anon373839 3y agoThese are extremely useful embedding models, and some are small enough to use in the frontend (with transformers.js) for on-device semantic search. One issue I’ve run into is that they produce a very good ranking, but the actual cosine similarity scores appear to be meaningless. (E.g., “chocolate chip cookies” and “PLS6;YJBXSRF&/” could have a similarity of 0.8.) Consequently, I’ve had a hard time selecting sensible cutoff values to balance recall and precision. Has anyone found a good approach to this?
- ilaksh 3y agomaybe the normalize_embeddings flag on encode?
- Clueed 3y agoNot sure about the specific implementation here but the very definition of cosine similarity includes normalization. [0] [0] https://en.wikipedia.org/wiki/Cosine_similarity https://en.wikipedia.org/wiki/Cosine_similarity
- eigenvalue 3y agoYes, check out my library for vector similarity that has various other measures which are more discriminative: https://github.com/Dicklesworthstone/fast_vector_similarity https://github.com/Dicklesworthstone/fast_vector_similarity pip install fast_vector_similarity
- refibrillator 3y agoEven if you choose a more sophisticated similarity measure as suggested by another commenter, you’ll still need to set a threshold on that metric to perform binary classification. In my experience there are two paths forward, the one I recommend is to train an MLP classifier on the embeddings to produce a binary classification (ie similar or not). The advantage is that you no longer need to set a numeric threshold on a distance metric, however you will need labeled training data to define what is “similar” in the context of your use case. The other path is to calculate the statistics of pair-wise distances for every record in some unlabeled dataset you have, and use the resulting distribution to inform choice of threshold. This will at least give you an estimate of what % of records will be classified as similar for a given threshold value.
- mapmeld 3y agoMajor kudos to this library for supporting Matryoshka/nested/adaptive embeddings (https://huggingface.co/blog/matryoshka https://huggingface.co/blog/matryoshka), which I needed to train a model recently
- sroussey 3y agoIs there an open source Matryoshka embedding model?
- edshiro 3y agoI don't have much experience with embeddings... Could someone more knowledgeable suggest when it would make sense to use the SentenceTransformers library vs for instance relying on the OpenAI API to get embeddings for a sentence?
- deleted 3y ago[deleted]
- rolisz 3y agoUp until a month ago, the OpenAI embeddings where very poor. But they recently released a new model which is much better then they're previous one. Now it depends un specific usecase (domain, language, length of texts)
- montebicyclelo 3y agoIt's fairly easy to use, not that compute intensive (e.g. can run on even a small-ish CPU VM), the embeddings tend to perform well and you can avoid sending your data to a third party. Also, there are models fine tuned for particular domains on HF-hub, that can potentially give better embeddings for content in that domain.
- edshiro 3y agoI see - thanks for the clarifications I presume if your customers are enterprise companies then you may opt to use this library vs sending their data to OpenAI etc. And you can get more customisation/fine-tuning from this library too.
- estreeper 3y agoJust to add to this, a great resource is the Massive Text Embedding Benchmark (MTEB) leaderboard which you can use to find good models to evaluate, and there are many open models that outperform i.e. OpenAI's text-embedding-ada-002, currently ranked #46 for retrieval, which you can use with SentenceTransformers. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
- 3y ago
- marban 3y agoAlso to be used with https://maartengr.github.io/BERTopic/index.html https://maartengr.github.io/BERTopic/index.html
- deepsdev 3y agoI have used this library for a few years and is reliable. As someone mentioned, sometimes 2 things that are not related can have the same cosine similarity. Easy to use and get started with.
- estreeper 3y agoI'm curious how people are handling multi-lingual embeddings. I've found LASER[1] which originally had the idea to embed all languages in the same vector space, though it's a bit harder to use than models available through SentenceTransformers. LASER2 stuck with this approach, but LASER3 switched to language-specific models. However, I haven't found benchmarks for these models, and they were released about 2 years ago. Another alternative would be to translate everything before embedding, which would introduce some amount of error, though maybe it wouldn't be significant. 1. https://github.com/facebookresearch/LASER https://github.com/facebookresearch/LASER
- VHRanger 3y agoThe transformer models handle multilingual directly. For good old embedding models (eg. GLoVe), you have a few choices: 1. LASER as you mentioned. The performance tends to suck though. 2. Language prediction + one embedding model per supported language. Libraries like whichlang make this nice, and MUSE has embedding modes aligned per language for 100ish languages. Fastembed is a good library for this. Note that for most people, 32 dimension glove is all they need if you benchmark it. As the length of the text you're embedding goes up, or as the specificity goes up (eg. You have only medical documents and want difference between them) you'll need richer embeddings (more dimensions, or a transformer model, or both) People never benchmark their embeddings and I find it incredible how they end up with needlessly overenginneered systems.
- gregw134 3y agoAny idea which model has the best performance across languages? I'm checking out model performance on the Huggingface leaderboard and the top models for English aren't even in the top 20 for Polish, French and Chinese
- VHRanger 3y agoDepends on what your usecase is. For the normal user that just wants something across languages, the minilm-paraphrase-multilingual in the OP library is great. Of you want better than that (either bigger model, or specifically for a subset of languages, etc.) then you need to think about your task, priorities, target languages, etc.
- andai 3y agoHas anyone used FlagEmbedding? I'm testing a model that comes with examples for both SentenceTransformers and FlagEmbedding, but it's hard to find any information about it.
- riku_iki 3y agoit doesn't look they fine-tune those models on any of modern foundational models, which likely produces huge performance gap compared to OpenAI embedding for example..
- andai 3y agoWhat is everyone using embeddings for, and which mdels? I built a RAG last year (for searching long documents) but found it a bit disappointing. I tested it with OpenAI and with SentenceTransformers (instructor-xl or a smaller variant). Apparently they've come a long way since then though. Currently I'm working on an old fashioned search engine (tf-if + keyword expansion). Apparently that can work better than vector databases in some cases: https://news.ycombinator.com/item?id=38703943 https://news.ycombinator.com/item?id=38703943
- itissid 3y agoI recall paragraph2Vec. It was the earliest way to experiment with position embeddings that I recall reading. At first when I read it, it felt kind of crazy/weird that it worked: You feed in the paragraph ID(yes just an integer of the para where the word was) via its own layer along with the word to CBOW/SkipGram set up. Then throw away that part after training. Then, during inference, you attach a new randmly initialized layer for where that old layer was and "re-train" just that part before generating the embedding for words.
- Der_Einzige 3y ago>tfw it's 2024 and people STILL aren't using span compression to implement a "medium term" memory (RAG is long term and the context length is short term) for LLMs. >tfw it's 2024 and we just accept that the context "falls out" of the model if we push it beyond it's regular context length So everyone forgot that we can put large N number of tokens into small N number of embeddings because???
- register 3y agoWhat do you mean by span compression? We have experimented with various embedding contes lengths and we have found that bigger embeddings aren't the ones providing the best recall. We have hit the best results with something between 65% to 75% of the maximum embedding context length. We have been using OpenAI embeddings models though.
- spxneo 3y agohow are people using this with/without LLM? is the feature more of "reliable and accurate without hallucination" than LLM? where are you using it? text interface?
- gregw134 3y agoHow are you guys deciding what parts of a document to turn into embeddings? I've heard paragraph embeddings aren't that reliable, so I'm planning on using tf-idf first to extract keywords from a document, and then just create embeddings from those keywords.
- nestorD 3y agoTake your document, cut it (cleanly!) into pieces small enough to fit into your sentence embedder's context window, and generate several embeddings that all point to the same document. I would recommend against merging (averaging, etc.) the embeddings (unless you want a blurry idea of what your document contains), as well as feeding very large pieces of text to the embedder (some models have massive context lengths, but the result is similarly vague).
- gregw134 3y ago> [recommend against] feeding very large pieces of text to the embedder Sounds right, I've heard this from multiple sources. That's why I'm leaning towards just embedding the keywords.
- Havoc 3y agoWhen trying to find similarity between whole docs one would feed the entire doc though, right?
- therealdrag0 3y agoKeyword would leave you with semantics of word definitions but lose sentence meaning/context right?
- gregw134 3y agoI'm sure it would lose a ton of meaning, but for me it's easier to fit into a traditional search pipeline.
- jacky2wong 3y agoI love this library - I'm surprised cohere doesn't jump in and have its logo in the front!