3 ms·

MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

PaulHoule 4y ago
I got clustering up and running with that in about ten minutes with k-means from scikit learn and wow, all the articles about Ukraine got clustered, all the sports, all the deep learning, all the reinforcement learning. It is way faster than my LDA clustering engine and I think gets better results. See sbert.net for a library that puts it on your fingertips. (With DBSCAN it shows great promise for clustering news articles about the same topic..)
- sharemywin 4y ago
  found this on their site thanks for the reference. https://www.sbert.net/docs/pretrained_models.html https://www.sbert.net/docs/pretrained_models.html
rahimnathwani 4y ago
Yesterday I read about this embedding model that can be run locally. I read about it here: https://news.ycombinator.com/item?id=35236275 https://news.ycombinator.com/item?id=35236275 But it's been mentioned on HN a few times in the past 3 months: https://news.ycombinator.com/item?id=35055808 https://news.ycombinator.com/item?id=35055808 https://news.ycombinator.com/item?id=34691307 https://news.ycombinator.com/item?id=34691307 https://news.ycombinator.com/item?id=34034940 https://news.ycombinator.com/item?id=34034940 https://news.ycombinator.com/item?id=34007942 https://news.ycombinator.com/item?id=34007942 It can be used as am embedding model in llama_index (fka gpt_index). Here's a code snippet from one of their users: from langchain.embeddings.huggingface import HuggingFaceEmbeddings from llama_index import LangchainEmbedding def get_embed_model(): model_name = "sentence-transformers/all-MiniLM-L6-v2" embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name=model_name)) return embed_model get_embed_model() index = GPTSimpleVectorIndex(documents, embed_model=embed_model)
sharemywin 4y ago
anybody have thoughts on this: https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/ https://engineering.fb.com/2017/03/29/data-infrastructure/fa... for doing the similarity search? I've also be thinking about looking into this: https://milvus.io/ https://milvus.io/
- PaulHoule 4y ago
  Faiss is great.
- monatis 4y ago
  Similarity search becomes a part of many technical stacks. Qdrant can be deployed as a service for fast, lightweight and filterable vector search. It's opensource, and you can also get up and running instantly and for free wit the cloud offering. You can try here https://qdrant.tech https://qdrant.tech

I got clustering up and running with that in about ten minutes with k-means from scikit learn and wow, all the articles about Ukraine got clustered, all the sports, all the deep learning, all the reinforcement learning. It is way faster than my LDA clustering engine and I think gets better results. See sbert.net for a library that puts it on your fingertips. (With DBSCAN it shows great promise for clustering news articles about the same topic..)

sharemywin 4y ago
found this on their site thanks for the reference. https://www.sbert.net/docs/pretrained_models.html https://www.sbert.net/docs/pretrained_models.html

rahimnathwani 4y ago

Yesterday I read about this embedding model that can be run locally. I read about it here: https://news.ycombinator.com/item?id=35236275 https://news.ycombinator.com/item?id=35236275 But it's been mentioned on HN a few times in the past 3 months: https://news.ycombinator.com/item?id=35055808 https://news.ycombinator.com/item?id=35055808 https://news.ycombinator.com/item?id=34691307 https://news.ycombinator.com/item?id=34691307 https://news.ycombinator.com/item?id=34034940 https://news.ycombinator.com/item?id=34034940 https://news.ycombinator.com/item?id=34007942 https://news.ycombinator.com/item?id=34007942 It can be used as am embedding model in llama_index (fka gpt_index). Here's a code snippet from one of their users: from langchain.embeddings.huggingface import HuggingFaceEmbeddings from llama_index import LangchainEmbedding def get_embed_model(): model_name = "sentence-transformers/all-MiniLM-L6-v2" embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name=model_name)) return embed_model get_embed_model() index = GPTSimpleVectorIndex(documents, embed_model=embed_model)

sharemywin 4y ago

anybody have thoughts on this: https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/ https://engineering.fb.com/2017/03/29/data-infrastructure/fa... for doing the similarity search? I've also be thinking about looking into this: https://milvus.io/ https://milvus.io/

PaulHoule 4y ago
Faiss is great.
monatis 4y ago
Similarity search becomes a part of many technical stacks. Qdrant can be deployed as a service for fast, lightweight and filterable vector search. It's opensource, and you can also get up and running instantly and for free wit the cloud offering. You can try here https://qdrant.tech https://qdrant.tech