4 ms·
Having played around with embeddings and built a few production use-cases with them.. They are awesome and enable a lot of cool applications. But, if you're bui
by Ozzie_osman 3y ago
Having played around with embeddings and built a few production use-cases with them.. They are awesome and enable a lot of cool applications. But, if you're building in a particular domain, you'll hit limits with any off-the-shelf embeddings model. One way to think about this is that the off-the-shelf embedding models have a lot of dimensions, but some of those dimensions may and some may not that matter for your application (classification, content similarity, clustering, etc). In other words, a vector may be close to another vector in dimensions you don't care about.
I look forward to seeing better tooling and literature around fine-tuning embedding models.
- eldenring 3y agoFine-tuning an entire language model to solve this problem is like using a sledgehammer on a nail. We have had tools for this for years, for example just label some data and train an SVM on your embedding space for classification.
- layoric 3y agoFrom my understanding, embedding models are just one layer of a modern LLM that do the similarity part as described in the post. The translation of representing content as a part of a common vocabulary in a vector. My understanding is that embedding models can be fine tuned in isolation to relate a vocabulary entries to each other, so fine tuning a whole LLM is not required, and the required compute resources would be far smaller.
- ahoho 3y agosentence-transformers has reasonably good tooling around this