4 ms·
I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that
by simonw 2y ago
I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available.
I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They later relicense their older models under Apache 2.0 licenses.
This gives me confidence that I can continue to use my calculated vectors in the future even if Nomic's model is no longer available because I can run the local one instead.
Nomic Embed Vision 1.5 for example started out as CC-BY-NC-4.0 but was later relicensed to Apache 2.0: https://www.nomic.ai/blog/posts/nomic-embed-vision https://www.nomic.ai/blog/posts/nomic-embed-vision
- throwup238 2y agoIn my experience, a non-finetunable closed source API is a complete nonstarter for a large fraction of possible use cases, especially the higher value ones like law and engineering. Most of these embedding models are trained too much on colloquial use of language on the internet that has little overlap with how terms of art are used, and without the ability to fine tune it to a specific use case, they're only really useful for generic use cases and even then they can be limited.
- mahjongmen 2y agoHey! Since we focus on Enterprise use-cases, we made sure to include training data from domains like you mentioned above. While in very specific use-cases finetuning may be helpful, but we also do offer that as a customization service (just not available via SaaS)
- serjester 2y agoHave you looked at fine tuning linear adaptors to sit on top of the embedding models? This works with any model (proprietary or open) and I think in practice this is significantly easier to implement anyways.
- mahjongmen 2y agoHey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we were to start doing so, I would take into account the worry of ensuring our users have a way of accessing our models. One aspect here that isn't factored is also efficiency. Yes there might be strong open weight models but if you're punching at the 7bn+ weight class your serving requirements are vastly different from a throughput efficiency perspective (also your query-inference speed). All food for thought. That being said, if for your use-case, Nomic Embed Vision 1.5 is better than Embed-v4.0, happy to hop on a call to discuss where the differential may be.
- simonw 2y agoI don't doubt the new Cohere model is better - but one of the features I value most from an embedding model is having an escape hatch, so I can continue to calculate vectors using that same model far into the future if something happens to the hosting provider. This matters for embedding models because I'm presumably building up a database of many millions of vectors for later similarity comparisons - so I need to know I'll be able to embed an arbitrary string in the future in order for that investment to still make sense. Size doesn't matter much to me, I don't even need to be able to run that model, it's more about having an insurance policy for my own peace of mind. (Even a covenant that says "in the event that Cohere goes out of business this model will be made available under license X" would address this itch for me.)
- mahjongmen 2y agoI'll start off with, I'm not one of our founders and REALLY wouldn't want to be publicly held accountable for policies or commitments until I've been able to get internal alignment on things I say. That being said, since I do manage our Search and Retrieval offering, if we were to deprecate any of our embedding models (which is generally the risk of closed-source models), I will make sure that there is an "escape hatch" for users. Heard on what your concerns are though :)