2 ms·
What makes this different from something like gensim? They have vector search for doc2vec embeddings.
by planetsprite 5y ago
What makes this different from something like gensim? They have vector search for doc2vec embeddings.
- cyrusthegreat 5y agoGensim is great for generating certain types of embeddings, but not for operationalizing them. It doesn’t do approximate nearest neighbor lookup which is a deal breaker for most models that use embeddings at scale. It also do not manage versioning so you end up having to hack a workflow around it to manage embedding. Finally, it’s not really data infrastructure like this is, so you end up doing hacky things like copying all your embeddings to every docker file. With regards to serving embeddings, gensim is just a library that supports in-memory brute force nearest neighbour look ups.
- ymroddi 5y agogensim actually allows you to use both annoy and nmslib with gensim generated vectors as part of the api. https://radimrehurek.com/gensim/similarities/nmslib.html https://radimrehurek.com/gensim/similarities/nmslib.html https://radimrehurek.com/gensim/similarities/annoy.html https://radimrehurek.com/gensim/similarities/annoy.html