3 ms·
ML noob here: what's the migration path from a set of embeddings made with model A to model B, if the source isn't available?
by franky47 3y ago
ML noob here: what's the migration path from a set of embeddings made with model A to model B, if the source isn't available?
- minimaxir 3y agoFully rebuild your vector store on the new embeddings.
- vessenes 3y agoSadly, re-embed. Or if you're really, really stuck, you could take the source embeddings, some of the sources, get destination new embeddings, then train a small model to update. It's unlikely this will do anything but lower your quality compared to re-embedding everything. This lock in effect is one reason I've avoided using OpenAI's embedding models; at least if it's open source, you'll be able to embed everything on an open model you have control of. The idea of committing to a large datastore using embedding APIs makes me feel very uncomfortable.
- sroussey 3y agoRebuild. You should setup your system are arbitrary number of embeddings for a given piece of text. In production that may mean just one, or maybe two while rebuilding.
- avereveard 3y agoSame technique as for mixed password hashing, you store the embedding model name and encode and search per each model stored, until the multiple embedding cost at search time become larger than embedding the old data again in the new for. At