4 ms·
Yes, you need all of the dimensions. All of the dimensionality reduction techniques, including SOTA ones (UMAP or better) are going to massively harm your embed
by Der_Einzige 4y ago
Yes, you need all of the dimensions. All of the dimensionality reduction techniques, including SOTA ones (UMAP or better) are going to massively harm your embeddings.
- tucnak 4y agoPerhaps I didn't word by question correctly, I'm looking to compare capability of Sentence transformers vs. OpenAI Ada-based embeddings relative to their respective dimensionality?
- MacsHeadroom 4y agoNo you don't need the extra dimensions and OpenAI is generally the worst at everything except being the first to market. Also, ditto your comments on vector database shilling. Vector Databases are just like any other database in that I'll host them myself. I don't need a dedicated VC backed company for a database.
- Der_Einzige 4y agoJust because you say it doesn't make it true. Dimensionality reduction is an extremely destructive operation. Losing even the wrong single vector component of an embedding is massively damaging to down stream performance.