4 ms·
> dimensions to an arbitrary dimensionality is not a typical dimensionality reduction technique so that likely requires a special training/alignment technique t
by summarity 3y ago
> dimensions to an arbitrary dimensionality is not a typical dimensionality reduction technique so that likely requires a special training/alignment technique that's novel
Very basic techniques (e.g. SuperBit random projection) have been extremely effective with OpenAI embeddings in the past. E.g. all embeddings on findsight.ai are OpenAI Ada embeddings stored as SuperBit signatures with a code length of 10,000 (i.e. 157 integers each), and there's almost no recall loss compared to the full vectors.
- brigadier132 3y agoHow low in dimensions have you been able to go without significant recall loss?
- osmarks 3y agohttps://github.com/facebookresearch/faiss/wiki/Vector-codecs https://github.com/facebookresearch/faiss/wiki/Vector-codecs has some good things available too.
- huac 3y agodo you construct an index over the superbit signatures to perform approximate search or do you perform exact search?
- summarity 3y agoParallelised exact search, which is good enough because calculating the hamming similarity is just an XOR+POPCNT per vector component and an addition. But of course you could put this into an HNSW graph for approximate search for >10M vectors. Or do LSH first for even larger data sets.
- sjkoelle 3y agohow does this compare with PCA?