3 ms·
Train an autoencoder to reduce your vector dimensions down to something more workable. It’s unlikely you’ll be able to search against such enormous vectors in a
by janalsncm 3y ago
Train an autoencoder to reduce your vector dimensions down to something more workable. It’s unlikely you’ll be able to search against such enormous vectors in a reasonable amount of time anyways.
Another option is to shard your vectors into N pieces, where N*k is the length of your vector. Since cosine similarity doesn’t care about order, it will be fine. The only requirement is that the k-th vector can only be compared with other k-th vectors for similarity. The benefit of this approach is that it can be parallelized easily.
- nl 3y agoThe autoencoder was what I was thinking too. I was hoping to avoid training anything though! Search speed is good (although the datasize is pretty small - hundreds of vectors). So the sharding idea I'd: slice up the vector compare each sub vector with the corresponding sub vector ones from other vectors getting the similarities. sum the similarities choose the maximum Is this the general idea? Is there an implementation of this you've seen?
- janalsncm 3y agoIf you’re looking for something out of the box, I can’t help you unfortunately. I do ML at a company where we have our own in-house tooling. I agree with your algorithm, it is equivalent to cosine similarity. But you might also consider whether you need all the shards at all. Maybe you can get away with a half or even a tenth of them. If you have some metric like ndcg you can measure the drop in performance and consider the trade off.