2 ms·
The autoencoder was what I was thinking too. I was hoping to avoid training anything though! Search speed is good (although the datasize is pretty small - hund
by nl 3y ago
The autoencoder was what I was thinking too. I was hoping to avoid training anything though!
Search speed is good (although the datasize is pretty small - hundreds of vectors).
So the sharding idea I'd:
slice up the vector
compare each sub vector with the corresponding sub vector ones from other vectors getting the similarities.
sum the similarities
choose the maximum
Is this the general idea? Is there an implementation of this you've seen?
- janalsncm 3y agoIf you’re looking for something out of the box, I can’t help you unfortunately. I do ML at a company where we have our own in-house tooling. I agree with your algorithm, it is equivalent to cosine similarity. But you might also consider whether you need all the shards at all. Maybe you can get away with a half or even a tenth of them. If you have some metric like ndcg you can measure the drop in performance and consider the trade off.