4 ms·
The author describes calculating cosine similarity of high dimensional vectors. If these are sparse binary vectors why not just store a list of nonzero indexes
by janalsncm 3y ago
The author describes calculating cosine similarity of high dimensional vectors. If these are sparse binary vectors why not just store a list of nonzero indexes instead? That way your “similarity” is just the length of the intersection of the two sets of indexes. Maybe I’m missing something.
- azornathogron 3y agoYou seem to be missing that that's already how it's implemented. https://www.marginalia.nu/log/69-creepy-website-similarity/ https://www.marginalia.nu/log/69-creepy-website-similarity/
- marginalia_nu 3y agoYes, that's how it's done. A dense (bitmap) representation of matrix wouldn't fit in memory, would require about a PB of RAM unless my napkin math is off. The cardinality of this dataset is in the 100s of millions. (An additional detail is I'm actually using a tiny fixed width bloom filter to make it go even faster)
- socksy 3y agoI'm curious if you had a look at GraphBLAS (ala Redis Graph etc)?
- marginalia_nu 3y agoNope, did try a few off the shelf approaches, but ultimately figured it was easier to build something myself.