2 ms·
Do your vectors change a lot? What have you tried so far? As a strong baseline, storing 10e6 512-bit vectors in memory should take about 640MiB of RAM. A binar
by gcr 3y ago
Do your vectors change a lot?
What have you tried so far? As a strong baseline, storing 10e6 512-bit vectors in memory should take about 640MiB of RAM. A binary Hamming distance between a query and a gallery is [popcnt(query xor elt) for elt in gallery], which should be plenty fast implemented in vectorized SSE instructions. I’d be shocked if this took more than 10ms per query per thread on modern hardware.
- Loic 3y agoYou are spot on, 10ms internal time for 100000 molecules, 60ms with the "stack on top": https://www.chemeo.com/similar?smiles=CCCCO https://www.chemeo.com/similar?smiles=CCCCO It is using a manually coded, most likely not that efficient, with popcnt.
- fbdab103 3y agoThese are not bit vectors, but float64. I could probably convert to float32 to cut down the ram. This is a one time job, so no changes. My initial attempt has been using the scikit-learn cosine similarity function[0] which lets you take a query matrix against a target one. The actual calculation is pretty fast and seems quite optimized (at least, I can see all of my cores max out when it does the calculation). Then I iterate across the results matrix, sorting for best matches, and record best seen to date. [0] https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html https://scikit-learn.org/stable/modules/generated/sklearn.me...