3 ms·

> even if I have a vastly different perspective on how to accelerate problems like this Do tell more! After the initial optimization, I did hint at an approac

by nkurz 6y ago

> even if I have a vastly different perspective on how to accelerate problems like this Do tell more! After the initial optimization, I did hint at an approach to Andrew that I thought could get a further large speedup. Essentially, the idea was to "rotate" all the stored data 90 degrees, so that instead of counting the features present in each compound you read lists of compounds that contain a given feature, storing the hits in some very fast custom data structure. He wasn't particularly interested, likely correctly realizing that it would be a lot of work for an uncertain amount of gain. The question wasn't really whether I could achieve further speedup (although there was question as to how much), rather (as alluded to in the paper) whether he would be able to sufficiently increase sales to justify the additional development cost and added complexity of the codebase.

dalke 6y ago
Towards the end of the writing the paper, another paper came out on "RISC: rapid inverted-index based search of chemical fingerprints", https://doi.org/10.1021/acs.jcim.9b00069 https://doi.org/10.1021/acs.jcim.9b00069 which does something along those lines. It was close enough that I published the pre-print "RISC and dense fingerprints" at https://doi.org/10.26434/chemrxiv.8218517.v1 https://doi.org/10.26434/chemrxiv.8218517.v1 to examine its claims. I found that their RISC implementation was faster than chemfp for low bit densities (<~5%), which includes the popular 2048-bit ECFP/Morgan fingerprints for smaller radii, and uncommonly high similarity thresholds. Otherwise, chemfp was faster. So while there's certainly something to investigate there, I think it's better to focus that effort on truly sparse fingerprints and count fingerprints, rather than nominally dense bit fingerprints. Just needs money and time. ;) Plus, part of the focus was on making chemfp a really good baseline for these sorts of timing tests.

by nkurz 6y ago

dalke 6y ago

Towards the end of the writing the paper, another paper came out on "RISC: rapid inverted-index based search of chemical fingerprints", https://doi.org/10.1021/acs.jcim.9b00069 https://doi.org/10.1021/acs.jcim.9b00069 which does something along those lines. It was close enough that I published the pre-print "RISC and dense fingerprints" at https://doi.org/10.26434/chemrxiv.8218517.v1 https://doi.org/10.26434/chemrxiv.8218517.v1 to examine its claims. I found that their RISC implementation was faster than chemfp for low bit densities (<~5%), which includes the popular 2048-bit ECFP/Morgan fingerprints for smaller radii, and uncommonly high similarity thresholds. Otherwise, chemfp was faster. So while there's certainly something to investigate there, I think it's better to focus that effort on truly sparse fingerprints and count fingerprints, rather than nominally dense bit fingerprints. Just needs money and time. ;) Plus, part of the focus was on making chemfp a really good baseline for these sorts of timing tests.