tanujjain

4 ms·

1.
by tanujjain 7y ago
The cnn methods in the package are particularly robust against resolution differences. In fact, if it's just a simple up/downscale that differentiates 2 images, then even hashing algorithms could be expected to do a good job.
2.
by tanujjain 7y ago
The api already supports returning the hamming distances/cosine similarities along with the duplicate file list which can be used to sort the files. Please refer the docs for 'find_duplicates' function for more.
3.
by tanujjain 7y ago
Only realized now that the 100-200ms time you refer to is for a single search and not for 400,000 searches. The package already achieves this brute-force speed. In fact, the package also implements bktree, which, depending upon the distance
4.
by tanujjain 7y ago
The hashes are 16 character hexadecimals represented as strings. Had a quick look at the faiss package and it looks promising. Would consider it for the next versions.
5.
by tanujjain 7y ago
As I pointed in other comments, the current implementation does not focus on the scale problem. However, using the 'scores' attribute of the 'find_duplicates' function, one could obtain the hamming distance/cosine s
6.
by tanujjain 7y ago
Just had a look, thanks for the pointer. And yes, it was designed for a dark purpose. Will try to find the comparisons with the currently implemented hashing methods and see if there's merit to implementing it.
7.
by tanujjain 7y ago
Thanks for the pointer, will check it out.
8.
by tanujjain 7y ago
That sounds good! Would be great if you can share the code, or even better, make a PR to the repo.
9.
by tanujjain 7y ago
We tried that approach, but it was way too slow.
10.
by tanujjain 7y ago
Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n'
11.
by tanujjain 7y ago
Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Ge