2 ms·
Hashing text isn't hard but to do it efficiently you may need to know the distribution. Imagine you want to dedupe 1000 text documents each 1TB in size. Hashin
by _cs2017_ 7y ago
Hashing text isn't hard but to do it efficiently you may need to know the distribution.
Imagine you want to dedupe 1000 text documents each 1TB in size. Hashing naively would require reading 1000 TB and would be horribly inefficient. Hashing the short prefix would be better but you need to know how long of a prefix is sufficient to avoid too many collisions.
Sorting might be safer from efficiency perspective in this case unless you know a bit more about your data set.
About 3rd party service: let's say you have images of faces and you want to dedupe them. A third party face comparison service could be a reasonable option. (And as another comment suggested, third party could be a different group in the same company, whose code base isn't trivial to modify.)
- ww520 7y agoIsn't sorting 1000 of 1TB strings even more inefficient? You need to read and compare the 1TB strings 1000 * log(1000) or 3000 times, while hashing only needs 1000 times.
- _cs2017_ 7y agoTypically, you can tell the order between two strings early, after you read just the starting few characters. So for comparison you would usually read maybe 10 bytes per file on average, or 3 x 1000 x 10 bytes = 30KB in total. Or maybe 30MB if the strings have long common prefixes. Of course, in a highly specialized distribution, all the strings could be identical except for the last few characters, and then sorting would be horrible. Ultimately you have to have some idea about the distribution of your data to say anything about average complexity.