3 ms·

Hashing text is one of the easiest, most common uses of hashing. As for your first example “you may be relying on a 3rd party service that compares things”, ha

• ezrast 8y ago
  Maybe not, but I bet "someone else overrode == and you've been told not to mess with it" has.
• _cs2017_ 8y ago
  Hashing text isn't hard but to do it efficiently you may need to know the distribution. Imagine you want to dedupe 1000 text documents each 1TB in size. Hashing naively would require reading 1000 TB and would be horribly inefficient. Hashing the short prefix would be better but you need to know how long of a prefix is sufficient to avoid too many collisions. Sorting might be safer from efficiency perspective in this case unless you know a bit more about your data set. About 3rd party service: let's say you have images of faces and you want to dedupe them. A third party face comparison service could be a reasonable option. (And as another comment suggested, third party could be a different group in the same company, whose code base isn't trivial to modify.)
  • ww520 8y ago
    Isn't sorting 1000 of 1TB strings even more inefficient? You need to read and compare the 1TB strings 1000 * log(1000) or 3000 times, while hashing only needs 1000 times.
    • _cs2017_ 8y ago
      Typically, you can tell the order between two strings early, after you read just the starting few characters. So for comparison you would usually read maybe 10 bytes per file on average, or 3 x 1000 x 10 bytes = 30KB in total. Or maybe 30MB if the strings have long common prefixes. Of course, in a highly specialized distribution, all the strings could be identical except for the last few characters, and then sorting would be horrible. Ultimately you have to have some idea about the distribution of your data to say anything about average complexity.