I don't have experience with image duplication, but if you can make a decent hash a 2.3 billion item hashtable is really cheap. If you need to do something clo

I don't have experience with image duplication, but if you can make a decent hash a 2.3 billion item hashtable is really cheap. If you need to do something closer to pairwise (for instance, because you can't make a cheap hash of images which papers over differences in compression), make the hash table for the text descriptions, then compare the images within buckets. Of the few 5 or 6 text fields I just spot checked (not even close to random selection) the worst false positive I found (in the 12M data set) was 3 pairs of two duplicates with the same description. On the other hand I found one set of 76 identical images with the same description.