3 ms·

Huh, there's a ton of duplicates in the data set... I would have expected that it would be worthwhile to remove those. Maybe multiple descriptions of the same t

• minimaxir 4y ago
  Per the project page: https://laion.ai/blog/laion-400-open-dataset/ https://laion.ai/blog/laion-400-open-dataset/ > There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated. I am surprised that image-to-image dupes aren't removed, though, as the cosine similarity trick the page mentions would work for that too.
  • kaibee 4y ago
    I assume having multiple captions for the same image is very helpful actually.
    • minimaxir 4y ago
      Scrolling through the sorted link from the GP, there are a few dupes with identical images and captions, so that doesn't always work either.
  • gchamonlive 4y ago
    Isn't it really expensive to dedupe images based on content? As you have to compare every image to every other image in the dataset? How could one go about deduping images? Maybe using something similar to rsync protocol? Cheap hash method, then a more expensive one, then a full comparison, maybe. Even so 2B+ images... and you are talking about saving on storage costs, mostly which is quite cheap these days.
    • ALittleLight 4y ago
      You don't have to compare all images to one another and doing so wouldn't reliably dedupe - what if one image is slightly different resolution, different image type, different metadata, etc? They would different hashes but still basically the same data. I think the way you do it is to train a model to represent images as vectors. Then you put those vectors into a BTree which will allow you to efficiently query for the "nearest neighbor" to an image on log(n) time. You calibrate to find a distance that picks up duplicates without getting too many non-duplicates and then it's n log(n) time rather than n^2. If that's still too slow there is also a thing called ANNOY which lets you do approximate nearest neighbor faster.
    • minimaxir 4y ago
      Convert the images to embeddings, and perform an approximate nearest neighbor search on them and identify images that are very close together (e.g. with faiss, which the page alludes to using). It's performant enough even at scale.
    • deleted 4y ago
      [deleted]
    • gpm 4y ago
      I don't have experience with image duplication, but if you can make a decent hash a 2.3 billion item hashtable is really cheap. If you need to do something closer to pairwise (for instance, because you can't make a cheap hash of images which papers over differences in compression), make the hash table for the text descriptions, then compare the images within buckets. Of the few 5 or 6 text fields I just spot checked (not even close to random selection) the worst false positive I found (in the 12M data set) was 3 pairs of two duplicates with the same description. On the other hand I found one set of 76 identical images with the same description.
    • wongarsu 4y ago
      There are hash algorithms for image similarity. For a toy example, imagine scaling the image to 8x8px, making it grayscale, and using those 64 bytes as hash. That way you only have to hash each picture once, and can find duplicates by searching for hashes with a low hamming distance (number of bit flips) to each other, which is very fast. Of course actual hash algorithms are a bit cleverer, there are a number to choose from depending on what you want to consider a duplicate (cropping, flips, rotations, etc)
  • djoldman 4y ago
    At a minimum a hash should be computed for each image and dupes removed. I haven't read the paper so they might have already done so.