4 ms·
Agreed. Do you think maybe these datasets should include checksums so that they might be recovered with more certainty, from less obvious sources? P.S.: Also,
by DarkLinkXXXX 10y ago
Agreed. Do you think maybe these datasets should include checksums so that they might be recovered with more certainty, from less obvious sources?
P.S.: Also, archive.org retroactively removes (or delists?) site content when blocked by robots.txt, as domain parkers often do, so I could see that causing trouble down the line. https://archive.org/post/1019415/retroactive-robotstxt-removal-of-past-crawls-aka-oakland-archive-policy https://archive.org/post/1019415/retroactive-robotstxt-remov...
- krasin 10y agoStrict hashsums (like MD5 / SHA-1 / SHA-2) don't make too much sense here, as the images from the 'less obvious sources' might be thumbnails of different sizes. Or, probably, the same size but reencoded (not always a good idea, but happens in the practice). Would a fuzzy hash, like the output from a known and fixed classifier work for that?
- krasin 10y agoI have added OriginalSize and OriginalMD5 columns to images.csv in the dataset: https://github.com/openimages/dataset/commit/526555b43ab74b6ca571540f98d7af0d1f8b1ef2 https://github.com/openimages/dataset/commit/526555b43ab74b6... still thinking about a soft hash.