deduping definitely not going to work on proprietary encrypted data but for web services like tumblr, flickr, youtube or hulu, deduping must be a killer app -

deduping definitely not going to work on proprietary encrypted data but for web services like tumblr, flickr, youtube or hulu, deduping must be a killer app - there's a huge amount of overlapping imaging data between them and it makes little sense not to analyze it. I'm sure google is deduping youtube stuff on a massive scale. So it depends on the data and application of course. Then again, purely deduping based on byte comparison is not going to cut it - you need to look into logical deduping (not storing same image in different resolutions, formats etc)