3 ms·
This analysis is flawed. 1) most data in the cloud is going to be proprietary. How much is GE's internal factory data going to overlap with Starbucks' financia
by ramanujan 17y ago
This analysis is flawed.
1) most data in the cloud is going to be proprietary. How much is GE's internal factory data going to overlap with Starbucks' financials?
2) You would need global access at the byte level to truly dedupe systems. So the 50 biggest companies on S3 are giving some random company read access to all their data? And allowing them to compare it to all the data of their competitors?
3) And to do what? to save a few percent max in storage costs on S3? That's not going to dominate your cost structure by a longshot.
Outside of a web crawl I doubt there is that much redundancy.
The only one who can dedupe at scale and with trust would be Amazon themselves, and they would only do it if it weren't a huge headache to keep track of.
- borism 17y agodeduping definitely not going to work on proprietary encrypted data but for web services like tumblr, flickr, youtube or hulu, deduping must be a killer app - there's a huge amount of overlapping imaging data between them and it makes little sense not to analyze it. I'm sure google is deduping youtube stuff on a massive scale. So it depends on the data and application of course. Then again, purely deduping based on byte comparison is not going to cut it - you need to look into logical deduping (not storing same image in different resolutions, formats etc)