4 ms·

> As far as I can tell, getting good 'de-duplication' technology is more expensive right now than just buying disks. Working at the abstraction layer of disks

> As far as I can tell, getting good 'de-duplication' technology is more expensive right now than just buying disks. Working at the abstraction layer of disks is misleading because there are so many other pieces that go into making a service like S3. Instead I'm proposing an arbitrage that works one layer up -- on top of S3. > Also, how much would de-duplication help? I mean, how much stuff is stored on s3 that is uncompressed? (or the same compressed file?) It's a good question. The answer is unclear. Certainly if you're willing to get exotic with redundancy elimination (as opposed to dedup, which is a subset) and large-data-set compression algorithms there just are going to be economies of scale. What's not clear is (a) how large those economies become, or (b) how exotic your algorithms have to be to capture those economies. > ..I could turn a profit renting uncompressed drive space at $0.05/gigabyte/month.. I wouldn't want to compete with Amazon S3. Data safety and uptime would keep me up at night like a mofo. I just wouldn't want to run that company, though I can imagine folks like MS doing it. > Costs in the outsourced infrastructure market, as far as I can tell, have always been dominated by marketing. This is a really interesting question. The landscape is obviously changing with the current batch of cloud infrastructure services. I do think there are pieces of cloud storage that will not be commoditized, most notably the peace of mind around data safety and uptime. I wouldn't want to use a small cloud storage provider. Thus: (a) they will be able to charge a premium for a long time, and (b) there will be a smallish number of providers, meaning marketing will be less important in the future than having a good product. (I'm not sure how much the API's and other development tools will be commoditizable. MSFT historically does a great job of embrace & extend when it comes to those parts, though their recent cloud services are too wedded to Windows and .NET.) (EDIT: added responses to more of your original post.)

lsc 17y ago
>Instead I'm proposing an arbitrage that works one layer up -- on top of S3. Is transfer from s3 to ec2 free? in which case, that might work. Otherwise you'd get eaten alive by bandwidth charges between s3 and your compression/uncompression box. >I wouldn't want to compete with Amazon S3. Data safety and uptime would keep me up at night like a mofo. I just wouldn't want to run that company, though I can imagine folks like MS doing it. If you are storing data compressed in your special way on s3, you have the same problems. using non-ecc ram (or ecc ram configured to not crash when bad) and have bad ram on one of your compression boxes? data is corrupted. gone. compression boxes go down? you now have your dreaded uptime issues. Of course, if there is some bug in your compression/deduplication/whatever software, again, you have data loss. I have personally lost much money and reputation trying to make my disk system 'better' by adding complexity and flexibility. After that pain, I now just try to keep the data as simple as possible (mirrors, and stripes of mirrors) I haven't had data loss since I abandoned my SAN. Complexity == admin error == data loss. In a reasonably designed hardware system, data loss is pretty rare without admin error. >I do think there are pieces of cloud storage that will not be commoditized, most notably the peace of mind around data safety and uptime. I wouldn't want to use a small cloud storage provider. Thus: (a) they will be able to charge a premium for a long time, and (b) there will be a smallish number of providers, meaning marketing will be less important in the future than having a good product. If s3 costs keep going down 15% a year (and hard drive costs keep going down 50% a year) s3 will very shortly become a 'premium' provider. While yes, there is room for premium providers, it will be a whole lot cheaper for anyone with significant storage needs to run their own stuff (or to go with a smaller player) I mean, paying premium prices makes sense sometimes, but not all the time. If you are speaking of premium markets, really, I'm the wrong guy to ask; I'm way over on the other end of the spectrum. edit: added more "simple is better"
- adamsmith 17y ago
  Very good points, all of them!! You are absolutely right that you would have to get serious about data integrity. ECC is such a good example.

lsc 17y ago

>Instead I'm proposing an arbitrage that works one layer up -- on top of S3. Is transfer from s3 to ec2 free? in which case, that might work. Otherwise you'd get eaten alive by bandwidth charges between s3 and your compression/uncompression box. >I wouldn't want to compete with Amazon S3. Data safety and uptime would keep me up at night like a mofo. I just wouldn't want to run that company, though I can imagine folks like MS doing it. If you are storing data compressed in your special way on s3, you have the same problems. using non-ecc ram (or ecc ram configured to not crash when bad) and have bad ram on one of your compression boxes? data is corrupted. gone. compression boxes go down? you now have your dreaded uptime issues. Of course, if there is some bug in your compression/deduplication/whatever software, again, you have data loss. I have personally lost much money and reputation trying to make my disk system 'better' by adding complexity and flexibility. After that pain, I now just try to keep the data as simple as possible (mirrors, and stripes of mirrors) I haven't had data loss since I abandoned my SAN. Complexity == admin error == data loss. In a reasonably designed hardware system, data loss is pretty rare without admin error. >I do think there are pieces of cloud storage that will not be commoditized, most notably the peace of mind around data safety and uptime. I wouldn't want to use a small cloud storage provider. Thus: (a) they will be able to charge a premium for a long time, and (b) there will be a smallish number of providers, meaning marketing will be less important in the future than having a good product. If s3 costs keep going down 15% a year (and hard drive costs keep going down 50% a year) s3 will very shortly become a 'premium' provider. While yes, there is room for premium providers, it will be a whole lot cheaper for anyone with significant storage needs to run their own stuff (or to go with a smaller player) I mean, paying premium prices makes sense sometimes, but not all the time. If you are speaking of premium markets, really, I'm the wrong guy to ask; I'm way over on the other end of the spectrum. edit: added more "simple is better"

adamsmith 17y ago
Very good points, all of them!! You are absolutely right that you would have to get serious about data integrity. ECC is such a good example.