4 ms·
You're right that they are doing this at scale. It's just that their business model -- charging per GB, instead of per GB of new data that nobody else is stori
by adamsmith 17y ago
You're right that they are doing this at scale.
It's just that their business model -- charging per GB, instead of per GB of new data that nobody else is storing yet -- leaves them susceptible to arbitrage. I.e. customers can steal the economies of scale if they collaborate.
You know how Starbucks charges $5 for a small and $6 for a large? It's the same thing. If two people want smalls, but collaborate, they can just split a large and save $2 each. It's the same idea, except that arbitrage becomes more realistic when it's electronic.
(I've ignored compression in the whole discussion for simplicity. If AWS wants a business model that isn't arbitragable based on compression they would have to charge based on how compressable your data is. Also, next generation compression algorithms -- another blog post on that later perhaps -- could achieve compression rates that have the same economies of cross-customer scale that data deduplication does, so really it would have to be price based on how compressable your data is __given all of the other data S3 is storing__.)
(EDIT: made starbucks example simpler.)
- deleted 17y ago[deleted]
- tienshiao 17y agoI'd like to think users with large data sets are already deduping and/or compressing their data. Of course, the users still win by eating into the margins of the company performing the arbitrage. Deduping and compression at a higher level may be easier though less efficient. I had built a prototype mail system using a twisted POP server, qpsmtpd, and S3 as a mail store that deduped and compressed email bodies.
- Andys 17y agoThe problem is it wouldn't be difficult for Amazon to start charging based on how much storage space you're consuming, which would pull the rug from under their feet.
- notmyname 17y agoI'm wondering what the logistics of this would be. Say 1 customer is storing 100GB in a cloud storage service that does internal de-duping. A second customer uploads exactly the same 100GB of data. What is each charged? The whole 100GB*rate? Does each customer pay half? Do prices change for remaining customers when one "copy" of the data is removed? Without charging each customer for their total apparent storage use, I don't see how any customer can have any predictability as to their monthly bill.
- siculars 17y agoThis is exactly the point. Billing under this some dedupe scheme would be a nightmare. The only scale that is to be had is for the provider. Either Amazon in this example, or some other service built on top of Amazon (or self hosted) that kept the vagaries of price fluctuation away from the customer. Notice that the way Amazon does price arbitrage with their compute nodes allows then to pull the power on them at any time when the price point moves above what you had payed for it. Anybody feel good about that happening to their data?
- houseabsolute 17y ago> You're right that they are doing this at scale. Not in my experience. My product uses >5PB data and we don't bother to do significant deduping. I've only heard of a few products at Google that do, and that's only when you can anticipate that most objects in the system are going to be duplicated many times, which is decidedly not the case in heterogeneous storage like S3's.