4 ms·

"Koffler says it's not necessarily hard to move data out of S3, it just may be more expensive than putting it in." This is a ridiculous statement. PUTs are $0.

"Koffler says it's not necessarily hard to move data out of S3, it just may be more expensive than putting it in." This is a ridiculous statement. PUTs are $0.01 for 1,000 requests, and GETs are $0.01 for 10,000. Deletion is free. Data transfer out per GB is comparable to the monthly cost per GB. Sure, putting data in is cheap because you don't get charged for bandwidth, but storing it costs just as much per month as moving it out would. If you've got somewhere better to store your data than S3, you'll come out ahead price-wise after a single month, so you should do it. That's not lock-in, not even close. Convenience-wise, yes, there's a lot of lock-in with AWS, no question. But that's a different problem, it's not fair to imply that you're locked in because of charges to actually get your data out.

jacquesm 14y ago
You'd be pretty silly if you had all your data only stored on S3 without a backup under your own control. So moving out should come at a cost of $0. You should be able to cancel your account at a moments notice and not lose a single bit, if only because the other side can do the same.
- jshen 14y ago
  where do you put this backup? Another cloud provider? Servers you manage? Etc. I think you're understating the cost of keeping such a backup, and the complexity that may come with it.
  - jacquesm 14y ago
    I think you're underestimating the cost of not keeping such a backup. That's a terminal mistake if there ever was one.
    - blake8086 14y ago
      I would love to hear both of your estimates of the costs.
      - jrockway 14y ago
        Not having a backup: your company goes out of business. Having a backup: $1000 a year for a dedicated server you physically own.
        dangrossman 14y ago
        Not all businesses would go out of business if some data was not backed up. If Chartbeat lost its data, it would be a minor inconvenience that the feature of their real-time dashboard where you can rewind and look back at an earlier point of the day wouldn't work until it repopulated the next day. They probably wouldn't even lose a customer over it. When they were a young company without much funding, it might've made business sense to only back up their accounts and not the historical data for the sites being tracked. Not all data can be backed up for only $1000 a year either; it's not just a matter of storage costs and a server. A write-heavy service like Foursquare can't just turn off and do a dump at night; to have backups they need enough server capacity to replicate data as it comes in. For those servers to keep up, they have to be as beefy as the ones they're replicating -- a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.
        beagle3 14y ago
        > a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth. You are mixing data, memory and processing power. In the case Amazon crashes and burns, they need that CPU+Ram+bandwidth etc. near the data. But to just keep a copy of the data (not available online) they don't need a really beefy server -- just store all the updates from a node, and replay them later at your leisure. 5400rpm will often be fast enough for this purpose (of course, you would need days to recover in this setup ... but that might be ok if past data is not needed online) Take twitter or foursquare for example: They need everyone's most recent max(4, tweets/checkins in the last day) available online at any given time. But if last years' tweets/checkins are not available online when the system is degraded in case of backup restoration etc -- that may be acceptable.
dalore 14y ago
I don't think he meant in terms of bandwith. Creating an s3 bucket and using it is quite easy. If you wanted to take that an replicate it by having some sort of redundant storage built yourself that's going to be expensive. That's how I took it anyhow.
- bermanoid 14y ago
  Well, in that case, the "lock in" comes from S3 being better and cheaper than the alternatives. In which case that's something to be applauded, not criticized.
jacobbijani 14y ago
What about AWS import/export? You just ship them drives and they plug it in, charged by the hour. http://aws.amazon.com/importexport/ http://aws.amazon.com/importexport/

jacquesm 14y ago

You'd be pretty silly if you had all your data only stored on S3 without a backup under your own control. So moving out should come at a cost of $0. You should be able to cancel your account at a moments notice and not lose a single bit, if only because the other side can do the same.

jshen 14y ago
where do you put this backup? Another cloud provider? Servers you manage? Etc. I think you're understating the cost of keeping such a backup, and the complexity that may come with it.
- jacquesm 14y ago
  I think you're underestimating the cost of not keeping such a backup. That's a terminal mistake if there ever was one.
  - blake8086 14y ago
    I would love to hear both of your estimates of the costs.
    - jrockway 14y ago
      Not having a backup: your company goes out of business. Having a backup: $1000 a year for a dedicated server you physically own.
      - dangrossman 14y ago
        Not all businesses would go out of business if some data was not backed up. If Chartbeat lost its data, it would be a minor inconvenience that the feature of their real-time dashboard where you can rewind and look back at an earlier point of the day wouldn't work until it repopulated the next day. They probably wouldn't even lose a customer over it. When they were a young company without much funding, it might've made business sense to only back up their accounts and not the historical data for the sites being tracked. Not all data can be backed up for only $1000 a year either; it's not just a matter of storage costs and a server. A write-heavy service like Foursquare can't just turn off and do a dump at night; to have backups they need enough server capacity to replicate data as it comes in. For those servers to keep up, they have to be as beefy as the ones they're replicating -- a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.
        beagle3 14y ago
        > a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth. You are mixing data, memory and processing power. In the case Amazon crashes and burns, they need that CPU+Ram+bandwidth etc. near the data. But to just keep a copy of the data (not available online) they don't need a really beefy server -- just store all the updates from a node, and replay them later at your leisure. 5400rpm will often be fast enough for this purpose (of course, you would need days to recover in this setup ... but that might be ok if past data is not needed online) Take twitter or foursquare for example: They need everyone's most recent max(4, tweets/checkins in the last day) available online at any given time. But if last years' tweets/checkins are not available online when the system is degraded in case of backup restoration etc -- that may be acceptable.

dalore 14y ago

I don't think he meant in terms of bandwith. Creating an s3 bucket and using it is quite easy. If you wanted to take that an replicate it by having some sort of redundant storage built yourself that's going to be expensive. That's how I took it anyhow.

bermanoid 14y ago
Well, in that case, the "lock in" comes from S3 being better and cheaper than the alternatives. In which case that's something to be applauded, not criticized.

jacobbijani 14y ago

What about AWS import/export? You just ship them drives and they plug it in, charged by the hour. http://aws.amazon.com/importexport/ http://aws.amazon.com/importexport/