3 ms·

I would love to hear both of your estimates of the costs.

jrockway 14y ago
Not having a backup: your company goes out of business. Having a backup: $1000 a year for a dedicated server you physically own.
- dangrossman 14y ago
  Not all businesses would go out of business if some data was not backed up. If Chartbeat lost its data, it would be a minor inconvenience that the feature of their real-time dashboard where you can rewind and look back at an earlier point of the day wouldn't work until it repopulated the next day. They probably wouldn't even lose a customer over it. When they were a young company without much funding, it might've made business sense to only back up their accounts and not the historical data for the sites being tracked. Not all data can be backed up for only $1000 a year either; it's not just a matter of storage costs and a server. A write-heavy service like Foursquare can't just turn off and do a dump at night; to have backups they need enough server capacity to replicate data as it comes in. For those servers to keep up, they have to be as beefy as the ones they're replicating -- a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.
  - beagle3 14y ago
    > a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth. You are mixing data, memory and processing power. In the case Amazon crashes and burns, they need that CPU+Ram+bandwidth etc. near the data. But to just keep a copy of the data (not available online) they don't need a really beefy server -- just store all the updates from a node, and replay them later at your leisure. 5400rpm will often be fast enough for this purpose (of course, you would need days to recover in this setup ... but that might be ok if past data is not needed online) Take twitter or foursquare for example: They need everyone's most recent max(4, tweets/checkins in the last day) available online at any given time. But if last years' tweets/checkins are not available online when the system is degraded in case of backup restoration etc -- that may be acceptable.
    - deleted 14y ago
      [deleted]
- bermanoid 14y ago
  Absolutely right, assuming the data you keep on S3 is mission critical (for me it's mostly just archival stuff, logs and such, that we never touch but have no particular reason to delete since S3 is so cheap, so we don't worry about backups). But even if your data does matter, spending $1000 to back it up separately is not a no-brainer, depending what your data is actually worth to your business and how likely you think it is that S3 will lose it. Assuming your data is worth a million dollars, then spending $1000 to back it up implies that you think there's at least a 1 in 1000 chance that S3 will lose it all. I don't know what probability I'd assign to "S3 loses all my data", but I do know that the particular data I store there is not valuable enough to make it worth backing up elsewhere. If my data actually mattered, I'd work out the math and consider alternatives. I will say this, as a word of warning - I would never use Amazon's durability claims ("Amazon S3 is designed to provide 99.999999999% durability of objects over a given year") when doing such a risk vs. cost assessment, because the real risk is not that the expected failure modes will "get you", but that something totally unknown and unpredictable will happen, i.e. someone at Amazon totally fucks up a staging push and sends it live to production instead, and the replication system goes nuts and starts corrupting everyone's data irretrievably before anyone realizes what's going on, or something like that. SLAs and durability claims don't account for stuff like that, but it's still a risk. Of course, there's also the "cover my ass" and "I'm not paying for this anyways" factors to consider - if you would likely lose your job if the data in question was lost, and your personal finances are not affected one bit by the company spending an extra $1k/$10k/$100k on backups, then even if math tells you it's a waste for the company, you might as well do it, because as long as it's not too much of a pain to set up it will help protect you. The calculus always changes when it's other people's money you're playing with...
- jshen 14y ago
  that's not a very reliable server if it's $1000/year. Are you really trusting that with data that's so important your company will go out of business if you lost it? Are you trolling? But the real cost is the time people spend working on it. Sys admins, programming the process to get it onto the server, etc. I don't think any of the data I have in s3 would cause my company to go out of business if we lost it. Most of them are thumbnails from videos that could be regenerated.
- benjiweber 14y ago
  Try maintaining a backup of a bucket with tens of millions of objects that are constantly changing. Not such a trivial or cheap problem to solve now. Downloading the bucket listing alone can take hours. Btw if anyone has any suggestions I'd love to hear them. So far the best approaches seem to be either to upload to multiple providers in the first place or to organise the buckets in such a way that bucket listings are not required to copy all files changed since a certain date.

Not having a backup: your company goes out of business. Having a backup: $1000 a year for a dedicated server you physically own.

dangrossman 14y ago
Not all businesses would go out of business if some data was not backed up. If Chartbeat lost its data, it would be a minor inconvenience that the feature of their real-time dashboard where you can rewind and look back at an earlier point of the day wouldn't work until it repopulated the next day. They probably wouldn't even lose a customer over it. When they were a young company without much funding, it might've made business sense to only back up their accounts and not the historical data for the sites being tracked. Not all data can be backed up for only $1000 a year either; it's not just a matter of storage costs and a server. A write-heavy service like Foursquare can't just turn off and do a dump at night; to have backups they need enough server capacity to replicate data as it comes in. For those servers to keep up, they have to be as beefy as the ones they're replicating -- a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth.
- beagle3 14y ago
  > a year or so ago, that was 64GB RAM per database for their Mongo instances. That's definitely more than $1000 a year's worth. You are mixing data, memory and processing power. In the case Amazon crashes and burns, they need that CPU+Ram+bandwidth etc. near the data. But to just keep a copy of the data (not available online) they don't need a really beefy server -- just store all the updates from a node, and replay them later at your leisure. 5400rpm will often be fast enough for this purpose (of course, you would need days to recover in this setup ... but that might be ok if past data is not needed online) Take twitter or foursquare for example: They need everyone's most recent max(4, tweets/checkins in the last day) available online at any given time. But if last years' tweets/checkins are not available online when the system is degraded in case of backup restoration etc -- that may be acceptable.
  - deleted 14y ago
    [deleted]
bermanoid 14y ago
Absolutely right, assuming the data you keep on S3 is mission critical (for me it's mostly just archival stuff, logs and such, that we never touch but have no particular reason to delete since S3 is so cheap, so we don't worry about backups). But even if your data does matter, spending $1000 to back it up separately is not a no-brainer, depending what your data is actually worth to your business and how likely you think it is that S3 will lose it. Assuming your data is worth a million dollars, then spending $1000 to back it up implies that you think there's at least a 1 in 1000 chance that S3 will lose it all. I don't know what probability I'd assign to "S3 loses all my data", but I do know that the particular data I store there is not valuable enough to make it worth backing up elsewhere. If my data actually mattered, I'd work out the math and consider alternatives. I will say this, as a word of warning - I would never use Amazon's durability claims ("Amazon S3 is designed to provide 99.999999999% durability of objects over a given year") when doing such a risk vs. cost assessment, because the real risk is not that the expected failure modes will "get you", but that something totally unknown and unpredictable will happen, i.e. someone at Amazon totally fucks up a staging push and sends it live to production instead, and the replication system goes nuts and starts corrupting everyone's data irretrievably before anyone realizes what's going on, or something like that. SLAs and durability claims don't account for stuff like that, but it's still a risk. Of course, there's also the "cover my ass" and "I'm not paying for this anyways" factors to consider - if you would likely lose your job if the data in question was lost, and your personal finances are not affected one bit by the company spending an extra $1k/$10k/$100k on backups, then even if math tells you it's a waste for the company, you might as well do it, because as long as it's not too much of a pain to set up it will help protect you. The calculus always changes when it's other people's money you're playing with...
jshen 14y ago
that's not a very reliable server if it's $1000/year. Are you really trusting that with data that's so important your company will go out of business if you lost it? Are you trolling? But the real cost is the time people spend working on it. Sys admins, programming the process to get it onto the server, etc. I don't think any of the data I have in s3 would cause my company to go out of business if we lost it. Most of them are thumbnails from videos that could be regenerated.
benjiweber 14y ago
Try maintaining a backup of a bucket with tens of millions of objects that are constantly changing. Not such a trivial or cheap problem to solve now. Downloading the bucket listing alone can take hours. Btw if anyone has any suggestions I'd love to hear them. So far the best approaches seem to be either to upload to multiple providers in the first place or to organise the buckets in such a way that bucket listings are not required to copy all files changed since a certain date.