9 ms·

>Don’t blow anything into the Cloud that you don’t have a personal copy of. I don't understand this logic. Amazon's S3 offers service level agreements with fai

by nsfyn55 11y ago

>Don’t blow anything into the Cloud that you don’t have a personal copy of. I don't understand this logic. Amazon's S3 offers service level agreements with failure rates that at one point implied the statistical likelihood of losing an object to be once in "thousands of years". When dealing with any sort of stable storage this is simply something I cannot offer. I couldn't produce a set up locally with the resources I have for making guarantees on the decade level let alone millennia. With that said I keep personal copies, but the authoritative copy is what's in the cloud, because its a hell of a lot more stable. TL;DR I hear this argument all the time. The cloud isn't perfect but its a hell of a lot closer to anything I could achieve. "not invented here" syndrome won't save your data.

jacquesm 11y ago
Until your control panel gets hacked and you lose all your data. See: codespaces.com (assuming it wasn't an inside job or a dumb mistake, but even then the same rules apply). So no, DON'T BLOW ANYTHING INTO THE CLOUD THAT YOU DON'T HAVE A COPY OF. It has nothing to do with 'not invented here', it has everything to do with your inability to outsource your responsibilities. The degree to which people rely on others to take care of their stuff is a huge blind-spot. Jason's advice is spot on in this respect, no matter what the up-time guarantees of the cloud solution you are using (and no matter what the redundancies), if you store all your data in the cloud without an off-line copy your company is 3 mouse clicks away from being history.
- nsfyn55 11y ago
  > Until your control panel gets hacked and you lose all your data This is just alarmism. If you really wanted to demonstrate your point you would show me some data. The data would demonstrate that over the millions and millions of users on a number of cloud platforms that their rates of data loss are significantly higher than your "home spun" storage. Then you would take out the outliers and show an honest distribution on the most stable vendors(since I'd expect the majority of data loss over all vendors to have some amount of locality within a specific vendor). The result of this will be the rate of data loss on the most reliable cloud platforms. You can then compare this against your own success rates. If you have ever(even once) lost even a single byte of data(for any reason), I do believe you will find yourself hopelessly outmatched.
  - dspillett 11y ago
    > demonstrate that ... on a number of cloud platforms ... that their rates of data loss are significantly higher than your "home spun" storage You seem to be missing the point. People are not saying "don't use cloud storage at all" They are saying "don't use cloud storage exclusively" You might trust that your cloud providers will never be hacked, that your individual account on them will never be hacked, that they will never suffer a catastrophic failure, that they will never go out of business, that they will never be fully or intermittently down at the moment you need to retrieve some data, and so on and so forth, based on their claims and documentation, or that you won't cock up at some point and cause data loss in your account, but I prefer to have my own copy (or copies) as well as the ones that are "in the cloud". For truly essential data at least one of those copies is both offline and offsite.
    - nsfyn55 11y ago
      >For truly essential data at least one of those copies is both offline and offsite. But the fact remains that where ever you choose to put it "offline and offsite" the odds of it being lost are orders of magnitude greater then its persistent and redundant storage on a reputable cloud provider. Even if you put it on the most stable storage you can find and lock it in a underground safe. You can't guarantee its integrity a handful of years from now let alone centuries. To put it in other words you are advocating storing your money in a mattress because you don't "trust the banks"
      - dmoo 11y ago
        Maybe ask the people of Greece if they "trust the banks". Shit happens, a local copy of your stuff does no harm at all in the same way as some cash on the hip is cool in case your card stops working.
        nsfyn55 11y ago
        Do you keep your money in a shoe box under your bed or stuffed in your mattress? I'd hope no, but then that begs the question "where do you keep it?" I'd assume not in a bank because any bank can fail at any moment and they are only insured by an instutition as flaky at the United States Treasury department. Governments collapse all the time just ask the Soviet Union. Right? The failure probabilities are relative. And everyone here is equivocating. As if the likelihood of failure is evenly dispersed. You indicate that storing customer data in the cloud is "playing fast and loose" but I'd argue the opposite. Not having a cloud backup is what is "fast and loose" Imagine you are an independent bank. You need to move your customers deposits. Do you load sacks of cash into your corporate minivan or hire an armored car service? Well lets look closer. With the latter you are giving up control, right? You have no control over the quality measures an armored car service might take. Yet somehow not contracting them seems like foolishness. The reason is obvious. Its because you know in this case that transporting money isn't your expertise. Its not something you can focus adequate time and resources on perfecting. You also can't spread the risk of failure across a lot of customers, absorb that failure, and make your service better for the next go. The exact same logic applies to long term storage of data. If that isn't your only function its extremely hard to get it right.
- fixermark 11y ago
  What's the relative risk that my specific control panel will get hacked and I, specifically, will lose all my data, vs. the risk that data on my local store will be compromised by a virus on my computer encrypting my whole hard drive and refusing to let go until I send XYZ bitcoins to a specific address? There's risks and then there's risks. As always, you can spend some time and money to mitigate some risk, and decide some risk is low enough that you don't care to mitigate it. I wouldn't just assume one risk vector is higher than another without some concrete numbers.
  - jacquesm 11y ago
    There's risks and there's bankruptcy. If you feel like gambling then be my guest. What's the risk of your house catching fire? Are you insured? What are the risks of getting burgled? Your car stolen, a car accident? Are you insured? What are the risks of having a bad health issue happen to you? Are you insured? I'll bet that you don't know the answers to any of those questions, and yet, you are probably insured against all of them. As for your question in more detail: it happens often enough that for me if you operate your business 'in the cloud' and you don't have a back-up of your data outside of the cloud to guard against catastrophic data loss due to either malice or accident that I'll happily fail you. Think 'sysadmin was depressive, wiped out company' (or maybe you let him/her go and they took revenge or any one of a number of other scenarios that would instantly terminate your existence online if you had not guarded against it). Risk management is consequences * incidence versus cost to mitigate. If the cost to mitigate is negligible, the risk is measureable and the consequences are terminal then it is a no-brainer to protect yourself against this. At least one of my customers wished they had set up a backup facility for their critical data (too bad, end of story for them) and there is one published case that we all know about. That's two that you can count on and most likely other trouble shooters have similar stories. I see catastrophic cloud data loss as having a much higher chance of happening than many other items on the list of stuff that I verify before greenlighting an investment.
    - fixermark 11y ago
      I see what you mean; I read "DON'T BLOW ANYTHING INTO THE CLOUD THAT YOU DON'T HAVE A COPY OF" and thought of only my personal data (because if it's user data, it's originating from the cloud and I'm not blowing it into the cloud; if anything, I'm pulling it out of the cloud). Though if we unbox that a bit... How do we balance the risk of loss due to your cloud being owned vs. loss due to the risk of your users' PII being violated if someone attacks your in-house (ostensibly in-house reliability and security-managed) copies? Better make sure you're spending the money on security best-practices on all your copies, not just the "live" ones. Side-note: What are your thoughts on two separate cloud providers as providing sufficient insurance? Say, having your data live in Google Cloud and at rest in Amazon S3?
- Domenic_S 11y ago
  The 3-2-1 rule is a rule for a reason. 3 copies 2 formats 1 copy off-site The cloud is a great place for that 1 off-site copy.
  - jacquesm 11y ago
    Assuming the rest of your data isn't in the cloud, yes. Otherwise reverse the situation. But you got it perfectly.
  - phinullfermata 11y ago
    Modern 3-2-1: 3 backup services 2 formats 1 local copy I guess that could get expensive depending on the services.
    - jacquesm 11y ago
      Yes, it could be. The trade-off is usually measured by the cost to the company of re-creating or losing the data entirely. So for some stuff such a system is feasible, for others it isn't and then you'll have to compromise. Log data for instance does not need to be stored like that unless there are legal rules saying that you have to keep it no matter what. For each business this is a delicate affair and you should spend some time on figuring out what you really absolutely have to have and what you could lose without losing the company or getting in trouble with the law.
  - ajross 11y ago
    No argument at all about the last sentence, but you should have skipped the first four lines as wildly premature optimization. The vast majority of the world would be much better served by (to appropriate your jargon) a 2-1-1 rule, because it's a straightforward treatment for a straightforward problem that is easily implemented. Yes, "format skew" and double-failure of backup solutions does indeed happen, but at a much lower incidence than "oh crap I deleted it!".
    - jacquesm 11y ago
      I think the 'three copies' rule refers to cyclic backups, not necessarily three copies of the one piece of data. This protects against data corruption that is not detected immediately. Another common way of doing this (and one that I prefer) is one where you rotate out a backup medium with ever larger intervals. So one gets set aside per week, then one gets set aside per month and so on. That gives you a series of snapshots in time that will allow you to pinpoint with some accuracy when an event happened. Longer ago you'll have less accuracy but this can help a lot in trying to triangulate who or what messed up. Just being able to answer the question of whether or not 'x' happened before 'y' was hired or after can help in narrowing down the number of suspects in case of a breach or other nastiness. It also prevents against back-ups for whatever reason not wanting to be reloaded (and you should guard against that by loading your back-up immediately after you make it, even so, the medium might fail the next time you try a read). Better still if there is a streaming log of everything but only very few companies can afford that sort of solution for all their data. Those can be hard to restore from (by replaying) so there too a snapshot system can help.
xenophonf 11y ago
I think you missed the point. S3 isn't free. You pay for that service: and if you’re a person they are giving it to you without you signing anything accompanied by cash or payment that says “and I mean it“.
- nsfyn55 11y ago
  Are you talking to me? I never said it was free. No storage solution is free.
  - vive-la-liberte 11y ago
    He's saying that the article does not apply for S3 and other paid services.
    - nsfyn55 11y ago
      Yeah I guess I can see that. But then again I would never give important data to a 3rd party without some sort of contractual arrangement.
      - jacquesm 11y ago
        How will a contract save your ass if your data is lost? You can't outsource your responsibility.
        nsfyn55 11y ago
        What is it with you?Will all your home brew nonsense save your ass when your data is lost. I am basing all of my arguments on "what is the statistically most likely to fail" and "given limited resources where would you store your authoritative copy". The answers to these are "whatever you come up with will have a higher likelihood of failure" and "the authoritative copy goes into the cloud" You must be a terrible gambler. "Will playing the odds save your ass if you lose your money on good bet" no. But making good bets will increase your return regardless of the outcome of any given bet.
pjc50 11y ago
S3 seems great today, and is undoubtedly more technically reliable than any home solution, but history is littered with companies that closed down on very short notice leaving their users' data deleted or inaccessible. Not to mention that billing disputes or legal action could also affect your ability to get data from them. (Jason Scott is speaking from experience here, as one of the people called in to do emergency archive response when these cloud businesses shut down.) Edit: jacquesm points out the sudden death of cloudspaces.com, which I'd forgotten about: http://www.infoworld.com/article/2608076/data-center/murder-in-the-amazon-cloud.html http://www.infoworld.com/article/2608076/data-center/murder-... A single computer security problem (which happen on a daily basis, although not usually at this severity) could enable your cloud to be deleted. The "counterparty risk" is small but ineradicable. The finance industry re-learnt this with Bear Stearns recently.
- nsfyn55 11y ago
  Yeah but once again this problem is statistical in nature. Admittedly the cloud isn't perfect but the question remains. Does the "Counter Party" risk outweigh the chances that my home spun solution would fail? Since the likelihood of failure on my part is orders of magnitude higher then the "counter party" risk would have to be huge, like really enormous. e.g Amazon would constantly have to be on the brink of imminent collapse. This is not the case. If Amazon went out of business there is no chance at this point that the loss would be total and catastrophic. Its essentially a bank at this point. Its shutdown would be orderly.
  - grayclhn 11y ago
    Bank shutdowns are not orderly because banks are important; they're orderly because banks are heavily regulated. If Amazon has an orderly shutdown, it would be because of the selflessness of its owners and managers, which is a pretty fickle thing to bet on.
    - nsfyn55 11y ago
      I certainly think this level of regulation is on the horizon given the number of businesses who use S3 as an authoritative repository. If Amazon needed to shutdown S3 even today I can see the government asserting a heavy hand in it.
Spooky23 11y ago
You need to think strategically about what you put on the cloud, because it's easy to overdo it. The decision inputs for an individual, startup and enterprise are all different. The cost of getting stuff out of Amazon is very dear. While the statistical likelihood that Amazon will lose your stuff is low, the likelihood of Amazon or a competitor doing something to cause you to rethink your use of AWS (or force you to move) is much higher. Wanting to reduce your AWS costs in the long term is a near certainty. As vendors move away from perpetually licensed software this gets more important. What happens when a vendor (say Microsoft for strawman purposes) decides that some function that is important to you must interface with Azure, and Azure only? What if Amazon decides upstream network transfers are no longer free? Those sorts of changes can break your business, and vendors like Amazon/Microsoft/Google are fully enabled to change those terms.
nickpsecurity 11y ago
We've been doing it for a long time. It's called some reliable servers in multiple locations with copies of the same data plus some offline backups (esp tape). There's many non-technical, Fortune 500 companies whose data has lasted longer than Amazon's existence. Cheapest version of this involves two high-capacity boxes at two locations each with maybe a colo agreement. Let's you keep replacing stuff as servers and drives fail. I've even done it with embedded boxes (VIA Artigo's) in the past where data volume wasn't high.
limaoscarjuliet 11y ago
This is the perfect point of the article - use the cloud (hey, it's "free") but KEEP A COPY. Otherwise you will get boned one day!
- zenogais 11y ago
  Might as well just change "use the cloud" to "use a computer".
kordless 11y ago
I argue frequently that the cloud causes cognitive dissonance. People want the ease of use and reliability of centralized outsourced services, but other people want data privacy guarantee and control of those services. If anything is a threat to one or the other, it's the fact we fail to understand the various requirements from distinctly different use cases. These polarized "requirements" are usually rationalized away by each group. As a data privacy advocate, I believe that I can provide a reliable storage solution to my company without using a centralized cloud service, which then guarantees my privacy because "I'm in control". As an advocate of centralized cloud services, such as Amazon, I present that their team is better at security and reliability than any other team on the planet and that I can encrypt something and trust that my key management is secure. Both of these arguments have fallacies and assumptions. The solution is to challenge ourselves to build better solutions. At what point in time did technology advancement ever slow down? At what point will we ever stop and say "Y'all, this here compute system is good enough and should be centralized/decentralized!"? Never, I say.
kybernetyk 11y ago
Do you control Amazon? No? Then you're not the captain - you're just a sailor and someone else is in control of your data. Now go put your stuff up on S3 but please keep a backup of the things you upload there.
- nsfyn55 11y ago
  I do keep a back up of things, but the back up is on S3 because that is the most stable storage option available to me. Everything else is the risky copy. >Do you control Amazon? No? Then you're not the captain This is textbook NIH syndrome. Rather than looking at the relative probabilities you look at something from an ideological perspective. Your argument isn't "Its safer with me" its "I want to be in control of it" This attitude is generally harmful. Think about your money. Are you in control of the bank? no right, but you still keep your money there instead of in a shoe box under your bed. Why? Because the bank is better at protecting your money than you are. They have big heavy metal doors and men with guns to move it from place to place. Amazon is like a bank for your data.
  - kybernetyk 11y ago
    > This is textbook NIH syndrome. I'm not saying you shouldn't use S3. You should just make sure that it's not the only service you have the only copy of your data hosted. >Rather than looking at the relative probabilities you look at something from an ideological perspective. No, I'm looking here solely at the risk of putting all your eggs into a basket you don't control.
pfortuny 11y ago
The problem with the "once in "thousands of years"." argument is that you never know when that once happens and it may well be... tomorrow and never more in the next couple of thousands of years. That is what happened (once again) to John Meriwether and Long-Term Capital Management... A "once every 10.000 years event" which took place... today.
srean 11y ago
> I don't understand this logic. Curious if Upton Sinclair would have anything to do with it. Lets put it this way, I know enough not to store data on the cloud(s) exclusively.
jellicle 11y ago
Well, unless Amazon blocks your access to your account. Have you accounted for that in your failure rate calculations? Ask Bernie Sanders about the cloud and the implications of losing access to it at the whim of the cloud provider.
spydum 11y ago
What is your recourse if service level is not met? A monthly credit? Your data is probably worth more to you than the value of its hosting. Too many people rest on the laurels of SLA.
jokr004 11y ago
> Amazon's S3 offers service level agreements with failure rates that at one point implied the statistical likelihood of losing an object to be once in "thousands of years". Where things like that in 2009 when this article was written?
rsync 11y ago
"TL;DR I hear this argument all the time. The cloud isn't perfect but its a hell of a lot closer to anything I could achieve. "not invented here" syndrome won't save your data." I think there is an easy rebuttal that should be considered... First, the "nines" rating of any service or resiliency is just gibberish. Go find the statistical likelihood of money market funds "breaking the buck" or of CDS blowing up - both in 2007/2008. Those had a lot of nines too and a lot of very smart , well qualified people attesting to those nines (in venues even more serious than IT). A highly complex system becomes incomprehensible, even to the people that built it. Those nines mean nothing. Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully. I don't care who does the calculation and how many nines they come up with - if you load FreeBSD on two bare metal servers and put them in two different datacenters and run them with any kind of conservative and cautious sysadminning you'll have a better solution. Yes, it will be more expensive.[1][2] The standard closure to a comment like this is to refer to Talebs Black Swan and Antifragile books ... which you certainly should read ... but even more important is "Normal Accidents" by Charles Perrow[3] which I hope will convince you to stop looking for complex things that never fail, and instead look for simple things that fail gracefully. [1] ... but we have a HN-Readers discount - just ask! [2] You know who we are. [3] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
- nsfyn55 11y ago
  >Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully. This is from you and this is from the wikipedia article on NIH(not invented here) syndrome ... >Not invented here (NIH) is the philosophical principle of not using third party solutions to a problem because of their external origins. False pride often drives an enterprise to use less-than-perfect invention in order to save face by ignoring, boycotting, or otherwise refusing to use or incorporate obviously superior solutions by others. I am not saying don't have a copy, I am saying the if you have several copies the safe one is in the cloud.
- 11y ago
e40 11y ago
What if you make a mistake a manually delete something in the cloud? What if a program has a bug in it and something gets deleted? If your data is precious, you need multiple copies of it, no matter where they live.

by nsfyn55 11y ago

jacquesm 11y ago

Until your control panel gets hacked and you lose all your data. See: codespaces.com (assuming it wasn't an inside job or a dumb mistake, but even then the same rules apply). So no, DON'T BLOW ANYTHING INTO THE CLOUD THAT YOU DON'T HAVE A COPY OF. It has nothing to do with 'not invented here', it has everything to do with your inability to outsource your responsibilities. The degree to which people rely on others to take care of their stuff is a huge blind-spot. Jason's advice is spot on in this respect, no matter what the up-time guarantees of the cloud solution you are using (and no matter what the redundancies), if you store all your data in the cloud without an off-line copy your company is 3 mouse clicks away from being history.

nsfyn55 11y ago
> Until your control panel gets hacked and you lose all your data This is just alarmism. If you really wanted to demonstrate your point you would show me some data. The data would demonstrate that over the millions and millions of users on a number of cloud platforms that their rates of data loss are significantly higher than your "home spun" storage. Then you would take out the outliers and show an honest distribution on the most stable vendors(since I'd expect the majority of data loss over all vendors to have some amount of locality within a specific vendor). The result of this will be the rate of data loss on the most reliable cloud platforms. You can then compare this against your own success rates. If you have ever(even once) lost even a single byte of data(for any reason), I do believe you will find yourself hopelessly outmatched.
- dspillett 11y ago
  > demonstrate that ... on a number of cloud platforms ... that their rates of data loss are significantly higher than your "home spun" storage You seem to be missing the point. People are not saying "don't use cloud storage at all" They are saying "don't use cloud storage exclusively" You might trust that your cloud providers will never be hacked, that your individual account on them will never be hacked, that they will never suffer a catastrophic failure, that they will never go out of business, that they will never be fully or intermittently down at the moment you need to retrieve some data, and so on and so forth, based on their claims and documentation, or that you won't cock up at some point and cause data loss in your account, but I prefer to have my own copy (or copies) as well as the ones that are "in the cloud". For truly essential data at least one of those copies is both offline and offsite.
  - nsfyn55 11y ago
    >For truly essential data at least one of those copies is both offline and offsite. But the fact remains that where ever you choose to put it "offline and offsite" the odds of it being lost are orders of magnitude greater then its persistent and redundant storage on a reputable cloud provider. Even if you put it on the most stable storage you can find and lock it in a underground safe. You can't guarantee its integrity a handful of years from now let alone centuries. To put it in other words you are advocating storing your money in a mattress because you don't "trust the banks"
    - dmoo 11y ago
      Maybe ask the people of Greece if they "trust the banks". Shit happens, a local copy of your stuff does no harm at all in the same way as some cash on the hip is cool in case your card stops working.
      - nsfyn55 11y ago
        Do you keep your money in a shoe box under your bed or stuffed in your mattress? I'd hope no, but then that begs the question "where do you keep it?" I'd assume not in a bank because any bank can fail at any moment and they are only insured by an instutition as flaky at the United States Treasury department. Governments collapse all the time just ask the Soviet Union. Right? The failure probabilities are relative. And everyone here is equivocating. As if the likelihood of failure is evenly dispersed. You indicate that storing customer data in the cloud is "playing fast and loose" but I'd argue the opposite. Not having a cloud backup is what is "fast and loose" Imagine you are an independent bank. You need to move your customers deposits. Do you load sacks of cash into your corporate minivan or hire an armored car service? Well lets look closer. With the latter you are giving up control, right? You have no control over the quality measures an armored car service might take. Yet somehow not contracting them seems like foolishness. The reason is obvious. Its because you know in this case that transporting money isn't your expertise. Its not something you can focus adequate time and resources on perfecting. You also can't spread the risk of failure across a lot of customers, absorb that failure, and make your service better for the next go. The exact same logic applies to long term storage of data. If that isn't your only function its extremely hard to get it right.
fixermark 11y ago
What's the relative risk that my specific control panel will get hacked and I, specifically, will lose all my data, vs. the risk that data on my local store will be compromised by a virus on my computer encrypting my whole hard drive and refusing to let go until I send XYZ bitcoins to a specific address? There's risks and then there's risks. As always, you can spend some time and money to mitigate some risk, and decide some risk is low enough that you don't care to mitigate it. I wouldn't just assume one risk vector is higher than another without some concrete numbers.
- jacquesm 11y ago
  There's risks and there's bankruptcy. If you feel like gambling then be my guest. What's the risk of your house catching fire? Are you insured? What are the risks of getting burgled? Your car stolen, a car accident? Are you insured? What are the risks of having a bad health issue happen to you? Are you insured? I'll bet that you don't know the answers to any of those questions, and yet, you are probably insured against all of them. As for your question in more detail: it happens often enough that for me if you operate your business 'in the cloud' and you don't have a back-up of your data outside of the cloud to guard against catastrophic data loss due to either malice or accident that I'll happily fail you. Think 'sysadmin was depressive, wiped out company' (or maybe you let him/her go and they took revenge or any one of a number of other scenarios that would instantly terminate your existence online if you had not guarded against it). Risk management is consequences * incidence versus cost to mitigate. If the cost to mitigate is negligible, the risk is measureable and the consequences are terminal then it is a no-brainer to protect yourself against this. At least one of my customers wished they had set up a backup facility for their critical data (too bad, end of story for them) and there is one published case that we all know about. That's two that you can count on and most likely other trouble shooters have similar stories. I see catastrophic cloud data loss as having a much higher chance of happening than many other items on the list of stuff that I verify before greenlighting an investment.
  - fixermark 11y ago
    I see what you mean; I read "DON'T BLOW ANYTHING INTO THE CLOUD THAT YOU DON'T HAVE A COPY OF" and thought of only my personal data (because if it's user data, it's originating from the cloud and I'm not blowing it into the cloud; if anything, I'm pulling it out of the cloud). Though if we unbox that a bit... How do we balance the risk of loss due to your cloud being owned vs. loss due to the risk of your users' PII being violated if someone attacks your in-house (ostensibly in-house reliability and security-managed) copies? Better make sure you're spending the money on security best-practices on all your copies, not just the "live" ones. Side-note: What are your thoughts on two separate cloud providers as providing sufficient insurance? Say, having your data live in Google Cloud and at rest in Amazon S3?
Domenic_S 11y ago
The 3-2-1 rule is a rule for a reason. 3 copies 2 formats 1 copy off-site The cloud is a great place for that 1 off-site copy.
- jacquesm 11y ago
  Assuming the rest of your data isn't in the cloud, yes. Otherwise reverse the situation. But you got it perfectly.
- phinullfermata 11y ago
  Modern 3-2-1: 3 backup services 2 formats 1 local copy I guess that could get expensive depending on the services.
  - jacquesm 11y ago
    Yes, it could be. The trade-off is usually measured by the cost to the company of re-creating or losing the data entirely. So for some stuff such a system is feasible, for others it isn't and then you'll have to compromise. Log data for instance does not need to be stored like that unless there are legal rules saying that you have to keep it no matter what. For each business this is a delicate affair and you should spend some time on figuring out what you really absolutely have to have and what you could lose without losing the company or getting in trouble with the law.
- ajross 11y ago
  No argument at all about the last sentence, but you should have skipped the first four lines as wildly premature optimization. The vast majority of the world would be much better served by (to appropriate your jargon) a 2-1-1 rule, because it's a straightforward treatment for a straightforward problem that is easily implemented. Yes, "format skew" and double-failure of backup solutions does indeed happen, but at a much lower incidence than "oh crap I deleted it!".
  - jacquesm 11y ago
    I think the 'three copies' rule refers to cyclic backups, not necessarily three copies of the one piece of data. This protects against data corruption that is not detected immediately. Another common way of doing this (and one that I prefer) is one where you rotate out a backup medium with ever larger intervals. So one gets set aside per week, then one gets set aside per month and so on. That gives you a series of snapshots in time that will allow you to pinpoint with some accuracy when an event happened. Longer ago you'll have less accuracy but this can help a lot in trying to triangulate who or what messed up. Just being able to answer the question of whether or not 'x' happened before 'y' was hired or after can help in narrowing down the number of suspects in case of a breach or other nastiness. It also prevents against back-ups for whatever reason not wanting to be reloaded (and you should guard against that by loading your back-up immediately after you make it, even so, the medium might fail the next time you try a read). Better still if there is a streaming log of everything but only very few companies can afford that sort of solution for all their data. Those can be hard to restore from (by replaying) so there too a snapshot system can help.

xenophonf 11y ago

I think you missed the point. S3 isn't free. You pay for that service: and if you’re a person they are giving it to you without you signing anything accompanied by cash or payment that says “and I mean it“.

nsfyn55 11y ago
Are you talking to me? I never said it was free. No storage solution is free.
- vive-la-liberte 11y ago
  He's saying that the article does not apply for S3 and other paid services.
  - nsfyn55 11y ago
    Yeah I guess I can see that. But then again I would never give important data to a 3rd party without some sort of contractual arrangement.
    - jacquesm 11y ago
      How will a contract save your ass if your data is lost? You can't outsource your responsibility.
      - nsfyn55 11y ago
        What is it with you?Will all your home brew nonsense save your ass when your data is lost. I am basing all of my arguments on "what is the statistically most likely to fail" and "given limited resources where would you store your authoritative copy". The answers to these are "whatever you come up with will have a higher likelihood of failure" and "the authoritative copy goes into the cloud" You must be a terrible gambler. "Will playing the odds save your ass if you lose your money on good bet" no. But making good bets will increase your return regardless of the outcome of any given bet.

pjc50 11y ago

S3 seems great today, and is undoubtedly more technically reliable than any home solution, but history is littered with companies that closed down on very short notice leaving their users' data deleted or inaccessible. Not to mention that billing disputes or legal action could also affect your ability to get data from them. (Jason Scott is speaking from experience here, as one of the people called in to do emergency archive response when these cloud businesses shut down.) Edit: jacquesm points out the sudden death of cloudspaces.com, which I'd forgotten about: http://www.infoworld.com/article/2608076/data-center/murder-in-the-amazon-cloud.html http://www.infoworld.com/article/2608076/data-center/murder-... A single computer security problem (which happen on a daily basis, although not usually at this severity) could enable your cloud to be deleted. The "counterparty risk" is small but ineradicable. The finance industry re-learnt this with Bear Stearns recently.

nsfyn55 11y ago
Yeah but once again this problem is statistical in nature. Admittedly the cloud isn't perfect but the question remains. Does the "Counter Party" risk outweigh the chances that my home spun solution would fail? Since the likelihood of failure on my part is orders of magnitude higher then the "counter party" risk would have to be huge, like really enormous. e.g Amazon would constantly have to be on the brink of imminent collapse. This is not the case. If Amazon went out of business there is no chance at this point that the loss would be total and catastrophic. Its essentially a bank at this point. Its shutdown would be orderly.
- grayclhn 11y ago
  Bank shutdowns are not orderly because banks are important; they're orderly because banks are heavily regulated. If Amazon has an orderly shutdown, it would be because of the selflessness of its owners and managers, which is a pretty fickle thing to bet on.
  - nsfyn55 11y ago
    I certainly think this level of regulation is on the horizon given the number of businesses who use S3 as an authoritative repository. If Amazon needed to shutdown S3 even today I can see the government asserting a heavy hand in it.

Spooky23 11y ago

You need to think strategically about what you put on the cloud, because it's easy to overdo it. The decision inputs for an individual, startup and enterprise are all different. The cost of getting stuff out of Amazon is very dear. While the statistical likelihood that Amazon will lose your stuff is low, the likelihood of Amazon or a competitor doing something to cause you to rethink your use of AWS (or force you to move) is much higher. Wanting to reduce your AWS costs in the long term is a near certainty. As vendors move away from perpetually licensed software this gets more important. What happens when a vendor (say Microsoft for strawman purposes) decides that some function that is important to you must interface with Azure, and Azure only? What if Amazon decides upstream network transfers are no longer free? Those sorts of changes can break your business, and vendors like Amazon/Microsoft/Google are fully enabled to change those terms.

nickpsecurity 11y ago

We've been doing it for a long time. It's called some reliable servers in multiple locations with copies of the same data plus some offline backups (esp tape). There's many non-technical, Fortune 500 companies whose data has lasted longer than Amazon's existence. Cheapest version of this involves two high-capacity boxes at two locations each with maybe a colo agreement. Let's you keep replacing stuff as servers and drives fail. I've even done it with embedded boxes (VIA Artigo's) in the past where data volume wasn't high.

limaoscarjuliet 11y ago

This is the perfect point of the article - use the cloud (hey, it's "free") but KEEP A COPY. Otherwise you will get boned one day!

zenogais 11y ago
Might as well just change "use the cloud" to "use a computer".

kordless 11y ago

I argue frequently that the cloud causes cognitive dissonance. People want the ease of use and reliability of centralized outsourced services, but other people want data privacy guarantee and control of those services. If anything is a threat to one or the other, it's the fact we fail to understand the various requirements from distinctly different use cases. These polarized "requirements" are usually rationalized away by each group. As a data privacy advocate, I believe that I can provide a reliable storage solution to my company without using a centralized cloud service, which then guarantees my privacy because "I'm in control". As an advocate of centralized cloud services, such as Amazon, I present that their team is better at security and reliability than any other team on the planet and that I can encrypt something and trust that my key management is secure. Both of these arguments have fallacies and assumptions. The solution is to challenge ourselves to build better solutions. At what point in time did technology advancement ever slow down? At what point will we ever stop and say "Y'all, this here compute system is good enough and should be centralized/decentralized!"? Never, I say.

kybernetyk 11y ago

Do you control Amazon? No? Then you're not the captain - you're just a sailor and someone else is in control of your data. Now go put your stuff up on S3 but please keep a backup of the things you upload there.

nsfyn55 11y ago
I do keep a back up of things, but the back up is on S3 because that is the most stable storage option available to me. Everything else is the risky copy. >Do you control Amazon? No? Then you're not the captain This is textbook NIH syndrome. Rather than looking at the relative probabilities you look at something from an ideological perspective. Your argument isn't "Its safer with me" its "I want to be in control of it" This attitude is generally harmful. Think about your money. Are you in control of the bank? no right, but you still keep your money there instead of in a shoe box under your bed. Why? Because the bank is better at protecting your money than you are. They have big heavy metal doors and men with guns to move it from place to place. Amazon is like a bank for your data.
- kybernetyk 11y ago
  > This is textbook NIH syndrome. I'm not saying you shouldn't use S3. You should just make sure that it's not the only service you have the only copy of your data hosted. >Rather than looking at the relative probabilities you look at something from an ideological perspective. No, I'm looking here solely at the risk of putting all your eggs into a basket you don't control.

pfortuny 11y ago

The problem with the "once in "thousands of years"." argument is that you never know when that once happens and it may well be... tomorrow and never more in the next couple of thousands of years. That is what happened (once again) to John Meriwether and Long-Term Capital Management... A "once every 10.000 years event" which took place... today.

srean 11y ago

> I don't understand this logic. Curious if Upton Sinclair would have anything to do with it. Lets put it this way, I know enough not to store data on the cloud(s) exclusively.

jellicle 11y ago

Well, unless Amazon blocks your access to your account. Have you accounted for that in your failure rate calculations? Ask Bernie Sanders about the cloud and the implications of losing access to it at the whim of the cloud provider.

spydum 11y ago

What is your recourse if service level is not met? A monthly credit? Your data is probably worth more to you than the value of its hosting. Too many people rest on the laurels of SLA.

jokr004 11y ago

> Amazon's S3 offers service level agreements with failure rates that at one point implied the statistical likelihood of losing an object to be once in "thousands of years". Where things like that in 2009 when this article was written?

rsync 11y ago

"TL;DR I hear this argument all the time. The cloud isn't perfect but its a hell of a lot closer to anything I could achieve. "not invented here" syndrome won't save your data." I think there is an easy rebuttal that should be considered... First, the "nines" rating of any service or resiliency is just gibberish. Go find the statistical likelihood of money market funds "breaking the buck" or of CDS blowing up - both in 2007/2008. Those had a lot of nines too and a lot of very smart , well qualified people attesting to those nines (in venues even more serious than IT). A highly complex system becomes incomprehensible, even to the people that built it. Those nines mean nothing. Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully. I don't care who does the calculation and how many nines they come up with - if you load FreeBSD on two bare metal servers and put them in two different datacenters and run them with any kind of conservative and cautious sysadminning you'll have a better solution. Yes, it will be more expensive.[1][2] The standard closure to a comment like this is to refer to Talebs Black Swan and Antifragile books ... which you certainly should read ... but even more important is "Normal Accidents" by Charles Perrow[3] which I hope will convince you to stop looking for complex things that never fail, and instead look for simple things that fail gracefully. [1] ... but we have a HN-Readers discount - just ask! [2] You know who we are. [3] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents

nsfyn55 11y ago
>Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully. This is from you and this is from the wikipedia article on NIH(not invented here) syndrome ... >Not invented here (NIH) is the philosophical principle of not using third party solutions to a problem because of their external origins. False pride often drives an enterprise to use less-than-perfect invention in order to save face by ignoring, boycotting, or otherwise refusing to use or incorporate obviously superior solutions by others. I am not saying don't have a copy, I am saying the if you have several copies the safe one is in the cloud.
11y ago

e40 11y ago

What if you make a mistake a manually delete something in the cloud? What if a program has a bug in it and something gets deleted? If your data is precious, you need multiple copies of it, no matter where they live.