4 ms·

Posting as a reminder to myself that "in the cloud" != safe There's always room for computer error, and more like, human error. Imagine if something like this

• noneshallpass6 11y ago
  This gives most SaaS providers a bad name. The error here is not the engineer deleting the db, it's the complete lack of data restore testing. Complex restoring never works well when the first implementation is under the pressure of the real event. Other SaaS providers will be cursing such a big name tool making such a public mess.
  • w_t_payne 11y ago
    I can't remember who said this, but it seems apt: "Don't ask if they do regular backups -- ask if they do regular restores".
    • dspillett 11y ago
      Yep. My regular rant: until it is tested what you have the is just a file, or collection of files or other object(s), that might be a backup. You don't really have a backup until you have tested it.
      • jamiek88 11y ago
        Schroedingers backup.
    • gmarkov 11y ago
      http://www.joelonsoftware.com/items/2009/12/14.html http://www.joelonsoftware.com/items/2009/12/14.html
    • amelius 11y ago
      Good point. And in the cloud, restoring a backup can take ages, which is important to take into consideration. In fact, I'm not sure how to do it efficiently other than swapping the drives.
  • pmlnr 11y ago
    DR testing is hard, complicated and costs a lot. Yes, it should be done regularly, but it's not an easy task; I believe small(ish) companies simply can't afford it.
    • notalaser 11y ago
      Then perhaps small(ish) companies shouldn't hold data that's critical to their customers. Just like real engineering companies don't build nuclear reactors if they can't test their safety systems, cars if they can't afford to do crash testing and so on. DR is not a luxury. Systems that don't properly do DR aren't unoptimized or something, they're badly engineered.
      • olalonde 11y ago
        Doubt any business will go down as a result of a web based diagram tool being unavailable for a few hours.
        • notalaser 11y ago
          I wasn't referring to this specific case. Also, "don't worry, our services may suck, but not to the point where they bring down your business" is not exactly the kind of reliability one would want to aspire to.
    • joshyeager 11y ago
      Testing complicated failover systems is hard. Testing backups is not. Especially if you just do a full nightly backup and not something more complex. The options range from "manually restore a random 10%" to "write a moderately complex script to automatically restore and validate everything". Those all look like more trouble than they're worth before you start, but they aren't that hard. And you'll recover your whole investment the first time you need to restore something and it's just a simple tweak to your standard test practice.
    • 21 11y ago
      They seem to afford now to run 4 restores in parallel, using different methods. If they had done this exercise even once a year, they would have known better what to expect, or how much it would take.
  • 3stripe 11y ago
    Apologies if I've put the wrong slant on this — designer not a tech expert. Could still edit the title of the submission if there's something else it should say?
    • noneshallpass6 11y ago
      It wasn't aimed at you.
• Fargren 11y ago
  Well, with Dropbox the default setting is to keep local copies of all files in every computer connected to the system, so it probably wouldn't be catastrophic.
  • Grue3 11y ago
    Not until the first sync, that is. Then all your files go poof.
  • sillysaurus3 11y ago
    There was a bug where Dropbox wiped a few of someone's files. These changes were sync'd to the customer's computers, which dutifully wiped the files. I think he was able to recover most or all of his data (I forget if it was photos or what) but it was interesting to realize that even with Dropbox's stellar record and pedigree, errors like that can still happen. It makes you realize data is just so fragile. Our bits are unlikely to last centuries, which is kind of unfortunate. I've wanted to take on this problem in a systematic way somehow. It seems like bittorrent offers a way to make a digital time capsule -- theoretically, if you have seeders, your data will persist forever. So you could imagine setting up a "time capsule" of sorts. Example: $2,000 on Digital Ocean will get you 33 years of server time. So that's one seeder, "guaranteed" to last 33 years. I wonder how far the costs could be reduced? Could you reach the 1 century mark for less than $3k? Costs will continue to go down, so it should be possible to have 10 seeders for your time capsule. And then of course you have to think of what to put into your time capsule. There's not any guarantee the computing architecture will be the same a century from now. But somehow I suspect Python 2.7 might still run. :) You could write some kind of rudimentary AI program that responds to basic questions, maybe using the data embedded in the time capsule... The only part I can't figure out is the most obvious one: Digital Ocean probably won't be around 20 years from now, let alone a century from now. So even if you get a bunch of money together, how do you reliably transform that money into running servers over the course of a century?