4 ms·

At the time of the fire I used nodechef that hosted their services on OVH and seemingly all their backups as well (in the same datacenter). Turns out when extra

At the time of the fire I used nodechef that hosted their services on OVH and seemingly all their backups as well (in the same datacenter). Turns out when extraordinary events happen promises of backups and such aren't always kept. We lost some data because of that, luckily we had our own backups. A good reminder to make sure you have backups and that they are working correctly. No matter what promises anyone gives you should always have your own backup strategy that's disconnected from the vendor you use. It was Nodechefs fault, not OVH obviously but perhaps it's interesting for others.

Cthulhu_ 5y ago
Make sure to read the T's and C's and availability / retention rates closely; it's a process that involves decoding the legalese and trying to associate it with RL situations. Amazon's S3 for example offers a 99.999999999% data durability guarantee, with other bits implying they can withstand a datacenter going up in flames. But there's two caveats there; data availability is lower (so if that datacenter goes up in flame your data may not be lost, but it may also not be directly accessible until they restore their backups), and if they do lose data, what is the consequences to them? It'll be a financial compensation at best.
- terom 5y ago
  Breaking the S3 durability guarantees badly enough to get press attention would be a critical reputational hit, with the consequences going far beyond just the contractual SLA compensation for the customers. It would still heal with time, though. There was a period after some nasty incidents where the general advice was to avoid EBS for anything critical... but that's over now.
  - lbriner 5y ago
    Would it be critical though? Unless there are lots of alternatives at a similar availability and price-point, most people aren't just going to jump to another provider. Clouds sound largely interchangeable but you try swapping from Azure to AWS or vice-versa: for anything more than the simplest system, this would probably take 12 months+
  - Nextgrid 5y ago
    > badly enough to get press attention would be a critical reputational hit Would it? Equifax basically had the worst possible outcome and survived it just fine.
- staticelf 5y ago
  Well yes but Nodechef was used before I joined the company I work for. Needless to say, we migrated away from them due to that they didn't handle the situation to our satisfaction. Since then we've updated our view on backups a lot and it was a valuable lesson for us (or rather the managers). We didn't lose that much data and not anything important since we had backups but they were not daily. Even if you read the terms and conditions it's hard as a developer to really grok what legalese actually say in practice and since I work for a small company we can't really go out and hire a lawyer for each and every terms and conditions. Instead we now spend time to write our own backup strategy in case a new fire should occur, or something else like war.
- traceroute66 5y ago
  > Amazon's S3 for example offers a 99.999999999% data durability guarantee IIRC AWS S3 was subjected to formal methods design using TLA+, so as long as the underlying infrastructure is correctly implemented, I suspect the durability guarantee is on fairly solid foundations. Also of course, I suspect the durability guarantee is based on what option you select for your bucket in the S3 console.
mkj 5y ago

Cthulhu_ 5y ago

Make sure to read the T's and C's and availability / retention rates closely; it's a process that involves decoding the legalese and trying to associate it with RL situations. Amazon's S3 for example offers a 99.999999999% data durability guarantee, with other bits implying they can withstand a datacenter going up in flames. But there's two caveats there; data availability is lower (so if that datacenter goes up in flame your data may not be lost, but it may also not be directly accessible until they restore their backups), and if they do lose data, what is the consequences to them? It'll be a financial compensation at best.

terom 5y ago
Breaking the S3 durability guarantees badly enough to get press attention would be a critical reputational hit, with the consequences going far beyond just the contractual SLA compensation for the customers. It would still heal with time, though. There was a period after some nasty incidents where the general advice was to avoid EBS for anything critical... but that's over now.
- lbriner 5y ago
  Would it be critical though? Unless there are lots of alternatives at a similar availability and price-point, most people aren't just going to jump to another provider. Clouds sound largely interchangeable but you try swapping from Azure to AWS or vice-versa: for anything more than the simplest system, this would probably take 12 months+
- Nextgrid 5y ago
  > badly enough to get press attention would be a critical reputational hit Would it? Equifax basically had the worst possible outcome and survived it just fine.
staticelf 5y ago
Well yes but Nodechef was used before I joined the company I work for. Needless to say, we migrated away from them due to that they didn't handle the situation to our satisfaction. Since then we've updated our view on backups a lot and it was a valuable lesson for us (or rather the managers). We didn't lose that much data and not anything important since we had backups but they were not daily. Even if you read the terms and conditions it's hard as a developer to really grok what legalese actually say in practice and since I work for a small company we can't really go out and hire a lawyer for each and every terms and conditions. Instead we now spend time to write our own backup strategy in case a new fire should occur, or something else like war.
traceroute66 5y ago
> Amazon's S3 for example offers a 99.999999999% data durability guarantee IIRC AWS S3 was subjected to formal methods design using TLA+, so as long as the underlying infrastructure is correctly implemented, I suspect the durability guarantee is on fairly solid foundations. Also of course, I suspect the durability guarantee is based on what option you select for your bucket in the S3 console.

mkj 5y ago