9 ms·

Hetzner Cloud – Data loss incident

172 points by bratao 5y ago

Data loss incident (snapshots) Dear Customer, Unfortunately, we have to inform you that there was a data loss incident that affects a small amount of your snapshots on Hetzner Cloud. All snapshots you create are stored on our highly available storage systems. The snapshot contents are distributed over multiple internal servers and data is stored in a way that allows up to two separate disks to fail without impacting data integrity. This means the snapshot can still be accessed, even if two disks fail at the same time. Due to a recent, very unfortunate series of events in one of our clusters, multiple disks failed in short succession and caused a small number of snapshots to become unavailable. We immediately tried to recover the affected snapshots but unfortunately the data is lost and we have exhausted all our options. Affected snapshots in your account: XXXXXXXXX The snapshots have been removed from our system as they are no longer accessible. We sincerely hope this doesn’t cause too much trouble for you; we know losing data is the worst-case scenario. Also, we have added 20€ as Cloud Credits to your account (valid for one year). While we know that this will not bring back your data, we still hope that you will accept the gesture. In response to this we will re-evaluate our snapshot cluster data replication strategies as well as our strategies for replacing disks and rebuilding redundancy after replacement. Best Regards, Hetzner Cloud

wiredfool 5y ago
Hetzner has been doing emergency maintenance on their backup drive systems (BX-XX) lately. Both of mine on separate accounts have had the notifications.
tugapt 5y ago
tugapt 5y ago
raxxorraxor 5y ago
A snapshot of the current data? So the data is still available and another snapshot can be created?
- k8sToGo 5y ago
  Also old snapshots
- op00to 5y ago
  Snapshots are often used for deployment of systems. What happens if the snapshot you rely on as a base image gets lost?
  - traspler 5y ago
    Create a new one?
    - op00to 5y ago
      If the base snapshot is gone, all future snapshots that rely on that snapshot may be invalid. Depends on exactly how snapshots are implemented. Are they a delta of the previous snapshot? Are they a full copy of the device?
      - Kwpolska 5y ago
        Hetzner snapshots are fully independent disk images, nothing more, nothing less.
        op00to 5y ago
        Thanks! $20 seems kinda fair then, I guess.
  - justinholmes 5y ago
    Base images should be repeatable anyway using packer etc.
    - tetha 5y ago
      Even though this is a bit tricky on the hetzner cloud, because it requires the addition of a static route[1] to access the internet which isn't pushed by the DHCP, and I haven't been able to fully automate that in debians preseed for the life of me. Yes it's offtopic, but that's one really frustrating thing. Maybe someone here has a solution. 1: https://docs.hetzner.com/cloud/servers/iso-installation-gateway/#installation https://docs.hetzner.com/cloud/servers/iso-installation-gate...
Proven 5y ago
op00to 5y ago
Wow, twenty whole euro for blowing away your data. Let's hope Hetzner doesn't go bankrupt offering this immensely generous offer.
- yjftsjthsd-h 5y ago
  What would you like them to do?
  - op00to 5y ago
    Refund the total cost for a month at least?
    - cxcorp 5y ago
      Which would be less than the 20€ they've offered?
    - 10987654321 5y ago
      Just checked and for the location where I have my vm the price is: > Snapshots cost €0.0125/GB/month (incl. 25 % VAT) so €20 would cover a month for about 1.6TB of snapshots. Sounds pretty reasonable to me. But I guess it depends on how much data OP lost.
    - emteycz 5y ago
      They refunded much more than that...
    - unionpivo 5y ago
      my monthly hertzner cost is 4.5 eur (and yes this includes snapshots). For simple websites/blogs etc. you don't need much. I host 12 different sites on this setup. And I could put up more. I get around 0.5-1m views combined per month. They are mostly from one time zone, so traffic is not that spread out. That's without cloudflare in front. I used to have it, but a lot of people said they are getting captched to death. People* caught up in AWS bubble often don't realize how expensive it is. However I do use AWS at work extensively - right product for right occasion. * Not saying that's true for you. But I notice that a lot of people I work with are surprised, when they learn this.
c7DJTLrn 5y ago
That's not good but you do get what you pay for. Should never put all your eggs in one basket like that. The €20 of credit made air accelerate out of my nostrils.
- cle 5y ago
  > The €20 of credit made air accelerate out of my nostrils. Also they made sure you knew it's only valid for one year!
- systemvoltage 5y ago
  I came across tour of Hetzner datacenter[1] and it looks like a completely unprofessional setup, like castle on stilts. Does not inspire confidence. [1] https://www.youtube.com/watch?v=5eo8nz_niiM https://www.youtube.com/watch?v=5eo8nz_niiM
  - tobltobs 5y ago
    Could you tell us some of the red flags you see in this video? Considering it is a low cost hoster it looks fine to me, but I am not an expert.
  - jamal-kumar 5y ago
    If that's your standard for a garbage data centre... hmmm Yeah I've seen way, way worse. Stuff like power cables zip tied to network cables and fibre lines. DUST. delerict hardware just sitting everywhere. This is nice. What are you looking for, IBM dudes in white shirts running around?
  - throwaway71271 5y ago
    looks pretty good depends how localized the fire extinguishers are and how the power is distributed which is hard to see from the video not sure what you expect, using 1-2U boxes or blades or something wont change the setup that much, it pretty much boils down to 'how many machines will die in the same time on power/fire' and how can we cool them as cheap as possible.
  - brokenodometer 5y ago
bennyp101 5y ago
I wonder if this is related to the emails I've been getting since the start of March regarding maintenance on Storage Box Hosts? There were urgent maintenance ones on the 11th April as well.
- cersa8 5y ago
  The day I ordered my first Hetzner storage box (May 16) was the day I had my first Urgent Maintenance notice. Today the second one. "Urgent maintenance on Storage Box hosts XXX to XXX" Never had any trouble with their vps offerings, but I'm not liking it.
uuyi 5y ago
Lost an EBS snapshot on AWS once. The only way to provide some assurance in this space is to make sure your data is stored in more than one physically and commercially separate location.
- Aachen 5y ago
  This is the way. I get that not doing small-scale things yourself saves costs and that's why "cloud" exists, but if something is important to you or your organization, I cannot stress enough how useful having just a simple old data copy is. I've lost data to bad drives too often myself. Microsoft can erroneously close your paid-for account without recourse, or ransomware can try to extort you, but your data will simply not be lost if you have an offline (or more advanced: append-only) system with an hourly/daily/... backup on it. There are various options here but the one I happened to have read up on (and then contributed documentation to make it easier to use safely) is Restic with Rest-server in append-only mode. It's not perfect (e.g. no compression yet, only dedup) but it's pretty good. If you're an individual or small–medium business, put your favorite raspberry pi equivalent in a closet and invest a hundred bucks in a hard drive big enough for the important documents, pictures of loved ones, etc. A single drive shouldn't be your only backup, just don't put 100% of your trust in magic cloud. See also Atlassian this week: it may be down, but if you need to access the data of ticket xyz for a lead this week and you have a local data export, you can find the needed data.
  - mahastore 5y ago
    you do get what you paid for
    - Aachen 5y ago
      Not always. Pay for various games on Xbox, paid onedrive storage, pay for office licenses for the whole family, trip some false positive and still get your account banned with everything in it. You can talk to a wall or ask a judge to force Microsoft to give a reason, but then they can't actually show the reason because that's against some law in another country or something. It's fantastic. (Dutch, unfortunately) https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-nederlands-onedrive-account-opheffen-van-rechter.html https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-n... https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu.html https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu...
drfuchs 5y ago
Anybody have more color on the nature of the "recent, very unfortunate series of events in one of our clusters [such that] multiple disks failed in short succession"? What kind of "events"? A failed climate-control system? A berserk employee with a sledge hammer? Russian hackers? The Spanish Inquisition? It seems odd that they'd be quite so vague and circumspect about it. Why not just say what it was, so we're not all left to speculate, perhaps for the worst?
- throwawayboise 5y ago
  A batch of disks in a RAID array that were all from the same manufacturing lot. Such setups have been known to have multiple drives fail within hours/days of each other. Best practice is to mix manufacturers or at least manufacturing lots when you build the storage array.
  - jerf 5y ago
    There's a couple of other well-known ways to screw up redundancy too, though I hope they're in far enough to know them. One of my favorite is, if you want to store three copies of something on a lot of disks, randomly select which three disks you want to send the copies to so that each bit of content is stored on three random disks. It superficially seems very appealing; I have 3 copies of everything, nothing is likely to go wrong! However, what you've actually created is a situation where if any three disks go down, you're guaranteed to lose data because as you scale up, the probability that those three disks were the three randomly-chosen disks for some set of data goes to 1. The probability of three disks going down at some point also likewise trends to 1 as you scale up. Not saying that was the case here, just sharing it as one of the well-known pitfalls I find particularly interesting.
    - GauntletWizard 5y ago
      I keep hearing this pitfall, and then I keep seeing the results of idiots who don't understand that by not picking three totally at random, what they have created is a giant mess should any of those three fail, instead of a quick and easy to clean up mass that you are constantly raid rebuilding from.
jamal-kumar 5y ago
I've never trusted cloud provider snapshots because it's just paying more for something that might fail outside of your responsibility. I have a local backup box and a remote one and they both sync stuff in from rsync on a cron job from multiple servers. Works great. In case of catastrophic failure (Which has happened a few times), putting my eggs in more than one basket has been extremely successful.
rmetzler 5y ago
Better communication than Atlassian apparently.
joecool1029 5y ago
I remember being told years back with Digitalocean to not rely on their backups by one of the people that wrote their system. This made sense to me as they didn't even charge for it for the longest time. We do now use it some of the time and haven't had problems but there's always a second place we send images to (usually S3) as well as longer term offline archives.
- KptMarchewa 5y ago
  Even when you consider that you might lose some data in an unlikely event, there still is value proposition in this kind of services. Easy restore compared to external services.
deleted 5y ago
[deleted]
timcavel 5y ago
boxingrock 5y ago
multiple drive failure... i wonder if they were all the same brand / model / batch
jacquesm 5y ago
> All snapshots you create are stored on our highly available storage systems. Highly available right until they are not. Nice to see that your data is worth 20 bucks to Hetzner. I always liked them but this is a bit rude to put it mildly.
- xmaayy 5y ago
  The storage system is highly available, but the data is not guaranteed to be :)
- charcircuit 5y ago
  Availability and durability are different things. S3 for example is designed for an availability such that it can fail 1 in every 10,000 requests on average and for a durability that can lose one object for every 100,000,000,000 objects you have each year on average.
  - jacquesm 5y ago
    NSS.
cpach 5y ago
Folks, remember the 3-2-1 rule. IMO, it’s still extremely relevant, even in today’s cloud-centric world. For data you can’t afford to loose, please, for the love of ${DEITY}, don’t store it with just one vendor. You never know what happens. https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
- bombcar 5y ago
  I think that people forget that you need two copies of your data under your control - and when your data is primarily on the cloud it is not data under your control; it is under the cloud company's control. You should have at least one copy that is physically in your possession, offline preferably, so that you have access to it. For my cloud data, I sync it down to a local server and then sync that to removable offline media.
  - KronisLV 5y ago
    > For my cloud data, I sync it down to a local server and then sync that to removable offline media. This seems like an excellent approach. Personally i use BackupPC which internally uses rsync and allows for incremental backups, compression and deduplication of said backups as needed, then copying the data over to additional HDDs with cron. Works pretty decently on a slow residential connection, since it all happens in the background through WireGuard and one of my homelab boxes. Plus, BackupPC allows to test restoring backups with a single click, on a per-file or per-directory basis. Of course, internally the solution is a bit of an Eldritch mess with Perl and whatnot, so other pieces of software like Bacula might be more popular in comparison.
- 1970-01-01 5y ago
  You forgot to explain it: Make 3 copies of your systems (just data if you can rebuild infra within your RTO) Use 2 different media (compress then encrypt) Move 1 copy off-site. (print the encryption key and store that in a fireproof safe)
  - freemint 5y ago
jasonzemos 5y ago
There is no reason to rely on "triple replication" for data integrity. This has long been a solved problem. An appropriate erasure encoding can reduce the probability of loss ~ten-fold while consuming physically less space (i.e. 2x worth of replication). Companies forego this technology because they feel confident in their operational ability to address failures quickly and competently. That's what we're relying on for data integrity, not the math.
- toast0 5y ago
  This email doesn't detail the integrity scheme, only that data was protected if no more than two disks failed, and more than two disks failed. That could be three copies at 3x the storage cost, or it could be a RAID6 or raidz2 style system where the storage cost is essentially two disks out of the storage set size (which we don't know). You could certainly increase the required failure count by increasing the parity, but if the problem was something like all the disks in the storage system hit the same fatal disk firmware bug at the same time (as speculated elsewhere in the thread, and is unfortunately plausible), then that doesn't really help much.
- deleted 5y ago
  [deleted]
- throwaway81523 5y ago
  Erasure codes with lots of redundant drives is how online.net's C14 storage product works, but it is much more hassle than ordinary replication, since retrieval requires reading data from a lot of different drives to reassemble the shares, and similarly for splitting storage requests into shares and recording them. C14 is a Glacier-like product where you request a retrieval and your data is restore as an S3 object sometime (up to hours) later. That makes it easier, I expect. I agree that this would be good as a last-ditch backup for stuff like Hetzner cloud snapshots, but the primary storage for those snapshots probably has to be ordinary RAID. Fwiw I have a bunch of data in Hetzner Storagebox and have gotten several notices in the past few months that Storagebox would be temporarily offline for maintenance, which I assume meant raid rebuilds and/or drive swaps. The most recent of these was an "urgent" maintenance with less notice time than usual. It hasn't caused me any inconvenience, but I wonder if they have suffered a spate of drive failures recently.
idxodse 5y ago
It's really odd to see so many complaining when yesterday a thread about Atlassian was asking for precisely this type of communication. They communicated what went wrong, how it happened (in a nutshell), apologized and stated how they're planning to do better. They even threw in 20€ which when compared to the pricing of their snapshot storage is more than fair. Shit happens, things go wrong. If you lost very important data because you only stored it in one location, then you're equally at fault. Especially when taking into account that they are a rather cheap service. What more do you want?

172 points by bratao 5y ago

wiredfool 5y ago

Hetzner has been doing emergency maintenance on their backup drive systems (BX-XX) lately. Both of mine on separate accounts have had the notifications.

tugapt 5y ago

raxxorraxor 5y ago

A snapshot of the current data? So the data is still available and another snapshot can be created?

k8sToGo 5y ago
Also old snapshots
op00to 5y ago
Snapshots are often used for deployment of systems. What happens if the snapshot you rely on as a base image gets lost?
- traspler 5y ago
  Create a new one?
  - op00to 5y ago
    If the base snapshot is gone, all future snapshots that rely on that snapshot may be invalid. Depends on exactly how snapshots are implemented. Are they a delta of the previous snapshot? Are they a full copy of the device?
    - Kwpolska 5y ago
      Hetzner snapshots are fully independent disk images, nothing more, nothing less.
      - op00to 5y ago
        Thanks! $20 seems kinda fair then, I guess.
- justinholmes 5y ago
  Base images should be repeatable anyway using packer etc.
  - tetha 5y ago
    Even though this is a bit tricky on the hetzner cloud, because it requires the addition of a static route[1] to access the internet which isn't pushed by the DHCP, and I haven't been able to fully automate that in debians preseed for the life of me. Yes it's offtopic, but that's one really frustrating thing. Maybe someone here has a solution. 1: https://docs.hetzner.com/cloud/servers/iso-installation-gateway/#installation https://docs.hetzner.com/cloud/servers/iso-installation-gate...

Proven 5y ago

op00to 5y ago

Wow, twenty whole euro for blowing away your data. Let's hope Hetzner doesn't go bankrupt offering this immensely generous offer.

yjftsjthsd-h 5y ago
What would you like them to do?
- op00to 5y ago
  Refund the total cost for a month at least?
  - cxcorp 5y ago
    Which would be less than the 20€ they've offered?
  - 10987654321 5y ago
    Just checked and for the location where I have my vm the price is: > Snapshots cost €0.0125/GB/month (incl. 25 % VAT) so €20 would cover a month for about 1.6TB of snapshots. Sounds pretty reasonable to me. But I guess it depends on how much data OP lost.
  - emteycz 5y ago
    They refunded much more than that...
  - unionpivo 5y ago
    my monthly hertzner cost is 4.5 eur (and yes this includes snapshots). For simple websites/blogs etc. you don't need much. I host 12 different sites on this setup. And I could put up more. I get around 0.5-1m views combined per month. They are mostly from one time zone, so traffic is not that spread out. That's without cloudflare in front. I used to have it, but a lot of people said they are getting captched to death. People* caught up in AWS bubble often don't realize how expensive it is. However I do use AWS at work extensively - right product for right occasion. * Not saying that's true for you. But I notice that a lot of people I work with are surprised, when they learn this.

c7DJTLrn 5y ago

That's not good but you do get what you pay for. Should never put all your eggs in one basket like that. The €20 of credit made air accelerate out of my nostrils.

cle 5y ago
> The €20 of credit made air accelerate out of my nostrils. Also they made sure you knew it's only valid for one year!
systemvoltage 5y ago
I came across tour of Hetzner datacenter[1] and it looks like a completely unprofessional setup, like castle on stilts. Does not inspire confidence. [1] https://www.youtube.com/watch?v=5eo8nz_niiM https://www.youtube.com/watch?v=5eo8nz_niiM
- tobltobs 5y ago
  Could you tell us some of the red flags you see in this video? Considering it is a low cost hoster it looks fine to me, but I am not an expert.
- jamal-kumar 5y ago
  If that's your standard for a garbage data centre... hmmm Yeah I've seen way, way worse. Stuff like power cables zip tied to network cables and fibre lines. DUST. delerict hardware just sitting everywhere. This is nice. What are you looking for, IBM dudes in white shirts running around?
- throwaway71271 5y ago
  looks pretty good depends how localized the fire extinguishers are and how the power is distributed which is hard to see from the video not sure what you expect, using 1-2U boxes or blades or something wont change the setup that much, it pretty much boils down to 'how many machines will die in the same time on power/fire' and how can we cool them as cheap as possible.
- brokenodometer 5y ago

bennyp101 5y ago

I wonder if this is related to the emails I've been getting since the start of March regarding maintenance on Storage Box Hosts? There were urgent maintenance ones on the 11th April as well.

cersa8 5y ago
The day I ordered my first Hetzner storage box (May 16) was the day I had my first Urgent Maintenance notice. Today the second one. "Urgent maintenance on Storage Box hosts XXX to XXX" Never had any trouble with their vps offerings, but I'm not liking it.

uuyi 5y ago

Lost an EBS snapshot on AWS once. The only way to provide some assurance in this space is to make sure your data is stored in more than one physically and commercially separate location.

Aachen 5y ago
This is the way. I get that not doing small-scale things yourself saves costs and that's why "cloud" exists, but if something is important to you or your organization, I cannot stress enough how useful having just a simple old data copy is. I've lost data to bad drives too often myself. Microsoft can erroneously close your paid-for account without recourse, or ransomware can try to extort you, but your data will simply not be lost if you have an offline (or more advanced: append-only) system with an hourly/daily/... backup on it. There are various options here but the one I happened to have read up on (and then contributed documentation to make it easier to use safely) is Restic with Rest-server in append-only mode. It's not perfect (e.g. no compression yet, only dedup) but it's pretty good. If you're an individual or small–medium business, put your favorite raspberry pi equivalent in a closet and invest a hundred bucks in a hard drive big enough for the important documents, pictures of loved ones, etc. A single drive shouldn't be your only backup, just don't put 100% of your trust in magic cloud. See also Atlassian this week: it may be down, but if you need to access the data of ticket xyz for a lead this week and you have a local data export, you can find the needed data.
- mahastore 5y ago
  you do get what you paid for
  - Aachen 5y ago
    Not always. Pay for various games on Xbox, paid onedrive storage, pay for office licenses for the whole family, trip some false positive and still get your account banned with everything in it. You can talk to a wall or ask a judge to force Microsoft to give a reason, but then they can't actually show the reason because that's against some law in another country or something. It's fantastic. (Dutch, unfortunately) https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-nederlands-onedrive-account-opheffen-van-rechter.html https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-n... https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu.html https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu...

drfuchs 5y ago

Anybody have more color on the nature of the "recent, very unfortunate series of events in one of our clusters [such that] multiple disks failed in short succession"? What kind of "events"? A failed climate-control system? A berserk employee with a sledge hammer? Russian hackers? The Spanish Inquisition? It seems odd that they'd be quite so vague and circumspect about it. Why not just say what it was, so we're not all left to speculate, perhaps for the worst?

throwawayboise 5y ago
A batch of disks in a RAID array that were all from the same manufacturing lot. Such setups have been known to have multiple drives fail within hours/days of each other. Best practice is to mix manufacturers or at least manufacturing lots when you build the storage array.
- jerf 5y ago
  There's a couple of other well-known ways to screw up redundancy too, though I hope they're in far enough to know them. One of my favorite is, if you want to store three copies of something on a lot of disks, randomly select which three disks you want to send the copies to so that each bit of content is stored on three random disks. It superficially seems very appealing; I have 3 copies of everything, nothing is likely to go wrong! However, what you've actually created is a situation where if any three disks go down, you're guaranteed to lose data because as you scale up, the probability that those three disks were the three randomly-chosen disks for some set of data goes to 1. The probability of three disks going down at some point also likewise trends to 1 as you scale up. Not saying that was the case here, just sharing it as one of the well-known pitfalls I find particularly interesting.
  - GauntletWizard 5y ago
    I keep hearing this pitfall, and then I keep seeing the results of idiots who don't understand that by not picking three totally at random, what they have created is a giant mess should any of those three fail, instead of a quick and easy to clean up mass that you are constantly raid rebuilding from.

jamal-kumar 5y ago

I've never trusted cloud provider snapshots because it's just paying more for something that might fail outside of your responsibility. I have a local backup box and a remote one and they both sync stuff in from rsync on a cron job from multiple servers. Works great. In case of catastrophic failure (Which has happened a few times), putting my eggs in more than one basket has been extremely successful.

rmetzler 5y ago

Better communication than Atlassian apparently.

joecool1029 5y ago

I remember being told years back with Digitalocean to not rely on their backups by one of the people that wrote their system. This made sense to me as they didn't even charge for it for the longest time. We do now use it some of the time and haven't had problems but there's always a second place we send images to (usually S3) as well as longer term offline archives.

KptMarchewa 5y ago
Even when you consider that you might lose some data in an unlikely event, there still is value proposition in this kind of services. Easy restore compared to external services.

deleted 5y ago

[deleted]

timcavel 5y ago

boxingrock 5y ago

multiple drive failure... i wonder if they were all the same brand / model / batch

jacquesm 5y ago

> All snapshots you create are stored on our highly available storage systems. Highly available right until they are not. Nice to see that your data is worth 20 bucks to Hetzner. I always liked them but this is a bit rude to put it mildly.

xmaayy 5y ago
The storage system is highly available, but the data is not guaranteed to be :)
charcircuit 5y ago
Availability and durability are different things. S3 for example is designed for an availability such that it can fail 1 in every 10,000 requests on average and for a durability that can lose one object for every 100,000,000,000 objects you have each year on average.
- jacquesm 5y ago
  NSS.

cpach 5y ago

Folks, remember the 3-2-1 rule. IMO, it’s still extremely relevant, even in today’s cloud-centric world. For data you can’t afford to loose, please, for the love of ${DEITY}, don’t store it with just one vendor. You never know what happens. https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

bombcar 5y ago
I think that people forget that you need two copies of your data under your control - and when your data is primarily on the cloud it is not data under your control; it is under the cloud company's control. You should have at least one copy that is physically in your possession, offline preferably, so that you have access to it. For my cloud data, I sync it down to a local server and then sync that to removable offline media.
- KronisLV 5y ago
  > For my cloud data, I sync it down to a local server and then sync that to removable offline media. This seems like an excellent approach. Personally i use BackupPC which internally uses rsync and allows for incremental backups, compression and deduplication of said backups as needed, then copying the data over to additional HDDs with cron. Works pretty decently on a slow residential connection, since it all happens in the background through WireGuard and one of my homelab boxes. Plus, BackupPC allows to test restoring backups with a single click, on a per-file or per-directory basis. Of course, internally the solution is a bit of an Eldritch mess with Perl and whatnot, so other pieces of software like Bacula might be more popular in comparison.
1970-01-01 5y ago
You forgot to explain it: Make 3 copies of your systems (just data if you can rebuild infra within your RTO) Use 2 different media (compress then encrypt) Move 1 copy off-site. (print the encryption key and store that in a fireproof safe)
- freemint 5y ago

jasonzemos 5y ago

There is no reason to rely on "triple replication" for data integrity. This has long been a solved problem. An appropriate erasure encoding can reduce the probability of loss ~ten-fold while consuming physically less space (i.e. 2x worth of replication). Companies forego this technology because they feel confident in their operational ability to address failures quickly and competently. That's what we're relying on for data integrity, not the math.

toast0 5y ago
This email doesn't detail the integrity scheme, only that data was protected if no more than two disks failed, and more than two disks failed. That could be three copies at 3x the storage cost, or it could be a RAID6 or raidz2 style system where the storage cost is essentially two disks out of the storage set size (which we don't know). You could certainly increase the required failure count by increasing the parity, but if the problem was something like all the disks in the storage system hit the same fatal disk firmware bug at the same time (as speculated elsewhere in the thread, and is unfortunately plausible), then that doesn't really help much.
deleted 5y ago
[deleted]
throwaway81523 5y ago
Erasure codes with lots of redundant drives is how online.net's C14 storage product works, but it is much more hassle than ordinary replication, since retrieval requires reading data from a lot of different drives to reassemble the shares, and similarly for splitting storage requests into shares and recording them. C14 is a Glacier-like product where you request a retrieval and your data is restore as an S3 object sometime (up to hours) later. That makes it easier, I expect. I agree that this would be good as a last-ditch backup for stuff like Hetzner cloud snapshots, but the primary storage for those snapshots probably has to be ordinary RAID. Fwiw I have a bunch of data in Hetzner Storagebox and have gotten several notices in the past few months that Storagebox would be temporarily offline for maintenance, which I assume meant raid rebuilds and/or drive swaps. The most recent of these was an "urgent" maintenance with less notice time than usual. It hasn't caused me any inconvenience, but I wonder if they have suffered a spate of drive failures recently.

idxodse 5y ago

It's really odd to see so many complaining when yesterday a thread about Atlassian was asking for precisely this type of communication. They communicated what went wrong, how it happened (in a nutshell), apologized and stated how they're planning to do better. They even threw in 20€ which when compared to the pricing of their snapshot storage is more than fair. Shit happens, things go wrong. If you lost very important data because you only stored it in one location, then you're equally at fault. Especially when taking into account that they are a rather cheap service. What more do you want?