5 ms·

The answer to the question is there >We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it do

The answer to the question is there >We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.

kwhitefoot 6y ago
> Management said we don't have enough money to fix any of this. It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost. I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner. An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years. I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
- benlivengood 6y ago
  > I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups. If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
  - Symbiote 6y ago
    But you are then archiving data (not just backing it up), and will need to work out how to delete records you are no longer legally able to keep. (GDPR and earlier laws, etc.)
    - benlivengood 6y ago
      Deletion requests affect monthly backups as well, although it's sort of up in the air how many months the EU will accept as a reasonable business practice for keeping should-be-deleted user data. The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
  - kwhitefoot 6y ago
    Afford in the strict sense of have enough money, yes of course. But backup is not the same as archive even when the backup exists. It is usually not easily searchable, it is hard to be sure which version is authoritative and so on.
carschno 6y ago
I have no reason to doubt that this is factual. However, I have serious doubt that management will acknowledge they made that decision. Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was. I obviously don't know who is right here.
pedro2 6y ago
IMO Management will find the money pretty quick if the business survive. Up until now, they didn't know what 'backup not working' meant. Now they know. Not being bitter -- there was a failure of communication from both parts.

kwhitefoot 6y ago

> Management said we don't have enough money to fix any of this. It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost. I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner. An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years. I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.

benlivengood 6y ago
> I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups. If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
- Symbiote 6y ago
  But you are then archiving data (not just backing it up), and will need to work out how to delete records you are no longer legally able to keep. (GDPR and earlier laws, etc.)
  - benlivengood 6y ago
    Deletion requests affect monthly backups as well, although it's sort of up in the air how many months the EU will accept as a reasonable business practice for keeping should-be-deleted user data. The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
- kwhitefoot 6y ago
  Afford in the strict sense of have enough money, yes of course. But backup is not the same as archive even when the backup exists. It is usually not easily searchable, it is hard to be sure which version is authoritative and so on.

carschno 6y ago

I have no reason to doubt that this is factual. However, I have serious doubt that management will acknowledge they made that decision. Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was. I obviously don't know who is right here.

pedro2 6y ago

IMO Management will find the money pretty quick if the business survive. Up until now, they didn't know what 'backup not working' meant. Now they know. Not being bitter -- there was a failure of communication from both parts.