8 ms·

You can never have too many backups

97 points by CHB0403085482 4y ago

muttled 4y ago
To add to this: having the backups copy somewhere that has write-only permissions can be a life-saver in a crypto-ransom malware situation. Hackers are smart enough to delete or encrypt backups, and this is made far easier for them when admins have full read/write permissions to the backup location.
- bombcar 4y ago
  Offline backups are very important - not only because they can't be modified because they're disconnected, but because they force you to spend time going and getting them and that time can be important. Many backups are successfully destroyed in trying to back them up, and spending some time to think about what is happening can be worth it.
- SahAssar 4y ago
  rsync.net is great for this. They keep ZFS snapshots, so even though you have read/write access you will always have the snapshots stored. So you will be safe at least as long as you notice the corruption within the normal 7-day rolling window (there are options for other windows or manual snapshots too IIRC). This should also lead to the question of how up-to-date your backups need to be and is your solution setup for that (is it 1s, 1m, 1h, 1d, etc.)? Define an actual SLA for how much data you can loose and test for that.
- Noumenon72 4y ago
  Does "write-only permissions" somehow imply they can't overwrite the backup with an encrypted copy? I don't understand how read permissions would allow them to delete or replace something.
  - mgdlbp 4y ago
    GP meant write once, read many
  - ok_dad 4y ago
    I think the OP means something more like "immutable" or "copy on write", so you would never modify or delete previous data (or at least separate the modifications/deletions to a more-secure system).
- WalterBright 4y ago
  Yeah, it's a crying shame that hard drives no longer come with physical read-only switches. I have no idea why major drive users don't demand this from the makers.
  - dsr_ 4y ago
    At the desktop level, this would result in a whole new class of useless support calls. At the SOHO server level, this might be useful. But the alternatives are already available: unmount the disk, or unmount and unplug it, or unmount, unplug, and carry over to the safe, or unmount, unplug, and carry to the backup site... At the datacenter level, nobody has time to physically touch things.
    - WalterBright 4y ago
      If your drive is write-enabled, and you carry it over to your system to restore from backup, then the ransomware can intervene and encrypt it first. Besides, for the bozos, ship the drives with the write-enable turned on. They won't notice the difference.
- justin_oaks 4y ago
  The solution for this that I've been working with is having an S3 bucket with versioning enabled. The systems that send backups to that S3 bucket don't have permissions to modify previous versions. If one or all of those systems are compromised then they can't overwrite previous backups because the original data will still exist in old versions.
darkteflon 4y ago
The bigger story here is: checklists, checklists, checklists. They’re underrated!
- ghaff 4y ago
  And automation. So long as you confirm that the automation is doing what you think it's doing. Another comment mentioned an offline backup. And I agree. But it's also easy to go from having a month old offline backup to a year old to...
- ElevenLathe 4y ago
  Do you know of a good app for doing checklists of this type? Many TODO-list apps market themselves as "checklist apps" but they really just mean "yeah we've got checkboxes". They don't mean "our app lets you choose from a library of checklists for common tasks and pull a fresh one to tick off when you want to perform one of those tasks." One that seems close is ForeFlight (demo: https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s), but it's just one feature of a much larger app, is mobile only, and obviously targeted at aviation. I'm looking for similar functionality but more IT-oriented (or at least generic, not referring to airplane models, etc.) and preferably with an offline desktop "client" (it is especially critical that you have your checklist library available if the whole corporate network is down, for example) or, barring that, at least a desktop-friendly web UI. Honestly, if I knew there was a market for this, I would build it. It's either a great opportunity or the market isn't big enough.
  - WalterBright 4y ago
    > Do you know of a good app for doing checklists of this type? 1. Type it into a file. 2. Print it onto a piece of paper 3. Put a mark by each item with a pen as you finish it.
    - ChrisMarshallNY 4y ago
      This. I have a small sticky note pad on my desk. When we have a meeting about features, I jot down my todos. In some cases, I may do a screengrab, or submit a bug report (to myself), but, most times, it’s the PostIt. When I’m done with a task, I crumple the note, and drag it into the trash can.
yohannparis 4y ago
A good trick to convince oneselves to backup and the cost of redoing the work lost. Sometimes, just a couple of pages of documents, or a setup script, is worth weeks of work, ressources, money, etc. And it's nothing compare to the costs of creating multiple backups.
- AyyWS 4y ago
  Makes sense to backup user data, and script your server/app/database install and setup. If your RTO is very short, then full backups with fast restores are best.
neilv 4y ago
In an early startup, where developers are also doing ops, and there are periodic manual tasks (like daily/weekly/monthly running certain kinds of backups, eyeballing AWS dashboards and logs to check for anything alerts missed, etc.), using the same board and issue-tracker you use for development tasks helps. In one company, I used GitLab Board, and had special tags for the recurring and background tasks, which were sticky in Kanban columns right below the generally one-time development tasks. The deadline dates for recurring tasks kept changing to the next time the task had to be done. The Issue description had the concise immediate instructions for the steps of the task (with a link to the wiki for related and background info). And every time you did the task, you added a very quick Comment with any notes, and bumped the due date. It was very lightweight, and fit the simplicity I was advocating (e.g., the GitLab Kanban board says what everyone is doing, all info goes into GitLab in one form or another, minimal other tools and silos to juggle, etc.).
- thrdbndndn 4y ago
  > Kanban board Does it have specific meaning in software engineering? Isn't Kanban just billboard in Japanese?
  - SahAssar 4y ago
    Yes, it's usually a column-based system for tracking progress of tasks that move through various stages (eg. planning, development, testing, rollout, evaluation). See here: https://en.wikipedia.org/wiki/Kanban_board https://en.wikipedia.org/wiki/Kanban_board
    - neilv 4y ago
      Exactly. Sometimes my columns aren't waterfall phases, but more: Backlog, Blocked, Active, Done, Abandoned. The "Blocked" is important to highlight on the board: when someone can't do their thing, no sense floundering quietly -- highlight that, see who can unblock it, and what anyone else can do in the meantime. "Abandoned" is another one that I think is important, but might be unusual. If the task no longer makes sense, stop immediately, move it to Abandoned, and move on to a more worthwhile task. Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore. Later, we can reflect on what info we had, when, and what we can learn from that. (I usually really-really want a Gantt model as well, on the same database that the board view is coming from, but the tools aren't quite there yet.)
GnarfGnarf 4y ago
I backup to DVDs. I have every backup for the last ten years. And yes, I can still read them.
- ttkari 4y ago
  > And yes, I can still read them. Just curious - how often do you test this?
  - GnarfGnarf 4y ago
    I just tested 6 CDs from 1998. I can read all of them. One CD took a little persuading. They have been kept in a cool, dark place (my basement). I write an MD5 checksum for each file so they can be verified at any time in the future.
zie 4y ago
If you are backing up users data that is not yours to own and do with as you please, yes you absolutely can have to many backups. If you are required by law or policy to erase user data and it's in your backup(s), you now have a real problem on your hands. Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this.
- JadeNB 4y ago
  > Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this. I am confused by these sentences. Sometimes the law does it, but you're not aware of a law that does it?
  - alliao 4y ago
    laws have jurisdiction so he's probably saying he doesn't know all laws from all jurisdictions?
  - zie 4y ago
    Essentially yes. I can't know all laws for all jurisdictions(nor would I want to), but laws could make a carve-out provision. I'm not currently aware of any that matter in my jurisdiction(s). That doesn't mean there aren't some out there somewhere. Policies are of course a different thing, and they may or may not apply to you, from industry regulations or cyber security insurance or just company or trade group policies that might apply.
- bombcar 4y ago
  GDPR apparently does: https://www.itgovernance.eu/blog/en/the-gdpr-how-the-right-to-be-forgotten-affects-backups-2 https://www.itgovernance.eu/blog/en/the-gdpr-how-the-right-t... If data is on backup tapes you seem to be able to inform them of how long your retention policies are.
  - msh 4y ago
    There is a bit more than just informing users: If you decide to go down this route, you should bear in mind that other supervisory authorities might be stricter and that you must be able to demonstrate that it’s impractical to delete backup data. This will require, at the very least, a risk assessment, business impact assessment and data protection impact assessment.
015a 4y ago
Early in one of my companies, we had a cloud hosted database provider that only provided (at our meager level of spend) 7 days of daily backups for the main db. We had a specific customer, one of maybe ten, and a higher percentage of our revenue, who made an big bulk import oopsie-whoopsie; and given the specifics of this software product, did not notice for weeks. They wanted everything back to the way it was. We ended up being saved by our traditionally "crappy" local dev process; me and the other dev had copies of the prod db on our development machines. And, naturally, we rarely updated them; the last time I had sync'ed it down was, conveniently, about four weeks ago. So they had some data loss between that "backup" and the oopsie-whoopsie, but far less major than their issues with the current state. The most interesting thing to really come out of that was a set of tooling to precision restore an individual customer's data from a backup of this multi-tenanted database. That was something I see as pretty rare for engineers to consider in their backup strategy; you may have the backup, it may be easily restored, but it will take the world with it and maybe you don't want the whole world. We ended up using that a half dozen or more times over the next few years... against a far more comprehensive set of backups of course. Yeah, maybe it makes sense to store a history of every change made to every row; but for our needs it made a lot more sense to leverage backups and some glue scripts.
- bombcar 4y ago
  One of the important considerations of database design is “what might I want to restore” and there are often way more answers than just “how it was at a point in time”. And some of those restores should happen in the application layer not the db backup.
  - silvestrov 4y ago
    100% of my restores have been due to user/developer fuckups. 0% has been because of hardware failures. This is why I love that PostgreSQL backup files are plain text files. It is so easy to extract a single table (or parts of a table) using a simple perl/python/whatever script.
- andrewf 4y ago
  Sounds like the thing Atlassian had to build while on fire this year.
outside1234 4y ago
Or practice restoring too many times
Ologn 4y ago
Remember the old 3-2-1 backup rule 3 recent copies of important data, in 2 different media formats, with 1 copy off-site
thematrixturtle 4y ago
Once upon a time, a Japanese megacorp was designing a new factory. The master plans for the factory were stored on a USB thumb drive owned by the Chief Engineer, who was the only person authorized to make changes to the master plan. The Chief Engineer and crew went to inspect another factory for inspiration. There was a direct flight to the city, but by corporate policy they had to use the preferred airline, requiring a transfer. The Chief Engineer didn't want to lug his heavy laptop around, so he checked it in, with the thumbdrive still plugged in. Chief Engineer and crew arrived successfully at the factory. His suitcase did not. Enquiries with the airline told them the suitcase had been shipped to the wrong place and would be returned very soon. And that was the last they or the airline ever saw of it. There was no central server, no versioning, no backups -- the plans were now gone. Fortunately, since this was a Japanese company, the plans were printed out weekly for review at the status update meeting. So the plans were manually reconstructed from paper printouts, page by page, line by line, character by character, and one of the many, many people typing it in was the friend who told me this story. Unsurprisingly, they no longer work at Megacorp.
- wodenokoto 4y ago
  Something about that story reminds me of toy story being deleted and a single animator had taken a copy of the _entire_ project home to work on it from home. Another good one is Maersk. When they suffered a ransomware attack that deleted all access to all backups/replication servers, the only surviving backup was a replication server that happened to be offline during the attack, because it was situated in a country with very unstable internet and power supply.
- MontyCarloHall 4y ago
  > the Chief Engineer was the only person authorized to make changes to the master plan. > the plans were printed out weekly for review at the status update meeting. This is so quintessentially Japanese. Have a single point of failure in a rigid chain of command structure, but bake in a ton of paper redundancy. Nonetheless, I bet the lost laptop was an extremely advanced model. In the early 2000s, my dad worked with some Japanese collaborators. They brought over some impossibly thin laptops that seemed a decade ahead of what you could get in the US at the time…and proceeded to use them to send faxes.
sebazzz 4y ago
I got my sysops backups monitored in Nagios. A script that runs in NSClient++ checks the backup date, size, and the backup log for errors. Still, you also need to inspect the backup manually from time to time - just to be sure.
Helmut10001 4y ago
Agree! These are some of my rule of thumbs: - write all commands, ever entered to any shell, down. Of course, this also includes the procedure to create backups - do not treat all data equal. There's 'fast' (but small) data that needs more backups more often and there is 'slow' (but large) data that is infrequently updated, thus less often backed up. - 3-2-1 rule is a minimal setup; extend it wherever possible. Also, use 'migration periods' to replace one backup system with another; during this period, you use both backup systems until you gained enough trust in the new one to deprecate the old one. This should include actually testing restoring systems from backups in VMs at least once. My private 3-2-1+: - immediate sync of files to my internal nextcloud server, hosted offsite. That means any files exist at least once in to different locations - daily ZFS automatic snapshots at the server side and deprecation of old ones - weekly, my server starts another Odroid CS2 at another offsite location and rsync's any changes; the Odroid is shutdown afterwards - yearly, I attach a 16TB usb drive and send ZFS snapshots. - in addition to these 'basic' 3-2-1, the '+' means: daily, Nextcloud's document folder is duplicated to a different ZFS dataset on the server which is not actively write-referenced in any VM; Also, for some very important files, I keep more versions: password manager, dotfiles, config files etc. are a) versioned in my private Gitlab, also b) Nextcloud will keep versions and c) some of these files are synced to my phone; I also print out the most precious keys yearly and store those at a 'secure' location - here's a special tip: Backup your OTP codes! I realized this recently: Google Authenticator will not create backups. You can migrate your OTP codes to Aegis Authenticator [1], which allows you to create automatic, encrypted backups that can be automatically synced using Nextcloud [1]: https://f-droid.org/en/packages/com.beemdevelopment.aegis/ https://f-droid.org/en/packages/com.beemdevelopment.aegis/
Havoc 4y ago
I’ve taken to keeping as much of my info as possible in flat text files and flowing that into git. That in turn is easy to replicate and protect against ransomware and the like Big media files are a bit trickier

97 points by CHB0403085482 4y ago

To add to this: having the backups copy somewhere that has write-only permissions can be a life-saver in a crypto-ransom malware situation. Hackers are smart enough to delete or encrypt backups, and this is made far easier for them when admins have full read/write permissions to the backup location.

bombcar 4y ago
Offline backups are very important - not only because they can't be modified because they're disconnected, but because they force you to spend time going and getting them and that time can be important. Many backups are successfully destroyed in trying to back them up, and spending some time to think about what is happening can be worth it.
SahAssar 4y ago
rsync.net is great for this. They keep ZFS snapshots, so even though you have read/write access you will always have the snapshots stored. So you will be safe at least as long as you notice the corruption within the normal 7-day rolling window (there are options for other windows or manual snapshots too IIRC). This should also lead to the question of how up-to-date your backups need to be and is your solution setup for that (is it 1s, 1m, 1h, 1d, etc.)? Define an actual SLA for how much data you can loose and test for that.
Noumenon72 4y ago
Does "write-only permissions" somehow imply they can't overwrite the backup with an encrypted copy? I don't understand how read permissions would allow them to delete or replace something.
- mgdlbp 4y ago
  GP meant write once, read many
- ok_dad 4y ago
  I think the OP means something more like "immutable" or "copy on write", so you would never modify or delete previous data (or at least separate the modifications/deletions to a more-secure system).
WalterBright 4y ago
Yeah, it's a crying shame that hard drives no longer come with physical read-only switches. I have no idea why major drive users don't demand this from the makers.
- dsr_ 4y ago
  At the desktop level, this would result in a whole new class of useless support calls. At the SOHO server level, this might be useful. But the alternatives are already available: unmount the disk, or unmount and unplug it, or unmount, unplug, and carry over to the safe, or unmount, unplug, and carry to the backup site... At the datacenter level, nobody has time to physically touch things.
  - WalterBright 4y ago
    If your drive is write-enabled, and you carry it over to your system to restore from backup, then the ransomware can intervene and encrypt it first. Besides, for the bozos, ship the drives with the write-enable turned on. They won't notice the difference.
justin_oaks 4y ago
The solution for this that I've been working with is having an S3 bucket with versioning enabled. The systems that send backups to that S3 bucket don't have permissions to modify previous versions. If one or all of those systems are compromised then they can't overwrite previous backups because the original data will still exist in old versions.

darkteflon 4y ago

The bigger story here is: checklists, checklists, checklists. They’re underrated!

ghaff 4y ago
And automation. So long as you confirm that the automation is doing what you think it's doing. Another comment mentioned an offline backup. And I agree. But it's also easy to go from having a month old offline backup to a year old to...
ElevenLathe 4y ago
Do you know of a good app for doing checklists of this type? Many TODO-list apps market themselves as "checklist apps" but they really just mean "yeah we've got checkboxes". They don't mean "our app lets you choose from a library of checklists for common tasks and pull a fresh one to tick off when you want to perform one of those tasks." One that seems close is ForeFlight (demo: https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s), but it's just one feature of a much larger app, is mobile only, and obviously targeted at aviation. I'm looking for similar functionality but more IT-oriented (or at least generic, not referring to airplane models, etc.) and preferably with an offline desktop "client" (it is especially critical that you have your checklist library available if the whole corporate network is down, for example) or, barring that, at least a desktop-friendly web UI. Honestly, if I knew there was a market for this, I would build it. It's either a great opportunity or the market isn't big enough.
- WalterBright 4y ago
  > Do you know of a good app for doing checklists of this type? 1. Type it into a file. 2. Print it onto a piece of paper 3. Put a mark by each item with a pen as you finish it.
  - ChrisMarshallNY 4y ago
    This. I have a small sticky note pad on my desk. When we have a meeting about features, I jot down my todos. In some cases, I may do a screengrab, or submit a bug report (to myself), but, most times, it’s the PostIt. When I’m done with a task, I crumple the note, and drag it into the trash can.

yohannparis 4y ago

A good trick to convince oneselves to backup and the cost of redoing the work lost. Sometimes, just a couple of pages of documents, or a setup script, is worth weeks of work, ressources, money, etc. And it's nothing compare to the costs of creating multiple backups.

AyyWS 4y ago
Makes sense to backup user data, and script your server/app/database install and setup. If your RTO is very short, then full backups with fast restores are best.

neilv 4y ago

In an early startup, where developers are also doing ops, and there are periodic manual tasks (like daily/weekly/monthly running certain kinds of backups, eyeballing AWS dashboards and logs to check for anything alerts missed, etc.), using the same board and issue-tracker you use for development tasks helps. In one company, I used GitLab Board, and had special tags for the recurring and background tasks, which were sticky in Kanban columns right below the generally one-time development tasks. The deadline dates for recurring tasks kept changing to the next time the task had to be done. The Issue description had the concise immediate instructions for the steps of the task (with a link to the wiki for related and background info). And every time you did the task, you added a very quick Comment with any notes, and bumped the due date. It was very lightweight, and fit the simplicity I was advocating (e.g., the GitLab Kanban board says what everyone is doing, all info goes into GitLab in one form or another, minimal other tools and silos to juggle, etc.).

thrdbndndn 4y ago
> Kanban board Does it have specific meaning in software engineering? Isn't Kanban just billboard in Japanese?
- SahAssar 4y ago
  Yes, it's usually a column-based system for tracking progress of tasks that move through various stages (eg. planning, development, testing, rollout, evaluation). See here: https://en.wikipedia.org/wiki/Kanban_board https://en.wikipedia.org/wiki/Kanban_board
  - neilv 4y ago
    Exactly. Sometimes my columns aren't waterfall phases, but more: Backlog, Blocked, Active, Done, Abandoned. The "Blocked" is important to highlight on the board: when someone can't do their thing, no sense floundering quietly -- highlight that, see who can unblock it, and what anyone else can do in the meantime. "Abandoned" is another one that I think is important, but might be unusual. If the task no longer makes sense, stop immediately, move it to Abandoned, and move on to a more worthwhile task. Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore. Later, we can reflect on what info we had, when, and what we can learn from that. (I usually really-really want a Gantt model as well, on the same database that the board view is coming from, but the tools aren't quite there yet.)

GnarfGnarf 4y ago

I backup to DVDs. I have every backup for the last ten years. And yes, I can still read them.

ttkari 4y ago
> And yes, I can still read them. Just curious - how often do you test this?
- GnarfGnarf 4y ago
  I just tested 6 CDs from 1998. I can read all of them. One CD took a little persuading. They have been kept in a cool, dark place (my basement). I write an MD5 checksum for each file so they can be verified at any time in the future.

zie 4y ago

If you are backing up users data that is not yours to own and do with as you please, yes you absolutely can have to many backups. If you are required by law or policy to erase user data and it's in your backup(s), you now have a real problem on your hands. Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this.

JadeNB 4y ago
> Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this. I am confused by these sentences. Sometimes the law does it, but you're not aware of a law that does it?
- alliao 4y ago
  laws have jurisdiction so he's probably saying he doesn't know all laws from all jurisdictions?
- zie 4y ago
  Essentially yes. I can't know all laws for all jurisdictions(nor would I want to), but laws could make a carve-out provision. I'm not currently aware of any that matter in my jurisdiction(s). That doesn't mean there aren't some out there somewhere. Policies are of course a different thing, and they may or may not apply to you, from industry regulations or cyber security insurance or just company or trade group policies that might apply.
bombcar 4y ago
GDPR apparently does: https://www.itgovernance.eu/blog/en/the-gdpr-how-the-right-to-be-forgotten-affects-backups-2 https://www.itgovernance.eu/blog/en/the-gdpr-how-the-right-t... If data is on backup tapes you seem to be able to inform them of how long your retention policies are.
- msh 4y ago
  There is a bit more than just informing users: If you decide to go down this route, you should bear in mind that other supervisory authorities might be stricter and that you must be able to demonstrate that it’s impractical to delete backup data. This will require, at the very least, a risk assessment, business impact assessment and data protection impact assessment.

015a 4y ago

Early in one of my companies, we had a cloud hosted database provider that only provided (at our meager level of spend) 7 days of daily backups for the main db. We had a specific customer, one of maybe ten, and a higher percentage of our revenue, who made an big bulk import oopsie-whoopsie; and given the specifics of this software product, did not notice for weeks. They wanted everything back to the way it was. We ended up being saved by our traditionally "crappy" local dev process; me and the other dev had copies of the prod db on our development machines. And, naturally, we rarely updated them; the last time I had sync'ed it down was, conveniently, about four weeks ago. So they had some data loss between that "backup" and the oopsie-whoopsie, but far less major than their issues with the current state. The most interesting thing to really come out of that was a set of tooling to precision restore an individual customer's data from a backup of this multi-tenanted database. That was something I see as pretty rare for engineers to consider in their backup strategy; you may have the backup, it may be easily restored, but it will take the world with it and maybe you don't want the whole world. We ended up using that a half dozen or more times over the next few years... against a far more comprehensive set of backups of course. Yeah, maybe it makes sense to store a history of every change made to every row; but for our needs it made a lot more sense to leverage backups and some glue scripts.

bombcar 4y ago
One of the important considerations of database design is “what might I want to restore” and there are often way more answers than just “how it was at a point in time”. And some of those restores should happen in the application layer not the db backup.
- silvestrov 4y ago
  100% of my restores have been due to user/developer fuckups. 0% has been because of hardware failures. This is why I love that PostgreSQL backup files are plain text files. It is so easy to extract a single table (or parts of a table) using a simple perl/python/whatever script.
andrewf 4y ago
Sounds like the thing Atlassian had to build while on fire this year.

outside1234 4y ago

Or practice restoring too many times

Ologn 4y ago

Remember the old 3-2-1 backup rule 3 recent copies of important data, in 2 different media formats, with 1 copy off-site

thematrixturtle 4y ago

Once upon a time, a Japanese megacorp was designing a new factory. The master plans for the factory were stored on a USB thumb drive owned by the Chief Engineer, who was the only person authorized to make changes to the master plan. The Chief Engineer and crew went to inspect another factory for inspiration. There was a direct flight to the city, but by corporate policy they had to use the preferred airline, requiring a transfer. The Chief Engineer didn't want to lug his heavy laptop around, so he checked it in, with the thumbdrive still plugged in. Chief Engineer and crew arrived successfully at the factory. His suitcase did not. Enquiries with the airline told them the suitcase had been shipped to the wrong place and would be returned very soon. And that was the last they or the airline ever saw of it. There was no central server, no versioning, no backups -- the plans were now gone. Fortunately, since this was a Japanese company, the plans were printed out weekly for review at the status update meeting. So the plans were manually reconstructed from paper printouts, page by page, line by line, character by character, and one of the many, many people typing it in was the friend who told me this story. Unsurprisingly, they no longer work at Megacorp.

wodenokoto 4y ago
Something about that story reminds me of toy story being deleted and a single animator had taken a copy of the _entire_ project home to work on it from home. Another good one is Maersk. When they suffered a ransomware attack that deleted all access to all backups/replication servers, the only surviving backup was a replication server that happened to be offline during the attack, because it was situated in a country with very unstable internet and power supply.
MontyCarloHall 4y ago
> the Chief Engineer was the only person authorized to make changes to the master plan. > the plans were printed out weekly for review at the status update meeting. This is so quintessentially Japanese. Have a single point of failure in a rigid chain of command structure, but bake in a ton of paper redundancy. Nonetheless, I bet the lost laptop was an extremely advanced model. In the early 2000s, my dad worked with some Japanese collaborators. They brought over some impossibly thin laptops that seemed a decade ahead of what you could get in the US at the time…and proceeded to use them to send faxes.

sebazzz 4y ago

I got my sysops backups monitored in Nagios. A script that runs in NSClient++ checks the backup date, size, and the backup log for errors. Still, you also need to inspect the backup manually from time to time - just to be sure.

Helmut10001 4y ago

Agree! These are some of my rule of thumbs: - write all commands, ever entered to any shell, down. Of course, this also includes the procedure to create backups - do not treat all data equal. There's 'fast' (but small) data that needs more backups more often and there is 'slow' (but large) data that is infrequently updated, thus less often backed up. - 3-2-1 rule is a minimal setup; extend it wherever possible. Also, use 'migration periods' to replace one backup system with another; during this period, you use both backup systems until you gained enough trust in the new one to deprecate the old one. This should include actually testing restoring systems from backups in VMs at least once. My private 3-2-1+: - immediate sync of files to my internal nextcloud server, hosted offsite. That means any files exist at least once in to different locations - daily ZFS automatic snapshots at the server side and deprecation of old ones - weekly, my server starts another Odroid CS2 at another offsite location and rsync's any changes; the Odroid is shutdown afterwards - yearly, I attach a 16TB usb drive and send ZFS snapshots. - in addition to these 'basic' 3-2-1, the '+' means: daily, Nextcloud's document folder is duplicated to a different ZFS dataset on the server which is not actively write-referenced in any VM; Also, for some very important files, I keep more versions: password manager, dotfiles, config files etc. are a) versioned in my private Gitlab, also b) Nextcloud will keep versions and c) some of these files are synced to my phone; I also print out the most precious keys yearly and store those at a 'secure' location - here's a special tip: Backup your OTP codes! I realized this recently: Google Authenticator will not create backups. You can migrate your OTP codes to Aegis Authenticator [1], which allows you to create automatic, encrypted backups that can be automatically synced using Nextcloud [1]: https://f-droid.org/en/packages/com.beemdevelopment.aegis/ https://f-droid.org/en/packages/com.beemdevelopment.aegis/

Havoc 4y ago

I’ve taken to keeping as much of my info as possible in flat text files and flowing that into git. That in turn is easy to replicate and protect against ransomware and the like Big media files are a bit trickier