3 ms·
"think the KDE sysadmins were stupid" The problem is that they were not stupid, they were too clever for their own good. A stupid sysadmin would follow the 3-
by Ologn 14y ago
"think the KDE sysadmins were stupid"
The problem is that they were not stupid, they were too clever for their own good. A stupid sysadmin would follow the 3-2-1 backup rule and have on hand at least 3 backups of the data from the past 24 hours (or sooner, if backup strategy necessitated it). The KDE sysadmins and devs thought they were super clever and could avoid this rule from operations 101, and then found out the hard way that they could not.
This is really 2 separate discussions. One is the intricacies of git. The other is ignoring operations 101 by not making proper backups. Having over a decade in ops experience, I am more focused on the latter discussion.
As someone else here said, if they had followed a braindead checklist of how to do proper backups they would not have had this problem. Standard operations procedure will save you, even if you had made faulty assumptions about your architecture.
The main point is this: they broke an ops 101 rule. They had clever reasons and assumptions for doing so. These assumptions turned out to be wrong. Now people are arguing the point. It does not change the main point, that if they had followed the ops 101 backup rule, possible disaster would not have been knocking on the door.
- unoti 14y agoThe fascinating thing to me about the ops 101 rule you describe: 1) it's so obviously correct, but 2) for some reason, almost nobody has a good backup plan that they actually follow, where they actually have a process to periodically validate the restore procedure works. So few people actually do this, that I've come to think of this as a basic rule to understand, but an apparently advanced one to actually follow. You hit the nail on the head regarding overabundant cleverness. A lot of developers are too clever by half, too.
- GFischer 14y agoI worked for a company that actually did periodical restores of backups, and had a smaller scale version of the "Chaos Monkey" (per company procedures, someone would periodically delete on purpose a file at random - it could be part of an important product, documentation, accounts, an SQL database - and we were asked to get it back from backup). After I got to the "real world", I'm shocked that some extremely large companies (financial institutions that should know better) don't have anything like it in place, and have far worse and untested but expensive disaster recovery plans.