6 ms·

The number of commenters that think the KDE sysadmins were stupid enough to not know that "raid isn't a backup strategy" is depressing. You either didn't read t

The number of commenters that think the KDE sysadmins were stupid enough to not know that "raid isn't a backup strategy" is depressing. You either didn't read the article or you haven't understood it. Git repositories contain redundant information to perform consistency checks. If a bit flips in one of your repos, git fsck will immediately catch it. The KDE sysadmins thought these consistency checks were triggered when keeping a repository clone in sync and thus any FS level corruption would be caught at the first subsequent attempt to sync. If you think "Gee, if I know this, surely they do?!" then perhaps the answer is: "They probably do and this issue is more subtle than you initially thought. Reread carefully before implying how dumb others were."

pja 14y ago
Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. Which is why mirroring (of any description) is not a backup strategy. Yes, git --mirror should probably automatically invoke the moral equivalent of git fsck by default in order to catch internal repository inconsistency like the other git commands do, and the KDE team have been caught out by this. But they still don't have any protection against user error leading to loss of data with this setup as far as I can see, and that seems like a huge oversight.
- richo 14y ago
  Git doesn't nuke objects until you explicitly ask it to. It might sync broken objects, but it won't actually chose to destroy dangling objects without being asked to.
  - DougBTX 14y ago
    They say in the article that one step of the anongit sync is to delete projects from the anongit servers which have been deleted on the central server. If a user accidentally deletes a project, they should be able to recover it from a backup, but it doesn't sound like there are any backups. The same idea applies to the individual repos themselves. A git gc will delete objects unreferenced by the reflogs, eg deleted branches over a month old by default. They say that the total set of repos is about 25GB, so it would be reasonable to keep at least monthly backups for at a year or more.
    - sho_hn 14y ago
      We do ref backups on branch deletions FWIW.
      - DougBTX 14y ago
        Yea, that sounds like a good idea, can you do that in something like a pre-push hook? That plus long reflogs seems like it would make recovery of rebased/deleted branches really easy for end users.
        sho_hn 14y ago
        We don't allow force-pushes without admin involvement, so rebasing isn't so scary for us, but we do cover it that way. And we got the undo operation down to a command on the infra: http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes The rest of that Wiki page is also pretty interesting for a general look at what devs can do on git.kde.org - server-side personal clones, personal repositories, a personal trashcan for repositories, etc. We don't have a pretty web interface for it (yet?), but it has some nice features.
  - pja 14y ago
    (This is missing the wider point, but regardless: AFAIK (do correct me if I'm wrong) git clone --mirror will not copy dangling references & those refs will disappear from the original after a git gc, whenever that happens.) Human error is human error: sometimes people will do apparently egregiously stupid things if it's possible for them to do so, including things like deleting branches from a repository & then doing a git gc to "save space" shortly before realising that they weren't issuing commands to the shell they thought they were in. A backup strategy should be robust against dumb human mistakes as well as mechanical hardware errors, otherwise it's no backup strategy at all.
- Confusion 14y ago
  Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. There is no equivalent of rm -rf in git. If you remove files being versioned and commit them, the previous version still exists and the commit can be reverted. If you remove the metadata, no sync can happen, you are immediately warned (assuming good monitoring was set up) and can restore from one of the clones. If the entire machine crashes and burns, you whip out your git-server-create.sh, which provisions a new VM and restores from one of the clones. There is no additional data to be backup up with git. Any single developer with a recent clone of the repos can setup a new master git server.
  - tomp 14y ago
    git checkout master git reset --hard HEAD^ git push --force origin master will remove the latest commit. If the mirrored repositories then blindly use the origin's refs, and do a `git gc`, the latest commit will be removed.
    - Confusion 14y ago
      It is impossible to blindly use the origin refs in that case. Git doesn't deal well with rewriting history. If someone force pushes to a centrally shared repo, all hell breaks loose.
      - pyre 14y ago
        From the article: | New refs are pulled down to the downstream | repository; all updates, including forced updates, | are mirrored to the downstream repository. | As a result, making a mirror clone essentially | bypasses the safety checks in the repository And > If someone force pushes to a centrally shared > repo, all hell breaks loose. This would happen to users pulling/pushing from the central repo, but not to the mirrors.
        Confusion 14y ago
        In that case another developer would notice, would get in touch with his fellow developers, they would agree to force push a specific repository condition and all is well and back to normal. There is no need to restore anything from backups or mirrors in that scenario. So that's an entirely different threat. On the one hand we have FS corruption, which is basically bound to happen and is likely to impact many repositories. On the other hand we have someone (maliciously) force pushing an update on a single repository that throws away all commits. The first issue is one that needs a backup and restore procedure. The second is one that is at best an inconvenience and at worst a security problem. Unless you want to include the the worst case scenario someone gaining access to the machine and force pushing such a destructive update on all repos. In that case restoring from developer copies may actually be safer anyway, because the machine and all its backups, be they mirrors or otherwise, may be considered compromised. That's a nightmare for which this discussion about backups is just quibbling.
- michaelt 14y ago
  Surely every backup system has some equivalent of an rm -rf ? A disgruntled employee could phone the off site tape archival company and tell them to toss all the tapes in the shreader.
  - chris_wot 14y ago
    The probability of this occurring is far less than what happened here.
  - reeses 14y ago
    In the specific case of offsite tape archival, let's say someone who has the authority to do so requests that they destroy all tapes. Most service providers (who stay in business) have enough compliance features in place so that multiple authorized people have to be in collusion, a sufficiently senior executive must make the request, or there will be a "package" ready to be served to the client and relevant policing unit (police, FBI, whatever) so that charges can be brought against the malefactor efficiently. While you may be limited in financial compensation by your contract with the service provider, it is absolutely in their best interest to avoid the situation with procedures (they do not want to be a party to a crime) and if those fail, to provide extensive records of the movement and disposition of those tapes.
- mpyne 14y ago
  > Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. The master repository is coded so that those types of actions are not permitted. Developers cannot force push or delete branches that aren't reintegrated into some other head, without assistance from the sysadmins. After every routine push to a KDE git repository the new commit will be a descendant of the repo's original head. With that in mind it's more understandable as to why they felt it was possible to take advantage of Git's design in the backup planning. No "huge oversight" here, at least on this particular issue.
  - andrewvc 14y ago
    Ummm, still not a backup. What if there's an error in the code? What if an admin runs a script directly on the server and resets all repos by mistake? What if a bad release of git comes out and corrupts the master? What if a hacker comes in and wipes them all out? Only a backup is a backup. Backup, like security, is somewhat onion like.
    - derleth 14y ago
      What if your backup systems get compromised? What if your backup systems get destroyed by inclement weather? Not a backup then, obviously. Only a backup is a backup, until it's not a backup anymore.
      - chris_wot 14y ago
        That's kind of ridiculous. The point that most people are making is that if someone does something incredibly stupid, or there is corruption in the system that follows down the line (like what happened here), it doesn't matter whether you have a repository. A backup clearly would have helped here.
        mpyne 14y ago
        A backup of a corrupt repository would have been just as corrupt though. This is the big thing I can't figure out what people are not understanding. git does consistency checking for you already, tar|rsync|etc. don't, so it makes sense to take advantage of that. What we had was an instance of some of the underlying data becoming corrupt on the filesystem (with indications of that starting on Feb 22!). The big mistake was considering the source repositories as consistent and canonical at the remote anongit end, but the data would have been just as corrupt if we had scp'ed the repos from git.kde.org to the anongit mirrors around the world, since we would have bypassed git's internal checking in that way. Is it safe to rsync a running mysql database at random times, or are you supposed to use mysql-provided tools to perform a backup?
3amOpsGuy 14y ago
I think you might miss the point here, in the same way as the KDE SAs did. What folks tend to consider the meat of ops work, often boils to a big ole boring checklist. The problem is that you shouldn't just elect to skip a whole big section without some seriously good reasoning. This isn't a slur on you or the KDE guys, hindsight is 20/20. I'm confident though that I'm not alone, that there are plenty of other Ops folks here who read the story and also felt the described setup violated a deep principle and just made feel ill at ease. These failure scenarios are not common, but the do happen often enough that we know to prepare for them. As an example I'd point to how DBAs handle validation of replication - it's the same principle here. Just for completeness, an example reason for not having proper restore procedures in place might be 'this is not the prime record copy of the data and it takes less than 24h to regenerate this data therefore this will be out of scope during restore tests'.
- sho_hn 14y ago
  > I think you might miss the point here, in the same way as the KDE SAs did. Who're clearly not under the impression they can't make mistakes considering TFA is a write-up of design flaws in a mirroring system :). Look at it this way: If you think something obvious was overlooked, then it's good there's another report backing up your point. That's the value in everyone being open about their operations and experiences along the way - you only get better metrics for what works and what doesn't, in practice.
  - DougBTX 14y ago
    Yea, it is good to see a writeup like this. I'm sure some of the servers I work with don't have proper backups, but I was cringing all the same waiting for there to be a discussion about why the central git server itself couldn't be restored from a backup.
- Confusion 14y ago
  I don't understand what you are referring to. What 'whole big section' was skipped? What 'deep principle' was violated?
stiff 14y ago
I think the lesson we should all reflect on here is that when designing very critical systems one should not try to be too clever or optimize to the point of the correct working of the whole system being dependent on a single assumption. Old-school engineering used to have the concept of a safety factor, where you used a material twice as strong as the one that your equations have shown to be strong enough to sustain the conditions you expect it to operate under. Richard Hamming also put this very nicely: "Would you fly a plane if you knew it depended on some function being Lebesgue-integrable?". Was the scenario that something unwanted might get synced one day really that improbable or so unobvious nobody considered it? Hard to say in retrospect, but I would say no. Anyway, this might happen to every one of us, so I will use the time to reflect on the systems _I_ built and whatever assumptions _I_ could have done mistakingly.
Ologn 14y ago
"think the KDE sysadmins were stupid" The problem is that they were not stupid, they were too clever for their own good. A stupid sysadmin would follow the 3-2-1 backup rule and have on hand at least 3 backups of the data from the past 24 hours (or sooner, if backup strategy necessitated it). The KDE sysadmins and devs thought they were super clever and could avoid this rule from operations 101, and then found out the hard way that they could not. This is really 2 separate discussions. One is the intricacies of git. The other is ignoring operations 101 by not making proper backups. Having over a decade in ops experience, I am more focused on the latter discussion. As someone else here said, if they had followed a braindead checklist of how to do proper backups they would not have had this problem. Standard operations procedure will save you, even if you had made faulty assumptions about your architecture. The main point is this: they broke an ops 101 rule. They had clever reasons and assumptions for doing so. These assumptions turned out to be wrong. Now people are arguing the point. It does not change the main point, that if they had followed the ops 101 backup rule, possible disaster would not have been knocking on the door.
- unoti 14y ago
  The fascinating thing to me about the ops 101 rule you describe: 1) it's so obviously correct, but 2) for some reason, almost nobody has a good backup plan that they actually follow, where they actually have a process to periodically validate the restore procedure works. So few people actually do this, that I've come to think of this as a basic rule to understand, but an apparently advanced one to actually follow. You hit the nail on the head regarding overabundant cleverness. A lot of developers are too clever by half, too.
  - GFischer 14y ago
    I worked for a company that actually did periodical restores of backups, and had a smaller scale version of the "Chaos Monkey" (per company procedures, someone would periodically delete on purpose a file at random - it could be part of an important product, documentation, accounts, an SQL database - and we were asked to get it back from backup). After I got to the "real world", I'm shocked that some extremely large companies (financial institutions that should know better) don't have anything like it in place, and have far worse and untested but expensive disaster recovery plans.

pja 14y ago

Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. Which is why mirroring (of any description) is not a backup strategy. Yes, git --mirror should probably automatically invoke the moral equivalent of git fsck by default in order to catch internal repository inconsistency like the other git commands do, and the KDE team have been caught out by this. But they still don't have any protection against user error leading to loss of data with this setup as far as I can see, and that seems like a huge oversight.

richo 14y ago
Git doesn't nuke objects until you explicitly ask it to. It might sync broken objects, but it won't actually chose to destroy dangling objects without being asked to.
- DougBTX 14y ago
  They say in the article that one step of the anongit sync is to delete projects from the anongit servers which have been deleted on the central server. If a user accidentally deletes a project, they should be able to recover it from a backup, but it doesn't sound like there are any backups. The same idea applies to the individual repos themselves. A git gc will delete objects unreferenced by the reflogs, eg deleted branches over a month old by default. They say that the total set of repos is about 25GB, so it would be reasonable to keep at least monthly backups for at a year or more.
  - sho_hn 14y ago
    We do ref backups on branch deletions FWIW.
    - DougBTX 14y ago
      Yea, that sounds like a good idea, can you do that in something like a pre-push hook? That plus long reflogs seems like it would make recovery of rebased/deleted branches really easy for end users.
      - sho_hn 14y ago
        We don't allow force-pushes without admin involvement, so rebasing isn't so scary for us, but we do cover it that way. And we got the undo operation down to a command on the infra: http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes The rest of that Wiki page is also pretty interesting for a general look at what devs can do on git.kde.org - server-side personal clones, personal repositories, a personal trashcan for repositories, etc. We don't have a pretty web interface for it (yet?), but it has some nice features.
- pja 14y ago
  (This is missing the wider point, but regardless: AFAIK (do correct me if I'm wrong) git clone --mirror will not copy dangling references & those refs will disappear from the original after a git gc, whenever that happens.) Human error is human error: sometimes people will do apparently egregiously stupid things if it's possible for them to do so, including things like deleting branches from a repository & then doing a git gc to "save space" shortly before realising that they weren't issuing commands to the shell they thought they were in. A backup strategy should be robust against dumb human mistakes as well as mechanical hardware errors, otherwise it's no backup strategy at all.
Confusion 14y ago
Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. There is no equivalent of rm -rf in git. If you remove files being versioned and commit them, the previous version still exists and the commit can be reverted. If you remove the metadata, no sync can happen, you are immediately warned (assuming good monitoring was set up) and can restore from one of the clones. If the entire machine crashes and burns, you whip out your git-server-create.sh, which provisions a new VM and restores from one of the clones. There is no additional data to be backup up with git. Any single developer with a recent clone of the repos can setup a new master git server.
- tomp 14y ago
  git checkout master git reset --hard HEAD^ git push --force origin master will remove the latest commit. If the mirrored repositories then blindly use the origin's refs, and do a `git gc`, the latest commit will be removed.
  - Confusion 14y ago
    It is impossible to blindly use the origin refs in that case. Git doesn't deal well with rewriting history. If someone force pushes to a centrally shared repo, all hell breaks loose.
    - pyre 14y ago
      From the article: | New refs are pulled down to the downstream | repository; all updates, including forced updates, | are mirrored to the downstream repository. | As a result, making a mirror clone essentially | bypasses the safety checks in the repository And > If someone force pushes to a centrally shared > repo, all hell breaks loose. This would happen to users pulling/pushing from the central repo, but not to the mirrors.
      - Confusion 14y ago
        In that case another developer would notice, would get in touch with his fellow developers, they would agree to force push a specific repository condition and all is well and back to normal. There is no need to restore anything from backups or mirrors in that scenario. So that's an entirely different threat. On the one hand we have FS corruption, which is basically bound to happen and is likely to impact many repositories. On the other hand we have someone (maliciously) force pushing an update on a single repository that throws away all commits. The first issue is one that needs a backup and restore procedure. The second is one that is at best an inconvenience and at worst a security problem. Unless you want to include the the worst case scenario someone gaining access to the machine and force pushing such a destructive update on all repos. In that case restoring from developer copies may actually be safer anyway, because the machine and all its backups, be they mirrors or otherwise, may be considered compromised. That's a nightmare for which this discussion about backups is just quibbling.
michaelt 14y ago
Surely every backup system has some equivalent of an rm -rf ? A disgruntled employee could phone the off site tape archival company and tell them to toss all the tapes in the shreader.
- chris_wot 14y ago
  The probability of this occurring is far less than what happened here.
- reeses 14y ago
  In the specific case of offsite tape archival, let's say someone who has the authority to do so requests that they destroy all tapes. Most service providers (who stay in business) have enough compliance features in place so that multiple authorized people have to be in collusion, a sufficiently senior executive must make the request, or there will be a "package" ready to be served to the client and relevant policing unit (police, FBI, whatever) so that charges can be brought against the malefactor efficiently. While you may be limited in financial compensation by your contract with the service provider, it is absolutely in their best interest to avoid the situation with procedures (they do not want to be a party to a crime) and if those fail, to provide extensive records of the movement and disposition of those tapes.
mpyne 14y ago
> Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. The master repository is coded so that those types of actions are not permitted. Developers cannot force push or delete branches that aren't reintegrated into some other head, without assistance from the sysadmins. After every routine push to a KDE git repository the new commit will be a descendant of the repo's original head. With that in mind it's more understandable as to why they felt it was possible to take advantage of Git's design in the backup planning. No "huge oversight" here, at least on this particular issue.
- andrewvc 14y ago
  Ummm, still not a backup. What if there's an error in the code? What if an admin runs a script directly on the server and resets all repos by mistake? What if a bad release of git comes out and corrupts the master? What if a hacker comes in and wipes them all out? Only a backup is a backup. Backup, like security, is somewhat onion like.
  - derleth 14y ago
    What if your backup systems get compromised? What if your backup systems get destroyed by inclement weather? Not a backup then, obviously. Only a backup is a backup, until it's not a backup anymore.
    - chris_wot 14y ago
      That's kind of ridiculous. The point that most people are making is that if someone does something incredibly stupid, or there is corruption in the system that follows down the line (like what happened here), it doesn't matter whether you have a repository. A backup clearly would have helped here.
      - mpyne 14y ago
        A backup of a corrupt repository would have been just as corrupt though. This is the big thing I can't figure out what people are not understanding. git does consistency checking for you already, tar|rsync|etc. don't, so it makes sense to take advantage of that. What we had was an instance of some of the underlying data becoming corrupt on the filesystem (with indications of that starting on Feb 22!). The big mistake was considering the source repositories as consistent and canonical at the remote anongit end, but the data would have been just as corrupt if we had scp'ed the repos from git.kde.org to the anongit mirrors around the world, since we would have bypassed git's internal checking in that way. Is it safe to rsync a running mysql database at random times, or are you supposed to use mysql-provided tools to perform a backup?

3amOpsGuy 14y ago

I think you might miss the point here, in the same way as the KDE SAs did. What folks tend to consider the meat of ops work, often boils to a big ole boring checklist. The problem is that you shouldn't just elect to skip a whole big section without some seriously good reasoning. This isn't a slur on you or the KDE guys, hindsight is 20/20. I'm confident though that I'm not alone, that there are plenty of other Ops folks here who read the story and also felt the described setup violated a deep principle and just made feel ill at ease. These failure scenarios are not common, but the do happen often enough that we know to prepare for them. As an example I'd point to how DBAs handle validation of replication - it's the same principle here. Just for completeness, an example reason for not having proper restore procedures in place might be 'this is not the prime record copy of the data and it takes less than 24h to regenerate this data therefore this will be out of scope during restore tests'.

sho_hn 14y ago
> I think you might miss the point here, in the same way as the KDE SAs did. Who're clearly not under the impression they can't make mistakes considering TFA is a write-up of design flaws in a mirroring system :). Look at it this way: If you think something obvious was overlooked, then it's good there's another report backing up your point. That's the value in everyone being open about their operations and experiences along the way - you only get better metrics for what works and what doesn't, in practice.
- DougBTX 14y ago
  Yea, it is good to see a writeup like this. I'm sure some of the servers I work with don't have proper backups, but I was cringing all the same waiting for there to be a discussion about why the central git server itself couldn't be restored from a backup.
Confusion 14y ago
I don't understand what you are referring to. What 'whole big section' was skipped? What 'deep principle' was violated?

stiff 14y ago

I think the lesson we should all reflect on here is that when designing very critical systems one should not try to be too clever or optimize to the point of the correct working of the whole system being dependent on a single assumption. Old-school engineering used to have the concept of a safety factor, where you used a material twice as strong as the one that your equations have shown to be strong enough to sustain the conditions you expect it to operate under. Richard Hamming also put this very nicely: "Would you fly a plane if you knew it depended on some function being Lebesgue-integrable?". Was the scenario that something unwanted might get synced one day really that improbable or so unobvious nobody considered it? Hard to say in retrospect, but I would say no. Anyway, this might happen to every one of us, so I will use the time to reflect on the systems _I_ built and whatever assumptions _I_ could have done mistakingly.

Ologn 14y ago

"think the KDE sysadmins were stupid" The problem is that they were not stupid, they were too clever for their own good. A stupid sysadmin would follow the 3-2-1 backup rule and have on hand at least 3 backups of the data from the past 24 hours (or sooner, if backup strategy necessitated it). The KDE sysadmins and devs thought they were super clever and could avoid this rule from operations 101, and then found out the hard way that they could not. This is really 2 separate discussions. One is the intricacies of git. The other is ignoring operations 101 by not making proper backups. Having over a decade in ops experience, I am more focused on the latter discussion. As someone else here said, if they had followed a braindead checklist of how to do proper backups they would not have had this problem. Standard operations procedure will save you, even if you had made faulty assumptions about your architecture. The main point is this: they broke an ops 101 rule. They had clever reasons and assumptions for doing so. These assumptions turned out to be wrong. Now people are arguing the point. It does not change the main point, that if they had followed the ops 101 backup rule, possible disaster would not have been knocking on the door.

unoti 14y ago
The fascinating thing to me about the ops 101 rule you describe: 1) it's so obviously correct, but 2) for some reason, almost nobody has a good backup plan that they actually follow, where they actually have a process to periodically validate the restore procedure works. So few people actually do this, that I've come to think of this as a basic rule to understand, but an apparently advanced one to actually follow. You hit the nail on the head regarding overabundant cleverness. A lot of developers are too clever by half, too.
- GFischer 14y ago
  I worked for a company that actually did periodical restores of backups, and had a smaller scale version of the "Chaos Monkey" (per company procedures, someone would periodically delete on purpose a file at random - it could be part of an important product, documentation, accounts, an SQL database - and we were asked to get it back from backup). After I got to the "real world", I'm shocked that some extremely large companies (financial institutions that should know better) don't have anything like it in place, and have far worse and untested but expensive disaster recovery plans.