5 ms·

Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. Which is why mirroring (of any descriptio

by pja 14y ago

Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. Which is why mirroring (of any description) is not a backup strategy. Yes, git --mirror should probably automatically invoke the moral equivalent of git fsck by default in order to catch internal repository inconsistency like the other git commands do, and the KDE team have been caught out by this. But they still don't have any protection against user error leading to loss of data with this setup as far as I can see, and that seems like a huge oversight.

richo 14y ago
Git doesn't nuke objects until you explicitly ask it to. It might sync broken objects, but it won't actually chose to destroy dangling objects without being asked to.
- DougBTX 14y ago
  They say in the article that one step of the anongit sync is to delete projects from the anongit servers which have been deleted on the central server. If a user accidentally deletes a project, they should be able to recover it from a backup, but it doesn't sound like there are any backups. The same idea applies to the individual repos themselves. A git gc will delete objects unreferenced by the reflogs, eg deleted branches over a month old by default. They say that the total set of repos is about 25GB, so it would be reasonable to keep at least monthly backups for at a year or more.
  - sho_hn 14y ago
    We do ref backups on branch deletions FWIW.
    - DougBTX 14y ago
      Yea, that sounds like a good idea, can you do that in something like a pre-push hook? That plus long reflogs seems like it would make recovery of rebased/deleted branches really easy for end users.
      - sho_hn 14y ago
        We don't allow force-pushes without admin involvement, so rebasing isn't so scary for us, but we do cover it that way. And we got the undo operation down to a command on the infra: http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes The rest of that Wiki page is also pretty interesting for a general look at what devs can do on git.kde.org - server-side personal clones, personal repositories, a personal trashcan for repositories, etc. We don't have a pretty web interface for it (yet?), but it has some nice features.
- pja 14y ago
  (This is missing the wider point, but regardless: AFAIK (do correct me if I'm wrong) git clone --mirror will not copy dangling references & those refs will disappear from the original after a git gc, whenever that happens.) Human error is human error: sometimes people will do apparently egregiously stupid things if it's possible for them to do so, including things like deleting branches from a repository & then doing a git gc to "save space" shortly before realising that they weren't issuing commands to the shell they thought they were in. A backup strategy should be robust against dumb human mistakes as well as mechanical hardware errors, otherwise it's no backup strategy at all.
Confusion 14y ago
Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. There is no equivalent of rm -rf in git. If you remove files being versioned and commit them, the previous version still exists and the commit can be reverted. If you remove the metadata, no sync can happen, you are immediately warned (assuming good monitoring was set up) and can restore from one of the clones. If the entire machine crashes and burns, you whip out your git-server-create.sh, which provisions a new VM and restores from one of the clones. There is no additional data to be backup up with git. Any single developer with a recent clone of the repos can setup a new master git server.
- tomp 14y ago
  git checkout master git reset --hard HEAD^ git push --force origin master will remove the latest commit. If the mirrored repositories then blindly use the origin's refs, and do a `git gc`, the latest commit will be removed.
  - Confusion 14y ago
    It is impossible to blindly use the origin refs in that case. Git doesn't deal well with rewriting history. If someone force pushes to a centrally shared repo, all hell breaks loose.
    - pyre 14y ago
      From the article: | New refs are pulled down to the downstream | repository; all updates, including forced updates, | are mirrored to the downstream repository. | As a result, making a mirror clone essentially | bypasses the safety checks in the repository And > If someone force pushes to a centrally shared > repo, all hell breaks loose. This would happen to users pulling/pushing from the central repo, but not to the mirrors.
      - Confusion 14y ago
        In that case another developer would notice, would get in touch with his fellow developers, they would agree to force push a specific repository condition and all is well and back to normal. There is no need to restore anything from backups or mirrors in that scenario. So that's an entirely different threat. On the one hand we have FS corruption, which is basically bound to happen and is likely to impact many repositories. On the other hand we have someone (maliciously) force pushing an update on a single repository that throws away all commits. The first issue is one that needs a backup and restore procedure. The second is one that is at best an inconvenience and at worst a security problem. Unless you want to include the the worst case scenario someone gaining access to the machine and force pushing such a destructive update on all repos. In that case restoring from developer copies may actually be safer anyway, because the machine and all its backups, be they mirrors or otherwise, may be considered compromised. That's a nightmare for which this discussion about backups is just quibbling.
michaelt 14y ago
Surely every backup system has some equivalent of an rm -rf ? A disgruntled employee could phone the off site tape archival company and tell them to toss all the tapes in the shreader.
- chris_wot 14y ago
  The probability of this occurring is far less than what happened here.
- reeses 14y ago
  In the specific case of offsite tape archival, let's say someone who has the authority to do so requests that they destroy all tapes. Most service providers (who stay in business) have enough compliance features in place so that multiple authorized people have to be in collusion, a sufficiently senior executive must make the request, or there will be a "package" ready to be served to the client and relevant policing unit (police, FBI, whatever) so that charges can be brought against the malefactor efficiently. While you may be limited in financial compensation by your contract with the service provider, it is absolutely in their best interest to avoid the situation with procedures (they do not want to be a party to a crime) and if those fail, to provide extensive records of the movement and disposition of those tapes.
mpyne 14y ago
> Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. The master repository is coded so that those types of actions are not permitted. Developers cannot force push or delete branches that aren't reintegrated into some other head, without assistance from the sysadmins. After every routine push to a KDE git repository the new commit will be a descendant of the repo's original head. With that in mind it's more understandable as to why they felt it was possible to take advantage of Git's design in the backup planning. No "huge oversight" here, at least on this particular issue.
- andrewvc 14y ago
  Ummm, still not a backup. What if there's an error in the code? What if an admin runs a script directly on the server and resets all repos by mistake? What if a bad release of git comes out and corrupts the master? What if a hacker comes in and wipes them all out? Only a backup is a backup. Backup, like security, is somewhat onion like.
  - derleth 14y ago
    What if your backup systems get compromised? What if your backup systems get destroyed by inclement weather? Not a backup then, obviously. Only a backup is a backup, until it's not a backup anymore.
    - chris_wot 14y ago
      That's kind of ridiculous. The point that most people are making is that if someone does something incredibly stupid, or there is corruption in the system that follows down the line (like what happened here), it doesn't matter whether you have a repository. A backup clearly would have helped here.
      - mpyne 14y ago
        A backup of a corrupt repository would have been just as corrupt though. This is the big thing I can't figure out what people are not understanding. git does consistency checking for you already, tar|rsync|etc. don't, so it makes sense to take advantage of that. What we had was an instance of some of the underlying data becoming corrupt on the filesystem (with indications of that starting on Feb 22!). The big mistake was considering the source repositories as consistent and canonical at the remote anongit end, but the data would have been just as corrupt if we had scp'ed the repos from git.kde.org to the anongit mirrors around the world, since we would have bypassed git's internal checking in that way. Is it safe to rsync a running mysql database at random times, or are you supposed to use mysql-provided tools to perform a backup?

by pja 14y ago

richo 14y ago

Git doesn't nuke objects until you explicitly ask it to. It might sync broken objects, but it won't actually chose to destroy dangling objects without being asked to.

DougBTX 14y ago
They say in the article that one step of the anongit sync is to delete projects from the anongit servers which have been deleted on the central server. If a user accidentally deletes a project, they should be able to recover it from a backup, but it doesn't sound like there are any backups. The same idea applies to the individual repos themselves. A git gc will delete objects unreferenced by the reflogs, eg deleted branches over a month old by default. They say that the total set of repos is about 25GB, so it would be reasonable to keep at least monthly backups for at a year or more.
- sho_hn 14y ago
  We do ref backups on branch deletions FWIW.
  - DougBTX 14y ago
    Yea, that sounds like a good idea, can you do that in something like a pre-push hook? That plus long reflogs seems like it would make recovery of rebased/deleted branches really easy for end users.
    - sho_hn 14y ago
      We don't allow force-pushes without admin involvement, so rebasing isn't so scary for us, but we do cover it that way. And we got the undo operation down to a command on the infra: http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes http://community.kde.org/Sysadmin/GitKdeOrgManual#ohnoes The rest of that Wiki page is also pretty interesting for a general look at what devs can do on git.kde.org - server-side personal clones, personal repositories, a personal trashcan for repositories, etc. We don't have a pretty web interface for it (yet?), but it has some nice features.
pja 14y ago
(This is missing the wider point, but regardless: AFAIK (do correct me if I'm wrong) git clone --mirror will not copy dangling references & those refs will disappear from the original after a git gc, whenever that happens.) Human error is human error: sometimes people will do apparently egregiously stupid things if it's possible for them to do so, including things like deleting branches from a repository & then doing a git gc to "save space" shortly before realising that they weren't issuing commands to the shell they thought they were in. A backup strategy should be robust against dumb human mistakes as well as mechanical hardware errors, otherwise it's no backup strategy at all.

Confusion 14y ago

Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. There is no equivalent of rm -rf in git. If you remove files being versioned and commit them, the previous version still exists and the commit can be reverted. If you remove the metadata, no sync can happen, you are immediately warned (assuming good monitoring was set up) and can restore from one of the clones. If the entire machine crashes and burns, you whip out your git-server-create.sh, which provisions a new VM and restores from one of the clones. There is no additional data to be backup up with git. Any single developer with a recent clone of the repos can setup a new master git server.

tomp 14y ago
git checkout master git reset --hard HEAD^ git push --force origin master will remove the latest commit. If the mirrored repositories then blindly use the origin's refs, and do a `git gc`, the latest commit will be removed.
- Confusion 14y ago
  It is impossible to blindly use the origin refs in that case. Git doesn't deal well with rewriting history. If someone force pushes to a centrally shared repo, all hell breaks loose.
  - pyre 14y ago
    From the article: | New refs are pulled down to the downstream | repository; all updates, including forced updates, | are mirrored to the downstream repository. | As a result, making a mirror clone essentially | bypasses the safety checks in the repository And > If someone force pushes to a centrally shared > repo, all hell breaks loose. This would happen to users pulling/pushing from the central repo, but not to the mirrors.
    - Confusion 14y ago
      In that case another developer would notice, would get in touch with his fellow developers, they would agree to force push a specific repository condition and all is well and back to normal. There is no need to restore anything from backups or mirrors in that scenario. So that's an entirely different threat. On the one hand we have FS corruption, which is basically bound to happen and is likely to impact many repositories. On the other hand we have someone (maliciously) force pushing an update on a single repository that throws away all commits. The first issue is one that needs a backup and restore procedure. The second is one that is at best an inconvenience and at worst a security problem. Unless you want to include the the worst case scenario someone gaining access to the machine and force pushing such a destructive update on all repos. In that case restoring from developer copies may actually be safer anyway, because the machine and all its backups, be they mirrors or otherwise, may be considered compromised. That's a nightmare for which this discussion about backups is just quibbling.

michaelt 14y ago

Surely every backup system has some equivalent of an rm -rf ? A disgruntled employee could phone the off site tape archival company and tell them to toss all the tapes in the shreader.

chris_wot 14y ago
The probability of this occurring is far less than what happened here.
reeses 14y ago
In the specific case of offsite tape archival, let's say someone who has the authority to do so requests that they destroy all tapes. Most service providers (who stay in business) have enough compliance features in place so that multiple authorized people have to be in collusion, a sufficiently senior executive must make the request, or there will be a "package" ready to be served to the client and relevant policing unit (police, FBI, whatever) so that charges can be brought against the malefactor efficiently. While you may be limited in financial compensation by your contract with the service provider, it is absolutely in their best interest to avoid the situation with procedures (they do not want to be a party to a crime) and if those fail, to provide extensive records of the movement and disposition of those tapes.

mpyne 14y ago

> Relying on consistency checks will not save you if someone does the git equivalent of rm -rf on the master repository. The master repository is coded so that those types of actions are not permitted. Developers cannot force push or delete branches that aren't reintegrated into some other head, without assistance from the sysadmins. After every routine push to a KDE git repository the new commit will be a descendant of the repo's original head. With that in mind it's more understandable as to why they felt it was possible to take advantage of Git's design in the backup planning. No "huge oversight" here, at least on this particular issue.

andrewvc 14y ago
Ummm, still not a backup. What if there's an error in the code? What if an admin runs a script directly on the server and resets all repos by mistake? What if a bad release of git comes out and corrupts the master? What if a hacker comes in and wipes them all out? Only a backup is a backup. Backup, like security, is somewhat onion like.
- derleth 14y ago
  What if your backup systems get compromised? What if your backup systems get destroyed by inclement weather? Not a backup then, obviously. Only a backup is a backup, until it's not a backup anymore.
  - chris_wot 14y ago
    That's kind of ridiculous. The point that most people are making is that if someone does something incredibly stupid, or there is corruption in the system that follows down the line (like what happened here), it doesn't matter whether you have a repository. A backup clearly would have helped here.
    - mpyne 14y ago
      A backup of a corrupt repository would have been just as corrupt though. This is the big thing I can't figure out what people are not understanding. git does consistency checking for you already, tar|rsync|etc. don't, so it makes sense to take advantage of that. What we had was an instance of some of the underlying data becoming corrupt on the filesystem (with indications of that starting on Feb 22!). The big mistake was considering the source repositories as consistent and canonical at the remote anongit end, but the data would have been just as corrupt if we had scp'ed the repos from git.kde.org to the anongit mirrors around the world, since we would have bypassed git's internal checking in that way. Is it safe to rsync a running mysql database at random times, or are you supposed to use mysql-provided tools to perform a backup?