5 ms·

Lessons from PostgreSQL's Git transition

75 points by corbet 16y ago

timmorgan 16y ago
GitHub makes a downloadable tarball of each repository available for download, which is nice for those who don't have (or don't want to install) Git. I understand that the Postgres team isn't using GitHub, but it makes me think -- could they use an oldschool tarball to update their BuildFarm nodes who cannot run Git? Or better yet, rsync, so only changes are transferred? Anyway, this is a decent writeup. I've wondered before about large project migration to Git.
- pyre 16y ago
  Someone suggested that in the comments, and the reply was that the amount of bandwidth to pull down the tarball is greater than just syncing the changes. It seems like they could use some sort of centralized checkout system + rsync to pull down changes. Possibly using gitolite's master/slave mirroring functionality.
- plaes 16y ago
  IIRC, they even provide two-way (commit and fetch) subversion->git gateway...
  - mrduncan 16y ago
    You're correct: http://github.com/blog/626-announcing-svn-support http://github.com/blog/626-announcing-svn-support http://github.com/blog/644-subversion-write-support http://github.com/blog/644-subversion-write-support
DannoHung 16y ago
Great pains taken to preserve the full commit history... does anyone know how valuable it really is? (Not to imply that it is worthless, but how much effort went into preserving it vs how much effort would be spent if it were not available). More of a general VCS conversion question, I suppose.
- neilc 16y ago
  Depends on the project, but for a long-running project with a large code base and many contributors (many of whom are no longer actively involved in the project), it can be very valuable. In the case of PostgreSQL, the commit history goes back to ~1995, and most changes are described with a detailed commit message, which can be very helpful when understanding why a certain piece of code behaves the way that it does.
  - ciupicri 16y ago
    But shouldn't the comments from the source code provide an explanation?
    - silentbicycle 16y ago
      Commit messages are a better place than comments for explaining the rationale behind cross-cutting changes.
      - nettdata 16y ago
        And knowing what files were touched for the various commits is a big advantage as well.
      - ciupicri 16y ago
        I admit that I've been in situations where I had to find out in what change a piece of code was committed, but on the other hand the code and the comments weren't too great. Also one of the purposes was to find out the author, so that I could pass the bug to him. I'm curious in what situations would a commit message be more appropriate than a comment. A couple of examples would be great. By the way, is there something like Perforce's Time-lapse View tool[1] for git? [1] http://www.perforce.com/perforce/press/pr55.html http://www.perforce.com/perforce/press/pr55.html
        silentbicycle 16y ago
        Example: "Gathered & cut this code from files A, B, and C, because Client X changed their mind about situation Y, and now we can drop those special cases." Anything involving moving / deleting a lot of code. Adding comments to explain deleted code in several places really doesn't make sense. I don't know about the viewer, but I'm going to look into this soon. We've used p4 for years and are strongly considering switching to git, and if so, I will be doing a lot of helping/explaining during the transition. I really dislike p4 as a VCS (though it was probably the best choice at the time), but agree its diff-viewing tool is actually quite good.
jarin 16y ago
"Yes, that's correct. No merge commits. To submit a patch, extract it as a context diff and e-mail it. Committers are to apply the patch under their own names, without branch history. The project has decided, more-or-less, to use Git like it was CVS as far as commits to the main repository are concerned. Rather than adapt the PostgreSQL project's workflow to Git, Git would be adapted to the project's workflow." I hope they fix that culture issue soon, one of the biggest strengths of Git is its merging.
- selenamarie 16y ago
  Peter notes in the LWN comments that committers use merging in their personal repos regularly. The restriction only applies to commits to the master. Many people share their individual repos and branches to git.postgresql.org (and GitHub). The point is to keep our commit history clean while we figure out best practices for the project.
  - Groxx 16y ago
    I've been wondering what the point of keeping a "clean" history is. In what way is less information better? All this means is you're destroying the real history, rewriting it to look more like a straight line, which is not at all how it was developed. If you want such a history to reflect how it was developed, there are wonderful version control programs out there. Like RCS. Why not just use that?
    - silentbicycle 16y ago
      While I can't speak for the parent comment, one reason people try to keep clean commit histories in git is that you can easily undo, cherry pick, etc. individual commits. If all of the changes necessary to add a feature / fix a major bug are in one commit, backporting that commit becomes much easier. If those changes are broken up in multiple commits ("fix this issue" + "oops, forgot that file (and unrelated changes in the same file)"), then it's more trouble. It usually doesn't make sense to have the repository in those intermediate states anyway, once the primary commit is done. History rewriting can make those commits atomic. Generally speaking, "clean" git histories have relevant history combined, but no significant information lost.
dochtman 16y ago
Everyone here should subscribe to LWN and get this kind of content sooner.

75 points by corbet 16y ago

GitHub makes a downloadable tarball of each repository available for download, which is nice for those who don't have (or don't want to install) Git. I understand that the Postgres team isn't using GitHub, but it makes me think -- could they use an oldschool tarball to update their BuildFarm nodes who cannot run Git? Or better yet, rsync, so only changes are transferred? Anyway, this is a decent writeup. I've wondered before about large project migration to Git.

pyre 16y ago
Someone suggested that in the comments, and the reply was that the amount of bandwidth to pull down the tarball is greater than just syncing the changes. It seems like they could use some sort of centralized checkout system + rsync to pull down changes. Possibly using gitolite's master/slave mirroring functionality.
plaes 16y ago
IIRC, they even provide two-way (commit and fetch) subversion->git gateway...
- mrduncan 16y ago
  You're correct: http://github.com/blog/626-announcing-svn-support http://github.com/blog/626-announcing-svn-support http://github.com/blog/644-subversion-write-support http://github.com/blog/644-subversion-write-support

DannoHung 16y ago

Great pains taken to preserve the full commit history... does anyone know how valuable it really is? (Not to imply that it is worthless, but how much effort went into preserving it vs how much effort would be spent if it were not available). More of a general VCS conversion question, I suppose.

neilc 16y ago
Depends on the project, but for a long-running project with a large code base and many contributors (many of whom are no longer actively involved in the project), it can be very valuable. In the case of PostgreSQL, the commit history goes back to ~1995, and most changes are described with a detailed commit message, which can be very helpful when understanding why a certain piece of code behaves the way that it does.
- ciupicri 16y ago
  But shouldn't the comments from the source code provide an explanation?
  - silentbicycle 16y ago
    Commit messages are a better place than comments for explaining the rationale behind cross-cutting changes.
    - nettdata 16y ago
      And knowing what files were touched for the various commits is a big advantage as well.
    - ciupicri 16y ago
      I admit that I've been in situations where I had to find out in what change a piece of code was committed, but on the other hand the code and the comments weren't too great. Also one of the purposes was to find out the author, so that I could pass the bug to him. I'm curious in what situations would a commit message be more appropriate than a comment. A couple of examples would be great. By the way, is there something like Perforce's Time-lapse View tool[1] for git? [1] http://www.perforce.com/perforce/press/pr55.html http://www.perforce.com/perforce/press/pr55.html
      - silentbicycle 16y ago
        Example: "Gathered & cut this code from files A, B, and C, because Client X changed their mind about situation Y, and now we can drop those special cases." Anything involving moving / deleting a lot of code. Adding comments to explain deleted code in several places really doesn't make sense. I don't know about the viewer, but I'm going to look into this soon. We've used p4 for years and are strongly considering switching to git, and if so, I will be doing a lot of helping/explaining during the transition. I really dislike p4 as a VCS (though it was probably the best choice at the time), but agree its diff-viewing tool is actually quite good.

jarin 16y ago

"Yes, that's correct. No merge commits. To submit a patch, extract it as a context diff and e-mail it. Committers are to apply the patch under their own names, without branch history. The project has decided, more-or-less, to use Git like it was CVS as far as commits to the main repository are concerned. Rather than adapt the PostgreSQL project's workflow to Git, Git would be adapted to the project's workflow." I hope they fix that culture issue soon, one of the biggest strengths of Git is its merging.

selenamarie 16y ago
Peter notes in the LWN comments that committers use merging in their personal repos regularly. The restriction only applies to commits to the master. Many people share their individual repos and branches to git.postgresql.org (and GitHub). The point is to keep our commit history clean while we figure out best practices for the project.
- Groxx 16y ago
  I've been wondering what the point of keeping a "clean" history is. In what way is less information better? All this means is you're destroying the real history, rewriting it to look more like a straight line, which is not at all how it was developed. If you want such a history to reflect how it was developed, there are wonderful version control programs out there. Like RCS. Why not just use that?
  - silentbicycle 16y ago
    While I can't speak for the parent comment, one reason people try to keep clean commit histories in git is that you can easily undo, cherry pick, etc. individual commits. If all of the changes necessary to add a feature / fix a major bug are in one commit, backporting that commit becomes much easier. If those changes are broken up in multiple commits ("fix this issue" + "oops, forgot that file (and unrelated changes in the same file)"), then it's more trouble. It usually doesn't make sense to have the repository in those intermediate states anyway, once the primary commit is done. History rewriting can make those commits atomic. Generally speaking, "clean" git histories have relevant history combined, but no significant information lost.

dochtman 16y ago

Everyone here should subscribe to LWN and get this kind of content sooner.