6 ms·

Use Jenkins "Build Per Branch" for Git Feature Branch Development

74 points by tednaleid 14y ago

Ramone 14y ago
But then it's not really continuous integration anymore is it? Having a CI server and practicing continuous integration are not the same thing.
- tednaleid 14y ago
  So what benefits are you saying are lost/given up by this model that another model preserves? Also, feature branch development is just how we're using it. The code monitors git for any new branches and automatically clones jobs for new branches. This would support a more traditional master branch with release branches created periodically. Those release branches would automatically get Jenkins jobs created for them instead of someone manually having to do it.
  - Ramone 14y ago
    I'm saying a CI server is unnecessary for these branches that aren't ready to integrate with master. These are short-lived branches with 1 or 2 contributors, right? Just run the build/tests locally. There's no advantage to this overhead with branches that won't last longer than a day.
    - tednaleid 14y ago
      There are lots of advantages for us. What if the full build and test suite takes longer to run than you want to occupy your computer? Unit tests are quick, but the full set of unit/integration/functional tests? Plus ensuring that artifacts like .war files can be successfully created, and that it runs fine on the target system rather than just on your dev laptop (which for most devs is not the same platform as they deploy to). What if you want to show off your work-in-progress by having a job auto-deploy it to a test server so that your customers or QA can play with it before you're ready to merge to master? Sometimes a day is too long to wait to show something off. Commit and publish as a branch and you can keep working while someone else looks at it. We have another dev code review every pull request to master to ensure code quality as well as spreading knowledge of what's happening across the team. The overhead is surprisingly low. Branches literally take about 15 seconds to make, and pull requests are no more than a minute or two unless you really go into testing instructions.
      - Ramone 14y ago
        Well every team is different, so I shouldn't try to say I know the right thing for your team. What you're talking about though is farming out your branch builds/tests/deploys to jenkins, which has nothing to do with CI, but still sounds like your team finds it useful. Carry on if that's the case. While I wouldn't personally solve the problem that way, I'm wrong to suggest that jenkins should only be used for CI.
    - gouranga 14y ago
      I agree entirely with that approach and it's how I operate.
- russell_h 14y ago
  Continuous integration doesn't mean all development happens in master. A policy that works well for me is that anything that will require more than 2 minor commits to accomplish should happen in a feature branch. If tests pass in the branch (and you probably want to insert code review/sign-off at this stage as well), merge to master. If tests pass in master, deploy. From what I can tell, this just makes it easy to run tests in branches. My team uses something similar that kicks off a build whenever someone comments "build it" in a GitHub pull request, and posts the results to the pull request discussion.
  - Ramone 14y ago
    It sounds like we agree that long-lived branches are not continuous integration at least. We probably differ in that I think short-lived branches and pull requests are more overhead than most teams require. Keep in mind that CI does expect daily check-ins to master. That's how short those branches can live to satisfy that requirement. Anything else is not CI... which is not the end of the world either, but I don't see the point of using a CI server at all anymore. Just build/test your stuff locally.
ddlatham 14y ago
One disadvantage is that with multiple, short-lived branches, manually creating CI to confirm that all tests continue to pass becomes far too much overhead. It must be automated for this model to work and be trusted. "Jenkins Build Per Branch" is the answer to that problem. We use something close to this model, but there's no need to create separate Jenkins jobs for each branch. We have a single job that takes the git branch as a parameter and can run all of the automated tests (and other jobs for builds, etc.). There is a script that watches for new pull requests and automatically triggers the Jenkins job to run all the tests on that pull request to verify that they pass before the reviewer merges in to master.
- tednaleid 14y ago
  Thats an interesting approach. If a branch is broken, which causes the job to fail, how do you communicate that and monitor when it's actually fixed (rather than another branch getting run which "fixes" it)?
  - ddlatham 14y ago
    We have email notifications on test failures, so the creator and reviewer of the pull request can see it fails. Haven't done it yet, but would like to extend it to report back the result as a comment directly on the pull request.
- tomstuart 14y ago
  FYI this is something that Travis (http://travis-ci.org/ http://travis-ci.org/) can now do; for example, see the travisbot comments on https://github.com/sinatra/sinatra/pull/521 https://github.com/sinatra/sinatra/pull/521. It's not finished yet, but they've got more information at http://about.travis-ci.org/blog/announcing-pull-request-support/ http://about.travis-ci.org/blog/announcing-pull-request-supp....
- steverb 14y ago
  We do the same using Team City and Git (on the .NET stack) although we don't merge to master until the feature in question has gone through manual QA as well. We have 5 QA servers and a dashboard where the tester can point any of the QA servers to a feature branch. Works awesomely.
nsxwolf 14y ago
Is there something like this for SVN?
- tednaleid 14y ago
  Not that I'm aware of, but modifying the code to support SVN would be pretty easy to do. All it really needs from the VCS is the current list of branch names. If you modified the GitApi class (https://github.com/entagen/jenkins-build-per-branch/blob/master/src/main/groovy/com/entagen/jenkins/GitApi.groovy https://github.com/entagen/jenkins-build-per-branch/blob/mas...) to talk to SVN instead of git, everything else should work fine. This project was presented at a user meeting tonight and one of the attendees there was planning on adding Mercurial support.
  - nsxwolf 14y ago
    Will take a look. Thanks.
    - sulsomat 14y ago
      mind dropping a note here how it went? really keen on having this for svn as well (i know, i know, so last century - but at my work place we have only limited resources and migrating to another scm is a pretty resource intensive task). how about forking the module on github?
      - lt 14y ago
        Count me in as another one interested on this.
tantalor 14y ago
Travis automatically does this, http://about.travis-ci.org/docs/user/build-configuration/ http://about.travis-ci.org/docs/user/build-configuration/ Basically, it builds whichever branch you push, as long as that branch has a config. I think this is better because your config is in SCM instead of off in jenkins-land (even though it is highly travis-specific).
- enobrev 14y ago
  For that exact reason, I usually to put all my build scripts into a bash file, and just call that bash file from jenkins.
gvalkov 14y ago
A while ago, I wrote a similar tool[1] that I never quite got around to releasing/documenting. Unlike the OP's nicely integrated plugin, mine is an external script (which you would run from cron) that checks the repository for new refs and creates jobs accordingly. Have a look at the config file[2] for a general idea of what it does. [1] https://github.com/gvalkov/jenkins-autojobs https://github.com/gvalkov/jenkins-autojobs [2] https://github.com/gvalkov/jenkins-autojobs/blob/master/doc/git.config.yaml https://github.com/gvalkov/jenkins-autojobs/blob/master/doc/...
pbiggar 14y ago
Circle (my CI-as-a-service startup - https://circleci.com https://circleci.com) does this by default. For nearly any reason that you have a branch - for staging or deployment, for features, for tags, for bug fixes - there really is no reason not to have this. Unless of course, everyone gets a notification for every build in every branch. Then you'll have practically every developer in your organization hating this feature! I'm sure Jenkins has a way of controlling that though.

74 points by tednaleid 14y ago

But then it's not really continuous integration anymore is it? Having a CI server and practicing continuous integration are not the same thing.

tednaleid 14y ago
So what benefits are you saying are lost/given up by this model that another model preserves? Also, feature branch development is just how we're using it. The code monitors git for any new branches and automatically clones jobs for new branches. This would support a more traditional master branch with release branches created periodically. Those release branches would automatically get Jenkins jobs created for them instead of someone manually having to do it.
- Ramone 14y ago
  I'm saying a CI server is unnecessary for these branches that aren't ready to integrate with master. These are short-lived branches with 1 or 2 contributors, right? Just run the build/tests locally. There's no advantage to this overhead with branches that won't last longer than a day.
  - tednaleid 14y ago
    There are lots of advantages for us. What if the full build and test suite takes longer to run than you want to occupy your computer? Unit tests are quick, but the full set of unit/integration/functional tests? Plus ensuring that artifacts like .war files can be successfully created, and that it runs fine on the target system rather than just on your dev laptop (which for most devs is not the same platform as they deploy to). What if you want to show off your work-in-progress by having a job auto-deploy it to a test server so that your customers or QA can play with it before you're ready to merge to master? Sometimes a day is too long to wait to show something off. Commit and publish as a branch and you can keep working while someone else looks at it. We have another dev code review every pull request to master to ensure code quality as well as spreading knowledge of what's happening across the team. The overhead is surprisingly low. Branches literally take about 15 seconds to make, and pull requests are no more than a minute or two unless you really go into testing instructions.
    - Ramone 14y ago
      Well every team is different, so I shouldn't try to say I know the right thing for your team. What you're talking about though is farming out your branch builds/tests/deploys to jenkins, which has nothing to do with CI, but still sounds like your team finds it useful. Carry on if that's the case. While I wouldn't personally solve the problem that way, I'm wrong to suggest that jenkins should only be used for CI.
  - gouranga 14y ago
    I agree entirely with that approach and it's how I operate.
russell_h 14y ago
Continuous integration doesn't mean all development happens in master. A policy that works well for me is that anything that will require more than 2 minor commits to accomplish should happen in a feature branch. If tests pass in the branch (and you probably want to insert code review/sign-off at this stage as well), merge to master. If tests pass in master, deploy. From what I can tell, this just makes it easy to run tests in branches. My team uses something similar that kicks off a build whenever someone comments "build it" in a GitHub pull request, and posts the results to the pull request discussion.
- Ramone 14y ago
  It sounds like we agree that long-lived branches are not continuous integration at least. We probably differ in that I think short-lived branches and pull requests are more overhead than most teams require. Keep in mind that CI does expect daily check-ins to master. That's how short those branches can live to satisfy that requirement. Anything else is not CI... which is not the end of the world either, but I don't see the point of using a CI server at all anymore. Just build/test your stuff locally.

ddlatham 14y ago

One disadvantage is that with multiple, short-lived branches, manually creating CI to confirm that all tests continue to pass becomes far too much overhead. It must be automated for this model to work and be trusted. "Jenkins Build Per Branch" is the answer to that problem. We use something close to this model, but there's no need to create separate Jenkins jobs for each branch. We have a single job that takes the git branch as a parameter and can run all of the automated tests (and other jobs for builds, etc.). There is a script that watches for new pull requests and automatically triggers the Jenkins job to run all the tests on that pull request to verify that they pass before the reviewer merges in to master.

tednaleid 14y ago
Thats an interesting approach. If a branch is broken, which causes the job to fail, how do you communicate that and monitor when it's actually fixed (rather than another branch getting run which "fixes" it)?
- ddlatham 14y ago
  We have email notifications on test failures, so the creator and reviewer of the pull request can see it fails. Haven't done it yet, but would like to extend it to report back the result as a comment directly on the pull request.
tomstuart 14y ago
FYI this is something that Travis (http://travis-ci.org/ http://travis-ci.org/) can now do; for example, see the travisbot comments on https://github.com/sinatra/sinatra/pull/521 https://github.com/sinatra/sinatra/pull/521. It's not finished yet, but they've got more information at http://about.travis-ci.org/blog/announcing-pull-request-support/ http://about.travis-ci.org/blog/announcing-pull-request-supp....
steverb 14y ago
We do the same using Team City and Git (on the .NET stack) although we don't merge to master until the feature in question has gone through manual QA as well. We have 5 QA servers and a dashboard where the tester can point any of the QA servers to a feature branch. Works awesomely.

nsxwolf 14y ago

Is there something like this for SVN?

tednaleid 14y ago
Not that I'm aware of, but modifying the code to support SVN would be pretty easy to do. All it really needs from the VCS is the current list of branch names. If you modified the GitApi class (https://github.com/entagen/jenkins-build-per-branch/blob/master/src/main/groovy/com/entagen/jenkins/GitApi.groovy https://github.com/entagen/jenkins-build-per-branch/blob/mas...) to talk to SVN instead of git, everything else should work fine. This project was presented at a user meeting tonight and one of the attendees there was planning on adding Mercurial support.
- nsxwolf 14y ago
  Will take a look. Thanks.
  - sulsomat 14y ago
    mind dropping a note here how it went? really keen on having this for svn as well (i know, i know, so last century - but at my work place we have only limited resources and migrating to another scm is a pretty resource intensive task). how about forking the module on github?
    - lt 14y ago
      Count me in as another one interested on this.

tantalor 14y ago

Travis automatically does this, http://about.travis-ci.org/docs/user/build-configuration/ http://about.travis-ci.org/docs/user/build-configuration/ Basically, it builds whichever branch you push, as long as that branch has a config. I think this is better because your config is in SCM instead of off in jenkins-land (even though it is highly travis-specific).

enobrev 14y ago
For that exact reason, I usually to put all my build scripts into a bash file, and just call that bash file from jenkins.

gvalkov 14y ago

A while ago, I wrote a similar tool[1] that I never quite got around to releasing/documenting. Unlike the OP's nicely integrated plugin, mine is an external script (which you would run from cron) that checks the repository for new refs and creates jobs accordingly. Have a look at the config file[2] for a general idea of what it does. [1] https://github.com/gvalkov/jenkins-autojobs https://github.com/gvalkov/jenkins-autojobs [2] https://github.com/gvalkov/jenkins-autojobs/blob/master/doc/git.config.yaml https://github.com/gvalkov/jenkins-autojobs/blob/master/doc/...

pbiggar 14y ago

Circle (my CI-as-a-service startup - https://circleci.com https://circleci.com) does this by default. For nearly any reason that you have a branch - for staging or deployment, for features, for tags, for bug fixes - there really is no reason not to have this. Unless of course, everyone gets a notification for every build in every branch. Then you'll have practically every developer in your organization hating this feature! I'm sure Jenkins has a way of controlling that though.