4 ms·

Jenkins is fantastic because you can plug it into just about everything, and trigger your job that does $whatever either on a timer, manually or with an HTTP GE

Jenkins is fantastic because you can plug it into just about everything, and trigger your job that does $whatever either on a timer, manually or with an HTTP GET request. When I started at this company, Jenkins was nothing more than a web server used to launch an all-encompassing build script on SVN commit. Since then, I've redesigned all builds using plugins such that Jenkins now manages the whole process (previous admins had reinvented so. f*cking. much. stuff Jenkins does natively) and we're using Jenkins to automate just about everything we can plug it into. This and Powershell, which although its syntax is mind-bending and in some cases just outright appalling, is a decent attempt at a scripting language with exceptional power over Windows.

autotune 10y ago
You also get the benefit of it the jobs being built on a shared environment. If developer A wants to build a job, $THING-A and developer B wants to build a their job, $THING-B, there's no having to look through code for environment variables or other differing, machine-specific issues, that might pop up when building with something like, for example, Terraform.
- gargravarr 10y ago
  The build server was still the shared environment, but all it did was call a batch script that did everything else, up to and including emailing the build breaker (see above comment). There was also copy-pasted jobs for each branch, rather than a single master build-branch job that took parameters.
strictfp 10y ago
Noo, builds shouldn't rely on Jenkins. I root for the previous admins.
- kirubakaran 10y ago
  Can you please go into more detail?
  - strictfp 10y ago
    My view is that the build which runs on the CI server should be close to identical to the build which a dev can run on a workstation. This way you don't have to check in anything to debug build errors. The build may complement the normal build, in ways such as emailing, but shoulldn't modify it.
    - gargravarr 10y ago
      I will agree with you there; the builds run through our CI servers can't be run on a local workstation. I'm still not sure why, but the build script simply will not work if run locally. To run from the command line, I have to invoke msbuild and nodejs manually. This isn't a big problem since most devs build through Visual Studio anyway, but I do fully understand your argument.
      - strictfp 10y ago
        Currently we have a docker-based test runner which works differently on the CI server. It should be identical, but of course the bugs you trigger are usually the difficult ones caused by unexpected timing problems, file permissions or subtle component interaction. If you get stuck trying to fix an error only triggered on the CI server, it's a huge time waster and commit log polluter. Especially if the security policy is such that you aren't easily allowed shell access on the build slave. If devs have full access, it's not so much of a problem actually. But still very annoying whe you are dependent on your VCS and CI server to build your project properly.
- gargravarr 10y ago
  By all means. Please picture the following: -A folder full of batch files, each 2,000 lines long, that actually coordinate the builds -A folder full of .exe's like 'findAndReplace.exe', 'EmailBuildBreakingAuthorsFromSVN.exe', replicating native Jenkins functionality, written in-house -SVN.exe must be installed on the build node; if the build fails, the working copy is usually locked and must be manually Cleaned Up -All build nodes run Windows XP Pro (including the Jenkins master) This is what I had to deal with when I took over the cluster last year!!! For the record, the whole cluster now runs on Win2012 with regular security updates, and most builds use the .Net plugin to build the VS project using msbuild. Those that can't, use Powershell scripts. It has been a VERY long road...
  - strictfp 10y ago
    Ok, I wouldn't go that far. Also, windows builds are a pain :)
deleted 10y ago
[deleted]
Psilidae 10y ago
Got any tips you could share regarding the Jenkins setup? I'm going to be needing to rework our current Jenkins build system—which sounds pretty similar to your "all-encompassing build script"—as we're about to migrate from Perforce to SVN. I know what we've got setup is pretty bad, but I don't know much about best practices for build organization.
- gargravarr 10y ago
  The best advice I can give is to use the Jenkins plugins wherever possible. Jenkins can do almost everything either natively or via its extensive library of plugins. In the same vein, always use Jenkins version control; our old build scripts used to shell out to svn.exe and run the checkouts that way, into a static folder. This meant that we couldn't run builds in parallel, and if an SVN operation were interrupted, the working copy would lock and would need to be manually cleaned up. For complex builds, write a script that is kept in the root of the project, then after checkout, simply do ./build.sh. We used to keep all our build scripts in a separate repository which would have to be checked out onto the build node in preparation; invariably, the scripts never got updated. This way, the product and the steps required to build it are kept together, and are very unlikely to come out of sync. Use labels as much as possible; I have our cluster set up so that builds that require dependencies are bound to label '<product>' and I add/remove the labels from the build nodes as the dependencies are added. This also means that if one node breaks and is unable to build <product>, I just remove that label and Jenkins will stop scheduling builds on it. It becomes much easier to manage than restricting builds to hostnames. Most importantly, ensure you can always build from a fresh checkout. I cannot tell you how annoying it gets when a build fails because some tiny thing is out of alignment on a fresh build node. I also recommend imaging your build nodes so they're quick and easy to deply, and BACK UP YOUR JENKINS CONFIG! :) When I came back from my Christmas break, it was to discover that the Jenkins server I had worked oh so hard on had been inadvertently trashed by one of the guys on my team; the disk had run out of space, so he'd remoted onto the server, gone into the Jenkins folder and deleted everything older than a month. He hadn't thought to restrict this to the build artefacts folders. Basically lobotomised the server, and that was when we discovered we'd been backing up the build NODES, but not the MASTER itself! I was able to salvage some of the config and rebuilt the server on the latest version of Jenkins. As a result, the Jenkins folder on the master is now an SVN working copy with all the config files under version control, and a scheduled task runs daily to commit the changes. It has been a very painful road to get this thing stable! It would be considerably easier if people weren't trying to nuke all my hard work while my back was turned...!

autotune 10y ago

You also get the benefit of it the jobs being built on a shared environment. If developer A wants to build a job, $THING-A and developer B wants to build a their job, $THING-B, there's no having to look through code for environment variables or other differing, machine-specific issues, that might pop up when building with something like, for example, Terraform.

gargravarr 10y ago
The build server was still the shared environment, but all it did was call a batch script that did everything else, up to and including emailing the build breaker (see above comment). There was also copy-pasted jobs for each branch, rather than a single master build-branch job that took parameters.

strictfp 10y ago

Noo, builds shouldn't rely on Jenkins. I root for the previous admins.

kirubakaran 10y ago
Can you please go into more detail?
- strictfp 10y ago
  My view is that the build which runs on the CI server should be close to identical to the build which a dev can run on a workstation. This way you don't have to check in anything to debug build errors. The build may complement the normal build, in ways such as emailing, but shoulldn't modify it.
  - gargravarr 10y ago
    I will agree with you there; the builds run through our CI servers can't be run on a local workstation. I'm still not sure why, but the build script simply will not work if run locally. To run from the command line, I have to invoke msbuild and nodejs manually. This isn't a big problem since most devs build through Visual Studio anyway, but I do fully understand your argument.
    - strictfp 10y ago
      Currently we have a docker-based test runner which works differently on the CI server. It should be identical, but of course the bugs you trigger are usually the difficult ones caused by unexpected timing problems, file permissions or subtle component interaction. If you get stuck trying to fix an error only triggered on the CI server, it's a huge time waster and commit log polluter. Especially if the security policy is such that you aren't easily allowed shell access on the build slave. If devs have full access, it's not so much of a problem actually. But still very annoying whe you are dependent on your VCS and CI server to build your project properly.
gargravarr 10y ago
By all means. Please picture the following: -A folder full of batch files, each 2,000 lines long, that actually coordinate the builds -A folder full of .exe's like 'findAndReplace.exe', 'EmailBuildBreakingAuthorsFromSVN.exe', replicating native Jenkins functionality, written in-house -SVN.exe must be installed on the build node; if the build fails, the working copy is usually locked and must be manually Cleaned Up -All build nodes run Windows XP Pro (including the Jenkins master) This is what I had to deal with when I took over the cluster last year!!! For the record, the whole cluster now runs on Win2012 with regular security updates, and most builds use the .Net plugin to build the VS project using msbuild. Those that can't, use Powershell scripts. It has been a VERY long road...
- strictfp 10y ago
  Ok, I wouldn't go that far. Also, windows builds are a pain :)

deleted 10y ago

[deleted]

Psilidae 10y ago

Got any tips you could share regarding the Jenkins setup? I'm going to be needing to rework our current Jenkins build system—which sounds pretty similar to your "all-encompassing build script"—as we're about to migrate from Perforce to SVN. I know what we've got setup is pretty bad, but I don't know much about best practices for build organization.

gargravarr 10y ago
The best advice I can give is to use the Jenkins plugins wherever possible. Jenkins can do almost everything either natively or via its extensive library of plugins. In the same vein, always use Jenkins version control; our old build scripts used to shell out to svn.exe and run the checkouts that way, into a static folder. This meant that we couldn't run builds in parallel, and if an SVN operation were interrupted, the working copy would lock and would need to be manually cleaned up. For complex builds, write a script that is kept in the root of the project, then after checkout, simply do ./build.sh. We used to keep all our build scripts in a separate repository which would have to be checked out onto the build node in preparation; invariably, the scripts never got updated. This way, the product and the steps required to build it are kept together, and are very unlikely to come out of sync. Use labels as much as possible; I have our cluster set up so that builds that require dependencies are bound to label '<product>' and I add/remove the labels from the build nodes as the dependencies are added. This also means that if one node breaks and is unable to build <product>, I just remove that label and Jenkins will stop scheduling builds on it. It becomes much easier to manage than restricting builds to hostnames. Most importantly, ensure you can always build from a fresh checkout. I cannot tell you how annoying it gets when a build fails because some tiny thing is out of alignment on a fresh build node. I also recommend imaging your build nodes so they're quick and easy to deply, and BACK UP YOUR JENKINS CONFIG! :) When I came back from my Christmas break, it was to discover that the Jenkins server I had worked oh so hard on had been inadvertently trashed by one of the guys on my team; the disk had run out of space, so he'd remoted onto the server, gone into the Jenkins folder and deleted everything older than a month. He hadn't thought to restrict this to the build artefacts folders. Basically lobotomised the server, and that was when we discovered we'd been backing up the build NODES, but not the MASTER itself! I was able to salvage some of the config and rebuilt the server on the latest version of Jenkins. As a result, the Jenkins folder on the master is now an SVN working copy with all the config files under version control, and a scheduled task runs daily to commit the changes. It has been a very painful road to get this thing stable! It would be considerably easier if people weren't trying to nuke all my hard work while my back was turned...!