9 ms·

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We ar

by mkobit 11y ago

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do everything" workflow system. The easy path that Jenkins provides, to me, seems like the wrong one - it makes it easy to just stuff everything in there because it "can" do it. This seems to leads to tons of copy/paste, drift, all types of different work being represented, and it is starting to become unmanageable. Have others seen this happen when using Jenkins? How have you dealt with it?

richardwhiuk 11y ago
Yes - we've definitely seen that. The most common thing we've see if that you start with a jenkins job that just runs a build process, and then you end up with a jenkins job that has a huge shell script which calls the build process somewhere in the middle, that's unversioned. To resolve that we've tried to push the shell script into a file in the repository, which Jenkins then checks out and runs, which makes it easier to maintain and faster to set up new build machines.
christop 11y ago
Netflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-controlled, as opposed to clicking everything together via the web UI. See https://jenkins-ci.org/2.0/ https://jenkins-ci.org/2.0/
- oblio 11y ago
  Some things are already available, see this: https://github.com/jenkinsci/workflow-plugin/blob/master/README.md#introduction https://github.com/jenkinsci/workflow-plugin/blob/master/REA...
- vorg 11y ago
  For the last 4 months, Groovy has been known as "Apache Groovy".
  - trollingineer 11y ago
    Since your birth, you have been known as loser-vorg.
lintoma 11y ago
It's really the same Jenkins Ami but sharded for different teams. One of the things Spinnaker does is plugin Jenkins jobs as a reusable stage that is parametrized and scoped to an application deploy. This has allowed one team to move from 40 Jenkins jobs created via job dsl plugin to just 6
sytse 11y ago
I think the copy/paste is not inherently bad as long as it is version controlled and visible to everyone, that is why we have all CI configuration in the repo with gitlab-ci.yml I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).
- onetwotree 11y ago
  Agreed, we have a basically arbitrary number of runners (near releases it gets into the hundreds), and one master handles it just fine.
  - sytse 11y ago
    Glad to hear that. We'll soon announce an autoscaling runner that allows you to boot up new instances automatically. I love that you sometimes run hundreds of runners, would you like to do a guest post? Email me at website@sytse.com
onetwotree 11y ago
I think most people do. We did a couple of things to get out of that state at Conjur. First, we made all of our builds into docker containers. Under this system, slaves (we call 'em executors now) only contain very basic software - docker, git, and make to be specific. This means that our builds are entirely self contained, and we don't have to worry about, for example, messing around with RVM on executors. If the containerized build works locally, it pretty much always works on Jenkins as well. Second, we started using the job dsl plugin to manage configuration and brought in some autoscaling and machine identity. Beyond what I do as a platform engineer to turn my projects into jenkins builds, I don't completely understand it (nor do I have to, which is a good thing!), so I'll let our DevOps guy take it from here: https://blog.conjur.net/scaling-jenkins-with-machine-identity https://blog.conjur.net/scaling-jenkins-with-machine-identit...
- jacques_chester 11y ago
  You might like to look at Concourse[1], which makes explicit, checked-in pipelines of containerized builds its central model. I am very bullish on Concourse. [1] https://concourse.ci https://concourse.ci
Rafert 11y ago
I've seen something similar happen, switched to https://www.go.cd/ https://www.go.cd/
- jacques_chester 11y ago
  GoCD is very difficult to version-control and the interface is, to put it politely, in need of some love. At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out. Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with Concourse pipelines.
  - dominotw 11y ago
    >now pretty much the whole of Cloud Foundry is being built with Concourse pipelines. What are some of your thoughts on, * How it compares to Jenkins pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin * Could it have been written a jenkins plugin instead of a whole new CI software. I like some of the features of concourse pipeline but it doesn't have support for wide range of remoting/plugins that jenkins supports.
    - jacques_chester 11y ago
      I'll defer to the authors for their experiences with Jenkins: http://concourse.ci/concourse-vs.html http://concourse.ci/concourse-vs.html As for writing a plugin, no, it would not have been possible. Concourse has an entirely different model of operation and needs easy access to containerisation facilities to achieve it. Concourse doesn't really think in terms of "plugins". What you become accustomed to is wondering "is there a resource type for this?". Right now I work on a software repo with a moderately complex API for uploading final binaries and releasing them to clients to download. Instead of telling people to write scripts or install a plugin, I can point them to the resource that another team has written. "Just add the resource". Released software is now just a stream of events, no different from git commits, S3 files, points in time, Tracker stories etc etc. Every resource has the same interface so it makes it possible to click together stuff into clever combinations, rather than lashing together awkwardly and hoping it'll work.
javajosh 11y ago
Is it more reasonable to try to avoid this problem in the first place, or to accept that it's likely to happen and deal with it then? Both options seem reasonable to me (and I'm trying to get my company to adopt Jenkins).
escapee 11y ago
We fought the copy+paste drift for a while. Most jobs were very similar, but just different enough that debugging things when something went wrong was often both time consuming and frustrating. Ultimately, we took an approach similar to Travis CI, or Gitlab CI [1], only using shell scripts since that plugs into Jenkins easily enough. Every project has a CI script and a release script in a common location relative to the project root that takes care of everything needed to take a fresh clone of a repository, run the tests, and deploy the project (depending on a few environment variables) if the tests pass. We have 1-click operation to set up a new job in Jenkins, and it handles all the configuration based on an XML template. Everyone understands that they're not supposed to make manual tweaks to the jobs once they're set up, and a year later, things are working pretty smooth. [1]: http://doc.gitlab.com/ce/ci/quick_start/README.html http://doc.gitlab.com/ce/ci/quick_start/README.html
- killface 11y ago
  This way is the only way we've found to do it, even on a smaller scale. Once you get above 20-25 jobs, or create a self service way for teams to create projects+github+jira+cloudformation+jenkins etc, you have to aggressively standardize. Often this means standardizing on the lowest common denominator (shell script). thanks for sharing.
yamitty11 11y ago
> A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. Could use something like this if you are using github https://github.com/groupon/DotCi https://github.com/groupon/DotCi Job configurations can be version controlled and reviewed like everything else. Also, the plugin does other optimizations like storing job/build data in a db so jenkins doesn't slow down as you create more builds/jobs. So you don't need "25 Jenkins masters". For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
paulddraper 11y ago
Jenkins DSL plugin and/or configuration management (chef/puppet) is necessary.
bmoyles 11y ago
We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles the very basic cases with Jenkins for more complex solutions. Our biggest problems are probably plugin creep (and the poor state of plugin maintenance in some cases), the cluttered UI with chunks of configuration hidden behind "Advanced" buttons, the inability to store job configuration with code a la Travis (which may be better with workflow), and handling routine maintenance across shards (without something in place like Operations Center), API inconsistencies (and the inability to fully manage via the API, forcing things like Groovy scripts over Jenkins remoting...). Once we have some direction on where to go, I'm sure we'll follow up with some more blog posts and share things at our meetups.
- jacques_chester 11y ago
  I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built. I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc). People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource. So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.
  - geodel 11y ago
    I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?
    - jacques_chester 11y ago
igor47 11y ago
I've spoken to multiple companies about how they do builds, and it's incredible (to me) how many of them use Jenkins. In the end, almost everyone ends up with the kinds of problems you can see on this thread: * Plugins are useless * Configuration drifts all over the place * Jenkins ends up just being used as a job runner to run shell scripts * Every repository or project implements similar but subtly different build scripts When we faced this exact problem while writing Deployboard (https://www.youtube.com/watch?v=tgmJa7FciDg https://www.youtube.com/watch?v=tgmJa7FciDg, especially starting at around 18:45), specifically the build system there. I explicitly wanted to avoid having engineers write shell scripts. People don't really know how to write them properly, nobody ever tests them, and they're just not really treated the same way you treat "real code". Instead, we ended up using resque. We're a rails shop and already use resque extensively in production. Resque scales really really well, so we were pretty confident that we would be able to run builds on this sytem indefinitely without needing to split anything into separate clusters. And resque jobs are ruby, so they can be written and tested just like ruby. As a result, we were able to standardize on just a few jobs (e.g., BuildRailsJob, GradleBuildJob, NodeBuildJob) with a few arguments for each job. In the process, we wrote a lot of really nice primitives, so if we do need a different kind of job (or a modification to an existing job) then those can be made pretty easily. On the whole, I've been extremely pleased with the resulting system.
- agentgt 11y ago
  However your not tipping your hat to the huge values Jenkins provides: * Reporting such as unit test reporting * Emailing and notification * A well known plugin system * User security * Master / Slave management Jenkins is not just a job runner. Its a job runner that collects reports, maintains history of the job being run, manages user security, has a public api (REST) and plugin system. That is a lot of stuff to implement on your own. Really Jenkins just needs to fix the configuration drift issue.
- dominotw 11y ago
  lets you have Jenkinsfile in your repo https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin .ci.yml like .travis.yml https://github.com/groupon/DotCi https://github.com/groupon/DotCi solves every single problem you have listed. Looks like jenkins has discoverability issues. I am curious what kind of things you tried with jenkins before writing your own build system.
allengeorge 11y ago
I've seen jenkins devolve into this at one of the previous companies I worked at. There were (at one point) two or three people simply tasked with writing, improving and debugging jenkins plugins as well as random configuration issues and failures. Not a pleasant experience. Our team ended up forgoing the plugins and simply writing shell scripts (checked into our repository) that would handle various pieces of the build workflow. Our jenkins job then became: Call script 1 Call script 2 ... I'm sure there are better alternatives, but at that time it allowed us to version changes to our build, and turned jenkins into nothing more than a glorified task runner - which we were fine with.

by mkobit 11y ago

richardwhiuk 11y ago

Yes - we've definitely seen that. The most common thing we've see if that you start with a jenkins job that just runs a build process, and then you end up with a jenkins job that has a huge shell script which calls the build process somewhere in the middle, that's unversioned. To resolve that we've tried to push the shell script into a file in the repository, which Jenkins then checks out and runs, which makes it easier to maintain and faster to set up new build machines.

christop 11y ago

Netflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-controlled, as opposed to clicking everything together via the web UI. See https://jenkins-ci.org/2.0/ https://jenkins-ci.org/2.0/

oblio 11y ago
Some things are already available, see this: https://github.com/jenkinsci/workflow-plugin/blob/master/README.md#introduction https://github.com/jenkinsci/workflow-plugin/blob/master/REA...
vorg 11y ago
For the last 4 months, Groovy has been known as "Apache Groovy".
- trollingineer 11y ago
  Since your birth, you have been known as loser-vorg.

lintoma 11y ago

It's really the same Jenkins Ami but sharded for different teams. One of the things Spinnaker does is plugin Jenkins jobs as a reusable stage that is parametrized and scoped to an application deploy. This has allowed one team to move from 40 Jenkins jobs created via job dsl plugin to just 6

sytse 11y ago

I think the copy/paste is not inherently bad as long as it is version controlled and visible to everyone, that is why we have all CI configuration in the repo with gitlab-ci.yml I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).

onetwotree 11y ago
Agreed, we have a basically arbitrary number of runners (near releases it gets into the hundreds), and one master handles it just fine.
- sytse 11y ago
  Glad to hear that. We'll soon announce an autoscaling runner that allows you to boot up new instances automatically. I love that you sometimes run hundreds of runners, would you like to do a guest post? Email me at website@sytse.com

onetwotree 11y ago

I think most people do. We did a couple of things to get out of that state at Conjur. First, we made all of our builds into docker containers. Under this system, slaves (we call 'em executors now) only contain very basic software - docker, git, and make to be specific. This means that our builds are entirely self contained, and we don't have to worry about, for example, messing around with RVM on executors. If the containerized build works locally, it pretty much always works on Jenkins as well. Second, we started using the job dsl plugin to manage configuration and brought in some autoscaling and machine identity. Beyond what I do as a platform engineer to turn my projects into jenkins builds, I don't completely understand it (nor do I have to, which is a good thing!), so I'll let our DevOps guy take it from here: https://blog.conjur.net/scaling-jenkins-with-machine-identity https://blog.conjur.net/scaling-jenkins-with-machine-identit...

jacques_chester 11y ago
You might like to look at Concourse[1], which makes explicit, checked-in pipelines of containerized builds its central model. I am very bullish on Concourse. [1] https://concourse.ci https://concourse.ci

Rafert 11y ago

I've seen something similar happen, switched to https://www.go.cd/ https://www.go.cd/

jacques_chester 11y ago
GoCD is very difficult to version-control and the interface is, to put it politely, in need of some love. At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out. Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with Concourse pipelines.
- dominotw 11y ago
  >now pretty much the whole of Cloud Foundry is being built with Concourse pipelines. What are some of your thoughts on, * How it compares to Jenkins pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin * Could it have been written a jenkins plugin instead of a whole new CI software. I like some of the features of concourse pipeline but it doesn't have support for wide range of remoting/plugins that jenkins supports.
  - jacques_chester 11y ago
    I'll defer to the authors for their experiences with Jenkins: http://concourse.ci/concourse-vs.html http://concourse.ci/concourse-vs.html As for writing a plugin, no, it would not have been possible. Concourse has an entirely different model of operation and needs easy access to containerisation facilities to achieve it. Concourse doesn't really think in terms of "plugins". What you become accustomed to is wondering "is there a resource type for this?". Right now I work on a software repo with a moderately complex API for uploading final binaries and releasing them to clients to download. Instead of telling people to write scripts or install a plugin, I can point them to the resource that another team has written. "Just add the resource". Released software is now just a stream of events, no different from git commits, S3 files, points in time, Tracker stories etc etc. Every resource has the same interface so it makes it possible to click together stuff into clever combinations, rather than lashing together awkwardly and hoping it'll work.

javajosh 11y ago

Is it more reasonable to try to avoid this problem in the first place, or to accept that it's likely to happen and deal with it then? Both options seem reasonable to me (and I'm trying to get my company to adopt Jenkins).

escapee 11y ago

We fought the copy+paste drift for a while. Most jobs were very similar, but just different enough that debugging things when something went wrong was often both time consuming and frustrating. Ultimately, we took an approach similar to Travis CI, or Gitlab CI [1], only using shell scripts since that plugs into Jenkins easily enough. Every project has a CI script and a release script in a common location relative to the project root that takes care of everything needed to take a fresh clone of a repository, run the tests, and deploy the project (depending on a few environment variables) if the tests pass. We have 1-click operation to set up a new job in Jenkins, and it handles all the configuration based on an XML template. Everyone understands that they're not supposed to make manual tweaks to the jobs once they're set up, and a year later, things are working pretty smooth. [1]: http://doc.gitlab.com/ce/ci/quick_start/README.html http://doc.gitlab.com/ce/ci/quick_start/README.html

killface 11y ago
This way is the only way we've found to do it, even on a smaller scale. Once you get above 20-25 jobs, or create a self service way for teams to create projects+github+jira+cloudformation+jenkins etc, you have to aggressively standardize. Often this means standardizing on the lowest common denominator (shell script). thanks for sharing.

yamitty11 11y ago

> A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. Could use something like this if you are using github https://github.com/groupon/DotCi https://github.com/groupon/DotCi Job configurations can be version controlled and reviewed like everything else. Also, the plugin does other optimizations like storing job/build data in a db so jenkins doesn't slow down as you create more builds/jobs. So you don't need "25 Jenkins masters". For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

paulddraper 11y ago

Jenkins DSL plugin and/or configuration management (chef/puppet) is necessary.

bmoyles 11y ago

We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles the very basic cases with Jenkins for more complex solutions. Our biggest problems are probably plugin creep (and the poor state of plugin maintenance in some cases), the cluttered UI with chunks of configuration hidden behind "Advanced" buttons, the inability to store job configuration with code a la Travis (which may be better with workflow), and handling routine maintenance across shards (without something in place like Operations Center), API inconsistencies (and the inability to fully manage via the API, forcing things like Groovy scripts over Jenkins remoting...). Once we have some direction on where to go, I'm sure we'll follow up with some more blog posts and share things at our meetups.

jacques_chester 11y ago
I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built. I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc). People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource. So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.
- geodel 11y ago
  I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?
  - jacques_chester 11y ago

igor47 11y ago

I've spoken to multiple companies about how they do builds, and it's incredible (to me) how many of them use Jenkins. In the end, almost everyone ends up with the kinds of problems you can see on this thread: * Plugins are useless * Configuration drifts all over the place * Jenkins ends up just being used as a job runner to run shell scripts * Every repository or project implements similar but subtly different build scripts When we faced this exact problem while writing Deployboard (https://www.youtube.com/watch?v=tgmJa7FciDg https://www.youtube.com/watch?v=tgmJa7FciDg, especially starting at around 18:45), specifically the build system there. I explicitly wanted to avoid having engineers write shell scripts. People don't really know how to write them properly, nobody ever tests them, and they're just not really treated the same way you treat "real code". Instead, we ended up using resque. We're a rails shop and already use resque extensively in production. Resque scales really really well, so we were pretty confident that we would be able to run builds on this sytem indefinitely without needing to split anything into separate clusters. And resque jobs are ruby, so they can be written and tested just like ruby. As a result, we were able to standardize on just a few jobs (e.g., BuildRailsJob, GradleBuildJob, NodeBuildJob) with a few arguments for each job. In the process, we wrote a lot of really nice primitives, so if we do need a different kind of job (or a modification to an existing job) then those can be made pretty easily. On the whole, I've been extremely pleased with the resulting system.

agentgt 11y ago
However your not tipping your hat to the huge values Jenkins provides: * Reporting such as unit test reporting * Emailing and notification * A well known plugin system * User security * Master / Slave management Jenkins is not just a job runner. Its a job runner that collects reports, maintains history of the job being run, manages user security, has a public api (REST) and plugin system. That is a lot of stuff to implement on your own. Really Jenkins just needs to fix the configuration drift issue.
dominotw 11y ago
lets you have Jenkinsfile in your repo https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin .ci.yml like .travis.yml https://github.com/groupon/DotCi https://github.com/groupon/DotCi solves every single problem you have listed. Looks like jenkins has discoverability issues. I am curious what kind of things you tried with jenkins before writing your own build system.

allengeorge 11y ago

I've seen jenkins devolve into this at one of the previous companies I worked at. There were (at one point) two or three people simply tasked with writing, improving and debugging jenkins plugins as well as random configuration issues and failures. Not a pleasant experience. Our team ended up forgoing the plugins and simply writing shell scripts (checked into our repository) that would handle various pieces of the build workflow. Our jenkins job then became: Call script 1 Call script 2 ... I'm sure there are better alternatives, but at that time it allowed us to version changes to our build, and turned jenkins into nothing more than a glorified task runner - which we were fine with.