7 ms·

Jenkins's biggest strength is also its biggest weakness: plugins. Any development shop that has been using Jenkins for a while is using at least a bunch of plug

Jenkins's biggest strength is also its biggest weakness: plugins. Any development shop that has been using Jenkins for a while is using at least a bunch of plugins. Plugins are not stable, they break every now and then. They require constant update with new Jenkins versions. They get abandoned by their creators (hell, many plugins still don't support pipeline). It's a fundamental issue with how Jenkins is set up that I don't know how they can get away with unless they abandon the whole plugin architecture all together. But obviously that's not a solution.

victorin 8y ago
Agree. The same happened with Eclipse IDE
- TremendousJudge 8y ago
  And Firefox to some extent, except they actually fixed it by requiring all addons to be reimplemented. Lesser of two evils and all that
  - foobarian 8y ago
    Less related but Minecraft servers were also a plugin mess. With N plugins, similar odds for any given plugin to break across an update, and frequent updates you'd pretty much always have broken plugins.
oblio 8y ago
They kind of made several plugins "blessed": Pipeline, Blue Ocean, Git, etc. The core package plus these "blessed" plugins is a lot more stable than throwing every random plugin on top of a base installation. Just write a bit of glue script code and you're golden.
- isbvhodnvemrwvn 8y ago
  They still have their own issues. Blue Ocean requires a lot of stuff I have no need for (like github support) which in some cases conflict with stuff I do need (like bitbucket support)
  - chinhodado 8y ago
    Same. Every time there's a Blue Ocean update it requires you to update two dozens other plugins, many of which I don't use and can't get rid of (like the github one). And more annoying is the fact that you can't "select all" to update all plugins, you have to select them one by one.
    - geerlingguy 8y ago
      There’s a select all link at the bottom of the plugin updates page...
    - michaelneale 8y ago
      well glad to see people using blue ocean, but yeah - that isn't a good look (and one of the aims mentioned is to get away from this pain, for good). (also evergreen should take care of the updating, ideally). I will be happy when it does as don't want to spend any more time thinking about this!
- deleted 8y ago
  [deleted]
tiny-dancer 8y ago
I agree, the next big next step would shifting the plug-in system (wiki, downloader, etc) to the "global pipeline libraries" level.
sytse 8y ago
I agree this weakness comes from plugins. Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. So you end up with a slow pace of development while you still end up breaking installations on upgrade. If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. This is what I think they will do with Cloud Native Jenkins. Essentially abandoning plugins. Jenkins Evergreen keeps only the essential plugins. This means they can run better tests. And when introducing new functionality you can update the essential plugins. With GitLab CI we add new functionality in the main code base, avoiding the need for needless configuration and ensuring everything still works when updating. I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-with-the-three-variants-of-jenkins/ https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w...
- zippie 8y ago
  Thank you for all the work on GitLab, we are using AutoDevops extensively. Any thoughts on how AD morphs or adapts with knative? It seems like Jenkins is fully knative with CRD support.
  - sytse 8y ago
    You're welcome, thanks for commenting. Auto DevOps can probably benefit from Knative in two ways. Use Knative build https://github.com/knative/docs/blob/master/build/builder-contract.md https://github.com/knative/docs/blob/master/build/builder-co... for building images. And use Knative serving https://github.com/knative/serving https://github.com/knative/serving to run review apps that don't use resources when not in use https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_90052063 https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_900...
- dominotw 8y ago
  > Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. This is a very simplistic explanation bordering on FUD. Jenkins defines something called 'Extension points' you can introduce new functionality as long as you don't break extension point contract you can continue to add functionality. For example, Greenballs plugin[1] is almost 11 yrs old and still works. Surely jenkins added new functionality in past 11 yrs. > If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. T Another comical statement. You only need to write tests against the contract of extension point and make sure you don't break the contract. > I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.. https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.... This is full of misinformation too. eg: You can checkin Jenkinsfile in the root of your git repo too, you don't have to copy it around. I don't want to attribute maliciousness to you but hope you correct the blog post. 1. https://github.com/jenkinsci/greenballs-plugin/ https://github.com/jenkinsci/greenballs-plugin/
jstrachan 8y ago
yeah, I see 'plugins' being around for a while but docker steps becoming the more cloud native long term alternative; being more reusable stand alone & not requiring changing a Jenkins Master (or even requiring a Jenkins master for ephemeral build pods)
jacques_chester 8y ago
The deeper issue with plugins is that they create global state across everything. Switching the execution engine to Kubernetes will help a lot with operational pain. But consider the comparison to Concourse, which also predates Kubernetes. What drove Concourse into being wasn't the lack of Kubernetes (being born during the Diego project, itself a container scheduler), it was the amount of time and pain and unsafety that came from relying on the status quo. Kawaguchi's agenda is bold and necessary, but I think it's going to take a while to get through even half of it. But the world is better off when Jenkins improves, simply because of its phenomenal installation base. We all talk about Travis and Circle and Drone and Concourse and Gitlab here, but I would bet folding money that over 75% of actual bits going through CI are going through some version of Jenkins. Disclosure: I work for Pivotal, we sponsor Concourse.
- dominotw 8y ago
  I soon as I saw the word concourse, I look at the username, sure enough it was you. You bring up Concourse comparision every single jenkins thread. I've considered it the past but looked like there is almost no adoption outside pivotal. would you say the adoption increasing in 2018, I would like to give it another spin if possible.
  - jacques_chester 8y ago
    > You bring up Concourse comparision every single jenkins thread. Because it's the reference point I know best. It would be silly of me to compare it to microbiology or the internals of Travis, both of which I'm much less familiar with than Concourse. > would you say the adoption increasing in 2018, I would like to give it another spin if possible. It's nowhere near as popular as Jenkins, by at least 2 orders of magnitude, maybe 3. If that is important to you, wait a bit.
  - baylisscg 8y ago
    We looked at it. It was our second choice. Ultimately, its entirely self-hosted nature was it's greatest strength and weakness. We like self-hosting but don't have the resources to get it going at the moment. It's interesting enough that we'll keep revisiting it.
michaelneale 8y ago
Part of the ideas mentioned are to resolve this stability, and not depend on in process plugins (a new extensibility architecture that won't hurt stability). There are many things in plugins which should be core functionality (and will be).
- peterwwillis 8y ago
  That's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Like, there's no official backup functionality? And why is version control not standard in 2018? This isn't something you just bolt on, or incorporate as a response to competing products. I think they should abandon all hope of Jenkins being competitive. They should remain the weird old school universal tool it always was, and let it become relegated to legacy systems, like the Apache web server. Jenkins was useful, but it's living in the past and trying to solve the wrong problems.
  - michaelneale 8y ago
    >That's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Oh there are core bundled plugins, official etc - they are just core functionality that happens to be implemented by plugins. >And why is version control not standard in 2018? that is and always has been - "git" support used to be not in by default, but that was a while ago (it is included now).
kohsuke 8y ago
Author of the post. I agree that it is our biggest strength & weakness, I acknowledge that problem and put forward some solutions in the doc. One piece of the solution is to embrace core and a bunch of important plugins together as the foundation. Normal users shouldn't be asked to pick & choose the basics like that, and we want to lock down the combination of the versions in that group. Whether those are behind the scene plugins or not from contributors' perspective is an implementation detail. Another piece of the solution is to grow more extensibility mechanism beyond the current in-process plugin. There's a thing called "Pipeline shared libraries" in Jenkins, which is a good example of this. It lets developers create higher level pipeline primitives by composing other existing ones. There's some mechanism to share those with the community, too, although not as sophisticated as plugins. From users' perspective, it extends capabilities of Jenkins just like plugins, but in a way that doesn't create the kind of instability a bad plugin can -- its impact is local to one build, for example. Then there's the container-as-a-building-block extensibility, Jenkins Evergreen, and more...

victorin 8y ago

Agree. The same happened with Eclipse IDE

TremendousJudge 8y ago
And Firefox to some extent, except they actually fixed it by requiring all addons to be reimplemented. Lesser of two evils and all that
- foobarian 8y ago
  Less related but Minecraft servers were also a plugin mess. With N plugins, similar odds for any given plugin to break across an update, and frequent updates you'd pretty much always have broken plugins.

oblio 8y ago

They kind of made several plugins "blessed": Pipeline, Blue Ocean, Git, etc. The core package plus these "blessed" plugins is a lot more stable than throwing every random plugin on top of a base installation. Just write a bit of glue script code and you're golden.

isbvhodnvemrwvn 8y ago
They still have their own issues. Blue Ocean requires a lot of stuff I have no need for (like github support) which in some cases conflict with stuff I do need (like bitbucket support)
- chinhodado 8y ago
  Same. Every time there's a Blue Ocean update it requires you to update two dozens other plugins, many of which I don't use and can't get rid of (like the github one). And more annoying is the fact that you can't "select all" to update all plugins, you have to select them one by one.
  - geerlingguy 8y ago
    There’s a select all link at the bottom of the plugin updates page...
  - michaelneale 8y ago
    well glad to see people using blue ocean, but yeah - that isn't a good look (and one of the aims mentioned is to get away from this pain, for good). (also evergreen should take care of the updating, ideally). I will be happy when it does as don't want to spend any more time thinking about this!
deleted 8y ago
[deleted]

tiny-dancer 8y ago

I agree, the next big next step would shifting the plug-in system (wiki, downloader, etc) to the "global pipeline libraries" level.

sytse 8y ago

I agree this weakness comes from plugins. Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. So you end up with a slow pace of development while you still end up breaking installations on upgrade. If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. This is what I think they will do with Cloud Native Jenkins. Essentially abandoning plugins. Jenkins Evergreen keeps only the essential plugins. This means they can run better tests. And when introducing new functionality you can update the essential plugins. With GitLab CI we add new functionality in the main code base, avoiding the need for needless configuration and ensuring everything still works when updating. I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-with-the-three-variants-of-jenkins/ https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w...

zippie 8y ago
Thank you for all the work on GitLab, we are using AutoDevops extensively. Any thoughts on how AD morphs or adapts with knative? It seems like Jenkins is fully knative with CRD support.
- sytse 8y ago
  You're welcome, thanks for commenting. Auto DevOps can probably benefit from Knative in two ways. Use Knative build https://github.com/knative/docs/blob/master/build/builder-contract.md https://github.com/knative/docs/blob/master/build/builder-co... for building images. And use Knative serving https://github.com/knative/serving https://github.com/knative/serving to run review apps that don't use resources when not in use https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_90052063 https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_900...
dominotw 8y ago
> Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. This is a very simplistic explanation bordering on FUD. Jenkins defines something called 'Extension points' you can introduce new functionality as long as you don't break extension point contract you can continue to add functionality. For example, Greenballs plugin[1] is almost 11 yrs old and still works. Surely jenkins added new functionality in past 11 yrs. > If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. T Another comical statement. You only need to write tests against the contract of extension point and make sure you don't break the contract. > I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.. https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.... This is full of misinformation too. eg: You can checkin Jenkinsfile in the root of your git repo too, you don't have to copy it around. I don't want to attribute maliciousness to you but hope you correct the blog post. 1. https://github.com/jenkinsci/greenballs-plugin/ https://github.com/jenkinsci/greenballs-plugin/

jstrachan 8y ago

yeah, I see 'plugins' being around for a while but docker steps becoming the more cloud native long term alternative; being more reusable stand alone & not requiring changing a Jenkins Master (or even requiring a Jenkins master for ephemeral build pods)

jacques_chester 8y ago

The deeper issue with plugins is that they create global state across everything. Switching the execution engine to Kubernetes will help a lot with operational pain. But consider the comparison to Concourse, which also predates Kubernetes. What drove Concourse into being wasn't the lack of Kubernetes (being born during the Diego project, itself a container scheduler), it was the amount of time and pain and unsafety that came from relying on the status quo. Kawaguchi's agenda is bold and necessary, but I think it's going to take a while to get through even half of it. But the world is better off when Jenkins improves, simply because of its phenomenal installation base. We all talk about Travis and Circle and Drone and Concourse and Gitlab here, but I would bet folding money that over 75% of actual bits going through CI are going through some version of Jenkins. Disclosure: I work for Pivotal, we sponsor Concourse.

dominotw 8y ago
I soon as I saw the word concourse, I look at the username, sure enough it was you. You bring up Concourse comparision every single jenkins thread. I've considered it the past but looked like there is almost no adoption outside pivotal. would you say the adoption increasing in 2018, I would like to give it another spin if possible.
- jacques_chester 8y ago
  > You bring up Concourse comparision every single jenkins thread. Because it's the reference point I know best. It would be silly of me to compare it to microbiology or the internals of Travis, both of which I'm much less familiar with than Concourse. > would you say the adoption increasing in 2018, I would like to give it another spin if possible. It's nowhere near as popular as Jenkins, by at least 2 orders of magnitude, maybe 3. If that is important to you, wait a bit.
- baylisscg 8y ago
  We looked at it. It was our second choice. Ultimately, its entirely self-hosted nature was it's greatest strength and weakness. We like self-hosting but don't have the resources to get it going at the moment. It's interesting enough that we'll keep revisiting it.

michaelneale 8y ago

Part of the ideas mentioned are to resolve this stability, and not depend on in process plugins (a new extensibility architecture that won't hurt stability). There are many things in plugins which should be core functionality (and will be).

peterwwillis 8y ago
That's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Like, there's no official backup functionality? And why is version control not standard in 2018? This isn't something you just bolt on, or incorporate as a response to competing products. I think they should abandon all hope of Jenkins being competitive. They should remain the weird old school universal tool it always was, and let it become relegated to legacy systems, like the Apache web server. Jenkins was useful, but it's living in the past and trying to solve the wrong problems.
- michaelneale 8y ago
  >That's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Oh there are core bundled plugins, official etc - they are just core functionality that happens to be implemented by plugins. >And why is version control not standard in 2018? that is and always has been - "git" support used to be not in by default, but that was a while ago (it is included now).

kohsuke 8y ago

Author of the post. I agree that it is our biggest strength & weakness, I acknowledge that problem and put forward some solutions in the doc. One piece of the solution is to embrace core and a bunch of important plugins together as the foundation. Normal users shouldn't be asked to pick & choose the basics like that, and we want to lock down the combination of the versions in that group. Whether those are behind the scene plugins or not from contributors' perspective is an implementation detail. Another piece of the solution is to grow more extensibility mechanism beyond the current in-process plugin. There's a thing called "Pipeline shared libraries" in Jenkins, which is a good example of this. It lets developers create higher level pipeline primitives by composing other existing ones. There's some mechanism to share those with the community, too, although not as sophisticated as plugins. From users' perspective, it extends capabilities of Jenkins just like plugins, but in a way that doesn't create the kind of instability a bad plugin can -- its impact is local to one build, for example. Then there's the container-as-a-building-block extensibility, Jenkins Evergreen, and more...