7 ms·

I feel so bad for them. Jenkins is really a technology anti-pattern. I keep saying I'll do this, but I really need to write a series of blog posts to elucidate

I feel so bad for them. Jenkins is really a technology anti-pattern. I keep saying I'll do this, but I really need to write a series of blog posts to elucidate all the ways in which Jenkins is just bad for your business. If you can use any alternative, do it, and for Bob's sake, pay for a solution. Stop trying to cobble together some crap with shitty free tools when CI/CD is critical to the velocity, quality, and reliability of your products.

mdeeks 5y ago
As a counter point, we had major issues with just paying for it via CircleCI. Excessive downtimes, a UI so slow it was nearly unusable, etc. We decided at the time we couldn't bet our company on it and we moved to Jenkins. We have far more control, build time and build costs are lower, uptime is better (All of this at the expense of initial dev time and maintenance of course). Generally we are a SaaS company, but paying for a solution just doesn't always work when you get larger and when there are limited options out there.
- diroussel 5y ago
  Try gitlab. You can run it yourself. Great up time. And best of all it’s much easier to run and upgrade than Jenkins. Jenkins and it’s plugins can get quite brittle and is quite an art to automate the installation and config of all those plugins.
  - mdeeks 5y ago
    I'm interested in trying Gitlab, but this does not look easy to run (from their docs): https://plantuml.gitlab-static.net/png/U9oDLbrlsZ0KVVUli9gNBHk55LHQQK8TKAf9rSRAOuUGIGnuDGwp7RNLjF--wzZv83RRNe3SutFlkSVEDJTA4wdpFSVz9vGGcUApEBvzqLGAml5zHb8y9fo8XCeuFiD4OShNEBearsiwh0YVnLy8p12k-yFUT8oGphPKxwZ4ln36ISOvJJJBX39PxfY-9slyG1XNSOnV-WLzopI7QDI_c4px5pvhmLBwn7uQmg0WA1l0qMPpbQOVqEzsLITwBhEzYUOAlyDtaco884xoFXDCPn97mzbiDXtuHIVmdVDQP58yuuWEZJEcfAq2BH7-YYSyL-1GsG-4LeaDXNoOIq8ty5odFxW8ZcQpgqcl5_GEiXwvPWetrbTTkrJpJEcjf8jlzt4SVnVTxfTCu0t9kUxeZ5D9XEvsZwgl3fQYghIGjvBMUNJHPMVaoeVXaP-mDcnAD5aJHOl2gNi8cdGtpd9xoYgNjciRmj7WzXDiNRFjHhVupLooFP6lRotDYLWTRBJ9RjKyIhAWiEkfIODVBeupgHAgSXahAddWXGqRgzhbpSgselEAC0eKgTsJD6Mga5By2ldsK01pAXM3yoSQXwoWh0uLOdRviZyOdien8ScEjkc92VwBMLV72_XWWlBJCg1Fk-I_zBGI7kloKq4kh-sFvuyENdPEgwcOjkC0wRXaiKayoulJ-7NzmxmJ2vWnP4lTqCcAu4hP88wUTqnJXCoyXGDZ9wvxjs0WjkC0kAc5a1kPWDgX2QV63KKVCaIOq05Wr58KuiAYejaQsx4qfMBPa1E0xBX4g1pLaD8EwqfE0F7qXDYLe1hq5OMmKXApjjhxgXvbvRpsyN98mVRQIm1EWhq6bdczMq6it1GFhALwKBDY26p9RIzvBy0xyRIiZKBbPGThx7NNr1q6I-4-MXdfGqtbGRGbtLlZQm_LDwVd93KtN7Xp4BgX8eM_FUWFlOpcMG00 https://plantuml.gitlab-static.net/png/U9oDLbrlsZ0KVVUli9gNB...
    - dfcowell 5y ago
      Unless you’re a very large organization, you’re not going to be managing each of those components individually. The “Omnibus installation” as they call it is pretty easy to use. Just bring a Postgres DB and a redis instance.
      - crummy 5y ago
        You don't even need to BYO databases - they have an omnibus docker compose setup that runs everything for you. https://docs.gitlab.com/omnibus/docker/#install-gitlab-using-docker-compose https://docs.gitlab.com/omnibus/docker/#install-gitlab-using...
        richwater 5y ago
        The discussion chain is everything that's wrong with modern engineering. A complex topic progressively simplified and abstracted to just "Run this container", ensuring no one in your org actually understands how it works, how to maintain it and how to debug it.
        crummy 5y ago
        The GP was interested in "trying" Gitlab. If you want to try it out, but are wary of the time involved, isn't a docker-compose installation a great solution?
        Aeolun 5y ago
        Gitlab is a complex system. I wouldn’t expect anyone to suddenly understand my companies’ product either.
nawgz 5y ago
Can you expand more on how it would qualify as an "anti-pattern"? I agree it is slow, has issues with its built-in coverage and capabilities, and has an oldschool UI; but it is at its fundamental core a pipeline runner. It is a decent pipeline runner even, which when it comes down to it is the core of each other CI product [that I've seen]. So to hear it described as an "anti-pattern", when realistically it seems to BE the pattern - just poorly executed, is a bit unintuitive to me.
- sakisv 5y ago
  (not the person you replied to, but allow me to give a couple of personal annoyances - keeping in mind that they're a couple of years old and things may have improved since) My main problem with Jenkins was that its architecture made it extremely difficult to automate its provisioning without having to click through the ui at all. This led to the second big problem which is that updating either Jenkins or one of its plug-ins when they got a new cve (which was every other day) was quite stressful because you could never be sure if something would break - especially for plugins that depended on other plugins (case in point this post) I have since moved to concourse which has a much more sane architecture - at least for these things.
- orlovs 5y ago
  Probably not possible to describe. Jenkins is just tool, if used wrong it will bite to operators. Problem with jenkins is that it's takes time to setup workflow. You configure to run plugin-install-tool, JaaC, Jobdsl, Shared Libs, some credential store etc. But when it's up and running, oh boy, it's factory. I have run jenkins with 5k+ jobs. All auto gen, no manual interventions etc. Gitlab-CI (same as github actions) I like due opinionated approach. It makes things easier if setup is simple, but when you need exceptions or special cases, hack's begins. But, yes, I have seen so many badly implemented jenkinses.
- 0xbadcafebee 5y ago
  1. it isn't designed as a cloud-native configuration-as-code immutable service. The way it stores and loads configs, jobs, logs, build workspaces, etc is all 1990s tech. Every modern replacement does these things much better. These inherent design flaws set up all the later problems. 2. configuration as code is an afterthought, so it doesn't work very well. 3. the only way to manage Jenkins as described initially requires learning four different DSLs, although for developers to write jobs only requires learning three (JobDSL to load jobs from JCasC, Jenkinsfile for simple pipelines, Groovy for complex ones). This is ridiculous. 4. the plugins are atrocious, there's too many of them, they don't have good enough features much of the time, managing and upgrading them is always a pain. 5. CloudBees doesn't even maintain the core stuff correctly. The current Jenkins container comes with a new plugin manager which, by default, does not respect pinned plugin versions. That's literally the most basic thing you can do for operational stability. I filed a bug in January, and they didn't feel like fixing it, so I got them to merge a note at the bottom of their README instead mentioning the bug. 6. pipeline libraries are a costly maintenance and development pain. Having to write Groovy code just to write pipelines is horrible. Jenkinsfiles are, although much better than Groovy, still an over-complicated, unintuitive mess. 7. there's no simple way to deploy, maintain, test, and upgrade a Jenkins cluster. You have to maintain multiple clusters, increasing cost and complexity. 8. since most people don't set it up right (because it is so overcomplicated), the jobs, server configurations, and build history are not backed up, there's no version control. So when something goes wrong, the whole thing is hosed. Unless a Jenkins expert took 6 weeks to set it up perfectly. 9. due to all the above problems, you end up with a million different Jenkinses, all in various states of insecurity, brokenness, and wildly different configuration, making them incompatible with each other. This makes for a gigantic maintenance cost that never ends. 10. literally all of it is completely proprietary to Jenkins. Unless you build it and the jobs in a very particular way (which makes it impractical to use) none of it can be re-used in a different system. That's off the top of my head. There's more reasons. The point is, organizations will invest literally thousands of man-hours in making Jenkins work, slowing down their product development, forcing everyone to use this old-ass over-complicated piece of junk. If they took the same amount of cash they could buy literally any proprietary CI/CD system and do everything much faster and better. But the organization doesn't see the hidden costs until it's too late and they desperately want to replace it. Jenkins is not just bad, it actively holds back your organization.
coredog64 5y ago
I'll note that Cloudbees offers commercial support for Jenkins. Having said that, I had a former employer that was a Cloudbees customer and the support we got was typically "Have you tried turning it off and back on?" That drove us back to OSS Jenkins. Although, AIUI, said employer has now moved to Harness.
aranchelk 5y ago
Modern Jenkins using JenkinsFiles and cloud provisioning can work pretty well. All things being equal, it's my preference to not use Jenkins for new projects, but anti-pattern is IMO a significant overstatement.

mdeeks 5y ago

As a counter point, we had major issues with just paying for it via CircleCI. Excessive downtimes, a UI so slow it was nearly unusable, etc. We decided at the time we couldn't bet our company on it and we moved to Jenkins. We have far more control, build time and build costs are lower, uptime is better (All of this at the expense of initial dev time and maintenance of course). Generally we are a SaaS company, but paying for a solution just doesn't always work when you get larger and when there are limited options out there.

diroussel 5y ago
Try gitlab. You can run it yourself. Great up time. And best of all it’s much easier to run and upgrade than Jenkins. Jenkins and it’s plugins can get quite brittle and is quite an art to automate the installation and config of all those plugins.
- mdeeks 5y ago
  I'm interested in trying Gitlab, but this does not look easy to run (from their docs): https://plantuml.gitlab-static.net/png/U9oDLbrlsZ0KVVUli9gNBHk55LHQQK8TKAf9rSRAOuUGIGnuDGwp7RNLjF--wzZv83RRNe3SutFlkSVEDJTA4wdpFSVz9vGGcUApEBvzqLGAml5zHb8y9fo8XCeuFiD4OShNEBearsiwh0YVnLy8p12k-yFUT8oGphPKxwZ4ln36ISOvJJJBX39PxfY-9slyG1XNSOnV-WLzopI7QDI_c4px5pvhmLBwn7uQmg0WA1l0qMPpbQOVqEzsLITwBhEzYUOAlyDtaco884xoFXDCPn97mzbiDXtuHIVmdVDQP58yuuWEZJEcfAq2BH7-YYSyL-1GsG-4LeaDXNoOIq8ty5odFxW8ZcQpgqcl5_GEiXwvPWetrbTTkrJpJEcjf8jlzt4SVnVTxfTCu0t9kUxeZ5D9XEvsZwgl3fQYghIGjvBMUNJHPMVaoeVXaP-mDcnAD5aJHOl2gNi8cdGtpd9xoYgNjciRmj7WzXDiNRFjHhVupLooFP6lRotDYLWTRBJ9RjKyIhAWiEkfIODVBeupgHAgSXahAddWXGqRgzhbpSgselEAC0eKgTsJD6Mga5By2ldsK01pAXM3yoSQXwoWh0uLOdRviZyOdien8ScEjkc92VwBMLV72_XWWlBJCg1Fk-I_zBGI7kloKq4kh-sFvuyENdPEgwcOjkC0wRXaiKayoulJ-7NzmxmJ2vWnP4lTqCcAu4hP88wUTqnJXCoyXGDZ9wvxjs0WjkC0kAc5a1kPWDgX2QV63KKVCaIOq05Wr58KuiAYejaQsx4qfMBPa1E0xBX4g1pLaD8EwqfE0F7qXDYLe1hq5OMmKXApjjhxgXvbvRpsyN98mVRQIm1EWhq6bdczMq6it1GFhALwKBDY26p9RIzvBy0xyRIiZKBbPGThx7NNr1q6I-4-MXdfGqtbGRGbtLlZQm_LDwVd93KtN7Xp4BgX8eM_FUWFlOpcMG00 https://plantuml.gitlab-static.net/png/U9oDLbrlsZ0KVVUli9gNB...
  - dfcowell 5y ago
    Unless you’re a very large organization, you’re not going to be managing each of those components individually. The “Omnibus installation” as they call it is pretty easy to use. Just bring a Postgres DB and a redis instance.
    - crummy 5y ago
      You don't even need to BYO databases - they have an omnibus docker compose setup that runs everything for you. https://docs.gitlab.com/omnibus/docker/#install-gitlab-using-docker-compose https://docs.gitlab.com/omnibus/docker/#install-gitlab-using...
      - richwater 5y ago
        The discussion chain is everything that's wrong with modern engineering. A complex topic progressively simplified and abstracted to just "Run this container", ensuring no one in your org actually understands how it works, how to maintain it and how to debug it.
        crummy 5y ago
        The GP was interested in "trying" Gitlab. If you want to try it out, but are wary of the time involved, isn't a docker-compose installation a great solution?
        Aeolun 5y ago
        Gitlab is a complex system. I wouldn’t expect anyone to suddenly understand my companies’ product either.

nawgz 5y ago

Can you expand more on how it would qualify as an "anti-pattern"? I agree it is slow, has issues with its built-in coverage and capabilities, and has an oldschool UI; but it is at its fundamental core a pipeline runner. It is a decent pipeline runner even, which when it comes down to it is the core of each other CI product [that I've seen]. So to hear it described as an "anti-pattern", when realistically it seems to BE the pattern - just poorly executed, is a bit unintuitive to me.

sakisv 5y ago
(not the person you replied to, but allow me to give a couple of personal annoyances - keeping in mind that they're a couple of years old and things may have improved since) My main problem with Jenkins was that its architecture made it extremely difficult to automate its provisioning without having to click through the ui at all. This led to the second big problem which is that updating either Jenkins or one of its plug-ins when they got a new cve (which was every other day) was quite stressful because you could never be sure if something would break - especially for plugins that depended on other plugins (case in point this post) I have since moved to concourse which has a much more sane architecture - at least for these things.
orlovs 5y ago
Probably not possible to describe. Jenkins is just tool, if used wrong it will bite to operators. Problem with jenkins is that it's takes time to setup workflow. You configure to run plugin-install-tool, JaaC, Jobdsl, Shared Libs, some credential store etc. But when it's up and running, oh boy, it's factory. I have run jenkins with 5k+ jobs. All auto gen, no manual interventions etc. Gitlab-CI (same as github actions) I like due opinionated approach. It makes things easier if setup is simple, but when you need exceptions or special cases, hack's begins. But, yes, I have seen so many badly implemented jenkinses.
0xbadcafebee 5y ago
1. it isn't designed as a cloud-native configuration-as-code immutable service. The way it stores and loads configs, jobs, logs, build workspaces, etc is all 1990s tech. Every modern replacement does these things much better. These inherent design flaws set up all the later problems. 2. configuration as code is an afterthought, so it doesn't work very well. 3. the only way to manage Jenkins as described initially requires learning four different DSLs, although for developers to write jobs only requires learning three (JobDSL to load jobs from JCasC, Jenkinsfile for simple pipelines, Groovy for complex ones). This is ridiculous. 4. the plugins are atrocious, there's too many of them, they don't have good enough features much of the time, managing and upgrading them is always a pain. 5. CloudBees doesn't even maintain the core stuff correctly. The current Jenkins container comes with a new plugin manager which, by default, does not respect pinned plugin versions. That's literally the most basic thing you can do for operational stability. I filed a bug in January, and they didn't feel like fixing it, so I got them to merge a note at the bottom of their README instead mentioning the bug. 6. pipeline libraries are a costly maintenance and development pain. Having to write Groovy code just to write pipelines is horrible. Jenkinsfiles are, although much better than Groovy, still an over-complicated, unintuitive mess. 7. there's no simple way to deploy, maintain, test, and upgrade a Jenkins cluster. You have to maintain multiple clusters, increasing cost and complexity. 8. since most people don't set it up right (because it is so overcomplicated), the jobs, server configurations, and build history are not backed up, there's no version control. So when something goes wrong, the whole thing is hosed. Unless a Jenkins expert took 6 weeks to set it up perfectly. 9. due to all the above problems, you end up with a million different Jenkinses, all in various states of insecurity, brokenness, and wildly different configuration, making them incompatible with each other. This makes for a gigantic maintenance cost that never ends. 10. literally all of it is completely proprietary to Jenkins. Unless you build it and the jobs in a very particular way (which makes it impractical to use) none of it can be re-used in a different system. That's off the top of my head. There's more reasons. The point is, organizations will invest literally thousands of man-hours in making Jenkins work, slowing down their product development, forcing everyone to use this old-ass over-complicated piece of junk. If they took the same amount of cash they could buy literally any proprietary CI/CD system and do everything much faster and better. But the organization doesn't see the hidden costs until it's too late and they desperately want to replace it. Jenkins is not just bad, it actively holds back your organization.

coredog64 5y ago

I'll note that Cloudbees offers commercial support for Jenkins. Having said that, I had a former employer that was a Cloudbees customer and the support we got was typically "Have you tried turning it off and back on?" That drove us back to OSS Jenkins. Although, AIUI, said employer has now moved to Harness.

aranchelk 5y ago

Modern Jenkins using JenkinsFiles and cloud provisioning can work pretty well. All things being equal, it's my preference to not use Jenkins for new projects, but anti-pattern is IMO a significant overstatement.