13 ms·
Shifting Gears
- chinhodado 8y agoJenkins's biggest strength is also its biggest weakness: plugins. Any development shop that has been using Jenkins for a while is using at least a bunch of plugins. Plugins are not stable, they break every now and then. They require constant update with new Jenkins versions. They get abandoned by their creators (hell, many plugins still don't support pipeline). It's a fundamental issue with how Jenkins is set up that I don't know how they can get away with unless they abandon the whole plugin architecture all together. But obviously that's not a solution.
- victorin 8y agoAgree. The same happened with Eclipse IDE
- TremendousJudge 8y agoAnd Firefox to some extent, except they actually fixed it by requiring all addons to be reimplemented. Lesser of two evils and all that
- foobarian 8y agoLess related but Minecraft servers were also a plugin mess. With N plugins, similar odds for any given plugin to break across an update, and frequent updates you'd pretty much always have broken plugins.
- oblio 8y agoThey kind of made several plugins "blessed": Pipeline, Blue Ocean, Git, etc. The core package plus these "blessed" plugins is a lot more stable than throwing every random plugin on top of a base installation. Just write a bit of glue script code and you're golden.
- isbvhodnvemrwvn 8y agoThey still have their own issues. Blue Ocean requires a lot of stuff I have no need for (like github support) which in some cases conflict with stuff I do need (like bitbucket support)
- chinhodado 8y agoSame. Every time there's a Blue Ocean update it requires you to update two dozens other plugins, many of which I don't use and can't get rid of (like the github one). And more annoying is the fact that you can't "select all" to update all plugins, you have to select them one by one.
- geerlingguy 8y agoThere’s a select all link at the bottom of the plugin updates page...
- michaelneale 8y agowell glad to see people using blue ocean, but yeah - that isn't a good look (and one of the aims mentioned is to get away from this pain, for good). (also evergreen should take care of the updating, ideally). I will be happy when it does as don't want to spend any more time thinking about this!
- deleted 8y ago[deleted]
- tiny-dancer 8y agoI agree, the next big next step would shifting the plug-in system (wiki, downloader, etc) to the "global pipeline libraries" level.
- sytse 8y agoI agree this weakness comes from plugins. Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. So you end up with a slow pace of development while you still end up breaking installations on upgrade. If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. This is what I think they will do with Cloud Native Jenkins. Essentially abandoning plugins. Jenkins Evergreen keeps only the essential plugins. This means they can run better tests. And when introducing new functionality you can update the essential plugins. With GitLab CI we add new functionality in the main code base, avoiding the need for needless configuration and ensuring everything still works when updating. I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-with-the-three-variants-of-jenkins/ https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w...
- zippie 8y agoThank you for all the work on GitLab, we are using AutoDevops extensively. Any thoughts on how AD morphs or adapts with knative? It seems like Jenkins is fully knative with CRD support.
- sytse 8y agoYou're welcome, thanks for commenting. Auto DevOps can probably benefit from Knative in two ways. Use Knative build https://github.com/knative/docs/blob/master/build/builder-contract.md https://github.com/knative/docs/blob/master/build/builder-co... for building images. And use Knative serving https://github.com/knative/serving https://github.com/knative/serving to run review apps that don't use resources when not in use https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_90052063 https://gitlab.com/gitlab-org/gitlab-ee/issues/3585#note_900...
- dominotw 8y ago> Because the plugins are not part of the main code base you can't introduce new functionality without breaking them. This is a very simplistic explanation bordering on FUD. Jenkins defines something called 'Extension points' you can introduce new functionality as long as you don't break extension point contract you can continue to add functionality. For example, Greenballs plugin[1] is almost 11 yrs old and still works. Surely jenkins added new functionality in past 11 yrs. > If you add functionality to the main codebase you can keep running your tests to ensure nothing breaks. T Another comical statement. You only need to write tests against the contract of extension point and make sure you don't break the contract. > I have just written a more extensive analysis of the blog post in https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.. https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w.... This is full of misinformation too. eg: You can checkin Jenkinsfile in the root of your git repo too, you don't have to copy it around. I don't want to attribute maliciousness to you but hope you correct the blog post. 1. https://github.com/jenkinsci/greenballs-plugin/ https://github.com/jenkinsci/greenballs-plugin/
- jstrachan 8y agoyeah, I see 'plugins' being around for a while but docker steps becoming the more cloud native long term alternative; being more reusable stand alone & not requiring changing a Jenkins Master (or even requiring a Jenkins master for ephemeral build pods)
- jacques_chester 8y agoThe deeper issue with plugins is that they create global state across everything. Switching the execution engine to Kubernetes will help a lot with operational pain. But consider the comparison to Concourse, which also predates Kubernetes. What drove Concourse into being wasn't the lack of Kubernetes (being born during the Diego project, itself a container scheduler), it was the amount of time and pain and unsafety that came from relying on the status quo. Kawaguchi's agenda is bold and necessary, but I think it's going to take a while to get through even half of it. But the world is better off when Jenkins improves, simply because of its phenomenal installation base. We all talk about Travis and Circle and Drone and Concourse and Gitlab here, but I would bet folding money that over 75% of actual bits going through CI are going through some version of Jenkins. Disclosure: I work for Pivotal, we sponsor Concourse.
- dominotw 8y agoI soon as I saw the word concourse, I look at the username, sure enough it was you. You bring up Concourse comparision every single jenkins thread. I've considered it the past but looked like there is almost no adoption outside pivotal. would you say the adoption increasing in 2018, I would like to give it another spin if possible.
- jacques_chester 8y ago> You bring up Concourse comparision every single jenkins thread. Because it's the reference point I know best. It would be silly of me to compare it to microbiology or the internals of Travis, both of which I'm much less familiar with than Concourse. > would you say the adoption increasing in 2018, I would like to give it another spin if possible. It's nowhere near as popular as Jenkins, by at least 2 orders of magnitude, maybe 3. If that is important to you, wait a bit.
- baylisscg 8y agoWe looked at it. It was our second choice. Ultimately, its entirely self-hosted nature was it's greatest strength and weakness. We like self-hosting but don't have the resources to get it going at the moment. It's interesting enough that we'll keep revisiting it.
- michaelneale 8y agoPart of the ideas mentioned are to resolve this stability, and not depend on in process plugins (a new extensibility architecture that won't hurt stability). There are many things in plugins which should be core functionality (and will be).
- peterwwillis 8y agoThat's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Like, there's no official backup functionality? And why is version control not standard in 2018? This isn't something you just bolt on, or incorporate as a response to competing products. I think they should abandon all hope of Jenkins being competitive. They should remain the weird old school universal tool it always was, and let it become relegated to legacy systems, like the Apache web server. Jenkins was useful, but it's living in the past and trying to solve the wrong problems.
- michaelneale 8y ago>That's just the thing. Why has there been so little adoption of commonly needed functionality as part of core, or at least as officially supported plugins? Oh there are core bundled plugins, official etc - they are just core functionality that happens to be implemented by plugins. >And why is version control not standard in 2018? that is and always has been - "git" support used to be not in by default, but that was a while ago (it is included now).
- kohsuke 8y agoAuthor of the post. I agree that it is our biggest strength & weakness, I acknowledge that problem and put forward some solutions in the doc. One piece of the solution is to embrace core and a bunch of important plugins together as the foundation. Normal users shouldn't be asked to pick & choose the basics like that, and we want to lock down the combination of the versions in that group. Whether those are behind the scene plugins or not from contributors' perspective is an implementation detail. Another piece of the solution is to grow more extensibility mechanism beyond the current in-process plugin. There's a thing called "Pipeline shared libraries" in Jenkins, which is a good example of this. It lets developers create higher level pipeline primitives by composing other existing ones. There's some mechanism to share those with the community, too, although not as sophisticated as plugins. From users' perspective, it extends capabilities of Jenkins just like plugins, but in a way that doesn't create the kind of instability a bad plugin can -- its impact is local to one build, for example. Then there's the container-as-a-building-block extensibility, Jenkins Evergreen, and more...
- lima 8y agoOpenShift's Jenkins <-> Kubernetes integration plugin is pretty neat. Authentication, SSH secrets and - most importantly - running each build in an ephemeral pod works out of the box. https://docs.openshift.com/container-platform/3.9/dev_guide/dev_tutorials/openshift_pipeline.html https://docs.openshift.com/container-platform/3.9/dev_guide/...
- deleted 8y ago[deleted]
- 0x8BADF00D 8y agoYou can just use helm to install the default chart for Jenkins, and override the specific configuration you need. But it’s still very clunky to work with, and spinning up a pod only happens when the build queue is not empty. It’s extremely slow.
- jstrachan 8y agoOpenShift's Jenkins integration is good. Though its even cooler to use Jenkins X on OpenShift as you get automated CI/CD pipelines + Environments, Preview Environments on Pull Requests and GitOps based Promotion between environments.
- mbubb 8y agoI love working with Jenkins - I know it is a pain to keep up to date but for me it has become a way as a sole or small team syseng to manage all kinds of stuff. "Jenkins-Ansible-Github" where you have a Jenkinsfile sitting in the git repo you are bulding/ deploying etc., has been a pretty good set of tools to manage heterogeneous environments.
- batbomb 8y agoI wish we could get encrypted credentials a lá travis in a Jenkinsfile. I’ve found most configuration for a job can be in a git repo but you have to manage some things through the web interface, and it’s not that easy securely managing credentials for a Jenkins installation, even with Folders and Roles
- jstrachan 8y agoBTW we use the kubernetes credentials provider plugin in Jenkins X which exposes Kubernetes Secrets as Jenkins Credentials; then the `credentials` step in the `Jenkinsfile` encrypts them from any build logs
- csanchez 8y agoyou can fetch the credentials from Kubernetes secrets using this plugin https://jenkinsci.github.io/kubernetes-credentials-provider-plugin/ https://jenkinsci.github.io/kubernetes-credentials-provider-... and manage both your Ops and Jenkins credentials the same way
- Rapzid 8y agoTravis recommends storing secrets that dont change between branches via the project settings. Also, secure envars in the Travis config appears to interfere with deep merging on triggered builds.. Hopefully they can do better :)
- Thaxll 8y agoI'm trying Gitlab atm, it's great to see something simpler than Jenkins to do CI/CD.
- dsumenkovic 8y agoGlad to hear that. We'd love to hear your feedback about GitLab CI/CD.
- Rowern 8y agoI've been working with gitlab CI for the last year. Here are some of my feedbacks: - 6 months ago we seriously considered moving away because it was really unstable (even when running on private runners) but now its a lot smoother - with private runners you can have a very powerful CI without having to manage a master (as Jenkins) for a fraction of the costs (runner with docker-machine on spot instances) - beware that if your CI flow is more complex than just a simple pipeline to build and deploy your project (we have a project for our code, that then trigger a project for end-to-end tests, that then trigger a deploy to our env) you will need to do a lot of boilerplate code (you will need to manually manage artifacts if they need to be shared between jobs) - variables from a triggered pipeline should be available through the API and made more visible in the UI - we do not use kubernetes so eveything CD is off the plate for us (environment and monitoring tab are useless) - DO NOT USE THE BUILT IN CACHE, it's super slow and will fail unexpectedly (simply do cp to s3 and it will never fail) - IF YOU USE THE BUILT IN CACHE, parallelism will be hard (you cannot populate part of the cache from a job, another part from another job and in the next step use the result of both cache) - triggers are weird, its a curl to an API endpoint but it does not use the normal auth mechanism and it will answer with a useless json (please add the project id, variables etc to the result of the trigger it's a must have for anyone that needs to parse the output) - the gitlab API is top notch except on the CI part... - be ready to restart some jobs 2-3 times if gitlab is deploying a new version ;) - be ready to have some random errors that can be fixed by a retry - it will seem a good idea to run gitlab-runner on every laptop of your team to reduce cost. DO NOT DO THAT, if you are more than 2 in your team the guy in charge of making the CI run (me) will make you restart you docker, delete a specific image, restart gitlab-runner, etc... invest 1 day to setup the docker machine on spot - please show in some way when a job triggered another one (maybe a section in the YML, or even better check make us populate an env var with a link to the triggered pipeline or anything) - design your pipeline so that if a part fails you can restart it without breaking everything (I'm looking at you terraform) This list seem really long but, I have worked with Jenkins and even if more stable the steady improvements and addition to gitlab CI still make it my first choice for my needs.
- toredash 8y agoI'm surprised that I haven't seen more of this announcement in my media streams. I'm not a big fan of Jenkins, I find it overly complex and a Swiss army knife. When you can do anything, you often end up with poor implementations (IMO). If the tools you have are restrictive but useful enough, I find it easier to adopt my workflow for the tools instead of demanding that my complex workflow fit into this one tool.
- cfontes 8y agoNice to know there is a plan and it's refreshing to see that they can understand most of the problems from the customer perspective now. I hope they address the constant shifts in focus with this plan and Jenkins can secure it's market spot, it really deserves it from a historical point of view in the least. It should not be a Nokia or a Xerox, it's better than that and has been a major tool for the industry. The whole CRD is a great way to move forward, but Argo is looking great right now and it's way ahead, if they manage to finish it soon and make it production ready it will be hard to beat it. The problem is that every segment has it's player now and there are some big ones, GiLab, GoCD, Spinnaker, Concourse... So many tools and the difference is that most of them have more focus than Jenkins does, they also have newer code and more speed, each has a niche but Jenkins has the market share, it will be an interesting match. Jenkins is fighting a bit of a uphill battle but with a huge army. I hope they keep it simple, focus in being the best in one or 2 things and then scale to other areas, that is my 2 cents.
- twic 8y agoFirst i've heard of Argo CI. Why would i want that rather than Concourse? Which can now be deployed on Kubernetes, it seems: https://github.com/helm/charts/tree/master/stable/concourse https://github.com/helm/charts/tree/master/stable/concourse
- jacques_chester 8y agoConcourse's components (ATC, DB, Workers) can run on Kubernetes, but it is still handling the scheduling of containers it creates. Delegating container scheduling to Kubernetes is the next major epic on the Core track for Concourse. As for Argo: I am not particularly in favour of Turing-complete YAML. Disclosure: I really like Concourse. I work for Pivotal, which sponsors Concourse development.
- cfontes 8y agoValid point. Don't fly you end up creating a turing-complete config file as well? I remember it being stored as some kind of Yaml like syntax.
- Aissen 8y agoI'm surprised they have such a good understanding of Jenkins' shortcomings. It's a good first step in fixing them. Although to be fair, this has been coming a long time as the post says; but having Cloudbees' CTO publicly acknowledging those is even better.
- oblio 8y agoHe's not only Cloudbees CTO, he's the original developer and architect. And before someone lambasts him for the Jenkins architecture, Jenkins/Hudson was created in 2005, when things were a lot different, and Jenkins managed to create an entire subgenre of software and lead it to the current day. Jenkins hasn't aged gracefully but how many software products from any category have even survived 5 or 10 years? :)
- mooreds 8y agoI can think of a number (Linux, Eclipse, vim, mysql, etc), but as a percentage of total software produced, it's very small.
- deleted 8y ago[deleted]
- kohsuke 8y agoThank you for the encouragement. Indeed I wanted to capture the shortcomings correctly, because I truly believe in the power of the community to solve them. Looking at this thread, I feel my summary is largely validated. We've been working to solve those, and we'll step up even more so. Exciting times!!
- mothsonasloth 8y agoI think this is too much too late for Jenkins. I can't speak for other countries but in London a lot of companies are now using Gitlab or Circle CI. I migrated all my builds (12 projects) to Gitlab CI. After figuring out the first CI pipeline using DockerInDocker, it was easy to then setup the remaining pipelines. Self hosting Gitlab was perfect for our needs (private docker registry). I use Gitlab for personal use too. I wonder if they will get rid of Ruby in the future though and go Java to make it more performant, as it does slow down sometimes. The Jenkins box is still running though, more out of sentimental value :)
- sytse 8y agoThanks for using GitLab! I'm glad to hear you found it easy to set up. I just wrote an article in response to the OP https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-with-the-three-variants-of-jenkins/ https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-w... We're working on making GitLab more performant. It is mostly fixes to our code, the parts where ruby is a problem are already rewritten in Go. GitLab self-hosted should be fast if it has enough memory, so make sure you check on its memory consumption.
- yebyen 8y ago> The current legacy version of Jenkins needs to be restarted once a day by an administrator Is this true? Do you have a source that says this? We have a Jenkins instance that Kubernetes is configured to scale down from 1 replica to 0 at night, and up to 1 again in the morning, so if it is true we never would have noticed. (It hasn't always run on this cron cycle, which is why I'm a little incredulous at this claim, but if it's given in the OP or somewhere else easy to find this, I'll concede it... ah... found it: > It’s not unheard of that somebody restarts Jenkins every day.) Honestly I don't understand this about "making a version of Jenkins that runs well on Kubernetes" – this is the _only way_ I have ever run Jenkins, and I think it runs already extraordinarily well for our purposes. I'm thrilled that they are making it their focus, and I'll concede also that our use of it is pretty narrow, but I haven't had these issues. We installed it from the stable helm chart nearly 2 years ago and have hardly needed to make any tweaks. We are not tracking every K8S release, so maybe that's why I haven't noticed Jenkins falling behind, and we also haven't tried GitLab seriously (heard great things, but my work is very risk-averse when it comes to new technologies, and to be honest we rarely try new things on a short cycle once a given problem has been solved adequately for us... we are also not primarily a development shop, so maybe it makes sense.) > The article doesn't mention how Cloud Native Jenkins addresses the problem, maybe it doesn't allow plugins. Like I've been saying, we've always used the stable helm chart for Jenkins and started maintaining our own values.yaml about a year and a half ago. Over time we have had less need to change the templates as more configuration got moved into values.yaml. When I have needed a plugin or other configuration that is able to be set in values.yaml, that's easy and almost makes maintaining Jenkins fun. It is a little obtuse that I have to maintain my list of plugins and their latest versions there manually, but this could be something that gets resolved in Cloud-native Jenkins if they are ultimately providing an operator or something like that. (Breaking a rule by commenting before I've read all of the content, but I liked your article and wanted to give you some feedback since you posted it.) For configuration that doesn't live in values.yaml, Jenkins chart maintains a Persistent Volume where configuration and build artifacts/history are stored. It is easy enough to take backups of that with the ThinBackup plugin, and the storage costs of that are sure not breaking the bank. > Services interacting through Kubernetes CRDs in order to promote better reuse and composability And there it is! That's the big announcement from today. Knative is still early but this news from Jenkins sounds supportive and I should really read the whole article / watch the video now.
- nrclark 8y agoI love Jenkins, and use it professionally. With that said, I'd really like to see better documentation on the Jenkinsfile Pipeline format. I've tried to get started with it a few times, and haven't had tons of success. Stuff like "How do I pull in secrets", and "How do I control a plugin". I appreciate that it's Groovy-based, but that's not particularly helpful information (for a hack like me, at least). The snippet-generator is nice, but it doesn't necessarily produce working code. Especially for things like getting secrets into a build. And it doesn't give me a broader picture for "How do I even write one of these from an empty text-box". I recently tried the job-to-Pipeline exporter plugin, and that didn't work on my jobs - it generated stuff that didn't match the input job, and also wasn't structured like the example snippets Jenkins provides natively. Maybe some kind of a sandbox I could experiment in? Or a REPL or something? It would really help to have something that gave great discoverability, with fast feedback. Faster than I can get by editing a job, saving it, running it, waiting, then realizing I still don't have the syntax right.
- 0x8BADF00D 8y agoUnfortunately, it’s not very easy to do. If you want syntax validation, you can validate it by POSTing to an endpoint on your Jenkins master. If you want a “sandbox”, you can just replay a pipeline run and make modifications. Both are not very useful IMO and slow down development pace substantially. Don’t get me started on integrating groovy shared libs.
- andbberger 8y agoSame here - I really struggled to set up declarative pipelines starting out. The docs don't do a great job of distinguishing between the full groovy syntax and the new declarative syntax and there is a relative dearth of examples. I think the swiss army knife nature of Jenkins contributes to this - there's just so much you can do.
- jstrachan 8y agoI think as part of the new Jenkins architecture we should be able to make it much easier to stop at a point in a pipeline & open a terminal/REPL to test out steps. Also I'm hoping for a nice validated YAML based pipeline syntax that should make editing/validating pipelines easier
- curtis 8y agoThe biggest deficiency I found in Jenkins is that GUI-based job configuration is great for simple setups and one-off jobs, but the moment you throw in any sort of parameterization it becomes a real headache. At that point you really need to be able to configure your jobs in code.
- dharmab 8y agoAnd the Jenkinsfile documentation is relatively bare and reliant on examples.
- jstrachan 8y agoone solution to the problem, if you are building apps for kubernetes is to use Jenkins X which automates all your CI/CD pipelines: https://jenkins-x.io/ https://jenkins-x.io/
- isbvhodnvemrwvn 8y agojenkins-dsl does wonders in this department. When infrastructure-as-code plugin gets stable we'll have a fully immutable Jenkins setup.
- kohsuke 8y agoExactly our thought! The whole Jenkins Pipeline is around the notion that your job definition should be version controlled (and you don't necessarily have to lose GUI, see our "blue ocean pipeline editor" that now comes out of the box. Then the newest kid on the block is https://github.com/jenkinsci/configuration-as-code-plugin https://github.com/jenkinsci/configuration-as-code-plugin, which I referred to in my doc.
- honkycat 8y agoThe worst part of my job is configuring our Jenkins server and managing builds in their dumbass groovy based DSL. I'm willing to bet that most people just want to build GitHub repos. Then why do we have to do this mess to get a decently repeatable deployment strategy: https://coderanger.net/jenkins/ https://coderanger.net/jenkins/ I should not have to crack open plugin source code in order to configure the plugin programmatically. It's dumb and bad. Also groovy is a bag language. Managing Jenkins pipeline library deps is a pain. Also yeah, plugins break constantly and upgrading them is always a nightmare.
- jstrachan 8y agoif you are happy to use Kubernetes then if you switch to Jenkins X you never need to configure a Jenkins server or create a groovy based DSL again: https://jenkins-x.io/ https://jenkins-x.io/
- cfontes 8y agoInstall GoCD and become a happy person again.
- zmmmmm 8y ago> Also groovy is a bag language. This seems to be a repeated pattern that is really giving Groovy a bad reputation: it keeps getting embedded as an extension point / scripting solution inside other products. It is sold as "it's almost the same as Java, so we don't need any documentation for it" - and the result is that people with little to no Groovy knowledge end up trying to use it and get incredibly frustrated with it. I'm curious if your conclusion above is based only on encountering it inside other things (Gradle,Jenkins, etc) or if it's actually from analysing its characteristics as a language more generically? (FWIW, Groovy is probably my favorite language, but I use it as a full stack language for application development, quite a different mode to how most other people encounter it).
- bdcravens 8y agoThe focus on cloud-first Jenkins is interesting considering Codebees's acquisition of Codeship earlier this year. Obviously Kohsuke would be biased to Jenkins, but as CTO, I'd imagine the corporate goals take precedence.
- jstrachan 8y agoeven from an OSS goals perspective I'm looking forward to seeing better alignment, reuse and interoperability between Jenkins, Jenkins X & things like CodeShip & Knative Build
- tannhaeuser 8y agoI'm wishing Jenkins all the best. I know it since the Hudson times as the de-facto CI system for Java (and Cruise Control before that as my first encounter with CI). OT: does anybody know a CI system based on plain Makefiles, convention-over-configured for autotools-like default targets, and supporting file-suffix based build and test rules for C + JS + custom compilers and such?
- jstrachan 8y agoinvoke `make` from inside your `Jenkinfile`? :)
- slavik81 8y agoThis is nice to see. I wish them luck on their project, because it is not going to be easy.
- jonthepirate 8y agoI worked on Jenkins at Lyft and completely set it up for DoorDash. If anybody needs help with their Jenkins setup, hit me up I give free advice and have a few blog posts on the matter. If you happen to be using AWS, GitHub, and Slack, we at DoorDash have developed lots of goodies for streamlining things. We have secured our Jenkins behind our VPN, created load balanced Jenkins clusters, built a shared Groovy library for all of the Jenkins behaviors that are useful for each of our microservices, implemented a Flask app that receives each of the GitHub webhooks which starts pipelines instantly (rather than git polling), setup Okta integration, interfaced with our internal secrets store, and implemented a way to map GitHub users to Slack users allowing us to Slack message people when they are mentioned in GitHub (when their PR's receive LGTM's etc.) When new microservices launch, Folders automatically appear in Jenkins configured correctly for the service's pipelines. If any of this sounds good let me know, maybe we open source some of our work. I love working on Jenkins and am happy to help advise you on how to scale, secure, demystify your own Jenkins setup. Links on my HN profile page.
- michaelneale 8y agoGlad to hear that - you should put your config up on a blog post somewhere sometime.
- tbugrara 8y agoOh god, please!
- kohsuke 8y agoYou should be presenting in a future Jenkins World event!
- rosshemsley 8y agoI was seduced by BlueOcean and tried using Jenkins for CI, using Github and AWS ECS builders (which felt like a common enough use-case). Unfortunately it ended up costing an astonishing amount of engineering time to get working and maintain, with builds frequently stalled or failing. Since moving to CircleCI 2.0 enterprise (admittedly far from perfect) and Airflow, we have _dramatically_ reduced eng. time spent managing our job scheduling. The core of our problem was how fragile and complex the Jenkins ecosystem seems to be: any change to the config or settings and it would easily burn a day of engineering, due to random bugs and hard to understand error messages. In the end, no one wanted to touch it! I think there's a great project hidden somewhere here, but just getting the basic "everyday" stuff done with it can be a real PITA.
- kohsuke 8y agoI'm sorry to hear the bad experience. I recognize those challenges in my pitch, we have various efforts already under way to address them, and with this gear shifting, I think we'll be combining those in a compelling way. For example, defining Jenkins config in YAML in Git is a key piece to solve a fear of config change, and this is called "Jenkins Configuration as Code" and is under way for a while now. Cloud Native Jenkins will also split single process "master" into many build-as-a-function kind of processes, so it isolates builds and allows changes to be rolled out more incrementally. There's more focus on us owning a bigger responsibilities around "basic every day stuff," too.
- CamouflagedKiwi 8y agoI'm impressed about how honest the author is about the shortcomings of Jenkins as it is now. Very appropriate that he mentions being in a local optimum - that is where most organisations end up with Jenkins. The server nearly immediately becomes a snowflake, most stuff is configured through the GUI rather than code, probably some people know it's not ideal but getting to something better requires changing everything and people know how it works now. Having said that, I think the conclusion is wrong. The next-generation CI already exists (CircleCI, Gitlab, etc), attempting to evolve Jenkins into that seems like a punishing task given the huge legacy and relatively little strategic advantage. Don't want to take anything away from them blazing the trail, but in the same way RCS and CVS did that and eventually bowed out of the game. Jenkins should gracefully do the same.
- kohsuke 8y agoThanks for your thought. I took your main question to be "why bother?" I think a part of it is that I fundamentally believe in an extensible system. The world of software development is so diverse, and we have smart people everywhere. So I always felt that the best thing a geek like me can do to other geeks is to give them a shoulder to build on. I don't think that's a solved problem, and to me, that'll always be an important value of the Jenkins project, more so than any code. I think a part of it is the responsibility to users. Jenkins is very widely used software, and it's an incredibly important part of the software development process for many. I appreciate that kind of trust, and I want to deliver better software for them. I think people in the community shares the same passion. As CTO of CloudBees, serving our users and customers, and broadening the adoption base are an obviously important business goal. So the interests are aligned there as well. And finally, I think this kind of "reinvention of the brand" happens all the time. Windows got reinvented from 95 to NT, Firefox got reinvented a few times. There are many other examples less famous but closer to my part of the universe, like Maven 2, GlassFish 3, ...
- ak217 8y agoHaving used Hudson/Jenkins for many years, I recently considered setting it up for a new project, and backed away mostly due to the issues Kohsuke describes. We ended up choosing GitLab instead. GitLab has been pulling ahead in features and usability, compared to other things I've tried. Right now, different projects I'm involved with use a combination of GitLab Enterprise, Travis, Circle, and Google Cloud Build. Of those, GitLab accommodates the heaviest and most sophisticated workloads, without having to go through too much trouble to set up, maintain, and instruct developers how to use it (certainly less trouble than Jenkins). I highly recommend taking a critical look at all of these services, to see which best fits your needs.
- Sir_Cmpwn 8y agoIf you don't mind, I'll shill my service as another option: builds.sr.ht. It's still in closed alpha, but it's being used seriously by several open-source projects for complex build automation. It also deploys itself, here's the build manifest which does it: https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml And an example of a build which used it: https://builds.sr.ht/~sircmpwn/job/6974 https://builds.sr.ht/~sircmpwn/job/6974 If you or anyone else would like to try it, please let me know. I used Jenkins for a long time (and still do at work), Travis for a while as well, also tried Drone and Circle, but none of them were exactly right. I think builds.sr.ht does it very well.
- baylisscg 8y agoAt ${DayJob} Jenkins is our default of yore. Returning to refresh a 1.x install for one group's product we're faced with the poster child for a Jenkins install gone bad. Looking at you Chuck Norris plugin. We can't upgrade and we can't migrate to a fresh install due to how Jenkins handles plugins. So we're left with a critical chunk of infrastructure that's a time bomb. Ultimately instead of making the jump to 2.x and Jenkinsfiles we're trialing Buildkite with great success so far and the confidence that we can jump ship to CloudBuild, TravisCI, Concourse, CircleCI, ect should we need to.
- misterbowfinger 8y agoHate to be that person, but what's "Cloud Native" even mean? Is there a glossary for buzzwords somewhere?