17 ms·
Jenkins Is Getting Old
- stabbles 8y agoJenkins's plugin manager is absolutely terrible. If you're stuck on an older version of Jenkins, you better not click the "refresh" button in the plugin management page, cause otherwise the page is just filled with red warnings saying that the latest version of each plugin is incompatible or has dependencies that are incompatible with your current Jenkins version. There is afaik no way to install the last plugin that was compatible with your version of Jenkins.
- buzer 8y ago> There is afaik no way to install the last plugin that was compatible with your version of Jenkins. Probably not directly, but if you know the version, you can download the HPI and install it manually. Jenkin's Docker build also contains install-plugins.sh (https://github.com/jenkinsci/docker/blob/7b4153f20a61d9c579b5f653a4a99fb543f26c41/install-plugins.sh https://github.com/jenkinsci/docker/blob/7b4153f20a61d9c579b...) that you can use to install specific version of plugin via command line.
- stabbles 8y agoThanks! That script definitely gives some pointers on how to update things.
- jennbriden 8y agoCheck this out, too. Free forever: https://www.cloudbees.com/products/cloudbees-jenkins-distribution https://www.cloudbees.com/products/cloudbees-jenkins-distrib.... We have included Beekeeper, which is an implementation of the plugin manager that provides a list of known-compatible, recommended plugins that CloudBees verifies and tests with each long-term support release of Jenkins.
- Cedricgc 8y agohttps://outline.com/VvVhg6 https://outline.com/VvVhg6
- rglullis 8y agoYours is not the first post that contains nothing more than an outline version of the link. Why is this becoming a thing? The page seems perfectly readable as it is...
- notacoward 8y agoIt's readable as long as you're willing to log in via Google or Facebook. Some people aren't.
- rglullis 8y agoI did neither and it opened fine even when I opened the link on incognito mode. Maybe some A/B testing thing? Anyway, good to know there is a reason for it. One of my pet projects is a self-hosted open source paywall-buster and read-it-later service. I wonder if the HN admins would be interested in integrating with the site?
- chrisweekly 8y agoI'm not an HN admin, but I'd love to learn more about your pet project.
- rglullis 8y agoIt is still very basic, but basically it is an alternative to pocket and wallabag, implemented in Python/Django/Celery - I actually wanted to use this as a way to learn Go, but working on the idea itself plus the potential work (activity pub, IPFS integration) became more interesting than working on learning a new language, so I changed to something I more familiar with. Anyway, the code is at https://github.com/lullis/nofollow https://github.com/lullis/nofollow. It would be great to have more people curious enough to actually run this.
- rjeli 8y agoI’d never used Jenkins before, a few weeks ago I needed to do some CI that was more complicated than just pytest or whatever. Installed jenkins, put together a dockerfile, connected the github hooks and It Just Works. I’m a happy customer
- bluGill 8y agoThe problem is people want to do really complex things with Jenkins. A few hundred repos, each with dozens of builds; Multiple operating systems... There is a reason a team of 10 works full time to administer our CI system: it is complex (the CI system is more than Jenkins)
- js2 8y agoAnother container based option is Screwdriver, which started as a Yahoo in-house solution built on top of Jenkins. But it is now fully open source and no longer makes use of Jenkins in any way: http://screwdriver.cd/ http://screwdriver.cd/
- dpedu 8y agoThis looks really good! For a long time Jenkins has seemed like the only option for flexible jobs. Tools like Drone and Travis and the many other that use a "config file" approach just aren't able to scale against your needs. They're fine for a run-of-the-mill build but that generally just isn't enough.
- bradrydzewski 8y agoDrone is working on support for Starlark, a python-inspired configuration language used by the Bazel Build system [1]. For complex pipelines, this should be more analogous to Jenkins scripting and may improve scaling Drone for larger projects (time will tell). [1] https://github.com/bazelbuild/starlark https://github.com/bazelbuild/starlark
- 0xdeadbeefbabe 8y agoIs buildbot old too? I sure enjoyed the flexibility. https://buildbot.net/ https://buildbot.net/
- herge 8y agoOne complaint about buildbot is that if you get too creative, your buildmaster.cfg gets very hard to maintain, but if you stay very diligent, just having python (and being able to print or log whatevery is happening) makes debugging and having complex setups be very easy. Also, if you ever need to schedule jobs/tasks (not just ci builds) across multiple machines, buildbot is great because all you need is a master, and slave python processes which just need a network connection to the master.
- durnygbur 8y agoWe need SPA CI in Angular, NgRx, and with Material Design:) /s
- deleted 8y ago[deleted]
- bovermyer 8y agoThe most interesting part of this is the very last paragraph. There really isn't a good self-hosted solution for build metadata/metrics, unless you write something yourself. It would be interesting to see a GUI dashboard for Drone/whatever builds and pipelines.
- bradrydzewski 8y agoThere was an interesting community-driven project to create a dashboard for Drone, see https://github.com/drone/drone-wall https://github.com/drone/drone-wall. I would love to see more like it.
- qznc 8y agoI never understood the purpose of dashboards. In the context of CI, I see three use cases and there is always a better solution than a dashboard. 1. Detect broken builds. It is better to actively notify somebody by mail, chat, whatever. 2. Prevent broken master. It is better to reject the pull request/patch before it becomes a problem. 3. Analyse the system. It is better to download the data and enable the use of whatever analysis tools are suitable.
- jayd16 8y agoDirect emails can't replace the collaborative "hey why are these still red" moments without spamming the whole team. They can show non-event information such as build time trends at a glance.
- hardwaresofton 8y agoDisclaimer: I'm pretty biased towards Gitlab -- write about the things you can do it from time to time and they gave me some free swag once. Best CI I've ever used is Gitlab CI[0]. The runner is completely open source[1] and you can use your own runner with your gitlab.com (or local instance) projects -- set it up in an autoscaling group[2] for savings. I run https://runnerrental.club https://runnerrental.club but Gitlab also recently released the ability to pay for minutes in 11.8 [3], so my product is more-or-less dead in the water but I don't mind since Gitlab is such an excellent tool, I'm glad to see them fill the need. But back to Gitlab CI -- the YAML configuration documentation[4] is pretty fantastic -- Most easy things are easy and hard things are possible. I suspect that one could run an entire startup like circleci/travis/drone based on just the software that Gitlab has open sourced and made available already. [0]: https://docs.gitlab.com/ee/ci/ https://docs.gitlab.com/ee/ci/ [1]: https://gitlab.com/gitlab-org/gitlab-runner https://gitlab.com/gitlab-org/gitlab-runner [2]: https://docs.gitlab.com/runner/configuration/runner_autoscale_aws https://docs.gitlab.com/runner/configuration/runner_autoscal... [3]: https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#purchase-add-on-ci-runner-minutes https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#p... [4]: https://docs.gitlab.com/ee/ci/yaml/ https://docs.gitlab.com/ee/ci/yaml/
- StavrosK 8y agoI will second this, Gitlab has the best CI I've ever used, and I don't know what it is. The UI is just so clean, it does what I need to and is easy to configure. I put all my projects on Gitlab mostly because of the CI, but also because of the other great features.
- jl-gitlab 8y agoGitLab product director for CI/CD here - thanks so much for the feedback, everyone. It's really great to read how much you're getting value out of what we built. We have an overall CI/CD direction page up at https://about.gitlab.com/direction/cicd https://about.gitlab.com/direction/cicd which you can drill down into the individual stages plans from. Feedback is always welcome, we love building things in partnership with real users. You can reach me at jason@gitlab.com any time.
- littleweep 8y agoI think a better headline would have been something more like "Jenkins doesn't have the features I require". There are plenty of examples of older pieces of software or older languages that still work perfectly fine today.
- kissgyorgy 8y agoYou should really read the article before you comment on it.
- asgeirn 8y agoOur setup runs Jenkins master and slaves as Kubernetes pods, with plugins limited to only the very few required to get GitHub integration and slaves working. Jobs are configured by adding an entire GitHub organization. All repositories with corresponding branches, pull requests and tags are automatically discovered and built based on the existence of a Jenkinsfile. Everything is built by slaves using Docker, either with Dockerfile or using builder images. Job history and artifacts are purged after a few weeks, since everything of importance is deployed to Bintray or Docker repositories. By keeping Jenkins constrained in this fashion, we have no performance issues.
- asey 8y agoThat is exactly how we're doing it as well, though I am interested in checking out Cloudbees'Jenkins. We've recently incorporated Zalenium (selenium grid which autoscales nicely natively in kubernetes) - just had to work a little magic with automatic service creation during builds.
- bryanlarsen 8y agoJenkins-X is really cool. I love how you can group services together into an environment and deploy that to a single Kubernetes namespace, and how well it uses github releases. I really hope Gitlab's Auto-Devops team is looking at it closely and stealing all the great ideas it has because that's what our team is using.
- kkapelon 8y agoJenkins X requires Kubernetes and Helm. Not all companies are deploying like this. (I work for Codefresh, a competitor of Jenkins X)
- deleted 8y ago[deleted]
- vladletter 8y agoThey should definitely dig more into Jenkins X: it evolves a lot and it's pretty new but they're gonna have a stable version soon!
- FrenchyJiby 8y agoI sincerely wish I could move away from Jenkins for the reasons stated in TFA (GUI-oriented, slow, hard to backup/config, test-in-production mentality and boundless plugins) but I've never found something that fits the bill. The much-touted repo integrations (travis, circle...) all have an exclusive focus on build-test-deploy CI of single repos. But when you have many similar repos (modules) with similar build steps you want to manage, and want to have a couple of pipelines around those, and manage the odd Windows build target, these just give up (it's docker or bust). Sadly, only Jenkins is generic enough, much as it pains me to admit. Anyone got a sane alternative to jenkins for us poor souls?
- hpaavola 8y agoI don't think Jenkins is gui oriented, slow or hard to backup/config, but I did enjoy using TeamCity few yers back. Sure it costs you arm and a leg, but it worked well without any plugins.
- allover 8y ago> But when you have many similar repos (modules) with similar build steps you want to manage How many teams do you have? In all seriousness, if you aren't talking at least one team per repo, have you considered a monorepo setup? Aren't you burning time managing those many similar repos with many similar build steps? That said, even in a monorepo, I still prefer Jenkins compared to cleaner seeming cloud offerings due to its limitless flexibility.
- FrenchyJiby 8y agoInternal libraries and similar fun stuff. Common build step ~~ same packager commands run on them. Management is fairly simple with a template + seed jobs. It's just ... everything else is annoying. I don't understand what you mean by one team per repo? I agree, as I keep saying at $WORK: Jenkins is the least-worst system out there.
- phowat 8y agoI'm quite happy with Buildbot.
- khanan 8y agoUhm, this sounds like somebody who is unable to has used Jenkins for a LONG TIME, but gotten lazy in "keeping up with the jonses" and realised he is out of date and takes ACTUAL WORK, even after you are 35. Get studying.
- tinix 8y agoI receive vulnerability notifications for Jenkins, pretty much regularly... mostly XSS and RCE. https://www.cvedetails.com/vulnerability-list/vendor_id-15865/product_id-34004/Jenkins-Jenkins.html https://www.cvedetails.com/vulnerability-list/vendor_id-1586... I'm just waiting for Apache to adopt it, and then it'll sit and fester like everything else in the Apache graveyard, full of vulnerabilities and slowly decaying. Those are just Jenkins core exploits too... there are so many many more for Jenkins plugins.... https://www.cvedetails.com/vulnerability-list/vendor_id-15865/Jenkins.html https://www.cvedetails.com/vulnerability-list/vendor_id-1586...
- mfer 8y agoJenkins is now part of the CD Foundation (https://cd.foundation/ https://cd.foundation/) which is one of the linux foundation sub-foundations. Don't expect it to show up in the apache foundation.
- 1ris 8y agoI don't think tinix was excpeting it to literally become a apache project - he was just saying its in a state of decay that apache is infamous for.
- VectorLock 8y agoLast place I was at had their unmanaged Jenkins servers get compromised and used to run crypto miners.
- jennbriden 8y agoWere they using an older version of Jenkins on the public internet? There's been a randomized GUID applied to the initial Jenkins admin password, which you can only access if you have direct access to the Jenkins install. I think this was added in 2016.
- VectorLock 8y agoIt was an older version with a vulnerability but as far as I know not a default password.
- outworlder 8y agoI agree with this article on all points. Currently we have a massive jenkins pipeline sprawl that's difficult to maintain. It is also difficult to create new jobs in Jenkins itself, specially if you are using pipelines. My average is around 100 test builds before I can get a full pipeline success for anything of modest complexity. If all you are doing is using jenkins to run simple bash scripts, you may be able to get away with it. The problems start when you want to add some logic to the pipeline – which you are doing, otherwise why bother with a pipeline? First things first: are you going to use the scripting pipeline, or the declarative pipeline? The declarative pipeline is a bit better, but it lacks examples, has lots of bugs (I've littered my code with references to JENKINS-XXXXX) and is very restrictive (arguably by design). Of course, you can have 'script' blocks inside your pipeline->stages->stage->steps blocks. Then you want to take advantage of parallelization or conditional steps, and to visualize that you want Blue Ocean. Problem is, not all plugins are compatible with Blue Ocean, it also doesn't have all features, so you drop down to 'old' jenkins often. People will want to have a whole bunch of tools with incompatible versions in their builders. Not all are supported natively, so you need to figure out your versioning. Once you figure all that, congratulations. Next guy to automate something will either find a similar pipeline to copy from, or will endure all the pain again. At this point you may want to use Groovy. Groovy was absolutely the wrong tool for the job. Yes, I get it that it works with Java, which Jenkins is based on. Still it is the wrong choice. You see, the kind of things you want to automate often involve passing commands around, be them bash, ansible, SQL statements, what have you. Groovy's string escaping rules will ensure your life will be pretty miserable (https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e412...) You could get around most of these by perhaps moving most of the logic to containers and then running those. There again you'll run into problems with declarative pipelines, random things won't work and you'll be scratching your head. However, if you are going to do that anyway, you're better off using a more modern system for CI, any system. Drone was already mentioned, there's also Concourse and a bunch of others. For CD, you can use Spinnaker as well. Or maybe keep jenkins around but forget all the fancy stuff. Delegate all the 'thinking' to scripts and pretend the more recent development has never happened. You'll be saner that way.
- humbleMouse 8y ago
- deleted 8y ago[deleted]
- willcodeforfoo 8y agoThere are lots of alternatives for Jenkins as a CI builder but I haven't found many for Jenkins as a web-based cron with a nice UI that keeps track of history, has tons of options for notifications on failures, and quickly allows re-building (even with user-supplied parameters!) We use Jenkins for running backup jobs, periodically updating data, and building quick little jobs for support staff to run--infrequently enough that they don't warrant adding to our admin app but frequently enough that bothering at developer adds up.
- cpitman 8y agoFor periodic admin tasks, you might consider giving Ansible AWX a try (https://github.com/ansible/awx https://github.com/ansible/awx). It is a web interface for managing Ansible playbooks, and lets you configure jobs that another user can then execute and provide parameters. It keeps track of all run results, has a fair number of supported integrations for notifications, and can schedule runs. I work for Red Hat, and AWX is the upstream community project for Ansible Tower, which we provide support for. AWX is one of our newer open source projects (we open sourced it after acquiring Ansible), so you'll sometimes have better luck searching for information on "Ansible Tower".
- chrisswanda 8y agoJenkins is to CI, as Nagios is to monitoring.
- geggam 8y agoBoth work ?
- user5994461 8y agoThey both work perfectly fine for the very limited use cases they were meant to, at the time they were made, with the limited resources that were available. Just like foot is the best way to travel and commute to work every day, as proven by the last 10000 years of people using theirs.
- mschuster91 8y agoNo, the key difference is that Nagios has perfectly viable alternatives (Zabbix admin here and it's not the only one). Jenkins however has no alternatives. Bamboo or whatever it is called now is typical Atlassian crapware. Expensive as fuck, eats more resources than the stuff it builds, and did I mention it is yet another half assed product that got shoddily integrated into the usual Atlassian lineup? Gitlab CI is great for anything that is code (think build, test, deploy), but it is not suited for abstracting "non-development" jobs which can perfectly be automated in Jenkins (e.g. creation of a dev environment with fresh data from production). Plus it is Docker and the runners are polling - which means at minimum 10s startup time compared to milliseconds for a Jenkins shellscript job running on a ssh connected slave! Github and friends are cloud which is a big no-no. We're placing too much power in the hands of AWS, GCE and Azure already, no way in hell it is a good idea to put private source code to a cloud provider.
- ethagnawl 8y ago"cripple-wear" made me cringe. Shouldn't it be "-ware", anyways?
- sanitz 8y agoI recommend https://concourse-ci.org/ https://concourse-ci.org/
- agentx3r 8y agoI really love the ideas/architecture behind Concourse, but there's a few things that disqualified it in favour of Jenkins for a new CI pipeline during prototyping: - no hooks to manage worker scaling (https://github.com/concourse/concourse/issues/993 https://github.com/concourse/concourse/issues/993). Our builds are _heavy_ and we'd run up an enormous AWS bill without something like (https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin) - no way to restart a build, resorting to 'empty commits', which is a huge red flag for useability (https://github.com/concourse/concourse/issues/413 https://github.com/concourse/concourse/issues/413) - limited documentation/examples (network effect)
- tedk-42 8y agoWow I never knew Jenkins could scale its workers like that. We definitely have performance issues with our Concourse cluster running in K8s. It's got it's own set of dedicated nodes but we need to scale the workers better as they're ofter under heavy load throughout the day when the devs are pushing code and a bunch of tests are running in Concourse (and PR checks as well)! Concourse definitely feels more refined than Jenkins. Like other commenters have said, it's a steep learning curve to grasp how things move between Tasks/Jobs.
- GeneticGenesis 8y agoUnrelated to the content, but the stock photos throughout this article felt completely needless to me. We're not 5 year olds, we don't need pictures unless they add to the content.
- linuxftw 8y agoTrying to integrate Jenkins w/ Github Enterprise proved to be a real pain for me. It wasn't clear what git* plugin is the best to use for how I wanted to setup builds. It's literally impossible to deploy Jenkins in a repeatable manner with just configuration files, you must get it online and use the api to configure such simple things as API keys (eg, specify the ID of the key so your jobs actually reference the key and don't break). Another big pain point, for me was, no obvious openstack integration. I wanted my jobs to run on ephemeral instances so all my dependencies could be defined in the job themselves, not having to rely that the Jenkins worker is setup just right. I think Gitlab runners get this right, but I have not investigated too far.
- mathewpeterson 8y agoYou definitely can configure Jenkins in a repeatable manor. See https://wiki.jenkins.io/display/JENKINS/Groovy+Hook+Script https://wiki.jenkins.io/display/JENKINS/Groovy+Hook+Script You can write all the configuration in groovy and it will execute when the Jenkins process starts up. There is also "Jenkins Configuration as Code" plugin which allows you to have a single yaml file to configure most of the Jenkins system. https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/README.md https://github.com/jenkinsci/configuration-as-code-plugin/bl...
- linuxftw 8y agoI managed to accomplish the task at hand ~2.5 years ago, so yes, it's possible to deploy jenkins in a repeatable manner. I guess what I'm driving at is that it's not very declarative and it's cumbersome.
- dvtrn 8y agoI'm not at all ashamed, nor a single bit remorseful to comment on the fact that it took a catastrophic data loss for a team I once worked with to finally sit down and look at our CI/CD pipeline before deciding "Maybe jenkins is overkill for what we need". Which was something I had been kvetching about for months and expressly warned, multiple times to our release manager as a point of concern given how quickly plugin vulnerabilities are reported (as someone comment on elsewhere in the thread). The day finally came when someone from one of our other offices went to update some infrastructure as code repos, poof. Jenkins server gone. They didn't have a roll back plan, and to complete the trifecta, they somehow also killed all of the instance volume backups. An entire sprint was summarily dedicated to creating a new build pipeline, I had resumes out the door the next day. This article hits so many of our pain points I joked to a current coworker who followed me out of that place that I wanted to print it and mail it to our former RM.
- Iwan-Zotow 8y agoWHat do they/you use now?
- dvtrn 8y agoNo clue. Didn't stick around long enough to find out, that moment was officially the last straw for me. There was a lot of hemming and hawing, I left and product, project and dev still hadn't made their minds up. Ops (where I lived) would give the team ideas and proposals, dev gatekept everything from tickets to the VCS. This was one of those shops that hired people and gave them devops job titles, but demanded they maintain very monolithic status quos with everything from tickets and stand-ups to release management.
- Pokepokalypse 8y agohm. same.
- robax 8y agoI'm still pretty new to devops and I personally love Jenkins but man is it a PITA at times. My biggest gripe is that every plug-in seems to be no longer actively maintained. We really on a huge list of plug-ins and in some cases upgrading Jenkins will break plug-ins. It's gotten to the point where I'm afraid to update to even minor or patch versions.
- KaiserPro 8y agoI have been with jenkins since about 2010. At its peak I had to shepherd ~50 masters of varying ages, plugins and levels of customisations. Yes, it allowed repeatable builds, and it can be, in certain circumstances a very good replacement for cron jobs where you need to administer >5000 machines. However, it has an offensive UI, the plugin system appears to be designed to deliberately cause things to break, versioning the flows is difficult, unless you rely on plugins (which break) or want to learn groovy. Automating and making jenkins it's self repeatable is a soul sapping task. In short, jenkins needs to disappear. Yes, cirle is crap, bamboo suck massive genitals. However I have had great success with gitlab runners. Now that gitlab is actually reliable, they are a brilliant first class integration into a git workflow.
- michaelneale 8y agoI think this "lesson" is not lost on many, including those that built and work with Jenkins and plugins. The Jenkins X project moved to a very different approach, a fresh start. see https://cd.foundation/ https://cd.foundation/ & https://jenkins-x.io https://jenkins-x.io (it works with Jenkins if you like, but natively uses the new pipelines and no SPOF masters). Might not be for everyone but worth a look. disclosure: cofounder of cloudbees here (wanted to comment on original article but not able to for some reason on medium).
- chucky_z 8y agoWhy the dependence on Kubernetes everywhere in these new tools?
- michaelneale 8y agoBut to your point (sorry I missed it first) - assuming kubernetes is like assuming an OS, you get to do things like preview apps and other benefits. But if that isn’t for you https://www.cloudbees.com/products/cloudbees-jenkins-distribution https://www.cloudbees.com/products/cloudbees-jenkins-distrib... aims to simplify the Jenkins config.
- michaelneale 8y ago
- cjbprime 8y agoIt's true that Jenkins is old, and has systemic architecture problems which mean that I'm sure we haven't seen the last RCE for it. RCEs in build systems are a nightmare because build systems necessarily end up knowing secret tokens. But drone.io is no kind of replacement. It only works with Docker containers! Even small Jenkins installs can quickly end up with Windows targets, mobile builds, etc.
- bradrydzewski 8y ago> It only works with Docker containers This will only be true for another week or two. We have devised a framework for alternate runtimes, including running pipelines directly on the host machine or in traditional virtual machines, that will be released end of week. Reference issue https://github.com/drone/drone/issues/2680 https://github.com/drone/drone/issues/2680
- rmoriz 8y agoDoes it support running on macOS and Windows, too? CI for native apps is IMHO the most common lock-in for Jenkins at the moment. Anything web or container related can almost always be implemented using pretty much any other CI/CD solution out there.
- bradrydzewski 8y agoYes, it supports windows, macos, and linux. I expect it will also support bsd although I have not tested this yet. This should land in master some time this week.
- sudofail 8y agoWhat I'd like to see is a CI/CD engine backend. Most CI/CD solutions have relatively the same feature set: - Code builds - Code deploys - Git Hooks - Parallel builds - Matrix builds - Build as configuration - Etc If there were an engine that supported these features, the community could create a number of frontends that target their specific needs. Jenkins was held back, in my opinion, for a number of years by its antiquated UI. And while Jenkins does have an API of sorts, writing a new UI for it is not a simple task. I've used Drone for a while, and it gets a lot of these things right, but at the end of the day it is a holistic, integrated solution. I'd like to see a composable, modular CI/CD framework.
- eddieh 8y agoSounds like you want https://buildbot.net https://buildbot.net
- bogomipz 8y agoWhat is a "matrix build" in the context of CI/CD?
- sudofail 8y agoWhen your software needs to build on, or run on multiple targets, it's useful to define it in a matrix. Here's an example: https://0-8-0.docs.drone.io/matrix-builds/ https://0-8-0.docs.drone.io/matrix-builds/
- falsedan 8y agoI'd argue that the components are: - job config - agents & resources - logs/results - triggers/schedules - web UI - system config/'plugins' - user directory with optional permission/ACL and default ancillaries for secrets, artifacts, alerts, etc., with the idea that they could all be replaced with a purpose-built product that's actually good at it (i.e. Vault, Artifactory, PagerDuty/Sensu/whatever) I think deployment is a separate problem space to CI and shouldn't be included
- epage 8y agoJenkins is terrible but I've not found something better yet. For me, the great thing is that it is an automation platform where we can integrate builds, scheduled tasks (cron), and very basic end-user UIs with Plugins that let me customize the whole experience. Azure Pipelines comes close in capabilities. If they made it easier to manually start a job and for them to accept their parameters from the user, then it'd probably have everything i need.
- solatic 8y agoOP really needs to try Concourse. Same container-based workflow as Drone that is touted as a solution, but more mature, and much more testable than Drone. Concourse really hits his requirements for ops-friendliness and testability. It's easy to upgrade because the web host and CI workers are completely stateless, and the work you do with Concourse is easy to test because the jobs themselves are all completely self-contained. Because Concourse forces you to factor your builds into inputs, tasks, and outputs, it becomes somewhat straightforward to test your outputs by replacing them with mock outputs. The main issue with Concourse is that it has a huge learning curve to use effectively. When you have many pipelines, you end up learning the undocumented meta pipeline pattern to help manage them. You end up implementing stuff like mock outputs by yourself, since it's not really a first-class concept. Concepts like per-branch versioning that have been in products like Jenkins for years are only now entering development as "spaces". All of the configuration can be version controlled, but it's all YAML, so to wrangle everything together, you end up using something which will compile down to YAML instead. RBAC much improved in v5 but still leaves much to be desired. There are no manual steps, only manual jobs. Job triggering is relatively slow due to a fundamental design tradeoff where webhooks trigger resource checks which trigger jobs instead of triggering jobs directly, to make resources easier to maintain. Nobody really tells you this on the front page. It has its flaws. But if you delve into it you see very quickly that it's an incredibly solid product. The ability to run one-off jobs from developer workstations on CI workers, and to easily SSH into those CI worker containers from developer workstations, is incredibly helpful for debugging issues. Because everything is stateless, everything tends to "just work". If you go in with your eyes open about its limitations, you'll have a good time.
- bradrydzewski 8y agoTekton [1] works in a similar manner where the pipeline stages define inputs, outputs and tasks. The great part about Tekton is it provides a set of building blocks that can be integrated into a larger system. I hope to integrate Tekton into Drone [2] and allow individual projects to choose their pipeline execution engine. Projects can choose to start with a more basic engine, knowing they have the option to grow into something more powerful (and complex) when they need to. [1] https://tekton.dev/ https://tekton.dev/ [2] https://github.com/drone/drone/issues/2680 https://github.com/drone/drone/issues/2680
- acd 8y agoIt is good to support open source software vs closed source. Gitlab is a modern good open source alternative to Jenkins which is also open source. Gitlab has its configuration files default in yaml files which is easy to read for humans. Jenkins uses XML which is hard for humans to read. One should not use close source solutions just because it is easy to use. Just because it is easy does not mean it is good for you in the long run.
- breatheoften 8y agoI want pause on breakpoint <x> in cloud worker <y> from my IDE — tooling which does not assist me to bring that about is working against my needs.
- matsilva 8y agoClassic product promo by poo pooing on someone else.
- sethammons 8y agoOur org is happy with buildkite. We moved to that after fighting Jenkins for a number of years.
- michaelneale 8y agoCo-founder of cloudbees here, just some corrections: 1) there are (since 1 year) restartable builds/stages in open source (a bit over a year IIRC) - when the article mentions that they are not in open source. 2) Jenkins X is something VERY DIFFERENT from Jenkins (despite the name), it is "master less", yes requires a kube cluster to power it, but it has no Jenkins instances as you know it, in a lot of ways quite a bit different as it uses Tekton as the engine for running pipelines (which has a bunch of advantages). So I wouldn't group it in with the same challenges and constraints. It is something new that shares a name and some of the ideas. Jenkins, along with Spinnaker and Tekton and Jenkins X are now part of https://cd.foundation https://cd.foundation - worth a look to see how they are evolving (expect some changes).
- manigandham 8y agoUnfortunately people are looking for simplicity and this is going in the opposite direction. A kubernetes cluster is just offloading a large part to yet another component you have to run. Spinnaker is another completely separate system. This is the opposite of what teams want. Drone, Gitlab, Teamcity as mentioned in these comments is a far better approach for 99% of companies who want a solid working solution.
- michaelneale 8y agoOn the point of simplicity, much pain has been described here about running Jenkins - https://www.cloudbees.com/products/cloudbees-jenkins-distribution https://www.cloudbees.com/products/cloudbees-jenkins-distrib... is hopefully a better way to handle that. On kubernetes - imagine if that was already managed for you, all the powerful things that can be done on top vs your self (preview apps, owasp vulnerability testing, progressive delivery - all without writing a pipeline - this is stuff that can be done. I note gitlab is mentioned a lot here: they noted this power as well (see auto devops) and have started building things on top of kubernetes too.
- michaelneale 8y agoAgree Kube is not for everyone, no question, I was just trying to clarify what was in the article. (if you are offloading complexity - I would hope it is to a GKE or EKS or AKS or similar, in which case it is very much offloaded...) I didn't mean to imply that you mix all those things together from the CDF - was just mentioning some interesting projects (some are unrelated) in the mix. Agree also on simplicity - myself I don't like to run anything, so CodeShip is what we have for that (but it sounds like you are referring to self hosting only solutions?) (edit: and thanks for insightful comment)
- 2StepsOutOfLine 8y agoTangentally I've had great success with Drone. Holding similar frustrations that other commenters are expressing here it was a breath of fresh air to see how dead simple making custom steps in Drone was compared to actually needing to write code for Jenkins. Every step in Drone is just a container, so if I want a Golang build container I can just set the step image to Golang:latest and start running build commands. And if you're wanting to encapsulate some logic into a "plugin" you can just pre-make your own docker image and from there any settings you pass in with the steps are converted to environment variables that can be accessed inside the container. Many of the default drone plugins like docker registry build, etc. Are just a couple line bash scripts on GitHub.
- fnord77 8y agogitlab CI is awesome. too bad it is not a standalone thing.
- jl-gitlab 8y agoYou can actually use it standalone, for example with GitHub repos: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/githu... (Disclaimer: GitLab employee here)
- jennbriden 8y agoHello, I see a lot of great feedback in this post. I am a product manager working at CloudBees, the primary corporate sponsor of Jenkins. Jenkins is now in the Continuous Delivery Foundation as well. While it is easy to bash on an inanimate object, there are some very dedicated and empathetic people who care deeply about the project. Some of those people do this work in their off-hours and some to this work as part of their daily work activities AND also in their off hours. In that spirit, we want to make Jenkins better and created a separate group at CloudBees in the Product and Engineering teams late last year. They focus on open source work for Jenkins and on some proprietary things for CloudBees Core (built on Jenkins). We also have a dedicated user experience/product designer who started working on the project a few months ago. One of the first things he and I worked on was creating a curated, tailored version of Jenkins via the CloudBees Jenkins Distribution. This distribution will focus more and more over time on a guided workflow for continuous integration and delivery with Jenkins. These patterns will also be shared with open source Jenkins - some through direct contributions and others through suggestions (better plugin categorization, documentation, etc.). Please use this comment thread to share your constructive, honest feedback about how we can improve Jenkins.
- clvx 8y agoJenkins configuration as code. It's hard to configure without going to all the files and options. You have several things to take care of: server config files, plugins, credentials, projects, etc. Going through xml files is not fun. Also, having predetermine pipeline plugin configuration based on runner types would be nice. For example, if k8s I would expect k8s plugin + credentials + organization plugin. In many cases, the only thing I want is a pipeline which I can deliver code. Do something with jenkins declarative and get rid of groovy script. The latter makes a mess in your files, the former is very limited. At the end you have to write groovy if you want things more bespoken. It would be nice to just use any language in a container as drone does for your plugins.
- mechastorm 8y agoI think its already mentioned about Jenkins Configuration as Code. But in general stick to a configuration method, I have had to migrate from JJB > groovy scripts > init.d groovy > Jenkinsfile. And can we PLEASE open up the issues tab on the Github repos especially for plugins. There is currently no way to provide any feedback/report issues on these plugins because the "issues" tab is disabled. Our current options are - Put a comment on the plugin wiki page - Hunt down the the relevant support forum for the plugin - Find the original source repo and post an issue All of which are not ideal and does nothing to directly help the development of the plugins.
- keithnz 8y agoWe currently use Appveyor (good) and TeamCity (okish) and feed into Octopus Deploy. We have ~50ish projects (web, backend systems, embedded firmware) that all go through these systems. Works really well!
- infamousjoeg 8y agoThis completely fails to mention the security issues with Jenkins, as well. https://www.cyberark.com/threat-research-blog/tripping-the-jenkins-main-security-circuit-breaker-an-inside-look-at-two-jenkins-security-vulnerabilities/ https://www.cyberark.com/threat-research-blog/tripping-the-j...
- llamataboot 8y agoJenkins also routinely has massive security holes. Last major exploit I heard about - the matrix.org exploit, was from privilege escalation through a Jenkins vulnerability [1] [1] https://www.zdnet.com/article/matrix-hack-forces-servers-offline-user-credentials-leaked/ https://www.zdnet.com/article/matrix-hack-forces-servers-off... -- This isn't nec a knock against Jenkins itself, but certainly a knock against thousands of orgs running their own unpatched Jenkins servers, often on the same machine as their other apps
- sisu2019 8y agoNormally I have no issues getting things to work but I bounced off trying to get modern Jenkins to build a github gradle project three times now. I even used to have one, way back, that built maven projects from SVN and deployed into artifactory but I can't seem to manage the same now. I thought it was me but maybe jenkins just became really awful in the meantime.
- bcoughlan 8y agoI spent quite a while grappling with Jenkins pipelines, and while it definitely has its warts, once you get over the hump and get one pipeline the way you want it, it's quite easy to use it as a starter for other projects. Personally I go to great lengths to minimise the plugin dependencies and push the logic into bash scripts or Maven/Gradle. The Jenkinsfile just calls the bash scripts, so I can run it before I change it. Where I use plugins in the pipeline I document what they are and what they do in case we ever need to go from scratch. It is hard to test but the linter picks up quite a lot of errors before push, and if you're doing a multi-branch pipeline you can just test your changes by pushing to your own branch. It's a long cycle, but the variation should go down the more you have them. It has also been indispensable as an ad-hoc task scheduler. For example, I have a job that cleans up old branch artifacts every Saturday night. It was easy to alert on failures and see the results and history of past runs. I don't know anything else that would have fit the bill. When you make software over 10 years old in such a fast-changing environment, the legacy of the software can weigh you down. I've worked on similar products that were tied to their legacy counterparts (sharing a database), and every change meant carrying the baggage of backwards compatibility with the legacy platform. If I could do it all again, I'd have done it the Basecamp way: Get a clean break from the legacy version, and have a migration path to the new system. I would like to see CloudBees take all they've learned over the years, take the great work they've done on Blue Ocean and their core job scheduling system and put it all together without carrying the history with it.
- zhynn 8y agoInteresting, not a single mention of Vespene, or more relevantly: https://medium.com/@michaeldehaan/discontinuing-vespene-17fd9c0f3079 https://medium.com/@michaeldehaan/discontinuing-vespene-17fd... Michael DeHaan worked on it after Ansible, but he got zero traction, and so shuttered the project. I think he should have stuck it out longer, because people hate Jenkins.
- stan_123 8y agoNo mention of Jenkins Shared Libraries? When Jenkinsfile pipelines start getting complex, or need to make them more generic, you can create groovy lib-s in a separate repository. Single line to load the lib dynamically at start of Jenkinsfile. Then you can call library functions/tasks passing appropriate params within your pipeline. But as anything Jenkins, documentation lacking or out of date.
- deboflo 8y agoAWS CodePipeline with CodeBuild supports GitHub webhooks, export artifacts between stages, many types of caching (S3 artifact caching, Docker caching, file/volume caching), Secrets Manager environment variable integration, queueing between stages, pipeline retries, stage retries, manual approval steps, integration with CloudWatch and many of services like Lambda, and provisioning all of the above via API, CLI, SDK, or CloudFormation.
- scarface74 8y agoAnd the beauty of CodeBuild is that if you need a custom build environment just for your special snowflake build, it’s as easy as creating a Docker container and it won’t interfere with anyone else’s build environment.
- robrtsql 8y ago> queueing between stages Doesn't it only support a queue depth of 1 at each stage? Last I checked, if you have a change that is queued up waiting to get into a stage, and a newer change comes along, that newer change will supersede the old one. That makes CodePipeline only good for workflows where you only care about deploying the 'newest' of something. I think the worst thing about CodePipeline is how hard they make it to run custom code from your pipeline. Your options are Lambda (limited to 15 minutes per run, or you have to rewrite your Lambda function to be called once every 30 seconds, essentially using the first run to start your deployment and every subsequent run to check to see if it's done yet or not) or CodePipeline Custom Actions (where you have to write the AWS SDK code for interacting with CodePipeline). The AWS Developer Tools team could learn a thing or two from Azure Pipelines. They did it "right", IMO: you can create a 'bash' stage in your pipeline which runs whatever script you want from the build agent (which can either be hosted my Microsoft or hosted yourself). That's all I really want. CodePipeline could support that with a custom action but it's more stuff that I would have to set up.
- solarengineer 8y agoSee https://build.gocd.org/ https://build.gocd.org/ for the live instance of GoCD used to build Gocd (apache licensed CD orchestration tool backed by Thoughtworks).
- closeparen 8y agoOf all the undifferentiated heavy lifting you can offload to your cloud provider, CI makes a lot of sense: - You probably want to store artifacts in its object store anyway. - Rent exactly the right amount of compute for the concurrent builds running in any given minute. - If your projects have sufficiently simple build-and-test entrypoints i.e. "make" then you're highly portable and not too worried about lock-in. You don't even need to be deploying to the cloud provider you're building on.
- killjoywashere 8y ago</read-article> Um, so doesn't his workflow buy an essentially infinite maintenance problem with all his containers? This sounds like tomorrow's Node left-pad debacle in a different form.
- FraaJad 8y agoHow does sourcehut (sr.ht)'s build service compare to the alternatives mentioned here?
- RickJWagner 8y agoIt seems like KNative wants to move into that space. Regular turnover in the Programming world. You have to re-learn everything once or twice a decade.
- liveoneggs 8y agogroovy pipelines solve a lot but cost a lot. Finding where to draw the line is really really hard. Anyway jenkins works, is free, is easy to setup, and easy to use for everyone except the one writing the groovy :) groovy sucks, jenkins works (high praise, really), but you get what you get
- mrferos 8y agoWe use Jenkins at work and have found a pretty damned sweet spot. Let me start by saying that we used to use GitLab, a lot of it was because of the CI but I didn’t have a great experience trying to manage it on top of Kubernetes, they’ve since introduced a Kube native package and I’ve been told it’s much easier, but with the deployed omnibus we ran into a lot of issues with runners randomly disconnecting, it became frustrating to the point where I had developers not wanting to use GitLab and finding interesting ways to work around it. So I set up Jenkins on a dedicated EC2 instance with a large EBS volume for workplace storage and installed the Jenkins plugin then I wrote a Jenkins library package that exposes a function to read a gitlab yaml file and generates the appropriate stages with parallel steps that execute as pods in khbernetes - took about a week to get the things we actually used from Gitlab CI and their YAML DSL working correctly. Now we very happily use Jenkins, mostly through YAML but in the occasions where things are much easier going directly to Groovy to use interface with plugins, developers can.
- mrferos 8y agoI want to specify this is my own experience and I think a lot of our own issues may have been from mismanaging our self-deployed setup. I’ve had a lot more experience managing Jenkins. GitLab and their approach to CI (easy to use yaml) really facilitated developers writing CI, which increased our software quality overall.
- yebyen 8y agoHave you used the Jenkins Job DSL plugin? I'm just getting started using it, but it seems like the solution to scaling up to a lot of Jenkins jobs. There's a good talk about it, and since you're one of only two people in the thread who used the word DSL and you are having a good experience with Jenkins, I thought I'd ask. My config is similar except my single EC2 node is actually running Kubernetes via kubeadm, it's a single node Kubernetes cluster and has enough room for about 3 of my worker pods to execute concurrently before the rest have to wait. (But that's just my setup and has nothing to do with Job DSL.) For me, managing Jenkins via the helm chart has been the best part of the deal, but I'm a pretty big fan of Helm already...
- pjmlp 8y agoNot at all. In what concerns Java and .NET projects, Jenkins is just a mature tool that gets the job done. We have no reason to consider anything else.
- peterwwillis 8y agoGetting old? It's ancient. Nobody should use Jenkins for new projects. I even started writing my own piece about it on a plane back from a recent vacation.
- sabujp 8y agodrone is like google cloud build and probably and whatever amazon has
- EngineerBetter 8y agoI'm a very strong Concourse proponent. We've been using it exclusively for three years, and it's better than anything we've experienced that our customers use (I run a consultancy). We run Concourse London User Group, and the first few instances were like some kind of Jenkins survivor's support group.
- ogtifs 8y agoAnyone looking for a more back-to-basics self-hosted CI might be interested in Laminar[1] - this article reminds me why I was inspired to write it [1] https://laminar.ohwg.net/ https://laminar.ohwg.net/
- oblio 8y agoIt's not distributed, from what I see. I don't see stuff about agents. Oh, it does, via DYI SSH, sort of. Could work, I guess.
- ogtifs 8y agoRight, the philosophy is more about making it easy to DIY stuff and getting out of your way, rather than endlessly tacking on features and integrations.
- eeeeeeeeeeeee 8y agoWe ran Jenkins in-house and it was a miserable experience. Always breaking. Any time we needed to update it we knew it was going to break, and it did. Debugging startup failures was always a pain if you had a moderately complex setup. There were times where it was quicker to wipe the entire configuration and start over than trying to debug it. We switched to CircleCI and it did everything we needed and we never had to lose half a day to CI problems again.
- jdhzzz 8y agoI work at a big enterprise that uses Jenkins. We support a large group of legacy applications and the waterfall type development processes that lately have been papered over with agile-lite. I work about 80% of the time administering (Cloudbees) Jenkins. I have lived through Freestyle projects, migrating to Job DSL, and now we are moving to declarative pipelines in shared libraries. I agree with 95% of the article's pain points. Still, I'm a fan, perhaps because each iteration is significantly less painful from the previous one. Still Jenkins works well for us in our environment. Our waterfall style development processes make for ops-friendly, cookie-cutter type builds: one deployable asset per repository, one deploy target, all projects laid out the same way... Jenkins shared libraries provide powerful ways to capture the commonalities among the projects and provide a way for us to manage them centrally, while providing some (small) flexibility for customizations by the developement teams. Shared libraries also allow the ability to test changes using selected versions (branches) of the pipeline for specific builds. Looking forward to what's next (I still have a few mortgage payments to make...).
- RealSpamapS 8y agoAnyone who agrees with the OP should give Zuul-ci.org a look. I run Zuul in production with GitHub and AWS, and it's been really useful for us. It scales to the moon, parallelizes everything, and having all job/pipeline configuration in git means you can test things like CI job changes before they're committed. Cross-repo pre-merge dependency management means I can build whole speculative futures in PR's before it lands and breaks the build. Funny story, the Zuul build status badge is just a static image, because Zuul does not allow code to merge if it fails tests: https://zuul-ci.org/docs/zuul/user/badges.html https://zuul-ci.org/docs/zuul/user/badges.html Full disclosure: I worked on a defunct zuul based service for a little while, and still am a core developer with lots of code in Zuul. A few presentations I've given are here: http://fewbar.com/the-build-is-never-broken/ http://fewbar.com/the-build-is-never-broken/ http://fewbar.com/zuul-ci-crossing-streams/ http://fewbar.com/zuul-ci-crossing-streams/