33 ms·
The worst thing about Jenkins is that it works (2019)
- craig 3y ago> Warning – I’m going to rant for many, many paragraphs. My advice is to skim I had a good chuckle at this.
- gjvc 3y agoAvoid Jenkins, and if you can, try https://www.jetbrains.com/teamcity https://www.jetbrains.com/teamcity "TeamCity Professional is free – even for commercial use – and has no limitations on features, number of users, or build time. It allows you to configure up to 100 builds and run up to 3 builds in parallel, which is more than enough for most projects"
- xkekjrktllss 3y agofree until it's not, of course
- aloisdg 3y agoor enshittificated
- DarkNova6 3y agoJetbrains is among the very few companies I’ll trust for the time being
- ldjkfkdsjnv 3y agoJetbrains products are outstanding
- rqtwteye 3y agoSame here. Although I am super nervous that they will either cash out by going public or selling to some investor. That will be the end of it.
- juunpp 3y agoThey aren't a silly con valley startup and they've been around for 20+ years. https://www.jetbrains.com/company/ https://www.jetbrains.com/company/
- chrisandchris 3y agoMaybe they will at one point, but JetBrains so far did never take VC capital (and if I remember correctly said "no" more than once) [1]. That doesn't tell about the future, but they probably had many options to cash out and never did. [1] https://www.bloomberg.com/news/articles/2020-12-18/czech-startup-founders-turn-billionaires-without-vc-help https://www.bloomberg.com/news/articles/2020-12-18/czech-sta...
- devmor 3y agoAlways praying that they never will. I’ve been using their products since my first “real” developer job over 12 years ago and I never want to switch.
- juunpp 3y ago"For the time being" is an understatement. The company has been around for 20+ years, and while I haven't used TeamCity, if it is anything like their other products, you can be certain it is built with a high level of attention to detail and that that won't change anytime soon. It is one of the few companies left I license stuff from.
- xkekjrktllss 3y agoIt is a capitalist enterprise, which means control by its very owners is contingent on making enough profit to pay the bills, in the very best of cases. This is not my opinion.
- irrational 3y agoYeah, but this is JetBrains. They are one of the few companies whose products are so much better than anyone else’s that I’d pay for them, even as an independent developer. They’ve been around for decades and are still more than popular enough that making money shouldn’t be their issue. My company of 80,000+ employees has a JetBrains license for every developer.
- xkekjrktllss 3y ago...for now.
- DarkNova6 3y agoIt's not a publicly traded company and not obligated to increase value for shareholders. Also, they are located in Czech Republic which has a very different business mentality, pursuing product quality over short term profits.
- runiq 3y ago> TeamCity Professional is free – even for commercial use – and has no limitations on features, number of users, or build time. Yeah, no. There's obviously gonna be a catch in the future.
- chrisandchris 3y agoMaybe, yes. But it's exactly the same license / pricing since i first used it and that's somewhat around 5-7 years. Quite some time so far. I would recommend TeamCity to everyone, it's awesome.
- capableweb 3y agoJust as a note: TeamCity originally launched around 2005 sometime, so it's coming up to be 20 years old soon. I think TeamCity Professional, when self-hosting, has been free since always.
- AaronM 3y agoThe catch is your limited to a small number of parallel builds (3). Our CI pipeline would take hours to run with that limitation
- solardev 3y agoTo be fair, I've never seen a company turn around a licensing fiasco as well as Jetbrains did. Initial announcement, 2015: https://blog.jetbrains.com/blog/2015/09/03/introducing-jetbrains-toolbox/ https://blog.jetbrains.com/blog/2015/09/03/introducing-jetbr... Community anger a couple days later: https://bytecrafter.blogspot.com/2015/09/how-jetbrains-lost-years-of-customer.html https://bytecrafter.blogspot.com/2015/09/how-jetbrains-lost-... Significant changes after hearing the feedback, a couple weeks later: https://blog.jetbrains.com/blog/2015/09/18/final-update-on-the-jetbrains-toolbox-announcement/ https://blog.jetbrains.com/blog/2015/09/18/final-update-on-t... I'd trust Jetbrains -- as a company with a few decades of experience serving devs and only devs -- much more than your typical get-rich-quick cloud SaaS startup that's just looking for a quick exit. Yes, they need income too, but they're one of those few tech companies not aiming for hypergrowth, just a sustained user base. Even though VScode gets all the love these days, and Atlassian and Github and CircleCI etc. get all the enterprise use, they've made some really cool products of their own too (Space, TeamCity). And their IDEs continue to be excellent. It's not like TeamCity even has a secret catch... their cloud version DOES have a monthly cost already. It's only the self-hosted version that's free.
- marginalia_nu 3y agoYeah, I use TeamCity for both my test and production setups. I've only had positive experiences with it, very nice seamless integration with Jetbrains' IDEs as well.
- callmeal 3y ago>configure up to 100 builds and run up to 3 builds in parallel Funnily enough, Jenkins does not have that limitation.
- gjvc 3y agoIt has plenty of others which more than make up for it though.
- gjvc 3y agoThe responses to this post are hilarious.
- lmm 3y agoTeamCity is even worse than Jenkins. Jenkins randomly breaks and you can eventually fix it, or kludge some kind of workaround. TeamCity randomly breaks and you call support and they can't fix it.
- deleted 3y ago[deleted]
- prabhatsharma 3y agoGithub actions works great Even AWS codebuild is good.
- deleted 3y ago[deleted]
- opentokix 3y agoReally like the one line dissmissal of gitlab ci - haha, what a n00b.
- anon23432343 3y agoJenkins works... No it dosen't or maybe it does for java which I don't care about. The ui looks like from the 90s. Finding anything in bigger projects/companies is impossible... I have browser bookmarks to find anything. The log view is annoying at best. I have nothing good to say about jenkins, it feels like it is stuck in the 2000s and refuses to change.
- colordrops 3y ago"refuses to change". Is there an active team working on it or is this anthropomorphism of a software project?
- xp84 3y agoThe 90s were the last actual advance in UI design anyway. Design has drifted in a terrible direction, in the past decade or two especially. Some of this is justified (by some) by optimizing for mobile, but that’s precisely the kind of thing I wouldn’t want creeping into my CI tooling anyway. I don’t do that on a phone.
- deleted 3y ago[deleted]
- dangus 3y agoThis idea that design peaked in the 90s does a big hand wave over a whole bunch of horrible examples of the user interface from that era. Is the Space Jam website peak UI design? What about Microsoft’s “hell of tabs” settings dialog boxes? Or the original Amazon home page: https://www.versionmuseum.com/images/websites/amazon-website/amazon-website%5E1997%5Eamzn-homepage-restored.png https://www.versionmuseum.com/images/websites/amazon-website... What link do I click to find my order status? Where do I go to search for a book by keyword? What will happen if I click the link for “first time customers click here?” Same deal with Yahoo!, what to the “new” and “cool” and “more yahoos” buttons do? http://2.bp.blogspot.com/-JAp1M-Q_0_Y/TkAwsyueWxI/AAAAAAAAAMk/EO918aOa_TM/s1600/1996_yahoo_internetexplorer.jpg http://2.bp.blogspot.com/-JAp1M-Q_0_Y/TkAwsyueWxI/AAAAAAAAAM... If I want to check a build status on my phone what’s the excuse for the web page to not display nicely on that device? You really think there’s no good reason for me to want to do that? Doesn’t modern CSS allow you to completely customize the viewing experience for both platforms in the same codebase?
- sethammons 3y agoWhile we are plugging ci tools, I enjoyed buildkite at my last gig
- dafelst 3y agoSame, we are using it now and it is a much nicer experience than Jenkins
- muglug 3y agoNeeds a (2019) Richard was my manager in early 2019, and was awesome.
- kureikain 3y agoJenkins has improve a bot. before github action has self host runner, Jenkins has worker node years ago. using jenkins declarative pipeline, almost everything can explain in code. updating jenkins is a walk in the past. the most tedious part is dealing with no longer maitenance or support plugin. make sure to never expose jenkins out. use a relayer to bring webhook traffic in
- lacrimacida 3y agoI had a gig (circa 2014) where Jenkins was used in atrocious ways where it was basically running the show. Yet there were no problems with Jenkins itself, it was running smoothly
- fullspectrumdev 3y agoI’ve found that people will use CI tools like a hammer once they get them working - everything’s a nail, even screws or bolts. You often end up with these glorious Rube Goldberg machines where Jenkins does everything and becomes a single point of failure - if the instance falls over, everything’s fucked.
- FireBeyond 3y agoYes, one multinational company who was a customer at my last employer lamented their "dystopian Jenkins hellscape", "glued together with 300+ Python scripts".
- lmm 3y agoI once worked somewhere that had two very different Jenkins instances - one that was the typical dev CI instance, and one that was a job scheduler for all the prod batch jobs. It actually worked very well.
- zdw 3y agoAll of the test runners have their issues. I've had as much problems with Jenkins as I've had with Github Runners and similar solutions. The best way out of this is to subdivide responsibilities: - Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. - Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable. That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.
- nerdponx 3y agoThis is basically what we do, using TeamCity. DevOps sets up all the permissions and VCS integration, and helps figure out all the AWS IAM stuff. Then we get a nice clean little sandbox to run our build and deploy scripts in, which we commit to version control, including CloudFormation templates. It works really well: it's a comfortable separation of responsibilities that allows our team to move quickly with relatively little risk, and it involves almost no hands-on attention from DevOps once it's up and running, so they can direct their efforts elsewhere.
- sevagh 3y ago>- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. This always breaks because then each platform has a specific way of defining env vars or secrets for those Makefiles and bash scripts. End result is devs still can't really run CI "proper" the way it's configured in a runner.
- zdw 3y agoThis is fundamentally an interface problem. As you called out Env vars are a pretty common and well supported interface, across multiple platforms. Do you have a better option to suggest?
- ath3nd 3y ago> “everybody should be writing and maintaining their own CI jobs” And, of course, everybody should be using whatever language and version they want to, and run their software on whatever cloud they want. The notion of this makes me super uneasy, but at the same time it's a good explanation why many big orgs have 100s of engineers when they can probably easily do with 10s of them.
- dig1 3y agoJenkins is not perfect by any means, but I can say it is much better than existing solutions (taking pros and cons after trying and using many of them). Sure, it doesn't have fancy UI or flying buttons, nor is it rust/PWA/react/whatever-rage-is-these-days, but it does its job - it works, is free, and has been very well-maintained for ages. Download a single file and run it. Want to extend it? There is a plugin for everything under the sun (but be careful, you can easily bloat it). IMHO, I don't want bells and whistles from the CI tool. CI tools are dry; I see them as a car drive shaft - it must work properly, and I don't expect it to be pretty. If you need to showcase it, you are probably doing something wrong. And one more thing: I expect Jenkins to be around and free in the next 10 years. I doubt that will be the case for "insert-your-favorite-ci-from-some-company".
- 2OEH8eoCRo0 3y agoIt's infrastructure. It should be boring, unchanging, and without surprises. It should be so reliable that we take it for granted.
- Nextgrid 3y agoAgreed - I'd much rather spend more time working around Jenkins' peculiarities once than constantly deal with change for the sake of change because some PM wants more "engagement" to justify their job and promotion. Tools need to be boring and get out of my way - something "modern" tooling forgets since everyone's paycheck depends on forgetting that.
- troupo 3y agoJenkins BlueOcean is one of the best "fancy UIs"
- i386 3y agoThanks! My team and I made that and it was a lot of fun :)
- vijaybritto 3y agoUsing that everyday all day. Thank you!!
- deleted 3y ago[deleted]
- btbuilder 3y agoThe kubernetes plug-in for Jenkins works well for us, and as the checkout/clone happens in the pod you don’t have the workspace bind mount problem. We are using docker-in-docker at the moment but looking at buildah. We do parallel windows container builds as well. Jenkins is a bit like Perl IMO: there are ways to use it that avoid the majority of its sharp edges (eg putting logic in bash and just call bash, avoid groovy) if you know the perils that lurk.
- sigwinch28 3y ago> We are uding docker-in-docker at the moment You can also run a "less privileged" container with all the features of Docker by using rootless buildkit in Kubernetes. Here are some examples: https://github.com/moby/buildkit/tree/master/examples/kubernetes https://github.com/moby/buildkit/tree/master/examples/kubern... https://github.com/moby/buildkit/blob/master/examples/kubernetes/job.rootless.yaml https://github.com/moby/buildkit/blob/master/examples/kubern... It's also possible to run dedicated buildkitd workers and connect to them remotely.
- AaronM 3y agoThe biggest issue I have with the plugin is the container takes a significant amount of resources to run. I don't know how that compares to other solutions like github though. Being able to run a really lightweight container in argowf has its advantages
- pyjarrett 3y agoI don't mind Jenkins pipeline limitations because my philosophy is that CI should be as simple and contain as little build logic as possible. Every job should be a few simple commands or a python script or something I can run locally. This simplifies reproducing and debugging issues caught by CI -- there's nothing worse than wasting time trying to reproduce what happened.
- demizer 3y agoJenkins is rock solid. Been using it for three years to deliver highly critical software. The development experience is the worst of all ci tools since no one wants to setup jenkinspipelineunit to test the pipelines. Dagger might be the salvation here, but even that tool didn't support Jenkins out of the gate. And for those that are reading this, the statement about pipeline groovy being a trap is exactly right. Avoid writing groovy at all costs. No one on the team wants to get good groovy and if you do, you're weird. The way we do it is to just use the Jenkins file to call the scripts (or tools) with bare minimal script code. Edit, forgot to add that I recently interviewed a bunch of candidates for a high paying job at a ai hardware company and every one of those candidates stated they worked on or near a project that involved moving from Jenkins to Gitlab CI.
- rightbyte 3y agoOh ye keep it simple. I worked as a Jenkins admin for half a year and one day I woke up from a nightmare and realized that I've made some sort of parallelization batch job processor communicating via outputting bat-files, in Groovy, to cut down build times.
- deleted 3y ago[deleted]
- bigstrat2003 3y ago> And for those that are reading this, the statement about pipeline groovy being a trap is exactly right. Avoid writing groovy at all costs. I disagree with this pretty strongly. I've worked with Jenkins where it was all freestyle jobs and it was a nightmare to maintain. Pipelines written in groovy are essential to doing Jenkins well, imo. It's also not exactly hard to write groovy. It's basically Java, not some weird esoteric language nobody knows.
- maratc 3y agoI'd suggest you wait until you spend a whole day on script approval and @NonCPS idiosyncrasies before you define Jenkins pipeline Groovy DSL as "basically Java". This doesn't mean you shouldn't be writing it, you should — but for the reasons completely different from its supposed "similarity to Java".
- loloquwowndueo 3y agoConcourse CI is great, it felt like it would be easier to write Concourse resources for the custom things we needed rather than fighting Jenkins anymore.
- solardev 3y agoDoes Jenkins let you test workflows locally? That's the thing I hate that about CircleCI, Github Actions, etc... having to "commit, upload, wait" to test every minor change =/
- Jnr 3y agoI put all the important steps in scripts (makefiles or as commands in package.json for JS projects) which can be run without Jenkins. Pipeline in the Jenkinsfile only runs those steps separately in the correct container build stage to collect separate step results. If done correctly, the Jenkinsfile should be easy to read by a person who has never written a Jenkins pipeline before. With this approach it is hard to mess things up by creating a monster pipeline which no one can understand, and if necessary, you can quickly change to different CI runner at a later point in time.
- Riverheart 3y agoJenkins has a “Replay” button on completed jobs and allows you to edit the Jenkinsfile before rerunning it. That is the easiest way to test changes without going through version control.
- happymellon 3y agoInteresting, the side snipe about Docker > One of the mostly-false promises of Docker, as it was sold to me by the true believers who introduced me to it, was that, if you do it right, you can run the same docker image, and therefore have basically the same environment in production, in CI, and on your local development machine. I literally have never worked anywhere that hasn't used the same container in dev/test/prod. People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image?
- dijksterhuis 3y ago> People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image? Tensorflow used to generate multiple images for different use cases: cpu only, GPU, cpu with Jupyterhub, GPU with Jupyterhub. So each release would generate minimum 4x container builds. CPU only images were useful for local testing of ml code at CLI (does it run) when a GPU might no be available. Jupyterhub variants for simple GUI for development work. GPU for obvious reasons (run the damn thing). I can’t remember if they used to do builds for their release against multiple CUDA / CuDNN versions… that might have been PyTorch.
- AdrienPoupa 3y agoPHP dev here, we have extensions for development that make no sense in production, xdebug for example. You need it for breakpoints and debugging in general but it should not be installed in production. So we extend our production image and install it on top of it. Similarly, we include Composer (package manager) in the dev image only as we only need the installed dependencies in production but not the package manager. Our dev image is a flavor of the production one, really.
- comprev 3y agoWould multi-stage Docker builds not help here? Composer executes in one step and the result artefacts are copied into a "clean" PHP image without Composer installed.
- hardwaresofton 3y agoGitlab CI is still the best CI in the game IMO, but GitHub Actions gives it an incredible run for it's money because of how easy action re-use is. I meant to make a blog post about this, but here's a good a place as any: GitLab absolutely innovated many hard parts of CI/CD as a platform-native piece, but it feels like they lose slightly to GitHub on what GitHub does best -- social virality for developers. The problem with losing slightly there is that the advantage compounds; if developers find it easy to make and share, then they make and share which makes more people make and share and generates tons of value for a platform. It may not be safe, but being able to: - uses: some-developer/some-repo@some-branch Is absolutely amazing. Everything else (generally) GitLab has, and (likely) did first, but the social viral stuff GitHub just gets right in a different way than GitLab does. No one would accuse GitLab of being a "copy" in recent memory like they did in the bad old days, but IMHO they need to just copy the shit out of when GitHub gets social things (like this) right.
- capableweb 3y ago> GitLab absolutely innovated many hard parts of CI/CD as a platform-native piece As someone who only used GitLabs CI briefly when it initially launched, what hard parts did they innovate on exactly? As far as I could tell, it's a run-of-the-mill CI/CD platform, for better or worse, but nothing I'd call "innovative". But again, maybe since the first time I tried it when it launched, it has changed, and I missed something really cool they did.
- starttoaster 3y agoInnovative is a strong word, but they always had the best support for a variety of deployment archetypes, including baremetal, kubernetes, docker, docker+machine (autoscales VMs in a public cloud to run many Docker executor runners.) They also had JWT authentication support built into GitLab before GitHub even had a CI offering at all. The GitLab kubernetes agent makes their CI/build tool more like a CD tool for organizations that primarily deploy to kubernetes clusters, which is fairly unique for a CI tool to act as a CD tool, though with scripts people often just make the CI tools do deployments anyway. My experience is limited to GitHub Actions, GitLab CI, and Jenkins. I've never been in an organization that managed to crack the code on running Jenkins in a sane way (always a poorly maintained, bloated mess of plugins, unfortunately) so I can't say I count Jenkins as a contender in this conversation. One could argue it's unfair to discount Jenkins because of organizations that manage it poorly, but there's something to be said about a tool that is so easy to accidentally run in an unmaintainable way. I also have some limited experience with Drone and Circle, but not enough to talk about them confidently.
- anymouse123456 3y agoBeen using git hooks with in-repo Makefiles and small bash scripts to good effect for some years now. It's fast, simple and took a couple hours to set up. Rarely ever needs attention. It just works.
- ysofunny 3y agothe worst think about Java (in general, Jenkins is Java) is that it solves a real problem that companies have. I put, however, that only companies have the problem, not programmers, and not even regular (sized) engineering teams; except when they have a fast employee turnover speed. I do not have a very clear idea, but roughly speaking I'm saying that java does really well in the "too many engineers are working on the same code but not at the same time" problem space
- intelVISA 3y agoIsn't Go the New Java in that sense? I am no fan but it seems industrial strength for this kinda use case.
- mcfedr 3y agoOf everything I've tried, gitlab co is my favourite. It's just a shell script, and that makes it super simple to replicate yourself locally, and - I think most importantly - it's really obvious what it's doing. I think that's my biggest problem with Jenkins and GitHub and such is all the plugins that do something, but without digging through the source code you cannot work out what they really do
- mschuster91 3y agoI'm so much more happy with GitLab CI. The only thing that is easy to implement in Jenkins, but an outright horror in anything else, is the usual "helper scripts" stuff: sync the production database to integration (and remove PII in the process), run reports and data analysis... the best way I've figured out to implement this is to create an empty branch (git checkout --orphan helper_xyz) and then run pipelines against this branch either manually or on a scheduler, but it sucks. And this is what keeps Jenkins afloat, despite it being a consistent source of CVEs both in core and in its myriad of plugins that don't bother to follow semantic versioning so every upgrade is playing Russian roulette with your company.
- happymellon 3y agoAside from the docker snipe I've already commented on, everywhere that I've had to deal with Jenkins it's been an issue because it's always a pet. There are always issues, and like the author, it's "only" about 25% of the time. A quarter of all builds fail due to things absolutely not related to that specific build. > oh, we were resource constraint at that one second, retry and it'll probably work > oh, we can't replicate that error message from a tool that you have no visibility over, retry and it'll probably work > oh, we have no idea why something timed out, retry and it will probably work Now when I have a failed build I don't actually trust the build before I don't trust that there was an edge case in an integration test that failed. Because it's almost always something outside of the code. [Edit] downvotes but no constructive responses? The issue isn't necessarily Jenkins, it's the type of people who use Jenkins to roll their own everything.
- mqus 3y ago> oh, we were resource constraint at that one second, retry and it'll probably work Had the same issues with (self-managed) gitlab CI. This is not jenkins-specific and usually something you can fix much easier in jenkins (other than just retrying) > oh, we can't replicate that error message from a tool that you have no visibility over, retry and it'll probably work This could maybe be better in other, more declarative CIs but honestly, its pretty much the same. The only thing some of them have is a local runner. > oh, we have no idea why something timed out, retry and it will probably work Jenkins has this, yes. But use another CI with docker and you can get the same and much worse. But in essence, yes, at our company too, jenkins is a pet. But this means that someone knows it from the inside and outside and knows which things to fix in your pipeline. Good luck with trying to extend gitlab CI (or others) to fix things you need and/or opening an issue with them. Gitlab has quite some high-profile issues sitting around for years with customer requests piling up. In jenkins, you'll find an extension that will do exactly what you want, or you could roll your own.
- happymellon 3y agoYou appear to be reading more into what I'm saying. I didn't say this was exclusive to Jenkins, just that Jenkins has always been a subset of this problem. Selfhosting this, and wanting to make it completely customised to fit the whim of whatever someone in compliance/security/accounting decided is an issue. It doesn't help that Blue Ocean is a terrible UI, but that's just an ugly facade to a system that enables bad habits. Other systems can be abused, but whenever Jenkins appears it is always abused.
- shadycuz 3y ago> On a previous team I had used Concourse CI to some extent, but I wasn’t really blown away by the experience. Travis and Circle were mentioned. I was a fool. I should have committed to seriously researching some of the contenders and making a more informed decision, but I lacked the willpower and the discernment. The whole post can be summed up as he had very little CICD experience. Made lots of beginner mistakes, which is easy to do in Jenkins. Then decided to write a post where all his complaints about Jenkins are not only wrong but are the issues that plague all the other CICD tools. > So instead of writing Bash directly, you’re writing Bash inside Groovy Why are you doing that? You have a fully featured programming language and you are running `sh('npm install')`. You could do this instead https://github.com/DontShaveTheYak/jenkins-std-lib/blob/master/jobs/js/node/npm_example.groovy https://github.com/DontShaveTheYak/jenkins-std-lib/blob/mast... . How is bash inside of YAML better? > The trouble is: Groovy is a much, much worse language for executing commands than Bash. Bash is interpreted, has a REPL that is great for experimentation, does require a ton of imports, and has lightweight syntax. Groovy has none of these things. Groovy has a language server, linters and a vscode IDE plugins. They are probably not as stable or full featured as the bash ones, but they are available and very few take advantage of them. Again, how is YAML+Bash better? > The way that developers test their Groovy steps is by triggering a job on the remote Jenkins server to run them. The feedback loop is 2 orders of magnitude slower than it is for just executing Bash locally. This is a rookie mistake. For about 60-75% of pipelines you can run them locally in a docker container on your local machine. You can even set up hot code reload so as you change your pipeline the Jenkins reloads it. You can also configure the job to kick off a build when it reloads the code. When Jenkins is configured correctly it has the fastest feedback loop of any CICD tool on the market. GitHub actions comes in a close second since it can also be run locally but you cant run a "clone" of what you run in production, like having the same secrets, so it gets second place. Beside Jenkins and GitHub actions, I dont know of any solutions for the other tools. You can run a GitHub action on Jenkins. It's a very deep and complex system. It's like an iceberg and so many engineers dont leave the surface before deciding it sucks and one of the YAML CICD tools is better. Sure the YAML alternatives are EASY to get started with and to do basic stuff with. But they are Terrible at anything complex. While Jenkins is not easy to get started with, once mastered, you can build complex pipelines with ease. I get that I'm a Jenkins fanboy. Most of the things I mentioned above, I either contribute to or I'm the author of. I know Jenkins has issues. I know it has hurt lots of people, I read the complaints online. But it's still the best out there. The best software in the world is not written in bash or yaml and the same is true of the best CICD pipelines in the world. It's a shame very few people get to see/use those pipelines.
- dagmx 3y agoI truly, truly dislike Jenkins but haven’t found a better locally hosting CI system that is repo agnostic. My issues with Jenkins: 1. Groovy is a really annoying language to configure stuff in. I’ve never written it for other things but it has so many little idiosyncrasies that catch you out. 2. Jenkins doesn’t have much in the way of resource management. It’s easy for a job to take over the entire node and Jenkins doesn’t have the smarts out of the box to monitor this and put things in a different node. 3. So many random job killing bugs that boil down to poor job isolation and resource tracking. Every now and then killing one job can take down another. 4. The UI and UX is bad. I know there are better interfaces for it, but they all just have such limited interface exposure of state beyond “running” and “failed”. Estimate time is useless because it aggregates failed jobs. 5. Inter job dependency management and tracking is poor and annoying to setup. It’s doable, but really should be a better first class citizen. In many ways, I start looking to other tools to fill my CI need that aren’t meant for CI: Render farm managers. Film farm rendering is largely similar to CI at a structural level. But they have much higher throughput of individual jobs and have built around making the most of that. There’s better inter job dependency tracking and resource management because of that, and they’re built to expose more information up front to help non-tech savvy artists or render wranglers who need to see status quickly at a glance on a few hundred jobs at once.
- gregopet 3y agoI've written web servers in Groovy for at least 5 years.. Groovy is great, once you get to know it, but some aspects of its design certainly don't "click" for a while. I'd be using a syntax for years when I'd read in some blog post "Oh that's just xyz with a shorter syntax" and I'd go Ooooh! Great little language though, IMHO one of the best for quick prototyping.
- dagmx 3y agoI don’t doubt at all that Groovy is great for doing stuff when you’re used to it and spend a reasonable amount of time with it. I’ve heard great things. But things like the differences between single quotes and double quotes and a bunch of other subtleties like that, make it really frustrating when the only time I use it is updating build logic. I’d honestly prefer a more consistent language like Python or Lua. Even if I don’t code them all day, they have more resources to lookup and fewer gotchas.
- camgunz 3y agoThe most interesting part IMO was at the end when op was noting his new company doesn't have teams/engineers create and maintain their own jobs. I've worked lots of different places now and something that's always frustrated me is how reticent engineering orgs are to reevaluate their processes and policies. Like it looks like his first org could have moved on from this problem by adopting the second org's policy. There might be other tradeoffs, but IME no org even wants to hear about this stuff, and they definitely don't have metrics for evaluating anything such that they might say "huh, this isn't working."
- ebiester 3y agoThere are tradeoffs. What happens when the CI team is a bottleneck? What happens when you have to frontload tasks you might not need because you’re afraid your release could be compromised? What happens when the CI team says no and it introduces complexity in your system? Or you end up doing “shadow CI” on your machines to avoid conflict and escalation? The complexity has to live somewhere. It is always worth evaluating the optimum place for your organization at any given stage.
- camgunz 3y agoSure, of course. My point is just that engineering orgs are generally super conservative and thus consistently fall prey to sunk cost fallacies everywhere. In a very irritatingly bureaucratic sense, you have to have a process for improving your processes, or you end up making awful decisions just because they're the least amount of change, and that metric always wins.
- krick 3y agoWell, to be fair, pretty much each of these points apply to all other CI tools in existence.
- Smaug123 3y agoJenkins is, like, uniquely awful though. Aside from the sheer staggering number of bugs in its core functionality (I encounter on average maybe one bug per new pipeline I write), it is also configured in almost the worst language I've ever touched (considerably worse than Nix, for example, which is really saying something). "Blue Ocean is Incomplete and Unintuitive" is entirely correct - you have to drop back to the old interface for essentially everything, and also Blue Ocean has the absolutely unforgiveable behaviour that it responds to any ESC keypress, even if the OS has focus.
- theptip 3y ago> The head of SRE championed Gitlab CI. I resisted this idea because I, the relatively inexperienced manager of a nascent team, was daunted by the prospect of trying to supplant Jenkins, Github, and JIRA all at once. I think this is a really big scoping mistake. There is a clean glide path, and I have no idea why you'd think you need to replace Jira. You can use Gitlab Runners in Github (https://docs.gitlab.com/ee/user/project/integrations/github.html https://docs.gitlab.com/ee/user/project/integrations/github....). Then with your mirrored repo you can try out the Gitlab features and realize it's better to use Gitlab for your repo if you're aso using it for CI. Having switched back and forth over the years (with tens of engineers using the repos), it's really not a lot of work to migrate between Github and Gitlab. Finally, if you like Jira, just use that? Sure there are advantages to having everything in one place, but if you're already using Jira+Github then you'll get an equivalent experience with Jira+Gitlab. I think this is one of those things that if you had actually prototyped the migration, you would find that it's much easier than you thought it was going to be.
- ebiester 3y agoGitlab is 29/month. It’s hard to stomach 29 + the money for GitHub and Jira, depending on the features you need. (Minimum 11/mo)
- stickfigure 3y agoOP's company is paying for a dedicated CI team. Unless they have special needs, they probably aren't concerned about the CI hosting costs themselves.
- fullspectrumdev 3y agoI’ve worked at some places that were pretty huge, but the finance department absolutely fucking nickle and dime over everything. Even the smallest EC2 instances had to be accounted for with a business reason that they would accept, with audits happening every few days. Asking for a licence for something upfront was like pulling teeth.
- postalrat 3y agoThe best part about jekins is that it works. The worst part is that many people dont want to work with jenkins.
- jurassic 3y agoHaving worked with a huge Jenkins deployment at a large company somewhat recently (>10,000 jobs), I found it worked okay enough that company leadership never felt it was worth the pain if switching. But all the friction points added up to a system that was rarely touched by anyone who hadn’t learned where all the bodies were buried. Over time that meant as an engineering org we were underinvesting in CI; there was a lot of quality-oriented stuff beyond unit testing that never got implemented in CI because the typical dev was unaware of how to change it or fearful about trying to change it. The relative accessibility of Github Actions (and other config as code alternatives) transforms the average dev’s relationship to CI from consumer to owner/developer/maintainer, and I think that is extremely worthwhile even if these tools bring some new problems of their own.
- ryukoposting 3y agoAs someone who has only ever been involved in CI/CD at smaller companies, I take for granted that our pipelines are maintainable by the developers writing the code being tested/packaged. That just seems like a positive indicator for QA culture.
- flimsypremise 3y agoThe real failure here was the failure to recognize the sunk cost fallacy. My team attempted the pursue the "supported" path with Concourse for a recent project, and I watched them bang their heads for a week, realized it wasn't going well, set a meeting where we all discussed and decided to move to GitHub actions. Now everyone is happy and the project is back on track and I don't have a write a blog post like this one about Concourse.
- _whiteCaps_ 3y agoI really like Concourse but unless you've got someone who's fully drunk the koolaid, it can be hard to a team onboard. It's got some very strong opinions on how builds should be deterministic. "Why can't I just click re-run this failed job!?" If you're doing something that doesn't line up with Pivotal / Cloud Foundry workflows, you're probably going to have a bad time.
- deleted 3y ago[deleted]
- redserk 3y agoThere are a large number of quirks I’ve had with the Kubernetes runner on Jenkins causing flaky behavior, but when it works, it’s mostly fine after figuring out Groovy. The 1 in 50 time a job needs to be kicked off again due to some strange error is annoying though. More often than not, I end up creating a sandbox folder in Jenkins and cloning workflows to work on them by hand without having 90 commits of misc errors and typos. This has helped speed up some iteration. Sort of off topic from the article, I think where almost every single CI engine falls short is the extremely limited scope they cater to - that every job ever made must be solved with the cargo cult ideology where every CI job mustn’t ever have human input, everything must be perfectly automated CI. This is wonderful when it is possible to do, but sucks when it isn’t feasible. Jenkins’ strength is that it’s actually a workflow engine masquerading as a CI tool, and that gives a lot of flexibility for a number of bespoke processes. Do you want to wait for someone to click a button to grant permission to run a job because the test might spin up extremely expensive resources and you’d really rather someone baby sit it? Jenkins has you covered. Or maybe it just isn’t possible (or worth the time) for your team at a point in time to automate all input to a job, so spin up a Jenkins job that lets someone manually specify parameters. Maybe your flow can trigger a release of software but you want a human to come behind to do a manual once-over as a final check, then also capture who/why approved it. More CI tools need to accept the reality that not everyone can implement the perfect ideal of CI jobs. Some jobs will require a human step somewhere, be it custom input to a job or waiting for a response.
- Jnr 3y agoThe part about using Docker is wrong. You can use the docker plugin and just run any image you want, and then run the code inside that specified image. You can pass additional parameters to also run it within the context of any particular docker build stage. I use the same Dockerfile for production builds and for running all the CI related tasks, without any problems. And the same single pipeline is used for PR hooks and releases. Post also mentions Blue Ocean UI, but that one is deprecated and should not be used at all. Jenkins is far from perfect, but unless you can use Gitlab (and sadly you can not in every case), it is the most feature rich self-hosted tool out there. Just make sure you set it up with Jenkins Configuration as a Code plugin out of the box.
- deleted 3y ago[deleted]
- jamesu 3y agoI stayed away from jenkins for a long while because of its bad reputation, but after diving into it for a recent project I was surprisingly impressed how easy it was to set everything up (with the docker image at least), and it seemed that there was a solution for just about every convoluted build scenario I could come up with. In contrast, I've tried out other tools and encountered: confusing uis, too much focus on docker, broken interop with forges, poor documentation, too much yaml, ... I guess the flexibility with jenkins is probably both a blessing and a curse.
- chologrande 3y agoThe blog perfectly sums up the feelings of anyone who has worked deep in jenkins. It works, but goddamn it's hard not to hate. Dagger is the answer to jenkins woes. It's saved my sanity and made CI development tolerable again. * Real language support. No groovy, no yaml. * Debugging available in your native IDE tools. * Clear docs in any language of your choice. * Reproducible builds you can run locally. Just need docker. This all comes with the bonus that it can actually just consume GHA yaml with some of the tooling folks have made.
- doubloon 3y agoat my company this is all very easy. there is no version control of any kind and also no automated testing. solves a lot of problems.
- nathants 3y agocloud is so good now it’s hard to justify not doing something bespoke. ec2 spot is insanely cheaper than turnkey cicd, and better in almost every way. i’m delighted to pay 30% over infra cost for convenience, but not 500%. and it better actually be convenient, not just have a good landing page and sales team. this month i learned localzones have even better spot prices. losangeles-1 is half the spot price of us-west-2. for a runner, do something like this, but react to an http call instead of a s3 put[1]. for a web ui do something like this[2]. s3, lambda, and ec2 spot are a perfect fit for cicd and a lot more. 1. https://github.com/nathants/libaws/tree/91b1c27fc947e067ed4607848b1c5b5cf48c4636/examples/complex/s3-ec2 https://github.com/nathants/libaws/tree/91b1c27fc947e067ed46... 2. https://github.com/nathants/aws-exec/tree/e68769126b5aae0e3515797d0d8401b917f6e16a https://github.com/nathants/aws-exec/tree/e68769126b5aae0e35...
- awinter-py 3y ago> Your editor won’t syntax highlight the Bash inside Groovy. You can’t run “shellcheck” (or any sort of Linter) on the Bash inside the groovy. You can’t very easily execute your shell commands to test them. this lands, and not specifically for jenkins. CI generally is inscrutable and difficult to test every CI has its quirk github CI feels fundamentally worse than gitlab because job artifacts are native in gitlab -- this makes a million things easier. github's own action for attaching things to releases is deprecated. the upload-artifact action doesn't work in nektos/act (the only way to test locally)
- fireflash38 3y agoOne of the most frustrating things about gitlab ci is testing rules. There's just no way to do it without sending it live, especially for things that might be more rare (tags, releases, default branches, etc). Why that isn't a priority to let people see what the jobs/vars of a pipeline would be without filly executing it is beyond me.
- awinter-py 3y agoyeah, really need a repl for CI. have run into exactly the same issue, conditional cases are impossible to test ahead of time
- fireflash38 3y agoJenkins is amazing for one thing in particular, and I've not really seen it replicated well elsewhere (like gitlab ci): common job execution. Letting people run jobs on/against shared equipment, rather than having people run scripts locally. I wouldn't recommend it for general CI other than that. Think of it like a butler, able to do small jobs, not as your complete home assistant.
- AceJohnny2 3y agoWhat would people recommend for a self-hosted CI system in 2023? I'm currently leaning towards Jenkins, because neighboring teams are using it (with some pain due to their scale, but theirs is 10x-100x ours), and I have some ~10y-old experience with it. (Our builds are compiling for embedded targets, with a Makefile-based system)
- AceJohnny2 3y agoI've noticed the Avahi folks use Woodpecker CI [1], which I hadn't heard of before, because it supports priorities and pre-emption [2] [1] https://woodpecker-ci.org/ https://woodpecker-ci.org/ [2] https://social.treehouse.systems/@marcan/111258515328251078 https://social.treehouse.systems/@marcan/111258515328251078
- kbenson 3y agoBecause not every company wants to allow their source code to the cloud. Basically all the reasons organization self host their code repositories (and a bunch do) all apply.
- AceJohnny2 3y agoNote that I asked "what" do people recommend, not "why" do people recommend...
- danielheath 3y agoBuildkite is mixed-hosting; source code stays on your machines, on-prem if you like, but build logs / history / SCM triggers are hosted. Unclear whether that’s enough for your situation but it’s by far my favourite because of how it lets you store the CI pipeline in the repository (so changes are reviewed via normal channels etc).
- MilStdJunkie 3y agoYou can run a local gitlab instance, so I'd assume self-hosting as a rider on that. Unless I've been making some very bad mistakes. GitLab's not too bad. I had a Gradle pile I had been using but once I knocked out some old Java bits I wanted to sleek my "required tools" down a bit. I've integrated some OpenProject stuff into GL also. Maybe someday . . for "minimal" clients . . I can just bring in the GitLab and call it a day, so far as server stuff is concerned. That day is not today, however. Customers in my business want a lot - like, a LOT a lot - of project toys, and homebrewing those in GitLab is beyond my skills.
- thayne 3y agoThe complaint about a long feedback loop is true of every CI system I've used (jenkins, travis, gitlab, github actions). None of them really have a good way to test a pipeline locally. The best solution here is to put as much of the build logic as possible in scripts and your build system.
- dahart 3y agoI’ve long had a very similar argument for why commuting in the Bay Area* sucks: because it’s just tolerable enough that people keep doing it. It’s not unmanageable, but it drains your life little by little every day. * This, of course, applies to all long commutes everywhere, like how it applies not only to Jenkins, but lots of other software too… especially Jira.
- lmm 3y agoJenkins is the worst CI system except for all the others. Everything feels like a pile of hacks upon hacks, maintaining it is tedious and unrewarding, new functionality is always bolted on and undiscoverable. But every alternative implements the first 20% of features that provide 80% of the functionality and then never gets around to implementing the other 80% (and interestingly every attempt to "modernize" Jenkins, such as "Blue Ocean" or "Declarative Pipelines" as mentioned in the article, suffers from exactly the same problem). I wish there was a nicer tool that worked. But there isn't.
- shitcoder 3y agoI really resonate with this post after working with Jenkins for ages. In my experience I wasn't very happy with Bash in the end either and moved to Javascript for reliability, performance and tighter integration with the Cloud were deploying to. It also allowed us to run locally and avoid having to deploy to Jenkins to test. The discoverability issue is also a massive issue, declarative pipelines massively over promise its capability. Pipeline configuration was mostly still done in the UI because extensions had 0 documentation about how to declaratively define its configuration. Searching on the internet for the magic incantations as the author puts it is such a waste of time and I just compromised to get it running. I'm glad this post is out there to articulate these common challenges when you invest so much into Jenkins.
- johnnyAghands 3y agoWow, this title sums up in one sentence what we all hate about Jenkins. It's like the equivelent of timeouts being worse than fast fails.
- Sophistifunk 3y agoAs somebody who has spent his share of time interacting with Jenkins as a user, and messing around inside it as a developer, I am definitely biased. For or against, I'm not sure tho ;-) But the problem with Jenkins is the problem with all CI. Our (software in general, not Jenkins specifically) build process are all godforsaken nightmares that would put HP Lovecraft and Rube Goldberg into rubber rooms. You can't add a bunch of extra automation to any of these systems and get something better, let alone a system that needs to be able to automate so many different and unique snowflake horrors.
- eternityforest 3y agoMaybe it's time for languages themselves to start defining what CI means, the way they currently define how builds and packages work. They already include test frameworks. It would be cool to have, at least as an experiment, a language with no snowflake horrors at all, if it runs at all, it's in the language or framework's One True Way All Tools Understand. Static typing, maybe even linter warnings as fatal errors, anything that makes adding CI a hassle just doesn't exist. Maybe it could even refuse to run without full test coverage. So you could take any project at all, click "Turn on CI" and that's that, there's only one CI config that exists. I'm guessing the number of use cases would be tiny to nonexistant in the startup centric tech world, but they might be some of the least stressful projects ever made!
- Sophistifunk 3y agoThe problem is that a real green-field re-imagination of software development needs to go back to the 70s, and back to the (these days mostly fake) metal, meaning it's a massive job that will take years to even know if you might make it. You need to start with something simple-but-useless, and grow it from there, rather than attacking a single layer or two of our current enormous tower of babel. It will be difficult to adopt because it must by definition be alien to our current workflows, and it will take forever to find users. The only way to find success is to slowly and surely build something that is so obviously good existing devs will be willing to give up tools in which they've invested years and (for many of them) their identities, just to play with it. It requires an iron clad vision, willingness to be a pariah among the coding masses and a dictator internally, excellent communication in order to present the vision and progress to the outsiders who are interested, and piles and piles of money to pay for it for years before it becomes sustainable, however you plan to make it so.
- throwaway2037 3y agoThis is my favourite quote: The worst thing about Jenkins is that it works. It can meet your needs. With a liiittle more effort, or by adopting sliiiightly lower standards, or with a liiiiitle more tolerance for pain, you can always get Jenkins to do aaaaalmost what you want it to. This is great, opinionated writing!
- hoseja 3y ago>The head of SRE championed Gitlab CI. I resisted this Well there's your problem right there.
- hwc 3y agoI agree with most of this. I end up with a lot of sh 'make all' sh 'make test' In my `Jenkinsfile`, but there's not a great way to test things locally.
- baud147258 3y agoWe're using Jenkins at work. For building artifacts, it's alright as our use case is simple. But then we're also using for our automated tests, which were alredy kind of a dumpster fire in terms of reliability (both because of mistakes made when automated testing was put into place and because of issues inherent to our product) and I don't think Jenkins is helping a lot here, even after consequent investment in dev time.
- mdwagner 3y agoI agree, Jenkins is awful. We also started with Bash scripts, but eventually moved to using Ansible playbooks, which is much nicer than Groovy. Alternatively, I recommend using Earthly as well.