3 ms·

Ask HN: What challenges did you face once you gave devs access to Jenkins jobs?

5 points by shtubi 5y ago

Hey, I am a DevOps engineer at a medium-sized company. Our team is super busy and we have a lot on our plate. Recently we decided to see where we can improve our productivity. One of our main responsibilities is to provide dev teams with different services they require (usually they open a ticket, sending a slack msg, etc…). As part of our efforts to improve productivity, we recently decided to assemble the most common popular “services” provided to devs and automate them using Jenkins. In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs. On one hand, it does allow devs to act on their own, on the other hand, I get a lot of backfires from different personas... Devs * Why did this job fail? * Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created" * Devs make mistakes and deploy to the wrong environment. * Why did my dev environment get terminated? * "Update me if you modify parameters" * etc... Team lead * Let’s add a manual approval for these jobs * Make sure we have an audit log for every dev operation * Make sure we devs are not spinning to many machines * The developer team leader asked BI about these Jenkins Jobs because it became critical for developers' day-to-day. * etc… Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins :) Do you have any suggestions? How do you manage it within your company? Thanks!

boguyona 5y ago
We also were in this phase. So first, we searched for some other tool to expose scripts, and we even tried Rundeck, but then we just built a custom "DevOps Portal" that made sense for our requirements.
iwwr 5y ago
It looks like the challenge is tracking cloud resources created as part of a CI infrastructure and not Jenkins itself. Jenkins can be nothing but a scheduling and execution engine, not quite a self-service platform for business workflows. It may help more to define more precisely a few scenarios where you experience problems, so people can be more specific with suggestions. Lacking more specific information, you might consider putting automatic self destruct / deprovisioning for created cloud resources, e.g. 7 days, so you don't worry as much about what is created. What also helps is idempotency in these scripts, to just tell developers to retry things if there are transient errors.
- shtubi 5y ago
  Thanks for the comprehensive answer. I did apply a destruct mechanism, but many times devs need the resource for longer or shorted periods, which makes it hard for them & me to manage it :/
  - iwwr 5y ago
    How about to ask developers to provide a job parameter DURATION, and some suitable default like 0.5 days?
- boguyona 5y ago
  "Jenkins can be nothing but a scheduling and execution engine, not quite a self-service platform for business workflows." -you are totally right. Do you have suggestions for a tool for a "self-service platform for business workflows"?
  - iwwr 5y ago
    No idea as of yet. There are some better ways to represent workflows so they are more easily composable and/or visible, there could be a startup in there :) Some things tried: totally custom solution (most flexible), GitLab (unlike Jenkins, you can define a pipeline as a graph), BPMN (pretty heavy Java solution, you still need to build a frontend, there are a lot of tooling but kind of big overhead and low flexibility), Airflow (good visualisation, good for short duration workflows, there is some infrastructure overhead).