5 ms·

Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural. People (on my team) don't understand pip

Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural. People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well. That's still an option with pipelines, but it feels much more discouraged (to e.g. write a shared pipeline job that runs a multi-line shell script). It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen (as opposed to 20 jobs that did the same thing for 200 repos). Unfortunately, it also seems like declarative pipeline is limited enough that we end up writing a lot of Groovy. I sorta get why Groovy became the official scripting language, but it's like pulling teeth getting people to learn even basic Groovy. I realize, again, that a lot of this is a cultural problem. Mostly just trying to give a counterpoint that pipelines don't always look as clean as the article implies, especially for orgs with a lot of repos (where committing the same 'test.sh' file to each repo doesn't make a lot of sense).

grogenaut 9y ago
Why not make it easy to reuse functions to do the steps the groovy is doing and build all of the common steps in a reusable way? Eg what the article is suggesting
- BlanketApple 9y ago
  Yeah, we've done that. And again, this is totally a cultural problem. But inevitably I end up writing that shared groovy code (because I have the most familiarity), and so when things go wrong, or it needs to be changed, I end up being the one having to debug it. Definitely a huge bus factor. It'd be nice if it was a slightly more common language (e.g. Python) so it wasn't such a pain.
  - grogenaut 9y ago
    Aah. Cultural problems are different. What's the main issue there? Making a Python language plugin wouldn't be hard though.
Someone 9y ago
”It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen” I never look at the main screen. You can create tabs that segment those 200 jobs. If your job names are consistent (as they should), you can do the segmenting by regular expression (I’m not sure whether creating tabs requires a plug-in) As to doing (almost) the same thing for different projects: you can POST XML job definitions to the web interface, so if you have the rights to create jobs and to run code on your local system, you can script job creation. In my experience, that’s the way to keep job definitions consistent. To figure out what the XML to POST should look like, grab it from your browser by GETting job/jobname/config.xml (https://support.cloudbees.com/hc/en-us/articles/218353308-How-to-update-job-config-files-using-the-REST-API-and-cURL- https://support.cloudbees.com/hc/en-us/articles/218353308-Ho...) Yes, that duplicates lots of stuff in the Jenkins job definitions, but you shouldn’t treat that as source, but as the output of your job creation scripts.
- tokenizerrr 9y ago
  If you get to that point, check out the Job DSL plugin.
  - Someone 9y ago
    I would have to convince the system administrator to install that plug-in. I also don’t like having to use yet another language (or two, counting Groovy and the Job DSL separately), but that, I could live with.
jonthepirate 9y ago
Rather than run monolithic Jenkins with a kagillion jobs in it that will easily bog down, you could consider running distributed Jenkins in which each team, or even each microservice, gets its own Jenkins Cluster with a small number of slaves and a small number of jobs. See my earlier post in this thread where I posted a YouTube video of the method I use for doing this using a CloudFormation template. Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.

grogenaut 9y ago

Why not make it easy to reuse functions to do the steps the groovy is doing and build all of the common steps in a reusable way? Eg what the article is suggesting

BlanketApple 9y ago
Yeah, we've done that. And again, this is totally a cultural problem. But inevitably I end up writing that shared groovy code (because I have the most familiarity), and so when things go wrong, or it needs to be changed, I end up being the one having to debug it. Definitely a huge bus factor. It'd be nice if it was a slightly more common language (e.g. Python) so it wasn't such a pain.
- grogenaut 9y ago
  Aah. Cultural problems are different. What's the main issue there? Making a Python language plugin wouldn't be hard though.

Someone 9y ago

”It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen” I never look at the main screen. You can create tabs that segment those 200 jobs. If your job names are consistent (as they should), you can do the segmenting by regular expression (I’m not sure whether creating tabs requires a plug-in) As to doing (almost) the same thing for different projects: you can POST XML job definitions to the web interface, so if you have the rights to create jobs and to run code on your local system, you can script job creation. In my experience, that’s the way to keep job definitions consistent. To figure out what the XML to POST should look like, grab it from your browser by GETting job/jobname/config.xml (https://support.cloudbees.com/hc/en-us/articles/218353308-How-to-update-job-config-files-using-the-REST-API-and-cURL- https://support.cloudbees.com/hc/en-us/articles/218353308-Ho...) Yes, that duplicates lots of stuff in the Jenkins job definitions, but you shouldn’t treat that as source, but as the output of your job creation scripts.

tokenizerrr 9y ago
If you get to that point, check out the Job DSL plugin.
- Someone 9y ago
  I would have to convince the system administrator to install that plug-in. I also don’t like having to use yet another language (or two, counting Groovy and the Job DSL separately), but that, I could live with.

jonthepirate 9y ago

Rather than run monolithic Jenkins with a kagillion jobs in it that will easily bog down, you could consider running distributed Jenkins in which each team, or even each microservice, gets its own Jenkins Cluster with a small number of slaves and a small number of jobs. See my earlier post in this thread where I posted a YouTube video of the method I use for doing this using a CloudFormation template. Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.