5 ms·

GitHub's CI pipelines are behind GitLab? My lord, here I was thinking about moving to GitHub Enterprise when our subscription expires because everything in GitL

GitHub's CI pipelines are behind GitLab? My lord, here I was thinking about moving to GitHub Enterprise when our subscription expires because everything in GitLab feels like an unfinished weekend project. Seems like nobody can get anywhere near the functionality of Jenkins.

mooreds 5y ago
> Seems like nobody can get anywhere near the functionality of Jenkins. I've heard Jenkins is a nightmare to administer (not sure about the new versions). Seems like an opportunity: the power of Jenkins with modern sensibility.
- cyberpunk 5y ago
  Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept. The servers themselves are quite simple (single .jar, etc). These days we have containers, and it's made CI a much better thing.
  - mikepurvis 5y ago
    I agree with this. The quality of your Jenkins experience will be pretty much inversely proportional to how much groovy script is in your Jenkinsfiles. Every new permutation of parameters is a new chance for something to go wrong, and there's basically no sane way to unit test or validate any of your "helper" functions, so they're all write-once-change-never. Which is not necessarily a problem inherent in Jenkins— Jenkins just gives you more than enough rope to hang yourself. GitLab CI can get silly too, but culturally it inherits from the much simpler Travis model, where you're basically expected to just be wrapping some other build tool (eg, tox from the Python world) rather than rolling the whole thing yourself.
    - outworlder 5y ago
      Groovy is arguably the wrong language choice, even if it worked fine in other aspects. The string handling situation alone is crazy, which is compounded if you are using it with bash (which has its own crazy ideas about string escaping). That's a not too uncommon scenario. You end up with this: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e412...
      - mikepurvis 5y ago
        To be fair, basically everything that interops/embeds bash like this ends up in a similar place— you see the same double-escape ridiculousness in CMakelists files and Nix package definitions.
  - outworlder 5y ago
    > Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept Yes. One thousand times this. If you are running simple scripts it's fine. If you are using the declarative pipeline, it's fine. The moment you start adding Groovy you'll be down a path that is filled with sadness and anger. Mind you, even the folks behind Jenkins will advise you not to use any complex Groovy scripts(including for performance reasons - you can easily overwork the jenkins master). I've been focusing on Concourse because it forces the usage of containers for everything. You don't have to care about what's installed in the worker node, you just use a container that has the stuff you want. Simple inputs and outputs. You _can_ do the same sort of thing with Jenkins (but be aware of all the bugs still open regarding containers). But Jenkins doesn't force you to do anything, nor it gives you an easy and out of the box solution to string containers together. Left unchecked, you have your reproducible builds running in completely unreproduceable magical build machines – that noone really understands how it all works. Did a migration of a few hundreds of pipelines from one server to another and it exposed a lot of dependencies we didn't know we had. Plugins, jars, packages installed in build machines, you name it. If you must use Jenkins, please try to avoid Groovy to the max, write anything that's non-trivial as an executable (even if it is a bash script) and call it from the main pipeline. Use containers if you can to avoid build machine dependencies. Try to use declarative pipelines too unless your jobs are very simple, and avoid the 'script' blocks. Do not use the scripting pipeline to avoid inviting groovy to your home. You can thank me in a couple of years.
    - gscho 5y ago
      I worked on a Jenkins cluster with around 150 agents and many, many builds per day. It all ran so smoothly and I swear it was because there was zero groovy.
- specialist 5y ago
  Jenkins (and the like) are build script obfuscation frameworks.
- kerblang 5y ago
  Just to contradict the complaining about groovy scripts: I reduced something like 100 jobs to 2 using groovy in jenkins, and it has been a godsend. I'm not having any problems with it. Mind you, the Groovy stuff was implemented as an afterthought, and if you aren't handy with Groovy (I am) it can be challenging to learn. But compared to the horrible GUI-driven alternative, it was a no-brainer in the end. Really, the more you're doing CI, the more you want scripts - doesn't have to be Groovy. I'd honestly be perfectly fine just doing things with a bash shell on a vanilla linux install. I only use Jenkins because my team insists on it, for what? So we can put a web UI in front of CI. I'd even tend to agree that with supply-chain attacks being what they are, and jenkins being a never-ending fountain of security patches, putting a web UI in front of CI isn't worth it. But at least the groovy part makes life bearable.
skofgar 5y ago
And here I am planning on replacing Jenkins with GitLab CI.. do you really think Jenkins is better? Do you have any examples or thoughts you can share?
- orf 5y ago
  We did a talk on it, happy to answer any questions. It’s definitely worth it. A lot of our Jenkins issues came down to “we are not very good at managing Jenkins”, but even if we where the massive jump in productivity, flexibility and team independence we saw after rolling it out makes me believe it’s just better. https://m.youtube.com/watch?v=k83hTlb9pMc https://m.youtube.com/watch?v=k83hTlb9pMc
- cyberpunk 5y ago
  Your plan is solid (source: building ci stuff for >10 years). I wouldn't really suggest much else, although argo is getting pretty good if you're deploying to k8s, I'd still keep gitlab around for creating your images.
- vultour 5y ago
  Really the most egregious thing: Manual job runs. They're a first-class citizen in Jenkins, where it lets you make a rudimentary UI by having several types for each variable (e.g. a selectbox). In GitLab the manual run just gives you a list of defined variables and a small text input next to each. Then there's dependencies between jobs which has gotten better lately but again it seems less developed compared to Jenkins. I also managed to kill the web UI with some of my pipelines when using dependent jobs because the javascript fails somewhere deep when trying to visualise the pipeline and the page just stays blank. Honestly it really depends on what you're doing. If you're mostly replacing automated builds and tests triggered by git changes then GitLab works just fine.
pid-1 5y ago
I agree GitLab's CI has many edge cases. More importantly, it feels like they stopped making it better to focus on other stuff. That said, GH actions is really new and full of imitations / weird stuff as well. I still think GL offers more bang for the buck in 2021. Another option is using something external like Drone CI, Circle CI, etc... Some of those look nice, but I've never tested myself.
wmfiv 5y ago
You're looking for TeamCity, https://www.jetbrains.com/teamcity/ https://www.jetbrains.com/teamcity/.
cyberpunk 5y ago
Seriously, they're miles behind. It doesn't even really support Kubernetes, you have to have long running 'runner' servers ala Jenkins and you need to have all your build tools installed on those. It's miserable, fragile and I've dearly, dearly missed gitlab CI the last few times I've had to go anywhere near actions.
- easton 5y ago
  It’s probably because Actions is a fork of Azure Pipelines, which was originally designed to coordinate runners on Windows (where containers aren’t nearly as flexible or compatible as they are on Linux). Microsoft came up with some fancy internal magic to have hot pools of ephemeral VMs ready to assign in seconds, which nobody outside can really replicate (I don’t think), and then everyone sits in sadness while they try to debug why their tests run fine on “runner-1” but not on “runner-2”.
  - aliswe 5y ago
    some sources?
secondaryacct 5y ago
And jenkins is centuries behind Teamcity. My company extracted us from github+teamcity paradise to force fit us to gitlab, it's not going well. Try both and for real, build a real project pipeline and see if that works for your dev before moving to either.
- rossmohax 5y ago
  I never worked with Teamcity, but created a fairly complicated Gitlab CI pipelines. What do you miss in Gitlab CI?
swozey 5y ago
Github makes you do a ton of goofy side work to get simple things you'll expect in Gitlab/etc to work. Want to check out multiple repos? You'll need someone with org admin to create a service account and to manually pull each repo in a CI step with that service accounts token. This means tons of people are using their own PATs to get around "who is the github org admin?" sort of stuff. I could go on and on, Github Actions are horrid compared to Gitlab. I wish, wish wish I could go back but I don't have that power here. Github doesn't have scoped issues like Gitlab, the issue board is so lacking, "projects" is stale and featureless so many things. But I AM the one who deals with the aftermath. I've spent days fixing one simple github action and I have plenty I don't even wind up deploying they're so problematic.
- rossmohax 5y ago
  No idea why you've been downvoted. Gitlab CI is lightyears ahead of Github Actions, it's a fact.
  - deleted 5y ago
    [deleted]
Aeolun 5y ago
> feels like an unfinished weekend project Jenkins has felt like an unfinished weekend project for going on ten years now. It may once have been state of the art, but that was in a time when people barely knew what a xmlhttprequest was.

mooreds 5y ago

> Seems like nobody can get anywhere near the functionality of Jenkins. I've heard Jenkins is a nightmare to administer (not sure about the new versions). Seems like an opportunity: the power of Jenkins with modern sensibility.

cyberpunk 5y ago
Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept. The servers themselves are quite simple (single .jar, etc). These days we have containers, and it's made CI a much better thing.
- mikepurvis 5y ago
  I agree with this. The quality of your Jenkins experience will be pretty much inversely proportional to how much groovy script is in your Jenkinsfiles. Every new permutation of parameters is a new chance for something to go wrong, and there's basically no sane way to unit test or validate any of your "helper" functions, so they're all write-once-change-never. Which is not necessarily a problem inherent in Jenkins— Jenkins just gives you more than enough rope to hang yourself. GitLab CI can get silly too, but culturally it inherits from the much simpler Travis model, where you're basically expected to just be wrapping some other build tool (eg, tox from the Python world) rather than rolling the whole thing yourself.
  - outworlder 5y ago
    Groovy is arguably the wrong language choice, even if it worked fine in other aspects. The string handling situation alone is crazy, which is compounded if you are using it with bash (which has its own crazy ideas about string escaping). That's a not too uncommon scenario. You end up with this: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e412...
    - mikepurvis 5y ago
      To be fair, basically everything that interops/embeds bash like this ends up in a similar place— you see the same double-escape ridiculousness in CMakelists files and Nix package definitions.
- outworlder 5y ago
  > Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept Yes. One thousand times this. If you are running simple scripts it's fine. If you are using the declarative pipeline, it's fine. The moment you start adding Groovy you'll be down a path that is filled with sadness and anger. Mind you, even the folks behind Jenkins will advise you not to use any complex Groovy scripts(including for performance reasons - you can easily overwork the jenkins master). I've been focusing on Concourse because it forces the usage of containers for everything. You don't have to care about what's installed in the worker node, you just use a container that has the stuff you want. Simple inputs and outputs. You _can_ do the same sort of thing with Jenkins (but be aware of all the bugs still open regarding containers). But Jenkins doesn't force you to do anything, nor it gives you an easy and out of the box solution to string containers together. Left unchecked, you have your reproducible builds running in completely unreproduceable magical build machines – that noone really understands how it all works. Did a migration of a few hundreds of pipelines from one server to another and it exposed a lot of dependencies we didn't know we had. Plugins, jars, packages installed in build machines, you name it. If you must use Jenkins, please try to avoid Groovy to the max, write anything that's non-trivial as an executable (even if it is a bash script) and call it from the main pipeline. Use containers if you can to avoid build machine dependencies. Try to use declarative pipelines too unless your jobs are very simple, and avoid the 'script' blocks. Do not use the scripting pipeline to avoid inviting groovy to your home. You can thank me in a couple of years.
  - gscho 5y ago
    I worked on a Jenkins cluster with around 150 agents and many, many builds per day. It all ran so smoothly and I swear it was because there was zero groovy.
specialist 5y ago
Jenkins (and the like) are build script obfuscation frameworks.
kerblang 5y ago
Just to contradict the complaining about groovy scripts: I reduced something like 100 jobs to 2 using groovy in jenkins, and it has been a godsend. I'm not having any problems with it. Mind you, the Groovy stuff was implemented as an afterthought, and if you aren't handy with Groovy (I am) it can be challenging to learn. But compared to the horrible GUI-driven alternative, it was a no-brainer in the end. Really, the more you're doing CI, the more you want scripts - doesn't have to be Groovy. I'd honestly be perfectly fine just doing things with a bash shell on a vanilla linux install. I only use Jenkins because my team insists on it, for what? So we can put a web UI in front of CI. I'd even tend to agree that with supply-chain attacks being what they are, and jenkins being a never-ending fountain of security patches, putting a web UI in front of CI isn't worth it. But at least the groovy part makes life bearable.

skofgar 5y ago

And here I am planning on replacing Jenkins with GitLab CI.. do you really think Jenkins is better? Do you have any examples or thoughts you can share?

orf 5y ago
We did a talk on it, happy to answer any questions. It’s definitely worth it. A lot of our Jenkins issues came down to “we are not very good at managing Jenkins”, but even if we where the massive jump in productivity, flexibility and team independence we saw after rolling it out makes me believe it’s just better. https://m.youtube.com/watch?v=k83hTlb9pMc https://m.youtube.com/watch?v=k83hTlb9pMc
cyberpunk 5y ago
Your plan is solid (source: building ci stuff for >10 years). I wouldn't really suggest much else, although argo is getting pretty good if you're deploying to k8s, I'd still keep gitlab around for creating your images.
vultour 5y ago
Really the most egregious thing: Manual job runs. They're a first-class citizen in Jenkins, where it lets you make a rudimentary UI by having several types for each variable (e.g. a selectbox). In GitLab the manual run just gives you a list of defined variables and a small text input next to each. Then there's dependencies between jobs which has gotten better lately but again it seems less developed compared to Jenkins. I also managed to kill the web UI with some of my pipelines when using dependent jobs because the javascript fails somewhere deep when trying to visualise the pipeline and the page just stays blank. Honestly it really depends on what you're doing. If you're mostly replacing automated builds and tests triggered by git changes then GitLab works just fine.

pid-1 5y ago

I agree GitLab's CI has many edge cases. More importantly, it feels like they stopped making it better to focus on other stuff. That said, GH actions is really new and full of imitations / weird stuff as well. I still think GL offers more bang for the buck in 2021. Another option is using something external like Drone CI, Circle CI, etc... Some of those look nice, but I've never tested myself.

wmfiv 5y ago

You're looking for TeamCity, https://www.jetbrains.com/teamcity/ https://www.jetbrains.com/teamcity/.

cyberpunk 5y ago

Seriously, they're miles behind. It doesn't even really support Kubernetes, you have to have long running 'runner' servers ala Jenkins and you need to have all your build tools installed on those. It's miserable, fragile and I've dearly, dearly missed gitlab CI the last few times I've had to go anywhere near actions.

easton 5y ago
It’s probably because Actions is a fork of Azure Pipelines, which was originally designed to coordinate runners on Windows (where containers aren’t nearly as flexible or compatible as they are on Linux). Microsoft came up with some fancy internal magic to have hot pools of ephemeral VMs ready to assign in seconds, which nobody outside can really replicate (I don’t think), and then everyone sits in sadness while they try to debug why their tests run fine on “runner-1” but not on “runner-2”.
- aliswe 5y ago
  some sources?

secondaryacct 5y ago

And jenkins is centuries behind Teamcity. My company extracted us from github+teamcity paradise to force fit us to gitlab, it's not going well. Try both and for real, build a real project pipeline and see if that works for your dev before moving to either.

rossmohax 5y ago
I never worked with Teamcity, but created a fairly complicated Gitlab CI pipelines. What do you miss in Gitlab CI?

swozey 5y ago

Github makes you do a ton of goofy side work to get simple things you'll expect in Gitlab/etc to work. Want to check out multiple repos? You'll need someone with org admin to create a service account and to manually pull each repo in a CI step with that service accounts token. This means tons of people are using their own PATs to get around "who is the github org admin?" sort of stuff. I could go on and on, Github Actions are horrid compared to Gitlab. I wish, wish wish I could go back but I don't have that power here. Github doesn't have scoped issues like Gitlab, the issue board is so lacking, "projects" is stale and featureless so many things. But I AM the one who deals with the aftermath. I've spent days fixing one simple github action and I have plenty I don't even wind up deploying they're so problematic.

rossmohax 5y ago
No idea why you've been downvoted. Gitlab CI is lightyears ahead of Github Actions, it's a fact.
- deleted 5y ago
  [deleted]

Aeolun 5y ago

> feels like an unfinished weekend project Jenkins has felt like an unfinished weekend project for going on ten years now. It may once have been state of the art, but that was in a time when people barely knew what a xmlhttprequest was.