5 ms·

I know everyone hates on Jenkins these days, but (at least for the workloads I am building) Concourse feels like a toy. My issues: - everything has to be in a

by mugsie 9y ago

I know everyone hates on Jenkins these days, but (at least for the workloads I am building) Concourse feels like a toy. My issues: - everything has to be in a docker container (not all things can run in garden) - Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible - each install of concourse may need an entirely different version of fly - There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews) Jenkins + Jenkinsfiles + Pipelines provides all of the good bits of Concourse, with none of the Pivitol enforced workflow + tools. I get why CloudFoundry uses Concourse - the build process is so arcane that you basically need to run the full CI locally, but I really do not get the hype for other projects.

wlamartin 9y ago
> not all things can run in garden Can you provide an example of what you'd want to do but you aren't able to?
- mugsie 9y ago
  Booting VMs is kind of the main one.
  - wlamartin 9y ago
    Booting VMs on your worker seems like exactly the kind of debris concourse is encouraging the avoidance of by enforcing the containerisation of your builds. That said, I can't think of a reason why you couldn't boot a VM in a privileged Garden container, have you tried?
    - mugsie 9y ago
      Yeah - I can't remember the actual failure, but it did not work out well. We also need the VMs to survive across build steps. We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes
      - wlamartin 9y ago
        Ah that's a shame, I still suspect it would _probably_ work with the right tweaking but I appreciate there's a hurdle there that might be frustrating to overcome and that's no good. We also test a PaaS and require VM spin up but we call out to GCP/AWS as necessary.
      - smgoller 9y ago
        I second what wlamartin said. We have pipelines that do integration tests that spin up 30 instance clusters in AWS. Trying to spin up something similar inside a concourse worker just isn't worth trying.
      - jacques_chester 9y ago
        > We also need the VMs to survive across build steps. You might represent them as a pool resource, so they can be safely handed from job to job. A lot of teams do this for stuff where they want to share an expensive, long-lived, stateful resource. Disclosure: I work at Pivotal.
        mugsie 9y ago
        The VMs are per job - once the CI run tests that the orchestration has created the right system config on the VMs, and that basic integration tests work, we destroy the VMs. I have worked in two companies now with Cloud Foundry distros, and both of them have started using Concourse, and then moved to Jenkins (for non Cloud Foundry projects - I shudder to think what would happen to a distro that attempted to not use Concourse)
        jacques_chester 9y ago
        I'm still lost. You said at the top that you need the VMs to survive across build steps (I read that as jobs), but now they're per-job? The Concourse->Jenkins thing will definitely happen without better docs, examples and widespread understanding. Concourse is non-obvious to newbies and the error messages need a lot of love. Or any love. I don't quite understand the parenthetical, could you elaborate?
pokemongoaway 9y ago
Not everyone... Jenkins is probably the most versatile! I'm still a heavy user and enjoy using the official docker image these days :)
gabrielr 9y ago
> Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible Have you tried `fly watch`? It will stream logs from in-flight builds, or show the full output of completed ones.
- mugsie 9y ago
  sure - I can use fly. if I got to the page, find the link to download the right version of fly, then remember the archine incantation to login, find a job, and find the logs. Or - I can go to my Jenkins build page, and click "Show full log". For me, I find the initial quick searching through logs much quicker in a browser. And, I have found that on longer builds the web UIs log streaming can actually crash the browser.
  - jadeklerk 9y ago
    The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.
    - mugsie 9y ago
      Well, to be clear, it's fly login --team-name internal --concourse-url https://<url> https://<url> -k (I think) Vs just going to the jenkins UI.
      - jadeklerk 9y ago
        It's fly login -t <my target> You can save targets. Also, if you want logs to appear somewhere you can pretty easily set up a after-fail job to ftp em somewhere.
joncampbelldev 9y ago
For a lightweight pipeline based CI tool (with no docker requirements) you may like buildkite. I use it for almost everything I used to do in jenkins, although I still keep jenkins around as a glorified cron job / easy generic task runner. (not associated with buildkite, just a very happy customer)
- mugsie 9y ago
  buildkite looks nice - but on premise is a current requirement, as not all our repos are on the internet. Looks like a cool product though
  - joncampbelldev 9y ago
    buildkite can be done entirely on your own infrastructure (for the purposes of source control and builds), its an interesting setup: - the ui is SaaS, but it doesn't care about your repos or build agents - the build agent runs on your machines (they communicate outwards only with the SaaS product to decide if they should start building) - all pipeline config is kept inside your repo - you can hook up any source control to connect to buildkites webhooks, they just happen to have an integration to the common SaaS source control tools (github etc) A nice extra is that running your own build agents lets you keep costs down. If you are on AWS they have a "one-click" setup for a cloud formation template that gives you an auto-scaling build environment
samueloph 9y ago
Jenkins has some serious problems when you want to work with CD. Its currently impossible* to use Jenkins in a CD environment where you want to deploy by tagging your git project[1] and allow rollbacks because Jenkins doesn't treat tag's hashes (only the commits the tag points to). Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. It may look as a silly usecase, but i believe its one that mostly fits when you have a dynamic server farm (hosting on aws with autoscalling, for example) and don't want to auto-deploy on master commit. [1]https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/xbX9-xM9BQAJ https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/... edit-> * aktually its not impossible, but pretty hacky and ugly, it would be nice to be able to rely on jenkins own tools for this.
- mugsie 9y ago
  > Jenkins has some serious problems when you want to work with CD. When you want to work with CD in a particular way. A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged. If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side. > Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item.
  - samueloph 9y ago
    >> Jenkins has some serious problems when you want to work with CD. > When you want to work with CD in a particular way. > A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged. > If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side. Well, i must confess i agree with you and my first idea was this, i will probably end up pushing for that anyway, but i will have to convince some devs that they should not be afraid of git reverts. When i first mentioned the idea, some devs with deploy rights said they were skeptical about this because they were afraid of having a git revert shitstorm in a SOS situation. >> Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. > Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item. Not exactly, on this case github only lets me create webhooks with 'create' action, which is trigerred "Any time a Branch or Tag is created.", so new branches would trigger deploys.
jadeklerk 9y ago
> - There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews) A CI that runs one job (test) for lots of branches is differnet than a multi-pipeline CI that runs lots of jobs for a single branch (which is what concourse aims to be).
- mugsie 9y ago
  Sure, but if a job fails for a transient reason, there should be a way to retrigger it - that is just a basic feature of nearly all CIs, no matter how many jobs there is.
  - jadeklerk 9y ago
    Of course, and you can - you can manually hit the + button to retrigger a build, and if you want specific inputs you can select the exact one you want in the resources tab (which is slightly less common, since you probably haven't been committing since your build broke, but it's totally doable).
jadeklerk 9y ago
> - Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible What? It's just STDOUT.. Are your logs going to a file or something?
- mugsie 9y ago
  Well, yeah. Different stages go to different log files, so you don't end up with a 10k log file.

by mugsie 9y ago

wlamartin 9y ago

> not all things can run in garden Can you provide an example of what you'd want to do but you aren't able to?

mugsie 9y ago
Booting VMs is kind of the main one.
- wlamartin 9y ago
  Booting VMs on your worker seems like exactly the kind of debris concourse is encouraging the avoidance of by enforcing the containerisation of your builds. That said, I can't think of a reason why you couldn't boot a VM in a privileged Garden container, have you tried?
  - mugsie 9y ago
    Yeah - I can't remember the actual failure, but it did not work out well. We also need the VMs to survive across build steps. We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes
    - wlamartin 9y ago
      Ah that's a shame, I still suspect it would _probably_ work with the right tweaking but I appreciate there's a hurdle there that might be frustrating to overcome and that's no good. We also test a PaaS and require VM spin up but we call out to GCP/AWS as necessary.
    - smgoller 9y ago
      I second what wlamartin said. We have pipelines that do integration tests that spin up 30 instance clusters in AWS. Trying to spin up something similar inside a concourse worker just isn't worth trying.
    - jacques_chester 9y ago
      > We also need the VMs to survive across build steps. You might represent them as a pool resource, so they can be safely handed from job to job. A lot of teams do this for stuff where they want to share an expensive, long-lived, stateful resource. Disclosure: I work at Pivotal.
      - mugsie 9y ago
        The VMs are per job - once the CI run tests that the orchestration has created the right system config on the VMs, and that basic integration tests work, we destroy the VMs. I have worked in two companies now with Cloud Foundry distros, and both of them have started using Concourse, and then moved to Jenkins (for non Cloud Foundry projects - I shudder to think what would happen to a distro that attempted to not use Concourse)
        jacques_chester 9y ago
        I'm still lost. You said at the top that you need the VMs to survive across build steps (I read that as jobs), but now they're per-job? The Concourse->Jenkins thing will definitely happen without better docs, examples and widespread understanding. Concourse is non-obvious to newbies and the error messages need a lot of love. Or any love. I don't quite understand the parenthetical, could you elaborate?

pokemongoaway 9y ago

Not everyone... Jenkins is probably the most versatile! I'm still a heavy user and enjoy using the official docker image these days :)

gabrielr 9y ago

> Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible Have you tried `fly watch`? It will stream logs from in-flight builds, or show the full output of completed ones.

mugsie 9y ago
sure - I can use fly. if I got to the page, find the link to download the right version of fly, then remember the archine incantation to login, find a job, and find the logs. Or - I can go to my Jenkins build page, and click "Show full log". For me, I find the initial quick searching through logs much quicker in a browser. And, I have found that on longer builds the web UIs log streaming can actually crash the browser.
- jadeklerk 9y ago
  The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.
  - mugsie 9y ago
    Well, to be clear, it's fly login --team-name internal --concourse-url https://<url> https://<url> -k (I think) Vs just going to the jenkins UI.
    - jadeklerk 9y ago
      It's fly login -t <my target> You can save targets. Also, if you want logs to appear somewhere you can pretty easily set up a after-fail job to ftp em somewhere.

joncampbelldev 9y ago

For a lightweight pipeline based CI tool (with no docker requirements) you may like buildkite. I use it for almost everything I used to do in jenkins, although I still keep jenkins around as a glorified cron job / easy generic task runner. (not associated with buildkite, just a very happy customer)

mugsie 9y ago
buildkite looks nice - but on premise is a current requirement, as not all our repos are on the internet. Looks like a cool product though
- joncampbelldev 9y ago
  buildkite can be done entirely on your own infrastructure (for the purposes of source control and builds), its an interesting setup: - the ui is SaaS, but it doesn't care about your repos or build agents - the build agent runs on your machines (they communicate outwards only with the SaaS product to decide if they should start building) - all pipeline config is kept inside your repo - you can hook up any source control to connect to buildkites webhooks, they just happen to have an integration to the common SaaS source control tools (github etc) A nice extra is that running your own build agents lets you keep costs down. If you are on AWS they have a "one-click" setup for a cloud formation template that gives you an auto-scaling build environment

samueloph 9y ago

Jenkins has some serious problems when you want to work with CD. Its currently impossible* to use Jenkins in a CD environment where you want to deploy by tagging your git project[1] and allow rollbacks because Jenkins doesn't treat tag's hashes (only the commits the tag points to). Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. It may look as a silly usecase, but i believe its one that mostly fits when you have a dynamic server farm (hosting on aws with autoscalling, for example) and don't want to auto-deploy on master commit. [1]https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/xbX9-xM9BQAJ https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/... edit-> * aktually its not impossible, but pretty hacky and ugly, it would be nice to be able to rely on jenkins own tools for this.

mugsie 9y ago
> Jenkins has some serious problems when you want to work with CD. When you want to work with CD in a particular way. A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged. If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side. > Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item.
- samueloph 9y ago
  >> Jenkins has some serious problems when you want to work with CD. > When you want to work with CD in a particular way. > A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged. > If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side. Well, i must confess i agree with you and my first idea was this, i will probably end up pushing for that anyway, but i will have to convince some devs that they should not be afraid of git reverts. When i first mentioned the idea, some devs with deploy rights said they were skeptical about this because they were afraid of having a git revert shitstorm in a SOS situation. >> Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it. > Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item. Not exactly, on this case github only lets me create webhooks with 'create' action, which is trigerred "Any time a Branch or Tag is created.", so new branches would trigger deploys.

jadeklerk 9y ago

> - There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews) A CI that runs one job (test) for lots of branches is differnet than a multi-pipeline CI that runs lots of jobs for a single branch (which is what concourse aims to be).

mugsie 9y ago
Sure, but if a job fails for a transient reason, there should be a way to retrigger it - that is just a basic feature of nearly all CIs, no matter how many jobs there is.
- jadeklerk 9y ago
  Of course, and you can - you can manually hit the + button to retrigger a build, and if you want specific inputs you can select the exact one you want in the resources tab (which is slightly less common, since you probably haven't been committing since your build broke, but it's totally doable).

jadeklerk 9y ago

> - Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible What? It's just STDOUT.. Are your logs going to a file or something?

mugsie 9y ago
Well, yeah. Different stages go to different log files, so you don't end up with a 10k log file.