3 ms·

I think there are two aspects to arbitrary builds: 1. Periodic jobs (not linked to a pushed commit) 2. Not linked to a git repo At GitLab we try to have 'inf

by sytse 10y ago

I think there are two aspects to arbitrary builds: 1. Periodic jobs (not linked to a pushed commit) 2. Not linked to a git repo At GitLab we try to have 'infrastructure as code'. We think builds should be under version control to allow for collaboration. But periodic jobs make a lot of sense and it is discussed in https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 https://gitlab.com/gitlab-org/gitlab-ce/issues/2989

jacques_chester 10y ago
I think one thing GitlabCI lost in taking inspiration from Concourse (amongst other tools!) was the centrality of resources. It wasn't obvious at first why this is so important, I and most others fixated on the other visible differences (containers for everything, declarative config, pipeline view etc). There are a lot of features that don't need to be added to the core because they can be resources instead. For git-triggered builds, I use the git resource. Periodic, I use the time resource or a cron resource. S3 triggered, I use the S3 resource. Triggered on release of new software to PivNet, I use pivnet-resource. Triggered on a new docker image, I use docker image resource. And so on. Any resource that has a 'get' operation can trigger a build. 3rd parties can add triggers without needing to avoid interference with other resources, because every operation is isolated. Disclosure: I work for Pivotal, which sponsors Concourse.
- sytse 10y ago
  Concourse is great and it was certainly an inspiration for GitLab CI. What I liked most and what we're still working on are cross project triggers https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 The use of resources sounds cool.
swozey 10y ago
The lack of arbitrary jobs in Gitlab is one of my huge pet peeves regarding it. I've used Gitlab-ee for a year now and really enjoy it for CI/CD stuff. But sometimes I want to throw up a quick shell job that is something super simple, like backing up a database, cleaning up docker images, things like that. I'm used to storing these in Jenkins/Bamboo so that they aren't left hidden on a server in a cron job. I can do this by setting a manual build that I just run occasionally, but what if I forget to run it for months? In Jenkins I could say "run this job for no reason whatsoever other than the fact that 30 days has gone since it last ran." This IMO is a huge use case for Ops. At this point I'll likely need to run Gitlab and Jenkins which I really don't want to have to do.
- sytse 10y ago
  Thanks for being a customer of GitLab EE and great to hear you enjoy the CD/CD stuff. Sounds like you really need the possibility to run job with a periodic interval. Does https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 that I mentioned before address all you need? Chatops sounds cool, it would be nice to have a slash command to trigger a build. I've created a feature proposal for it https://gitlab.com/gitlab-org/gitlab-ce/issues/25866 https://gitlab.com/gitlab-org/gitlab-ce/issues/25866
  - swozey 10y ago
    Periodic jobs is great for overnight regression testing, things of that nature and the few examples I gave above. That'll be a great (re)addition. But still, the ability to create random jobs like in Jenkins without having to create a repo, etc would be a huge plus along with easing ops teams into performing server maintenance via CI.
    - sytse 10y ago
      If you use one 'catchall' repo for your random jobs you would only have to modify a file instead of creating a repo when adding a new job. Do you think this is acceptable considering the added convenience of having the job under version control?
      - swozey 10y ago
        So perhaps something like an infrastructure/tasks repo, then make various "environments" for assigning different env variables (like server IPs, etc). I'll fiddle around with this and see if its viable. I just really don't want to have to create a project/repo for every single little tiny task/report I want automatically done. I like to run various reports, for example run a report monthly and diff my AWS/GCE firewall rules to show if anything that shouldn't have changed has changed then email if the there is a diff. Same with instance lists, load balancers, things of that nature. Then all of the server stuff, hourly snapshots using aws-cli or gcloud, yaddayadda. Once the periodical thing is back in I can give that a shot. I've only used the travis style builds for about a year so I'm not very advanced with them. I think when I can do a when: monthly that will help greatly. Re: ChatOps I've upvoted the ticket. I'd LOVE it if there was a slack: step that Gitlab could use that went beyond the checkboxes under integrations> - for instance I'd like the ability to echo whatever under a Slack step to a channel, for instance. I can do this buy making my own step, running a slack container and doing webhooks but that's clunkier than I'd like, especially when the GItlab server already has my slack tokens/etc. Right now I need to run a container that pulls that info to webhook to slack. slack: if: successful channel: deploys echo: "$buildtag has been deployed, you can access this build at $server1 and $server2" curl: #posttodatadog "$buildtag released on $date"
merb 10y ago
well I know you have great idea's but sadly a lot of the focus of gitlab at the moment is either monetizing with EE (which I can understand, I need to develop for my income aswell..) and the other with adding features to CE (which of course leads to the first). Sadly this does not fix bugs. And the issue tracker of gitlab-ce grows and grows and the CI is not really stable. The new permission model was great, but break stuff. Unfortunatly at the moment me and my team is discussing to use gitlab + jenkins instead of gitlab + gitlab for vcs/ci. just because webhooks + jenkins is way more stable than the gitlab way (and actually webhooks are push based, while gitlab ci polls.) This is sad since gitlab ci was quick and easy, while even the new dsl for jenkins is way harder to get right. but we actually can't retry half of our builds. Another way would be using the shell executor where we actually loose some functionality. (we use the docker executor). I know your vision (and I think some ideas are great), but sadly I think too much centralization is harmful after going with gitlab since v8, somethings does not need to be reinvented by gitlab (they might just need a improvement).
- dimitrie 10y ago
  Thank you for your thoughts regarding your use of GitLab CI. As UX Designer at the gitlab CI team I can say that in the upcoming release there will be a focus on making the system more robust and fixing bugs. Stability and dependability are certainly a priority and are increasingly so. It's about finding the right balance between focussing our scope and broadening it. My last note is on your mention of the growing issue tracker. The issue tracker is not only about defining bugs and problems, but also about idea exploration. To my knowledge the more GitLab will be known, the more it will grow. It is everyones best interest to let it flourish, but in an organised way. We are hard at work in that area as well! For example, we have dedicated Issue Triage Specialists which keep track and answer a lot of the newly created ones.
- sytse 10y ago
  We spend a lot of development time fixing bugs. Over the last year GitLab has gotten a lot more stable. Of course it is not perfect and the rapid addition of new features creates new bugs. The issue tracker is growing because we also keep feature proposals there. GitLab CI doesn't poll the repository. The GitLab Runner does poll GitLab for new jobs. We're currently working on making sure that GitLab can deal with those requests more efficiently. Polling makes setting up Runners a lot easier for our users. Not being able to retry half of your builds sounds very bad. Please contact support@ our domain and add a link to this comment to receive help with that. It was a big thing to add CI to GitLab and we had a lot of concerns about it before we did so. But after doing it the improvements in the interface, CD functionality, Pages, review apps, auto deploy, and many other things convinced us that this is the right path. GitLab will work with other tools but out of the box it will offer you a great experience. For the benefits of that please see https://about.gitlab.com/2016/11/14/idea-to-production/ https://about.gitlab.com/2016/11/14/idea-to-production/

by sytse 10y ago

jacques_chester 10y ago

I think one thing GitlabCI lost in taking inspiration from Concourse (amongst other tools!) was the centrality of resources. It wasn't obvious at first why this is so important, I and most others fixated on the other visible differences (containers for everything, declarative config, pipeline view etc). There are a lot of features that don't need to be added to the core because they can be resources instead. For git-triggered builds, I use the git resource. Periodic, I use the time resource or a cron resource. S3 triggered, I use the S3 resource. Triggered on release of new software to PivNet, I use pivnet-resource. Triggered on a new docker image, I use docker image resource. And so on. Any resource that has a 'get' operation can trigger a build. 3rd parties can add triggers without needing to avoid interference with other resources, because every operation is isolated. Disclosure: I work for Pivotal, which sponsors Concourse.

sytse 10y ago
Concourse is great and it was certainly an inspiration for GitLab CI. What I liked most and what we're still working on are cross project triggers https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 The use of resources sounds cool.

swozey 10y ago

The lack of arbitrary jobs in Gitlab is one of my huge pet peeves regarding it. I've used Gitlab-ee for a year now and really enjoy it for CI/CD stuff. But sometimes I want to throw up a quick shell job that is something super simple, like backing up a database, cleaning up docker images, things like that. I'm used to storing these in Jenkins/Bamboo so that they aren't left hidden on a server in a cron job. I can do this by setting a manual build that I just run occasionally, but what if I forget to run it for months? In Jenkins I could say "run this job for no reason whatsoever other than the fact that 30 days has gone since it last ran." This IMO is a huge use case for Ops. At this point I'll likely need to run Gitlab and Jenkins which I really don't want to have to do.

sytse 10y ago
Thanks for being a customer of GitLab EE and great to hear you enjoy the CD/CD stuff. Sounds like you really need the possibility to run job with a periodic interval. Does https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 that I mentioned before address all you need? Chatops sounds cool, it would be nice to have a slash command to trigger a build. I've created a feature proposal for it https://gitlab.com/gitlab-org/gitlab-ce/issues/25866 https://gitlab.com/gitlab-org/gitlab-ce/issues/25866
- swozey 10y ago
  Periodic jobs is great for overnight regression testing, things of that nature and the few examples I gave above. That'll be a great (re)addition. But still, the ability to create random jobs like in Jenkins without having to create a repo, etc would be a huge plus along with easing ops teams into performing server maintenance via CI.
  - sytse 10y ago
    If you use one 'catchall' repo for your random jobs you would only have to modify a file instead of creating a repo when adding a new job. Do you think this is acceptable considering the added convenience of having the job under version control?
    - swozey 10y ago
      So perhaps something like an infrastructure/tasks repo, then make various "environments" for assigning different env variables (like server IPs, etc). I'll fiddle around with this and see if its viable. I just really don't want to have to create a project/repo for every single little tiny task/report I want automatically done. I like to run various reports, for example run a report monthly and diff my AWS/GCE firewall rules to show if anything that shouldn't have changed has changed then email if the there is a diff. Same with instance lists, load balancers, things of that nature. Then all of the server stuff, hourly snapshots using aws-cli or gcloud, yaddayadda. Once the periodical thing is back in I can give that a shot. I've only used the travis style builds for about a year so I'm not very advanced with them. I think when I can do a when: monthly that will help greatly. Re: ChatOps I've upvoted the ticket. I'd LOVE it if there was a slack: step that Gitlab could use that went beyond the checkboxes under integrations> - for instance I'd like the ability to echo whatever under a Slack step to a channel, for instance. I can do this buy making my own step, running a slack container and doing webhooks but that's clunkier than I'd like, especially when the GItlab server already has my slack tokens/etc. Right now I need to run a container that pulls that info to webhook to slack. slack: if: successful channel: deploys echo: "$buildtag has been deployed, you can access this build at $server1 and $server2" curl: #posttodatadog "$buildtag released on $date"

merb 10y ago

well I know you have great idea's but sadly a lot of the focus of gitlab at the moment is either monetizing with EE (which I can understand, I need to develop for my income aswell..) and the other with adding features to CE (which of course leads to the first). Sadly this does not fix bugs. And the issue tracker of gitlab-ce grows and grows and the CI is not really stable. The new permission model was great, but break stuff. Unfortunatly at the moment me and my team is discussing to use gitlab + jenkins instead of gitlab + gitlab for vcs/ci. just because webhooks + jenkins is way more stable than the gitlab way (and actually webhooks are push based, while gitlab ci polls.) This is sad since gitlab ci was quick and easy, while even the new dsl for jenkins is way harder to get right. but we actually can't retry half of our builds. Another way would be using the shell executor where we actually loose some functionality. (we use the docker executor). I know your vision (and I think some ideas are great), but sadly I think too much centralization is harmful after going with gitlab since v8, somethings does not need to be reinvented by gitlab (they might just need a improvement).

dimitrie 10y ago
Thank you for your thoughts regarding your use of GitLab CI. As UX Designer at the gitlab CI team I can say that in the upcoming release there will be a focus on making the system more robust and fixing bugs. Stability and dependability are certainly a priority and are increasingly so. It's about finding the right balance between focussing our scope and broadening it. My last note is on your mention of the growing issue tracker. The issue tracker is not only about defining bugs and problems, but also about idea exploration. To my knowledge the more GitLab will be known, the more it will grow. It is everyones best interest to let it flourish, but in an organised way. We are hard at work in that area as well! For example, we have dedicated Issue Triage Specialists which keep track and answer a lot of the newly created ones.
sytse 10y ago
We spend a lot of development time fixing bugs. Over the last year GitLab has gotten a lot more stable. Of course it is not perfect and the rapid addition of new features creates new bugs. The issue tracker is growing because we also keep feature proposals there. GitLab CI doesn't poll the repository. The GitLab Runner does poll GitLab for new jobs. We're currently working on making sure that GitLab can deal with those requests more efficiently. Polling makes setting up Runners a lot easier for our users. Not being able to retry half of your builds sounds very bad. Please contact support@ our domain and add a link to this comment to receive help with that. It was a big thing to add CI to GitLab and we had a lot of concerns about it before we did so. But after doing it the improvements in the interface, CD functionality, Pages, review apps, auto deploy, and many other things convinced us that this is the right path. GitLab will work with other tools but out of the box it will offer you a great experience. For the benefits of that please see https://about.gitlab.com/2016/11/14/idea-to-production/ https://about.gitlab.com/2016/11/14/idea-to-production/