5 ms·

Gitlab's CI/CD is amazing. Of all the systems I've seen, it's the easiest to (a) deploy your own runners easily, and (b) mix private and public runners (i.e. th

Gitlab's CI/CD is amazing. Of all the systems I've seen, it's the easiest to (a) deploy your own runners easily, and (b) mix private and public runners (i.e. the equivalent of using a free plan on Travis, with the benefit that for heavy-duty jobs you can run your own machines). That and first-class support for docker (no more being at the whims of what Travis chooses to deploy in their images), make it a really great system. But we had been on Github, and that turned into something of a pain. First we tried mirroring Github to Gitlab. This worked, but (a) it added a minute or two of latency to starting jobs in the CI, (b) since it went through one account, all the build emails went to a single person, and (c) the mirroring happened on our own infrastructure, which turned out not to be so reliable. So we got tired of this approach. Eventually we decided to move development to Gitlab, but we didn't want to move our users, so we left all the user-facing elements on Github (issues, PRs, etc.). This also kind of works but has its own set of issues. Things might have been different if we'd started on Gitlab from the very beginning, but as things are, I think having the repository on Github and the CI on Gitlab would be the best of both worlds for us.

jgh 9y ago
Have you tried CircleCI? I've found it really straightforward to use in a lot of the same ways you mention, except for perhaps the docker support -- it uses docker images and you can of course have a docker image that supports docker, but it doesnt upload to dockerhub out of the box. I'm curious to know how the two compare because I haven't tried Gitlab before.
- eslaught 9y ago
  So last time I checked (and I sent an email to sales to confirm) CircleCI doesn't have support for a configuration where you want to use both public and private runners. I want to use my own machines to build certain jobs (perhaps because those jobs require heavy computation), but I want to use the free, shared, cloud-based runners for the rest (to increase overall throughput). And I want to do this while keeping the master CI instance (the node which keeps track of which jobs are running and serves the webpage which shows the job status) on hosted infrastructure so that I don't have to manage it myself. Nearly everything that exists today, except Gitlab, forces you to choose between shared runners in the cloud and doing it completely yourself (including hosting the master CI instance). Gitlab allows you to connect both shared and private runners to a hosted CI instance, so we get the benefit of private machines, the throughput of shared machines, and ease-of-use of not having to host the CI ourselves. Edit: And note, to use private runners with basically any of the commercial CI's, you have to pay big money for a enterprise instance (even if you're open source). Whereas the ability to connect private runners to Gitlab comes for free (I think on all accounts? but definitely open source ones).
  - jgh 9y ago
    Ah interesting. Thanks for the reply.
  - stock_toaster 9y ago
    We use buildkite at work, and it works well. Buildkite hosts the CI master, and you run the workers on your own infrastructure -- as many as you want. So you don't get the free public agents like you want, but you do get private runners.
    - _frog 9y ago
      My company's also using Buildkite, and I've been hugely impressed with how easy it is to work with, and how flexible it's been for us.
  - ragesh 9y ago
    Disclosure: I work at Shippable. I think the latest features we launched at Shippable meets all of your requirements. You can mix and match on-demand and dedicated nodes, allocate them to specific projects if you'd like, all of this is available in our SaaS edition, and pricing is (in my opinion) quite reasonable.
eslaught 9y ago
Edit2: This is wrong, see replies below. An update, having now tried this on a toy Gitlab/Github instance. It looks like this still relies on Gitlab's support for mirroring an external repo, which relies on polling. This is less than awesome, because it means we're back into the (many) minutes latency range from pushing changes to seeing them appear in the mirrored repo (and therefore CI). When I did this before, we had to set up a mirror on our own machines to get this latency down into the ~1 minute range, otherwise it was unusable. What appears to have been added was syncing the build status from CI builds back to Github. That part seems to work as intended. There does appear to be a Gitlab API for triggering a mirror update, but I don't see an easy way to get that hooked up to Github out of the box. The CI status feature would be much more aweseome with the addition of push-based mirroring. Edit: And the problem with mirroring manually like we did before is that there isn't a clean way to mirror pull requests (it requires manipulating web APIs, not just pushing a Git repo). So if we went back to that route, we wouldn't really get the seamless experience we've been hoping for.
- williamchia 9y ago
  Hi Eslaught - So the new CI/CD for GitHub feature should be solving the pain points of the old manual mirroring process. When you set up a project and Oauth to GitHub it should automatically implement 3 components: - Bidirectional Mirroring https://gitlab.com/gitlab-org/gitlab-ee/issues/3745 https://gitlab.com/gitlab-org/gitlab-ee/issues/3745 - push webhook to GitLab to trigger CI/CD immediately once a code is committed (e.g. "push mirroring") https://gitlab.com/gitlab-org/gitlab-ee/issues/4691 https://gitlab.com/gitlab-org/gitlab-ee/issues/4691 - GitHub project service integration (to webhook CI status back to GitHub.) https://gitlab.com/gitlab-org/gitlab-ee/issues/3836 https://gitlab.com/gitlab-org/gitlab-ee/issues/3836 Is this not working for you? (may be a bug) Did you set up the toy instance by creating a CI/CD project and use the GitHub Oauth?
  - Snappy 9y ago
    FWIW, if you created the connection between GitLab and GitHub for your project before 10.6, you'll have to remove the connection and then re-add it to get the push-based mirroring to happen. Otherwise, yes, it will still poll.
reddit_sucks 9y ago
It also seems massively overpriced. $99/user/month? Pff. No one is going to pay that. Certainly not large enterprise businesses.
Improvotter 9y ago
I'm planning on open sourcing my service and was planning on using GitHub because I wanted to get people involved. However, currently I'm using a local server with Jenkins to poll GitHub for the deployment with a separate repository for deployment configuration. This isn't ideal and I'm not a huge fan of Jenkins itself in the first place. (It took a huge amount of time to setup!) Could you explain more about how you've got public and private CI/CD setup? I'd love to get the tests to run publicly, but I'd love to deploy privately and not have all of that information right there in the repo because the plan is to offer it as a service as well.

jgh 9y ago

Have you tried CircleCI? I've found it really straightforward to use in a lot of the same ways you mention, except for perhaps the docker support -- it uses docker images and you can of course have a docker image that supports docker, but it doesnt upload to dockerhub out of the box. I'm curious to know how the two compare because I haven't tried Gitlab before.

eslaught 9y ago
So last time I checked (and I sent an email to sales to confirm) CircleCI doesn't have support for a configuration where you want to use both public and private runners. I want to use my own machines to build certain jobs (perhaps because those jobs require heavy computation), but I want to use the free, shared, cloud-based runners for the rest (to increase overall throughput). And I want to do this while keeping the master CI instance (the node which keeps track of which jobs are running and serves the webpage which shows the job status) on hosted infrastructure so that I don't have to manage it myself. Nearly everything that exists today, except Gitlab, forces you to choose between shared runners in the cloud and doing it completely yourself (including hosting the master CI instance). Gitlab allows you to connect both shared and private runners to a hosted CI instance, so we get the benefit of private machines, the throughput of shared machines, and ease-of-use of not having to host the CI ourselves. Edit: And note, to use private runners with basically any of the commercial CI's, you have to pay big money for a enterprise instance (even if you're open source). Whereas the ability to connect private runners to Gitlab comes for free (I think on all accounts? but definitely open source ones).
- jgh 9y ago
  Ah interesting. Thanks for the reply.
- stock_toaster 9y ago
  We use buildkite at work, and it works well. Buildkite hosts the CI master, and you run the workers on your own infrastructure -- as many as you want. So you don't get the free public agents like you want, but you do get private runners.
  - _frog 9y ago
    My company's also using Buildkite, and I've been hugely impressed with how easy it is to work with, and how flexible it's been for us.
- ragesh 9y ago
  Disclosure: I work at Shippable. I think the latest features we launched at Shippable meets all of your requirements. You can mix and match on-demand and dedicated nodes, allocate them to specific projects if you'd like, all of this is available in our SaaS edition, and pricing is (in my opinion) quite reasonable.

eslaught 9y ago

Edit2: This is wrong, see replies below. An update, having now tried this on a toy Gitlab/Github instance. It looks like this still relies on Gitlab's support for mirroring an external repo, which relies on polling. This is less than awesome, because it means we're back into the (many) minutes latency range from pushing changes to seeing them appear in the mirrored repo (and therefore CI). When I did this before, we had to set up a mirror on our own machines to get this latency down into the ~1 minute range, otherwise it was unusable. What appears to have been added was syncing the build status from CI builds back to Github. That part seems to work as intended. There does appear to be a Gitlab API for triggering a mirror update, but I don't see an easy way to get that hooked up to Github out of the box. The CI status feature would be much more aweseome with the addition of push-based mirroring. Edit: And the problem with mirroring manually like we did before is that there isn't a clean way to mirror pull requests (it requires manipulating web APIs, not just pushing a Git repo). So if we went back to that route, we wouldn't really get the seamless experience we've been hoping for.

williamchia 9y ago
Hi Eslaught - So the new CI/CD for GitHub feature should be solving the pain points of the old manual mirroring process. When you set up a project and Oauth to GitHub it should automatically implement 3 components: - Bidirectional Mirroring https://gitlab.com/gitlab-org/gitlab-ee/issues/3745 https://gitlab.com/gitlab-org/gitlab-ee/issues/3745 - push webhook to GitLab to trigger CI/CD immediately once a code is committed (e.g. "push mirroring") https://gitlab.com/gitlab-org/gitlab-ee/issues/4691 https://gitlab.com/gitlab-org/gitlab-ee/issues/4691 - GitHub project service integration (to webhook CI status back to GitHub.) https://gitlab.com/gitlab-org/gitlab-ee/issues/3836 https://gitlab.com/gitlab-org/gitlab-ee/issues/3836 Is this not working for you? (may be a bug) Did you set up the toy instance by creating a CI/CD project and use the GitHub Oauth?
- Snappy 9y ago
  FWIW, if you created the connection between GitLab and GitHub for your project before 10.6, you'll have to remove the connection and then re-add it to get the push-based mirroring to happen. Otherwise, yes, it will still poll.

reddit_sucks 9y ago

It also seems massively overpriced. $99/user/month? Pff. No one is going to pay that. Certainly not large enterprise businesses.

Improvotter 9y ago

I'm planning on open sourcing my service and was planning on using GitHub because I wanted to get people involved. However, currently I'm using a local server with Jenkins to poll GitHub for the deployment with a separate repository for deployment configuration. This isn't ideal and I'm not a huge fan of Jenkins itself in the first place. (It took a huge amount of time to setup!) Could you explain more about how you've got public and private CI/CD setup? I'd love to get the tests to run publicly, but I'd love to deploy privately and not have all of that information right there in the repo because the plan is to offer it as a service as well.