3 ms·

Owning someone's CI system via the CI system is all well and good, but not surprising in the slightest. All these things expose web-based admin interfaces, so o

Owning someone's CI system via the CI system is all well and good, but not surprising in the slightest. All these things expose web-based admin interfaces, so of course they're vulnerable at the very least to social engineering. The real trick is owning someone's CI system via the code under test. And it's even more versatile because it works even for projects which keep their executor scripts checked in to source. 1) Check out and gain an understanding of the target's CI scripts 2) Create a pull request (or whatever) which makes some reasonable code change but also introduces a vulnerability into the build/test scripts themselves. For example, curl'ing a domain you control. 3) Kick off pre-commit runs of all the tests to see if your patch should be accepted. 4) Own all the devices testing your patch.

sytse 10y ago
Even when you have control of a project tested by the CI server you should not have control of the CI server itself. GitLab comes with integrated CI. But all the tests are ran by GitLab Runner on external machines. You can spin up (and down) a new machines per test with Runner Autoscale if you want. As far as I could see GitLab CI is not vulnerable to the items mentioned in the article (apart from only enforcing password length, not complexity). But CI security is hard so if you find anything please let us know https://about.gitlab.com/disclosure/ https://about.gitlab.com/disclosure/
- phasmantistes 10y ago
  Yes, sandboxing and ephemeral machines are far and away the best defense a CI system has against compiling/testing malicious code. But the vast majority of CI systems don't actually use ephemeral executors because imaging between runs is slow and rebuilding your whole checkout/object cache is expensive. You don't need to own the CI server in order to own a project. You can take over all of the individual executors and replace their linker with your own version that links against an attacker-supplied backdoored fork of openssl (for an extreme and topical example). Doesn't even require constantly-running code which might be detected by a monitoring system, and naturally persists across reboots (but not reimagings). I'm glad that GitLab is paying attention to these things. The Travis which is tightly integrated with GitHub also seems to do a good job, although I haven't looked into it extensively. But plenty of orgs run self-hosted CI that simply isn't up to proper security standards in this respect.
  - sytse 10y ago
    Rebuilding is expensive. We have several caching mechanisms but there is no silver bullet and caching itself can lead to security problems. For a deeper review of security problems we're currently focussed on please see https://gitlab.com/gitlab-org/gitlab-ce/issues/21911 https://gitlab.com/gitlab-org/gitlab-ce/issues/21911 and please feel free to contribute there.
    - Snappy 10y ago
      For a discussion on caching and artifacts, please see https://gitlab.com/gitlab-org/gitlab-ce/issues/21913 https://gitlab.com/gitlab-org/gitlab-ce/issues/21913.
- drewcrawford 10y ago
  Heavy Gitlab CI user here. The real problem is isolating the runners themselves. You can assign runners to individual projects for isolation, but when the runners can't be AWSed (iOS hardware, for example) you have to make tough choices about buying more stuff to sit under your desk right next to the stuff you could be using that's idle. IMO some basic isolation measures like clearing the cache between projects would be good. Killing active processes when a job ends.
  - sytse 10y ago
    Thanks for using GitLab heavily! When running GitLab Runner on metal you can already opt to start with a fresh clone instead of updating I think. I'm not sure about killing active processes, feel free to create an issue to discuss further.

sytse 10y ago

Even when you have control of a project tested by the CI server you should not have control of the CI server itself. GitLab comes with integrated CI. But all the tests are ran by GitLab Runner on external machines. You can spin up (and down) a new machines per test with Runner Autoscale if you want. As far as I could see GitLab CI is not vulnerable to the items mentioned in the article (apart from only enforcing password length, not complexity). But CI security is hard so if you find anything please let us know https://about.gitlab.com/disclosure/ https://about.gitlab.com/disclosure/

phasmantistes 10y ago
Yes, sandboxing and ephemeral machines are far and away the best defense a CI system has against compiling/testing malicious code. But the vast majority of CI systems don't actually use ephemeral executors because imaging between runs is slow and rebuilding your whole checkout/object cache is expensive. You don't need to own the CI server in order to own a project. You can take over all of the individual executors and replace their linker with your own version that links against an attacker-supplied backdoored fork of openssl (for an extreme and topical example). Doesn't even require constantly-running code which might be detected by a monitoring system, and naturally persists across reboots (but not reimagings). I'm glad that GitLab is paying attention to these things. The Travis which is tightly integrated with GitHub also seems to do a good job, although I haven't looked into it extensively. But plenty of orgs run self-hosted CI that simply isn't up to proper security standards in this respect.
- sytse 10y ago
  Rebuilding is expensive. We have several caching mechanisms but there is no silver bullet and caching itself can lead to security problems. For a deeper review of security problems we're currently focussed on please see https://gitlab.com/gitlab-org/gitlab-ce/issues/21911 https://gitlab.com/gitlab-org/gitlab-ce/issues/21911 and please feel free to contribute there.
  - Snappy 10y ago
    For a discussion on caching and artifacts, please see https://gitlab.com/gitlab-org/gitlab-ce/issues/21913 https://gitlab.com/gitlab-org/gitlab-ce/issues/21913.
drewcrawford 10y ago
Heavy Gitlab CI user here. The real problem is isolating the runners themselves. You can assign runners to individual projects for isolation, but when the runners can't be AWSed (iOS hardware, for example) you have to make tough choices about buying more stuff to sit under your desk right next to the stuff you could be using that's idle. IMO some basic isolation measures like clearing the cache between projects would be good. Killing active processes when a job ends.
- sytse 10y ago
  Thanks for using GitLab heavily! When running GitLab Runner on metal you can already opt to start with a fresh clone instead of updating I think. I'm not sure about killing active processes, feel free to create an issue to discuss further.