5 ms·

Yeah but I mean... that's why CI and CD are separate things. CI should not need any privileges. CI builds should be hermetic (no network access, no persistence,

Yeah but I mean... that's why CI and CD are separate things. CI should not need any privileges. CI builds should be hermetic (no network access, no persistence, ideally fully reproducible from what's going in). CI should not talk to servers on your network, let alone have credentials, especially for production systems.

NAHWheatCracker 5y ago
CI for me has to access repositories. Eg. downloading libraries from PyPI, Maven, NPM, etc... For private repositories, that means access to credentials. Probably read-only credentials, but it requires network access. Would you be suggesting that everyone should commit all dependencies?
- formerly_proven 5y ago
  That's a fair point. I think for many projects which only consume dependencies the answer can be "yes, just commit all your dependencies" in many instances. Pulling from internal repos shouldn't be critical though as long as the read-only tokens you're using can only access what the developers can read anyway; pulling "unintended" code in a CI should never be able to escalate because we're running anything a dev pushed.
  - NAHWheatCracker 5y ago
    I've heard of doing this, although not worked on such a project. Committing node_modules seems wild to me and I foresee issues with committing all the Python wheels necessary for different platforms on some projects. I'm a proponent of lock-file approaches, which gain 99% of the benefits with far less pain. It requires network access, though.
    - emteycz 5y ago
      Yarn v2 is much better for committing dependencies than committing node_modules. However, it's problematic. Only use it if you're certain it will solve a specific problem you have. Consider using Docker to build images that include a snapshot of node_modules.
- nightpool 5y ago
  Sure, but that's a completely different scenario than the one the OP/TFA is talking about. We're talking about disclosing privileged information or access tokens that a single developer on their own shouldn't be able to handle without code review, like being able to use your CI system to access production accounts. Software libraries don't fall under this category, since the developer would already need to have accessed them to write the code in the first place!
  - NAHWheatCracker 5y ago
    I was focused on the "no network access" aspect of formerly_proven's comment. "no network access" would make CI nearly pointless from my perspective. I agree that there are tokens and variables that are dangerous to expose via CI, but throwing the baby out with the bathwater confused me.
- derefr 5y ago
  You could "just" 1. stand up a private package mirror that you control, that uses a whitelist for what packages it is willing to mirror; 2. configure your project's dependency-fetching logic to fetch from said mirror; 3. configure CI to only allow outbound network access to your package mirror's IP. The disadvantage — but also the point — of this, is that it is then a release manager's responsibility, not a developer's responsibility, to give the final say-so for adding a dependency to the project (because only the release manager has the permissions to add packages to the mirror.)
  - NAHWheatCracker 5y ago
    That's an interesting approach. I'm not keen on the bureaucratic aspect, which leads to more friction than it's worth in my experience. I guess that's beside the point if your goal is only to reduce risk of compromised CI/CD.
    - mulmen 5y ago
      You can compromise and give the developers the ability to add dependencies as well. In a separate “dependency mirror” package. But those have to be code reviewed like anything else. So you have a paper trail but adding a dependency is lower friction. And you still can’t accidentally (or maliciously!) pull in an evil dependency without at least two people involved.
pc86 5y ago
Based on what? Every CI system I've ever seem has pulled dependencies in from the network.
- inetknght 5y ago
  git clone --recursive --branch "$commitid" "$repourl" "$repodir" img="$(docker build --network=none -f "$dockerfile" "$repodir")" docker run --rm -ti --network=none "$img" Sure, CI pulls in from the network... but execution occurs without network.
  - staticassertion 5y ago
    Can you explain wtf is happening here?
    - structural 5y ago
      After the CI system (with network access) pulls down the code, including submodules, this code is then placed into a container with no network access to perform the actual build.
      - pharmakom 5y ago
        Is such a clean separation possible? I’ve seen some crazy things…
        inetknght 5y ago
        For well-designed projects it's not only possible but required.
    - inetknght 5y ago
      I'd assumed some familiarity with common CI systems (and assumed the commands would be tailored to your use case). Let me walk it into some more depth. First: > git clone --recursive --branch "$commitid" "$repourl" "$repodir" The `git clone` will take your git repository URL as the $repourl variable. It will also take your commit id (commit hash or tagged version which a pull request points to) as the $commitid variable (`--branch $commitid`). It will also take a $repodir variable which points to the directory that will contain the contents of the cloned git repository and already checked out at the commit id specified. It will do so recursively (`--recursive`): if there are submodules then they will also automatically be cloned. This of course assumes that you're cloning a public repository and/or that any credentials required have already been set up (see `man git-config`). Then: > img="$(docker build --network=none -f "$dockerfile" "$repodir")" Okay so this is sort've broken: you'd need a few more parameters to `docker build` to get it to work "right". But as-is, `docker build` usually has network access so `--network=none` will specify that the build process will not have access to the network. I hope your build system doesn't automatically download dependencies because that will fail (and also suggests that the build system may be susceptible to attack). You specify a dockerfile to build using `-f "$dockerfile"`. Finally, you specify the build context using "$repodir" -- and that assumes that your whole git repository should be available to the dockerfile. However, `docker build` will write a lot more than just the image name to standard output and so this is where some customization would need to occur. Suffice to say that you can use `--quiet` if that's all you want; I do prefer to see the output because it normally contains intermediate image names useful for debugging the dockerfile. Finally: > docker run --rm -ti --network=none "$img" Finally, it runs the built image in a new container with an auto-generated name. `-ti` here is wrong: it will attach a standard input/output terminal and so if it drops you into an interactive program (such as bash) then it could hang the CI process. But you can remove that. It also assumes that your dockerfile correctly specifies ENTRYPOINT and/or CMD. When the container has exited then the container will automatically be removed (--rm) -- usually they linger around and pollute your docker host. Finally, the --network=none also ensures that your container does not have network access so your unit tests should also be capable of running without the network or else they will fail. You could use `--volume` to specify a volume with data files if you need them. You might also want to look at `--user` if you don't want your container to have root privileges... And of course if you want integration tests with other containers then you should create a dedicated docker network and specify its alias with `--network`: see `man docker-network-create`; you can use `docker network create -d internal` to create a network which shouldn't let containers out. Does that answer your question?
tremon 5y ago
Our CI systems have read access to the data definition store (which is a sql database right now), because we don't store interop data definitions in code. So fully hermetic no, because our code (repository) is not fully self-contained. The definition store has its own audits and change tracking, but it's separate from the interface code.
- derefr 5y ago
  That seems fine, in the same way that e.g. an XML library fetching DTDs from a known public URL is fine. However, it'd probably be better if you could have the CI framework collect and inject that information into the build using some hard-coded deterministic logic, rather than giving the build itself (developer-driven Arbitrary Code Execution) access to that capability. Same idea as e.g. injecting Kubernetes Secrets into Pods as env-vars at the controller level, rather than giving the Pod itself the permission to query Secrets out of the controller through its API.

NAHWheatCracker 5y ago

CI for me has to access repositories. Eg. downloading libraries from PyPI, Maven, NPM, etc... For private repositories, that means access to credentials. Probably read-only credentials, but it requires network access. Would you be suggesting that everyone should commit all dependencies?

formerly_proven 5y ago
That's a fair point. I think for many projects which only consume dependencies the answer can be "yes, just commit all your dependencies" in many instances. Pulling from internal repos shouldn't be critical though as long as the read-only tokens you're using can only access what the developers can read anyway; pulling "unintended" code in a CI should never be able to escalate because we're running anything a dev pushed.
- NAHWheatCracker 5y ago
  I've heard of doing this, although not worked on such a project. Committing node_modules seems wild to me and I foresee issues with committing all the Python wheels necessary for different platforms on some projects. I'm a proponent of lock-file approaches, which gain 99% of the benefits with far less pain. It requires network access, though.
  - emteycz 5y ago
    Yarn v2 is much better for committing dependencies than committing node_modules. However, it's problematic. Only use it if you're certain it will solve a specific problem you have. Consider using Docker to build images that include a snapshot of node_modules.
nightpool 5y ago
Sure, but that's a completely different scenario than the one the OP/TFA is talking about. We're talking about disclosing privileged information or access tokens that a single developer on their own shouldn't be able to handle without code review, like being able to use your CI system to access production accounts. Software libraries don't fall under this category, since the developer would already need to have accessed them to write the code in the first place!
- NAHWheatCracker 5y ago
  I was focused on the "no network access" aspect of formerly_proven's comment. "no network access" would make CI nearly pointless from my perspective. I agree that there are tokens and variables that are dangerous to expose via CI, but throwing the baby out with the bathwater confused me.
derefr 5y ago
You could "just" 1. stand up a private package mirror that you control, that uses a whitelist for what packages it is willing to mirror; 2. configure your project's dependency-fetching logic to fetch from said mirror; 3. configure CI to only allow outbound network access to your package mirror's IP. The disadvantage — but also the point — of this, is that it is then a release manager's responsibility, not a developer's responsibility, to give the final say-so for adding a dependency to the project (because only the release manager has the permissions to add packages to the mirror.)
- NAHWheatCracker 5y ago
  That's an interesting approach. I'm not keen on the bureaucratic aspect, which leads to more friction than it's worth in my experience. I guess that's beside the point if your goal is only to reduce risk of compromised CI/CD.
  - mulmen 5y ago
    You can compromise and give the developers the ability to add dependencies as well. In a separate “dependency mirror” package. But those have to be code reviewed like anything else. So you have a paper trail but adding a dependency is lower friction. And you still can’t accidentally (or maliciously!) pull in an evil dependency without at least two people involved.

pc86 5y ago

Based on what? Every CI system I've ever seem has pulled dependencies in from the network.

inetknght 5y ago
git clone --recursive --branch "$commitid" "$repourl" "$repodir" img="$(docker build --network=none -f "$dockerfile" "$repodir")" docker run --rm -ti --network=none "$img" Sure, CI pulls in from the network... but execution occurs without network.
- staticassertion 5y ago
  Can you explain wtf is happening here?
  - structural 5y ago
    After the CI system (with network access) pulls down the code, including submodules, this code is then placed into a container with no network access to perform the actual build.
    - pharmakom 5y ago
      Is such a clean separation possible? I’ve seen some crazy things…
      - inetknght 5y ago
        For well-designed projects it's not only possible but required.
  - inetknght 5y ago
    I'd assumed some familiarity with common CI systems (and assumed the commands would be tailored to your use case). Let me walk it into some more depth. First: > git clone --recursive --branch "$commitid" "$repourl" "$repodir" The `git clone` will take your git repository URL as the $repourl variable. It will also take your commit id (commit hash or tagged version which a pull request points to) as the $commitid variable (`--branch $commitid`). It will also take a $repodir variable which points to the directory that will contain the contents of the cloned git repository and already checked out at the commit id specified. It will do so recursively (`--recursive`): if there are submodules then they will also automatically be cloned. This of course assumes that you're cloning a public repository and/or that any credentials required have already been set up (see `man git-config`). Then: > img="$(docker build --network=none -f "$dockerfile" "$repodir")" Okay so this is sort've broken: you'd need a few more parameters to `docker build` to get it to work "right". But as-is, `docker build` usually has network access so `--network=none` will specify that the build process will not have access to the network. I hope your build system doesn't automatically download dependencies because that will fail (and also suggests that the build system may be susceptible to attack). You specify a dockerfile to build using `-f "$dockerfile"`. Finally, you specify the build context using "$repodir" -- and that assumes that your whole git repository should be available to the dockerfile. However, `docker build` will write a lot more than just the image name to standard output and so this is where some customization would need to occur. Suffice to say that you can use `--quiet` if that's all you want; I do prefer to see the output because it normally contains intermediate image names useful for debugging the dockerfile. Finally: > docker run --rm -ti --network=none "$img" Finally, it runs the built image in a new container with an auto-generated name. `-ti` here is wrong: it will attach a standard input/output terminal and so if it drops you into an interactive program (such as bash) then it could hang the CI process. But you can remove that. It also assumes that your dockerfile correctly specifies ENTRYPOINT and/or CMD. When the container has exited then the container will automatically be removed (--rm) -- usually they linger around and pollute your docker host. Finally, the --network=none also ensures that your container does not have network access so your unit tests should also be capable of running without the network or else they will fail. You could use `--volume` to specify a volume with data files if you need them. You might also want to look at `--user` if you don't want your container to have root privileges... And of course if you want integration tests with other containers then you should create a dedicated docker network and specify its alias with `--network`: see `man docker-network-create`; you can use `docker network create -d internal` to create a network which shouldn't let containers out. Does that answer your question?

tremon 5y ago

Our CI systems have read access to the data definition store (which is a sql database right now), because we don't store interop data definitions in code. So fully hermetic no, because our code (repository) is not fully self-contained. The definition store has its own audits and change tracking, but it's separate from the interface code.

derefr 5y ago
That seems fine, in the same way that e.g. an XML library fetching DTDs from a known public URL is fine. However, it'd probably be better if you could have the CI framework collect and inject that information into the build using some hard-coded deterministic logic, rather than giving the build itself (developer-driven Arbitrary Code Execution) access to that capability. Same idea as e.g. injecting Kubernetes Secrets into Pods as env-vars at the controller level, rather than giving the Pod itself the permission to query Secrets out of the controller through its API.