3 ms·

I would do both. Make a build container that pulls in the Git hash that you want to build/test against. You can keep your dependencies, build process, unit and

I would do both. Make a build container that pulls in the Git hash that you want to build/test against. You can keep your dependencies, build process, unit and func tests in the container. Your dev team can use the same container image to build and test in the same way that you do, pulling in the latest code from Git and running a docker build. Now nobody needs to set up their environment because it's all just in the Docker image in the same way. You don't need all those build and test files in production, so you can build a new "deploy container" with just the artifacts from your build process. I would also record the versions of your dependencies, as well as the SHA hash of the build/test container you used, and the SHA hash of the Git repos you pulled from. You can also set things like flyway migrations to run each time the container runs. Now you can deploy this second container somewhere. You can then use the same build/test container you used before to run func tests against the deployed container, to double check it works in your deployed environment.

hinkley 8y ago
The former can be handily taken care of in your CI toolchain. You need a build agent with all of the CLI tools you need for your build. Many support Docker these days. Some can even spin up images on demand.
- peterwwillis 8y ago
  This is (for me) the best part of keeping your build process in a container: reducing the complexity of the build agent. The only software dependencies on your build agent are Docker and Git. Your CI/CD job pulls a Git repo with a Dockerfile, and runs 'docker build', and that pulls every dependency it needs.
  - hinkley 8y ago
    Agreed, and I can’t wait to work someplace like that again. We have a build agent with 9 versions of Node in it. It was 6 when I started here. Once a month I get to help someone figure out weird errors because one task has the wrong value in a drop down. It’s such busy work I can’t even. When I run the CI system, shit gets done and people are expected to be able to recreate build failures by themselves. Give a man a tool, make it reliable, then expect him to use it.
  - jacques_chester 8y ago
    I am obliged to refer you to Concourse, which is really really good at containerised build and automation: https://concourse-ci.org/ https://concourse-ci.org/
    - peterwwillis 8y ago
      I really like how stupid easy that index page is to read and understand. These seem like pretty good docs. Some of the documentation is missing, so I'm not sure if I missed this or not. But can you make a pipeline out of independently configured jobs? Or a pipeline of pipelines? Often you'll want to chain jobs or pipelines that are owned by different groups. I imagine they support this, if they support teams...
      - jacques_chester 8y ago
        The usual way to create pipelines of pipelines is that one pipeline puts to a resource that another pipeline will trigger on. For example, one pipeline might do a bunch of stuff and drop a file into a bucket. Unbeknownst to pipeline 1, pipeline 2 is triggered by the file drop and begins to do its own thing. The key here is that all statefulness is the responsibility of Resources -- Concourse does not do very much except bookkeeping about what Jobs have seen what versions of a Resource. So if you need multi-pipeline coordination you can do this through any shared state you please. One that I personally like is tagging repositories, these are a nice visible marker of activity to observers. At Pivotal each team tends to have several pipelines. Teams which rely on lots of upstream work (Release Engineering, Master Pipeline, CloudOps) tend to run many more pipelines which respond to activity by upstream teams. Similarly, Buildpacks has many pipelines which react to changes made in projects outside our control. When Node releases a new version, the pipeline checks out the sourcecode and begins to roll a new buildpack from scratch. At no point do the downstream teams need to coordinate with the upstream teams. I'd say that automation is one of our secret weapons, but I never shut up about it.

hinkley 8y ago

The former can be handily taken care of in your CI toolchain. You need a build agent with all of the CLI tools you need for your build. Many support Docker these days. Some can even spin up images on demand.

peterwwillis 8y ago
This is (for me) the best part of keeping your build process in a container: reducing the complexity of the build agent. The only software dependencies on your build agent are Docker and Git. Your CI/CD job pulls a Git repo with a Dockerfile, and runs 'docker build', and that pulls every dependency it needs.
- hinkley 8y ago
  Agreed, and I can’t wait to work someplace like that again. We have a build agent with 9 versions of Node in it. It was 6 when I started here. Once a month I get to help someone figure out weird errors because one task has the wrong value in a drop down. It’s such busy work I can’t even. When I run the CI system, shit gets done and people are expected to be able to recreate build failures by themselves. Give a man a tool, make it reliable, then expect him to use it.
- jacques_chester 8y ago
  I am obliged to refer you to Concourse, which is really really good at containerised build and automation: https://concourse-ci.org/ https://concourse-ci.org/
  - peterwwillis 8y ago
    I really like how stupid easy that index page is to read and understand. These seem like pretty good docs. Some of the documentation is missing, so I'm not sure if I missed this or not. But can you make a pipeline out of independently configured jobs? Or a pipeline of pipelines? Often you'll want to chain jobs or pipelines that are owned by different groups. I imagine they support this, if they support teams...
    - jacques_chester 8y ago
      The usual way to create pipelines of pipelines is that one pipeline puts to a resource that another pipeline will trigger on. For example, one pipeline might do a bunch of stuff and drop a file into a bucket. Unbeknownst to pipeline 1, pipeline 2 is triggered by the file drop and begins to do its own thing. The key here is that all statefulness is the responsibility of Resources -- Concourse does not do very much except bookkeeping about what Jobs have seen what versions of a Resource. So if you need multi-pipeline coordination you can do this through any shared state you please. One that I personally like is tagging repositories, these are a nice visible marker of activity to observers. At Pivotal each team tends to have several pipelines. Teams which rely on lots of upstream work (Release Engineering, Master Pipeline, CloudOps) tend to run many more pipelines which respond to activity by upstream teams. Similarly, Buildpacks has many pipelines which react to changes made in projects outside our control. When Node releases a new version, the pipeline checks out the sourcecode and begins to roll a new buildpack from scratch. At no point do the downstream teams need to coordinate with the upstream teams. I'd say that automation is one of our secret weapons, but I never shut up about it.