3 ms·
The usual way to create pipelines of pipelines is that one pipeline puts to a resource that another pipeline will trigger on. For example, one pipeline might d
by jacques_chester 8y ago
The usual way to create pipelines of pipelines is that one pipeline puts to a resource that another pipeline will trigger on.
For example, one pipeline might do a bunch of stuff and drop a file into a bucket. Unbeknownst to pipeline 1, pipeline 2 is triggered by the file drop and begins to do its own thing.
The key here is that all statefulness is the responsibility of Resources -- Concourse does not do very much except bookkeeping about what Jobs have seen what versions of a Resource.
So if you need multi-pipeline coordination you can do this through any shared state you please. One that I personally like is tagging repositories, these are a nice visible marker of activity to observers.
At Pivotal each team tends to have several pipelines. Teams which rely on lots of upstream work (Release Engineering, Master Pipeline, CloudOps) tend to run many more pipelines which respond to activity by upstream teams. Similarly, Buildpacks has many pipelines which react to changes made in projects outside our control. When Node releases a new version, the pipeline checks out the sourcecode and begins to roll a new buildpack from scratch. At no point do the downstream teams need to coordinate with the upstream teams.
I'd say that automation is one of our secret weapons, but I never shut up about it.