4 ms·

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ https://slack.engineering/

by theamk 4y ago

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD system out there except Jenkins and Buildkite, and both of them a pretty horrible at it unless you spend lots of time with custom scripts. At some point it will become easier to rewrite entire system rather than try to fight existing one.

derefnull 4y ago
Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful. I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.
cloudfive 4y ago
There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself. Also noteworthy: Slack is going to explain this in more depth next month: https://www.linkedin.com/video/event/urn:li:ugcPost:6962637599597355008/ https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...
- Mandatum 4y ago
  Or you can read the article: https://slack.engineering/remote-development-at-slack/ https://slack.engineering/remote-development-at-slack/
humbleMouse 4y ago
thrwawy74 4y ago
I would think running something within a Docker container is running it natively?
- theamk 4y ago
  Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.
- verdverm 4y ago
  How do you build a container from inside a container? Answering this gives you one example of a difference
longcommonname 4y ago
This is possible with github actions. Have one job create a matrix of other jobs for each of your runners. Use self hosted runners if you are worried about costs or have special requirements.
- theamk 4y ago
  Test build is dozens of gigabytes and millions of files and takes 20+ minutes. When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more. You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.
- verdverm 4y ago
  There are limits on build queue sizes unless you buy the enterprise plan. Also doesn't work if you are not hosting code on GitHub

by theamk 4y ago

derefnull 4y ago

Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful. I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.

cloudfive 4y ago

There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself. Also noteworthy: Slack is going to explain this in more depth next month: https://www.linkedin.com/video/event/urn:li:ugcPost:6962637599597355008/ https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Mandatum 4y ago
Or you can read the article: https://slack.engineering/remote-development-at-slack/ https://slack.engineering/remote-development-at-slack/

humbleMouse 4y ago

thrwawy74 4y ago

I would think running something within a Docker container is running it natively?

theamk 4y ago
Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.
verdverm 4y ago
How do you build a container from inside a container? Answering this gives you one example of a difference

longcommonname 4y ago

This is possible with github actions. Have one job create a matrix of other jobs for each of your runners. Use self hosted runners if you are worried about costs or have special requirements.

theamk 4y ago
Test build is dozens of gigabytes and millions of files and takes 20+ minutes. When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more. You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.
verdverm 4y ago
There are limits on build queue sizes unless you buy the enterprise plan. Also doesn't work if you are not hosting code on GitHub