4 ms·

Ask HN: Why do some companies build their own CICD orchestrators?

14 points by dev_0 4y ago

I saw Slack has something called Checkpoint

hansvm 4y ago
CICD can get out of hand pretty quickly. If an off-the-shelf solution works for you with minimal changes to your code and workflow then that's fantastic. Ofttimes they're simultaneously clunky and full of unnecessary (to you) features while not supporting the few things you care about. Writing a general purpose CICD probably isn't a good idea for most organizations IMO, but writing a CICD that handles just your use cases is super easy and can take less time than writing the workarounds necessary otherwise. Plus, when things inevitably go wrong the code is right there for you to fix. The second you notice your engineers are writing anything like custom batching logic to get around CICD limits, custom change detection because the general-purpose solution can't appropriately recognize transitive dependencies, ways out of the CICD sandbox to interact with some raw hardware device with vendored blobs that only work on XP, or anything else fancy then that's a strong indication that you would have been better off doing it yourself from the start.
cloudfive 4y ago
Maybe the question is really "why do some companies build a new in-house technology rather than use an existing one"? 1. Some companies are so slow and some teams so agile that creating a new working project takes less time than submitting a ticket for permission to install and demo software and get the required permission to purchase it. 2. Developers hate to deal with salespeople, so unless the CICD platform is OpenSource / free the developer might choose to build something simple. 3. Developers might be unaware that a solution exists to solve their specific problem. 4. The commercial solution to a small developer's problem might be too expensive. The developer estimates of time and money for building something in-house often neglect the "hidden costs" ... (including the long tail). The hidden costs often include : 1. opportunity costs - a developer can't do two things at once, so if they are writing a tool, they are not releasing new features or fixing defects. 2. Maintenance costs - keeping the thing running (when the host OS is upgraded, the network changes, the repository credentials expire, etc) 3. support costs - if this touches any other outside teams or customers, someone will have issues and need support. Maybe a new team has to be created to handle the Zendesk support tickets, provide Slack help, etc.
tyingq 4y ago
There's an article here explaining one reason why they put Checkpoint in front of Jenkins: https://slack.engineering/circuit-breakers/ https://slack.engineering/circuit-breakers/
theamk 4y ago
Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD system out there except Jenkins and Buildkite, and both of them a pretty horrible at it unless you spend lots of time with custom scripts. At some point it will become easier to rewrite entire system rather than try to fight existing one.
- derefnull 4y ago
  Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful. I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.
- cloudfive 4y ago
  There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself. Also noteworthy: Slack is going to explain this in more depth next month: https://www.linkedin.com/video/event/urn:li:ugcPost:6962637599597355008/ https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...
  - Mandatum 4y ago
    Or you can read the article: https://slack.engineering/remote-development-at-slack/ https://slack.engineering/remote-development-at-slack/
- humbleMouse 4y ago
- thrwawy74 4y ago
yrgulation 4y ago
I built one for a client. It was a rocketchat bot taking commands for making a release of a specific tag or latest master branch, a revert, showing what is currently deployed and printing a commit log vs previously deployed commit. Worked like a charm. Was a set of ansible scripts called by node. No need for anything fancy. Oh and it always expected a confirmation.
Gravityloss 4y ago
It has been tried, for example making diagrams in Rational Rose and generating C++ code from that. Then I assume filling in the code "details" manually. Probably someone else can comment on how that has worked. That was before agile or TDD were such big things.
- cloudfive 4y ago
  I was around when UML took off - but the only real success I have seen from automatically generating code (from diagrams or otherwise) has been in the API space recently since those contracts are easy to describe and generate with tools like Swagger

14 points by dev_0 4y ago

I saw Slack has something called Checkpoint

CICD can get out of hand pretty quickly. If an off-the-shelf solution works for you with minimal changes to your code and workflow then that's fantastic. Ofttimes they're simultaneously clunky and full of unnecessary (to you) features while not supporting the few things you care about. Writing a general purpose CICD probably isn't a good idea for most organizations IMO, but writing a CICD that handles just your use cases is super easy and can take less time than writing the workarounds necessary otherwise. Plus, when things inevitably go wrong the code is right there for you to fix. The second you notice your engineers are writing anything like custom batching logic to get around CICD limits, custom change detection because the general-purpose solution can't appropriately recognize transitive dependencies, ways out of the CICD sandbox to interact with some raw hardware device with vendored blobs that only work on XP, or anything else fancy then that's a strong indication that you would have been better off doing it yourself from the start.

cloudfive 4y ago

Maybe the question is really "why do some companies build a new in-house technology rather than use an existing one"? 1. Some companies are so slow and some teams so agile that creating a new working project takes less time than submitting a ticket for permission to install and demo software and get the required permission to purchase it. 2. Developers hate to deal with salespeople, so unless the CICD platform is OpenSource / free the developer might choose to build something simple. 3. Developers might be unaware that a solution exists to solve their specific problem. 4. The commercial solution to a small developer's problem might be too expensive. The developer estimates of time and money for building something in-house often neglect the "hidden costs" ... (including the long tail). The hidden costs often include : 1. opportunity costs - a developer can't do two things at once, so if they are writing a tool, they are not releasing new features or fixing defects. 2. Maintenance costs - keeping the thing running (when the host OS is upgraded, the network changes, the repository credentials expire, etc) 3. support costs - if this touches any other outside teams or customers, someone will have issues and need support. Maybe a new team has to be created to handle the Zendesk support tickets, provide Slack help, etc.

tyingq 4y ago

There's an article here explaining one reason why they put Checkpoint in front of Jenkins: https://slack.engineering/circuit-breakers/ https://slack.engineering/circuit-breakers/

theamk 4y ago

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD system out there except Jenkins and Buildkite, and both of them a pretty horrible at it unless you spend lots of time with custom scripts. At some point it will become easier to rewrite entire system rather than try to fight existing one.

derefnull 4y ago
Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful. I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.
cloudfive 4y ago
There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself. Also noteworthy: Slack is going to explain this in more depth next month: https://www.linkedin.com/video/event/urn:li:ugcPost:6962637599597355008/ https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...
- Mandatum 4y ago
  Or you can read the article: https://slack.engineering/remote-development-at-slack/ https://slack.engineering/remote-development-at-slack/
humbleMouse 4y ago
thrwawy74 4y ago

yrgulation 4y ago

I built one for a client. It was a rocketchat bot taking commands for making a release of a specific tag or latest master branch, a revert, showing what is currently deployed and printing a commit log vs previously deployed commit. Worked like a charm. Was a set of ansible scripts called by node. No need for anything fancy. Oh and it always expected a confirmation.

Gravityloss 4y ago

It has been tried, for example making diagrams in Rational Rose and generating C++ code from that. Then I assume filling in the code "details" manually. Probably someone else can comment on how that has worked. That was before agile or TDD were such big things.

cloudfive 4y ago
I was around when UML took off - but the only real success I have seen from automatically generating code (from diagrams or otherwise) has been in the API space recently since those contracts are easy to describe and generate with tools like Swagger