3 ms·

Every CI system in existence is reinventing the exact same wheel: "I want to run some random task" + event hooks + secrets + logs + plugins + integrations. It's

Every CI system in existence is reinventing the exact same wheel: "I want to run some random task" + event hooks + secrets + logs + plugins + integrations. It's so ridiculous that more of them keep being created - and are losing functionality. GHA has all these configs in YAML (there's a user-friendly config file...) but doesn't let you run paramaterized builds in their web UI? .....why? We shouldn't be writing all these jobs in a format that only works for one CI system. You spend months writing Jenkinsfiles, and then you move to CircleCI and have to rewrite all of them, and then move to Drone and have to rewrite all of them, and then move to CodeBuild/CodePipeline and have to rewrite all of them, and then move to GitHub Actions and have to rewrite all of them. Eventually we'll rewrite them all for something else. And why? To run the same exact tests on a slightly different system.

yebyen 5y ago
> doesn't let you run paramaterized builds in their web UI? It doesn't? https://github.blog/changelog/2021-11-10-github-actions-input-types-for-manual-workflows/ https://github.blog/changelog/2021-11-10-github-actions-inpu... Maybe it didn't 6 months ago, but it seems this is an option now, and well-documented. This is a topic of interest for me, I presented on Jenkins and GitOps to an audience at KubeCon who I suspect are almost all interested in moving away from Jenkins, or have been told to be interested in switching from Jenkins to something else, and I tried to get the idea across that they probably don't really need to switch the workflow tool even if it's ancient, ... But maybe should consider subbing out some of the important fiddly bits underneath it (like, I assume the vast majority of Jenkins users are building images with Docker, and if they're running on Kubernetes, they're many of them wondering what they will do, or how long they really have before they have to start worrying about the deprecation of dockershim and how their lives are going to change when their clusters won't be running Docker under the hood anymore?) I was actually arguing for a tool like Porter.sh to come in and make the boundaries of "what's in a build" super neat and tidy, organized, but also limited so the next time they feel compelled to switch workflow tooling, it will be a non-issue and can be over and finished inside of a single day's work. The problem isn't that your workflow tool is too old, it's that you've jammed too much arbitrary complexity inside of it, probably because the right abstraction was not made available to you at the time. "Switching off of Jenkins" just means building a second system and it comes with all the baggage of "second system syndrome" to do so. Sure it's difficult switching from Jenkins, when you've built this gigantic pipeline with 18 branches and 12 of them run in parallel, half of them are configured with different options passed to the docker build tool, half of them must use buildx, and the third half of them are unmaintained so we don't go in there... so put some guard rails up around the hard parts! And get somebody in there to take care of those cobwebs.
- 0xbadcafebee 5y ago
  I'm glad it's an option now! That was crazy that it took 3 years for this feature to show up, though. I think we need a few more "12 Factor App"-style guidelines for modern systems. 12 Factor goes a long way to abstract away the tendency for implementation lock-in. But we can probably create a few more guidelines specific to CI so it's portable. Use OAuth, use the same authorization layer as your VCS, tie secrets and artifacts to the VCS repo, make every plugin a Docker container. (I'm stealing these concepts from Drone because it has the best cloud-native design I've ever seen) The final unsolved bit is how to manage a DAG of jobs and hooks around every event in a portable way. We probably need a universal CI spec and API, but maybe that's too specific. Also: holy crap, I don't think I've seen Porter.sh before, I love the idea! Definitely going to look into that
- steveklabnik 5y ago
  That page 404s for me.
  - yebyen 5y ago
    Hmm, I wonder how the link got changed... it worked when I posted it? Pretty sure? Someone else found the link.
    - steveklabnik 5y ago
      ... now it no longer does. How odd.
- bewuethr 5y ago
  Link should be https://github.blog/changelog/github-actions-input-types-for-manual-workflows/ https://github.blog/changelog/github-actions-input-types-for...
qbasic_forever 5y ago
It's why I like tekton, it's a CI system for kubernetes so you're letting kubernetes solve all those problems. I have far more faith that k8s will be alive and kicking and not breaking core things like secrets, logs, container execution, etc. in painful ways in the future than anything else. In general though I try to put all the CI work into a simple shell or similar script and then just configure whatever CI system is in use to call it in the appropriate environment (in a container, bespoke VM, etc.). I agree putting all kinds of complexity in a unique CI system is just asking for trouble down the road as it's now basically a hard dependency for your code to be shippable.
jiggawatts 5y ago
A brilliant insight I came across once was that a low barrier to entry can prevent standardisation and quality. The original example was that in LISP it's so easy to add Object Oriented functionality that it's a mere afternoon's work for a graduate student. For comparison, it took Bjarne Stroustrup several years to extend C with objects to create C++. Hence there are only about 4-5 such languages out there, of which only two are popular. This means that every C++ program is object oriented in the "same way", whereas random OO LISP modules are incompatible and you can't merged them into one cohesive program or reuse the components nicely. It's almost trivial to come up with a build system. Like you said, it boils down to not much more than interpreting a simple sequence of steps, typically doing nothing more than just triggering shell execution directly or indirectly. A basic build system is a few days work for a talented programmer! Heck, a mere shell script will do in a pinch... Notice however how the industry is slowly consolidating on Docker and Kubernetes. I suspect that one reason for this is that these are hard technologies. Not just anyone can "whip up" a container build and orchestration system in a short period of time. Hence, there's only a handful of container systems commonly used in the wild, which has resulted in standardisation. Precisely the type of standardisation that you (and many others) have been craving. TL;DR: The most complex system will become the CI system standard to rule them all because it's complex.
- lispm 5y ago
  ANSI Common Lisp was the first standardized OOP language (1994) with its integrated Common Lisp Object System. The project to specify this object system took several years and a group of core designers, plus a larger feedback group.
  - jiggawatts 5y ago
    specification != implementation
    - lispm 5y ago
      The design & feedback was driven together with a portable implementation: Portable Common Loops. The team developed that implementation to validate their design ideas and to get practical feedback from a broader community.

yebyen 5y ago

> doesn't let you run paramaterized builds in their web UI? It doesn't? https://github.blog/changelog/2021-11-10-github-actions-input-types-for-manual-workflows/ https://github.blog/changelog/2021-11-10-github-actions-inpu... Maybe it didn't 6 months ago, but it seems this is an option now, and well-documented. This is a topic of interest for me, I presented on Jenkins and GitOps to an audience at KubeCon who I suspect are almost all interested in moving away from Jenkins, or have been told to be interested in switching from Jenkins to something else, and I tried to get the idea across that they probably don't really need to switch the workflow tool even if it's ancient, ... But maybe should consider subbing out some of the important fiddly bits underneath it (like, I assume the vast majority of Jenkins users are building images with Docker, and if they're running on Kubernetes, they're many of them wondering what they will do, or how long they really have before they have to start worrying about the deprecation of dockershim and how their lives are going to change when their clusters won't be running Docker under the hood anymore?) I was actually arguing for a tool like Porter.sh to come in and make the boundaries of "what's in a build" super neat and tidy, organized, but also limited so the next time they feel compelled to switch workflow tooling, it will be a non-issue and can be over and finished inside of a single day's work. The problem isn't that your workflow tool is too old, it's that you've jammed too much arbitrary complexity inside of it, probably because the right abstraction was not made available to you at the time. "Switching off of Jenkins" just means building a second system and it comes with all the baggage of "second system syndrome" to do so. Sure it's difficult switching from Jenkins, when you've built this gigantic pipeline with 18 branches and 12 of them run in parallel, half of them are configured with different options passed to the docker build tool, half of them must use buildx, and the third half of them are unmaintained so we don't go in there... so put some guard rails up around the hard parts! And get somebody in there to take care of those cobwebs.

0xbadcafebee 5y ago
I'm glad it's an option now! That was crazy that it took 3 years for this feature to show up, though. I think we need a few more "12 Factor App"-style guidelines for modern systems. 12 Factor goes a long way to abstract away the tendency for implementation lock-in. But we can probably create a few more guidelines specific to CI so it's portable. Use OAuth, use the same authorization layer as your VCS, tie secrets and artifacts to the VCS repo, make every plugin a Docker container. (I'm stealing these concepts from Drone because it has the best cloud-native design I've ever seen) The final unsolved bit is how to manage a DAG of jobs and hooks around every event in a portable way. We probably need a universal CI spec and API, but maybe that's too specific. Also: holy crap, I don't think I've seen Porter.sh before, I love the idea! Definitely going to look into that
steveklabnik 5y ago
That page 404s for me.
- yebyen 5y ago
  Hmm, I wonder how the link got changed... it worked when I posted it? Pretty sure? Someone else found the link.
  - steveklabnik 5y ago
    ... now it no longer does. How odd.
bewuethr 5y ago
Link should be https://github.blog/changelog/github-actions-input-types-for-manual-workflows/ https://github.blog/changelog/github-actions-input-types-for...

qbasic_forever 5y ago

It's why I like tekton, it's a CI system for kubernetes so you're letting kubernetes solve all those problems. I have far more faith that k8s will be alive and kicking and not breaking core things like secrets, logs, container execution, etc. in painful ways in the future than anything else. In general though I try to put all the CI work into a simple shell or similar script and then just configure whatever CI system is in use to call it in the appropriate environment (in a container, bespoke VM, etc.). I agree putting all kinds of complexity in a unique CI system is just asking for trouble down the road as it's now basically a hard dependency for your code to be shippable.

jiggawatts 5y ago

A brilliant insight I came across once was that a low barrier to entry can prevent standardisation and quality. The original example was that in LISP it's so easy to add Object Oriented functionality that it's a mere afternoon's work for a graduate student. For comparison, it took Bjarne Stroustrup several years to extend C with objects to create C++. Hence there are only about 4-5 such languages out there, of which only two are popular. This means that every C++ program is object oriented in the "same way", whereas random OO LISP modules are incompatible and you can't merged them into one cohesive program or reuse the components nicely. It's almost trivial to come up with a build system. Like you said, it boils down to not much more than interpreting a simple sequence of steps, typically doing nothing more than just triggering shell execution directly or indirectly. A basic build system is a few days work for a talented programmer! Heck, a mere shell script will do in a pinch... Notice however how the industry is slowly consolidating on Docker and Kubernetes. I suspect that one reason for this is that these are hard technologies. Not just anyone can "whip up" a container build and orchestration system in a short period of time. Hence, there's only a handful of container systems commonly used in the wild, which has resulted in standardisation. Precisely the type of standardisation that you (and many others) have been craving. TL;DR: The most complex system will become the CI system standard to rule them all because it's complex.

lispm 5y ago
ANSI Common Lisp was the first standardized OOP language (1994) with its integrated Common Lisp Object System. The project to specify this object system took several years and a group of core designers, plus a larger feedback group.
- jiggawatts 5y ago
  specification != implementation
  - lispm 5y ago
    The design & feedback was driven together with a portable implementation: Portable Common Loops. The team developed that implementation to validate their design ideas and to get practical feedback from a broader community.