14 ms·
Modern CI is too complex and misdirected
- salawat 6y agoDing ding ding. We have a winner. All a CI pipeline is is an artifact generator. All a CD pipeline is is an artifact shuffler that can kick off dependent CI jobs. The rest is just as the author mentions. Remote Code Execution as a service.
- hinkley 6y agoBut what we have is CI tools with integrations to a million things they shouldn't probably have integrations to. Most of the build should be handled by your build scripts. Most of the deploy should be handled by deploy scripts. What's left for a CI that 'stays in its lane' is fetching, scheduling, and reporting, and auth. Most of them could stand to be doing a lot more scheduling and reporting, but all evidence points to them being too busy being distracted by integrating more addons. There are plenty of addons that one could write that relate to reporting (eg, linking commit messages to systems of record), without trying to get into orchestration that should ultimately be the domain of the scripts. Otherwise, how do you expect people to debug them? I've been thinking lately I could make a lot of good things happen with a Trac-like tool that also handled CI and stats as first class citizens.
- Smaug123 6y ago"Build Systems à la Carte" is not so much ringing a bell as shattering it with the force of its dong. https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... To expand, the OP ends with an "ideal world" that sounds to me an awful lot like someone's put the full expressive power of Build Systems à la Carte into a programmable platform, accessible by API.
- jacques_chester 6y agoNitpick: I think you meant to write "gong".
- rubiquity 6y agoCall me crazy, but I don't think they did...
- throwawayboise 6y agoWhat is the name for this phenomenon: Observation: X is too complex/too time-consuming/too error-prone. Reaction: Create X' to automate/simplify X. Later: X' is too complex.
- temporama1 6y agoSoftware Development
- stepbeek 6y agoI don't know the name of the fallacy/phenomenon, but it always reminds me of this xkcd: https://xkcd.com/927/ https://xkcd.com/927/
- blacktriangle 6y agoThe alternative to this seems even worse. To deal with the overcomplexity we get things like Boot and Webpack, which aren't even really build tools, but tools for building build tools.
- cryptonector 6y ago"tech"
- simonw 6y agoIt genuinely hadn't crossed my mind that a CI system and a build system were different things - maybe because I usually work in dynamic rather than compiled languages? I've used Jenkins, Circle CI, GitLab and GitHub Actions and I've always considered them to be a "remote code execution in response to triggers relating to my coding workflow" systems, which I think covers both build and CI.
- bencalam 6y agoSame boat. I am surprised that I had to to scroll so far down to see this comment.
- oblio 6y agoDid your dynamic languages not have some sort of build tool? Heck, Javascript has Gulp, Grunt, Webpack, a million others. Ruby has rake, Python pipenv/poetry, I guess. Python is actually kind of outlier, I guess you're expected to write Python scripts to manage your packaging and such.
- chatmasta 6y agoThere is a lot of valid criticism here, but the suggestion that "modern CI" is to blame is very much throwing the baby out with the bathwater. The GitHub Actions / GitLab CI feature list is immense, and you can configure all sorts of wild things with it, but you don't have to. At our company our `gitlab-ci.yml` is a few lines of YAML that ultimately calls a single script for each stage. We put all our own building/caching/incremental logic in that script, and just let the CI system call it for us. As a nice bonus that means we're not vendor locked, as our "CI system" is really just a "build system" that happens to run mostly on remote computers in response to new commits. It's not exactly the same as the local build system, because development requirements and constraints are often distinct from staging/prod build requirements, and each CI paltform has subtle differences with regards to caching, Docker registries, etc. But it uses a lot of the same underlying scripts. (In our case, we rely a lot on Makefiles, Docker BuildKit, and custom tar contexts for each image). Regarding GitHub actions in particular, I've always found it annoyingly complex. I don't like having to develop new mental models around proprietary abstractions to learn how to do something I can do on my own machine with a few lines of bash. I always dread setting up a new GHA workflow because it means I need to go grok that documentation again. Leaning heavily on GHA / GL CI can be advantageous for a small project that is using standardized, cookie-cutter approaches, e.g. your typical JS project that doesn't do anything weird and just uses basic npm build + test + publish. In that case, using GHA can save you time because there is likely a preconfigured workflow that works exactly for your use case. But as soon as you're doing something slightly different from the standard model, relying on the cookie-cutter workflows becomes inhibitive and you're better off shoving everything into a few scripts. Use the workflows where they integrate tightly with something on the platform (e.g. uploading an artifact to GitHub, or vendor-specific branch caching logic), but otherwise, prefer your own scripts that can run just as well on your laptop as in CI. To be honest, I've even started avoiding the vendor caching logic in favor of using a single layer docker image as an ad-hoc FS cache.
- jayd16 6y agoThis makes no sense to me. Modern build systems have reproducible results based on strict inputs. Modern CI/CD handles tasks that are not strictly reproducible. The continuous aspect also implies its integrated to source control. I guess I don't understand the post if its not just semantic word games based on sufficient use of the word sufficient. Maybe the point is to talk about how great Taskcluster is but the only thing mentioned is security and that is handled with user permissions in Gitlab and I assume Github. Secrets are associated with project and branch permissions, etc. No other advantage is mentioned in detail. Can someone spell out the point of this article?
- mlthoughts2018 6y agoMany CI systems try to strictly enforce hermetic build semantics and disallow non-idempotent steps from being possible. For example, by associating build steps with an exact source code commit and categorically disallow a repeat of a successful step for that commit.
- brown9-2 6y agoYour CI pipeline builds and tests your project, which is the same thing your build system does, except they are each using different specifications of how to do that. The author argues this is a waste. I think by introducing continuous deployment you are changing the topic from what the author wrote (which strictly referred to CI).
- slaymaker1907 6y agoOne big difference in my opinion is that a CI system can (and should) allow for guarantees about provenance. Ideally code signing can only be done for your app from the CI server. This allows users/services to guarantee that the code they are running has gone through whatever workflows are necessary to be built on the build server (most importantly that all code has been reviewed). As a principle, I consider any codebase which can be compromised by corrupting a single person to be inherently vulnerable. Sometimes this is ok, but this is definitely not ok for critical systems. Obviously it is better to have a stronger safety factor and require more groups/individuals to be corrupted, but there are diminishing returns considering it is assumed said individuals are already relatively trustworthy. Additionally, it is really surprising to me how many code platforms make providing even basic guarantees like this one impossible.
- gilbetron 6y agoIt is something we're definitely concerned about: https://www.datadoghq.com/blog/engineering/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto/ https://www.datadoghq.com/blog/engineering/secure-publicatio...
- pydry 6y agoIt's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
- gilbetron 6y agoJoe Beda (k8s/Heptio) made this same point in one of his TGI Kubernetes videos: https://youtu.be/M_rxPPLG8pU?t=2936 https://youtu.be/M_rxPPLG8pU?t=2936 I agree 100%. Every time I see "nindent" in yaml code, a part of my soul turns to dust.
- theptip 6y ago> Every time I see "nindent" in yaml code, a part of my soul turns to dust. Yup. For this reason it's a real shame to me that Helm won and became the lingua franca of composable/configurable k8s manifests. The one benefit of writing in static YAML instead of dynamic <insert-DSL / language>, is that regardless of primary programming language, everyone can contribute; more complex systems like KSonnet start exploding in first-use complexity.
- qbasic_forever 6y agoI wouldn't say helm has won, honestly. The kubectl tool integerated Kustomize into it and it's sadly way too underutilized. I think it's just that the first wave of k8s tutorials that everyone has learned from were all written when helm was popular. But now with some years of real use people are less keen on helm. There are tons of other good options for config management and templating--I expect to see it keep changing and improving.
- theptip 6y agoAre there good kustomize bases for things like Redis, Postgres, Mysql, etc? My impression is that most "cloud-native" projects ship raw manifests and Helm charts, or just helm charts. By "won" I just mean in terms of community mind-share, not that they built the best thing. But I could be out of date there. I do like kustomize, but the programming model is pretty alien, and they only recently added Components to let you template a commonly-used block (say, you have a complex Container spec that you want to stamp onto a number of different Deployments). Plus last I looked, kustomize was awkward when you actually do need dynamic templating, e.g. "put a dynamic annotation onto this Service to specify the DNS name for this review app". Ended up having to use another program to do templating which felt awkward. Maybe mutators have come along enough since I last looked into this though.
- donatj 6y agoI was really disappointed when GitHub Actions didn't more closely resemble Drone CI. Drone's configuration largely comes down to a Docker container and a script to run in said Docker container.
- mattacular 6y agoNot familiar with Drone but just pointing out Github Actions can be used for lots of stuff besides CICD. That's just the most popular and obvious usage. I actually prefer being able to grab some existing actions plugins rather than having to write every single build step into a shell script like with eg. Aws codePipeline for every app. You don't have to use them, though. You could have every step be just shell commands with Github Actions.
- throwaway823882 6y agoDrone is the ultimate example of simplicity in CI. Automatic VCS-provider OAuth, automatic authorization for repos/builds, native secrets, plugins are containers, server configuration is just environment variables, SQL for stateful backend, S3 for logs, and a yaml file for the pipelines. There's really nothing else to it. And when you need a feature like dynamic variables, they support things you're already familiar with, like Bourne shell parameter expansion. Whoever created it deeply understands KISS. I think the only reason it doesn't take over the entire world is the licensing. (It's not expensive at all considering what you're getting, but most companies would rather pay 10x to roll their own than admit that maybe it's worth paying for software sometimes)
- jrockway 6y agoI think that modern CI is actually too simple. They all boil down to "get me a Linux box and run a shell script". You can do anything with that, and there are a million different ways to do everything you could possibly want. But, it's easy to implement, and every feature request can be answered with "oh, well just apt-get install foobarbaz3 and run quuxblob to do that." A "too complex" system, would deeply integrate with every part of your application, the build system, the test runner, the dependencies would all be aware of CI and integrate. That system is what people actually want ("run these browser tests against my Go backend and Postgres database, and if they pass, send the exact binaries that passed the tests to production"), but have to cobble together with shell-scripts, third-party addons, blood, sweat, and tears. I think we're still in the dark ages, which is where the pain comes from.
- Already__Taken 6y agoHard agree. I've been using gitlab CI/CD for a long time now. I almost want to say it's been around longer or as long as docker? It has a weird duality of running as docker images, but also really doesn't understand how to use container images IN the process. Why volumes don't just 1:1 map to artifacts and caching, always be caching image layers to make things super fast etc.
- sytse 6y ago"I almost want to say it's been around longer or as long as docker?" I had to look it up but GitLab CI has been around longer than Docker! Docker was released as open-source in March 2013. GitLab CI was first released in 2012.
- noir_lord 6y agoPretty much. Docker in my experience is the same way, people see docker as the new hotness then treat it like a Linux box with a shell script (though at least with the benefit you can shoot it in the head). One of the other teams had an issue with reproducibility on something they where doing so I suggested that they use a multistage build in docker and export the result out as an artefact they could deploy, they looked at me like I’d grown a second head yet have been using docker twice as long as me, though I’ve been using Linux for longer than all of them combined. It’s a strange way to solve problems all around when you think about what it’s actually doing. Also feels like people adopt tools and cobble shit together from google/SO, what happened to RTFM. If I’m going to use a technology I haven’t before the first thing I do is go read the actual documentation - I won’t understand it all on the first pass but it gives me an “index”/outline I can use when I do run into problems, if I’m looking at adopting a technology I google “the problem with foobar” not success stories, I want to know the warts not the gloss. It’s the same with books, I’d say two 3/4 of the devs I work with don’t buy programming books, like at all. It’s all cobbled together knowledge from blog posts, that’s fine but a cohesive book with a good editor is nearly always going to give your a better understanding than piecemeal bits from around the net, that’s not to say specific blog posts aren’t useful but the return on investment on a book is higher (for me, for the youngsters, they might do better learning from tiktok I don’t know..).
- nohuck13 6y ago"Bazel has remote execution and remote caching as built-in features... If I define a build... and then define a server-side Git push hook so the remote server triggers Bazel to build, run tests, and post the results somewhere, is that a CI system? I think it is! A crude one. But I think that qualifies as a CI system." --- Absolutely. The advisability of rolling your own CI aside, treating CI as "just another user" has real benefits, and this was a pleasant surprise for me when using Bazel. When your run the same build command (`say bazel test //...`) across development and CI, then: - you get to debug your build pipeline locally like code - the CI DSL/YAML files mostly contain publishing and other CI-specific information (this feels right) - the ability of a new user to pull the repo, build, and have everything just work, is constantly being validated by the CI. With a bespoke CI environment defined in a Docker image or YAML file this is harder. - tangentially: the remote execution API [2] is beautiful in its simplicity it's doing a simple core job. [1] OTOH: unless you have a vendor-everything monorepo like Google, integrating with external libraries/package managers is unnatural; hermetic toolchains are tricky; naively-written rules end up system-provided utilities that differ by host, breaking reproducibility, etc etc. [2] https://github.com/bazelbuild-remote-apis/blob/master/build/bazel/remote/execution/v2/remote_execution.prot https://github.com/bazelbuild-remote-apis/blob/master/build/...
- qznc 6y agoHow does Bazel deal with different platforms? For example, run tests on Windows, BSD, Android, Raspberry Pi, RISCv5, or even custom hardware?
- EdSchouten 6y agoPretty well! You can set up a build cluster that provides workers for any of these different platforms. Each of these platforms is identified by a different set of label values. Then you can run Bazel on your personal system to 'access' any of those platforms to run your build actions or tests. In other words: A 'bazel test' on a Linux box can trigger the execution of tests on a BSD box. (Full transparency: I am the author of Buildbarn, one of the major build cluster implementations for Bazel.)
- 6y ago
- tauntz 6y agoI built a CI service many moons ago (was https://greenhouseci.com https://greenhouseci.com back then) on the premise that _most_ companies don't actually require a hugely complicated setup and don't actually need all the flexibility / complexity that most CI systems have. After talking to tens and tens of companies with apparent complex CI requirements, I still stand by that assertion. When drilling down into _why_ they say they need all that configurability, it's sometimes as easy as "but FAANG is doing X so my 5 person team needs it as well".
- twoslide 6y agoAm I too old fashioned in thinking it’s good to define an acronym the first time it’s used? I think many well educated readers wouldn’t know CI
- tremon 6y agoIf you're in software development and haven't heard about Continuous Integration, you're definitely old-fashioned. Although fashion moves fast and breaks things here.
- mmphosis 6y agoToo Many Acronyms (TMA) Continuous Integration (CI) Continuous Delivery or Deployment (CD) Domain-Specific Language (DSL) Structured Query Language (SQL) YAML Ain't Markup Language (YAML) Windows Subsystem for Linux (WSL) Portable Operating System Interface (POSIX) Berkeley Software Distribution (BSD) Reduced Instruction Set Computer (RISC) Central processing unit (CPU) Operating System (OS) Quick EMUlator (QEMU) Total Addressable Market (TAM) Directed Acyclic Graph (DAG) Computer Science (CS) User Interface (UI) PR? workflow (Pull Request, Purchase Request, Public Relations, Puerto Rico) GitHub Actions (GHA) GitLab? or Graphics Library (GL) Facebook Apple Amazon Netflix Google (FAANG) The only acronym I care about is B: alias b='code build'
- hctaw 6y agoI've spent a ton of time thinking/working on this problem myself. I came to roughly the same conclusions about a year ago except for a few things: - Build system != CI. Both are in the set of task management. Centralized task management is in the set of decentralized tasks. Scheduling centralized tasks is easy, decentralized is very hard. Rather than equating one to the other, consider how a specific goal fits into a larger set relative to how tasks are run, where they run, what information they need, their byproducts, and what introspection your system needs into those tasks. It's quite different between builds and CI, especially when you need it decentralized. - On the market: What's the TAM of git? That's what we're talking about when it comes to revamping the way we build/test/release software. - There's a perverse incentive in CI today, which is that making your life easier costs CI platforms revenue. If you want to solve the business problem of CI, solve this one. - There are a number of NP Hard problems in the way of a perfect solution. Cache invalidation and max cut of a graph come to mind. - I don't know how you do any of this without a DAG. Yet, I don't know how you represent a DAG in a distributed system such that running tasks through it remain consistent and produce deterministic results. - Failure is the common case. Naive implementations of task runners assume too much success, and recovery from failure is crucial to making something that doesn't suck.
- cratermoon 6y agoHe doesn't mention Concourse, but let me say the complexity there is beast.
- jacques_chester 6y agoI've heard Concourse team members describe it by analogy to make, FWIW. One thing Concourse does well that I haven't seen replicated is strictly separating what is stateful and what isn't. That makes it possible to understand the history of a given resource without needing a hundred different specialised instruments.
- cratermoon 6y agoYes, if make threw away all the context and everything you had done prior to a target completing, and only gave the outputs of the target to the next target.
- godfryd2 6y agoThere is a new CI solution: https://kraken.ci https://kraken.ci. Workflows are defined in Starlark/Python. They can be arranged in DAG. Beside providing base CI features it also has features supporting big scale testing that are missing in other systems.
- bob1029 6y agoWe got tired of using external tools that were not well-aligned with our build/deployment use cases - non-public network environments. GitHub Actions, et. al. cannot touch the target environments that we deploy our software to. Our customers are also extremely wary of anything cloud-based, so we had to find an approach that would work for everyone. As a result, we have incorporated build & deployment logic into our software as a first-class feature. Our applications know how to go out to source control, grab a specified commit hash, rebuild themselves in a temporary path, and then copy these artifacts back to the working directory. After all of this is completed, our application restarts itself. Effectively, once our application is installed to some customer environment, it is like a self-replicating organism that never needs to be reinstalled from external binary artifacts. This has very important security consequences - we build on the same machine the code will execute on, so there are far fewer middle men who can inject malicious code. Our clients can record all network traffic flowing to the server our software runs on and definitively know 100% of the information which constitutes the latest build of their application. Our entire solution operates as a single binary executable, so we can get away with some really crazy bullshit that most developers cannot these days. Putting your entire app into a single self-contained binary distribution that runs as a single process on a single machine has extremely understated upsides these days.
- lstamour 6y agoSounds like Chrome, minus the build-on-the-customer's-machine part. Or like Homebrew, sort of. Also sounds like a malware dropper. That said, it makes sense. I would decouple the build-on-the-customer's machine part from the rest, having a CI system that has to run the same way on every customer's machine sounds like a bit of a nightmare for reproducibility if a specific machine has issues. I'd imagine you'd need to ship your own dependencies and set standards on what version of Linux, CPU arch and so on you'd support. And even then I'd feel safer running inside overlays like Docker allows for, or how Bazel sandboxes on Linux. Also reminds me a bit of Istio or Open Policy Agent in that both are really apps that distribute certificates or policy data and thus auto-update themselves?
- bob1029 6y ago
- gran_colombia 6y agoThe complaint is invalid. CI pipelines are build systems.
- jdfellow 6y agoI've been wishing for one of my small projects (3 developers) for some kind of "proof of tests" tool that would allow a developer to run tests locally, and add some sort of token to the commit message assuring that they pass. I could honestly do without a ton of the remote-execution-as-a-service in my current GitLab CI setup, and be happy to run my deployment automation scripts on my own machine if I could have some assurance of code quality from the team without CI linters and tests running in the cloud (and failing for reasons unrelated to the tests themselves half of the time).
- zomglings 6y agoYou are right. Small teams absolutely do not need to execute code remotely, especially if the cost is having an always on job server. My team writes test output to our knowledge base: bugout trap --title "$REPO_NAME tests: $(date -u +%Y%m%d-%H%M)" --tags $REPO_NAME,test,zomglings,$(git rev-parse HEAD) -- ./test.sh This runs test.sh and reports stdout and stderr to our team knowledge base with tags that we can use to find information later on. For example, to find all failed tests for a given repo, we would perform a search query that looked like this: "#<repo> #test !#exit:0". The knowledge base (and the link to the knowledge base entry) serve as proof of tests. We also use this to keep track of production database migrations.
- oblio 6y ago> You are right. Small teams absolutely do not need to execute code remotely, especially if the cost is having an always on job server. Super debatable. You should have some builds and tests run in a clean environment. Maybe you could do it in a Docker container. But otherwise, you want a remote server. Devs mess up their environment too much for a regular dev machine to be a reliable build machine.
- chriswarbo 6y agoGit pre-commit hooks can run your tests, but that's easy to skip. I don't know about a "proof of test" token. Checking such a token would presumably require some computation involving the repo contents; but we already have such a thing, it's called 'running the test suite'. A token could contain information about branches taken, seeds for any random number generators, etc. but we usually want test suites to be deterministic (hence not requiring any token). We could use such a token in property-based tests, as a seed for the random number generator; but it would be easier to just use one fixed seed (or, we could use the parent's commit ID).
- alexcroox 6y agoThat's why I love Buddy (buddy.works), you build your CI pipelines with a UI, all the config and logic is all easily configurable without having to know the magic config key/value combination. Need to add a secrets file or private key; just add it to the "filesystem" and it'l be available during the run, no awkwardly base64ing contents into an environment string. Unfortunately I have to use github actions/CircleCi for my iOS deployments still, but I read MacOS container support is coming soon.
- epage 6y agoAnything in CS can be generalized to its purest, most theoretical forms. The question is how usable is it and how much work does it take to get anything done. See - https://danstroot.com/2018/10/03/hammer-factories/ https://danstroot.com/2018/10/03/hammer-factories/ - https://web.archive.org/web/20120427101911/http://jacksonfish.com/blog/2012/03/12/the-day-bill-gates-called-me-rude-and-other-lessons-in-user-experience/ https://web.archive.org/web/20120427101911/http://jacksonfis... Bazel, for example, is tailored to the needs of reproducible builds and meets its audience where it is at, on the command line. People want fast iteration time and only occasionally need "everything" ran. Github Actions is tailored for completeness and meets is audience where its at, the PR workflow (generally, a web UI). The web UI is also needed for visualizing the complexity of completeness. I never find myself reproducing my build in my CI but do find I have a similar shape of needs in my CI but in a different way. Some things more tailored to CI that wouldn't fit within the design of something like Bazel include - Tracking differences in coverage, performance, binary bloat, and other "quality" metrics between a target branch and HEAD - Post relevant CI feedback directly on the PR
- choeger 6y agoThe biggest problem with any CI system is that you need an execution environment. Changing this environment should be the same as changing the code. Docker (or rather podman) has given us the tools to do this. Now if CI systems would allow me to build that container image myself, I could pretty much guarantee that local build/tests and CI build/tests can run inside the same environment. I hacked something like this for gitlab but it's ugly and slow. So in conclusion, I think that CI systems should expect container creation, some execution inside that container, and finally some reporting or artifact generation from the results.
- mlthoughts2018 6y agoDocker / containers are necessary but not sufficient. For example, in a machine learning CI / CD system, there could be a fundamental difference between executing the same step, with the same code, on CPU hardware vs GPU hardware.
- 10ko 6y agoI cannot stop thinking this guy is describing what we do at https://garden.io https://garden.io. He seems to go on describing the Stack Graph and the build/test/deploy/task primitives, the unified configuration between environments, build and test results caching, the platform agnosticism (even though we are heavy focused on kubernetes) and the fact that CI can be just a feature, not a product on itself. One thing I definitely don't agree with is: "The total addressable market for this idea seems too small for me to see any major player with the technical know-how to implement and offer such a service in the next few years." We just published the results from an independent survery we commissioned last week and one of the things that came out is: it doesn't matter the size of the company, the amount of hours teams spend mantaining this overly complex build systems, CI systems, Preview/Dev environments etc. is enormous and often is object of the biggest complaints across teams of Tech organizations. So yeah, I agree with the complexity bit but I think the author is overly positive about the current state of the art, at least in the cloud native world.
- mlthoughts2018 6y agoIsn’t Drone also an example of the author’s ideal solution?
- sly010 6y agoRe: correct language/abstraction At the highest level you want a purely functional DSL with no side effects. Preferably one that catches dependency cycles so it halts provably. On the lowest level, however, all your primitives are unix commands that are all about side effects. Yet, you want them to be reproducible, or at least idempotent so you can wrap them in the high level DSL. So you really need to separate those two worlds, and create some sort of "runtime" for the low level 'actions' to curb the side effects. * Even in the case of bazel, you have separate .bzl and BUILD files. * In the case of nix, you have nix files and you have the final derivation (a giant S expression) * In the case of CI systems and github actions, you have the "actions" and the "gui". Re: CI vs build system, I guess the difference is that build systems focus on artifacts, while CI systems also focus on side effects. That said, there are bazel packages to push docker images, so it's certainly a very blurry line.
- throwaway894345 6y ago> Re: CI vs build system, I guess the difference is that build systems focus on artifacts, while CI systems also focus on side effects. That said, there are bazel packages to push docker images, so it's certainly a very blurry line. I think the CI and build system have basically the same goals, but they're approaching the problem from different directions, or perhaps it's more accurate to say that "CI" is more imperative while build systems are more declarative. I really want a world with a better Nix or Bazel. I think purely functional builds are always going to be more difficult than throwing everything in a big side-effect-y container, but I don't think they have to be Bazel/Nix-hard.
- sly010 6y agoOut of curiosity, what's hard about bazel? From my experience the main issue is interoperability with third party build systems. i.e. using a cmake library that was not manually bazeled by someone.
- throwaway894345 6y agoIt’s been a few years since I tried to use it, but at the time the documentation was sparse and misleading. I couldn’t tell the right way to write my own rules and there were several concepts which were muddled. Some of the docs suggested I could extend it by implementing my own rules or macros and other docs suggested I had to delve into the extension interface which was a mess of convoluted Java (the kind of OOP that OOP advocates swear is Not True OOP). It certainly seemed like Bazel supports certain kinds of projects roughly aligned to languages, and if you needed to do anything interesting like codegen the answer was not at all obvious. The worst was that the docs all said that Bazel supported Python 3, but there were half a dozen different flags pertaining to Python 3, and after trying every permutation and asking for help, I discovered that the docs were simply wrong and Python 3 support had never actually been implemented due to various issues. It seems you need deep familiarity with Bazel in order to use it in even a basic capacity.
- lenkite 6y agoI so oppressed with YAML chosen as the configuration language for mainstream CI systems. How do people manage to live with this ? I always make mistakes - again and again. And I can never keep anything in my head. It's just not natural. Why couldn't they choose a programming language ? Restrict what can be done by all means, but something that has a proper parser, compiler errors and IDE/editor hinting support would be great. One can even choose an embedded language like Lua for restricted execution environments. Anything but YAML!
- kzrdude 6y agoI guess YAML has the best solution to nested scopes, just indentation
- zaat 6y agoVS code with the prettier extension is an IDE with hinting, parser and immediately show when you have (not compiler but) errors. If there is an extension for your CI system try installing it too.
- stitched2gethr 6y agoThis is what chef did and I enjoyed it, but it seems that's not the way most systems went.
- 0xbadcafebee 6y agoYAML is not a language, it's a data format. Why does nobody in the entire tech industry know the difference? I didn't even go to school and I figured it out. Most software today that uses YAML for a configuration file is taking a data format (YAML) applying a very shitty parser to create a data structure, and then feeding that data structure to a function, which then determines what other functions to call. There's no grammar, no semantics, no lexer, no operators, and no types, save those inherent to the data format it was encoded in (YAML). Sometimes they'll look like they include expressions, but really they're just function arguments.
- anonydsfsfs 6y agoThe gp isn't talking about YAML in itself, they're talking about YAML as it's used by mainstream CI systems. Github Actions, for example, encodes conditionals, functions, and a whole lot of other language elements in its YAML format. To say it's "just a data format" is like saying XSLT is "just a markup language" because it's written in XML.
- iudqnolq 6y agoI was thinking the other day that Jetpack compose or some other reactive framework might be the perfect build system. It has declaritivity by default, sane wrapping of IO, and excellent composability (just define a function!), plus of course a real language.
- neeleshs 6y agoAirflow is a pretty generic DAG execution platform. In fact, some people have built a CI (and CD) system with Airflow. https://engineering.ripple.com/building-ci-cd-with-airflow-gitlab-and-terraform-in-gcp/ https://engineering.ripple.com/building-ci-cd-with-airflow-g... https://airflowsummit.org/sessions/airflow-cicd/ https://airflowsummit.org/sessions/airflow-cicd/ etc
- thinkharderdev 6y agoI've always had a pipe dream of building a CI system on top of something like Airflow. A modern CD pipeline is just a long-running workflow, so it would be great to treat it just like any other data pipeline.
- qbasic_forever 6y agoWhy even have a difference between production and CI in the first place? I see a future where it's Kubernetes all the way down. My CI system is just a different production-like environment. The same job to deploy code to production is used to deploy code to CI. The same deployment that gets a pod running in prod will get a pod running in test under CI. Your prod system handles events from end-users, your CI system handles events from your dev environment and source repo. Everything is consistent and the same. Why should I rely on some bespoke CI system to re-invent everything that my production system already has to do?
- rosineygp 6y agothere are a lot of solutions for CI. I just create a small script and run using make https://github.com/rosineygp/mkdkr https://github.com/rosineygp/mkdkr in another words: https://xkcd.com/927/ https://xkcd.com/927/
- jugg1es 6y agoI always thought of CI to be a build server that pushes code directly to production automatically. I never liked how CI does away with the concept of a 'release' with a specific version. To me CI is synonymous with SaaS products, specifically products whose job is to serve individual client requests via a front end. I've never understood how CI is supposed to work with different types of software that isn't so transactional, or when the transactions are much longer running than a single HTTP request.
- lame88 6y agoI had to integrate Azure Pipelines and wanted to shoot myself in the face. The idea that you are simply configuring a pipeline yaml is just one big lie; it's code, in the world's shittiest programming language using YML syntax - code that you have no local runtime for, so you have to submit to the cloud like a set of punch cards to see that 10 minutes later it didn't work and to try again. Pipelines are code, pure and simple. The sooner we stop pretending it isn't, the better off we'll be.
- aliswe 6y agoYeah, agreed. Feels like programming a PHP/ASP website directly on the server through FTP, like we did in the 90's
- vladaionescu 6y agoThe vision of this article is a similar vision to earthly.dev, a company I have founded to pursue the issues presented here. We have built the build system part and are working on completing the vision with the CI part. We use buildkit underneath as a constraint solver and our workflows are heavily DAG based. Hundreds of CI pipelines run Earthly today. I don't fully agree with all the assumptions in the article, including with the fact that the TAM is limited here. CI has been growing at 19% CAGR and also I think there are possibilities for expanding into other areas once you are a platform.
- kvnhn 6y ago> In my ideal world, there is a single DAG dictating all build, testing, and release tasks. I've stewed on this as well, and I'll add my two cents: I'd argue this problem -- building and managing a DAG of tasks -- isn't just critical to build systems and CI. It's everywhere: cloud architecture, Dev/MLOps pipelines... I'd argue most programming is just building and managing a DAG of tasks (e.g. functions, and purposefully limiting to "acyclic" for this argument). This is why we always regress to Turing complete languages; they're great at building DAGs. So yes, I agree that some standard DAG language (probably Turing complete) would be great. But I'd extend it's reach into source code itself. Pass your DAG to Python, and it schedules and runs your tasks in the interpreter (or perhaps many interpreters). Pass your DAG to Kubernetes and it schedules and runs your tasks in containers. etc. inb4 Lisp