4 ms·

People get set in their ways. Sometimes entire industries need a shake-up. This never occurs voluntarily, and people will wail and thrash about even as you try

People get set in their ways. Sometimes entire industries need a shake-up. This never occurs voluntarily, and people will wail and thrash about even as you try to help them get out of their rut. Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A typical modern CI/CD pipeline is like a smorgasboard of worst-case scenarios for performance. Let me list just some of them: - Everything is typically done from scratch, with minimal or no caching. - Synchronous I/O from a single thread, often to a remote cloud-hosted replicated disk... for an ephemeral build job. - Tens of thousands of tiny files, often smaller than the physical sector size. - Layers upon layers of virtualisation. - Many small HTTP downloads, from a single thread with no pipelining. Often un-cached despite being identified by stable identifiers -- and hence infinitely cacheable safely. - Spinning up giant, complicated, multi-process workflows for trivial tasks such as file copies. (CD agent -> shell -> cp command) Bonus points for generating more kilobytes of logs than the kilobytes of files processed. - Repeating the same work over and over (C++ header compilation). - Generating reams of code only to shrink it again through expensive processes or just throw it away (Rust macros). I could go on, but it's too painful...

mschuster91 3y ago
> Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A lot of what you describe originates from lessons learned in "classic" build environments: - broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step) - someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers - its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke - OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software - attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials - colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better: - every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git) - no SSH or other access to the k8s nodes possible - every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented
- gpderetta 3y ago
  You forgot: - you can get lunch while you wait for your build to finish.
  - mschuster91 3y ago
    Not if your pipeline is decent. Parallelize and cache as much as you can. The only stack that routinely throws wrenches into pipeline optimization is Maven. I'd love to run, say, Sonarqube, OWASP Dependency Checker, regular unit tests and end-to-end tests in parallel in different containers, but Maven - even if you pass the entire target / */target folders through - insists on running all steps prior to the goal you attempt to run. It's not just dumb and slow, it makes the runner images large AF because they have to carry everything needed for all steps in one image, including resources like RAM and CPU.
- justinclift 3y ago
  > OS package upgrades breaking things Heh. docker.io package on Ubuntu did this recently, whereby it stopped honouring the "USER someuser" clause in Dockerfiles. Completely breaks docker builds. No idea if it's fixed yet, we just updated our systems to not pull in docker.io 20.10.25-0ubuntu1~20.04. or newer.
  - ilyt 3y ago
    Docker developers being clueless, what else is new...
- slowmovintarget 3y ago
  Indeed: Optimize to reduce developer time spent on bad builds first. One of the rules of this approach is to filter all extraneous variable input likely to disrupt the build results, especially and including artifacts from previous failed builds.
- jiggawatts 3y ago
  You’re the perfect example of the cranky experienced developer stuck in their ways and fighting against better solutions. Most of the issues you’ve described are consequences of yet more issues such as not caching with the correct cache key. I argue that all of the problems are eminently solvable. You’re arguing for leaving massive issues in the system because of… other massive issues. Only one of these two approaches to problem solving gets to a solution without massive problems.
api 3y ago
We've gotten very good at using containerization and virtualization to isolate and abstract away ugliness, and as a result we have been able to build unspeakable towers of horror that would have been impossible without these innovations.
jrockway 3y ago
A few weeks ago, I decided to lock myself in my apartment and write a build system. I have always felt like developers wait ages for Docker, and I wanted to see why. (Cycle times on the app I develop are a minute for my crazy combination of shell scripts that I use, or up to 5 minutes for what most of my teammates do. This is incomprehensibly unproductive.) It turns out, it's all super crazy at every level. Things like Docker use incredibly slow algorithms like SHA256 and gzip by default. For example, it takes 6 seconds to gzip a 150MB binary, while zstd -fast=3 achieves the same ratio and does it in 100 milliseconds! The OCI image spec allows Zstandard compression, so this is something you can just do to save build time and container startup time. (gzip, unsurprisingly, is not a speed demon when decompressing either.) SHA256, used everywhere in the OCI ecosystem, is also glacial; a significant amount of CPU used by starting or building containers is just running this algorithm. Blake3 is 17 times faster! (Blake2b, a fast and more-trusted hash than blake3, is about 6x faster.) But unfortunately, Docker/OCI only support SHA256, so you are stuck waiting every time you build or pull a container. (On the building side, you actually have to compute layer SHA256s twice; once for the compressed data and once for the uncompressed data. I don't know what happens if you don't do this, I just filled in every field the way the standard mandated and things worked.) This was on HN a couple years ago and was a real eye opener for me: https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ There are also things that Dockerfiles preclude, like building each layer in parallel. I don't use layers or shell commands for anything; I just put binaries into a layer. With a builder that doesn't use Dockerfiles, you can build all the layers in parallel and push some of the earlier layers while the later ones are building. (One of the reasons I wrote my own image assembler is because we produce builds for each architecture. The build machine has to run an arm64 qemu emulator so that a Dockerfile-based build can run `[` to select the right third-party binary to extract. This is crazy to me; the decision is static and unchanging, so no code needs to be run. But I know that it's designed for stuff like "FROM debian; RUN apt-get update; RUN apt-get upgrade" which is ... not needed for anything I do.) The other thing that surprises me about pushing images is how slow a localhost->localhost container push is. I haven't looked into why because the standard registry code makes me cry, but I plan to just write a registry that stores blobs on disk, share the disk between the build environment and the k8s cluster (hostpath provisioner or whatever), and have the build system just write the artifacts it's building into that directory; thus there is no push step required. When the build is complete, the artifacts are available for k8s to "pull". The whole thing is a work in progress, but with a week of hacking I got the cycle time down from 1 minute to about 5 seconds, and many more improvements are available. (Eventually I plan to build everything with Bazel and remote execution, but I needed the container builder piece for multi-architecture releases; Bazel will have to be invoked independently for each architecture because of its design, and then the various artifacts have to be assembled into the final image list.)

mschuster91 3y ago

> Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently. A lot of what you describe originates from lessons learned in "classic" build environments: - broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step) - someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers - its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke - OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software - attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials - colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better: - every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git) - no SSH or other access to the k8s nodes possible - every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented

gpderetta 3y ago
You forgot: - you can get lunch while you wait for your build to finish.
- mschuster91 3y ago
  Not if your pipeline is decent. Parallelize and cache as much as you can. The only stack that routinely throws wrenches into pipeline optimization is Maven. I'd love to run, say, Sonarqube, OWASP Dependency Checker, regular unit tests and end-to-end tests in parallel in different containers, but Maven - even if you pass the entire target / */target folders through - insists on running all steps prior to the goal you attempt to run. It's not just dumb and slow, it makes the runner images large AF because they have to carry everything needed for all steps in one image, including resources like RAM and CPU.
justinclift 3y ago
> OS package upgrades breaking things Heh. docker.io package on Ubuntu did this recently, whereby it stopped honouring the "USER someuser" clause in Dockerfiles. Completely breaks docker builds. No idea if it's fixed yet, we just updated our systems to not pull in docker.io 20.10.25-0ubuntu1~20.04. or newer.
- ilyt 3y ago
  Docker developers being clueless, what else is new...
slowmovintarget 3y ago
Indeed: Optimize to reduce developer time spent on bad builds first. One of the rules of this approach is to filter all extraneous variable input likely to disrupt the build results, especially and including artifacts from previous failed builds.
jiggawatts 3y ago
You’re the perfect example of the cranky experienced developer stuck in their ways and fighting against better solutions. Most of the issues you’ve described are consequences of yet more issues such as not caching with the correct cache key. I argue that all of the problems are eminently solvable. You’re arguing for leaving massive issues in the system because of… other massive issues. Only one of these two approaches to problem solving gets to a solution without massive problems.

api 3y ago

We've gotten very good at using containerization and virtualization to isolate and abstract away ugliness, and as a result we have been able to build unspeakable towers of horror that would have been impossible without these innovations.

jrockway 3y ago

A few weeks ago, I decided to lock myself in my apartment and write a build system. I have always felt like developers wait ages for Docker, and I wanted to see why. (Cycle times on the app I develop are a minute for my crazy combination of shell scripts that I use, or up to 5 minutes for what most of my teammates do. This is incomprehensibly unproductive.) It turns out, it's all super crazy at every level. Things like Docker use incredibly slow algorithms like SHA256 and gzip by default. For example, it takes 6 seconds to gzip a 150MB binary, while zstd -fast=3 achieves the same ratio and does it in 100 milliseconds! The OCI image spec allows Zstandard compression, so this is something you can just do to save build time and container startup time. (gzip, unsurprisingly, is not a speed demon when decompressing either.) SHA256, used everywhere in the OCI ecosystem, is also glacial; a significant amount of CPU used by starting or building containers is just running this algorithm. Blake3 is 17 times faster! (Blake2b, a fast and more-trusted hash than blake3, is about 6x faster.) But unfortunately, Docker/OCI only support SHA256, so you are stuck waiting every time you build or pull a container. (On the building side, you actually have to compute layer SHA256s twice; once for the compressed data and once for the uncompressed data. I don't know what happens if you don't do this, I just filled in every field the way the standard mandated and things worked.) This was on HN a couple years ago and was a real eye opener for me: https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ There are also things that Dockerfiles preclude, like building each layer in parallel. I don't use layers or shell commands for anything; I just put binaries into a layer. With a builder that doesn't use Dockerfiles, you can build all the layers in parallel and push some of the earlier layers while the later ones are building. (One of the reasons I wrote my own image assembler is because we produce builds for each architecture. The build machine has to run an arm64 qemu emulator so that a Dockerfile-based build can run `[` to select the right third-party binary to extract. This is crazy to me; the decision is static and unchanging, so no code needs to be run. But I know that it's designed for stuff like "FROM debian; RUN apt-get update; RUN apt-get upgrade" which is ... not needed for anything I do.) The other thing that surprises me about pushing images is how slow a localhost->localhost container push is. I haven't looked into why because the standard registry code makes me cry, but I plan to just write a registry that stores blobs on disk, share the disk between the build environment and the k8s cluster (hostpath provisioner or whatever), and have the build system just write the artifacts it's building into that directory; thus there is no push step required. When the build is complete, the artifacts are available for k8s to "pull". The whole thing is a work in progress, but with a week of hacking I got the cycle time down from 1 minute to about 5 seconds, and many more improvements are available. (Eventually I plan to build everything with Bazel and remote execution, but I needed the container builder piece for multi-architecture releases; Bazel will have to be invoked independently for each architecture because of its design, and then the various artifacts have to be assembled into the final image list.)