A few weeks ago, I decided to lock myself in my apartment and write a build system. I have always felt like developers wait ages for Docker, and I wanted to se

A few weeks ago, I decided to lock myself in my apartment and write a build system. I have always felt like developers wait ages for Docker, and I wanted to see why. (Cycle times on the app I develop are a minute for my crazy combination of shell scripts that I use, or up to 5 minutes for what most of my teammates do. This is incomprehensibly unproductive.) It turns out, it's all super crazy at every level. Things like Docker use incredibly slow algorithms like SHA256 and gzip by default. For example, it takes 6 seconds to gzip a 150MB binary, while zstd -fast=3 achieves the same ratio and does it in 100 milliseconds! The OCI image spec allows Zstandard compression, so this is something you can just do to save build time and container startup time. (gzip, unsurprisingly, is not a speed demon when decompressing either.) SHA256, used everywhere in the OCI ecosystem, is also glacial; a significant amount of CPU used by starting or building containers is just running this algorithm. Blake3 is 17 times faster! (Blake2b, a fast and more-trusted hash than blake3, is about 6x faster.) But unfortunately, Docker/OCI only support SHA256, so you are stuck waiting every time you build or pull a container. (On the building side, you actually have to compute layer SHA256s twice; once for the compressed data and once for the uncompressed data. I don't know what happens if you don't do this, I just filled in every field the way the standard mandated and things worked.) This was on HN a couple years ago and was a real eye opener for me: https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ There are also things that Dockerfiles preclude, like building each layer in parallel. I don't use layers or shell commands for anything; I just put binaries into a layer. With a builder that doesn't use Dockerfiles, you can build all the layers in parallel and push some of the earlier layers while the later ones are building. (One of the reasons I wrote my own image assembler is because we produce builds for each architecture. The build machine has to run an arm64 qemu emulator so that a Dockerfile-based build can run `[` to select the right third-party binary to extract. This is crazy to me; the decision is static and unchanging, so no code needs to be run. But I know that it's designed for stuff like "FROM debian; RUN apt-get update; RUN apt-get upgrade" which is ... not needed for anything I do.) The other thing that surprises me about pushing images is how slow a localhost->localhost container push is. I haven't looked into why because the standard registry code makes me cry, but I plan to just write a registry that stores blobs on disk, share the disk between the build environment and the k8s cluster (hostpath provisioner or whatever), and have the build system just write the artifacts it's building into that directory; thus there is no push step required. When the build is complete, the artifacts are available for k8s to "pull". The whole thing is a work in progress, but with a week of hacking I got the cycle time down from 1 minute to about 5 seconds, and many more improvements are available. (Eventually I plan to build everything with Bazel and remote execution, but I needed the container builder piece for multi-architecture releases; Bazel will have to be invoked independently for each architecture because of its design, and then the various artifacts have to be assembled into the final image list.)