5 ms·

What does fast CI even mean? CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that

What does fast CI even mean? CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that the cost of whatever CI runner relative to it should be nil. If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. A bit more on topic, if you're selling CI and your business fails it's because people you're providing no value - people can run their build scripts fine without you.

jve 3y ago
> If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them Yes and no. Caching stuff and knowing when to run tsc/clang/rustc also improves performance.
- deleted 3y ago
  [deleted]
- bluGill 3y ago
  Sure, but you have to be 100% perfect in guessing when the cache file must be rebuilt. If there is any situation at all where you use the cached file when you should have rebuilt your tool is not useful. Note that you can go too far. If you rebuild a file on linux even though the only changes were in a "#ifdef freebsd" (windows, mac...) section that is a waste of time, but not harmful. However we already have tools that do that.
  - esafak 3y ago
    This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.
    - duped 3y ago
      It's not a solved problem, very few builds in the wild are deterministic and reproducible. You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable. Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect. The other fundamental problem with this model is that it cannot model cyclical dependencies (a degenerate case: glibc needs python which needs glibc). Since you can't hash the output until you've hashed the input, and the input hash depends on the output hash you need a way to break the dependency cycle. Tools have varying ways of mitigating this but it's not exactly "solved" in an elegant sense. And finally, a lot of build steps are faster than hashing their input and outputs. So if you want the utmost performance you need heuristics or a model that allows for some things to be hashed and others to always be considered fresh.
      - danudey 3y ago
        > very few builds in the wild are deterministic and reproducible There's a difference between 'deterministic and reproducible' and 'predictable'. Sure I can (probably) build curl ten times and get ten technically different binaries, but the differences aren't relevant to the functioning of the actual result. Assuming my build environment isn't changing out from underneath me (i.e. if I'm using the same versions of the same dependencies on the same platform with the same compiler toolchain, and I'm not passing in different arguments or environment variables) then the end result should be functionally identical. > You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable. This is a false assertion. If I'm building (again, as an example) curl, then I don't need to be rebuilding the kernel, glibc, kerberos, libssh, gnutls, brotli, etc. all in order to get a working curl build; I just need to make sure the libraries and toolchains I have installed are valid. If I create a docker image with all of the dependencies and toolchains preinstalled, and then I build the exact same curl code inside of that docker container on two separate occasions, then the end result is going to be indistinguishable without hashing the build artifacts. > Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect. It doesn't need to be perfect, it just needs to be correct for the vast majority of cases (which are not somehow inherently broken already and just haven't failed yet). > a degenerate case: glibc needs python which needs glibc Because you're not rebuilding the entire world (in 99% of cases), this doesn't actually matter. If you're building glibc use the existing build of python that you already have installed, which is the same identical version to the one you used last time you built glibc. If I'm rebuilding glibc, do I also need to rebuild python? If the glibc code hasn't changed, but maybe python has so I get a different result? Well okay, has the python source changed? No? Okay, have the dependencies for python or glibc changed? No? Okay, well problem solved then. I'm not sure why people have this philosophy of "this problem isn't solved because of some imaginary, unrealistic corner case that either shouldn't exist in the first place, isn't actually a real problem, or that no one will run into". In 99% of cases it works, and if it's not right for a particular circumstance then just don't use it in that circumstance. Sometimes makefiles aren't the right tool. Sometimes a GUI isn't the right tool. Let's not argue that we shouldn't have GUIs because sometimes running a command on the CLI is easier, and let's not argue that an extremely basic caching system that makes sane assumptions that are almost always valid isn't a good idea. And the parent is right: if you correctly define your inputs and outputs, then the system works. What those inputs and outputs are is up to you to determine. Maybe for you, the outputs are a working build of glibc, and maybe the inputs are all of the tarballs necessary to build Linux from scratch and a working bootstrap environment, but if all of those inputs and that whole environment are identical to last time, what's the point of rebuilding any of them? Maybe there is one, but that's up to you to determine and model.
  - danudey 3y ago
    > you have to be 100% perfect in guessing when the cache file must be rebuilt Or, you make reasonable assumptions that are correct in most cases, and rely on the user to provide hints as to when the cache should be invalidated. Assuming your code, environment variables, and `args` are all factored into invalidating the cache, there are few situations in typical builds where using the cache is a bad idea. The biggest ones I can think of are if your build system is going out and fetching data itself to decide what to do, e.g. if you're downloading source code (e.g. a shared library) from an external source which may have updated it. That said, if you're just arbitrarily pulling down whatever the latest build of libcurl is or something, you're going to have a bad time eventually anyway, so you should either put a static version into an arg or just put the download into a separate step; at which point the cache will correctly invalidate when it's changed. Maybe if a build is only valid for 90 days from the build date or something (e.g. a preview build) and that's built into the binary at compile time, but in that case you can pass that in as an arg anyway and then your cache is going to be invalid every morning regardless. I'm not going to say it's never an issue, but the same code with the same environment and the same arguments, built on the same platform with the same versions of the same dependencies, using the same versions of the same tooling to run the same commands and copy the same artifacts to the same locations should very nearly never produce a valid output now which is not an equally valid output in the future unless it is fetching and acting on external information (which can be easily parameterized).
jcranmer 3y ago
Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.
- aidos 3y ago
  I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker. In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.
- alexvitkov 3y ago
  Incremental builds are something even enterprise build systems fail at. I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug. If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.
  - no_wizard 3y ago
    One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't. This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot
    - jcparkyn 3y ago
      Visual studio does give a fairly detailed explanation of what was compiled each time, in the output>build view.
  - pharmakom 3y ago
deleted 3y ago
[deleted]
jnwatson 3y ago
The only part of the Earthly pitch that resonated with me was the ability to run the CI locally. When debugging your CI, it is just faster to find problems when you can run on your own box. I have a gnarly GitHub Actions script that took forever to get right because the debug cycle was 10 minutes long.
- fireflash38 3y ago
  That's one of my biggest complaints about gitlab-ci. Theres basically no way to debug complex rules without just running it. They've got a way now to simulate for default branch, but that's like bare minimum necessary. No way to simulate based on vars or anything else. And no way to try locally, just gotta fucking send it live.
- ianwalter 3y ago
  I feel you on that one. There is also Act (https://github.com/nektos/act https://github.com/nektos/act) and Dagger.io.
  - jnwatson 3y ago
    I have a very complicated script and Act didn't work on it.
- smaudet 3y ago
  Yeah, the faster but is also a bit of a non-seller, they have to integrate the world/deal with bad build times that can't easily be parallelized (what if there are no modular dependencies, that your tool can understand?) except by tools that have nothing to do with their product. But there have been many times when debugging a CI has been necessary, having to figure out what has gone wrong on a remote system is often an issue...ofc often the remote system is itself the problem...
crabbone 3y ago
> CI is an overgrown shell script running your build and telling you when it fails. Oh, I wish it were that simple... But, I also don't know what they measured. But, I'll give you just one example. Jenkins default landing page is a disaster in terms of speed. It tries to display data on recent builds everywhere across your cluster. For even not so big clusters this can be hundreds to thousands of items to pull from individual nodes running those builds. I've managed to crash Jenkins more times than I could count by simply loading the landing page w/o a particular configuration added that prevents it from its default behavior. CI server will have its own database, typically, managing all sorts of CI entities: jobs, artifacts, users, secretes... It could be quite big and you'd do well taking care of proper indexing etc. CI will have its assortment of runners, which are often provisioned dynamically (think VM or Docker images that need to be deployed to runner nodes), distributing them across the cluster is also not an easy task, if you want that to work fast. Of course you also want to be able to distribute the artifacts across the cluster -- this also takes time and resources. CI will have its own bookkeeping necessary to, essentially, do garbage collection, reporting and self-diagnostic. In big enough clusters this stuff can have a very high latency, if you don't specifically work towards reducing that latency. > If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. Ever heard of ccache? But, really, this is also super-naive. To make builds fast in a distributed system that CI is you would also need to figure out how to distribute this cache, how to modularize the builds... You might have heard about how one of the most difficult things in programming is cache invalidation? -- That's only partly a joke.
jrockway 3y ago
The tricky part of CI is figuring out what work doesn't need to be done. That is how you save time.

jve 3y ago

> If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them Yes and no. Caching stuff and knowing when to run tsc/clang/rustc also improves performance.

deleted 3y ago
[deleted]
bluGill 3y ago
Sure, but you have to be 100% perfect in guessing when the cache file must be rebuilt. If there is any situation at all where you use the cached file when you should have rebuilt your tool is not useful. Note that you can go too far. If you rebuild a file on linux even though the only changes were in a "#ifdef freebsd" (windows, mac...) section that is a waste of time, but not harmful. However we already have tools that do that.
- esafak 3y ago
  This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.
  - duped 3y ago
    It's not a solved problem, very few builds in the wild are deterministic and reproducible. You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable. Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect. The other fundamental problem with this model is that it cannot model cyclical dependencies (a degenerate case: glibc needs python which needs glibc). Since you can't hash the output until you've hashed the input, and the input hash depends on the output hash you need a way to break the dependency cycle. Tools have varying ways of mitigating this but it's not exactly "solved" in an elegant sense. And finally, a lot of build steps are faster than hashing their input and outputs. So if you want the utmost performance you need heuristics or a model that allows for some things to be hashed and others to always be considered fresh.
    - danudey 3y ago
      > very few builds in the wild are deterministic and reproducible There's a difference between 'deterministic and reproducible' and 'predictable'. Sure I can (probably) build curl ten times and get ten technically different binaries, but the differences aren't relevant to the functioning of the actual result. Assuming my build environment isn't changing out from underneath me (i.e. if I'm using the same versions of the same dependencies on the same platform with the same compiler toolchain, and I'm not passing in different arguments or environment variables) then the end result should be functionally identical. > You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable. This is a false assertion. If I'm building (again, as an example) curl, then I don't need to be rebuilding the kernel, glibc, kerberos, libssh, gnutls, brotli, etc. all in order to get a working curl build; I just need to make sure the libraries and toolchains I have installed are valid. If I create a docker image with all of the dependencies and toolchains preinstalled, and then I build the exact same curl code inside of that docker container on two separate occasions, then the end result is going to be indistinguishable without hashing the build artifacts. > Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect. It doesn't need to be perfect, it just needs to be correct for the vast majority of cases (which are not somehow inherently broken already and just haven't failed yet). > a degenerate case: glibc needs python which needs glibc Because you're not rebuilding the entire world (in 99% of cases), this doesn't actually matter. If you're building glibc use the existing build of python that you already have installed, which is the same identical version to the one you used last time you built glibc. If I'm rebuilding glibc, do I also need to rebuild python? If the glibc code hasn't changed, but maybe python has so I get a different result? Well okay, has the python source changed? No? Okay, have the dependencies for python or glibc changed? No? Okay, well problem solved then. I'm not sure why people have this philosophy of "this problem isn't solved because of some imaginary, unrealistic corner case that either shouldn't exist in the first place, isn't actually a real problem, or that no one will run into". In 99% of cases it works, and if it's not right for a particular circumstance then just don't use it in that circumstance. Sometimes makefiles aren't the right tool. Sometimes a GUI isn't the right tool. Let's not argue that we shouldn't have GUIs because sometimes running a command on the CLI is easier, and let's not argue that an extremely basic caching system that makes sane assumptions that are almost always valid isn't a good idea. And the parent is right: if you correctly define your inputs and outputs, then the system works. What those inputs and outputs are is up to you to determine. Maybe for you, the outputs are a working build of glibc, and maybe the inputs are all of the tarballs necessary to build Linux from scratch and a working bootstrap environment, but if all of those inputs and that whole environment are identical to last time, what's the point of rebuilding any of them? Maybe there is one, but that's up to you to determine and model.
- danudey 3y ago
  > you have to be 100% perfect in guessing when the cache file must be rebuilt Or, you make reasonable assumptions that are correct in most cases, and rely on the user to provide hints as to when the cache should be invalidated. Assuming your code, environment variables, and `args` are all factored into invalidating the cache, there are few situations in typical builds where using the cache is a bad idea. The biggest ones I can think of are if your build system is going out and fetching data itself to decide what to do, e.g. if you're downloading source code (e.g. a shared library) from an external source which may have updated it. That said, if you're just arbitrarily pulling down whatever the latest build of libcurl is or something, you're going to have a bad time eventually anyway, so you should either put a static version into an arg or just put the download into a separate step; at which point the cache will correctly invalidate when it's changed. Maybe if a build is only valid for 90 days from the build date or something (e.g. a preview build) and that's built into the binary at compile time, but in that case you can pass that in as an arg anyway and then your cache is going to be invalid every morning regardless. I'm not going to say it's never an issue, but the same code with the same environment and the same arguments, built on the same platform with the same versions of the same dependencies, using the same versions of the same tooling to run the same commands and copy the same artifacts to the same locations should very nearly never produce a valid output now which is not an equally valid output in the future unless it is fetching and acting on external information (which can be easily parameterized).

jcranmer 3y ago

Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.

aidos 3y ago
I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker. In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.
alexvitkov 3y ago
Incremental builds are something even enterprise build systems fail at. I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug. If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.
- no_wizard 3y ago
  One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't. This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot
  - jcparkyn 3y ago
    Visual studio does give a fairly detailed explanation of what was compiled each time, in the output>build view.
- pharmakom 3y ago

deleted 3y ago

[deleted]

jnwatson 3y ago

The only part of the Earthly pitch that resonated with me was the ability to run the CI locally. When debugging your CI, it is just faster to find problems when you can run on your own box. I have a gnarly GitHub Actions script that took forever to get right because the debug cycle was 10 minutes long.

fireflash38 3y ago
That's one of my biggest complaints about gitlab-ci. Theres basically no way to debug complex rules without just running it. They've got a way now to simulate for default branch, but that's like bare minimum necessary. No way to simulate based on vars or anything else. And no way to try locally, just gotta fucking send it live.
ianwalter 3y ago
I feel you on that one. There is also Act (https://github.com/nektos/act https://github.com/nektos/act) and Dagger.io.
- jnwatson 3y ago
  I have a very complicated script and Act didn't work on it.
smaudet 3y ago
Yeah, the faster but is also a bit of a non-seller, they have to integrate the world/deal with bad build times that can't easily be parallelized (what if there are no modular dependencies, that your tool can understand?) except by tools that have nothing to do with their product. But there have been many times when debugging a CI has been necessary, having to figure out what has gone wrong on a remote system is often an issue...ofc often the remote system is itself the problem...

crabbone 3y ago

> CI is an overgrown shell script running your build and telling you when it fails. Oh, I wish it were that simple... But, I also don't know what they measured. But, I'll give you just one example. Jenkins default landing page is a disaster in terms of speed. It tries to display data on recent builds everywhere across your cluster. For even not so big clusters this can be hundreds to thousands of items to pull from individual nodes running those builds. I've managed to crash Jenkins more times than I could count by simply loading the landing page w/o a particular configuration added that prevents it from its default behavior. CI server will have its own database, typically, managing all sorts of CI entities: jobs, artifacts, users, secretes... It could be quite big and you'd do well taking care of proper indexing etc. CI will have its assortment of runners, which are often provisioned dynamically (think VM or Docker images that need to be deployed to runner nodes), distributing them across the cluster is also not an easy task, if you want that to work fast. Of course you also want to be able to distribute the artifacts across the cluster -- this also takes time and resources. CI will have its own bookkeeping necessary to, essentially, do garbage collection, reporting and self-diagnostic. In big enough clusters this stuff can have a very high latency, if you don't specifically work towards reducing that latency. > If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them. Ever heard of ccache? But, really, this is also super-naive. To make builds fast in a distributed system that CI is you would also need to figure out how to distribute this cache, how to modularize the builds... You might have heard about how one of the most difficult things in programming is cache invalidation? -- That's only partly a joke.

jrockway 3y ago

The tricky part of CI is figuring out what work doesn't need to be done. That is how you save time.