4 ms·

One improvement that would be nice in gitlab is an extension / similar feature to the FF_USE_FASTZIP flag. The default way to cache things is zip, which no mat

One improvement that would be nice in gitlab is an extension / similar feature to the FF_USE_FASTZIP flag. The default way to cache things is zip, which no matter the compression isn't a great fit, you really want a format that is stream-able like tar. When its stream-able you can download the tar and unpack it at the same time, not waiting for the full download before beginning to unpack. Obviously you would compress the tar (normally) with lz4 or zstd or similar, using this approach you should generally see a reduction in total time to get and unpack the cache. node_modules even more so since it generally has zillions of small files, so the unpack time can be quite high. Be nice if gitlab supported this approach OOTB. Edit: An even better way if your environment supports it is to use a mountable image (esp for node_modules) for your caches, this basically removes the entire unpack phase of making the cache available, instead of unpacking you just mount the image and off you go. In macOS this looks like a sparseimage via attached hdituil, or in linux a squashfs image with a writable overlay mount (you need a privileged container for this if in a container). Since the OOTB gitlab.com runners are a root-user linux VM (i think?) this approach should work quite well. SquashFS images are a great fit for node_modules especially as it moves the creation of the zillions of tiny files to cache-creation time rather than cache-use time. If you share the cache images via a hostPath mount or similar for existing images you can effectively make caches available in 0 seconds (just mount an already downloaded image and done).

john_cogs 5y ago
Thanks for the feedback. I passed it along to the pipeline authoring team.
n0w 5y ago
The later should already be possible. I believe you can specify a docker image to be used for running a job so you can bake things like node_modules into the image.
- nhoughto 5y ago
  Yep true, generally baking caches into build images is a bit fragile tho, the more caches and different types of jobs you have the larger the set of combinations of images you need to cover it. Often ends up easier to decouple build image from caches, and ideally calculate a cache key based on a yarn.lock or similar (not supported by gitlab). Also image pull performance is generally pretty poor from most registries, you will get much higher speeds out of a straight s3 download then pulling an image via docker or containerd, even tho ecr for ex is backed by s3 . I haven’t actually used ootb gitlab caching in quite a long time because of these limitations, but would be nice for it to work great ootb!
  - KronisLV 5y ago
    I guess it depends on what you're building. You can also leverage your Dockerfile to reuse the layers that haven't changed, for example see the order of commands i offered here: https://news.ycombinator.com/item?id=29393532 https://news.ycombinator.com/item?id=29393532 In some projects, you can just do something like the following (Java centric example, but applies to other stacks as well): - have Nexus (or another solution) running with a Docker registry next to your GitLab Runner nodes, internal network - setup Nexus to proxy all of the Maven/NuGet/whatever packages as well, in addition to Docker images in the registry - setup GitLab CI to build the base container images that you need weekly (e.g. Maven with whatever you need on top of it for builds, another base JDK image as a runner etc.), this includes not just stuff in your Dockerfile, but also those that you'll use for Runners (assuming Docker executor) - setup GitLab CI to build an intermediate base image for your project with all of the dependencies every morning (assuming that your project has lots of dependencies that don't change often though) - base your actual builds on that image, which is based on the Maven image, which will essentially cut out the need to pull everything from the Nexus Maven repo - it will still allow you to add new dependencies during the day (e.g. new library), which will be pulled from the Internet and then cached both in Docker layer cache for your current pom.xml file, as well as your Nexus intermediate repo You don't even need the intermediate dependency images, as long as you have control over which Runners your project has, with dedicated ones the Docker cache should be sufficient on its own. Right now i no longer have to fear Docker rate limits anymore and most of my builds don't even hit the Internet either at all. I applied some of those principles at a large enterprise project and the builds went from about 7 minutes to 3, with no drawbacks to speak of. Of course, the next step would be incremental code compilation and not wasting time there, however seeing as even some of the above is overkill for many scenarios, that's probably situational.
    - iduoad 5y ago
      Great insights Thank you KronisLV
    - nhoughto 5y ago
      Ah ok sounds like a good setup, I could never get past the slow image pull times for systems like that though? Maybe I missed a setting. Containerd especially was quite slow at pulling images. Image of 2GB~ could take a couple of minutes to come down, order of magnitude more than straight s3 download. Once you’ve got your images fine, but if your build nodes are elastic you can easily hit cold/new nodes and pay the penalty.
- dSebastien 5y ago
  Yep! That's the approach I took when I realized how slow Gitlab CI was with node-based projects: https://dsebastien.net/blog/2020-04-12-speeding-up-your-ci-cd-build-times-with-a-custom-docker-image https://dsebastien.net/blog/2020-04-12-speeding-up-your-ci-c... That helped a whole lot, but it was still far from great..
saurik 5y ago
> The default way to cache things is zip, which no matter the compression isn't a great fit, you really want a format that is stream-able like tar. When its stream-able you can download the tar and unpack it at the same time, not waiting for the full download before beginning to unpack. You absolutely can stream extract a zip file: the index at the end of a zip file is merely an optimization mechanism added to a format that is otherwise somewhat tar-like at its base. While it is theoretically possible to create files that uncompress correctly only via one or the other data structures (see http://www.saurik.com/id/17 http://www.saurik.com/id/17) the file wouldn't be compliant and no one normally does this. The only real tradeoff with a zip file vs tar+gzip is that the compression it does is per file, this means you gain random access (but don't lose streaming support) at the cost of being unable to share compression statistics across multiple small files (and so the resulting archive can be a lot larger).
- nhoughto 5y ago
  Yep Specially crafting a zip file can make it streamable but it’s not the general default. Can GNU unzip read from stdin? Not sure but I don’t think so?
iduoad 5y ago
I believe there are ongoing discussion on supporting other cache/artifact encryption/archiving. https://gitlab.com/gitlab-org/gitlab/-/issues/28330 https://gitlab.com/gitlab-org/gitlab/-/issues/28330 The mountable images for cache is a great idea, thank you

john_cogs 5y ago

Thanks for the feedback. I passed it along to the pipeline authoring team.

n0w 5y ago

The later should already be possible. I believe you can specify a docker image to be used for running a job so you can bake things like node_modules into the image.

nhoughto 5y ago
Yep true, generally baking caches into build images is a bit fragile tho, the more caches and different types of jobs you have the larger the set of combinations of images you need to cover it. Often ends up easier to decouple build image from caches, and ideally calculate a cache key based on a yarn.lock or similar (not supported by gitlab). Also image pull performance is generally pretty poor from most registries, you will get much higher speeds out of a straight s3 download then pulling an image via docker or containerd, even tho ecr for ex is backed by s3 . I haven’t actually used ootb gitlab caching in quite a long time because of these limitations, but would be nice for it to work great ootb!
- KronisLV 5y ago
  I guess it depends on what you're building. You can also leverage your Dockerfile to reuse the layers that haven't changed, for example see the order of commands i offered here: https://news.ycombinator.com/item?id=29393532 https://news.ycombinator.com/item?id=29393532 In some projects, you can just do something like the following (Java centric example, but applies to other stacks as well): - have Nexus (or another solution) running with a Docker registry next to your GitLab Runner nodes, internal network - setup Nexus to proxy all of the Maven/NuGet/whatever packages as well, in addition to Docker images in the registry - setup GitLab CI to build the base container images that you need weekly (e.g. Maven with whatever you need on top of it for builds, another base JDK image as a runner etc.), this includes not just stuff in your Dockerfile, but also those that you'll use for Runners (assuming Docker executor) - setup GitLab CI to build an intermediate base image for your project with all of the dependencies every morning (assuming that your project has lots of dependencies that don't change often though) - base your actual builds on that image, which is based on the Maven image, which will essentially cut out the need to pull everything from the Nexus Maven repo - it will still allow you to add new dependencies during the day (e.g. new library), which will be pulled from the Internet and then cached both in Docker layer cache for your current pom.xml file, as well as your Nexus intermediate repo You don't even need the intermediate dependency images, as long as you have control over which Runners your project has, with dedicated ones the Docker cache should be sufficient on its own. Right now i no longer have to fear Docker rate limits anymore and most of my builds don't even hit the Internet either at all. I applied some of those principles at a large enterprise project and the builds went from about 7 minutes to 3, with no drawbacks to speak of. Of course, the next step would be incremental code compilation and not wasting time there, however seeing as even some of the above is overkill for many scenarios, that's probably situational.
  - iduoad 5y ago
    Great insights Thank you KronisLV
  - nhoughto 5y ago
    Ah ok sounds like a good setup, I could never get past the slow image pull times for systems like that though? Maybe I missed a setting. Containerd especially was quite slow at pulling images. Image of 2GB~ could take a couple of minutes to come down, order of magnitude more than straight s3 download. Once you’ve got your images fine, but if your build nodes are elastic you can easily hit cold/new nodes and pay the penalty.
dSebastien 5y ago
Yep! That's the approach I took when I realized how slow Gitlab CI was with node-based projects: https://dsebastien.net/blog/2020-04-12-speeding-up-your-ci-cd-build-times-with-a-custom-docker-image https://dsebastien.net/blog/2020-04-12-speeding-up-your-ci-c... That helped a whole lot, but it was still far from great..

saurik 5y ago

> The default way to cache things is zip, which no matter the compression isn't a great fit, you really want a format that is stream-able like tar. When its stream-able you can download the tar and unpack it at the same time, not waiting for the full download before beginning to unpack. You absolutely can stream extract a zip file: the index at the end of a zip file is merely an optimization mechanism added to a format that is otherwise somewhat tar-like at its base. While it is theoretically possible to create files that uncompress correctly only via one or the other data structures (see http://www.saurik.com/id/17 http://www.saurik.com/id/17) the file wouldn't be compliant and no one normally does this. The only real tradeoff with a zip file vs tar+gzip is that the compression it does is per file, this means you gain random access (but don't lose streaming support) at the cost of being unable to share compression statistics across multiple small files (and so the resulting archive can be a lot larger).

nhoughto 5y ago
Yep Specially crafting a zip file can make it streamable but it’s not the general default. Can GNU unzip read from stdin? Not sure but I don’t think so?

iduoad 5y ago

I believe there are ongoing discussion on supporting other cache/artifact encryption/archiving. https://gitlab.com/gitlab-org/gitlab/-/issues/28330 https://gitlab.com/gitlab-org/gitlab/-/issues/28330 The mountable images for cache is a great idea, thank you