9 ms·

Disk I/O bottlenecks in GitHub Actions

106 points by jacobwg 2y ago

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
ValdikSS 2y ago
`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.
- nijave 2y ago
  Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step
  - wtallis 2y ago
    CI containers should probably run entirely from tmpfs.
    - jacobwg 2y ago
      We're having some success with doing this at the block level (e.g. in-memory writeback cache).
      - yjftsjthsd-h 2y ago
        Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
        jacobwg 2y ago
        The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
    - candiddevmike 2y ago
      We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
    - nijave 2y ago
      Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
      - wtallis 2y ago
        tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
  - jacobwg 2y ago
    I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
    - 3np 2y ago
      atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
    - formerly_proven 2y ago
      noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
  - kylegalbraith 2y ago
    This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
  - formerly_proven 2y ago
    You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html
  - chippiewill 2y ago
    > Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...
    - nijave 2y ago
      Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though
- Brian_K_White 2y ago
  "`dpkg` calls `fsync()` on all the unpacked files" Why in the world does it do that ???? Ok I googled (kagi). Same reason anyone ever does: pure voodoo. If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either. Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.
  - duped 2y ago
    "durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
    - Brian_K_White 2y ago
      Like I said.
      - levkk 2y ago
        Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
        Brian_K_White 2y ago
        I didn't say it synced. The file is simply "written" and available at that point. It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
        hiciu 2y ago
        I would like to introduce you to a few case studies on bugzilla: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slo...
- the8472 2y ago
  No need for eatmydata, dpkg has an unsafe-io option. Other options are to use an overlay mount with volatile or ext4 with nobarrier and writeback.
  - ValdikSS 2y ago
    unsafe-io eliminates most fsyncs, but not all of them.
- inetknght 2y ago
  So... the goal is to make `apt` to be web scale? (to be clear: my comment is sarcasm and web scale is a reference to a joke about reliability [0]) [0]: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
suryao 2y ago
TLDR: disk is often the bottleneck in builds. Use 'fio' to get performance of the disk. If you want to truly speed up builds by optimizing disk performance, there are no shortcuts to physically attaching NVMe storage with high throughput and high IOPS to your compute directly. That's what we do at WarpBuild[0] and we outperform Depot runners handily. This is because we do not use network attached disks which come with relatively higher latency. Our runners are also coupled with faster processors. I love the Depot content team though, it does a lot of heavy lifting. [0] https://www.warpbuild.com https://www.warpbuild.com
miohtama 2y ago
If you can afford, upgrade your CI runners on GitHub to paid offering. Highly recommend, less drinking coffee, more instant unit test results. Pay as you go.
- striking 2y ago
  As a Depot customer, I'd say if you can afford to pay for GitHub's runners, you should pay for Depot's instead. They boot faster, run faster, are a fraction of the price. And they are lovely people who provide amazing support.
  - suryao 2y ago
    [flagged]
    - mdeeks 2y ago
      I think you're being downvoted because this comes off a bit... I dunno... petty I guess? Maybe reply to the original post and let your product stand on it's own merits.
      - deleted 2y ago
        [deleted]
- kylegalbraith 2y ago
  This is what we focus on with Depot. Faster builds across the board without breaking the bank. More time to get things done and maybe go outside earlier. Trading Strategy looks super cool, by the way.
jacobwg 2y ago
A list of fun things we've done for CI runners to improve CI: - Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!) - Benchmarked EC2 instance types (m7a is the best x86 today, m8g is the best arm64) - "Warming" the root EBS volume by accessing a set of priority blocks before the job starts to give the job full disk performance [0] - Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub) - Configuring Docker with containerd/estargz support - Just generally turning kernel options and unit files off that aren't needed [0] https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initialize.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initial...
- 3np 2y ago
  > Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub) Are you not using a caching registry mirror, instead pulling the same image from Hub for each runner...? If so that seems like it would be an easy win to add, unless you specifically do mostly hot/unique pulls. The more efficient answer to those rate limits is almost always to pull less times for the same work rather than scaling in a way that circumvents them.
  - jacobwg 2y ago
    Today we (Depot) are not, though some of our customers configure this. For the moment at least, the ephemeral public IP architecture makes it generally unnecessary from a rate-limit perspective. From a performance / efficiency perspective, we generally recommend using ECR Public images[0], since AWS hosts mirrors of all the "Docker official" images, and throughput to ECR Public is great from inside AWS. [0] https://gallery.ecr.aws/ https://gallery.ecr.aws/
    - glenjamin 2y ago
      If you’re running inside AWS us-east-1 then docker hub will give you direct S3 URLs for layer downloads (or it used to anyway) Any pulls doing this become zero cost for docker hub Any sort of cache you put between docker hub and your own infra would probably be S3 backed anyway, so adding another cache in between could be mostly a waste
larusso 2y ago
So I had to read to the end to realize it’s a kinda infomercial. Ok fair enough. Didn’t know what depot was though.
crmd 2y ago
This is exactly the kind of content marketing I want to see. The IO bottleneck data and the fio scripts are useful to all. Then at the end a link to their product which I’d never heard of, in case you’re dealing with the issue at hand.
- kylegalbraith 2y ago
  Thank you for the kind words. We’re always trying to share our knowledge even if Depot isn’t a good fit for everyone. I hope the scripts get some mileage!
nodesocket 2y ago
I just migrated multiple ARM64 GitHub action Docker builds from my self hosted runner (Raspberry Pi in my homeland) to Blacksmith.io and I’m really impressed with the performance so far. Only downside is no Docker layer and image cache like I had on my self hosted runner, but can’t complain on the free tier.
- adityamaru 2y ago
  Have you checked out https://docs.blacksmith.sh/docker-builds/incremental-docker-builds https://docs.blacksmith.sh/docker-builds/incremental-docker-...? This should help setup a shared, persistent docker layer cache for your runners
  - nodesocket 2y ago
    Thanks for sharing. I have a custom bash script which does the docker builds currently and swapping to useblacksmith/build-push-action would take a bit of refactoring which I don't want to spend the time on now. :-)
kayson 2y ago
Bummer there's no free tier. I've been bashing my head against an intermittent CI failure problem on Github runners for probably a couple years now. I think it's related to the networking stack in their runner image and the fact that I'm using docker in docker to unit test a docker firewall. While I do appreciate that someone at Github did actually look at my issue, they totally missed the point. https://github.com/actions/runner-images/issues/11786 https://github.com/actions/runner-images/issues/11786 Are there any reasonable alternatives for a really tiny FOSS project?
crohr 2y ago
I'm maintaining a benchmark of various GitHub Actions providers regarding I/O speed [1]. Depot is not present because my account was blocked but would love to compare! The disk accelerator looks like a nice feature. [1]: https://runs-on.com/benchmarks/github-actions-disk-performance/ https://runs-on.com/benchmarks/github-actions-disk-performan...
r3tr0 2y ago
we are working on a platform that let's you measure this stuff in real-time for free. you can check us out at https://yeet.cx https://yeet.cx we also have a anonymous guest sandbox you can play with https://yeet.cx/play https://yeet.cx/play

106 points by jacobwg 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

ValdikSS 2y ago

`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.

nijave 2y ago
Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step
- wtallis 2y ago
  CI containers should probably run entirely from tmpfs.
  - jacobwg 2y ago
    We're having some success with doing this at the block level (e.g. in-memory writeback cache).
    - yjftsjthsd-h 2y ago
      Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
      - jacobwg 2y ago
        The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
  - candiddevmike 2y ago
    We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
  - nijave 2y ago
    Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
    - wtallis 2y ago
      tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
- jacobwg 2y ago
  I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
  - 3np 2y ago
    atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
  - formerly_proven 2y ago
    noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
- kylegalbraith 2y ago
  This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
- formerly_proven 2y ago
  You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html
- chippiewill 2y ago
  > Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...
  - nijave 2y ago
    Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though
Brian_K_White 2y ago
"`dpkg` calls `fsync()` on all the unpacked files" Why in the world does it do that ???? Ok I googled (kagi). Same reason anyone ever does: pure voodoo. If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either. Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.
- duped 2y ago
  "durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
  - Brian_K_White 2y ago
    Like I said.
    - levkk 2y ago
      Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
      - Brian_K_White 2y ago
        I didn't say it synced. The file is simply "written" and available at that point. It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
        hiciu 2y ago
        I would like to introduce you to a few case studies on bugzilla: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slo...
the8472 2y ago
No need for eatmydata, dpkg has an unsafe-io option. Other options are to use an overlay mount with volatile or ext4 with nobarrier and writeback.
- ValdikSS 2y ago
  unsafe-io eliminates most fsyncs, but not all of them.
inetknght 2y ago
So... the goal is to make `apt` to be web scale? (to be clear: my comment is sarcasm and web scale is a reference to a joke about reliability [0]) [0]: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

suryao 2y ago

TLDR: disk is often the bottleneck in builds. Use 'fio' to get performance of the disk. If you want to truly speed up builds by optimizing disk performance, there are no shortcuts to physically attaching NVMe storage with high throughput and high IOPS to your compute directly. That's what we do at WarpBuild[0] and we outperform Depot runners handily. This is because we do not use network attached disks which come with relatively higher latency. Our runners are also coupled with faster processors. I love the Depot content team though, it does a lot of heavy lifting. [0] https://www.warpbuild.com https://www.warpbuild.com

miohtama 2y ago

If you can afford, upgrade your CI runners on GitHub to paid offering. Highly recommend, less drinking coffee, more instant unit test results. Pay as you go.

striking 2y ago
As a Depot customer, I'd say if you can afford to pay for GitHub's runners, you should pay for Depot's instead. They boot faster, run faster, are a fraction of the price. And they are lovely people who provide amazing support.
- suryao 2y ago
  [flagged]
  - mdeeks 2y ago
    I think you're being downvoted because this comes off a bit... I dunno... petty I guess? Maybe reply to the original post and let your product stand on it's own merits.
    - deleted 2y ago
      [deleted]
kylegalbraith 2y ago
This is what we focus on with Depot. Faster builds across the board without breaking the bank. More time to get things done and maybe go outside earlier. Trading Strategy looks super cool, by the way.

jacobwg 2y ago

A list of fun things we've done for CI runners to improve CI: - Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!) - Benchmarked EC2 instance types (m7a is the best x86 today, m8g is the best arm64) - "Warming" the root EBS volume by accessing a set of priority blocks before the job starts to give the job full disk performance [0] - Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub) - Configuring Docker with containerd/estargz support - Just generally turning kernel options and unit files off that aren't needed [0] https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initialize.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initial...

3np 2y ago
> Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub) Are you not using a caching registry mirror, instead pulling the same image from Hub for each runner...? If so that seems like it would be an easy win to add, unless you specifically do mostly hot/unique pulls. The more efficient answer to those rate limits is almost always to pull less times for the same work rather than scaling in a way that circumvents them.
- jacobwg 2y ago
  Today we (Depot) are not, though some of our customers configure this. For the moment at least, the ephemeral public IP architecture makes it generally unnecessary from a rate-limit perspective. From a performance / efficiency perspective, we generally recommend using ECR Public images[0], since AWS hosts mirrors of all the "Docker official" images, and throughput to ECR Public is great from inside AWS. [0] https://gallery.ecr.aws/ https://gallery.ecr.aws/
  - glenjamin 2y ago
    If you’re running inside AWS us-east-1 then docker hub will give you direct S3 URLs for layer downloads (or it used to anyway) Any pulls doing this become zero cost for docker hub Any sort of cache you put between docker hub and your own infra would probably be S3 backed anyway, so adding another cache in between could be mostly a waste

larusso 2y ago

So I had to read to the end to realize it’s a kinda infomercial. Ok fair enough. Didn’t know what depot was though.

crmd 2y ago

This is exactly the kind of content marketing I want to see. The IO bottleneck data and the fio scripts are useful to all. Then at the end a link to their product which I’d never heard of, in case you’re dealing with the issue at hand.

kylegalbraith 2y ago
Thank you for the kind words. We’re always trying to share our knowledge even if Depot isn’t a good fit for everyone. I hope the scripts get some mileage!

nodesocket 2y ago

I just migrated multiple ARM64 GitHub action Docker builds from my self hosted runner (Raspberry Pi in my homeland) to Blacksmith.io and I’m really impressed with the performance so far. Only downside is no Docker layer and image cache like I had on my self hosted runner, but can’t complain on the free tier.

adityamaru 2y ago
Have you checked out https://docs.blacksmith.sh/docker-builds/incremental-docker-builds https://docs.blacksmith.sh/docker-builds/incremental-docker-...? This should help setup a shared, persistent docker layer cache for your runners
- nodesocket 2y ago
  Thanks for sharing. I have a custom bash script which does the docker builds currently and swapping to useblacksmith/build-push-action would take a bit of refactoring which I don't want to spend the time on now. :-)

kayson 2y ago

Bummer there's no free tier. I've been bashing my head against an intermittent CI failure problem on Github runners for probably a couple years now. I think it's related to the networking stack in their runner image and the fact that I'm using docker in docker to unit test a docker firewall. While I do appreciate that someone at Github did actually look at my issue, they totally missed the point. https://github.com/actions/runner-images/issues/11786 https://github.com/actions/runner-images/issues/11786 Are there any reasonable alternatives for a really tiny FOSS project?

crohr 2y ago

I'm maintaining a benchmark of various GitHub Actions providers regarding I/O speed [1]. Depot is not present because my account was blocked but would love to compare! The disk accelerator looks like a nice feature. [1]: https://runs-on.com/benchmarks/github-actions-disk-performance/ https://runs-on.com/benchmarks/github-actions-disk-performan...

r3tr0 2y ago

we are working on a platform that let's you measure this stuff in real-time for free. you can check us out at https://yeet.cx https://yeet.cx we also have a anonymous guest sandbox you can play with https://yeet.cx/play https://yeet.cx/play