4 ms·

Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI

by nijave 2y ago

Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step

wtallis 2y ago
CI containers should probably run entirely from tmpfs.
- jacobwg 2y ago
  We're having some success with doing this at the block level (e.g. in-memory writeback cache).
  - yjftsjthsd-h 2y ago
    Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
    - jacobwg 2y ago
      The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
- candiddevmike 2y ago
  We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
- nijave 2y ago
  Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
  - wtallis 2y ago
    tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
jacobwg 2y ago
I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
- 3np 2y ago
  atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
- formerly_proven 2y ago
  noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
kylegalbraith 2y ago
This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
formerly_proven 2y ago
You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html
chippiewill 2y ago
> Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...
- nijave 2y ago
  Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though

by nijave 2y ago

Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step

wtallis 2y ago

CI containers should probably run entirely from tmpfs.

jacobwg 2y ago
We're having some success with doing this at the block level (e.g. in-memory writeback cache).
- yjftsjthsd-h 2y ago
  Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
  - jacobwg 2y ago
    The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
candiddevmike 2y ago
We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
nijave 2y ago
Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
- wtallis 2y ago
  tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).

jacobwg 2y ago

I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.

3np 2y ago
atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
formerly_proven 2y ago
noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.

kylegalbraith 2y ago

This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.

formerly_proven 2y ago

You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html

chippiewill 2y ago

> Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...

nijave 2y ago
Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though