5 ms·

`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks

`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.

nijave 2y ago
Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step
- wtallis 2y ago
  CI containers should probably run entirely from tmpfs.
  - jacobwg 2y ago
    We're having some success with doing this at the block level (e.g. in-memory writeback cache).
    - yjftsjthsd-h 2y ago
      Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
      - jacobwg 2y ago
        The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
  - candiddevmike 2y ago
    We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
  - nijave 2y ago
    Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
    - wtallis 2y ago
      tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
- jacobwg 2y ago
  I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
  - 3np 2y ago
    atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
  - formerly_proven 2y ago
    noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
- kylegalbraith 2y ago
  This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
- formerly_proven 2y ago
  You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html
- chippiewill 2y ago
  > Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...
  - nijave 2y ago
    Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though
Brian_K_White 2y ago
"`dpkg` calls `fsync()` on all the unpacked files" Why in the world does it do that ???? Ok I googled (kagi). Same reason anyone ever does: pure voodoo. If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either. Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.
- duped 2y ago
  "durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
  - Brian_K_White 2y ago
    Like I said.
    - levkk 2y ago
      Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
      - Brian_K_White 2y ago
        I didn't say it synced. The file is simply "written" and available at that point. It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
        hiciu 2y ago
        I would like to introduce you to a few case studies on bugzilla: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slo...
the8472 2y ago
No need for eatmydata, dpkg has an unsafe-io option. Other options are to use an overlay mount with volatile or ext4 with nobarrier and writeback.
- ValdikSS 2y ago
  unsafe-io eliminates most fsyncs, but not all of them.
inetknght 2y ago
So... the goal is to make `apt` to be web scale? (to be clear: my comment is sarcasm and web scale is a reference to a joke about reliability [0]) [0]: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.

nijave 2y ago

Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit If you corrupt a CI node, whatever. Just rerun the step

wtallis 2y ago
CI containers should probably run entirely from tmpfs.
- jacobwg 2y ago
  We're having some success with doing this at the block level (e.g. in-memory writeback cache).
  - yjftsjthsd-h 2y ago
    Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
    - jacobwg 2y ago
      The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
- candiddevmike 2y ago
  We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching: https://etcha.dev/etchaos/ https://etcha.dev/etchaos/
- nijave 2y ago
  Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal. I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
  - wtallis 2y ago
    tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
jacobwg 2y ago
I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
- 3np 2y ago
  atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
- formerly_proven 2y ago
  noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
kylegalbraith 2y ago
This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
formerly_proven 2y ago
You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection. edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html https://docs.kernel.org/fault-injection/fault-injection.html
chippiewill 2y ago
> Docker is a big culprit Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it. A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mounts https://www.redhat.com/en/blog/container-volatile-overlay-mo...
- nijave 2y ago
  Well yeah, but indirectly through the usage of Docker, I mean. Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files Tearing down overlayfs is a huge issue, though

Brian_K_White 2y ago

"`dpkg` calls `fsync()` on all the unpacked files" Why in the world does it do that ???? Ok I googled (kagi). Same reason anyone ever does: pure voodoo. If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either. Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.

duped 2y ago
"durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
- Brian_K_White 2y ago
  Like I said.
  - levkk 2y ago
    Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
    - Brian_K_White 2y ago
      I didn't say it synced. The file is simply "written" and available at that point. It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
      - hiciu 2y ago
        I would like to introduce you to a few case studies on bugzilla: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slo...

the8472 2y ago

No need for eatmydata, dpkg has an unsafe-io option. Other options are to use an overlay mount with volatile or ext4 with nobarrier and writeback.

ValdikSS 2y ago
unsafe-io eliminates most fsyncs, but not all of them.

inetknght 2y ago

So... the goal is to make `apt` to be web scale? (to be clear: my comment is sarcasm and web scale is a reference to a joke about reliability [0]) [0]: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs