4 ms·

I recently made a system that uses overlays to provide work spaces for a complex build process. However, there is significant overhead paid for unmounting an de

I recently made a system that uses overlays to provide work spaces for a complex build process. However, there is significant overhead paid for unmounting an deleting the files in the overlay after all the work is done. I was thinking about changing the system such that I allocate a partition ahead of time, write all the overlays there, and on success just blow away the partition and with it the overlays. This is kind of a pain in the ass. Can anyone suggest a method for rapidly deleting all the data generated by using 100's of overlays? Maybe BtrFS snapshots would be better? What are the pros and cons? Thank you so much and I apologize for "anything" up front :)

viraptor 7y ago
I'm not sure I understand your use case well, but have you tried lvm's snapshots? That could be the simplest solution. If you're going to try btrfs, check if your system/tools handle the space overcommit correctly. Some ways to check the available space don't really play well with snapshots. (As in, they report less space available)
- nwellinghoff 7y ago
  The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo. The git repo is massive so giving each process its own copy is not practical. Instead each subprocess gets a unique overlay to do it's work. After a subprocess finishes if it has any artifacts that will be used by a later stage of the build they are copied to a "common" overlay which will get mounted on top of the original lower file system used in the previous phase. So the system needs to be able to use multiple lower overlays (RO) as it moves through each phase. After its all over all the data needs to be quickly deleted such that the system is free to perform the next job.
  - emptysea 7y ago
    Does this build system cover a similar space to Buck, Bazel, and Pants? If so, is there a reason for building your own vs working with an existing one?
  - yjftsjthsd-h 7y ago
    > The git repo is massive so giving each process its own copy is not practical. Just to make sure: This is per point-in-time, and not just because of history? You can usually make things somewhat cheaper just by doing a shallow clone. But of course that doesn't help if at any given point-in-time the repo is still too big.
    - nwellinghoff 7y ago
      correct its per point in time. Either way the overlays are a TON faster. :)
    - namibj 7y ago
      Even then, you can actually use multiple external worktrees, saving on the .git folder (or at least any significant size contributions of it).
      - the8472 7y ago
        I don't think git uses copy_file_range yet (at least I can't find it looking through its source), so it'll make expensive copies for each worktree. And he said even deleting the diffs in an overlay is too expensive, which normally should be cheaper than deleting a worktree.
  - viraptor 7y ago
    I'd definitely try lvm snapshots in that case. They free you from the specific choice of the filesystem. But I don't know your exact workload, so... testing and profiling some options. Make sure you try larger lvm snapshot blocks to lower the overhead - 4KB may still be the default and it kills performance.
  - beagle3 7y ago
    It is likely a lot of work, but sounds like working towards out-of-tree builds will pay great dividends going forward in your codebase.
  - deleted 7y ago
    [deleted]
  - fulafel 7y ago
    Another way to do this with just userspace tools could be git clone -l, combined with committing artifacts to a temporary branch to be used by later stages. But maybe the space savings would still be suboptimal because there would be copies of the artifacts both in the workspace and the git db?
  - rumanator 7y ago
    > The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo. Very interesting. Thanks for sharing!
  - rmetzler 7y ago
    I think there is a limit on how many layers Docker can handle in an image and if I recall correctly the number is like 63 layers. I don’t have any idea why it is like this but it could be chosen for performance reasons.
  - namibj 7y ago
    You can use a btrfs subvolume for the build location, and take snapshots at all rollback points. This is fast. Basically, you create an r/w snapshot at first that will evolve the current batch's "common" overlay. Proceed by making another r/w snapshot for the/each subprocess. Just work inside that folder (which behaves ~like a CoW bind-mount) and copy the artifacts into the "common" snapshot. btrfs sub del -c path/to/temp/build/snapshot will clean it up and ensure it won't reappear after a crash. You can skip the -c if you don't care about it reappearing. If you have enough space to handle up to maybe a couple minutes delay in garbage collecting, you don't need to force any special syncing or such, as it just happens in the background. As for the phases, I don't think this is a problem. btrfs snapshots are like git branches that don't allow merging, leaving explicit copy as the only recourse.
  - yiyus 7y ago
    I have done something similar using a ramfs for the work directories. That way when you are finished you simply save what you want and delete it. Building is much faster than to disk. If your data does not fit in RAM, just use an image file in the same way.
rzzzt 7y ago
Tangentially related: are there any good solutions for shipping data with containers (for which the CoW mechanism is not suited particularly well)? Is there a "hub" for volumes?
tlb 7y ago
Can you use a Linux tmpfs as the overlay? It's much faster to create/delete files, and you can simply unmount it at the end and its memory is immediately reclaimed.
- nwellinghoff 7y ago
  unfortunately, the overlays are too big and would eat too much RAM. I have not figured out why they get so big as they are only modifying a small amount of files. Sometimes I think the copy on write functionality is not working properly.
  - Hello71 7y ago
    make sure you have redirect_dir=on,metacopy=on
  - contingencies 7y ago
    Try adding initial /build-tmpfs then copying or mounting /lib{n} ... then making new /specific-build-tmps with subdirectories or readonly for mounts /lib{n}. This way writes are discarded after a build (you destroy /specific-build on completion). If you can't easily split files between disparate sources, use a COW filesystem only for /specific/build/tmpfs. Likely to exhibit substantially different performance characteristics, with easily limited memory requirements. Did a lot of this with ZFS in the past, worked very well.
the8472 7y ago
With recent kernels you can combine overlayfs and btrfs. btrfs subvols/snapshosts have their own costs, they can get slow once you accumulate thousands of them (it's fine if you just use a few at a time). But you can create a single btrfs subvolume, store your overlays in there and then delete the subvolume when you're done.
- nwellinghoff 7y ago
  Good idea.

viraptor 7y ago

I'm not sure I understand your use case well, but have you tried lvm's snapshots? That could be the simplest solution. If you're going to try btrfs, check if your system/tools handle the space overcommit correctly. Some ways to check the available space don't really play well with snapshots. (As in, they report less space available)

nwellinghoff 7y ago
The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo. The git repo is massive so giving each process its own copy is not practical. Instead each subprocess gets a unique overlay to do it's work. After a subprocess finishes if it has any artifacts that will be used by a later stage of the build they are copied to a "common" overlay which will get mounted on top of the original lower file system used in the previous phase. So the system needs to be able to use multiple lower overlays (RO) as it moves through each phase. After its all over all the data needs to be quickly deleted such that the system is free to perform the next job.
- emptysea 7y ago
  Does this build system cover a similar space to Buck, Bazel, and Pants? If so, is there a reason for building your own vs working with an existing one?
- yjftsjthsd-h 7y ago
  > The git repo is massive so giving each process its own copy is not practical. Just to make sure: This is per point-in-time, and not just because of history? You can usually make things somewhat cheaper just by doing a shallow clone. But of course that doesn't help if at any given point-in-time the repo is still too big.
  - nwellinghoff 7y ago
    correct its per point in time. Either way the overlays are a TON faster. :)
  - namibj 7y ago
    Even then, you can actually use multiple external worktrees, saving on the .git folder (or at least any significant size contributions of it).
    - the8472 7y ago
      I don't think git uses copy_file_range yet (at least I can't find it looking through its source), so it'll make expensive copies for each worktree. And he said even deleting the diffs in an overlay is too expensive, which normally should be cheaper than deleting a worktree.
- viraptor 7y ago
  I'd definitely try lvm snapshots in that case. They free you from the specific choice of the filesystem. But I don't know your exact workload, so... testing and profiling some options. Make sure you try larger lvm snapshot blocks to lower the overhead - 4KB may still be the default and it kills performance.
- beagle3 7y ago
  It is likely a lot of work, but sounds like working towards out-of-tree builds will pay great dividends going forward in your codebase.
- deleted 7y ago
  [deleted]
- fulafel 7y ago
  Another way to do this with just userspace tools could be git clone -l, combined with committing artifacts to a temporary branch to be used by later stages. But maybe the space savings would still be suboptimal because there would be copies of the artifacts both in the workspace and the git db?
- rumanator 7y ago
  > The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo. Very interesting. Thanks for sharing!
- rmetzler 7y ago
  I think there is a limit on how many layers Docker can handle in an image and if I recall correctly the number is like 63 layers. I don’t have any idea why it is like this but it could be chosen for performance reasons.
- namibj 7y ago
  You can use a btrfs subvolume for the build location, and take snapshots at all rollback points. This is fast. Basically, you create an r/w snapshot at first that will evolve the current batch's "common" overlay. Proceed by making another r/w snapshot for the/each subprocess. Just work inside that folder (which behaves ~like a CoW bind-mount) and copy the artifacts into the "common" snapshot. btrfs sub del -c path/to/temp/build/snapshot will clean it up and ensure it won't reappear after a crash. You can skip the -c if you don't care about it reappearing. If you have enough space to handle up to maybe a couple minutes delay in garbage collecting, you don't need to force any special syncing or such, as it just happens in the background. As for the phases, I don't think this is a problem. btrfs snapshots are like git branches that don't allow merging, leaving explicit copy as the only recourse.
- yiyus 7y ago
  I have done something similar using a ramfs for the work directories. That way when you are finished you simply save what you want and delete it. Building is much faster than to disk. If your data does not fit in RAM, just use an image file in the same way.

rzzzt 7y ago

Tangentially related: are there any good solutions for shipping data with containers (for which the CoW mechanism is not suited particularly well)? Is there a "hub" for volumes?

tlb 7y ago

Can you use a Linux tmpfs as the overlay? It's much faster to create/delete files, and you can simply unmount it at the end and its memory is immediately reclaimed.

nwellinghoff 7y ago
unfortunately, the overlays are too big and would eat too much RAM. I have not figured out why they get so big as they are only modifying a small amount of files. Sometimes I think the copy on write functionality is not working properly.
- Hello71 7y ago
  make sure you have redirect_dir=on,metacopy=on
- contingencies 7y ago
  Try adding initial /build-tmpfs then copying or mounting /lib{n} ... then making new /specific-build-tmps with subdirectories or readonly for mounts /lib{n}. This way writes are discarded after a build (you destroy /specific-build on completion). If you can't easily split files between disparate sources, use a COW filesystem only for /specific/build/tmpfs. Likely to exhibit substantially different performance characteristics, with easily limited memory requirements. Did a lot of this with ZFS in the past, worked very well.

the8472 7y ago

With recent kernels you can combine overlayfs and btrfs. btrfs subvols/snapshosts have their own costs, they can get slow once you accumulate thousands of them (it's fine if you just use a few at a time). But you can create a single btrfs subvolume, store your overlays in there and then delete the subvolume when you're done.

nwellinghoff 7y ago
Good idea.