5 ms·

A lot of utility in Docker comes from incremental (cached) builds based on overlay but in fact you can get it from any CoW system such as LVM/ZFS/BtrFS snapshot

A lot of utility in Docker comes from incremental (cached) builds based on overlay but in fact you can get it from any CoW system such as LVM/ZFS/BtrFS snapshots.

seabrookmx 7y ago
Docker defaults to overlayfs but you don't have to use it. If you use ZFS or another storage driver it will leverage their capabilities to provide the same functionality: https://docs.docker.com/storage/storagedriver/select-storage-driver/ https://docs.docker.com/storage/storagedriver/select-storage...
- deleted 7y ago
  [deleted]
- koffiezet 7y ago
  Docker on ZFS is a pain in the ass and slow as hell. And I say this as both a happy docker/container and ZFS user. But combining them is a bad idea.
  - seabrookmx 7y ago
    I definitely noticed slowness as well when I tried it. I'm currently using overlay because I leave my docker mount on my root partition since ZFS has always been a pain for root. I hear Ubuntu 19.10 is smoothing that over a bit though. It's too bad because I've ran into issues on lower memory boxes where the Linux block cache and the arc compete for memory and cause performance slowdowns or errors. If all the filesystems were ZFS this wouldn't be an issue. Also all the other ZFS benefits like resisting bitrot etc.
    - koffiezet 7y ago
      I just use overlay on top of ZFS. It's just the native ZFS docker backend that is very slow.
      - seabrookmx 7y ago
        Why didn't I think of that? So all you need to do is manually specify the volume driver to overlay2 when your FS is ZFS, and this just works?
flas9sd 7y ago
fyi, btrfs as /var/lib/docker/btrfs has a disadvantage under certain circumstances: running commands (du,rsync,..) on the subvolume folders altering the access time will incur a storage cost if the filesystem is mounted without noatime option: https://github.com/moby/moby/issues/39815 https://github.com/moby/moby/issues/39815 - not sure if it applies to ZFS as well (see https://lwn.net/Articles/499648/ https://lwn.net/Articles/499648/)
- szemet 7y ago
  Another difference that may matter for some: on block based CoW FS, shared executables and libs will not be shared among containers, because file cache is file based not block based. Using Overlayfs - or other file level CoW storage, when one starts several containers from the same image, shared libraries and executables will be loaded only once(so will be actually shared) across containers.
curt15 7y ago
Facebook, for one, uses Btrfs for its containers: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebo...
jzl 7y ago
In fact, this is explained in the article which also has an interesting anecdote about btrfs.

A lot of utility in Docker comes from incremental (cached) builds based on overlay but in fact you can get it from any CoW system such as LVM/ZFS/BtrFS snapshots.

seabrookmx 7y ago

Docker defaults to overlayfs but you don't have to use it. If you use ZFS or another storage driver it will leverage their capabilities to provide the same functionality: https://docs.docker.com/storage/storagedriver/select-storage-driver/ https://docs.docker.com/storage/storagedriver/select-storage...

deleted 7y ago
[deleted]
koffiezet 7y ago
Docker on ZFS is a pain in the ass and slow as hell. And I say this as both a happy docker/container and ZFS user. But combining them is a bad idea.
- seabrookmx 7y ago
  I definitely noticed slowness as well when I tried it. I'm currently using overlay because I leave my docker mount on my root partition since ZFS has always been a pain for root. I hear Ubuntu 19.10 is smoothing that over a bit though. It's too bad because I've ran into issues on lower memory boxes where the Linux block cache and the arc compete for memory and cause performance slowdowns or errors. If all the filesystems were ZFS this wouldn't be an issue. Also all the other ZFS benefits like resisting bitrot etc.
  - koffiezet 7y ago
    I just use overlay on top of ZFS. It's just the native ZFS docker backend that is very slow.
    - seabrookmx 7y ago
      Why didn't I think of that? So all you need to do is manually specify the volume driver to overlay2 when your FS is ZFS, and this just works?

flas9sd 7y ago

fyi, btrfs as /var/lib/docker/btrfs has a disadvantage under certain circumstances: running commands (du,rsync,..) on the subvolume folders altering the access time will incur a storage cost if the filesystem is mounted without noatime option: https://github.com/moby/moby/issues/39815 https://github.com/moby/moby/issues/39815 - not sure if it applies to ZFS as well (see https://lwn.net/Articles/499648/ https://lwn.net/Articles/499648/)

szemet 7y ago
Another difference that may matter for some: on block based CoW FS, shared executables and libs will not be shared among containers, because file cache is file based not block based. Using Overlayfs - or other file level CoW storage, when one starts several containers from the same image, shared libraries and executables will be loaded only once(so will be actually shared) across containers.

curt15 7y ago

Facebook, for one, uses Btrfs for its containers: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebo...

jzl 7y ago

In fact, this is explained in the article which also has an interesting anecdote about btrfs.