4 ms·

If you're using a recent Ubuntu, docker/zfs can lead to a large amount of snapshots. 'docker system prune' may help but has had zfs issues for me.

yjftsjthsd-h 5y ago
> but has had zfs issues for me. Can you give more details so I know what to watch out for? (Recently started running Docker+ZFS in prod, haven't seen it break yet but appreciate heads up)
- flemhans 5y ago
  The zfs storage driver isn't production ready yet.
  - r1ch 5y ago
    I love that they say this, then default to using it when ZFS is detected.
  - cosmin800 5y ago
    I am pretty sure Canonical will soon release a stable,production ready snap package for zfs.
    - alyandon 5y ago
      That gets updated arbitrarily at random intervals (because who doesn't want filesystem modules updated frequently!?!?!) and then purges logs every 72 hours so when you have a problem you'll never even know what was updated! I have to run snapd on one of my servers and I hate everything about it.
      - flemhans 5y ago
        I'm considering to run lxd as a snap because I want Debian 11 (for openzfs 2.0) and lxd at the same time. Am I in for a world of hurt?
        alyandon 5y ago
        I wouldn't say you are in for a world of hurt but you'll have to accept that the package will periodically update and in the process kill your containers/vms. My primary objections to snapd are: 1) updates are pushed/applied and services restarted on an arbitrary schedule which means you can't test updates prior to rolling them out or otherwise control when a service is taken offline by an update 2) snapd doesn't log to standard OS logging facilities and the logs it does have are cleared after 72 hours 3) snapd pollutes home directories with a directory called "snap" snapd is a poorly conceived project written by people that are profoundly ignorant of how real services and servers are actually operated. Despite having the above flaws repeatedly pointed out, the developers have shown zero interest in taking actual concrete actions to address them.
        flemhans 5y ago
        Thank you for the reply. It seems snapd is indeed a poorly conceived project. How would you work around it in a production setting if you HAD to use it? Auto-updating sounds like the biggest issue of those.
        zekrioca 5y ago
        Build LXD yourself. It is not that complicated if one updates go to something beyond 1.14. Only downside is that you may need to create your own init scripts and a lxd group, but that's not complicated either.
- anotherhue 5y ago
  there were a number of containers immune to 'docker rm', seemed it was trying to invoke 'zfs delete' on something that had already been removed (presumably moments ago as part of a previous step). Nasty workaround was to manually recreate the dataset (and children) so that it could remove them. The issue was common enough that I could google it so don't stress. More worryingly, 'prune' locked up for about 10 minutes before getting to do it's thing. pinning the cpu the whole time. This was on my devbox, so maybe it's my doing but did not inspire confidence. I've run FreeBSD ZFS for about a decade and openzfs/ubuntu in prod for kafka clusters so I'm no stranger - this docker usage was just particularly strange.
- e12e 5y ago
  (not op, but) I don't recall exactly what I experienced (in development- I think it was just terribly slow?) - I ended up carving out a volume (zvol) and formatting it with ext4 (or xfs?) for docker. It might actually have been this ms sql issue that led me to do that, actually: https://github.com/moby/moby/issues/33191 https://github.com/moby/moby/issues/33191
hddherman 5y ago
I've had bad experiences with Docker+Debian as well when the storage was on ZFS. I've had a much better experience by just keeping /var/log/containers on something simple, like ext4. Valuable data is still on ZFS and mounted inside the containers, no issues there.

yjftsjthsd-h 5y ago

> but has had zfs issues for me. Can you give more details so I know what to watch out for? (Recently started running Docker+ZFS in prod, haven't seen it break yet but appreciate heads up)

flemhans 5y ago
The zfs storage driver isn't production ready yet.
- r1ch 5y ago
  I love that they say this, then default to using it when ZFS is detected.
- cosmin800 5y ago
  I am pretty sure Canonical will soon release a stable,production ready snap package for zfs.
  - alyandon 5y ago
    That gets updated arbitrarily at random intervals (because who doesn't want filesystem modules updated frequently!?!?!) and then purges logs every 72 hours so when you have a problem you'll never even know what was updated! I have to run snapd on one of my servers and I hate everything about it.
    - flemhans 5y ago
      I'm considering to run lxd as a snap because I want Debian 11 (for openzfs 2.0) and lxd at the same time. Am I in for a world of hurt?
      - alyandon 5y ago
        I wouldn't say you are in for a world of hurt but you'll have to accept that the package will periodically update and in the process kill your containers/vms. My primary objections to snapd are: 1) updates are pushed/applied and services restarted on an arbitrary schedule which means you can't test updates prior to rolling them out or otherwise control when a service is taken offline by an update 2) snapd doesn't log to standard OS logging facilities and the logs it does have are cleared after 72 hours 3) snapd pollutes home directories with a directory called "snap" snapd is a poorly conceived project written by people that are profoundly ignorant of how real services and servers are actually operated. Despite having the above flaws repeatedly pointed out, the developers have shown zero interest in taking actual concrete actions to address them.
        flemhans 5y ago
        Thank you for the reply. It seems snapd is indeed a poorly conceived project. How would you work around it in a production setting if you HAD to use it? Auto-updating sounds like the biggest issue of those.
        zekrioca 5y ago
        Build LXD yourself. It is not that complicated if one updates go to something beyond 1.14. Only downside is that you may need to create your own init scripts and a lxd group, but that's not complicated either.
anotherhue 5y ago
there were a number of containers immune to 'docker rm', seemed it was trying to invoke 'zfs delete' on something that had already been removed (presumably moments ago as part of a previous step). Nasty workaround was to manually recreate the dataset (and children) so that it could remove them. The issue was common enough that I could google it so don't stress. More worryingly, 'prune' locked up for about 10 minutes before getting to do it's thing. pinning the cpu the whole time. This was on my devbox, so maybe it's my doing but did not inspire confidence. I've run FreeBSD ZFS for about a decade and openzfs/ubuntu in prod for kafka clusters so I'm no stranger - this docker usage was just particularly strange.
e12e 5y ago
(not op, but) I don't recall exactly what I experienced (in development- I think it was just terribly slow?) - I ended up carving out a volume (zvol) and formatting it with ext4 (or xfs?) for docker. It might actually have been this ms sql issue that led me to do that, actually: https://github.com/moby/moby/issues/33191 https://github.com/moby/moby/issues/33191

hddherman 5y ago

I've had bad experiences with Docker+Debian as well when the storage was on ZFS. I've had a much better experience by just keeping /var/log/containers on something simple, like ext4. Valuable data is still on ZFS and mounted inside the containers, no issues there.