4 ms·

I have tried to use Docker on very busy processing systems with short to medium running tasks. My efforts were on Ubuntu 16.04... because any non-LTS is almost

by cjhanks 10y ago

I have tried to use Docker on very busy processing systems with short to medium running tasks. My efforts were on Ubuntu 16.04... because any non-LTS is almost a non-starter. There were two major obstacles; the file system driver and the Docker daemon. I ended up settling on 1.11.{i forget}, because virtually every other version was unusable. As sad as it may sounds, I was tracking two metrics; probability the container will start and time until the docker daemon reaches a state of irrecoverable dead-lock. Personally I found containers (on a highly contentious system) launched at ~98% success rate with a time-to-dead-lock somewhere around 6500 container start/stop (it was a pretty fat tailed distribution). For a busy system... those numbers equate to an administrative headache. And on Ubuntu/Debian the defaults (the thing the majority of non-specialists are probably using) were far worse. The only filesystem driver which seemed to work at all was devicemapper direct-lvm. - If you have developers on your team who can spend their time figuring out the one magic incantation that makes Docker work most of the time, you're fine. Everybody else should follow his advice for using service providers.

mwpmaybe 10y ago
Were you using Device Mapper purposefully, or just because you couldn't get a more typical storage driver to work? The general setup instructions recommend either 1. installing linux-image-extra and using AUFS (for kernel versions < 4) or 2. modifying the dockerd invocation to specify the OverlayFS storage driver with `-s overlay` (for kernel versions >= 4). Admittedly, the latter strategy is a bit frustrating and counterintuitive on 16.04. When you install Docker it will start automatically and hang because it can't find the AUFS driver and Device Mapper isn't set up. The solution is to either 1. modify policy-rc.d to prevent services from automatically starting or 2. set up Device Mapper (install dmsetup and run `dmsetup mknodes`) before installing Docker and changing the storage driver. Unfortunately, these workarounds are not particularly well-documented. I would be interested to see how Docker 1.12 or 1.13 with a modern kernel and OverlayFS would be able to handle your described workload.
- user5994461 10y ago
  So you are saying that all default settings are unusable in production, to the point docker might not start at all, and the only cure is series of 5 obscure advanced system setup/configuration that are almost impossible to figure out by oneself and not documented, yet they should be totally obvious to anyone using Docker, right? FYI: It's because of this sort of bullshit that there are articles called "Docker in Production: An history of failure".
  - mwpmaybe 10y ago
    No, that's not what I'm saying. I'm saying that Ubuntu Server 16.04 LTS comes out of the box without AUFS and Device Mapper, and that you either need to enable one or the other before installing Docker or prevent Docker from starting automatically at install time. It sucks, yes, but your description of the problem and its solution is beyond hyperbolic. Googling "docker ubuntu install hang" takes you right to the GitHub issue with the solution at the bottom. And I think it's fixed in Docker 1.13 (it will prefer OverlayFS if AUFS and Device Mapper are not available), although I have not tested it.
    - user5994461 10y ago
      The GitHub issue: https://github.com/docker/docker/issues/23347 https://github.com/docker/docker/issues/23347 And there goes my hope of ever running Docker on Ubuntu with any success.
  - StreamBright 10y ago
    People like to waste their time on different things. :)
- cjhanks 10y ago
  I mean to say, device mapper was the only solution which worked in a production setting for me. I no longer work with that company and so I can't tell how those improvements would change the reliability. Presently I am using docker with btrfs on a low-throughput workload and it seems to work just fine. Tangentially related - I have been using OverlayFS without Docker for about a year now. It's pretty great. My read-layers can be on an NFS drive and the penalty for writes is relatively small. It basically gives me 90% of what I wanted from Docker in the first place.

by cjhanks 10y ago

mwpmaybe 10y ago

Were you using Device Mapper purposefully, or just because you couldn't get a more typical storage driver to work? The general setup instructions recommend either 1. installing linux-image-extra and using AUFS (for kernel versions < 4) or 2. modifying the dockerd invocation to specify the OverlayFS storage driver with `-s overlay` (for kernel versions >= 4). Admittedly, the latter strategy is a bit frustrating and counterintuitive on 16.04. When you install Docker it will start automatically and hang because it can't find the AUFS driver and Device Mapper isn't set up. The solution is to either 1. modify policy-rc.d to prevent services from automatically starting or 2. set up Device Mapper (install dmsetup and run `dmsetup mknodes`) before installing Docker and changing the storage driver. Unfortunately, these workarounds are not particularly well-documented. I would be interested to see how Docker 1.12 or 1.13 with a modern kernel and OverlayFS would be able to handle your described workload.

user5994461 10y ago
So you are saying that all default settings are unusable in production, to the point docker might not start at all, and the only cure is series of 5 obscure advanced system setup/configuration that are almost impossible to figure out by oneself and not documented, yet they should be totally obvious to anyone using Docker, right? FYI: It's because of this sort of bullshit that there are articles called "Docker in Production: An history of failure".
- mwpmaybe 10y ago
  No, that's not what I'm saying. I'm saying that Ubuntu Server 16.04 LTS comes out of the box without AUFS and Device Mapper, and that you either need to enable one or the other before installing Docker or prevent Docker from starting automatically at install time. It sucks, yes, but your description of the problem and its solution is beyond hyperbolic. Googling "docker ubuntu install hang" takes you right to the GitHub issue with the solution at the bottom. And I think it's fixed in Docker 1.13 (it will prefer OverlayFS if AUFS and Device Mapper are not available), although I have not tested it.
  - user5994461 10y ago
    The GitHub issue: https://github.com/docker/docker/issues/23347 https://github.com/docker/docker/issues/23347 And there goes my hope of ever running Docker on Ubuntu with any success.
- StreamBright 10y ago
  People like to waste their time on different things. :)
cjhanks 10y ago
I mean to say, device mapper was the only solution which worked in a production setting for me. I no longer work with that company and so I can't tell how those improvements would change the reliability. Presently I am using docker with btrfs on a low-throughput workload and it seems to work just fine. Tangentially related - I have been using OverlayFS without Docker for about a year now. It's pretty great. My read-layers can be on an NFS drive and the penalty for writes is relatively small. It basically gives me 90% of what I wanted from Docker in the first place.