6 ms·

Goodbye Docker on CentOS, Hello Ubuntu

71 points by rusher81572 11y ago

vonklaus 11y ago
Any chance you can link to a "getting started" docker on Ubuntu writeup?
- rusher81572 11y ago
  I should write one that compliments that post. Good idea =)
- Jedd 11y ago
  This one was posted on HN a few days ago. Not specifically Ubuntu-related, but includes Ubuntu. Also aimed at EC2 rather than DO, however one imagines the ideas behind a fully portable ersatz VM approach are, well, fully portable. https://news.ycombinator.com/item?id=10890233 https://news.ycombinator.com/item?id=10890233
cmurf 11y ago
I haven't used AUFS or overlay(fs), only devicemapper (thin provisioning) and Btrfs. Btrfs is faster than devicemapper, but even creating and removing containers seems slower than it ought to be considering how little delta there is. The create/remove time with 'btrfs sub crea/del' is much faster than docker create or remove (container), so I'm not really sure where the delays are.
- rusher81572 11y ago
  Good comment. Did you try on CentOS or Ubuntu?
  - cmurf 11y ago
    Fedora Cloud Atomic 23 on an Intel NUC.
lox 11y ago
For some background: https://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/ https://developerblog.redhat.com/2014/09/30/overview-storage... Overlay(fs) is likely going to be the way forward.
Patrick_Devine 11y ago
You can find some notes on tuning devicemapper here: https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html#44 https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-div... If you've got a new enough kernel though (i.e. 3.18+), you're best off using Overlay for your storage driver. It's fast and doesn't require a lot of tuning.
- rodgerd 11y ago
  > you're best off using Overlay for your storage driver. As long as you don't mind gifting root to your container tenants.
- elricL 11y ago
  Last I checked, OverlayFS did not play well with tools like Yum/Pip. Has this been fixed?
infodroid 11y ago
By default, Docker will use the AUFS storage backend if available, and then fall back to devicemapper on loopback. RHEL, CentOS, and Fedora do not ship the AUFS kernel module because it is not part of the mainline Linux kernel and is unlikely to be included in future, and these distros have an "upstream first, no out-of-tree bits" policy. Instead, they recommend using devicemapper on LVM [1][2]. The same advice is provided in the official Docker documentation [3]: > Docker hosts running the devicemapper storage driver default to a configuration mode known as loop-lvm... The mode is designed to work out-of-the-box with no additional configuration. However, production deployments should not run under loop-lvm mode... The preferred configuration for production deployments is direct lvm. You might consider using CentOS Atomic Host, which comes preconfigured with LVM thin pools. OverlayFS is also an alternative, but it can be problematic. It only implements a subset of the POSIX standard [4], which can cause some programs to fail. [1] http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/ http://www.projectatomic.io/blog/2015/06/notes-on-fedora-cen... [2] https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers https://access.redhat.com/documentation/en/red-hat-enterpris... [3] https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/ https://docs.docker.com/engine/userguide/storagedriver/devic... [4] https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/overl...
- mbreese 11y ago
  Yeah, I'm a bit surprised that the author didn't try direct LVM for their device. The speed up is noticeable.
- dvusboy 11y ago
  The author of the original blog is confusing the issue of base-image of a Docker image and the host OS on which the Docker daemon is running. They are completely orthogonal issues. Yes, there've been plenty of complaints about DM performance and space reclamation issues. By all means, switch the host OS from CentOS/Fedora to Ubuntu if it alleviates the problems. The base-image is a completely different matter. There is no reason to switch from CentOS/Fedora to Ubuntu just because you changed the host OS. This is point of the filesystem isolation Docker provides.
hhw 11y ago
From the op: "I enjoyed its minimal install to create a light environment, intuitive installation process, and it’s package manager." CentOS is neither minimal nor light. Ubuntu also isn't either of these things. Both of these distributions are more targeted towards convenience and ease of use, which means a lot of features/services that are generally unnecessary are enabled by default. The main reasons to use CentOS is compatibility with proprietary software made for RHEL, Ubuntu for people already familiar with its desktop version, or if needing to buy enterprise support for either. If the op is primarily looking for minimal and light, he should look at pretty much any other major popular Linux distribution like Debian proper, Slackware, or Gentoo before CentOS or Ubuntu.
- tomrod 11y ago
  I had a different takeaway from that particular line. Maybe I'm understanding it wrong. For many years I would craft a system from the Ubuntu minimal install. It appears CentOS has a similar installer.[0] With Ubuntu's one could create a pretty good minimal install. [0] https://www.centos.org/download/ https://www.centos.org/download/
  - rusher81572 11y ago
    That was my intention.
- Jedd 11y ago
  Yeah, this surprised me too, given the stated interest in a light-weight, server (no GUI) platform. But we see this a lot - people jump onto Ubuntu despite not having an interest in running Unity, Mir, upstart, etc, or having a slightly fancier installer, or different schedule on release of stable / long term support versions. I suspect it's simply that for many people unfamiliar with the various GNU/Linux distros, Ubuntu is the one they've heard of more often. And while most people marvel at the sophisticated package management system, few ponder why the package files don't have a .ubu suffix.
  - ricardobeat 11y ago
diebir 11y ago
I would rather say "goodbye docker", period. Having used it a bit, I have concluded that it is just not a very good or mature tool. Once you scratch under the surface, things cease to be easy or stop working at all. It makes you wonder how it's been this long and nobody stumbled upon an issue that you uncover 30 minutes into tool's use. Docker does have it's uses, but in the majority of cases you are better off using native OS package and dependency management (RPM/YUM in the case on Redhat-based distros). One very obvious thing is that package managers usually track versions and dependencies and allow for install actions to happen based on the versions delta. With Docker you just replace the whole environment, which is fine, unless some of the data context is outside of Docker. I think the majority of Docker uses are of a class "I can't figure out how to manage dependencies for a given OS, so I am going to skirt the issue by using Docker".
- meddlepal 11y ago
  Well... that might be a reason why it's popular. I think the very snappy boot up process for a container is also very appealing. For example, t2.micro's on AWS seem to take at least 30 seconds which is frustratingly slow if you're changing stuff often but only touching application/service dependencies.
  - creshal 11y ago
    It's not "docker or VMs", it's "docker or other container frameworks". Compared to those, Docker seems to come with a ludicrous list of caveats and workarounds.
    - meddlepal 11y ago
      Good point. I haven't kept up with the maturity status of other container frameworks though so I'm not sure how prevalent stuff like Rocket is in the wild.
- s986s 11y ago
  You say that as if its a bad thing. Isnt the evolution in development within making it more intuitive? I love the 10,000 line man page as much as the next dev, except I Dont. Docker solves a problem, standizing scalability and microservices without the need to think about whether its on the same or diff machines from developing the application. Honestly, I think it could be better. I wouldnt mind if the volume concept was easier for me to wrap my head around. I would like the cleanup process to be less worrysome. But thet have built an awesome solution and it opensource. What more can you ask for?
akeck 11y ago
I've had different trouble with Docker on CentOS 7. Last November, I created a POC Kubernetes cluster on top of the shipping CentOS 7 Docker infrastructure (1.8 at the time), and then beat the hell out of it. After a week or so, I lost a node to XFS file system corruption in the Docker image tree. The only solution I could find that worked was uninstalling Docker, wiping the Docker image tree in /var, and then reinstalling Docker. Kubernetes would then resume distributing containers to the node. Every node died in this manner at least once during the POC. With Kubernetes managing the containers, it wasn't a disaster - just really annoying.
codinghorror 11y ago
I strongly DO NOT recommend using devicemapper as storage in Docker. Every time we have tried, and every customer who has tried, has failed medium to long term, in bad ways. It became so painful that we literally blacklist devicemapper as a supported filesystem in the Discourse installer. We waited a year for this to "stabilize" but it never did.
stefanha 11y ago
Never had performance issues with the Docker device-mapper graph driver on Fedora 23. No tweaking was necessary. I imagine CentOS is similar although it may not use the latest and greatest features. There is not enough information in this blog post to say exactly what the problem is, but switching distros may be overkill here.

71 points by rusher81572 11y ago

vonklaus 11y ago

Any chance you can link to a "getting started" docker on Ubuntu writeup?

rusher81572 11y ago
I should write one that compliments that post. Good idea =)
Jedd 11y ago
This one was posted on HN a few days ago. Not specifically Ubuntu-related, but includes Ubuntu. Also aimed at EC2 rather than DO, however one imagines the ideas behind a fully portable ersatz VM approach are, well, fully portable. https://news.ycombinator.com/item?id=10890233 https://news.ycombinator.com/item?id=10890233

cmurf 11y ago

I haven't used AUFS or overlay(fs), only devicemapper (thin provisioning) and Btrfs. Btrfs is faster than devicemapper, but even creating and removing containers seems slower than it ought to be considering how little delta there is. The create/remove time with 'btrfs sub crea/del' is much faster than docker create or remove (container), so I'm not really sure where the delays are.

rusher81572 11y ago
Good comment. Did you try on CentOS or Ubuntu?
- cmurf 11y ago
  Fedora Cloud Atomic 23 on an Intel NUC.

lox 11y ago

For some background: https://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/ https://developerblog.redhat.com/2014/09/30/overview-storage... Overlay(fs) is likely going to be the way forward.

Patrick_Devine 11y ago

You can find some notes on tuning devicemapper here: https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html#44 https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-div... If you've got a new enough kernel though (i.e. 3.18+), you're best off using Overlay for your storage driver. It's fast and doesn't require a lot of tuning.

rodgerd 11y ago
> you're best off using Overlay for your storage driver. As long as you don't mind gifting root to your container tenants.
elricL 11y ago
Last I checked, OverlayFS did not play well with tools like Yum/Pip. Has this been fixed?

infodroid 11y ago

By default, Docker will use the AUFS storage backend if available, and then fall back to devicemapper on loopback. RHEL, CentOS, and Fedora do not ship the AUFS kernel module because it is not part of the mainline Linux kernel and is unlikely to be included in future, and these distros have an "upstream first, no out-of-tree bits" policy. Instead, they recommend using devicemapper on LVM [1][2]. The same advice is provided in the official Docker documentation [3]: > Docker hosts running the devicemapper storage driver default to a configuration mode known as loop-lvm... The mode is designed to work out-of-the-box with no additional configuration. However, production deployments should not run under loop-lvm mode... The preferred configuration for production deployments is direct lvm. You might consider using CentOS Atomic Host, which comes preconfigured with LVM thin pools. OverlayFS is also an alternative, but it can be problematic. It only implements a subset of the POSIX standard [4], which can cause some programs to fail. [1] http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/ http://www.projectatomic.io/blog/2015/06/notes-on-fedora-cen... [2] https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers https://access.redhat.com/documentation/en/red-hat-enterpris... [3] https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/ https://docs.docker.com/engine/userguide/storagedriver/devic... [4] https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/overl...

mbreese 11y ago
Yeah, I'm a bit surprised that the author didn't try direct LVM for their device. The speed up is noticeable.
dvusboy 11y ago
The author of the original blog is confusing the issue of base-image of a Docker image and the host OS on which the Docker daemon is running. They are completely orthogonal issues. Yes, there've been plenty of complaints about DM performance and space reclamation issues. By all means, switch the host OS from CentOS/Fedora to Ubuntu if it alleviates the problems. The base-image is a completely different matter. There is no reason to switch from CentOS/Fedora to Ubuntu just because you changed the host OS. This is point of the filesystem isolation Docker provides.

hhw 11y ago

From the op: "I enjoyed its minimal install to create a light environment, intuitive installation process, and it’s package manager." CentOS is neither minimal nor light. Ubuntu also isn't either of these things. Both of these distributions are more targeted towards convenience and ease of use, which means a lot of features/services that are generally unnecessary are enabled by default. The main reasons to use CentOS is compatibility with proprietary software made for RHEL, Ubuntu for people already familiar with its desktop version, or if needing to buy enterprise support for either. If the op is primarily looking for minimal and light, he should look at pretty much any other major popular Linux distribution like Debian proper, Slackware, or Gentoo before CentOS or Ubuntu.

tomrod 11y ago
I had a different takeaway from that particular line. Maybe I'm understanding it wrong. For many years I would craft a system from the Ubuntu minimal install. It appears CentOS has a similar installer.[0] With Ubuntu's one could create a pretty good minimal install. [0] https://www.centos.org/download/ https://www.centos.org/download/
- rusher81572 11y ago
  That was my intention.
Jedd 11y ago
Yeah, this surprised me too, given the stated interest in a light-weight, server (no GUI) platform. But we see this a lot - people jump onto Ubuntu despite not having an interest in running Unity, Mir, upstart, etc, or having a slightly fancier installer, or different schedule on release of stable / long term support versions. I suspect it's simply that for many people unfamiliar with the various GNU/Linux distros, Ubuntu is the one they've heard of more often. And while most people marvel at the sophisticated package management system, few ponder why the package files don't have a .ubu suffix.
- ricardobeat 11y ago

diebir 11y ago

I would rather say "goodbye docker", period. Having used it a bit, I have concluded that it is just not a very good or mature tool. Once you scratch under the surface, things cease to be easy or stop working at all. It makes you wonder how it's been this long and nobody stumbled upon an issue that you uncover 30 minutes into tool's use. Docker does have it's uses, but in the majority of cases you are better off using native OS package and dependency management (RPM/YUM in the case on Redhat-based distros). One very obvious thing is that package managers usually track versions and dependencies and allow for install actions to happen based on the versions delta. With Docker you just replace the whole environment, which is fine, unless some of the data context is outside of Docker. I think the majority of Docker uses are of a class "I can't figure out how to manage dependencies for a given OS, so I am going to skirt the issue by using Docker".

meddlepal 11y ago
Well... that might be a reason why it's popular. I think the very snappy boot up process for a container is also very appealing. For example, t2.micro's on AWS seem to take at least 30 seconds which is frustratingly slow if you're changing stuff often but only touching application/service dependencies.
- creshal 11y ago
  It's not "docker or VMs", it's "docker or other container frameworks". Compared to those, Docker seems to come with a ludicrous list of caveats and workarounds.
  - meddlepal 11y ago
    Good point. I haven't kept up with the maturity status of other container frameworks though so I'm not sure how prevalent stuff like Rocket is in the wild.
s986s 11y ago
You say that as if its a bad thing. Isnt the evolution in development within making it more intuitive? I love the 10,000 line man page as much as the next dev, except I Dont. Docker solves a problem, standizing scalability and microservices without the need to think about whether its on the same or diff machines from developing the application. Honestly, I think it could be better. I wouldnt mind if the volume concept was easier for me to wrap my head around. I would like the cleanup process to be less worrysome. But thet have built an awesome solution and it opensource. What more can you ask for?

akeck 11y ago

I've had different trouble with Docker on CentOS 7. Last November, I created a POC Kubernetes cluster on top of the shipping CentOS 7 Docker infrastructure (1.8 at the time), and then beat the hell out of it. After a week or so, I lost a node to XFS file system corruption in the Docker image tree. The only solution I could find that worked was uninstalling Docker, wiping the Docker image tree in /var, and then reinstalling Docker. Kubernetes would then resume distributing containers to the node. Every node died in this manner at least once during the POC. With Kubernetes managing the containers, it wasn't a disaster - just really annoying.

codinghorror 11y ago

I strongly DO NOT recommend using devicemapper as storage in Docker. Every time we have tried, and every customer who has tried, has failed medium to long term, in bad ways. It became so painful that we literally blacklist devicemapper as a supported filesystem in the Discourse installer. We waited a year for this to "stabilize" but it never did.

stefanha 11y ago

Never had performance issues with the Docker device-mapper graph driver on Fedora 23. No tweaking was necessary. I imagine CentOS is similar although it may not use the latest and greatest features. There is not enough information in this blog post to say exactly what the problem is, but switching distros may be overkill here.