4 ms·

This article mentions offhand that the storage drivers are unreliable, even for data volumes. Is that actually the case? Is there a serious risk that a databas

This article mentions offhand that the storage drivers are unreliable, even for data volumes. Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?

mstump 10y ago
I've been running a couple of petabytes in production with Docker and Cassandra for a couple years (around 2k nodes). I've rarely seen FS corruption, however I must qualify that this is on bare metal. They could be running into issues with the interaction with EBS? This is more of a screed than an argument backed by specific details and facts.
- tayo42 10y ago
  How do you handle making updates to configuration with all your nodes in containers? Do you blue green deploy the cluster or something? Run config management in the container?
  - mstump 10y ago
    Configuration comes from the environment. We store the configuration per cluster in a centralized store (C*, etcd, SimpleDB). We bake images that contain everything else. Depending on the customer and the tech involved we'll do blue-green by doing a controlled rolling push of the config or image after it makes it through the dev/test cycle. Also depending on the type of tech we'll store actual data on network or host volumes.
eicnix 10y ago
I can imagine that the probability of this happening is higher than when you run without docker. But on the other hand you have mechanisms like sharding and replications to deal with single machine/zone failure. I have run database clusters on kubernetes in production without running into this particular problem. The current state of container orchestrators for running databases is not optimal because one size does not fit all database types like with stateless applications. One solution for this problem are coreos operators which introduce third party resources into kubernetes that are specific to the database type and contain logic to manage this specific database type on kubernetes.
meta_AU 10y ago
While it is true that Docker has a variety of storage drivers for the overlay file system, some unreliable, if you are doing DB then you'd be using a host volume mount which is a bind mount. There should be no issues with the bind mount as it is not a part of Docker.
user5994461 10y ago
You'll want to read this, the links at the end and the comments: https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... There are major issues registered with AUFS, Overlay, Overlay2 and BTRFS. It's ridiculous.
- Kudos 10y ago
  But if you're bind mounting the data dir to the host you're not using any docker FS driver.
takeda 10y ago
In order for a database to provide such guarantees it needs some guarantees from an underlying hardware. Mainly that when disk say that data was written, it really was written, and that the data was written in the right order. The danger with overlay systems is that they might not provide these guarantees, which makes database writes unsafe. Given that current overlay drivers are unstable as it is, I doubt they put any effort to enforce such guarantees. Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM, unless you have a hardware that supports VT-d (a.k.a. PCI pass-through) and have it enabled.
- X-Istence 10y ago
  > Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM This is true for any and all file systems. ext4 can't provide those guarantees either because it's up to the hypervisor to do the right thing when the guest requests it to flush caches to disk. ZFS just happens to have made it very explicit that it requires certain guarantees to provide the data reliability, but those same requirements exist for other file systems.

mstump 10y ago

I've been running a couple of petabytes in production with Docker and Cassandra for a couple years (around 2k nodes). I've rarely seen FS corruption, however I must qualify that this is on bare metal. They could be running into issues with the interaction with EBS? This is more of a screed than an argument backed by specific details and facts.

tayo42 10y ago
How do you handle making updates to configuration with all your nodes in containers? Do you blue green deploy the cluster or something? Run config management in the container?
- mstump 10y ago
  Configuration comes from the environment. We store the configuration per cluster in a centralized store (C*, etcd, SimpleDB). We bake images that contain everything else. Depending on the customer and the tech involved we'll do blue-green by doing a controlled rolling push of the config or image after it makes it through the dev/test cycle. Also depending on the type of tech we'll store actual data on network or host volumes.

eicnix 10y ago

I can imagine that the probability of this happening is higher than when you run without docker. But on the other hand you have mechanisms like sharding and replications to deal with single machine/zone failure. I have run database clusters on kubernetes in production without running into this particular problem. The current state of container orchestrators for running databases is not optimal because one size does not fit all database types like with stateless applications. One solution for this problem are coreos operators which introduce third party resources into kubernetes that are specific to the database type and contain logic to manage this specific database type on kubernetes.

meta_AU 10y ago

While it is true that Docker has a variety of storage drivers for the overlay file system, some unreliable, if you are doing DB then you'd be using a host volume mount which is a bind mount. There should be no issues with the bind mount as it is not a part of Docker.

user5994461 10y ago

You'll want to read this, the links at the end and the comments: https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... There are major issues registered with AUFS, Overlay, Overlay2 and BTRFS. It's ridiculous.

Kudos 10y ago
But if you're bind mounting the data dir to the host you're not using any docker FS driver.

takeda 10y ago

In order for a database to provide such guarantees it needs some guarantees from an underlying hardware. Mainly that when disk say that data was written, it really was written, and that the data was written in the right order. The danger with overlay systems is that they might not provide these guarantees, which makes database writes unsafe. Given that current overlay drivers are unstable as it is, I doubt they put any effort to enforce such guarantees. Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM, unless you have a hardware that supports VT-d (a.k.a. PCI pass-through) and have it enabled.

X-Istence 10y ago
> Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM This is true for any and all file systems. ext4 can't provide those guarantees either because it's up to the hypervisor to do the right thing when the guest requests it to flush caches to disk. ZFS just happens to have made it very explicit that it requires certain guarantees to provide the data reliability, but those same requirements exist for other file systems.