3 ms·

What about when an SSH terminal gets locked up? If I'm running something and I want to abort it, I often find I can't because of the kernel thrashing. Often I s

What about when an SSH terminal gets locked up? If I'm running something and I want to abort it, I often find I can't because of the kernel thrashing. Often I see a nontrivial amount of CPU time given to kswapd but sometimes not, and more often I don't have any monitoring up to see what's going on, or the terminal I'm monitoring is also nonresponsive. This is especially a problem with Docker, I've found, because when I've got a container thrashing the system like this, I can't access the host or any other containers. Is this because I'm running single-core instances and the problem goes away once I have more than one thread available? Or should I be restricting workloads to a block device not used for caching? Does anyone else experience this?

deleted 13y ago
[deleted]
derefr 13y ago
Docker can set CPU, memory, and network-bandwidth quotas for containers. If you can guarantee that the aggregate total of each quota is less than the amount of that resource available to the host, then in the worst case you'll always have some of each left for a "rescue" container. This throws away one of the neat advantages of containers compared to VMs, though, which is that you can overcommit (or "thin-provision", if you want to be charitable) the host's resource allocation, since it's extremely unlikely that all your services will balloon in resource-consumption simultaneously. A less guaranteed, but more economical strategy, is to just set quotas on each container such that if it individually starts using more than, say, 80% of the host's resources for an extended period, it'll be terminated. This doesn't save you from a bad interaction between containers that makes everything explode in parallel (e.g. all your containers attempting to reconnect to a stuck service with no backoff) but it should save you in the majority of cases where each host has a heterogeneous container-load, and horizontal scale happens between hosts rather than internal to them. (The dev-side solution to this problem, though, is to just set up your software architecture such that any host that gets into such a state can hard-reboot, without you losing anything of consequence. http://12factor.net http://12factor.net -type containers excel at this, and it looks to be the strategy https://coreos.com https://coreos.com embraces as well.)
- AaronFriel 13y ago
  Docker supports (supported?) what LXC supported, which was relative CPU shares. I am not sure it would help if it supported something else (like quota minimum/maximum). When the IO subsystem is hit with a memory intensive workload that thrashes swap, all your guarantees go out the window. From my perspective (staring at top while an interactive terminal is not responding), it's the kernel that owns the CPU shares being used, and the IO volume, and it looks like the container is "almost idle" because most of the CPU time is being spent by the kernel. I don't know if it's that something Docker can fix. It might be something kernel devs can fix, though.

deleted 13y ago

[deleted]

derefr 13y ago

Docker can set CPU, memory, and network-bandwidth quotas for containers. If you can guarantee that the aggregate total of each quota is less than the amount of that resource available to the host, then in the worst case you'll always have some of each left for a "rescue" container. This throws away one of the neat advantages of containers compared to VMs, though, which is that you can overcommit (or "thin-provision", if you want to be charitable) the host's resource allocation, since it's extremely unlikely that all your services will balloon in resource-consumption simultaneously. A less guaranteed, but more economical strategy, is to just set quotas on each container such that if it individually starts using more than, say, 80% of the host's resources for an extended period, it'll be terminated. This doesn't save you from a bad interaction between containers that makes everything explode in parallel (e.g. all your containers attempting to reconnect to a stuck service with no backoff) but it should save you in the majority of cases where each host has a heterogeneous container-load, and horizontal scale happens between hosts rather than internal to them. (The dev-side solution to this problem, though, is to just set up your software architecture such that any host that gets into such a state can hard-reboot, without you losing anything of consequence. http://12factor.net http://12factor.net -type containers excel at this, and it looks to be the strategy https://coreos.com https://coreos.com embraces as well.)

AaronFriel 13y ago
Docker supports (supported?) what LXC supported, which was relative CPU shares. I am not sure it would help if it supported something else (like quota minimum/maximum). When the IO subsystem is hit with a memory intensive workload that thrashes swap, all your guarantees go out the window. From my perspective (staring at top while an interactive terminal is not responding), it's the kernel that owns the CPU shares being used, and the IO volume, and it looks like the container is "almost idle" because most of the CPU time is being spent by the kernel. I don't know if it's that something Docker can fix. It might be something kernel devs can fix, though.