4 ms·
Nice. Thanks for sharing lessons learned and tools that encode this knowledge. I work on container cluster management full time myself. I am focusing on AWS EC
by nzoschke 11y ago
Nice. Thanks for sharing lessons learned and tools that encode this knowledge.
I work on container cluster management full time myself. I am focusing on AWS ECS so the problems are very different technically but very similar conceptually.
The question is who watches the watchmen?
A container scheduler is supposed to be responsible for maintaining the entire health of the cluster. But if it has fundamental troubles in doing so, how do you automatically detect this and get it back into a working state?
On ECS I have a an agent container running one every instance that terminates the instance on observed failures. The most common problems I have observed are a bad disk (full, read-only, or too slow) and a locked up docker daemon.
I also schedule one more monitor process in the cluster that periodically monitors the ECS, EC2 and ASG APIs. A common failure is instances that lose ECS agent connectivity and need to be terminated.
All this hard won knowledge is encoded in the open source Convox platform: https://github.com/convox/rack https://github.com/convox/rack
The next problem is that sometimes this monitor container stops working due to the very problems it's trying to correct! I plan to move it to a Lambda task to remove the correlated failure.
But I always wonder. Why aren't these problems handled natively by Amazon and ECS?
The same question applies to this post. If you have to run additional monitoring to make kubernetes work reliably long term, can we consider that a kubernetes bug?
- alexk 11y agoI agree - detecting failures in a distributed system is very hard, especially if you don't know what's the root cause right away. I also think it would be very helpful to have a set of standard platform health and diagnostic checks for systems like these, so we can plug various forms of monitoring and roll our own app/deployment specific logic. I have no doubt that K8s will get there, it already has a very extensive self-testing e2e testing suite built into it's core. One of the reasons why we've implemented and open sourced satellite is to continue brainstorming and propose a possible solution to the problem of detecting failures on a running system.
- deemok 11y agoI don't think kubernetes is to blame for that. One of the facets of a really complex system is that there're way more moving parts that a generic system can handle or anticipate all possible failures. Sure, with time, kubernetes might get a mature native monitoring setup or expose a higher-level support for monitoring than what's currently available in terms of cAdvisor/heapster and health endpoints. So we built the simplest thing to achieve what we needed at certain point in hopes that the simpler it is, the less "monitoring the monitor itself will require".
- brendandburns 11y agoKubernetes handles most of this seemlessly for the cluster infrastructure. The central master handles node failures by removing nodes that aren't heartbeating. On the node, we require a process monitor for the kubelet (by default we use supervisord) but then the kubelet monitors Docker [and also does garbage collection and resource limiting ], and then all of the other node daemons (e.g. the kubernetes proxy) are run/monitored/restarted by the kubelet.
- alexk 11y agoFully agree - K8s has tons lots of self-healing capabilities as long as it functions correctly. Our goal was to extend this with a lower layer that will detect failures that are mission critical for k8s deployments to run properly like etcd, docker state and skydns