7 ms·
> In order to make that fix, we needed to access the Kubernetes control plane – which we could not do due to the increased load to the Kubernetes API servers.
by btown 2y ago
> In order to make that fix, we needed to access the Kubernetes control plane – which we could not do due to the increased load to the Kubernetes API servers.
Something that I wish all databases and API servers would do, and that few actually do in practice, is to allocate a certain amount of headroom (memory and CPU) to "break glass in case of emergency" sessions. Have an interrupt fired periodically that listens exclusively on a port that will only be used for emergency instructions (but uses equal security measures to production, and is only visible internally). Ensure that it can allocate against a preallocated block of memory; allow it to schedule higher-priority threads. A small concession to make in the usual course of business, but when it's useful it's vital.
- dotancohen 2y agoI have a 1 GiB file to rm in case of emergency on every filesystem I manage - both personal and professional. I've only had to delete it maybe three or four times that I remember, but it's been a system-saver each time. I've long considered writing a process that just consumes some CPU, memory, and bandwidth for the same emergency buffer. I've even considered that it should even examine `last` every second and free up those resources when I SSH in.
- tux3 2y agoIt is also common for filesystems to reserve a small percentage to the root user. I think the ext4 default is still 5% (which can be quite a bit more than 1GB on modern drives!)
- timschmidt 2y agoYup. 'tune2fs -m0' has saved my bacon more than once.
- dotancohen 2y agoI haven't used root directly for over a decade. Modern usage is to log in as an unprivileged user, and to use sudo for all root operations.
- skywhopper 2y agoWhat’s your point here? Using sudo is using root.
- tucnak 2y agoI think you missed the joke here
- tux3 2y agoI don't think they were joking? I personally tend to leave a root shell open (and both my feet remain largely hole-free to this day), but it's pretty common advice, to avoid accidentally typing something unfortunate like rm ./* into the wrong shell
- tucnak 2y agoYeah, no. If you're going to be doing `rm /` you might as well do `sudo rm /` just as easily. It's the same security model, and honestly the distinction is quite funny.
- tux3 2y agoNo, there's a distinction if you look again. People don't accidentally type sudo in front of commands, but people do type things into the wrong window, or the wrong tab.
- hank808 2y agoWe used to refer to this as a "cork file" back in the day.
- jiggawatts 2y agoMicrosoft SQL Server has a dedicated admin connection (DAC) feature.
- cies 2y agoIn Zig you have to handle the case malloc does not succeed.
- remram 2y agoYou always have a load-balancer in front of your control plane (apiservers) if you have more than 1, that's what I would use to break the glass. You need your engineers to know how to do that though. You don't even need to break everything, take 1 apiserver out for admin access (provided etcd is not overwhelmed too).
- btown 2y agoYep - my understanding of https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing https://github.com/kubernetes/kubeadm/blob/main/docs/ha-cons... is that Kubernetes doesn't usually control that load balancer (nor should it, since you could accidentally tell Kubernetes to take down the control plane LB, then not be able to get it back up again!). I suppose one could set up a "fast pass lane" kubeconfig that adds a header that haproxy would understand, and route to a priority class in its queue with e.g. https://www.haproxy.com/documentation/haproxy-configuration-tutorials/service-reliability/overload-protection/#http-request-priority-queue https://www.haproxy.com/documentation/haproxy-configuration-... . But there's no easy `kubectl --with-priority` (or, to my knowledge, good guidelines for the various gitops solutions) that follows this pattern out of the box.
- remram 2y agoI just connect to the backend server directly after taking that backend out of the load-balanced pool. Easier than going through the load-balancer in a special way.
- tcdent 2y agoResource exhaustion can be super frustrating, and always feels like a third world slum situation when it happens. Like, why would operating systems allow themselves to run out of headroom entirely in this day and age?
- TeMPOraL 2y ago> a third world slum situation when it happens > why would operating systems allow themselves to run out of headroom entirely in this day and age? Following the analogy, perhaps for the same reason the richest cities in the richest, most developed Western nations, are rapidly beginning to look very much like third-world slums? Over-optimization is the name of the game. All systems need some amount of slack in them to stay flexible, robust (and livable). Unfortunately, cutting into the slack is always profitable on the margin, so without top-down intervention, the slack will be cut into until it's gone entirely. "Look, these machines are utilized only to 75% of capacity; adding this new system will only increase it by 5%"; "look, we can save XX$/month by cutting on compute, the machines will still be maxing out at 90%, so we'll still have a buffer". "Oh, this new telemetry service will bump that only by 1%". "Look, there's so much free space here in between these blocks of flats; adding another block won't hurt." And so on. Until your machines are running at 99% capacity and you risk global outage every time someone sneezes near the server room. Until your city starts to look like London, and if you're from Central Europe like me, you may start to realize that being a few months or years behind on the most recent gadgets is small price to pay in exchange for cities that are affordable and clean.
- skywhopper 2y agoThey usually don’t? If they did, they would have just crashed. If they are still running, then they allowed themselves headroom. The real question is who gets to use that headroom and how? What happens when the administrative processes use “too many” resources? There are tons of tools built into modern OSes to manage prioritization of processes, etc. Whether it’s worth the tradeoff for you to deal with the operational overhead of harnessing those features is up to you.
- throwaway290 2y agoBecause it makes computer slower and people pay for compute. I think consumer systems do it. macos shows the "quit some app, you're out of ram" but the system itself works. but if you are asking that OS knows this is containers process and this is control plane process and treat them differently, I think no one does that. Imagine how frustrating would it be if your os does it wrongly and you end up paying 10x because the os nerfed your main process because it thought something more important was running
- xyzzy123 2y agoAs I understand it you can configure api rate limiting per-user but you'd need to work out "reasonable" values on your own to allow enough headroom for admin requests. This would require per-cluster testing (and is complex to test since you need to induce representative load) so I suppose hardly anyone does it.
- dilyevsky 2y agoThere’s a whole set of features built[0] to prevent this exact scenario - runaway controllers consuming all api resources. Not sure why it didn’t work, maybe they are running an old release [0] - https://kubernetes.io/docs/concepts/cluster-administration/flow-control/ https://kubernetes.io/docs/concepts/cluster-administration/f...