4 ms·

That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service

That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.

ithkuil 3y ago
Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incidents will be created for conditions that may solve themselves because of another deployment (possibly of another team) happening. Overcommit/bursting is an advanced cost saving feature. Let me say it again: splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature! The problem is that the request/limits feature is presented in the configuration spec and in the documentation in a deceptively simple way and we're tricked to think it's a basic feature. Not all companies have ops teams that are well equipped to do more sophisticated things. My advice for those teams who cannot setup full automation around capacity management is to just not use this advanced features. An alternative is to just use smaller dedicated nodes and (anti)affinity rules, so you always understand which pods go with which other pods. It's clunky but it's actually easier to reason about what's going to happen. EDIT: typos
- otterley 3y ago
  > splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature That's an interesting take. For those of us who once used microcomputers back in the 1980s like Commodore 64s, Apple IIs, and IBM PCs running MS-DOS, sure. But time-sharing systems have been around since the 1960s and Unix dates back to the 1970s, and multi-user logins and resource controls like ulimits have been part of the picture for a very long time. We've had plenty of time to get used to multitasking and resource contention and management. Is it complicated? It can be. But if you consider that containers are basically a program lifecycle/deployment and resource control abstraction on an OS that's had these features (maybe not cgroups and namespaces, but similar ones) since its birth, it's not really all that advanced.
  - ithkuil 3y ago
    The same arguments hold on old time-sharing systems. Setting up you limits in such a way that you can use excess capacity when available is very well suited (now and back then) for batch workloads. Scheduler priorities are a way to mix batch and interactive workloads. When you're not latency sensitive but aiming at optimizing throughput you can achieve a pretty good utilization of the underlying resource. This is what resource<limit setting in k8s is good for: batch workload. Batch workload still exists and it's important but the proportion of batch vs latency sensitive workload shifted significantly from the 70s to internet age. Request handlers that power that sit in the critical path for the user experience of a web or mobile app are not only latency sensitive but also _tail_ latency sensitive. Tail latency often poses significant problems because stragglers can often determine the total duration of the interaction as perceived by the user, even if only one straggler request suffers from slowdowns. While there are tricks to deal with tail latency while also improving utilization (i.e. reducing waste) they are hard to implement in practice and/or don't generalize well across workloads. One thing always works and is your best option as a starting point: stop worrying about idle CPU in an _average_ usage metric. The behaviour during CPU usage spikes (even spikes that are shorter than your scraping period of your favourite monitoring tool!) determine latency envelope of your service. Measure how much CPU you need during those and plan accordingly. Ideally it should be possible to some low priority (batch) process use that idle CPU, but AFAIK that's not currently possible with k8s. EDIT: forgot to mention that all this discussion matters only inasmuch you have strict latency targets. If you're ok to occasionally have very slow requests and your customers are ok with that and your bosses don't freak out because they spoke to that one customer that is very angry that their requests time out a few times every month ... I'm very happy for you! Not everybody has these requirements and you can go pretty far with a production system consisting of a couple of VM on AWS without k8s, containers, cgroups or whatever in the picture. I know people who do and it works just fine for them. But in order to understand what the fuss about all of this we need to frame this discussion as pertaining to batch vs controlled-latency. Otherwise it's hard to explain why there are so many otherwise intelligent people making choices that appear to be a bit silly, isn't it? Sure, there are people who are "just wrong" on the internet :-) but more often than not if somebody ends up with a different solution is because they have different goals and tradeoffs.
eightnoteight 3y ago
imagine amazon's order history microservice causes outage for ordering creation microservice. regardless of the monitoring and alerting, you never want a non critical system to cause outage on the most critical system you have

ithkuil 3y ago

Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incidents will be created for conditions that may solve themselves because of another deployment (possibly of another team) happening. Overcommit/bursting is an advanced cost saving feature. Let me say it again: splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature! The problem is that the request/limits feature is presented in the configuration spec and in the documentation in a deceptively simple way and we're tricked to think it's a basic feature. Not all companies have ops teams that are well equipped to do more sophisticated things. My advice for those teams who cannot setup full automation around capacity management is to just not use this advanced features. An alternative is to just use smaller dedicated nodes and (anti)affinity rules, so you always understand which pods go with which other pods. It's clunky but it's actually easier to reason about what's going to happen. EDIT: typos

otterley 3y ago
> splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature That's an interesting take. For those of us who once used microcomputers back in the 1980s like Commodore 64s, Apple IIs, and IBM PCs running MS-DOS, sure. But time-sharing systems have been around since the 1960s and Unix dates back to the 1970s, and multi-user logins and resource controls like ulimits have been part of the picture for a very long time. We've had plenty of time to get used to multitasking and resource contention and management. Is it complicated? It can be. But if you consider that containers are basically a program lifecycle/deployment and resource control abstraction on an OS that's had these features (maybe not cgroups and namespaces, but similar ones) since its birth, it's not really all that advanced.
- ithkuil 3y ago
  The same arguments hold on old time-sharing systems. Setting up you limits in such a way that you can use excess capacity when available is very well suited (now and back then) for batch workloads. Scheduler priorities are a way to mix batch and interactive workloads. When you're not latency sensitive but aiming at optimizing throughput you can achieve a pretty good utilization of the underlying resource. This is what resource<limit setting in k8s is good for: batch workload. Batch workload still exists and it's important but the proportion of batch vs latency sensitive workload shifted significantly from the 70s to internet age. Request handlers that power that sit in the critical path for the user experience of a web or mobile app are not only latency sensitive but also _tail_ latency sensitive. Tail latency often poses significant problems because stragglers can often determine the total duration of the interaction as perceived by the user, even if only one straggler request suffers from slowdowns. While there are tricks to deal with tail latency while also improving utilization (i.e. reducing waste) they are hard to implement in practice and/or don't generalize well across workloads. One thing always works and is your best option as a starting point: stop worrying about idle CPU in an _average_ usage metric. The behaviour during CPU usage spikes (even spikes that are shorter than your scraping period of your favourite monitoring tool!) determine latency envelope of your service. Measure how much CPU you need during those and plan accordingly. Ideally it should be possible to some low priority (batch) process use that idle CPU, but AFAIK that's not currently possible with k8s. EDIT: forgot to mention that all this discussion matters only inasmuch you have strict latency targets. If you're ok to occasionally have very slow requests and your customers are ok with that and your bosses don't freak out because they spoke to that one customer that is very angry that their requests time out a few times every month ... I'm very happy for you! Not everybody has these requirements and you can go pretty far with a production system consisting of a couple of VM on AWS without k8s, containers, cgroups or whatever in the picture. I know people who do and it works just fine for them. But in order to understand what the fuss about all of this we need to frame this discussion as pertaining to batch vs controlled-latency. Otherwise it's hard to explain why there are so many otherwise intelligent people making choices that appear to be a bit silly, isn't it? Sure, there are people who are "just wrong" on the internet :-) but more often than not if somebody ends up with a different solution is because they have different goals and tradeoffs.

eightnoteight 3y ago

imagine amazon's order history microservice causes outage for ordering creation microservice. regardless of the monitoring and alerting, you never want a non critical system to cause outage on the most critical system you have