3 ms·

Is there such a thing as 95% threshold CPU monitoring? Consider an application spikes (close to 100% on a core) for 2-3s on some web requests -- let's assume t

Is there such a thing as 95% threshold CPU monitoring? Consider an application spikes (close to 100% on a core) for 2-3s on some web requests -- let's assume this is normal (nothing can be done about it). Now, let's consider the average user of the system is idle for 2 minutes per web request. So, users won't see performance degradation unless $(active-users) > $(cores) during a 2-3 minute window. For most monitoring systems, CPU is reported as an average over a minute, and, even if it's pinned only 2-3s per 60s, that's only 5% usage. Presume a 2 CPU system with 5 users, who all happen to be in a conference call... and hitting the system at exactly the same time (but are otherwise mostly idle). The CPU graph might show 10-15% usage (no flag). Yet, those 5 users will report significant application performance issues (one of the users will have to wait 6-9s). What I'd like to monitor, as a system administration, is the 95% utilization of the CPUs -- that is, over the minute, throw away the bottom 94% (mostly idle cycles) and report to me the CPU utilization of the next highest percentile. This should show me those pesky CPU spikes. Anything do that?

barrkel 11y ago
You're more interested in response time than CPU usage; the customer is having a bad time whether the slow responses are due to CPU or IO or bad weather.
- clarkevans 11y ago
  That's another (important) measure, but, not the one I'm interested in. In the very short run, I can increase CPUs if there is hardware contention. The response time is an indicator of, but doesn't prove, CPU starvation. Request response time analysis is a longer term quality of service measure to identify and fix application (if it is even feasible). ... There's a reason why ISPs bill at the 95% -- it's directly correlated with costs of over-subscribed resources. If you're running a VM cluster, you've got a similar issue, only with CPUs (and memory) as well as than bandwidth. Most operational divisions don't have the luxury of fixing a vendor application. Instead, they have limited variables to play with: CPU cores and memory being the coarse levers. While one could argue this is an "application" issue and you'd be correct, it's irrelevant to my question. I'm asking what tools are available for system administrators to diagnose and address CPU starvation (under spiked usage). Current tools and techniques I'm aware of don't seem to measure this.
  - dap 11y ago
    Once you've identified long response times, it would also be possible to observe that the threads handling requests spent a good percentage of time waiting for CPU, which would point to CPU saturation as the problem. I'm not sure how you do this on GNU/Linux, but you can assess this on illumos with ptime(1) or prstat(1M). I think your concern about measuring CPU utilization is real, though. You can use frequent sampling and present samples on a heat map to deal with this problem. There are some examples here: http://www.brendangregg.com/HeatMaps/utilization.html http://www.brendangregg.com/HeatMaps/utilization.html
  - wang_li 11y ago
    The only metric that matters is the measure of the business task at hand. You might dig into CPU utilization after you've identified a problem with your application, but trying to identify an application problem by measuring CPU is like trying to determine where your shipment is by looking at engine RPM of the freight vehicle. That being said, historic CPU utilization is a useful metric for capacity planning.
  - evook 11y ago
    Don't you overestimate an evening with strace and tcpdump a little bit as a "quality of service measure"? In my experience the reasons for high request response times are very easy to find, but not necessarily easy to fix. So at best it takes 30 minutes to find the root of the problem and 30 minutes to come up with a fix, at worst it takes 30 minutes to find and half a man-year to fix an entire stack.
StreamBright 11y ago
I am not aware of such tool. It is a good idea to report p50, p90, p99, p99.9 for CPU utilization but I also think it is good to report avg as well. Application performance issues should be monitored from the app point of view (response latency) and if you have that you can drill down and pin point CPU issues. Generally speaking it is better to monitor metrics from upper layers than just look at OS graphs. Most of the monitoring systems cover these in one dashboard so you can easily track down issues.
thrownaway2424 11y ago
I don't really understand your question. Fundamentally a process is either on the CPU, or not. There's nothing between 0 and 100% CPU usage. CPU utilization only makes sense with a windowing function, for example in 90% of samples over the last second, some process was on the CPU. I think what you want for your application, if I understand any of it correctly, is the time spent by a runnable process waiting to get scheduled on a CPU. That would indicate contention. If a process runs immediately when it becomes runnable, then there's no contention, no matter what the windowed utilization looks like.

barrkel 11y ago

You're more interested in response time than CPU usage; the customer is having a bad time whether the slow responses are due to CPU or IO or bad weather.

clarkevans 11y ago
That's another (important) measure, but, not the one I'm interested in. In the very short run, I can increase CPUs if there is hardware contention. The response time is an indicator of, but doesn't prove, CPU starvation. Request response time analysis is a longer term quality of service measure to identify and fix application (if it is even feasible). ... There's a reason why ISPs bill at the 95% -- it's directly correlated with costs of over-subscribed resources. If you're running a VM cluster, you've got a similar issue, only with CPUs (and memory) as well as than bandwidth. Most operational divisions don't have the luxury of fixing a vendor application. Instead, they have limited variables to play with: CPU cores and memory being the coarse levers. While one could argue this is an "application" issue and you'd be correct, it's irrelevant to my question. I'm asking what tools are available for system administrators to diagnose and address CPU starvation (under spiked usage). Current tools and techniques I'm aware of don't seem to measure this.
- dap 11y ago
  Once you've identified long response times, it would also be possible to observe that the threads handling requests spent a good percentage of time waiting for CPU, which would point to CPU saturation as the problem. I'm not sure how you do this on GNU/Linux, but you can assess this on illumos with ptime(1) or prstat(1M). I think your concern about measuring CPU utilization is real, though. You can use frequent sampling and present samples on a heat map to deal with this problem. There are some examples here: http://www.brendangregg.com/HeatMaps/utilization.html http://www.brendangregg.com/HeatMaps/utilization.html
- wang_li 11y ago
  The only metric that matters is the measure of the business task at hand. You might dig into CPU utilization after you've identified a problem with your application, but trying to identify an application problem by measuring CPU is like trying to determine where your shipment is by looking at engine RPM of the freight vehicle. That being said, historic CPU utilization is a useful metric for capacity planning.
- evook 11y ago
  Don't you overestimate an evening with strace and tcpdump a little bit as a "quality of service measure"? In my experience the reasons for high request response times are very easy to find, but not necessarily easy to fix. So at best it takes 30 minutes to find the root of the problem and 30 minutes to come up with a fix, at worst it takes 30 minutes to find and half a man-year to fix an entire stack.

StreamBright 11y ago

I am not aware of such tool. It is a good idea to report p50, p90, p99, p99.9 for CPU utilization but I also think it is good to report avg as well. Application performance issues should be monitored from the app point of view (response latency) and if you have that you can drill down and pin point CPU issues. Generally speaking it is better to monitor metrics from upper layers than just look at OS graphs. Most of the monitoring systems cover these in one dashboard so you can easily track down issues.

thrownaway2424 11y ago

I don't really understand your question. Fundamentally a process is either on the CPU, or not. There's nothing between 0 and 100% CPU usage. CPU utilization only makes sense with a windowing function, for example in 90% of samples over the last second, some process was on the CPU. I think what you want for your application, if I understand any of it correctly, is the time spent by a runnable process waiting to get scheduled on a CPU. That would indicate contention. If a process runs immediately when it becomes runnable, then there's no contention, no matter what the windowed utilization looks like.