5 ms·

There is a newer alternative, pressure stall information[0], split by CPU, memory and IO pressure. htop can display the rolling PSI averages and one can also se

There is a newer alternative, pressure stall information[0], split by CPU, memory and IO pressure. htop can display the rolling PSI averages and one can also see the total time during which some tasks were stalled. They're also available per cgroup. [0] https://lwn.net/Articles/759781/ https://lwn.net/Articles/759781/

mardifoufs 5y ago
Now that makes me wonder how does Windows deal with this? Say when you open the task manager, what does the cpu usage % actually mean? I'd guess it's not measuring it in the same way Linux does, but I'd be surprised if it used pressure stall information?
- SuperQue 5y ago
  Windows users just look at CPU utilization, which is what most Linux users should be doing as well. Monitoring both total system and single core utilization. Of course, 100% CPU use also isn't a "problem", it's just a fact. PSI is the thing that actually reveals problems. I'm not aware of a Windows equivalent.
  - mardifoufs 5y ago
    Okay so my question might not make a lot of sense but how does an OS or I'd guess a kernel knows what's the CPU utilization? I'm pretty sure it's reported by the CPU, but how does the CPU measure that? From my understanding, CPUs have multiple "units" that can specialize in specific operations. Is that taken into account, does it just measure the utilization of it's "general use" units, and if so how does the CPU keep track of it's own usage? I'd google it but I honestly don't know what to look for past the OS/kernel layer
    - johntb86 5y ago
      The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran. Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization. There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.
tanelpoder 5y ago
By the way, PSI has the same shortcoming of not accounting the time of threads waiting for async I/O completion as I/O pressure.
- zaarn 5y ago
  Yes, but async I/O means the process can continue doing other, potentially useful, work, while waiting for async I/O to complete. So if a process has issues an async I/O, it's not blocked or dead, like a process who issued sync I/O. And on the topside, if you block on the io_submit and friends, async I/O counts towards the PSI and load average again. Ie, when you can't do useful work because the device queue is full, for example. So it's not as much of a shortcoming as you think, because if the system is I/O exhausted, async I/O processes will quickly contribute to load as well once the I/O queues are nearly full or full.
  - tanelpoder 5y ago
    The reason why I'm aware of this is that Oracle databases use libaio for some reads and most of the writes on Linux, including transaction log writes. There are plenty of cases where you don't exhaust the block device I/O queues - io_submit() finishes quickly without blocking, but then the I/O issuing process wants to wait for the asynchronously submitted I/O completion, before moving on. It will use io_getevents() with the "timeout > 0" argument, so it will sleep (in S mode), waiting for I/O completion. If it had submitted a synchronous I/O, it would have increased Linux load, but if configured to use libaio, it won't. So, the I/O component of Linux system load and PSI do not include the async I/O waiting threads that decide to synchronoysly wait for asynchronously submitted I/O requests :-) I see MySQL is using libaio in some places too (the io_submit/io_getevents show up in syscalls), but I haven't looked deeper into whether the getevents are "willing to wait" or not. But any application using libaio, that at some point uses io_getevents() in "willing to wait" mode, would be affected by this discrepancy.
    - zaarn 5y ago
      I still think that is a reasonable caveat; the process didn't get hung up waiting for IO because IO was exhausted when it issues the write, rather it got hung because it deliberately chose to wait for IO to be completed. That isn't a load factor, it was the application doing it. Ideally Oracle DB (and MySQL) should track this and offer a performance metric somewhere (It's systemd service for example) of how long it's waiting for AIO completion. The crucial point there is that while the application is slow, the system remains as reactive as the PSI number indicates. Just the application didn't.

mardifoufs 5y ago

Now that makes me wonder how does Windows deal with this? Say when you open the task manager, what does the cpu usage % actually mean? I'd guess it's not measuring it in the same way Linux does, but I'd be surprised if it used pressure stall information?

SuperQue 5y ago
Windows users just look at CPU utilization, which is what most Linux users should be doing as well. Monitoring both total system and single core utilization. Of course, 100% CPU use also isn't a "problem", it's just a fact. PSI is the thing that actually reveals problems. I'm not aware of a Windows equivalent.
- mardifoufs 5y ago
  Okay so my question might not make a lot of sense but how does an OS or I'd guess a kernel knows what's the CPU utilization? I'm pretty sure it's reported by the CPU, but how does the CPU measure that? From my understanding, CPUs have multiple "units" that can specialize in specific operations. Is that taken into account, does it just measure the utilization of it's "general use" units, and if so how does the CPU keep track of it's own usage? I'd google it but I honestly don't know what to look for past the OS/kernel layer
  - johntb86 5y ago
    The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran. Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization. There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.

tanelpoder 5y ago

By the way, PSI has the same shortcoming of not accounting the time of threads waiting for async I/O completion as I/O pressure.

zaarn 5y ago
Yes, but async I/O means the process can continue doing other, potentially useful, work, while waiting for async I/O to complete. So if a process has issues an async I/O, it's not blocked or dead, like a process who issued sync I/O. And on the topside, if you block on the io_submit and friends, async I/O counts towards the PSI and load average again. Ie, when you can't do useful work because the device queue is full, for example. So it's not as much of a shortcoming as you think, because if the system is I/O exhausted, async I/O processes will quickly contribute to load as well once the I/O queues are nearly full or full.
- tanelpoder 5y ago
  The reason why I'm aware of this is that Oracle databases use libaio for some reads and most of the writes on Linux, including transaction log writes. There are plenty of cases where you don't exhaust the block device I/O queues - io_submit() finishes quickly without blocking, but then the I/O issuing process wants to wait for the asynchronously submitted I/O completion, before moving on. It will use io_getevents() with the "timeout > 0" argument, so it will sleep (in S mode), waiting for I/O completion. If it had submitted a synchronous I/O, it would have increased Linux load, but if configured to use libaio, it won't. So, the I/O component of Linux system load and PSI do not include the async I/O waiting threads that decide to synchronoysly wait for asynchronously submitted I/O requests :-) I see MySQL is using libaio in some places too (the io_submit/io_getevents show up in syscalls), but I haven't looked deeper into whether the getevents are "willing to wait" or not. But any application using libaio, that at some point uses io_getevents() in "willing to wait" mode, would be affected by this discrepancy.
  - zaarn 5y ago
    I still think that is a reasonable caveat; the process didn't get hung up waiting for IO because IO was exhausted when it issues the write, rather it got hung because it deliberately chose to wait for IO to be completed. That isn't a load factor, it was the application doing it. Ideally Oracle DB (and MySQL) should track this and offer a performance metric somewhere (It's systemd service for example) of how long it's waiting for AIO completion. The crucial point there is that while the application is slow, the system remains as reactive as the PSI number indicates. Just the application didn't.