5 ms·

Is Your Linux Version Hiding Interrupt CPU Usage from You?

133 points by tanelpoder 6y ago

tinus_hn 6y ago
The truth is out there! The conspiracy is real!
- dang 6y ago
  Please don't do this here.
deleted 6y ago
[deleted]
chrismorgan 6y ago
Mods (dang): could we get the fragment specifier (#how-to-measure-interrupt-cpu-overhead-when-irq-time-accounting-is-disabled) removed from the URL?
londons_explore 6y ago
Does this CPU time accounting have any noticeable performance impact? I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.
- shakna 6y ago
  > Does this CPU time accounting have any noticeable performance impact? Yes. > Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact. [0] https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html
krzyk 6y ago
CONFIG_IRQ_TIME_ACCOUNTING according to the docs: CONFIG_IRQ_TIME_ACCOUNTING: Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small *performance impact*. If in doubt, say N here. So I didn't enable this option in my kernels afraid of loosing performance on my laptop.
- tanelpoder 6y ago
  (Author here). It would be interesting to test the performance difference. I downloaded the source of one of my distro-provided kernels & recompiled it with different config setting (to compare perf with just one thing changed), but got into some trouble with my drivers and ran out of time. I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit. In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue. edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.
happyflower 6y ago
On a typical laptop, what level of interrupts is too high? And what can be done about it?
- joosters 6y ago
  The tool 'powertop' is very useful here, it will show what is causing any form of wakeup on your system, and offers a 'Tunables' tab with suggested settings to improve some aspects.
  - LinuxBender 6y ago
    Powertop is great. You can even run it in non-interactive mode. That said, it only optimizes for lower power usage and lower interrupts. You can further reduce interrupts by disabling devices. If there are devices you don't need, you can blacklist them in /etc/modprobe.d/ provided you understand the potential dependencies other devices may have on the drivers. This can free up a tiny bit of memory as well.
irqquestion 6y ago
Can someone explain this in further detail. What is causing interrupt CPU usage on my machine. When I run powertop I see this: 629 mW 2,6 ms/s 158,2 Interrupt [0] HI_SOFTIRQ Is this related or another story and what can I do about it?
- hedora 6y ago
  Try disabling hardware devices until it goes away.
archi42 6y ago
Ahh, yes, that's actually a great point to be aware of, thanks @tanelpoder. Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.
andy_threos_io 6y ago
Irq accounting depends on architecture and code usage, e.g. like some kind of barrier (serializing) before the RDTSC. on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.). The RDTSC and fence timings (Latency, Reciprocal throughput) [0]: AMD Zen 1 SFENCE 4 ~20 LFENCE 1 0.25 MFENCE 7 ~70 RDTSC 37 36 AMD Zen 2 SFENCE 1 1 LFENCE 1 11 MFENCE 7 ~78 RDTSC 37 37 AMD Zen 3 SFENCE 1 1 LFENCE 1 10 MFENCE 6 ~60 RDTSC 44 36 Intel Haswell LFENCE 2 4 MFENCE 3 33 SFENCE 2 5 RDTSC 15 24 Intel Broadwell LFENCE 2 4 MFENCE 3 33 SFENCE 2 6 RDTSC 15 24 Intel Skylake LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 Intel Skylake-X LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 Intel Coffee Lake LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 [0] https://www.agner.org/optimize/instruction_tables.pdf https://www.agner.org/optimize/instruction_tables.pdf
- tanelpoder 6y ago
  I guess the fence instructions are there for avoiding the CPU out-of-order execution getting the timestamp too early (or late) compared to the interrupt service routine's start & end?
3np 6y ago
Thank you, this is something I somewhat recently realized I don't know how to measure and troubleshoot properly - I've never been able to give a confident answer to if context switching/interrupts are a bottleneck or not, just "the number looks highish and I don't have anything else to blame".

133 points by tanelpoder 6y ago

tinus_hn 6y ago

The truth is out there! The conspiracy is real!

dang 6y ago
Please don't do this here.

deleted 6y ago

[deleted]

chrismorgan 6y ago

Mods (dang): could we get the fragment specifier (#how-to-measure-interrupt-cpu-overhead-when-irq-time-accounting-is-disabled) removed from the URL?

londons_explore 6y ago

Does this CPU time accounting have any noticeable performance impact? I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.

shakna 6y ago
> Does this CPU time accounting have any noticeable performance impact? Yes. > Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact. [0] https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html

krzyk 6y ago

CONFIG_IRQ_TIME_ACCOUNTING according to the docs: CONFIG_IRQ_TIME_ACCOUNTING: Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small *performance impact*. If in doubt, say N here. So I didn't enable this option in my kernels afraid of loosing performance on my laptop.

tanelpoder 6y ago
(Author here). It would be interesting to test the performance difference. I downloaded the source of one of my distro-provided kernels & recompiled it with different config setting (to compare perf with just one thing changed), but got into some trouble with my drivers and ran out of time. I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit. In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue. edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.

happyflower 6y ago

On a typical laptop, what level of interrupts is too high? And what can be done about it?

joosters 6y ago
The tool 'powertop' is very useful here, it will show what is causing any form of wakeup on your system, and offers a 'Tunables' tab with suggested settings to improve some aspects.
- LinuxBender 6y ago
  Powertop is great. You can even run it in non-interactive mode. That said, it only optimizes for lower power usage and lower interrupts. You can further reduce interrupts by disabling devices. If there are devices you don't need, you can blacklist them in /etc/modprobe.d/ provided you understand the potential dependencies other devices may have on the drivers. This can free up a tiny bit of memory as well.

irqquestion 6y ago

Can someone explain this in further detail. What is causing interrupt CPU usage on my machine. When I run powertop I see this: 629 mW 2,6 ms/s 158,2 Interrupt [0] HI_SOFTIRQ Is this related or another story and what can I do about it?

hedora 6y ago
Try disabling hardware devices until it goes away.

archi42 6y ago

Ahh, yes, that's actually a great point to be aware of, thanks @tanelpoder. Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.

andy_threos_io 6y ago

Irq accounting depends on architecture and code usage, e.g. like some kind of barrier (serializing) before the RDTSC. on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.). The RDTSC and fence timings (Latency, Reciprocal throughput) [0]: AMD Zen 1 SFENCE 4 ~20 LFENCE 1 0.25 MFENCE 7 ~70 RDTSC 37 36 AMD Zen 2 SFENCE 1 1 LFENCE 1 11 MFENCE 7 ~78 RDTSC 37 37 AMD Zen 3 SFENCE 1 1 LFENCE 1 10 MFENCE 6 ~60 RDTSC 44 36 Intel Haswell LFENCE 2 4 MFENCE 3 33 SFENCE 2 5 RDTSC 15 24 Intel Broadwell LFENCE 2 4 MFENCE 3 33 SFENCE 2 6 RDTSC 15 24 Intel Skylake LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 Intel Skylake-X LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 Intel Coffee Lake LFENCE 2 4 MFENCE 4 33 SFENCE 2 6 RDTSC 20 25 [0] https://www.agner.org/optimize/instruction_tables.pdf https://www.agner.org/optimize/instruction_tables.pdf

tanelpoder 6y ago
I guess the fence instructions are there for avoiding the CPU out-of-order execution getting the timestamp too early (or late) compared to the interrupt service routine's start & end?

3np 6y ago

Thank you, this is something I somewhat recently realized I don't know how to measure and troubleshoot properly - I've never been able to give a confident answer to if context switching/interrupts are a bottleneck or not, just "the number looks highish and I don't have anything else to blame".