5 ms·

Linux file scanners are the worst for this, or were until recently. I have no idea why it was so hard not to occasionally consume 100% CPU and tons of disk IO

Linux file scanners are the worst for this, or were until recently. I have no idea why it was so hard not to occasionally consume 100% CPU and tons of disk IO at inconvenient times, but multiple different indexers have exactly the same issue. I bet you could static analyze to find pure functions, then add a caching layer automatically. A lot of slowness seems to be because a cache wasn't used. Or because something stateless should have been stateful(As in fully recomputing stuff that depends on previous inputs rather than calculating one item at a time).

smoldesu 5y ago
MacOS' indexer is equally as bad. The amount of time's I've seen 10 mdworker threads pinning my CPU is staggering...
- als0 5y ago
  AFAIK this problem seems to have gone away with the M1 based Macs. Can anyone confirm?
  - smoldesu 5y ago
    I'd imagine it's not an issue since Grand Central Dispatch will only assign it to a single core cluster, which leaves the other cores free to do their own processing.
  - ceeplusplus 5y ago
    I believe indexing tasks only run on e-cores, at least from quick glances at Activity Monitor. I also see more kernel code run on e-cores when I'm compiling code, not sure if Apple is moving blocking syscalls onto e-cores and swapping threads or something (intuition tells me this shouldn't lead to perf gains due to context switch cost, but who knows).
    - MBCook 5y ago
      I’ve been led to believe that all non-interactive tasks only run on the efficiency cores by default. Of course code can request alternate priorities. It’s entirely possible that Apple has written things like the spotlight indexer to explicitly make sure they run that way as well, or it may just be an intentional result of the policy.
- astrange 5y ago
  Spotlight runs at a low CPU priority, so it doesn’t use a lot of power (theoretically) and on Intel won’t increase P-states. The CPU % doesn’t mean much at all except it’s runnable.
  - smoldesu 5y ago
    In practice my CPU hits ~80c and makes the system nearly unusable. I just disabled Spotlight indexing altogether.
    - sliken 5y ago
      To be fair backups are one of many things (like compiling, video conferencing, or even just looking at the laptop wrong) that makes x86-64 macs run their fans run out. My particular case is an i9 in a mbp 16". It throttles heavily while making impressive noise any just about any light workload that's not pure idle.
sliken 5y ago
The major problems I see are one of two cases. Often just naive implementations run a filescan, run as fast as they can, the OS starts using more ram for caching, then at some point applications need more ram. That triggers a reclaiming pages from cache, writing dirty pages, reading pages from the disk for the application, all while the scan is keeping the I/O pipeline full. The user observes this as a slow/laggy machine, and if you are unlucky enough to have a i7/i9 in a laptop often a fan running flat out. A second case is where a vendor offers "unlimited" backups for a flat price. This creates an incentive for the vendor to make an INCREDIBLY inefficient backup client that consumes substantial ram and CPU resources locally and minimal storage remotely. Typically this means that you aren't allowed to use more efficient clients, and have to use the vendors client. Here's looking at you crashplan.
- Tagbert 5y ago
  And that is why many of us moved from Crashplan to Backblaze and never looked back.
invalidator 5y ago
I set this: echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load My backups and indexing are already running nice, so now they loaf along at 1.6GHz. The CPUs only spin up to full speed when I run non-nice work. There's room to tune more. In the old days I would start large jobs (large builds, etc) with nice so they wouldn't interfere with interactive use. Now they would run slower instead of fast-but-yielding. I can always flip ignore_nice_load off if I want the old behavior for competing fast jobs. It's an easy 80% solution, especially because most indexers and backups come preconfigured to run nice.

smoldesu 5y ago

MacOS' indexer is equally as bad. The amount of time's I've seen 10 mdworker threads pinning my CPU is staggering...

als0 5y ago
AFAIK this problem seems to have gone away with the M1 based Macs. Can anyone confirm?
- smoldesu 5y ago
  I'd imagine it's not an issue since Grand Central Dispatch will only assign it to a single core cluster, which leaves the other cores free to do their own processing.
- ceeplusplus 5y ago
  I believe indexing tasks only run on e-cores, at least from quick glances at Activity Monitor. I also see more kernel code run on e-cores when I'm compiling code, not sure if Apple is moving blocking syscalls onto e-cores and swapping threads or something (intuition tells me this shouldn't lead to perf gains due to context switch cost, but who knows).
  - MBCook 5y ago
    I’ve been led to believe that all non-interactive tasks only run on the efficiency cores by default. Of course code can request alternate priorities. It’s entirely possible that Apple has written things like the spotlight indexer to explicitly make sure they run that way as well, or it may just be an intentional result of the policy.
astrange 5y ago
Spotlight runs at a low CPU priority, so it doesn’t use a lot of power (theoretically) and on Intel won’t increase P-states. The CPU % doesn’t mean much at all except it’s runnable.
- smoldesu 5y ago
  In practice my CPU hits ~80c and makes the system nearly unusable. I just disabled Spotlight indexing altogether.
  - sliken 5y ago
    To be fair backups are one of many things (like compiling, video conferencing, or even just looking at the laptop wrong) that makes x86-64 macs run their fans run out. My particular case is an i9 in a mbp 16". It throttles heavily while making impressive noise any just about any light workload that's not pure idle.

sliken 5y ago

The major problems I see are one of two cases. Often just naive implementations run a filescan, run as fast as they can, the OS starts using more ram for caching, then at some point applications need more ram. That triggers a reclaiming pages from cache, writing dirty pages, reading pages from the disk for the application, all while the scan is keeping the I/O pipeline full. The user observes this as a slow/laggy machine, and if you are unlucky enough to have a i7/i9 in a laptop often a fan running flat out. A second case is where a vendor offers "unlimited" backups for a flat price. This creates an incentive for the vendor to make an INCREDIBLY inefficient backup client that consumes substantial ram and CPU resources locally and minimal storage remotely. Typically this means that you aren't allowed to use more efficient clients, and have to use the vendors client. Here's looking at you crashplan.

Tagbert 5y ago
And that is why many of us moved from Crashplan to Backblaze and never looked back.

invalidator 5y ago

I set this: echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load My backups and indexing are already running nice, so now they loaf along at 1.6GHz. The CPUs only spin up to full speed when I run non-nice work. There's room to tune more. In the old days I would start large jobs (large builds, etc) with nice so they wouldn't interfere with interactive use. Now they would run slower instead of fast-but-yielding. I can always flip ignore_nice_load off if I want the old behavior for competing fast jobs. It's an easy 80% solution, especially because most indexers and backups come preconfigured to run nice.