5 ms·

> Why does gnu parallel use all the cores when we tell it to use only 8? We only saw eight git clone processes at a time, but a large number of git index-pack p

by bhaney 2y ago

> Why does gnu parallel use all the cores when we tell it to use only 8? We only saw eight git clone processes at a time, but a large number of git index-pack processes that maxed out all 32 cores on my laptop. I’m guessing git’s index-pack is a forked subprocess and allows parallel to start another git clone? Parallel is running 8 git clone jobs at once (as asked for) and each git clone is starting as many index-pack threads as it wants. Temporarily setting pack.threads to 1 (via git config) would help here.

chrismorgan 2y ago
This is an increasingly common problem: two layers, both trying to utilise the entire machine by parallelising by the number of CPUs/cores, and so the inner layer gets N² threads/processes. Being quadratic growth, this means that the problem becomes worse the more CPUs you have. With 32 cores, 32² = 1024, though in practice Parallel was told 8 so it’d cap out at only 256 index-pack processes. But that’s a lot of memory needed to support that, especially given that it’s doing no good. The solution is for only one of the two layers to parallelise. —⁂— Given that you talked about pack.threads, here’s its description from `man git-config`: > Specifies the number of threads to spawn when searching for best delta matches. This requires that git-pack-objects(1) be compiled with pthreads otherwise this option is ignored with a warning. This is meant to reduce packing time on multiprocessor machines. The required amount of memory for the delta search window is however multiplied by the number of threads. Specifying 0 will cause Git to auto-detect the number of CPUs and set the number of threads accordingly.
- PhilipRoman 2y ago
  >The solution is for only one of the two layers to parallelise If you have a common scheduling API, you can manage this much more elegantly. For example make(1) can control the concurrency level across recursive invocations by using a "job server" https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... With this, you can have, for example, 3 top level subprocesses, each spawning multiple threads of their own, but never exceeding the CPU count. Alternatively, parallel could make it's subprocesses think that they're running on a 1-core machine, although this may have some subtle side affects.
  - slimsag 2y ago
    Reminds me of a Zig proposal I saw recently to make the std.Thread.Pool API respect the Linux jobserver and macOS dispatcher automatically out of the box: https://github.com/ziglang/zig/issues/20274 https://github.com/ziglang/zig/issues/20274
    - PhilipRoman 2y ago
      Those are some really interesting proposals. I agree that a lot of code ignores resource cleanup, especially when it comes to driver-userspace communication. For example, driver authors just implement a netlink/ioctl system which manages persistent state in kernel space, even though they could bind the state to a device file descriptor, which automatically gets cleaned up upon process exit (but still can be handed to another process, if really needed)
- rwmj 2y ago
  The solution is a global scheduler (the one thing that Kubernetes does well). It would be aware of what every program needs, not just in terms of CPU, but also memory, devices, time etc. and split the available resources fairly.
notpushkin 2y ago
> Temporarily setting pack.threads to 1 (via git config) Or just use `git -c pack.threads=1 clone`: https://git-scm.com/docs/git#Documentation/git.txt--cltnamegtltvaluegt https://git-scm.com/docs/git#Documentation/git.txt--cltnameg...
shae 2y ago
Thank you, didn't know this. I'll add it to the article.

by bhaney 2y ago

chrismorgan 2y ago

This is an increasingly common problem: two layers, both trying to utilise the entire machine by parallelising by the number of CPUs/cores, and so the inner layer gets N² threads/processes. Being quadratic growth, this means that the problem becomes worse the more CPUs you have. With 32 cores, 32² = 1024, though in practice Parallel was told 8 so it’d cap out at only 256 index-pack processes. But that’s a lot of memory needed to support that, especially given that it’s doing no good. The solution is for only one of the two layers to parallelise. —⁂— Given that you talked about pack.threads, here’s its description from `man git-config`: > Specifies the number of threads to spawn when searching for best delta matches. This requires that git-pack-objects(1) be compiled with pthreads otherwise this option is ignored with a warning. This is meant to reduce packing time on multiprocessor machines. The required amount of memory for the delta search window is however multiplied by the number of threads. Specifying 0 will cause Git to auto-detect the number of CPUs and set the number of threads accordingly.

PhilipRoman 2y ago
>The solution is for only one of the two layers to parallelise If you have a common scheduling API, you can manage this much more elegantly. For example make(1) can control the concurrency level across recursive invocations by using a "job server" https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... With this, you can have, for example, 3 top level subprocesses, each spawning multiple threads of their own, but never exceeding the CPU count. Alternatively, parallel could make it's subprocesses think that they're running on a 1-core machine, although this may have some subtle side affects.
- slimsag 2y ago
  Reminds me of a Zig proposal I saw recently to make the std.Thread.Pool API respect the Linux jobserver and macOS dispatcher automatically out of the box: https://github.com/ziglang/zig/issues/20274 https://github.com/ziglang/zig/issues/20274
  - PhilipRoman 2y ago
    Those are some really interesting proposals. I agree that a lot of code ignores resource cleanup, especially when it comes to driver-userspace communication. For example, driver authors just implement a netlink/ioctl system which manages persistent state in kernel space, even though they could bind the state to a device file descriptor, which automatically gets cleaned up upon process exit (but still can be handed to another process, if really needed)
rwmj 2y ago
The solution is a global scheduler (the one thing that Kubernetes does well). It would be aware of what every program needs, not just in terms of CPU, but also memory, devices, time etc. and split the available resources fairly.

notpushkin 2y ago

> Temporarily setting pack.threads to 1 (via git config) Or just use `git -c pack.threads=1 clone`: https://git-scm.com/docs/git#Documentation/git.txt--cltnamegtltvaluegt https://git-scm.com/docs/git#Documentation/git.txt--cltnameg...

shae 2y ago

Thank you, didn't know this. I'll add it to the article.