6 ms·

Why was there a limitation on Windows? I can't find any such limit for Linux.

lofties 1y ago
Windows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.
- okanat 1y ago
  Every operating system, that's relevant in 2025, needs that concept. It is called NUMA. At some point you cannot model the system ignoring the memory affinity / closeness to the cores. Modern AMD CPUs are literally consist of core groups on chiplets. It is better for an OS to make decisions / expose APIs for cores that are physically so far away from each other that moving data back-and-forth over the RAM, system bus or interconnect has significant time penalties.
lmm 1y ago
Seems like this is a general Windows thing per https://learn.microsoft.com/en-us/windows/win32/procthread/processor-groups https://learn.microsoft.com/en-us/windows/win32/procthread/p... - applications that want to run on more than 64 CPUs need to be written with dedicated support for doing so.
deleted 1y ago
[deleted]
dwattttt 1y ago
The linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how many logical processors are on the system if you pass ALL_PROCESSOR_GROUPS, but that API was only added in Windows 7 (that said, that was more than 15 years ago, they probably could've found a moment to add and test a conditional call to it).
- dspillett 1y ago
  It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably could've found a moment to add and test a conditional call to it I suspect it hasn't been an issue much at all until recently. Any single block of data worth spinning up that many threads for compressing is going to be very large, you don't want to split something into too small chunks for compression or you lose some benefit of the dynamic compression dictionary (sharing that between threads would add a lot of inter-thread coordination work, killing any performance gain even if the threads are running local enough on the CPU to share cache). Compression is not an inherently parallelizable task, at least not “embarrassingly” so like some processes. Even when you do have something to compress that would benefit for more than 64 separate tasks in theory, unless it is all in RAM (or on an incredibly quick & low latency drive/array) the process is likely to be IO starved long before it is compute starved, when you have that much compute resource to hand. Recent improvements in storage options & CPUs (and the bandwidth between them) have presumably pushed the occurrences of this being worthwhile (outside of artificial tests) from “practically zero” to “near zero, but it happens”, hence the change has been made. Note that two or more 7-zip instances working on different data could always use more than 64 threads between them, if enough cores to make that useful were available.
  - dwattttt 1y ago
    Are you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into. The referenced text suggests applications will "work", but that isn't really explicit.
    - Dylan16807 1y ago
silon42 1y ago
Maybe WaitForMultipleObjects limit of 64 (MAXIMUM_WAIT_OBJECTS) applies? An ugly limitation on an API that initially looks superior to Linux equivalents.
monocasa 1y ago
A lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions. Because of that, transitioning a software thread to another processor group is a manual process that has to be managed by user space.
- zik 1y ago
  Wow. That's surprisingly lame.
  - Const-me 1y ago
    The NT kernel dates back to 1993. Computers didn’t exceed 64 logical processors per system until around 2014. And doing it back then required a ridiculously expensive server with 8 Intel CPUs. The technical decision Microsoft made initially worked well for over two decades. I don’t think it was lame; I believe it was a solid choice back then.
    - monocasa 1y ago
      I mean, x86 didn't, but other systems had been exceeding 64 cores since the late 90s. And x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it.
      - Const-me 1y ago
        > other systems had been exceeding 64 cores since the late 90s. Windows didn’t run on these other systems, why would Microsoft care about them? > x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it For publicly accessible web servers, Linux overtook Windows around 2005. Then in 2006 Amazon launched EC2, and the industry started that massive transition to the clouds. Linux is better suited for clouds, due to OS licensing and other reasons.
whalesalad 1y ago
Windows is a terrible operating system.
xxs 1y ago
WaitForMultipleObjects is limited to 64... since forever.

Windows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.

okanat 1y ago
Every operating system, that's relevant in 2025, needs that concept. It is called NUMA. At some point you cannot model the system ignoring the memory affinity / closeness to the cores. Modern AMD CPUs are literally consist of core groups on chiplets. It is better for an OS to make decisions / expose APIs for cores that are physically so far away from each other that moving data back-and-forth over the RAM, system bus or interconnect has significant time penalties.

lmm 1y ago

Seems like this is a general Windows thing per https://learn.microsoft.com/en-us/windows/win32/procthread/processor-groups https://learn.microsoft.com/en-us/windows/win32/procthread/p... - applications that want to run on more than 64 CPUs need to be written with dedicated support for doing so.

deleted 1y ago

[deleted]

dwattttt 1y ago

The linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how many logical processors are on the system if you pass ALL_PROCESSOR_GROUPS, but that API was only added in Windows 7 (that said, that was more than 15 years ago, they probably could've found a moment to add and test a conditional call to it).

dspillett 1y ago
It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably could've found a moment to add and test a conditional call to it I suspect it hasn't been an issue much at all until recently. Any single block of data worth spinning up that many threads for compressing is going to be very large, you don't want to split something into too small chunks for compression or you lose some benefit of the dynamic compression dictionary (sharing that between threads would add a lot of inter-thread coordination work, killing any performance gain even if the threads are running local enough on the CPU to share cache). Compression is not an inherently parallelizable task, at least not “embarrassingly” so like some processes. Even when you do have something to compress that would benefit for more than 64 separate tasks in theory, unless it is all in RAM (or on an incredibly quick & low latency drive/array) the process is likely to be IO starved long before it is compute starved, when you have that much compute resource to hand. Recent improvements in storage options & CPUs (and the bandwidth between them) have presumably pushed the occurrences of this being worthwhile (outside of artificial tests) from “practically zero” to “near zero, but it happens”, hence the change has been made. Note that two or more 7-zip instances working on different data could always use more than 64 threads between them, if enough cores to make that useful were available.
- dwattttt 1y ago
  Are you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into. The referenced text suggests applications will "work", but that isn't really explicit.
  - Dylan16807 1y ago

silon42 1y ago

Maybe WaitForMultipleObjects limit of 64 (MAXIMUM_WAIT_OBJECTS) applies? An ugly limitation on an API that initially looks superior to Linux equivalents.

monocasa 1y ago

A lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions. Because of that, transitioning a software thread to another processor group is a manual process that has to be managed by user space.

zik 1y ago
Wow. That's surprisingly lame.
- Const-me 1y ago
  The NT kernel dates back to 1993. Computers didn’t exceed 64 logical processors per system until around 2014. And doing it back then required a ridiculously expensive server with 8 Intel CPUs. The technical decision Microsoft made initially worked well for over two decades. I don’t think it was lame; I believe it was a solid choice back then.
  - monocasa 1y ago
    I mean, x86 didn't, but other systems had been exceeding 64 cores since the late 90s. And x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it.
    - Const-me 1y ago
      > other systems had been exceeding 64 cores since the late 90s. Windows didn’t run on these other systems, why would Microsoft care about them? > x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it For publicly accessible web servers, Linux overtook Windows around 2005. Then in 2006 Amazon launched EC2, and the industry started that massive transition to the clouds. Linux is better suited for clouds, due to OS licensing and other reasons.

whalesalad 1y ago

Windows is a terrible operating system.

xxs 1y ago

WaitForMultipleObjects is limited to 64... since forever.