3 ms·

There is no mention in the article whether the software suite was vetted for support of more than 64 threads on Win32. The API has a peculiar weakness that limi

There is no mention in the article whether the software suite was vetted for support of more than 64 threads on Win32. The API has a peculiar weakness that limits thread scheduling to a single processor group by default and a group can have no more than 64 hardware threads. To get above this limit, the application must explicitly adjust the processor affinity of its threads to include the additional hardware threads. MS was not in a hurry to adjust the C++ STL and their OpenMP runtime after the basic processor group API appeared in Vista. I am not sure if they managed to do it by now. Some of the benchmark results look to me like the missing scaling from 64 to 128 hardware threads on Windows might be caused by this.

tpxl 7y ago
There was another phoronix article recently that mentioned this, so they must know this exists. Either way, if it doesn't work in the benchmark, it doesn't really matter who's at fault (as long as the benchmark isn't completely synthetic)?
- muststopmyths 7y ago
  How is it a "benchmark" if it doesn't fully utilize the operating system's APIs to maximize performance ? I am not defending the utter shittiness of the Win32 Processor group API design, which is the usual overly complicated Win32 API that only makes sense to an NT kernel developer. I can see why no application developer bothered to incorporate that in their software. It's basically asking you to do thread scheduling yourself for >64 cores. But I'm always annoyed to see people whinging about Windows performance when they're using "cross-platform" applications (that are using the lowest common denominator APIs) to supposedly measure perf. For an application that knows the OS environment it's running in, the "scheduler" or the "kernel" is very unlikely to be an actual impediment most of the time. I use quotes because those words are usually used to mean "my app doesn't run as I expect and it can't be my fault". This is true of all operating systems. An old MSDN doc [1] actually says "The reason for initially limiting all threads to a single group is that 64 processors is more than adequate for the typical application.". There is no newer doc that I could find, so perhaps the design hasn't changed in 8 or 10. "XYZ should be more than adequate for the typical application" is like a Microsoft basic design principle or something :-) [1]https://docs.microsoft.com/en-us/previous-versions/windows/hardware/design/dn653313(v=vs.85)?redirectedfrom=MSDN https://docs.microsoft.com/en-us/previous-versions/windows/h...
  - firethief 7y ago
    > How is it a "benchmark" if it doesn't fully utilize the operating system's APIs to maximize performance ? Why should it do that? Isn't it better to reflect how applications will realistically use the APIs?
  - rowanG077 7y ago
    It depends on what you want to measure. You want to measure the maximum pure power the OS can possibly give you? Then sure it's not a fair benchmark. If you want to measure "real world" performance it is a fair benchmark.
monocasa 7y ago
It's not just the API, it's the scheduler in NT itself that won't move threads from one process group of up to 64 hardware threads (on a 64bit system) to another and so it has to be manually managed by the application if you want to scale out farther than that on NT. Given that it's a fundamental limitation of the NT scheduler (not present in Linux), it seems like it'd be on the table for "yeah, windows makes this way harder, and a lot of applications won't scale the same way on windows as their Linux versions will", rather than "oh, that just doesn't count because they aren't using it right". EDIT: As an aside, this kind of thing is exactly why Linux doesn't provide binary compatibility on the driver level. It's easy to paint yourself into a corner by making decisions that were perfectly sane 20 years ago. Now NT has fundamental limitations, hitting even harder in kernel space where nearly every driver out there has some macros compiled in that touched these structures. It's bad enough at the syscall layer, but it's even worse when you can't change things with code that's directly modifying internal structures. This is exactly why Linux won't provide a driver ABI, and why it's a good thing.
- zamadatix 7y ago
  I don't think it's a fundamental binary API limitation type thing as the issue does not exist in Windows Enterprise or Server. This was covered the last time this was posted https://www.anandtech.com/show/15483/amd-threadripper-3990x-review/3 https://www.anandtech.com/show/15483/amd-threadripper-3990x-...
  - monocasa 7y ago
    > the issue does not exist in Windows Enterprise or Server. It does. There's maximum of 64 threads in a processor group. This is because the affinity mask in tons of internal data structures inside NT are pointer width (so 64 bits on a 64bit platform). Server and Enterprise's scheduler adjustments are just around making better decisions balancing which processor group a new process is assigned to at creation time. You can read more about processor groups and the manual work by user space needed to manage them on all flavors of windows that support them here: https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups https://docs.microsoft.com/en-us/windows/win32/procthread/pr...
908B64B197 7y ago
The 64 processor per processor group limitation makes sense to me. A lot of >64 processor configs on the market are NUMA systems where it's probably best to pin threads to the NUMA node where their data will be hosted. They also used a version of Windows aimed at desktop/graphical uses and not Server. I wouldn't be surprised if Desktop Windows didn't like having apps use a lot of CPU (making the system feel unresponsive) but server didn't mind.

tpxl 7y ago

There was another phoronix article recently that mentioned this, so they must know this exists. Either way, if it doesn't work in the benchmark, it doesn't really matter who's at fault (as long as the benchmark isn't completely synthetic)?

muststopmyths 7y ago
How is it a "benchmark" if it doesn't fully utilize the operating system's APIs to maximize performance ? I am not defending the utter shittiness of the Win32 Processor group API design, which is the usual overly complicated Win32 API that only makes sense to an NT kernel developer. I can see why no application developer bothered to incorporate that in their software. It's basically asking you to do thread scheduling yourself for >64 cores. But I'm always annoyed to see people whinging about Windows performance when they're using "cross-platform" applications (that are using the lowest common denominator APIs) to supposedly measure perf. For an application that knows the OS environment it's running in, the "scheduler" or the "kernel" is very unlikely to be an actual impediment most of the time. I use quotes because those words are usually used to mean "my app doesn't run as I expect and it can't be my fault". This is true of all operating systems. An old MSDN doc [1] actually says "The reason for initially limiting all threads to a single group is that 64 processors is more than adequate for the typical application.". There is no newer doc that I could find, so perhaps the design hasn't changed in 8 or 10. "XYZ should be more than adequate for the typical application" is like a Microsoft basic design principle or something :-) [1]https://docs.microsoft.com/en-us/previous-versions/windows/hardware/design/dn653313(v=vs.85)?redirectedfrom=MSDN https://docs.microsoft.com/en-us/previous-versions/windows/h...
- firethief 7y ago
  > How is it a "benchmark" if it doesn't fully utilize the operating system's APIs to maximize performance ? Why should it do that? Isn't it better to reflect how applications will realistically use the APIs?
- rowanG077 7y ago
  It depends on what you want to measure. You want to measure the maximum pure power the OS can possibly give you? Then sure it's not a fair benchmark. If you want to measure "real world" performance it is a fair benchmark.

monocasa 7y ago

It's not just the API, it's the scheduler in NT itself that won't move threads from one process group of up to 64 hardware threads (on a 64bit system) to another and so it has to be manually managed by the application if you want to scale out farther than that on NT. Given that it's a fundamental limitation of the NT scheduler (not present in Linux), it seems like it'd be on the table for "yeah, windows makes this way harder, and a lot of applications won't scale the same way on windows as their Linux versions will", rather than "oh, that just doesn't count because they aren't using it right". EDIT: As an aside, this kind of thing is exactly why Linux doesn't provide binary compatibility on the driver level. It's easy to paint yourself into a corner by making decisions that were perfectly sane 20 years ago. Now NT has fundamental limitations, hitting even harder in kernel space where nearly every driver out there has some macros compiled in that touched these structures. It's bad enough at the syscall layer, but it's even worse when you can't change things with code that's directly modifying internal structures. This is exactly why Linux won't provide a driver ABI, and why it's a good thing.

zamadatix 7y ago
I don't think it's a fundamental binary API limitation type thing as the issue does not exist in Windows Enterprise or Server. This was covered the last time this was posted https://www.anandtech.com/show/15483/amd-threadripper-3990x-review/3 https://www.anandtech.com/show/15483/amd-threadripper-3990x-...
- monocasa 7y ago
  > the issue does not exist in Windows Enterprise or Server. It does. There's maximum of 64 threads in a processor group. This is because the affinity mask in tons of internal data structures inside NT are pointer width (so 64 bits on a 64bit platform). Server and Enterprise's scheduler adjustments are just around making better decisions balancing which processor group a new process is assigned to at creation time. You can read more about processor groups and the manual work by user space needed to manage them on all flavors of windows that support them here: https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups https://docs.microsoft.com/en-us/windows/win32/procthread/pr...

908B64B197 7y ago

The 64 processor per processor group limitation makes sense to me. A lot of >64 processor configs on the market are NUMA systems where it's probably best to pin threads to the NUMA node where their data will be hosted. They also used a version of Windows aimed at desktop/graphical uses and not Server. I wouldn't be surprised if Desktop Windows didn't like having apps use a lot of CPU (making the system feel unresponsive) but server didn't mind.