5 ms·

> Could we do better? Assuredly. There are many AVX-512 that we are not using yet. We do not use ternary Boolean operations (vpternlog). We are not using the ne

> Could we do better? Assuredly. There are many AVX-512 that we are not using yet. We do not use ternary Boolean operations (vpternlog). We are not using the new powerful shuffle functions (e.g., vpermt2b). We have an example of coevolution: better hardware requires new software which, in turn, makes the hardware shine. > Of course, to get these new benefits, you need recent Intel processors with adequate AVX-512 support AVX-512 support can be confusing because it’s often referred to as a single instruction set. AVX-512 is actually a large family of instructions that have different availability depending on the CPU. It’s not enough to say that a CPU has AVX-512 because it’s not a binary question. You have to know which AVX-512 instructions are supported on a particular CPU. Wikipedia has a partial chart of AVX-512 support by CPU: https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 Note that some instructions that are available in one generation of CPU are can actually be unavailable (superseded, usually) in the next generation of the CPU. If you go deep enough into AVX-512 optimization, you essentially end up targeting a specific CPU for the code. This is not a big deal if you’re deploying software to 10,000 carefully controlled cloud servers with known specifications, but it makes general use and especially consumer use much harder.

mikepurvis 4y ago
Are there good libraries for doing runtime feature detection? Eg, include three versions of hot function X in the binary, and have it seamlessly insert the correct function pointer at startup? Or have the function contain multiple bodies and just JMP to the correct block of code? I know you can do this yourself, but last time I looked it was a heavily manual process— you had to basically define a plugin interface and dynamically load your selected implementation from a separate shared object. What are the barriers to having compilers able to be hinted into transparently generating multiple versions of key functions?
- colejohnson66 4y ago
  Check out Agner Fog's vectorclass library: https://github.com/vectorclass/version2 https://github.com/vectorclass/version2
- bremac 4y ago
  I'm unsure about library support, but gcc and clang support function multi-versioning (FMV), which resolves the function based on CPUID the first time the function is called. This LWN article has some additional information: https://lwn.net/Articles/691932/ https://lwn.net/Articles/691932/
  - mikepurvis 4y ago
    TIL! I guess it makes sense that popular numeric libraries like BLAS, Eigen, and so-on would take advantage of this, but I wonder how widely used it is overall.
    - bee_rider 4y ago
      For BLAS, it has enough legacy that the vendors just make their own libraries, so it would probably be best to use Intel MKL (which has it's own internal dispatching, but I don't think it uses FMV) if you are targeting Intel, and maybe AMD's branch of BLIS if you are targeting AMD. I believe Eigen comes with it's own version of BLAS built in, but it can be made to link to MKL at least, probably BLIS if you are really stubborn.
- loeg 4y ago
  GCC has offered Function Multiversioning for about a decade now (GCC ~4.8 or 4.9). GCC 6's resolver apparently uses CPUID to resolve the ifunc once at program start: https://lwn.net/Articles/691932/ https://lwn.net/Articles/691932/ . Clang added it in 7.0.0: https://releases.llvm.org/7.0.0/tools/clang/docs/AttributeReference.html#target-gnu-target https://releases.llvm.org/7.0.0/tools/clang/docs/AttributeRe... A nice presentation on it: https://llvm.org/devmtg/2014-10/Slides/Christopher-Function%20Multiversioning%20Talk.pdf https://llvm.org/devmtg/2014-10/Slides/Christopher-Function%...
  - indygreg2 4y ago
    While this IFUNC feature does exist and it is useful, when I performed binary analysis on every package in Ubuntu in January, I found that only ~11 distinct packages have IFUNCs. It certainly looks like this ELF feature is not really used much [in open source] outside of GNU toolchain-level software! https://gregoryszorc.com/blog/2022/01/09/bulk-analyze-linux-packages-with-linux-package-analyzer/ https://gregoryszorc.com/blog/2022/01/09/bulk-analyze-linux-...
    - TkTech 4y ago
      I've wanted to use them many times in the past, but the limited support on other compilers (looking at you MSVC) always made it a non-starter. If I have to support some other method of feature detection anyways, there's no point.
    - jeffbee 4y ago
      The use of ifunc in libc, openssl, and zlib covers just about everything people want from it. You want that optimized memcpy, you only need it in one place.
    - arthur2e5 4y ago
      The way ifunc (well, actually language-level FMV) works in GCC and clang is that the input source code, not the command-line switches, specifies what ISA extensions to build for on a candidate function. This naturally means that packagers and other vendors are not using it as much as you hope: they would need to have a separate patch to add these attributes for each architecture and that’s just not maintainable. Even Intel’s Clear Linux does not bother to patch individual libraries. They just use the glibc library multi-versioning feature to load from different directories depending on the cpuid. In my opinion, GCC and Clang could make the whole thing more ergonomic. Ideally you declare a function as "interesting for multi-versioning" in the source using an attribute, then in the command line define what -march's to actually clone for. Kinda like ICC’s /Qax. (On second thought preprocessor defs are sufficient, duh.)
- Scene_Cast2 4y ago
  For vectorizable workloads, there's Halide. It can be set up in multiple ways, one of which is a JIT that auto-detects CPU capabilities and cache sizes (to better optimize scheduling).
robocat 4y ago
To add, they are using[2] the relatively recent VBMI2 instructions of AVX512. This article[1] talks about the advantages of VBMI on IceLake released 2021. [1] https://www.singlestore.com/blog/a-programmers-perspective/ https://www.singlestore.com/blog/a-programmers-perspective/ comments https://news.ycombinator.com/item?id=28179111 https://news.ycombinator.com/item?id=28179111 [2] https://news.ycombinator.com/item?id=31522464 https://news.ycombinator.com/item?id=31522464
paulmd 4y ago
> AVX-512 support can be confusing because it’s often referred to as a single instruction set. This has been broadly overplayed. There are two main families: consumer AVX-512 and server AVX-512. Server gets some additional VNNI instructions and BFLOAT16 support, consumer gets VPOPCNTDQ and IFMA/VBMI. That's basically all you need to know The chart looks confusing in Wikipedia because it's ordered by date, not by "series" and generation, and some intrepid wikipedia editors have munged up the features even then. This is what the feature set looks like to me: https://i.imgur.com/idAjB1X.png https://i.imgur.com/idAjB1X.png Xeon Phi is sorta its own thing for sure, they did a weird 4-wide architecture and it got special 4-wide VNNI instructions to match. And since it's been abandoned it never was updated. But there's zero reason for average consumer/server software to target Xeon Phi anyway these days - it's abandoned and it was almost entirely a supercomputing/HPC product even when it wasn't. The other instructions are (almost) a straight superset within their series, so you just need to know if you're targeting server or consumer. Laptop gets things a bit earlier because Intel started 10nm first on laptops, and 14nm was stuck with older architectures and then a backported architecture, so it's not quite a strict series in terms of dates, but generation-on-generation they haven't ever regressed a feature within a given "series". The only exception is Alder Lake doesn't get anything because lol Intel - they disabled AVX-512 entirely, but if you got a model that supports it then it's also a superset of all previous consumer generations. Again, all you really need to remember is, consumer gets better FMA and server gets VNNI/BFLOAT, that's the major difference. Big deal, who cares. But "it's just strict supersets of features within server/client" wouldn't get the clicks/exasperated outcry of "gosh isn't this just an intractible mess!". It's not, and they've never regressed a feature within a series. Also for the record, AVX2/AVX1/SSE/MMX look really shitty if you split them up in this same fashion. There were like 4 different "sets" of SSE4 feature sets alone and some of them were abandoned and never used again, and both AMD and Intel adopted them all at different dates and sometimes dropped them back out, so if you made a "feature set vs architecture" table it would be a giant fucking mess for SSE as well, let alone if you did it by year instead of by architectural series. Yet I never hear anyone bring up how awful and how confusing AVX and SSE are, just AVX-512. Feature sets are messy! https://en.wikipedia.org/wiki/SSE4 https://en.wikipedia.org/wiki/SSE4 Eventually, AVX-512 support will converge around the sets of features that are useful, and some of the less-useful features may eventually be deprecated, just like the SSE4a instruction set did. It's gonna be fine.

mikepurvis 4y ago

Are there good libraries for doing runtime feature detection? Eg, include three versions of hot function X in the binary, and have it seamlessly insert the correct function pointer at startup? Or have the function contain multiple bodies and just JMP to the correct block of code? I know you can do this yourself, but last time I looked it was a heavily manual process— you had to basically define a plugin interface and dynamically load your selected implementation from a separate shared object. What are the barriers to having compilers able to be hinted into transparently generating multiple versions of key functions?

colejohnson66 4y ago
Check out Agner Fog's vectorclass library: https://github.com/vectorclass/version2 https://github.com/vectorclass/version2
bremac 4y ago
I'm unsure about library support, but gcc and clang support function multi-versioning (FMV), which resolves the function based on CPUID the first time the function is called. This LWN article has some additional information: https://lwn.net/Articles/691932/ https://lwn.net/Articles/691932/
- mikepurvis 4y ago
  TIL! I guess it makes sense that popular numeric libraries like BLAS, Eigen, and so-on would take advantage of this, but I wonder how widely used it is overall.
  - bee_rider 4y ago
    For BLAS, it has enough legacy that the vendors just make their own libraries, so it would probably be best to use Intel MKL (which has it's own internal dispatching, but I don't think it uses FMV) if you are targeting Intel, and maybe AMD's branch of BLIS if you are targeting AMD. I believe Eigen comes with it's own version of BLAS built in, but it can be made to link to MKL at least, probably BLIS if you are really stubborn.
loeg 4y ago
GCC has offered Function Multiversioning for about a decade now (GCC ~4.8 or 4.9). GCC 6's resolver apparently uses CPUID to resolve the ifunc once at program start: https://lwn.net/Articles/691932/ https://lwn.net/Articles/691932/ . Clang added it in 7.0.0: https://releases.llvm.org/7.0.0/tools/clang/docs/AttributeReference.html#target-gnu-target https://releases.llvm.org/7.0.0/tools/clang/docs/AttributeRe... A nice presentation on it: https://llvm.org/devmtg/2014-10/Slides/Christopher-Function%20Multiversioning%20Talk.pdf https://llvm.org/devmtg/2014-10/Slides/Christopher-Function%...
- indygreg2 4y ago
  While this IFUNC feature does exist and it is useful, when I performed binary analysis on every package in Ubuntu in January, I found that only ~11 distinct packages have IFUNCs. It certainly looks like this ELF feature is not really used much [in open source] outside of GNU toolchain-level software! https://gregoryszorc.com/blog/2022/01/09/bulk-analyze-linux-packages-with-linux-package-analyzer/ https://gregoryszorc.com/blog/2022/01/09/bulk-analyze-linux-...
  - TkTech 4y ago
    I've wanted to use them many times in the past, but the limited support on other compilers (looking at you MSVC) always made it a non-starter. If I have to support some other method of feature detection anyways, there's no point.
  - jeffbee 4y ago
    The use of ifunc in libc, openssl, and zlib covers just about everything people want from it. You want that optimized memcpy, you only need it in one place.
  - arthur2e5 4y ago
    The way ifunc (well, actually language-level FMV) works in GCC and clang is that the input source code, not the command-line switches, specifies what ISA extensions to build for on a candidate function. This naturally means that packagers and other vendors are not using it as much as you hope: they would need to have a separate patch to add these attributes for each architecture and that’s just not maintainable. Even Intel’s Clear Linux does not bother to patch individual libraries. They just use the glibc library multi-versioning feature to load from different directories depending on the cpuid. In my opinion, GCC and Clang could make the whole thing more ergonomic. Ideally you declare a function as "interesting for multi-versioning" in the source using an attribute, then in the command line define what -march's to actually clone for. Kinda like ICC’s /Qax. (On second thought preprocessor defs are sufficient, duh.)
Scene_Cast2 4y ago
For vectorizable workloads, there's Halide. It can be set up in multiple ways, one of which is a JIT that auto-detects CPU capabilities and cache sizes (to better optimize scheduling).

robocat 4y ago

To add, they are using[2] the relatively recent VBMI2 instructions of AVX512. This article[1] talks about the advantages of VBMI on IceLake released 2021. [1] https://www.singlestore.com/blog/a-programmers-perspective/ https://www.singlestore.com/blog/a-programmers-perspective/ comments https://news.ycombinator.com/item?id=28179111 https://news.ycombinator.com/item?id=28179111 [2] https://news.ycombinator.com/item?id=31522464 https://news.ycombinator.com/item?id=31522464

paulmd 4y ago

> AVX-512 support can be confusing because it’s often referred to as a single instruction set. This has been broadly overplayed. There are two main families: consumer AVX-512 and server AVX-512. Server gets some additional VNNI instructions and BFLOAT16 support, consumer gets VPOPCNTDQ and IFMA/VBMI. That's basically all you need to know The chart looks confusing in Wikipedia because it's ordered by date, not by "series" and generation, and some intrepid wikipedia editors have munged up the features even then. This is what the feature set looks like to me: https://i.imgur.com/idAjB1X.png https://i.imgur.com/idAjB1X.png Xeon Phi is sorta its own thing for sure, they did a weird 4-wide architecture and it got special 4-wide VNNI instructions to match. And since it's been abandoned it never was updated. But there's zero reason for average consumer/server software to target Xeon Phi anyway these days - it's abandoned and it was almost entirely a supercomputing/HPC product even when it wasn't. The other instructions are (almost) a straight superset within their series, so you just need to know if you're targeting server or consumer. Laptop gets things a bit earlier because Intel started 10nm first on laptops, and 14nm was stuck with older architectures and then a backported architecture, so it's not quite a strict series in terms of dates, but generation-on-generation they haven't ever regressed a feature within a given "series". The only exception is Alder Lake doesn't get anything because lol Intel - they disabled AVX-512 entirely, but if you got a model that supports it then it's also a superset of all previous consumer generations. Again, all you really need to remember is, consumer gets better FMA and server gets VNNI/BFLOAT, that's the major difference. Big deal, who cares. But "it's just strict supersets of features within server/client" wouldn't get the clicks/exasperated outcry of "gosh isn't this just an intractible mess!". It's not, and they've never regressed a feature within a series. Also for the record, AVX2/AVX1/SSE/MMX look really shitty if you split them up in this same fashion. There were like 4 different "sets" of SSE4 feature sets alone and some of them were abandoned and never used again, and both AMD and Intel adopted them all at different dates and sometimes dropped them back out, so if you made a "feature set vs architecture" table it would be a giant fucking mess for SSE as well, let alone if you did it by year instead of by architectural series. Yet I never hear anyone bring up how awful and how confusing AVX and SSE are, just AVX-512. Feature sets are messy! https://en.wikipedia.org/wiki/SSE4 https://en.wikipedia.org/wiki/SSE4 Eventually, AVX-512 support will converge around the sets of features that are useful, and some of the less-useful features may eventually be deprecated, just like the SSE4a instruction set did. It's gonna be fine.