4 ms·

Nice. It would be even nicer, if Zig would support hot-dispatch for SIMD, i.e. the idea that the compiler can emit multiple versions of the same function/code f

Nice. It would be even nicer, if Zig would support hot-dispatch for SIMD, i.e. the idea that the compiler can emit multiple versions of the same function/code for a number of vector widths simultaneously and the runtime selects the best (widest) option available for the hardware running the code - this is something ISPC does and is incredibly useful for targeting a range of architectures.

hansvm 3y ago
Zig the language doesn't do this right away, but: 1. IMO it'd be a bit more ziggish to branch at compile time. Zig has cross-compilation as a first-class feature, and you generally know the architecture you're targeting. 2. Whether you're branching at runtime or compile time, it'd be easy to build a vectored app with that behavior. The first thing that comes to mind is having code that's generic on the vector type (or bit width) and then just choosing which generic to instantiate in an inline for loop in your app's entrypoint. 3. A lot of the time you don't need that behavior. Select a vector width 8x too big, ensure your data is chunked into multiples of that, and rely on the compiler to break that down into a few instructions of the appropriate length. You can't effectively target a GPU that way, and it's not the same as hand-tuned assembly, but you get decent results on a vast array of problems.
- murkt 3y ago
  Counterpoint for 1: packages on PyPI or NPM. When I’m publishing an extension for another language. I really have no idea about their architecture. I’ve seen people just compile for the lowest common denominator. E.g. faiss-cpu on PyPI doesn’t even include AVX2 and is slow as heck.
- barrkel 3y ago
  Architecture yes, but specific feature set, including exact support for each incremental revision of the SIMD instruction and register set, not so much. If you're e.g. providing a docker image for a service, it's fine to build per architecture, but you don't want the hassle of producing a different one for SSE4, SSE4.1, SSE4.2, SSE4a and so on.
  - janwas 3y ago
    Fair point, but we find it useful to have a few clusters: SSE4, AVX2 (Haswell), Skylake, newer AVX-512 Icelake/Zen4. That's still manageable, we're just compiling the SIMD parts 4 times and binary size impact is very modest (we're not replicating the entire binary).
- anonymoushn 3y ago
  regarding point 2, it's not that simple. A lot of 256-bit-wide goodies are in AVX-512 and a lot of different sets of extensions exist.
- p0nce 3y ago
  Counterpoint: it's not all positive to do multiversioning. - You pay for it in build times, more IR, more optimizations. - a lot of software is for servers, in that case you mostly know the arch before-hand. Multi-versioning is best for consumer software. - The lowest spec computers, in consumer software, need the optimizations the most. And they don't have latest instructions. - SIMD has biggest gains from memory optimizations, and those do not necessarily require the latest and greatest instructions.
  - burntsushi 3y ago
    Indeed. This is how ripgrep works. It's compiled for just plain `x86_64`, but it looks for whether things like AVX2 are enabled. And if so, uses vector algorithms for substring and multi-substring search. The nice thing about dealing with strings is that the "coarse" requirement is already somewhat natural to the domain. But, this functionality is absolutely critical. It doesn't even have to be automatic. Just the ability to compile functions with certain ISA extensions enabled, and then only call them when the requisite CPU features are enabled is enough. In a nutshell: https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07be2bb66dc2659d9cf28ad32/src/memchr/x86/mod.rs https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07b...
- burntsushi 3y ago
  You can't branch at compile time. Well, you can, but that's less useful than branching at runtime. If you branch at compile time, then you need to produce binaries that have a minimum supported ISA extension. But if you branch at runtime, then you can produce maximally portable binaries that only use ISA extensions when they're available.
Laremere 3y ago
While Zig doesn't support this automatically, I think there's a path towards this thanks to comptime support. For example: const std = @import("std"); fn f(comptime width: comptime_int, value: i32) i32 { const v = @splat(width, @as(i32, value)); return @reduce(.Add, v); } pub fn main() !void { std.debug.print("1={d}\n", .{f(1, 5)}); std.debug.print("2={d}\n", .{f(2, 5)}); std.debug.print("4={d}\n", .{f(4, 5)}); } Here it creates 3 different versions of the function f at compile time, and then calls them each in succession. Running it prints: 1=5 2=10 4=20 In practice you'd need to set up a dispatch that chooses the function based on the hardware, and ensure that zig/LLVM are actually using the full width of the vectors when compiling.
kristoff_it 3y ago
It's planned: https://github.com/ziglang/zig/issues/1018 https://github.com/ziglang/zig/issues/1018
AndyKelley 3y ago
This is called [function multi-versioning](https://github.com/ziglang/zig/issues/1018 https://github.com/ziglang/zig/issues/1018).
exDM69 3y ago
The issue with runtime instruction set detection and dynamic dispatch is that it needs to be at a rather coarse level to be beneficial. Take a simple 4-wide dot product for example, on x86_64 you'd have 3-4 different implementations (SSE2, SSE3 w/ hadd, SSE4.2 w/ dpps). But the function itself is just a few clock cycles, and calling it via function pointer will eliminate any gains and you might as well compute it with a scalar loop at that point. This is further compounded by inhibiting compiler optimizations. You can't use the dot product function in higher level code expecting it to be inlined and further optimized (which is really the key to performance) if it's behind a dynamic dispatch. A sufficiently smart compiler could maybe propagate the dynamic dispatch above, so that all the dot products get inlined but all code using dot product would get emitted multiple times with different dot product implementations, with the dynamic dispatch only at the top level. This has a slight risk of combinatorial explosion, but there really aren't that many combinations of supported ISAs in real hardware out there. Another option you can use without any special compiler support is to take all your performance sensitive parts and pack them into a shared object/dll, compile multiple versions with different compiler options and choose the correct dll at runtime. Or even build the entire executable a few times and have some kind of launcher pick the correct one.
- kps 3y ago
  What you say about dynamic dispatch also applies to regular function calls, which is why I'm disappointed that Zig provides no visible distinction at the call site between direct and indirect calls (as K&R C did but ANSI C made optional). I understand the desire for magic-indirection ergonomics; I just don't think the tradeoffs work out the same for code vs data.
  - hansvm 3y ago
    If the function is comptime-known then you'll get a direct call, and you can mark it near the call site as being required to be comptime-known (e.g., by marking a function argument to another function with the "comptime" tag). To make it super explicit you can make a no-op function like `fn cknown(comptime f: anytype) @TypeOf(f) {return f;}` and then replace stuff like `foo(bar)` with `cknown(foo)(bar)`. It's a tiny bit harder to force an indirect call if that's desired for some reason; I think you'd need to write a slightly longer never inlined helper function to strip the constness from the pointer. It's doable though, just not directly provided by the language.

hansvm 3y ago

Zig the language doesn't do this right away, but: 1. IMO it'd be a bit more ziggish to branch at compile time. Zig has cross-compilation as a first-class feature, and you generally know the architecture you're targeting. 2. Whether you're branching at runtime or compile time, it'd be easy to build a vectored app with that behavior. The first thing that comes to mind is having code that's generic on the vector type (or bit width) and then just choosing which generic to instantiate in an inline for loop in your app's entrypoint. 3. A lot of the time you don't need that behavior. Select a vector width 8x too big, ensure your data is chunked into multiples of that, and rely on the compiler to break that down into a few instructions of the appropriate length. You can't effectively target a GPU that way, and it's not the same as hand-tuned assembly, but you get decent results on a vast array of problems.

murkt 3y ago
Counterpoint for 1: packages on PyPI or NPM. When I’m publishing an extension for another language. I really have no idea about their architecture. I’ve seen people just compile for the lowest common denominator. E.g. faiss-cpu on PyPI doesn’t even include AVX2 and is slow as heck.
barrkel 3y ago
Architecture yes, but specific feature set, including exact support for each incremental revision of the SIMD instruction and register set, not so much. If you're e.g. providing a docker image for a service, it's fine to build per architecture, but you don't want the hassle of producing a different one for SSE4, SSE4.1, SSE4.2, SSE4a and so on.
- janwas 3y ago
  Fair point, but we find it useful to have a few clusters: SSE4, AVX2 (Haswell), Skylake, newer AVX-512 Icelake/Zen4. That's still manageable, we're just compiling the SIMD parts 4 times and binary size impact is very modest (we're not replicating the entire binary).
anonymoushn 3y ago
regarding point 2, it's not that simple. A lot of 256-bit-wide goodies are in AVX-512 and a lot of different sets of extensions exist.
p0nce 3y ago
Counterpoint: it's not all positive to do multiversioning. - You pay for it in build times, more IR, more optimizations. - a lot of software is for servers, in that case you mostly know the arch before-hand. Multi-versioning is best for consumer software. - The lowest spec computers, in consumer software, need the optimizations the most. And they don't have latest instructions. - SIMD has biggest gains from memory optimizations, and those do not necessarily require the latest and greatest instructions.
- burntsushi 3y ago
  Indeed. This is how ripgrep works. It's compiled for just plain `x86_64`, but it looks for whether things like AVX2 are enabled. And if so, uses vector algorithms for substring and multi-substring search. The nice thing about dealing with strings is that the "coarse" requirement is already somewhat natural to the domain. But, this functionality is absolutely critical. It doesn't even have to be automatic. Just the ability to compile functions with certain ISA extensions enabled, and then only call them when the requisite CPU features are enabled is enough. In a nutshell: https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07be2bb66dc2659d9cf28ad32/src/memchr/x86/mod.rs https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07b...
burntsushi 3y ago
You can't branch at compile time. Well, you can, but that's less useful than branching at runtime. If you branch at compile time, then you need to produce binaries that have a minimum supported ISA extension. But if you branch at runtime, then you can produce maximally portable binaries that only use ISA extensions when they're available.

Laremere 3y ago

While Zig doesn't support this automatically, I think there's a path towards this thanks to comptime support. For example: const std = @import("std"); fn f(comptime width: comptime_int, value: i32) i32 { const v = @splat(width, @as(i32, value)); return @reduce(.Add, v); } pub fn main() !void { std.debug.print("1={d}\n", .{f(1, 5)}); std.debug.print("2={d}\n", .{f(2, 5)}); std.debug.print("4={d}\n", .{f(4, 5)}); } Here it creates 3 different versions of the function f at compile time, and then calls them each in succession. Running it prints: 1=5 2=10 4=20 In practice you'd need to set up a dispatch that chooses the function based on the hardware, and ensure that zig/LLVM are actually using the full width of the vectors when compiling.

kristoff_it 3y ago

It's planned: https://github.com/ziglang/zig/issues/1018 https://github.com/ziglang/zig/issues/1018

AndyKelley 3y ago

This is called [function multi-versioning](https://github.com/ziglang/zig/issues/1018 https://github.com/ziglang/zig/issues/1018).

exDM69 3y ago

The issue with runtime instruction set detection and dynamic dispatch is that it needs to be at a rather coarse level to be beneficial. Take a simple 4-wide dot product for example, on x86_64 you'd have 3-4 different implementations (SSE2, SSE3 w/ hadd, SSE4.2 w/ dpps). But the function itself is just a few clock cycles, and calling it via function pointer will eliminate any gains and you might as well compute it with a scalar loop at that point. This is further compounded by inhibiting compiler optimizations. You can't use the dot product function in higher level code expecting it to be inlined and further optimized (which is really the key to performance) if it's behind a dynamic dispatch. A sufficiently smart compiler could maybe propagate the dynamic dispatch above, so that all the dot products get inlined but all code using dot product would get emitted multiple times with different dot product implementations, with the dynamic dispatch only at the top level. This has a slight risk of combinatorial explosion, but there really aren't that many combinations of supported ISAs in real hardware out there. Another option you can use without any special compiler support is to take all your performance sensitive parts and pack them into a shared object/dll, compile multiple versions with different compiler options and choose the correct dll at runtime. Or even build the entire executable a few times and have some kind of launcher pick the correct one.

kps 3y ago
What you say about dynamic dispatch also applies to regular function calls, which is why I'm disappointed that Zig provides no visible distinction at the call site between direct and indirect calls (as K&R C did but ANSI C made optional). I understand the desire for magic-indirection ergonomics; I just don't think the tradeoffs work out the same for code vs data.
- hansvm 3y ago
  If the function is comptime-known then you'll get a direct call, and you can mark it near the call site as being required to be comptime-known (e.g., by marking a function argument to another function with the "comptime" tag). To make it super explicit you can make a no-op function like `fn cknown(comptime f: anytype) @TypeOf(f) {return f;}` and then replace stuff like `foo(bar)` with `cknown(foo)(bar)`. It's a tiny bit harder to force an indirect call if that's desired for some reason; I think you'd need to write a slightly longer never inlined helper function to strip the constness from the pointer. It's doable though, just not directly provided by the language.