2 ms·

You can prefetch memory though (but yeah, it indeed is very finicky to get right. So is any sort of decision compilers do, so there is that). Off topic, but the

by kaba0 4y ago

You can prefetch memory though (but yeah, it indeed is very finicky to get right. So is any sort of decision compilers do, so there is that). Off topic, but the OCaml GC does use prefetch instructions for I believe a 3-4x speedup in an already mature project, so there is that. SIMD is still the panacea of compiler optimizations, you won’t get significant performance improvements from trivial peephole optimizations like replacing * 2 with << 1, and the supposedly low-level C has no way of controlling that. Sure, vector sizes are different between processors, but so is many other things we already generalize/assume. But an optimizing compiler will change up your loops all around, inline your functions, etc, so just because it could correspond to some naive assembly, it won’t unless you compile it with some toy.

dzaima 4y ago
Ok, prefetching is a thing that could maybe be standardized as a hint, but even then there could be differences in how many bytes apart your prefetches optimally need to be, and what range they'll prefetch (might even be a property of the specific CPU at runtime, at which point things get really hairy to do properly). My point with SIMD is that pretty much all current things do, at worst, map to one instruction (maybe more for things like large loads/stores) on modern 64-bit architectures. But with SIMD the operations supported by various architectures are so diverse, such that taking the intersection (i.e. preserving things mapping mostly-1-to-1 at worst), you'll have an unusable thing (no non-constant shuffles, 8-bit shifts, float rounding, movemask, fault-only-first loads, compress, limited conversions (for 64-bit floats x86 only has i32↔f64, but NEON only has i64↔f64; and narrowing integers takes a sequence of shuffles on pre-AVX-512 x86)). And if you take the union of architectures, a large amount of operations will just suck on any architecture that's not where it was from. Never mind architectures without SIMD. Different vector sizes is a comparatively trivial problem to tackle (but then you get to the scalable SIMD sets, i.e. SVE & RISC-V V; and even those differ in an important way - SVE vectors are some multiple of 128 bits long, up to 2048, but rvv also guarantees it being a power of two, and can be up to 2^16 bits, at which point indexes in the vectors don't fit in 8-bit integers). Sure, the optimizing compiler will optimize things, but that's what it does - optimize things, so its actions should be preferable, and manually writing something equivalent to what the compiler did is still very possible, and often not even that weird (with the discussed exception of vectorization).
- janwas 4y ago
  This seems overly pessimistic :) We have actually implemented all of those things except first-fault loads in a cross-platform way at reasonable cost in https://github.com/google/highway https://github.com/google/highway. If any of those is the one missing piece that's preventing you from vectorizing, then it's totally worthwhile to emulate it. It may sound horrible to take 4-5 instructions instead of one, but it can still be very worthwhile. FYI for SVE, Arm has requested input and recently clarified that there will only be power of two vector lengths. Happy to discuss.
  - dzaima 4y ago
    Yeah, I've seen it! https://google.github.io/highway/en/master/quick_reference.html#speeding-up-code-for-older-x86-platforms https://google.github.io/highway/en/master/quick_reference.h... In many cases, yeah, emulating things can be fine, but in others, very much not so. If not doing trivial (i.e. autovectorizable) things, I'd imagine you'd still have to be at least somewhat careful about what things will get emulated, and see whether the gain from everything around it makes it worth it for your desired architectures, which is still far from a state of things I'd consider worthy of being standardized. Interesting note on SVE vector lengths though. I'll keep that in mind. Haven't gotten to exploring SVE/RVV much; don't have any personal hardware to optimize for to encourage it.
    - janwas 4y ago
      Yes, I appreciated your previous list of 'things it would be better to work around if possible'. But "a large amount of operations will just suck" did sound overly pessimistic :) We are talking about a small percentage of the ~220 ops in Highway. I agree it's important to benchmark the performance win in any case. Perhaps even automatically picking the best instruction set, because it's nontrivial to predict. As to SVE, it can help on AWS Graviton3 - but it's still only 2x256 bit vs 4x128 bit, so whether it's a win depends on whether you can/do use some of the newer instructions that were not yet supported in NEON.
      - 4y ago

by kaba0 4y ago

dzaima 4y ago

Ok, prefetching is a thing that could maybe be standardized as a hint, but even then there could be differences in how many bytes apart your prefetches optimally need to be, and what range they'll prefetch (might even be a property of the specific CPU at runtime, at which point things get really hairy to do properly). My point with SIMD is that pretty much all current things do, at worst, map to one instruction (maybe more for things like large loads/stores) on modern 64-bit architectures. But with SIMD the operations supported by various architectures are so diverse, such that taking the intersection (i.e. preserving things mapping mostly-1-to-1 at worst), you'll have an unusable thing (no non-constant shuffles, 8-bit shifts, float rounding, movemask, fault-only-first loads, compress, limited conversions (for 64-bit floats x86 only has i32↔f64, but NEON only has i64↔f64; and narrowing integers takes a sequence of shuffles on pre-AVX-512 x86)). And if you take the union of architectures, a large amount of operations will just suck on any architecture that's not where it was from. Never mind architectures without SIMD. Different vector sizes is a comparatively trivial problem to tackle (but then you get to the scalable SIMD sets, i.e. SVE & RISC-V V; and even those differ in an important way - SVE vectors are some multiple of 128 bits long, up to 2048, but rvv also guarantees it being a power of two, and can be up to 2^16 bits, at which point indexes in the vectors don't fit in 8-bit integers). Sure, the optimizing compiler will optimize things, but that's what it does - optimize things, so its actions should be preferable, and manually writing something equivalent to what the compiler did is still very possible, and often not even that weird (with the discussed exception of vectorization).

janwas 4y ago
This seems overly pessimistic :) We have actually implemented all of those things except first-fault loads in a cross-platform way at reasonable cost in https://github.com/google/highway https://github.com/google/highway. If any of those is the one missing piece that's preventing you from vectorizing, then it's totally worthwhile to emulate it. It may sound horrible to take 4-5 instructions instead of one, but it can still be very worthwhile. FYI for SVE, Arm has requested input and recently clarified that there will only be power of two vector lengths. Happy to discuss.
- dzaima 4y ago
  Yeah, I've seen it! https://google.github.io/highway/en/master/quick_reference.html#speeding-up-code-for-older-x86-platforms https://google.github.io/highway/en/master/quick_reference.h... In many cases, yeah, emulating things can be fine, but in others, very much not so. If not doing trivial (i.e. autovectorizable) things, I'd imagine you'd still have to be at least somewhat careful about what things will get emulated, and see whether the gain from everything around it makes it worth it for your desired architectures, which is still far from a state of things I'd consider worthy of being standardized. Interesting note on SVE vector lengths though. I'll keep that in mind. Haven't gotten to exploring SVE/RVV much; don't have any personal hardware to optimize for to encourage it.
  - janwas 4y ago
    Yes, I appreciated your previous list of 'things it would be better to work around if possible'. But "a large amount of operations will just suck" did sound overly pessimistic :) We are talking about a small percentage of the ~220 ops in Highway. I agree it's important to benchmark the performance win in any case. Perhaps even automatically picking the best instruction set, because it's nontrivial to predict. As to SVE, it can help on AWS Graviton3 - but it's still only 2x256 bit vs 4x128 bit, so whether it's a win depends on whether you can/do use some of the newer instructions that were not yet supported in NEON.
    - 4y ago