janwas

8 ms·

91.
by janwas 2y ago
The code is unfortunately not (yet) open source. The CPU with 50x is an SKX Gold, and it is similar for Zen4. I compute this ratio as #FMA * 4 / total system memory bandwidth. We are indeed not fully memBW bound :)
92.
by janwas 2y ago
OK, we agree that L1-resident workloads see a benefit. I also agree with your analysis if the loads actually come from memory. Let's look at a more interesting case. We have a dataset bigger than L3. We touch a small part of it with
93.
by janwas 2y ago
I agree server CPUs are underprovisioned for memBW. Each core's share is 2-4 GB/s, whereas each could easily drive 10 GB/s (Intel) or 20+ (AMD). I also agree "some" (for example low-arithmetic-intensity) workloads w
94.
by janwas 2y ago
It is not that the entire working set has to fit within SRAM. Kernels that reuse portions of their inputs several times, such as matmul, can be compute bound and there AMD's AVX-512 shines.
95.
by janwas 2y ago
Thanks for sharing :) Any thoughts on what kind of things you are looking for and didn't find? I cannot recall anyone saying this kind of thing is a bottleneck for them. We don't use std::range, but searching for a negative value
96.
by janwas 2y ago
As mentioned ("re-include our source file"), we are indeed able to put the SIMD code, as well as the self-#include of itself, in a load.cxx TU. Here is an example: https://github.com/google/gemma.cpp/blob
97.
by janwas 2y ago
What leads you to that conclusion? It is still possible to use #include in module implementations. We can use that to make the module implementation look like your example. Thus it ought to be possible, though I have not yet tried it.
98.
by janwas 2y ago
I think I do understand, this is exactly what we do. (MEGA_MACRO == HWY_NAMESPACE) Then we have a table of function pointers to &AVX2::foo, &AVX3::foo etc. As long as the module exports one single thing, which either calls into or e
99.
by janwas 2y ago
> Google Highway manage to pull it off but the trade off is a variadics interface that I personally find very difficult. I'm curious what you mean by 'variadics', and what exactly you find difficult? People new to Highway
100.
by janwas 2y ago
We use pragma target instead of compiler flags :)
101.
by janwas 2y ago
It is possible to avoid ODR violations :) We put the per-target code into unique namespaces, and export a function pointer to them.
102.
by janwas 2y ago
+1, dynamic dispatch is important. Our Highway library has extensive support for this. Detailed intro by kfjahnke here: https://github.com/kfjahnke/zimt/blob/multi_isa/examples/mul...
103.
by janwas 2y ago
+1, rounding f32 to bf16 is helpful. For the other direction, the approach we take in Highway/gemma.cpp is to load a full vector of bf16, then shift/AND to isolate the odd/even elements and convert to float. These can execute
104.
by janwas 2y ago
Thank you for sharing your thoughts! > it’s supposed to make writing faster code easier and more automatic Agree with this viewpoint. I suppose that makes it compiler-like in spirit, though much simpler. I also agree that waiting for inp
105.
by janwas 2y ago
Curious - how is/was performance helped by rewriting? Why not reach out to us, to see if it can be fixed in the library - wouldn't that be cheaper than rewriting?
106.
by janwas 2y ago
Wrappers can be zero-overhead, so any claim of better codegen vs the underlying intrinsics sounds dubious. "best result for the [higher] effort involved" also contradicts my experience, so I ask for evidence. One counterexample: o
107.
by janwas 2y ago
I did, and I truly do not understand why some people do this. As shown in the reddit comments on this article [1], the initial intrinsics version was quite suboptimal and clearly worse than portable code [2]. When not busy unnecessarily rew
108.
by janwas 2y ago
OK :) A thin abstraction, though. If comparing with alternative categories such as domain-specific language or autovectorization, I'd still classify Highway as intrinsics, just portable and easier to use.
109.
by janwas 2y ago
I agree not everything needs to be rewritten. And neither does code using an abstraction such as Highway, so we can stop beating that dead horse.
110.
by janwas 2y ago
Agreed. FWIW we demonstrated with JPEG XL (image codec, though also with animation 'video' support) that it is possible to write such kernels using the portable Highway intrinsics.
111.
by janwas 2y ago
Unfortunately a bit late :) Highway reached v1.0 about 2.5 years ago. How long would it take until Clang/GCC/MSVC are ready, and all users' distros have updated? Not to mention that the number of ops provided by std::experi
112.
by janwas 2y ago
Max performance is a stretch - recompilation would not utilize tensor cores, right? "too hard for mainstream programmers" seems overly pessimistic. I've run several workshops where devs have written dot-product kernels using
113.
by janwas 2y ago
I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions. The cost is that the
114.
by janwas 2y ago
My understanding is that there is a lot of hand-writing (not just fine-tuning) going on. AFAIK CuDNN and TensorRT are written directly as SASS, not CUDA. And the presence of FP8 in H100, but not A100, would likely require a complete rewrite
115.
by janwas 2y ago
Not sure why SIMT would help, it requires more compiler transforms than if the code is written for packets/vectors or whatever we want to call them. As you note, cross-lane is a key part of a good SIMD abstraction. Vulkan calls it &quo
116.
by janwas 2y ago
I like taking the trimmed mean of 10-20 runs, or if a run is quick, the (half-sample) mode of more runs. See robust_statistics.h.
117.
by janwas 2y ago
Right, though the out of bounds depends on VL, which makes it less generally useful than a known 128-bit limit. It also requires at least one extra instruction to get a mask.
118.
by janwas 2y ago
Good to know. There is more to shuffle instructions than just their speed, though. Arm TBX leaves out-of-bounds elements unchanged, and x86 PSHUFB can also zero the result and implicitly clear bits 4..6. Some SW does make use of that, so it
119.
by janwas 2y ago
Great writeup! One comment: PSHUFB does seem important, rather than just "nice to have". > This should be mostly mitigated because the base V extension only has the full lane-crossing-gather, hence a lot of software will use la
120.
by janwas 2y ago
This advice seems specific to an interpreter with "hundreds of slow paths" :)

More ›