janwas

11 ms·

121.
by janwas 2y ago
Sure, we generally don't want branches in the code. What makes intrinsics really interesting is that you can use wrapper libraries such as our Highway, then the code actually is portable and efficient.
122.
by janwas 2y ago
Or perf/TCO, or availability? :)
123.
by janwas 2y ago
Good points. gemma.cpp can also be an interesting benchmark, it also uses Highway.
124.
by janwas 2y ago
FWIW I ran a quick test of gemma.cpp on M3 Pro with 8 threads. Similar PaliGemma inference speed to an older AMD (Rome or Milan) with 8 threads. But the AMD has more cores than that, and more headroom :)
125.
by janwas 2y ago
Most of the SIMD code I write can consume tens of GB/s BW per core. From this perspective, it seems that 192 cores make for an imbalanced system.
126.
by janwas 2y ago
That's indeed short for some actual uses such as summarization, but AFAIK many/most? evals involve generating less than 200.
127.
by janwas 2y ago
We (gemma.cpp) recently started accumulating softmax terms into f64. There is at least one known case of this causing differing output, but after 200 tokens, hence unlikely to be detected in many benchmarks. Does anyone have experience with
128.
by janwas 2y ago
Thanks! Yes, that (rminimax/ratapprox) looks very interesting. I got the impression that this was separate from Sollya. Fair point about the correct rounding. We'd be fine with several ULPs.
129.
by janwas 2y ago
Possible misunderstanding: I mean a rational function in the sense of Padé approximation or CF [1], not just representing individual numbers as p/q. I did not find anything related to this in Sollya [2]. [1]: https://www.jst
130.
by janwas 2y ago
hm, why not Chebfun? Result is a rational polynomial so we have to divide, but that seems fine/fast on servers.
131.
by janwas 2y ago
Nice! Any way to subscribe for updates?
132.
by janwas 2y ago
In European inner cities, it can be faster to get around on bikes.
133.
by janwas 2y ago
I acknowledge Jevon and that demand may increase. But the original point was actually that specialized HW will be outcompeted by newer algorithms on more flexible HW. I have seen this play out several times, for example with brute force S
134.
by janwas 2y ago
Brute force hardware locks us into the local minimum of matmul. Stella Nera demonstrated better power efficiency than GPUs, using approx matmul. Maybe the next generation of special purpose HW can help, too. But they come after 2-4 years. W
135.
by janwas 2y ago
+1. Personal opinion: accelerators are useful today but have kept us in a local minimum which is certainly not the ideal. There are interesting approaches such as near linear low-rank approximation of attention gradients [1]. Would we rathe
136.
by janwas 2y ago
Indeed, though Highway is approaching a system library at this point. It is also a thin wrapper over the compiler's intrinsics headers: e.g. ~16KLOC for arm_neon.h, times the number of targets. That's a lot of code, but it is not
137.
by janwas 2y ago
Highway TL here, happy to discuss :)
138.
by janwas 2y ago
I generally agree with your points. On niceness of programming, our Highway SIMD library provides portable intrinsics that cover most of the nice SVE/RVV/AVX-512. Those intrinsics are reasonably emulated on other ISAs. Is it fair
139.
by janwas 2y ago
I do not understand this take :) The vast majority of our code is vector length agnostic, as is required for SVE and RVV. GPU perf/TCO is not always better than CPU, if you can even get enough GPUs, and the latency is also a concern an
140.
by janwas 2y ago
Going through Highway's set of shuffle ops: 64-bit OddEven/Reverse2/ConcatOdd/ConcatEven, OddEvenBlocks, SwapAdjacentBlocks, 8-bit Reverse, CombineShiftRightBytes, TableLookupBytesOr0 (=PSHUFB) and Broadcast especially f
141.
by janwas 2y ago
:) Yes indeed, no changes to the build required. Example: https://gcc.godbolt.org/z/KM3ben7ET I agree dispatch should be at a reasonably high level. This hasn't been a problem in my experience, we have been able t
142.
by janwas 2y ago
Was thinking of a shorter avl producing partial results merged into another reg. Something like a += b; a[0] += c[0]. Without avl we'd just have a write-after-write, but with it, we now have an additional input, and whether this happen
143.
by janwas 2y ago
> With 50+ years of figuring the basics out, RISC-V won't be making any major mistakes on the most important stuff. RVV does have significant departures from prior work, and some of them are difficult to understand: - the whole conc
144.
by janwas 2y ago
> it's a matter of the wrong configuration unlike x86 where it is a matter of the wrong instruction +1 to dzaima's mention of vrgather. The lack of fixed-pattern shuffle instructions in RVV is absolutely a wrong-instruction iss
145.
by janwas 2y ago
Cinebench might not be the most relevant benchmark, it uses lots of scalar instructions with fairly high branch mispredictions and low IPC: https://chipsandcheese.com/2021/02/22/analyzing-zen-2s-cineb... .
146.
by janwas 2y ago
I do not understand why ubiquity or baselines should be required in order to use CPU features :) Since many years, performance-critical libraries have used runtime/dynamic dispatch. Our github.com/google/highway intrinsics ev
147.
by janwas 2y ago
Unfortunately, AMD's masked AVX2 instructions reserve the right to fault even for masked-off elements :(
148.
by janwas 2y ago
Dispatching is actually heavily used. Image/video codecs, cryptography, and ML libraries routinely use it, because the lowest common denominator is very low indeed.
149.
by janwas 2y ago
Highway TL here. The integration is a work in progress; what has been integrated so far is VQSort :)
150.
by janwas 2y ago
I agree that SIMD and vector are basically interchangeable at a certain level. There is still a difference in the binutils, because SSE4 and AVX2 and AVX-512 have different instruction encodings per length. But yes, it is possible to write

More ›