janwas

13 ms·

151.
by janwas 2y ago
Yes, PPL and certain benchmarks do not detect differences from quantization. But recent work gives cause for concern, e.g., https://arxiv.org/pdf/2310.01382 , https://arxiv.org/pdf/2405.18137 .
152.
by janwas 2y ago
:) Confirmed working. We've just pushed the dev branch to main.
153.
by janwas 2y ago
Hi, main author here, happy to discuss :) Though with high latency, back in about 20h.
154.
by janwas 2y ago
Nice :) Our github.com/google/highway also uses this technique to compile your code multiple times, once per target CPU.
155.
by janwas 2y ago
Right, longer vectors let us write more elements at a time. I agree that the number of L1 load ports (or issue width) is also a parameter: that times the SIMD width gives us the bandwidth. It will be interesting to see what AMD Zen5 brings
156.
by janwas 2y ago
Note that compact/compress are not actually the key enablers: also with AVX-512 we use table lookups for u64 keys, because this allows us to actually partition a vector and write it both to the left and write sides, as opposed to compr
157.
by janwas 2y ago
For very wide, I'm thinking of Semidynamic's 2048-bit HW, which with LMUL=8 gives 2048 byte vectors, or the NEC vector machines. AFAIK it has not been publicly disclosed why Intel did not get AVX-512 into their e-cores, and I hear
158.
by janwas 2y ago
+1 for TwoSum [1]. To expand on that: Kahan summation is an approximation which is a bit cheaper, but not great if some inputs can also be negative. That's because it is basically FastTwoSum(a,b) in a loop, which only works if the expo
159.
by janwas 2y ago
I agree with you we do not only want "very wide SIMD", and it seems to me that 2x512-bit (Intel) or 4x256 (AMD) are actually a good middle ground. Sure, it's https://news.ycombinator.com/item?id=40465090 .
160.
by janwas 2y ago
On x86 at least, the cost of OoO is astonishing - more pJ per instruction dispatch than the operation itself. Amortizing that over more operations is the whole point of SIMD. I have not yet seen such data for Arm. That aside, see the "
161.
by janwas 2y ago
Anyone interested in getting such numbers could run github.com/google/gemma.cpp on Arm hardware with hwy::DisableTargets(HWY_ALL_NEON) or HWY_ALL_SVE to compare the two :) I'd be curious to see the result. Calling hwy::Dispat
162.
by janwas 2y ago
Fair point about movprfx, I understand they were short on encoding space. This can be mitigated by using *_x versions of intrinsics where masks are not used. The sizeless headache is anyway there if you want to support RISC-V V, which we do
163.
by janwas 2y ago
It's great you bring up cmp, helps to understand why 4x128 is not necessarily as good as 1x512. Quicksort, hardly a 'weird kernel', does comparisons followed by compaction. Because comparisons return a predicate, which have o
164.
by janwas 2y ago
With AVX-512 one can have a 128-byte table with one vector of lookups produced each cycle :)
165.
by janwas 2y ago
> PSNR is a terrible measure of quality Glad we can agree on that. Unfortunately many evals use that or plain SSIM, which is still L2 at heart. > Encoding speed is not really a concern for web uses hm, in many discussions with Jon of
166.
by janwas 2y ago
At a guess, probably because it matches or at least is in the same order of magnitude encode speed as JPEG XL? It starts to feel like bait and switch if we say "look what we can do with >1000x slower encode", without noting tha
167.
by janwas 2y ago
> which could lead to efficient encoding of AVIF photos Efficient in what sense? HW encoders usually only explore a fraction of the search space, and in the case of JPEG often result in 3..5 bit per pixel images.
168.
by janwas 2y ago
I'm curious what the evidence is for AVIF outperforming below 1bpp? Have you seen this more recent data that includes AVIF? https://cloudinary.com/labs/cid22
169.
by janwas 2y ago
Disclosure: I worked on JPEG XL, opinions are my own. The radio(network) on phones can consume more power than the SoC(CPU). Thus smaller size can translate into energy savings. As to hypothetical advantage, we are talking about HW _potenti
170.
by janwas 2y ago
hm, sounds almost like macros wouldn't make it assembly anymore :) I'm curious whether you know of any such inline asm wrapper? Seems that this gives the compiler less information than the intrinsics, which largely expand to built
171.
by janwas 2y ago
> it doesn't use intrinsics because they aren't actually easier to work with; they are not faster, not more portable Well golly, I'll just have to disagree based on >20 years of experience, including several in assembl
172.
by janwas 3y ago
For anyone interested in a C++ implementation, our github.com/google/gemma.cpp now supports this model.
173.
by janwas 3y ago
For C++, also check out our https://github.com/google/gemma.cpp/blob/main/gemma.cc , which has direct calls to MatVec.
174.
by janwas 3y ago
I think the previous code was using dot products, f32 instead of bf16.
175.
by janwas 3y ago
Nice result, congrats Justine! The bf16 dot instruction replaces 6 instructions: https://github.com/google/highway/blob/master/hwy/ops/x86_12... A 3-4x speedup vs SKX sounds quite plausible :)
176.
by janwas 3y ago
FWIW I have been writing SIMD since 20+ years and worked on JPEG XL, which also contains a good bit of vector code. BTW one anecdote: a colleague mentioned what should have been a quick 20 min patch to ffmpeg took a day because it was writt
177.
by janwas 3y ago
I believe these drawbacks are mostly solved with our github.com/google/highway library which provides portable intrinsics, reasonably documented and grouped into categories :)
178.
by janwas 3y ago
hm, from my point of view, "here are 4+ different ways to do it, depending on which CPU you are running on" sure sounds like fragmentation. Perhaps embedded systems engineers see things differently if they know exactly what CPU th
179.
by janwas 3y ago
I agree the situation changes if there is a single new spec, this is why I wrote "at the moment". Do you have any insight on when it's finished/ widely available? I'd guess two years?
180.
by janwas 3y ago
"Real truth" and "fan boys"? Let's have some facts: there are indeed a boatload of extensions, though perhaps not (yet) to the V extension itself. T-HEAD [1], Ventana's ternary op [2], Sifive also has a couple

More ›