janwas

14 ms·

181.
by janwas 3y ago
I am inclined to believe those numbers :) That's an interesting question. One partial answer is that the hash table speedup likely involves an interface change, replacing "insert single item" with batching. This is a generall
182.
by janwas 3y ago
Gemma.cpp was also affected, and now fixed ( https://github.com/google/gemma.cpp/pull/93 ). Thanks for the heads-up!
183.
by janwas 3y ago
We use it in gemma.cpp [1]. This hybrid of E5M2 and E4M3 decodes to bf16 in ~14 instructions, so we can do that on the fly during dot products. [1]: github.com/google/gemma.cpp
184.
by janwas 3y ago
:) Thanks for the mention. Highway/vqsort TL here, happy to discuss. PS: I used to work on JPEG XL. It is great to see these outstanding improvements, congrats to the team!
185.
by janwas 3y ago
For the 7B IT and a short factual query I see 5.3 tps on a 5 year old Skylake Gold 6154 CPU @ 3.00GHz, 16 threads. Expect a slight increase as we improve scalability. FYI using the NUQ (4.5-bit) quantization improves throughput by about 1.4
186.
by janwas 3y ago
We understand that some teams prefer to use CPU even when mobile GPU would be available. This code is also intended to facilitate research & experimentation, which may not fall under your definition of 'serious' :)
187.
by janwas 3y ago
FYI we have an experimental 4.5-bit format (NUQ) in gemma.cpp, our C++ inference engine, which from my testing works reasonably well.
188.
by janwas 3y ago
Hi, I devised the 4.5 (NUQ) and 8-bit (SFP) compression schemes. These are prototypes that enabled reasonable inference speed without any fine-tuning , and compression/quantization running in a matter of seconds on a CPU. We do not ye
189.
by janwas 3y ago
> but that algorithm requires costly preprocessing of the set to be matched. For string functions, input strings are usually very short > the advantage of longer vectors severely reduced when your input is unlikely to be longer than o
190.
by janwas 3y ago
Ah, Highway is C++ so that won't work for you then. And indeed CMPISTR is quite nonportable. I'm curious whether you are getting good performance from CMPISTR relative to AVX2 or even AVX-512, for those who have it? Wider vectors
191.
by janwas 3y ago
Great to see SIMD used in string algorithms :) For porting, have you considered using github.com/google/highway? This allows you to write the code once using 'portable' intrinsics, and run on >20 instruction sets. Dis
192.
by janwas 3y ago
Why are we not also talking about memory bandwidth? Personal opinion: this is the key. The latest Phi had about 100 GB/s in 2017. The contemporary Nvidia GTX 1080: 320 GB/s. When CPUs actually come with bandwidth and a decent vect
193.
by janwas 3y ago
Just saw your comment. I agree that remainders are tricky. We do have some ready-made algorithms in hwy/contrib/algo and hwy/contrib/unroller which avoid remainder handling. Any thoughts on how this can be improved?
194.
by janwas 3y ago
Yes :) https://arxiv.org/abs/2310.16795
195.
by janwas 3y ago
hm, any tool that enables people to write more SIMD is probably a good thing, but personally I find it preferable to have the SIMD integrated into the same toolchain. This allows us to (inlined!) call back into C++, use templates/class
196.
by janwas 3y ago
Thanks :) We're always curious to hear how it could be improved?
197.
by janwas 3y ago
This happens regularly in my experience :) Also, when using a SIMD wrapper library, it can actually be quite portable in practice.
198.
by janwas 3y ago
FWIW, I remember Intel manuals years ago warning that CPUID results might change across cores, but it seems software did not prepare :)
199.
by janwas 3y ago
OK, fair :) Do you have any data on how many tasks are actually compute-bound? From where I sit, we're often limited by memory bandwidth. When a CPU such as A64FX or even M2 shows up with decent bandwidth, lo and behold, they are often
200.
by janwas 3y ago
Thanks :) We fixed a performance bug (re-seeding an RNG on each sort).
201.
by janwas 3y ago
Measurements (by the same author) for Intel's x86-simd-sort and vqsort are here: https://github.com/Voultapher/sort-research-rs/blob/main/wri...
202.
by janwas 3y ago
Golly. There are probably already enough comments in other threads saying this, but I'm curious how you would justify the description 'footnote'.
203.
by janwas 3y ago
Interesting, thanks for sharing :) At the time we open-sourced Highway, the standardization process had already started and there were some discussions. I'm curious why stdlib is the only path you see to default? Compare the activity l
204.
by janwas 3y ago
That's before the performance bugfix we talked about, right? (We re-initialized the RNG on every sort.) Vqsort performance on M1 is indeed about half that of AVX-512. The NEON instruction set is missing some important operations for Qu
205.
by janwas 3y ago
Vqsort is indeed using SIMD :)
206.
by janwas 3y ago
Oh, I see, thanks for clarifying. Yes, I was referring only to "same source code" and agree our approach would generate multiple copies of the instructions.
207.
by janwas 3y ago
As mentioned, we implement what applications are using/requesting. Do we know how many permutations are used in ffmpeg? hm, I vaguely remember there was a vzeroupper problem, perhaps one fell through the cracks. Interesting, can you sh
208.
by janwas 3y ago
I'm curious what is the new momentum? It's difficult to understand the value proposition of std::experimental::simd. Standardization has taken many years, lost the "load/store" function naming which just about every
209.
by janwas 3y ago
We actually do use ternlog in several places in Highway :) Whenever we want to use a new immediate arg, we add new ops such as Not, Xor3, Or3, OrAnd, IfVecThenElse that also do something reasonable on other platforms. BTW this reminds me of
210.
by janwas 3y ago
We are able to do the same with Highway's RVV :)

More ›