6 ms·
New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy
- koe123 2y agoPerhaps somewhat off topic, does anyone know how stuff like ggml compares to runtimes (tensorflow lite, onnxruntime, etc.)?
- ilaksh 2y agoOn what hardware exactly?
- refulgentis 2y agoI'm intimately familiar, maintain ONNX and llama.cpp Flutter libraries across all 6 True Platforms. Quick opinionated TL;DR: - llama.cpp for LLMs and can do whisper with it's core dependency, GGML. - ONNX for everything else. - TF is the Apple of ML, it's great if you're completely wedded to Google ML ecosystem. Virtually dead outside that. (Something absurd ,like, 94%, of HF models are Pytorch) - only chance I'd have to do a direct comparison in inference performance is Whisper in ONNX vs. GGML, someone got my llama .cpp lib running with Whisper and didn't report significant perf difference
- svnt 2y agoWhat are the True Platforms, in this case?
- crthpl 2y agoI'm guessing MacOS, Linux, Android, Windows, iOS, and Web?
- refulgentis 2y agoCorrect
- a_wild_dandan 2y agoWhat’re your thoughts on MLX? It’s been phenomenal on my MBP.
- refulgentis 2y agoNo time* to try it unfortunately :( Sounds great, though, and Mac just kicks the pants of every other platform on local inference thanks to Metal, I imagine MLX must extend that lead to the point Qualcomm/Google has to a serious investment in open source acceleration. Cheapest iPhone from 2022 kicks the most expensive Android from 2023 (Pixel Fold) around the block, 2x on inference. 12 tkns/s vs. 6/s. * it sounded like a great idea to do an OpenAI LLM x search app. Then it sounded like a great idea to add embeddings locally for privacy (thus, ONNX). Then it sounded like a great idea to do a local LLM (thus, llama.cpp). Then it sounded like a great idea to differentiate by being on all platforms, supported equally. Really taxing. Think I went too far this time. It works but, jeez, the workload...hopefully, after release, it turns out maintenance load is relatively low
- catgary 2y agoI don’t think that’s a terribly fair description of Google - AWS’s chips (inferon and trainium) both have robust XLA/JAX support. Plus JAX now exports MLIR, so there is a really compelling JAX -> IREE pipeline so JAX models can more or less be deployed anywhere, even on bare metal embedded devices.
- a_wild_dandan 2y ago(in llama.cpp, for CPU)
- jart 2y agoI developed this originally for llamafile, which was included in the last two releases: https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.8.2 https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.8.2 Now we're upstreaming it to the llama.cpp project. There are other performance enhancements you can currently only get from llamafile, such as Kawrakow's work making K quants go much faster.
- breakingcups 2y agoIs that just because nobody has made an effort yet to port them upstream, or is there something inherently difficult about making those changes work in llama.cpp?
- leblancfg 2y agoI get the impression most llama.cpp users are interested in running models on GPU. AFAICT this optimization is CPU-only. Don't get me wrong – a huge one! – and opens the door to running llama.cpp on more and more edge devices.
- pclmulqdq 2y agoWait, computing SiLU directly using some numerical analysis is probably a lot faster than doing an exp each time. Is there a significant perf impact to doing this?
- jart 2y agoWith expf() most of the work had already been done and I could kill two birds with one stone. If you want to do the math for doing SiLU directly, that'd be an awesome change I'd happily merge into llamafile. You might even be able to get that into PyTorch and even bigger name projects.
- deleted 2y ago[deleted]
- SeanAnderson 2y agooff topic, but sheesh. I was skimming this and thought, "This seems like a crazy optimization. It's complex and the code in question has had a ton of eyes on it already." and then I saw the contributor and was like "Of course it's jart. It's always jart with the crazy good solutions." well done :)
- mhh__ 2y ago> adapted from arm limited optimized routine Shoulders of giants and all that
- neonsunset 2y agoMostly looks scary* because that's just how it is with intrinsics syntax in C and C++. As with many things there, this pain is mostly self-inflicted. There are C++ libraries that allow for C#-style SIMD and hardware instrinsics syntax as far as I'm aware. It comes at a disadvantage as you can't directly lookup mnemonics in ISAs documentation though. *not to seem as if dismissing the importance of the work done there, just highlighting that it could have been much more accessible to wider audience, but I'm not gonna suggest something that everyone here would consider preposterous such as rewriting inference back-end in C# just yet
- larodi 2y agoIts not something that they teach at asymptotic analysis lectures, right? which reminds me of professor who famously said - that constant that everyone's just overlooking can in engineering terms very much eat your head.
- boshalfoshal 2y agoUnrelated but I see this sentiment a lot. Algorithms (and related courses) are supposed to be largely mathematical/theory based so it sort of has to be hardware agnostic. Imo an algorithms course doesn't really need to teach you about lowering the constant factor (unless the optimizations for lowering said factor are reasonably abstracted away from physical HW), thats the purpose of Computer Architecture or Operating Systems course. And realistically in the real world you'd be benchmarking these things anyway if a supposedly "optimal" algorithm is really the bottleneck, and you'd apply context specific optimizations to speed it up.
- mhh__ 2y ago> replaces short[65536] look up table Is that not quite dim to begin with (having a LUT the size of the whole L1 cache?) or does it work surprisingly well because of some probabilistic fudging?
- andy99 2y agoWhy you (probably) shouldn't use a lookup table https://specbranch.com/posts/lookup-tables/ https://specbranch.com/posts/lookup-tables/ gives some discussion about when it's appropriate generally. My narrow experience is that you can do a lot of real time calculation before it's faster to do a lookup.
- o11c 2y agoOne case that doesn't mention explicitly: sometimes your problem is lucky enough that you have the choice between: 1. for each function, for each datum, call function with datum 2. for each datum, for each function, call function with datum For 1, if your main data is fetched predictably but does not saturate bandwidth, it can be beneficial to use even a fairly large LUT. Note also that while actual L1i is semi-ignorable, e.g. branch prediction benefits much more from remaining "in cache". For 2, assuming other functions also have their own miscellaneous data, the non-LUT approach is likely to win. But you should probably benchmark against #1. But sometimes you can get the best of both worlds: 3. for each arbitrary chunk of data (smaller than L3, or maybe even L2 depending on when your throughput saturates), for each function, for each datum in the chunk, call function with datum Yes, you should of course do whole-system benchmarks, but being able to guess helps you write a useful implementation to benchmark in the first place.
- pclmulqdq 2y agoIn the two cases you have specified here, #2 is almost always the winner on performance. So much so that in performance-sensitive code, many people will (justifiably) default to it without a benchmark. Computers almost always operate in a memory bandwidth bound state, and have comparatively idle cores, and #1 is likely to just be wasteful of the resource that will almost always be the binding constraint. Examples are ECS systems in games, and async run-to-completion runtimes on servers. HPC systems also tend to operate this way. Also, in the interest of disclosure, I wrote the blog post you are responding to.
- mysteria 2y agoHow much do these silu and softmax improvements affect the LLM inference speed as a whole? Correct me if I'm wrong but I feel that this change will only have a small effect as the majority of the time is spent doing matrix multiplications.
- terafo 2y agoOverwhelming majority of flops is indeed spent on matmuls, but softmax disproportionately uses memory bandwidth, so it generally takes much longer than you'd expect from just looking at flops.
- cgearhart 2y agoWhy does it disproportionately use bandwidth?
- jacobn 2y agoIn transformers the attention matrix is N*N, so there are a lot of values to go over. Typically makes it memory bandwidth bound, not compute bound.
- cgearhart 2y agoOooooh, I forgot that the self attention layer has a softmax. I thought this was referring to a softmax on the dense forward layer. Thanks! Next question: does the softmax in the SA block cause it to be bandwidth bound—won’t it have to materialize all the parameters of the N^2 matrix either way? Does SM cause redundant data reads?
- bjornsing 2y agoWouldn’t the softmax typically be “fused” with the matmul though?
- anewhnaccount2 2y agoYes but as far as I understand this is only really usefully possible with FlashAttention. (The main idea is that you have to use the log-sum-exp trick when computing the softmax, but can't compute the max activation incrementally so have to rescale everything.)
- deleted 2y ago[deleted]
- lxe 2y agoAt this point, is gguf/llama.cpp a more performant solution for unbatched inference on CUDA devices, or is exllamav2+flashattention still reigning supreme?
- GuuD 2y agoThe difference is negligible on 2x 4090. There are more important differences like 4 bit KV cache.
- mjcohen 2y agoAbout 20 years ago, I was programming for the Hughes radar signal processor, a highly parallel pipelined machine which accounted for much of Hughes success in radar processing. Anyway, I needed to compute e^x for 0 < x < 1. The processor had a multiply, so I used four 256 long tables of e^x for each possible 8-bit values in the 4 blocks in the 32-bit word, multiplied them to get the final value. It was about 5 times as fast as the previous best e^x routine. That machine was fun! It is obsolete now, but for many years is could process radar signals faster that processors that were nominally many times faster.
- evmar 2y agoIn case anyone else wasn't quite following this, I think the idea is e^x = e^(a+b+c+d) (where abcd are the bytes of x) = e^a * e^b * e^d * e*d and then you make a lookup table for each of those e^a, e^b values. (I fudged "a" here, where it's really more like "high byte << 24", but I think you just make your lookup table for e^a be a map from a => e^(a<<24), and similar for the other bytes.)
- dekhn 2y agoWhen I took Huffman's Cybernetics course as an undergrad, his tests were full of multiplication and you couldn't use a calculator, so he provided a table of exponents and logarithms, and you'd look up in one table, sum the values, then look up the result in another table. I was not a fan.
- gdevenyi 2y agoThis is literally what a sliderule is.
- bigbacaloa 2y agoIn their day, slide rules were the most efficient easily available option.
- 2y ago
- KaoruAoiShiho 2y agoThis does help the gguf usecase of partial offloading to GPU? It'll help the CPU part be faster too?
- Nuella19 2y ago[dead]
- petermcneeley 2y agoOne can vectorize LUTs as well https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#avxnewtechs=AVX2&text=gather https://www.intel.com/content/www/us/en/docs/intrinsics-guid... I wrote about the kinds of things that are possible with LUTs awhile back https://darkcephas.blogspot.com/2018/10/validating-utf8-strings-with-lookup.html https://darkcephas.blogspot.com/2018/10/validating-utf8-stri...
- boulos 2y agoYes, but a direct `exp` implementation is only like 10-20 FMAs depending on how much accuracy you want. No gathering or permuting will really compete with straight math.
- janwas 2y agoWith AVX-512 one can have a 128-byte table with one vector of lookups produced each cycle :)
- celrod 2y agoYes, I have an AVX512 double precision exp implementation that does this thanks to iperm2pd. This approach was also recommended by the Intel optimization manual -- a great resource. I just went with straight math for single-precision, though.
- rdevulap 2y agoOn similar lines, faster tanh https://github.com/microsoft/onnxruntime/pull/20612 https://github.com/microsoft/onnxruntime/pull/20612
- jart 2y agoGreat work. But what's their goal? Are they trying to make that GeLU approximation go faster? Things would probably go a lot faster going back to the erff().
- DeathArrow 2y agoWith all such optimizations, isn't llama too slow to be run on a CPU with a reasonable amount of parameters?
- nxpnsv 2y agoI don't see how that conclusion follows from this optimization.
- baq 2y agoModels get better for equal parameters, CPUs get faster and good folks at intel, amd and arm are probably working very hard to catch up with apple silicon in terms of memory architecture. I can see this be very relevant in a couple of years.
- jart 2y agoI'm gpu poor so I use cpu to run large models. For example, llamafile running mixtral 8x22b on my threadripper can chew through long legal documents and give me advice in a few minutes, using f16 weights and kahan summation. Show me someone who has that many graphics cards.
- kkzz99 2y agoWhat are your tokens/s on that setup?
- bsenftner 2y agoAre you "gpu poor" by choice? I would have thought you, of all people, would be "gpu wealthy" like few can imagine.
- jart 2y agoI'm a hobbyist developer. I haven't been employed in six years. In the past few months I've been fortunate enough to find work as a TVC for Mozilla, which has been wonderful. It's a good lifestyle that affords me a great deal of freedom and autonomy, but certainly not a lot of GPUs. I have an RTX 4090, a few XTX cards, and I rent stuff from Vast occasionally, but that's about it.