4 ms·
It fascinating to me that coming up on a year since Sapphire Rapids has been available in the public cloud, developers are still targeting AVX512 when they shou
by kiratp 3y ago
It fascinating to me that coming up on a year since Sapphire Rapids has been available in the public cloud, developers are still targeting AVX512 when they should be targeting VNNI and AMX.
https://github.com/ggerganov/llama.cpp/issues/2555 https://github.com/ggerganov/llama.cpp/issues/2555
- yjftsjthsd-h 3y agoThis project in particular seems to care about the long tail of hardware; note that the very first machine in this post is a box from 2020 with spinning rust disk. Granted, adding support for newer extensions is likely also good, but cost/benefit is in play.
- taneq 3y agoIs four years really 'long tail' these days? Our VM host box is from 2010 (and I had to rebuild llama.cpp locally without AVX to get it working :P )
- yjftsjthsd-h 3y agoFor cutting-edge LLM work, probably? I mean, I run mine on older hardware than that, but I'm a total hobbyist...
- d416 3y agoIt should be noted that while the HP Prodesk was released in 2020, the CPU’s Skylake architecture was designed in 2014. Architecture is a significant factor in this style of engineering gymnastics to squeeze the most out of silicon.
- refulgentis 3y agoFor LLMs...yeah. I imagine you're measuring in tokens/minute with that setup. So its possible, but...do you use it much? :)
- luyu_wu 3y agoI don't believe that is the target for a local LLM... Pretty sure we're talking about client-side computing, of which the newest supports only AVX-512 (and even that sketchily on Intel's side).
- kristianp 3y agoJust buy a new AMD processor that supports AVX512.
- baq 3y agoPeople with Sapphire Rapids options are not the target audience of these patches