3 ms·

73GB for the 1 bit model...

• naasking 2mo ago
  That probably includes the 51b ngrams too. It's possible that those could be streamed from NVMe on-demand. The Engram paper that developed this technique streamed from RAM to VRAM at only ~1% performance degradation, but these strix halo boxes and the spark have much slower memory, so it's possible moving down another rung on the memory hierarchy wouldn't affect their performance too much. This will almost certainly require changes to llama.cpp or vllm to do it right.
  • naasking 2mo ago
    This guy claims 6% throughout hit for this approach: https://x.com/0xBakeer/status/2092694905978237224?s=20 https://x.com/0xBakeer/status/2092694905978237224?s=20 Crazy how fast things move these days.
• petu 2mo ago
  It's not 1 bit. It's ~4bit for n-gram and ~2.8bit for the model. Not idea why it's called Q1, but likely it's preliminary quant just for PR testing / very likely to be remade after llama.cpp support is merged.