2 ms·

FYI the model is almost 150GB: https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct/tree/main https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct/tree/mai...

• moffkalast 2y ago
  Nobody runs these things at full precision, a 4 bit quant of 70B range models fits into 64 GB. Though you would need more for long context.
  • extheat 2y ago
    A simple equation to approximate it is `memory_in_gb = parameters_in_billions * (bits/8)` So at 32 bit full precision, 70 * (32 / 8) ~= 280GB fp16, 70 * (16 / 8) ~= 140GB 8 bit, 70 * (8 / 8) ~= 70GB 4 bit, 70 * (4 / 8) ~= 35GB However in things like llama.cpp quants sometimes it's mixed so some of the weights are Q5, some Q4, etc, so you usually want to take the higher number.
    • moffkalast 2y ago
      Well that and you also need a fair bit more space for the KV cache which can be a bit unpredictable. Models without GQA, flash attention or 4 bit cache support are really terrible in that regard, plus it depends on context length. Haven't found a good rule of thumb for that yet.
  • lostmsu 2y ago
    How much ELO on chatbot arena does 4 bit lose vs full fp16/bf16?
    • moffkalast 2y ago
      I think this is the only known benchmark that actually compares different quants: https://oobabooga.github.io/benchmark.html https://oobabooga.github.io/benchmark.html It's not really all that consistent, but larger models can be compressed more without as much loss.