4 ms·

For the instruction tuned models, Llama 3 8B is even significantly better than Llama 2 70B!

• rileyphone 3y ago
  To be fair, the Llama 2 instruction tuning was notably bad.
  • oersted 3y ago
    I see it more as an indirect signal for how good Llama 3 8B can get after proper fine-tuning by the community.
• seydor 3y ago
  how much vram does the 8B model use?
  • cjbprime 3y ago
    In general you can swap B for GB (and use the q8 quantization), so 8GB VRAM can probably just about work.
    • lolinder 3y ago
      If you want to not quantize at all, you need to double it for fp16—16GB.
      • cjbprime 3y ago
        Yes, but I think it's standard to do inference at q8, not fp16.
  • derac 3y ago
    You can use 5 bits per parameter with negligible loss of capability as a general rule. 4 bits for a tiny bit worse results. This is subject to changes in how good quantization is in general and on the specific model.