3 ms·

File size on disk is ~10GB.

• armcat 3y ago
  Interesting, so they are using single precision (fp32), so 2.7B x 4 Bytes = ~10 GB. With CUDA overhead and room for context, you would need at least 12GB VRAM. They could use half precision and half that VRAM requirement and save costs for everyone involved. Maybe there is a performance reason why they use full precision.
  • collaborative 3y ago
    Yes, the reason I asked is that when I see SLM I got all excited thinking "finally a small model that fits in cheap hardware for simpler tasks"
    • eightysixfour 3y ago
      Models at this size are regularly quantized to 5/4bit to reduce the size. While there is some degradation, it isn’t as substantial as you would expect.
      • deleted 3y ago
        [deleted]
• sigmar 3y ago
  That's just the parameters? So 32 bit parameters? This blogpost is incredibly misleading by putting Gemini nano-2's "size" as larger than Phi-2 (in Table 2 displaying only the number of parameters) and saying "Phi-2 matches or outperforms the recently-announced Google Gemini Nano 2, despite being smaller in size." Because Gemini nano parameters are 4 bit. So Gemini nano-2 is 1.6 GB (3.25/2) in size compared to Phi-2's 10GB
  • stavros 3y ago
    LLM size isn't measured in bytes, but in parameters.