4 ms·

if fp32, 4 bytes per param. but the weights may have been quantized to lower precision (eg fp16, so 2B/p)

• ponyous 3y ago
  GPT3: 175B * 4 = 600GB GPT4: 4 * 1.76T = 7TB That feels too much, but I have no idea tbh.
  • k__ 3y ago
    Does it? IDK... Nevertheless, it would be cool if it could be reduced 95% in the future, haha.