3 ms·

can you explain the math of how you got to 25 gpus? everyone seems to know these conversions and idk if i missed the memo or something

• coolspot 3y ago
  220 Bp * 8 = 1760 Bp 1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 consumer-grade GPUs (3090/4090) or 1760/80=22 professional-grade GPUs (A100/H100).
  • swyx 3y ago
    thank you for this! 1Bp ~ 1Gb VRAM 8bit is a very nice equivalence to hold in my head.
    • jsjohnst 3y ago
      If parameters are 8 bit (aka a byte), one billion parameters is roughly 1 billion bytes, aka 1 gigabyte. ;)
      • swyx 3y ago
        doh. of course
  • LarsDu88 3y ago
    I was incorrectly calculating based on 1 weight == 1 transistor which is totally wrong. This figure that you provided is more accurate. We can see that today, the MI300X can already run inference for some open source LLMs: https://www.youtube.com/watch?v=rYVPDQfRcL0 https://www.youtube.com/watch?v=rYVPDQfRcL0 There are almost certainly algorithmic optimizations still available. LLM-scale computing inside of consumer robots should be achievable by the end of the decade. In fact electric cars are probably the best "housing" for this sort of hardware.