4 ms·

220 Bp * 8 = 1760 Bp 1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 con

220 Bp * 8 = 1760 Bp 1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 consumer-grade GPUs (3090/4090) or 1760/80=22 professional-grade GPUs (A100/H100).

swyx 3y ago
thank you for this! 1Bp ~ 1Gb VRAM 8bit is a very nice equivalence to hold in my head.
- jsjohnst 3y ago
  If parameters are 8 bit (aka a byte), one billion parameters is roughly 1 billion bytes, aka 1 gigabyte. ;)
  - swyx 3y ago
    doh. of course
LarsDu88 3y ago
I was incorrectly calculating based on 1 weight == 1 transistor which is totally wrong. This figure that you provided is more accurate. We can see that today, the MI300X can already run inference for some open source LLMs: https://www.youtube.com/watch?v=rYVPDQfRcL0 https://www.youtube.com/watch?v=rYVPDQfRcL0 There are almost certainly algorithmic optimizations still available. LLM-scale computing inside of consumer robots should be achievable by the end of the decade. In fact electric cars are probably the best "housing" for this sort of hardware.