4 ms·
The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in
by bastawhiz 2mo ago
The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.
- wmf 2mo agoThat math doesn't really work. 8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2 8B model (Taalas) = 4 GB ROM = ~800 mm2