3 ms·
Here is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second.
by liuliu 4y ago
Here is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second. That means we need 8s per token for 30B model. Hmmm, the math is a bit off (or I need to look closer whether we can do more tokens per iteration with some batching).
- 0xbadc0de5 4y agoI think your math is missing some details. An RTX 4090 has a memory bandwidth of 1,008 GB/s. PCIe 4 x16 has a 32GB/s bandwidth. DDR4 RAM has 3200 MT/s transfer rate. An AMD 5900x can easily max that out. A good NVME can hit 7 GB/s read speeds or better. And the 4-bit CUDA kernels can pack 16x 4-bit ints into a single 64-bit transfer / register.
- visarga 4y agoYou can use Speculative Sampling, where a draft model is used to generate short continuations of a sequence, which are scored in parallel by the large model. The draft model can be small, and you only need to call the large model from time to time, so you can stream it from SSD or cheap RAM. Using LRDIMM DDR4 at the price of less than $1000 it is possible to stream GPT-3 five times a second, in 4bit quantisation. Multiply that with the 2-2.5x speedup from Speculative Sampling. >Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318