3 ms·
I mean, if we're being pedantic, I think it is possible to run llama.cpp with real memory savings this way... in a contrived situation where you were only gener
by btown 4y ago
I mean, if we're being pedantic, I think it is possible to run llama.cpp with real memory savings this way... in a contrived situation where you were only generating a single token with a single forward pass, and you were limited in RAM.
The new MMAP system would not require you to load the whole model from disk up front, but rather would allow you to load and immediately forget each layer of parameters as you do a forward pass through the GPT architecture. And if you had just tried to use normal swap paging to do this without MMAP, you'd essentially be doing 2x the loading work, since the page for the first layer likely got swapped out as you loaded the rest of the model into "memory."
Of course, as soon as you want to generate a new token, you'd need to reload all the pages of the model parameters. Every token generated would take as long as it would if you were loading the model from scratch.
But for a classification problem where you only need to generate one token, and you need or want to do this in a serverless environment where you'd need to load from disk anyways? I think you'd be able to do that workflow with significantly less RAM in the same amount of clock time, now.
- visarga 4y ago> Every token generated would take as long as it would if you were loading the model from scratch. Wouldn't make sense to keep most of the model in GPU (as much as it fits) and only load the remaining layers on each pass?
- alchemist1e9 4y agoIs it possible to do multi-GPU inference the way you describe and have say 4 GPUs but each doing different layers on a large model? If that is the case then perhaps a high end consumer system with max pcie 5 nvme bandwidth and multi gpus could do inference on large LLM models.
- rfoo 4y agoCongratulations on rediscovering how the original researchers run these models!
- alchemist1e9 4y agoDo they do it in a way that each GPU can have smaller memory than the model weights? I keep seeing references to these absolutely massive 48GB+ gpu memory sizes and those are extremely expensive. If more but smaller memory gpus can work that could open up a much larger range of hardware and individuals to run inference, especially if efficient load/unload of weight layers from raid pcie5 nvme devices is feasible.
- rfoo 4y agoYes? The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory. As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?
- alchemist1e9 4y agoGot it, so looking ahead to building hardware that is useful for inference would you agree with these considerations: - max PCI lanes for more GPUs. AMD cpus and boards probably better - PCIe 5 nvme. ideally enough to saturate bus - max GPUs with max memory. memory more important than FLOPs probably. for a given budget one tries to get as much as possible of above. How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
- rfoo 4y agoPCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother. PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device. max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-technical-analysis https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff. > How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? $10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.