2 ms·
It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk. Of course an n
by michaelt 2y ago
It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk.
Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second.
So in approximate terms, an LLM and quantisation level that can produce 10 tokens per second on a 4090 will produce 1 token per second from RAM and a token every 10 seconds from SSD.
- slimsag 2y agoI haven't been able to get llama.cpp's mmap logic to work on macOS
- tgtweak 2y agoYeah mmap and leaving the caching up to the OS or storage drivers instead of model-informed placement is not going to yield the same results. Observing data access patterns, pre-empting inference data requirements and understanding hardware latency when distributing the model can yield some pretty significant results, as that same approach does in other domains. This is adjacent to compiler optimization. It would be very meta to use AI to observe these access patterns and distribute the model accordingly based on usage, so that it can optimize placement for your given context domain.