4 ms·
Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though th
by jart 4y ago
Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though there's something far more important happening here. When you read() or memcpy() loaded memory from a 12GB file, then you need at least 24GB of RAM, because the copied memory pages are competing with the kernel's file cache pages. Once you go over that, the kernel is going to delete the file cache, thereby ensuring you have to do a full disk read every time. Using mmap() ensures the kernel knows they're both the same thing. That means you should be able to run models that are 2x larger than before, without compromising system stability.
- ithkuil 4y agoWould reading without the cache (e.g. O_DIRECT) can be used to achieve a similar effect?
- jart 4y agoWell I think that would guarantee your always hit disk for sure. The goal here is to be able to run the llama.cpp command repeatedly, possibly in shell scripts, and rely on the file caches being available so that the command loads instantly (we're talking like 1 second rather than 60 seconds to run, because disk is 100x slower).
- dekhn 4y agoO_DIRECT is absurdly slow in my my experience.
- the8472 4y agoO_DIRECT is basically telling the OS to load a block of data straight from disk into your process memory, skipping all the caching, sharing, scheduling, coalescing and readaheads the OS normally does for you. Which means you now either must do all that yourself in userspace or you can use io_uring to have it happen in the background and get notified when it's done. But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
- Tepix 4y agoHi, can something along these lines also be used to speed up loading of models running in the GPU memory? With a GDDR6X memory bandwidth of 936 GB/s and PCIe 4.0 x16 bandwidth of 64GB/s loading something like 20GB into the VRAM of an RTX 3090 shouldn't take longer than 1/2 a second or so, right? (assuming it is in the kernel cache)