6 ms·

Can someone break this down? Since this seems to be inferencing without having the entire model loaded into memory, is it possible this could be a way to relax

Can someone break this down? Since this seems to be inferencing without having the entire model loaded into memory, is it possible this could be a way to relax memory requirements of the 65b model?

layer8 4y ago
I don’t think so: > the gains here are mostly due to not copying memory anymore, and better cooperation with the kernel's page manager. We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved. If I understand correctly, this only provides a speed up when the model is already in the OS’s file system cache, and in any case you still have to load the entire model into memory.
- jart 4y ago
  Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though there's something far more important happening here. When you read() or memcpy() loaded memory from a 12GB file, then you need at least 24GB of RAM, because the copied memory pages are competing with the kernel's file cache pages. Once you go over that, the kernel is going to delete the file cache, thereby ensuring you have to do a full disk read every time. Using mmap() ensures the kernel knows they're both the same thing. That means you should be able to run models that are 2x larger than before, without compromising system stability.
  - ithkuil 4y ago
    Would reading without the cache (e.g. O_DIRECT) can be used to achieve a similar effect?
    - jart 4y ago
      Well I think that would guarantee your always hit disk for sure. The goal here is to be able to run the llama.cpp command repeatedly, possibly in shell scripts, and rely on the file caches being available so that the command loads instantly (we're talking like 1 second rather than 60 seconds to run, because disk is 100x slower).
    - dekhn 4y ago
      O_DIRECT is absurdly slow in my my experience.
      - the8472 4y ago
        O_DIRECT is basically telling the OS to load a block of data straight from disk into your process memory, skipping all the caching, sharing, scheduling, coalescing and readaheads the OS normally does for you. Which means you now either must do all that yourself in userspace or you can use io_uring to have it happen in the background and get notified when it's done. But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
  - Tepix 4y ago
    Hi, can something along these lines also be used to speed up loading of models running in the GPU memory? With a GDDR6X memory bandwidth of 936 GB/s and PCIe 4.0 x16 bandwidth of 64GB/s loading something like 20GB into the VRAM of an RTX 3090 shouldn't take longer than 1/2 a second or so, right? (assuming it is in the kernel cache)
adultSwim 4y ago
"We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecomment-1473271638 https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...
- jart 4y ago
  You still get the gain of zero loading time. The issue with loading a model is it ensures 100% of the file needs to be copied into memory. This change means we don't have to copy at all. The file cache pages can be made directly accessible to the matmul ops. You could have ten of these processes running at once, and they'd all share the same memory. However those pages still have to be pulled into memory. What I was hoping for, when I said that, is it's possible in some cases for mmap() to make things even faster than this change managed to accomplish. Some scientific computing applications use sparse datasets. If you skip loading, and use mmap(), then the system will only load pages as-needed on a 4096-byte basis. If the data were actually sparsely used, then with mmap() you could for instance load a 1TB file into memory on a system with 32GB of RAM, and no file cache, only touch a small portion, and it'll go fast and barely use any memory at all. That's not the case here sadly. It's still however a big improvement.

Can someone break this down? Since this seems to be inferencing without having the entire model loaded into memory, is it possible this could be a way to relax memory requirements of the 65b model?

layer8 4y ago

I don’t think so: > the gains here are mostly due to not copying memory anymore, and better cooperation with the kernel's page manager. We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved. If I understand correctly, this only provides a speed up when the model is already in the OS’s file system cache, and in any case you still have to load the entire model into memory.

jart 4y ago
Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though there's something far more important happening here. When you read() or memcpy() loaded memory from a 12GB file, then you need at least 24GB of RAM, because the copied memory pages are competing with the kernel's file cache pages. Once you go over that, the kernel is going to delete the file cache, thereby ensuring you have to do a full disk read every time. Using mmap() ensures the kernel knows they're both the same thing. That means you should be able to run models that are 2x larger than before, without compromising system stability.
- ithkuil 4y ago
  Would reading without the cache (e.g. O_DIRECT) can be used to achieve a similar effect?
  - jart 4y ago
    Well I think that would guarantee your always hit disk for sure. The goal here is to be able to run the llama.cpp command repeatedly, possibly in shell scripts, and rely on the file caches being available so that the command loads instantly (we're talking like 1 second rather than 60 seconds to run, because disk is 100x slower).
  - dekhn 4y ago
    O_DIRECT is absurdly slow in my my experience.
    - the8472 4y ago
      O_DIRECT is basically telling the OS to load a block of data straight from disk into your process memory, skipping all the caching, sharing, scheduling, coalescing and readaheads the OS normally does for you. Which means you now either must do all that yourself in userspace or you can use io_uring to have it happen in the background and get notified when it's done. But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
- Tepix 4y ago
  Hi, can something along these lines also be used to speed up loading of models running in the GPU memory? With a GDDR6X memory bandwidth of 936 GB/s and PCIe 4.0 x16 bandwidth of 64GB/s loading something like 20GB into the VRAM of an RTX 3090 shouldn't take longer than 1/2 a second or so, right? (assuming it is in the kernel cache)

adultSwim 4y ago

"We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecomment-1473271638 https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...

jart 4y ago
You still get the gain of zero loading time. The issue with loading a model is it ensures 100% of the file needs to be copied into memory. This change means we don't have to copy at all. The file cache pages can be made directly accessible to the matmul ops. You could have ten of these processes running at once, and they'd all share the same memory. However those pages still have to be pulled into memory. What I was hoping for, when I said that, is it's possible in some cases for mmap() to make things even faster than this change managed to accomplish. Some scientific computing applications use sparse datasets. If you skip loading, and use mmap(), then the system will only load pages as-needed on a 4096-byte basis. If the data were actually sparsely used, then with mmap() you could for instance load a 1TB file into memory on a system with 32GB of RAM, and no file cache, only touch a small portion, and it'll go fast and barely use any memory at all. That's not the case here sadly. It's still however a big improvement.