4 ms·

"We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the

• jart 4y ago
  You still get the gain of zero loading time. The issue with loading a model is it ensures 100% of the file needs to be copied into memory. This change means we don't have to copy at all. The file cache pages can be made directly accessible to the matmul ops. You could have ten of these processes running at once, and they'd all share the same memory. However those pages still have to be pulled into memory. What I was hoping for, when I said that, is it's possible in some cases for mmap() to make things even faster than this change managed to accomplish. Some scientific computing applications use sparse datasets. If you skip loading, and use mmap(), then the system will only load pages as-needed on a 4096-byte basis. If the data were actually sparsely used, then with mmap() you could for instance load a 1TB file into memory on a system with 32GB of RAM, and no file cache, only touch a small portion, and it'll go fast and barely use any memory at all. That's not the case here sadly. It's still however a big improvement.