4 ms·

There are some cool ideas in here. I've long been curious why people don't use mmap to re-use all those wonderful pages that got loaded (without reparsing the

• liuliu 4y ago
  That's actually the point of safetensors format, although it needs to take care of alignment. I actually use this technique to mmap directly to GPU buffer on Apple systems (although it seems to segfault on iOS 15.x systems, only supported on 16.x and above).
• arthurcolle 4y ago
  Can you explain how this is possible to do? Sorry I haven't gotten this low level with any of these models before and I'd really appreciate how to understand this.
  • delusional 4y ago
    This has nothing to do with the models, just standard *nix stuff. If you mmap the file readonly the pages can be shared by multiple processes without duplication since they are guaranteed to be the same.
  • dekhn 4y ago
    To run inference, you need to load a model from disk into RAM. Usually the model is written in a disk format that is convenient but needs to be parsed at runtime into a RAM-resident C application data structure. In this case, it looks like jart@ modified malloc to capture the memory generated by the loading process and serialized that to disk. When you run, the application calls mmap to make a virtual memory association with the bytes on disk- so any time you access RAM and it's not yet loaded, ti gets loaded from disk. At that point it gets saved by the kernel in a page cache and since the files on disk don't change, those pages can stay in memory longer than the process. So when the process restarts, all those RAM requests are immediately mapped to already-cached virtual memory, rather than reading from disk. The inference library here supports a data pointer that would point to the memory mapped location. This is faster than relying on the kernel's disk read cache; in that case, you'd still need to convert the data from the disk format to the in-memory format. Normally the data build process is run as an external program that writes the mmap-ready structure to disk (an example is the BLAST program which writes the DNA sequence data into an index structure that is mmapped at runtime). But in this case ti looks like using an instrumented malloc() helps simplify the process of building the disk structure.
    • arthurcolle 4y ago
      Thank you for taking the time to write this out. Very helpful for understanding. I remember using malloc to build out C data structures in my coursework but I must admit I haven't really done much practical work at this level. Thanks again, you are a scholar.
    • selfhoster11 4y ago
      Thank you for taking your time to write this up. For someone with a new-found interest in AI, this is invaluable information.
• csmpltn 4y ago
  > "I've long been curious why people don't use mmap to re-use all those wonderful pages that got loaded" That's because the people that develop these models are often data scientists and have little to no experience with systems programming, optimizations, etc.
  • dekhn 4y ago
    shhh, you're getting close to exposing my guaranteed employment secrets