4 ms·

You don’t need that much ram to use mmap(2)

• marginalia_nu 2y ago
  To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.
  • ninkendo 2y ago
    Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram. It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.
    • marginalia_nu 2y ago
      Really depends on the shape of the data. mmap can be suboptimal in many cases. For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.