3 ms·

Andy's obsession with mmap is interesting. I don't think it's the worst idea in some cases, like Prometheus. They use mmap to simplify the code and only keep th

Andy's obsession with mmap is interesting. I don't think it's the worst idea in some cases, like Prometheus. They use mmap to simplify the code and only keep the relevant (hot) parts of the index/data in memory. I guess the fact that the data is immutable helps, but that's the point; it's a good idea for some usecases and I don't agree with a blanket "mmap bad"

nikita 4y ago
We used mmap extensively and it took a gigantic effort to purge it out of the codebase in SingleStore. Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. The second one is that mmap prevents you from a tight control of memory in various pipelining scenarios when you want to have your memory overhead to be constant and not sizeof(data).
- menaerus 4y ago
  > Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. But the alternative we have suffers from the same issue, doesn't it? malloc is implemented in terms of sbrk and mmap/munmap, both of which are guarded by locks. And malloc will also have its own user-space locks on top of the ones from the kernel. I checked glibc and jemalloc implementation. > mmap prevents you from a tight control of memory ... when you want to have your memory overhead to be constant and not sizeof(data). I am not sure I understand this argument. It's trivial to implement the user-space custom MMAP allocator that gives you no less control over the memory than other "types" of allocators. Actually, it can give you more power because you can avoid the locks from user-space malloc implementations if you know that the use-case you're crafting it for is single-threaded. I used this technique in the past for short-living objects and in certain workloads it improved the performance by a factor.
  - sakras 4y ago
    When Andy rages about databases using mmap, he means using it to replace the buffer pool. In theory, it provides exactly what the database needs by paging the data in and out on demand. In practice this scales poorly to large number of threads due to TLB shootdowns, and as Nikita said, you give up control of memory usage (since it’s the OS managing it). Nobody is arguing that you shouldn’t call mmap to get yourself a few anonymous virtual pages to do your work in.
    - menaerus 4y ago
      I am probably missing something but I still don't see the difference given that buffer pool will otherwise be implemented in terms of malloc, and malloc is still implemented in terms of mmap so we're back to square one. However, I could imagine that it is possible for contention in kernel-space mmap locks could be artificially relieved because of the user-space locks in malloc. Currently I don't see how TLB shootdowns would be specific to mmap only but not to malloc but I may find some time to read the paper.
      - sakras 4y ago
        What we’re talking about is mmap-ing files vs doing some anonymous mapping and manually reading files. Anonymous mappings don’t incur all the TLB shootdowns that mmap’d files do.

nikita 4y ago

We used mmap extensively and it took a gigantic effort to purge it out of the codebase in SingleStore. Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. The second one is that mmap prevents you from a tight control of memory in various pipelining scenarios when you want to have your memory overhead to be constant and not sizeof(data).

menaerus 4y ago
> Lots of issues the biggest one is that munmap has a mutex and you incur a lot of contention. But the alternative we have suffers from the same issue, doesn't it? malloc is implemented in terms of sbrk and mmap/munmap, both of which are guarded by locks. And malloc will also have its own user-space locks on top of the ones from the kernel. I checked glibc and jemalloc implementation. > mmap prevents you from a tight control of memory ... when you want to have your memory overhead to be constant and not sizeof(data). I am not sure I understand this argument. It's trivial to implement the user-space custom MMAP allocator that gives you no less control over the memory than other "types" of allocators. Actually, it can give you more power because you can avoid the locks from user-space malloc implementations if you know that the use-case you're crafting it for is single-threaded. I used this technique in the past for short-living objects and in certain workloads it improved the performance by a factor.
- sakras 4y ago
  When Andy rages about databases using mmap, he means using it to replace the buffer pool. In theory, it provides exactly what the database needs by paging the data in and out on demand. In practice this scales poorly to large number of threads due to TLB shootdowns, and as Nikita said, you give up control of memory usage (since it’s the OS managing it). Nobody is arguing that you shouldn’t call mmap to get yourself a few anonymous virtual pages to do your work in.
  - menaerus 4y ago
    I am probably missing something but I still don't see the difference given that buffer pool will otherwise be implemented in terms of malloc, and malloc is still implemented in terms of mmap so we're back to square one. However, I could imagine that it is possible for contention in kernel-space mmap locks could be artificially relieved because of the user-space locks in malloc. Currently I don't see how TLB shootdowns would be specific to mmap only but not to malloc but I may find some time to read the paper.
    - sakras 4y ago
      What we’re talking about is mmap-ing files vs doing some anonymous mapping and manually reading files. Anonymous mappings don’t incur all the TLB shootdowns that mmap’d files do.