5 ms·

Discovering and exploring mmap using Go

107 points by brunoac 6y ago

thoughtsunific6 6y ago
9/11 never happened.
tedunangst 6y ago
Does this assume the file fits in address space? That's not always the case.
- jonp888 6y ago
  According to the developers of Mongo at least, it's not worth considering. You can't store more than 2GB of data on a 32-bit machine because they map the entire datastore into memory.
- kaslai 6y ago
  While the default behavior of this library is to map the entire file into address space (and also the default assumption that people have regarding mmap), you can map specific portions of a file into memory to avoid virtual address exhaustion. It's just far less convenient to do so, though typically still faster than the alternatives.
- fnord123 6y ago
  From the first paragraph: > One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory.
  - fnord123 6y ago
    I misread the comment. The file fits in address space? As sibling comment suggests, address space is v. big and it's very much a corner case (or very specific use case) for most people.
- pbalcer 6y ago
  All x86-64 CPUs support 48 bit address space, with 57 bits in the newest ones [1]. It's unlikely that address space exhaustion would be a problem for the vast majority of use cases. [1] - https://en.wikipedia.org/wiki/Intel_5-level_paging https://en.wikipedia.org/wiki/Intel_5-level_paging
slashvar2701 6y ago
This kind of articles, sounds like a reminder that system programming is a fundamental knowledge for software engineers.
- brunoac 6y ago
  Fundamentals are key. And it is very fun for those who really enjoy their craft
EdSchouten 6y ago
For people doing memory mapping in Go, I would strongly advise calling debug.SetPanicOnFault() and setting up a panic handler. Without it, your program will simply crash in case of I/O failures, file truncation, etc.. Here's some code I wrote some time ago that does exactly this: https://github.com/buildbarn/bb-storage/blob/c346ca331930f1bc5e4f9bde75de96ee3e6c8a9c/pkg/blockdevice/memory_mapped_block_device_unix.go#L46-L55 https://github.com/buildbarn/bb-storage/blob/c346ca331930f1b...
Matthias247 6y ago
Be aware that using mmap instead of read/write might mess up Go’s scheduling. The runtime has special handling of other blocking system calls which tried to minimize the impact of those on other running goroutines. With directly accessing memory locations mapped via mmap you won’t get this benefit.

107 points by brunoac 6y ago

thoughtsunific6 6y ago

9/11 never happened.

tedunangst 6y ago

Does this assume the file fits in address space? That's not always the case.

jonp888 6y ago
According to the developers of Mongo at least, it's not worth considering. You can't store more than 2GB of data on a 32-bit machine because they map the entire datastore into memory.
kaslai 6y ago
While the default behavior of this library is to map the entire file into address space (and also the default assumption that people have regarding mmap), you can map specific portions of a file into memory to avoid virtual address exhaustion. It's just far less convenient to do so, though typically still faster than the alternatives.
fnord123 6y ago
From the first paragraph: > One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory.
- fnord123 6y ago
  I misread the comment. The file fits in address space? As sibling comment suggests, address space is v. big and it's very much a corner case (or very specific use case) for most people.
pbalcer 6y ago
All x86-64 CPUs support 48 bit address space, with 57 bits in the newest ones [1]. It's unlikely that address space exhaustion would be a problem for the vast majority of use cases. [1] - https://en.wikipedia.org/wiki/Intel_5-level_paging https://en.wikipedia.org/wiki/Intel_5-level_paging

slashvar2701 6y ago

This kind of articles, sounds like a reminder that system programming is a fundamental knowledge for software engineers.

brunoac 6y ago
Fundamentals are key. And it is very fun for those who really enjoy their craft

EdSchouten 6y ago

For people doing memory mapping in Go, I would strongly advise calling debug.SetPanicOnFault() and setting up a panic handler. Without it, your program will simply crash in case of I/O failures, file truncation, etc.. Here's some code I wrote some time ago that does exactly this: https://github.com/buildbarn/bb-storage/blob/c346ca331930f1bc5e4f9bde75de96ee3e6c8a9c/pkg/blockdevice/memory_mapped_block_device_unix.go#L46-L55 https://github.com/buildbarn/bb-storage/blob/c346ca331930f1b...

Matthias247 6y ago

Be aware that using mmap instead of read/write might mess up Go’s scheduling. The runtime has special handling of other blocking system calls which tried to minimize the impact of those on other running goroutines. With directly accessing memory locations mapped via mmap you won’t get this benefit.