4 ms·

mmap is great. I worked on a project a couple months ago where we had ~20 threads concurrently fetching data over the wire and dumping it into a file. We just f

mmap is great. I worked on a project a couple months ago where we had ~20 threads concurrently fetching data over the wire and dumping it into a file. We just ftruncated the file to ~4TB (bigger than we'd ever need it to be) and mmapped'd it all into memory, initializing the first 8 bytes to be an atomic integer that kept track of the actual size. The data needed to end up on disk anyway, and we were more than happy to let the OS take care of that us. We also had very little random access (mostly sequential writes) so it worked really well.

beagle3 7y ago
Did you ever down-truncate it? Did your backup program take forever to crunch through non-existent 4TB (and take as much space to restore?) Atomic access wise, that’s fine as long as you don’t map it through a network file system if any sort; also, $DIETY help you recover a consistent state if the power goes out or you kernel panic Mmap is awesome, I use it all the time. But writing into mmaps robustly is hard. Best example to learn from I am familiar with is symas LMDB source code.
- dekhn 7y ago
  Unwritten pages in files that are extended don't actually take up space or appear as blocks; a good backup program should respect that. See https://en.wikipedia.org/wiki/Sparse_file https://en.wikipedia.org/wiki/Sparse_file
  - beagle3 7y ago
    Indeed, but many programs, backup or otherwise, or unaware of it. IIRC gzip/bzip/zstd don’t (will spend time reading and writing all the missing pages) whereS gnu tar does.
    - terrelln 7y ago
      Zstd by default will write sparse files when it detects runs of zeros during decompression, but isn't aware of it when reading a file.
- butterthebuddha 7y ago
  We down-truncated on process exit. I'll take a look at the symas LMDB source code, thanks for pointing that out.
downerending 7y ago
Beware of multi-host filesystems (e.g., GPFS). The required locking and sync can be far slower than you might get on a local filesystem. If nothing else, it's very useful to have a "don't use mmap" option in your code.
- microtherion 7y ago
  Another risk is that if the file system on which the file resides can get unmounted for any reason, access to the mmap'ed data will fail in nasty ways.
deleted 7y ago
[deleted]

beagle3 7y ago

Did you ever down-truncate it? Did your backup program take forever to crunch through non-existent 4TB (and take as much space to restore?) Atomic access wise, that’s fine as long as you don’t map it through a network file system if any sort; also, $DIETY help you recover a consistent state if the power goes out or you kernel panic Mmap is awesome, I use it all the time. But writing into mmaps robustly is hard. Best example to learn from I am familiar with is symas LMDB source code.

dekhn 7y ago
Unwritten pages in files that are extended don't actually take up space or appear as blocks; a good backup program should respect that. See https://en.wikipedia.org/wiki/Sparse_file https://en.wikipedia.org/wiki/Sparse_file
- beagle3 7y ago
  Indeed, but many programs, backup or otherwise, or unaware of it. IIRC gzip/bzip/zstd don’t (will spend time reading and writing all the missing pages) whereS gnu tar does.
  - terrelln 7y ago
    Zstd by default will write sparse files when it detects runs of zeros during decompression, but isn't aware of it when reading a file.
butterthebuddha 7y ago
We down-truncated on process exit. I'll take a look at the symas LMDB source code, thanks for pointing that out.

downerending 7y ago

Beware of multi-host filesystems (e.g., GPFS). The required locking and sync can be far slower than you might get on a local filesystem. If nothing else, it's very useful to have a "don't use mmap" option in your code.

microtherion 7y ago
Another risk is that if the file system on which the file resides can get unmounted for any reason, access to the mmap'ed data will fail in nasty ways.

deleted 7y ago

[deleted]