4 ms·

If the task is to just overwrite existing files without blocking, why not mmap()?

by riobard 12y ago

userbinator 12y ago
mmap() has no way of knowing how much you'll write to the file, where, in what order, or even whether you may read before doing any writing, so it still has to read the blocks into memory first. A block-aligned write() or pwrite(), however, doesn't require any reading.
quotemstr 12y ago
I wish people would stop bringing up mmap as a performance panacea. It's frequently worse than using IO system calls due to the need to actually take a page fault and map the file's backing page into the faulting processes' address space. mmap is not a magical fast path. It certainly shouldn't be your fault IO strategy.
rdtsc 12y ago
As others have suggested, if mmap is so fast, wouldn't it be expected that write/pwrite would just be mapped (pun intended) to mmap inside the standard library? [Heck for all I know, that might already be the case] I have had to actually write benchmark tests to show some people I work with that there was no conlusive difference between the two (with our data access patterns). mmap having the disadvantage that it allows you more ways to shoot yourself in the foot (ever seen SIGBUS signals?). Before that they swore up and down how mmap is this magic performance hack that was burried in there for ages and only the elites know about it.
- evan_miller 12y ago
  The semantics for write(2) and mmap(2) differ. Neither can be used to implement the other in full. It's possible to have access patterns which don't demonstrate the efficiencies of mmap but it would be incorrect to suggest they do not exist. The overhead of making a syscall to access an area of a file and copying the data being accessed is significant for many workloads.
- beagle3 12y ago
  My experience has been that mmap is not magical for writing in most workloads, but it is mostly magical for reading - I have yet to encounter a real life workload in which mmaping and using memory was NOT easier and at least as fast as reading. On 32-bit systems, however, it's easy to run out of usable address space - 64-bit makes it useful again.
  - _delirium 12y ago
    One workload where I find it worse: if my access pattern is sequential reads from the start of a file (no seeking or random access), or could reasonably be rewritten as such. In that case, using mmap() breaks some expected Unixy flexibility, because it demands real files, while there's no reason in this access pattern that your program should die if it finds a named pipe instead. Of course you could test for pipe and provide an alternate read path, but then you might as well just use that for real files too, instead of maintaining two paths.

If the task is to just overwrite existing files without blocking, why not mmap()?

by riobard 12y ago

If the task is to just overwrite existing files without blocking, why not mmap()?

mmap() has no way of knowing how much you'll write to the file, where, in what order, or even whether you may read before doing any writing, so it still has to read the blocks into memory first. A block-aligned write() or pwrite(), however, doesn't require any reading.

quotemstr 12y ago

I wish people would stop bringing up mmap as a performance panacea. It's frequently worse than using IO system calls due to the need to actually take a page fault and map the file's backing page into the faulting processes' address space. mmap is not a magical fast path. It certainly shouldn't be your fault IO strategy.

rdtsc 12y ago

As others have suggested, if mmap is so fast, wouldn't it be expected that write/pwrite would just be mapped (pun intended) to mmap inside the standard library? [Heck for all I know, that might already be the case] I have had to actually write benchmark tests to show some people I work with that there was no conlusive difference between the two (with our data access patterns). mmap having the disadvantage that it allows you more ways to shoot yourself in the foot (ever seen SIGBUS signals?). Before that they swore up and down how mmap is this magic performance hack that was burried in there for ages and only the elites know about it.

evan_miller 12y ago
The semantics for write(2) and mmap(2) differ. Neither can be used to implement the other in full. It's possible to have access patterns which don't demonstrate the efficiencies of mmap but it would be incorrect to suggest they do not exist. The overhead of making a syscall to access an area of a file and copying the data being accessed is significant for many workloads.
beagle3 12y ago
My experience has been that mmap is not magical for writing in most workloads, but it is mostly magical for reading - I have yet to encounter a real life workload in which mmaping and using memory was NOT easier and at least as fast as reading. On 32-bit systems, however, it's easy to run out of usable address space - 64-bit makes it useful again.
- _delirium 12y ago
  One workload where I find it worse: if my access pattern is sequential reads from the start of a file (no seeking or random access), or could reasonably be rewritten as such. In that case, using mmap() breaks some expected Unixy flexibility, because it demands real files, while there's no reason in this access pattern that your program should die if it finds a named pipe instead. Of course you could test for pipe and provide an alternate read path, but then you might as well just use that for real files too, instead of maintaining two paths.