5 ms·

Mmap(2)

59 points by luu 11y ago

rdtsc 11y ago
As a fun heuristic I have come to believe that mmap advocacy and use for storage is somewhat correlated with medium experience level. At first junior developers don't know about it, then, they read a blog, hear something at a meeting about it and all of the sudden they fall in love with mmap. Then they want to re-write every storage related thing using it claiming it will be lightning fast and efficient. "You just write to memory and kernel writes to file, that so awesome, here I refactored your storage backend to use it over the weekend!" Some might even venture to write whole database products based on it. But then one day they "grow up" and realize what a great pain mmap can be, their storage backend is being hammered constantly by input data and dirty pages are not flushed fast enough, and then they are they start blocking other processes for seconds at a time. THey start tweaking madvise setting trying to control behavior, adjusting number of pdflush threads, etc. Or, and also sigbus errors are fun...
- striking 11y ago
  And yet it exists, and works gloriously: http://symas.com/mdb/doc/ http://symas.com/mdb/doc/ The kernel is smarter than your app. Trying to force it to do something it doesn't want to do is a very bad idea.
  - mveety 11y ago
    Kernels generally aren't all that smart. They're usually dumb and simple programs that do things based on some average defaults that look okay and might work for most people. The kernel needs to be told what to do especially if data integrity is important. Like if some dude thought it was a great idea to store up writes and do them in large transactions and you're writing a filesystem that needs verifiable writes, you turn that off or force it off if you can't disable it otherwise. Relying on the utter sausage factory that is your average kernel is a recipe for disaster. That is especially true for Linux in my experience.
    - striking 11y ago
      I said the kernel is smarter than your app, not psychic. Of course you need to tell Linux what to do with specialized applications. However, any option worth disabling can be disabled. Nothing needs to be forced off like the writer of the article did. mmap'd writes should be flushed as soon as possible, not dragged out, especially in the SSD age we have today. What the writer did was actually counterintuitive and makes no sense. None.
  - marssaxman 11y ago
    The kernel does have access to more information than your app, but your app has a specific job to do, while the kernel must provide reasonable service to all possible applications and may not be optimized for your specific case.
- kev009 11y ago
  Not really sure the point you're making, maybe you need some form of back pressure? How would that differ with varying I/O models? Fear of the operating system interfaces is a heuristic for me, especially an unwillingness to work on their implementation when the company has more than on order of 100 engineers.
- rawnlq 11y ago
  What would be the "high experience level" alternative?
  - rewqfdsa 11y ago
    Read and write, most of the time. It's funny how code written by masters tends to look a lot like code written by novices, while intermediate-level code is full of baroque architecture and clever tricks that don't work. The difference between master code and novice code is that a master knows when he can get away with simplicity.
    - hyc_symas 11y ago
      When you've written a world's fastest anything then you'll be in position to judge. But not before then.
      - rewqfdsa 11y ago
        How do you know I haven't? Besides, I said "most of the time". There are obviously important applications for mmap.
      - rdtsc 11y ago
        A fastest for loop that does nothing qualifies? What about fastest product that run 1M TPS but crashes and is down 10 days per month? Is that still fastest?
- buserror 11y ago
  mmap() is still fantastic for mostly-read datasets. If you serialize and queue your writes, and 'replay' them with a little bit of heuristics about page alignments you can also help the kernel a lot with the flushing job. I've discovered mmap 25 years ago on SunOS, I haven't outgrown it yet -- it has served me well over the years, and I expect it will continue to do so ;-)
- beagle3 11y ago
  striking already mentioned mdb, and I will mention kdb+ I'm not sure what those pains are that you are talking about. mmap allows use to share buffers with the OS in a way that is much more efficient than you can do with plain reads; it costs you some asynchrony and (practically these days) the ability to use a 32-bit processor. Other than that, it is the same - you still have to reason about locality of reference and stuff.
- hyc_symas 11y ago
  The ignorance is strong with this one. Not worth the effort to enumerate all that's wrong here. The Apollo DOMAIN OS was entirely built around single-level store, i.e. memory-mapped files. It scaled well beyond anything Sun was building in those days, even across large networks.
vanviegen 11y ago
So what's the alternative? Doesn't data dirtied by a write() experience the same or similar messy flushing behavior?
- rewqfdsa 11y ago
  It depends. Are you using aio? O_DIRECT? write(2) gives you plenty more options and these days, it's no less efficient than mmap. The whole idea of system calls being expensive is obsolete now that we're not using heavyweight software interrupts to implement them. What the fuck do you think happens when you write to a mmap(2)ed page anyway? The kernel has to know the page is dirty somehow so that it can remember to flush it later. It learns about page dirtying by initially mapping the page read-only, letting the CPU fault when you try to write, marking the page dirty, and letting the write proceed. You're entering the kernel either way.
  - hyc_symas 11y ago
    "system calls being expensive is obsolete now" - utter nonsense. System calls are still expensive. Argument checking, copying data between user space and kernel space - none of that cost goes away.
    - mveety 11y ago
      It's far from nonsense. Everything that happens when you call a syscall nowadays is minimal compared to firing off interrupts to do the job. Memory operations are pretty cheap and the argument checking is probably only 20 or so instructions at most. A lot of the cost has gone away. If you still believe this, run dtrace on some random programs and be horrified over how many syscalls are being fired off.
      - eloff 11y ago
        Syscalls are hugely expensive. Not for those reasons, but because of the way they trash the cache, causing reduced performance for a long time after returning control to userspace. This is why to really achieve the performance the hardware is capable of, both for nvram ssds and for high speed network interfaces, you need to bypass the kernel completely. You can do a little better using calls that amortize the costs of the syscalls over multiple units of work, but even with the best optimizations it's still an order of magnitude less than what the hardware is capable of.
todd8 11y ago
These ideas have been around for a long time. I remember using the ideas a quarter of a century ago! See DEC SRC Report 24, "A simple and efficient implementation for small databases", by Birrell, et al. 1988. http://web.archive.org/web/20051215194340/http://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/abstracts/src-rr-024.html http://web.archive.org/web/20051215194340/http://gatekeeper.... I used to read these reports as they came out, very interesting.

59 points by luu 11y ago

rdtsc 11y ago

As a fun heuristic I have come to believe that mmap advocacy and use for storage is somewhat correlated with medium experience level. At first junior developers don't know about it, then, they read a blog, hear something at a meeting about it and all of the sudden they fall in love with mmap. Then they want to re-write every storage related thing using it claiming it will be lightning fast and efficient. "You just write to memory and kernel writes to file, that so awesome, here I refactored your storage backend to use it over the weekend!" Some might even venture to write whole database products based on it. But then one day they "grow up" and realize what a great pain mmap can be, their storage backend is being hammered constantly by input data and dirty pages are not flushed fast enough, and then they are they start blocking other processes for seconds at a time. THey start tweaking madvise setting trying to control behavior, adjusting number of pdflush threads, etc. Or, and also sigbus errors are fun...

striking 11y ago
And yet it exists, and works gloriously: http://symas.com/mdb/doc/ http://symas.com/mdb/doc/ The kernel is smarter than your app. Trying to force it to do something it doesn't want to do is a very bad idea.
- mveety 11y ago
  Kernels generally aren't all that smart. They're usually dumb and simple programs that do things based on some average defaults that look okay and might work for most people. The kernel needs to be told what to do especially if data integrity is important. Like if some dude thought it was a great idea to store up writes and do them in large transactions and you're writing a filesystem that needs verifiable writes, you turn that off or force it off if you can't disable it otherwise. Relying on the utter sausage factory that is your average kernel is a recipe for disaster. That is especially true for Linux in my experience.
  - striking 11y ago
    I said the kernel is smarter than your app, not psychic. Of course you need to tell Linux what to do with specialized applications. However, any option worth disabling can be disabled. Nothing needs to be forced off like the writer of the article did. mmap'd writes should be flushed as soon as possible, not dragged out, especially in the SSD age we have today. What the writer did was actually counterintuitive and makes no sense. None.
- marssaxman 11y ago
  The kernel does have access to more information than your app, but your app has a specific job to do, while the kernel must provide reasonable service to all possible applications and may not be optimized for your specific case.
kev009 11y ago
Not really sure the point you're making, maybe you need some form of back pressure? How would that differ with varying I/O models? Fear of the operating system interfaces is a heuristic for me, especially an unwillingness to work on their implementation when the company has more than on order of 100 engineers.
rawnlq 11y ago
What would be the "high experience level" alternative?
- rewqfdsa 11y ago
  Read and write, most of the time. It's funny how code written by masters tends to look a lot like code written by novices, while intermediate-level code is full of baroque architecture and clever tricks that don't work. The difference between master code and novice code is that a master knows when he can get away with simplicity.
  - hyc_symas 11y ago
    When you've written a world's fastest anything then you'll be in position to judge. But not before then.
    - rewqfdsa 11y ago
      How do you know I haven't? Besides, I said "most of the time". There are obviously important applications for mmap.
    - rdtsc 11y ago
      A fastest for loop that does nothing qualifies? What about fastest product that run 1M TPS but crashes and is down 10 days per month? Is that still fastest?
buserror 11y ago
mmap() is still fantastic for mostly-read datasets. If you serialize and queue your writes, and 'replay' them with a little bit of heuristics about page alignments you can also help the kernel a lot with the flushing job. I've discovered mmap 25 years ago on SunOS, I haven't outgrown it yet -- it has served me well over the years, and I expect it will continue to do so ;-)
beagle3 11y ago
striking already mentioned mdb, and I will mention kdb+ I'm not sure what those pains are that you are talking about. mmap allows use to share buffers with the OS in a way that is much more efficient than you can do with plain reads; it costs you some asynchrony and (practically these days) the ability to use a 32-bit processor. Other than that, it is the same - you still have to reason about locality of reference and stuff.
hyc_symas 11y ago
The ignorance is strong with this one. Not worth the effort to enumerate all that's wrong here. The Apollo DOMAIN OS was entirely built around single-level store, i.e. memory-mapped files. It scaled well beyond anything Sun was building in those days, even across large networks.

vanviegen 11y ago

So what's the alternative? Doesn't data dirtied by a write() experience the same or similar messy flushing behavior?

rewqfdsa 11y ago
It depends. Are you using aio? O_DIRECT? write(2) gives you plenty more options and these days, it's no less efficient than mmap. The whole idea of system calls being expensive is obsolete now that we're not using heavyweight software interrupts to implement them. What the fuck do you think happens when you write to a mmap(2)ed page anyway? The kernel has to know the page is dirty somehow so that it can remember to flush it later. It learns about page dirtying by initially mapping the page read-only, letting the CPU fault when you try to write, marking the page dirty, and letting the write proceed. You're entering the kernel either way.
- hyc_symas 11y ago
  "system calls being expensive is obsolete now" - utter nonsense. System calls are still expensive. Argument checking, copying data between user space and kernel space - none of that cost goes away.
  - mveety 11y ago
    It's far from nonsense. Everything that happens when you call a syscall nowadays is minimal compared to firing off interrupts to do the job. Memory operations are pretty cheap and the argument checking is probably only 20 or so instructions at most. A lot of the cost has gone away. If you still believe this, run dtrace on some random programs and be horrified over how many syscalls are being fired off.
    - eloff 11y ago
      Syscalls are hugely expensive. Not for those reasons, but because of the way they trash the cache, causing reduced performance for a long time after returning control to userspace. This is why to really achieve the performance the hardware is capable of, both for nvram ssds and for high speed network interfaces, you need to bypass the kernel completely. You can do a little better using calls that amortize the costs of the syscalls over multiple units of work, but even with the best optimizations it's still an order of magnitude less than what the hardware is capable of.

todd8 11y ago

These ideas have been around for a long time. I remember using the ideas a quarter of a century ago! See DEC SRC Report 24, "A simple and efficient implementation for small databases", by Birrell, et al. 1988. http://web.archive.org/web/20051215194340/http://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/abstracts/src-rr-024.html http://web.archive.org/web/20051215194340/http://gatekeeper.... I used to read these reports as they came out, very interesting.