17 ms·
Use mmap with care
- b4shout 8y agoThat one is amazing...!
- tobyhinloopen 8y agoI feel like you should use some kind of library for this instead of handling it all yourself.
- royjacobs 8y agoWhich library would you propose?
- ben-schaaf 8y agoA library for mmap would have not helped with any of the Windows or Breakpad issues. It would have also been more work up-front than mmap, since none of the problems of mmap were known to us at the time.
- natmaka 8y agoHow widely-used code laying on mmap (LMDB maybe being one of the prominent cases) rolls?
- ben-schaaf 8y agoAuthor here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
- evozer 8y agoDid you consider emulating mmap yourselves? "Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so." I feel like this could be done in c++ directly, by maintaining an internal cache for each file that keeps track of which parts of the file are loaded and uses read() to load chunks on demand. Error handling would be a lot simpler (no signals, just a failed read()) and there would be less OS-specific code.
- nneonneo 8y agoWell, then you have to implement some kind of plan for efficient caching - some kind of LRU scheme, for example, to prevent the cache from ballooning to unusable sizes - at which point you're reinventing the kernel page cache (poorly). mmap does have a big advantage here if you really need a lot of random accesses.
- maxlybbert 8y agoIt’s easy enough to read a file in chunks, parsing out the information as you go. This limits memory use as long as you release the chunks when you no longer need them. The operating system can swap out memory as-needed, even if you didn’t get the memory from mmap, so it’s irrelevant where you store the parsed data. Unless you actually need to read the file multiple times (compared to looking at the parsed in-memory data multiple times), this should be fast enough.
- ben-schaaf 8y agoThis is essentially how databases like PostgreSQL work, but in essence it only avoids the sys-call overhead. The OS is already caching the file, regardless of mmap, so using pread would have likely been enough for us. It totally would have been simpler overall, but each incremental step we made was significantly less work than the refactoring required for pread.
- quotemstr 8y ago> The OS is already caching the file Not necessarily. With O_DIRECT, pread() doesn't put pages into page cache: it just DMAs them directly into your process. Using O_DIRECT and the process-private caching we've been discussing, sophisticated programs (like databases) can (and do!) implement their own "page cache" systems. And because databases have access pattern information that the generic kernel VM subsystem doesn't, such a database can frequently do a better job doing this caching on its own.
- ben-schaaf 8y agoI might have undersold the performance advantage of writing your own cache, but let me reiterate the point I was trying to make: The reason we didn't consider doing so was because we weren't having a performance issue. Writing our own cache would be strictly more work than just using pread and accomplished the same thing.
- emilfihlman 8y agoJust use signalfd and refactor into an event loop without jumping around.
- nneonneo 8y agoThis does not work - signalfd cannot catch synchronous signals, per the man page: > Limitations > The signalfd mechanism can't be used to receive signals that are synchronously generated, such as the SIGSEGV signal This is because synchronous signals are fired at the thread that caused them, and signalfd read() calls can't be used to read signals fired at other threads.
- eeZah7Ux 8y ago"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.
- maxlybbert 8y agoI think it’s reasonable to say “it’s difficult to justify using mmap when other programs are expected to manipulate, and potentially even delete, the file while you are working with it.” But, honestly, that case will always be hard to handle, and mmap doesn’t make it any worse.
- nneonneo 8y agoI think the problem here is that mmap does make it worse - with (p)read, if you get an errno, you flag an error. with mmap, you have to handle a signal and jump back to an appropriate point in the code (and you can't do this in a cross-platform way), and then flag an error. Obviously, what the upper levels do with the error still has to be worked out, but you can hardly argue that sigsetjmp/siglongjmp + signal handlers + Windows SEH handlers is no worse than "ret = read(...); if (ret < 0) ERROR;"
- maxlybbert 8y agoIf the file changes out from under you, all you can do is reread the file. mmap gives you a pointer that you can read from, but you still have to loop through the file’s bytes. I’m pretty sure most programmers are capable of writing a signal handler that sets a flag (volatile sig_atomic_t) for “parsing failed,” and a loop that checks that flag in addition to checking whether the loop is finished for other reasons. Signal handling doesn’t have to be complicated.
- nneonneo 8y agoIt’s not quite as simple as setting a flag - if the signal is SIGBUS/SIGSEGV then returning from the handler normally will retry the faulting instruction. So, you either have to map dummy memory into place so the fault doesn’t reoccur, or use sigsetjmp/siglongjmp to jump out to a previously configured exception handler (as Sublime Text did here. I think you’ve mixed up synchronous signals - SEGV, BUS, ABRT - with asynchronous signals like QUIT, INT, USR1. Asynchronous signals can be handled easily with a flag and a loop as you mentioned; synchronous signals are much trickier and much more complicated.
- T3OU-736 8y agoThank you for the write-up. As an occasional scripter (a coder I am not), I found it very useful - a very nice and structured presentation of both coding practices, and of the challenges of supporting multiple environments with their own way of doing things. Showing the scope of change within the editor is a rather nice touch. Visualization of complexity, if you will.
- Hackbraten 8y agoSmall side note for completeness: The effects of oh-noes-my-file-is-gone can be somewhat mitigated by using the heuristics built into NSData (instead of using mmap directly). For example, you call NSData’s `dataWithContentsOfFile:options:error:` with the `NSDataReadingMappedIfSafe` option [1]. The framework will then transparently mmap the file unless it believes there’s an elevated risk of the file going away. Apple doesn’t disclose how NSData exactly makes that decision; however, I’ve found a few reports that say it uses mmap internally when the file is on the root filesystem, and fall back on an in-memory copy otherwise. It’s a rather dumb heuristics though, and may not solve the issue entirely. [1] https://developer.apple.com/documentation/foundation/nsdatareadingoptions/nsdatareadingmappedifsafe https://developer.apple.com/documentation/foundation/nsdatar...
- saagarjha 8y agoThe method also checks for things like whether the file is in /dev and whether filesystem compression is enabled.
- ben-schaaf 8y agoWouldn't that mean that networked files would be loaded entirely into RAM, negating the whole reason we started using mmap in the first place?
- yoodenvranx 8y agoSince I am a really big fan of Sublime Text I tried to switch from Gitkraken to Sublime Merge a few times but I just don't enjoy working with Sublime Merge. Is there any chance that merging and rebasing via drag-n-drop is coming to Sublime Merge? For me that's the one big feature which keeps me from switching from Gitkraken to Sublime Merge.
- daemin 8y agoI was using Sublime Merge on the weekend and after doing a few changes with my remotes using the normal command line, it exited (well crashed I assume). When things like this happen do you automatically get an error report? Because you mentioned that you use the error reporting library from Google in that article.
- ben-schaaf 8y agoWe do get crash reports. If you have some more details in relation to this it would be great to hear from you on our forums or on the issue tracker: https://github.com/SublimeTextIssues/Core https://github.com/SublimeTextIssues/Core
- gianpaj 8y agoI believe the correct link for SublimeMerge GH issues is: https://github.com/sublimehq/sublime_merge https://github.com/sublimehq/sublime_merge
- avar 8y agoIs there a post where it's covered why Sublime Merge implements things like packfile reading on its own, rather than using git's own plumbing? E.g. in this case presumably keeping a "git cat-file --batch" would do the trick. I contribute to git.git, and it would be interesting to know if there's inherent issues stopping you from doing that, or if it's implementation problems in some cases (e.g. missing plumbing commands or features). There's definitely interest from upstream in reviewing patches / helping if there's missing or inadequate plumbing. You'd get upstream features for free as they come along. E.g. presumably you haven't implemented the new MIDX format, but that speeds up pack file access by a lot for some use-cases, and presumably the boring bits of low-level git operations aren't much of a selling point in and of themselves. Aside from whether you'd use "git" itself, such a trick of using a slave process you'd talk to over IPC of some sort would cover some of the issues you wrote about, e.g. issue with sharing global state with libraries like Breakpad.
- loeg 8y agoThe license of git (GPL2) might be an issue for a commercial product. libgit2 is also GPL. (Also, IPC and fork+exec has overhead that mmap or thread in the same program does not.)
- avar 8y agoThere are no license issues with bundling the git command-line tools in a commercial product, there's multiple existing proprietary commercial applications built on top of git that do so. The libgit2 code is GPL with a linking exception, so you can use it (unlike "git" itself) as a C library in a proprietary commercial product. > IPC and fork+exec has overhead[...] The "git cat-file --batch" command is something you'd invoke once, and then as your program runs you keep feeding it SHA-1s on stdin and it spews out their content on stdout. So even on Windows the overhead of that should be fine. It's clear from ben-schaaf's other comments (which I read later) that one concern was the simplicity of downstream APIs being able to read the data using a normal C variable. But that just leaves more questions. People in this thread are mentioning pack files, assuming that a multi-GB "git object" must be in a pack, but I notice the original post doesn't say anything about it. If they're reading packs with this they'll need to parse it, resolve deltas etc. So likely the code that deals with the mmap()'d variable is small in any sane codebase (they're surely not doing delta resolution repeatedly all over the place...). If they're very large loose objects those will most likely be zlib compressed, so wouldn't this need to go through some intermediary API layer anyway? I guess if SM itself is adding them it could add them uncompressed. Since ben-schaaf mentioned this not being about performance, but about saving memory I thought this might be something like wanting to extract a small part of a 1GB object from git for display. That seems like a thing an editor might want to do. In that case "git cat-file --batch" would suck, but not for some intrinsic reason. An API could be added that could take the start/end of an object to print out.
- jlokier 8y agoYou've used thread_local for the sigjmp_buf. Are you sure the mechanism used by thread_local is safe to use in a signal handler? Relevant bug from Rust: "TLS accesses aren't async-signal-safe", https://github.com/rust-lang/rust/issues/43146 https://github.com/rust-lang/rust/issues/43146 I'm thinking even if it works on certain OSes, it's not guaranteed, because a signal handler's context is not a thread context - or is it? E.g. the thread_local mechanism might depend on compile-time options (affecting how thread_local, whether it allocates memory on demand, and how it relocates the memory block when loading a shared library), whether it's main program or an -fPIC shared library, and the type of thread library (different ways of implementing pthreads).
- ben-schaaf 8y agoConsidering we haven't had any mmap related crashes anymore I'd say its safe to assume that its good enough for our application.
- chinhodado 8y agoAbout Sublime Merge, is there any plan for plugins?
- ben-schaaf 8y agoWe do want to support plugins at some point, but we don't have a timeline yet.
- cbsmith 8y agoI'm just going to leave this right here: http://man7.org/linux/man-pages/man2/signalfd.2.html http://man7.org/linux/man-pages/man2/signalfd.2.html
- ben-schaaf 8y agoI know about signalfd, how would it help dealing with mmap? > The signalfd mechanism can't be used to receive signals that are synchronously generated, such as the SIGSEGV
- cbsmith 8y agoSo a common way to handle synchronous signals is via a mechanism much like signalfd, where your signal handler just appends to a pipe that can be read at a convenient time.
- nneonneo 8y agoEven without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded program on POSIX is incredibly painful. Honestly, pread is just a much better solution for 90% of use cases, and it works for large files on 32-bit systems (mmap does not!). If you're doing largely sequential things, fread/fseek often work remarkably well as they handle all the caching for you. mmap tends to shine performance-wise if you need random access to a file but access certain parts of the file frequently (for example, accessing the index in a header + contents of the file), because the page cache is literally designed for this type of usage. But the performance improvement is rarely worth the technical complexity.
- fpoling 8y agoAlternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.
- quotemstr 8y agoYou can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.
- tus87 8y agoYes, but if the file is on local disk its exactly the same situation as the page file. OP seems to be trying to cope with NFS errors....but if you are using mmap on NFS you have bigger problems...
- saagarjha 8y ago> Using setjmp and longjmping from a signal handler is actually unsafe. It seems to cause undefined behaviour, especially on MacOS. Have you considered making a dispatch_source_t of type DISPATCH_SOURCE_TYPE_SIGNAL and handling all signals in a dispatch queue, instead of trying do figure out what kind of behavior is legal in a signal handler? > If a library such as Breakpad registers for Mach exception messages, and handles those, it will prevent signals from being fired. This is of course at odds with our signal handling. The only workaround we've found so far involves patching Breakpad to not handle SIGBUS. Would it be possible to install your own handler before Breakpad does?
- ben-schaaf 8y ago> Have you considered making a dispatch_source_t I think that would have been considerably more work than finding the SO answer that says you need to use sigsetjmp, and would probably still conflict with Breakpad ;) > Would it be possible to install your own handler before Breakpad does? I may be wrong, but I think you can only register one exception handler per "task" (process), so Breakpad would override ours.
- lgg 8y agoWhen you install a mach exception handler you can get the port of the previous exception handler, which you can use to forward the messages your newly installed exception handler receives. Of course (as with all raw mach APIs) it is poorly documented and error prone.
- saagarjha 8y agoInterestingly, it seems like Breakpad does a task_get/set_exception_ports dance instead of using task_swap_exception_ports: https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/master/src/client/mac/handler/exception_handler.cc#675 https://chromium.googlesource.com/breakpad/breakpad/+/refs/h.... Doesn't this break if someone registers a handler between the two lines?
- Asooka 8y agoOh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] + someListOfNumbers[3]; If headerOffset is not a multiple of 4, bad things will happen. On x86 you'll get away with it... Unless you were summing many more ints and the compiler decides to use aligned SSE loads for speed[1]. On ARM you'll get a SIGBUS just for trying to read unaligned ints (IIRC). You can fix that by wrapping your ints or whatever you're trying to read in a packed struct, but it is yet another thing to keep in mind. [1] http://pzemtsov.github.io/2016/11/06/bug-story-alignment-on-x86.html http://pzemtsov.github.io/2016/11/06/bug-story-alignment-on-...
- saagarjha 8y agoWhy not use a memcpy here and not rely on undefined behavior? I’m sure the compiler will optimize out the call anyways, especially if you enforce alignment.
- ben-schaaf 8y ago> Oh I see you didn't get to caveat 5 > On x86 you'll get away with it... Guess we'll have to wait for the proliferation of a different architecture to encounter this one :)
- 0xffff2 8y agoDifferent architectures like ARM? Plenty of code will only ever run on an x86 chip, but a non-trivial amount will run on x86 and ARM at some point. I've certainly been bitten by this before.
- ben-schaaf 8y agoSure, but my point is that both Sublime products only support x86 currently. There's likely a fair amount of other stuff that would break if we ported to ARM.
- quotemstr 8y agoThere's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said, there's nothing wrong with mmap or SIGBUS error reporting in concept. I think I've said it before, but you can think of mmap of a disk file as basically adding a temporary dedicated swap file to the system, with all the pages backed by that "swap file" already "swapped" out unless already cached. Sometimes that's exactly what you want! The author's signal handler registration difficulties come from the POSIX signal API being awful, not from the idea that catching CPU traps is somehow bad. It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. When I ran [1] by the glibc people, though, the response from them was pretty stark and unwarranted hostility toward any use of signals at all, even in perfectly legitimate scenarios, like mmap SIGBUS or certain kinds of important JVM-style pointer check [2] optimizations. It's this "we won't change anything anywhere despite our views being out of step with real user needs" attitude that's responsible for a lot of weird friction in the Unix API surface. Practically every program mmaps disk files has trouble recovering from IO errors. If it were easier for multiple components in a process to share responsibility for handling a signal, we'd more often get subtleties like this right. A better signal API will improve the user experience. Tilting at signal-shaped windmills won't. [1] https://www.facebook.com/notes/daniel-colascione/toward-shareable-posix-signals/10157129032641102/ https://www.facebook.com/notes/daniel-colascione/toward-shar... [2] Relying on CPU traps lets you perform certain checks for free. Why write "if (myptr != nullptr) var = myptr" when you can just write "var = myptr" and handle the SIGSEGV when myptr ends up being nullptr? The latter option is zero-overhead in the non-nullptr case. It's also much more complicated, but if you're a VM, and you generate millions and millions of these "if (myptr != nullptr)" checks in JITed code, the size and speed win of eliminating the check starts to justify the complexity.
- Pahr3yah 8y agoIf you roughly know your access patterns in advance you can reduce the page fault costs with madvise.
- ysleepy 8y agoReally cool talk by Bryan Cantrill about the joys of mmap in a "simple" use-case: https://m.youtube.com/watch?v=vm1GJMp0QN4# https://m.youtube.com/watch?v=vm1GJMp0QN4#
- karussell 8y agoSuch a great talk - thanks a lot! Here is the direct link as it is the last of the lightning talks: https://www.youtube.com/watch?time_continue=2462&v=vm1GJMp0QN4 https://www.youtube.com/watch?time_continue=2462&v=vm1GJMp0Q...
- atemerev 8y agoThings are so much better if you are not writing apps for general public (mine are trading-related). You can tell your few clients — make sure that the access to mmapped file is exclusive — and get away with it. And yes, mmap is the awesomest thing out there.
- CoolGuySteve 8y agoI’ve moved away from mmap for trading in favor of a separate write thread. While mmap is fast, the combination of factors that can make it decide to stall your thread while it commits to disk is difficult to manage from an operational standpoint. A slight misconfiguration is all it takes to introduce a rare and hard to notice multi-millisecond delay. Whereas with a spinlocked sized-reserved vector, the fail state performance is however long it takes to allocate more space which is on the level of microseconds. You do pay 50-150ns for that spinlock though.
- jacobush 8y agoOr better yet, mmap in a read-write-thread?
- CoolGuySteve 8y agoYa sure, go nuts on the other thread. It’s not critical to trading. I normally fprintf all sorts of time stamps, log level info, and handle formatting over there. Just make sure you put it on another core to protect your cache. But if you want mmap, you do you.
- gpderetta 8y agoWritable mappings are problematic though, because of the unpredictable pauses caused by the TLB shutdowns required to maintain coherence of the dirty flag.
- sara17372 8y agoWorld's biggest cat Even you can't believe it's size See her video https://howto105.blogspot.com/2019/04/worlds-biggest-cat.html https://howto105.blogspot.com/2019/04/worlds-biggest-cat.htm... They found world's biggest fish Even you can't believe how big fish is this That's unbelievable http://bit.ly/2FHl4pM http://bit.ly/2FHl4pM Even some goats are better than humans See that she did and they made her as mayor of city http://bit.ly/2UjTcS5 http://bit.ly/2UjTcS5 See the video a dog shopping and lots of other acts like humans http://bit.ly/2DCpo9I http://bit.ly/2DCpo9I Even you can't believe there is a hen in world's biggest animals See the video how they looks and what they do https://cutt.ly/1rOU7Q https://cutt.ly/1rOU7Q
- rwmj 8y agoAnd that's just for reading. Writing, especially if you want to be sure when the writes hit the backing file, or in what order, or if you run out of disk space, is another kettle of problems. I wonder how Multics dealt with all this, since AIUI in that system everything was effectively an mmapped file.
- mcculley 8y agoDid Multics have anything like NFS? Everything is easier if your kernel has control of the underlying device.
- BeeOnRope 8y agoWell the problematic example given in the article was NTFS where the whole filesystem can disappear, but the problem applied to local files too, eg if their size is changed by another process.
- CoolGuySteve 8y agoI feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally). Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file. I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways.
- samcday 8y agoBut now you have two completely independent code paths. Both of which will need to go through the same maturation phase that the ST folks evidently went through with mmap. And if the code needs to evolve for other reasons, potentially both of these paths will need some love too. Seems like the worst choice in a situation like this!
- CoolGuySteve 8y agoThe read() API is so basic, mainly because it is synchronous and has clear return values, that I have a hard time believing it would present nearly as many issues.
- theoh 8y agoButler Lampson talks about the general concept of an overlay in operating system design. It's a slightly different case, on the surface, but the possible application in this case is that if mmap fails, you probably should have a fallback code path that tries to access the file normally. That's obviously not going to work for all applications. But it's something you consciously have to do anyway if mmap may not be available (e.g. when working across the network). The OS design should make it easy to move up and down in this hierarchy of access methods, but the application programmer still needs to know which "level" of access they have to the file. There's work to be done here in improving the design of the OS interface. That would lift part of the the burden of maintaining multiple paths in the application code. https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s https://www.youtube.com/watch?v=TRLJ6XdmgnA&t=11m35s slides: https://bwlampson.site/Slides/Hints%20and%20principles%20(HLF%202015).pdf https://bwlampson.site/Slides/Hints%20and%20principles%20(HL... see also: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-826-principles-of-computer-systems-spring-2002/lecture-notes/30.pdf https://ocw.mit.edu/courses/electrical-engineering-and-compu...
- emersion 8y agoNote that some BSDs have a MAP_ZERO flag which makes invalid accesses read zeros instead of triggering SIGBUS.
- loeg 8y agoWhich ones? I don't see it in manual pages for any of the ones I know about (Free, Net, Open, Dragonfly).
- snarfy 8y agoNow I'm curious how Vim deals with large files. I'm assuming the went the pread route.
- rocqua 8y agoVim needs to do a lot more special stuff, because it has to handle random insertions and deletions. Doing those things to a file (mmaped or otherwise) requires moving all data after the edit. If I recall correctly, vim/vi uses a linked list of 'chunks' that is dynamically merged. For long files I would expect some form of lazy loading of chunks.
- dullgiulio 8y agoThat's right. The key data structure is the rope: https://en.wikipedia.org/wiki/Rope_(data_structure) https://en.wikipedia.org/wiki/Rope_(data_structure)
- deleted 8y ago[deleted]
- snarfy 8y agoNice! I thought xi was the only editor that used ropes.
- raphlinus 8y agoThe other big problem with mmap is what happens when your file changes out from under you. This seems to be mostly for git packfiles, which I think can be treated as immutable by convention, but that's not strongly enforced anywhere. For reading, eg, program source files, I think mmap is hugely problematic. I've been arguing for a long time that operating systems should provide read only snapshots of files as a primitive, but that's a pretty big ask; it's especially hard to do when the file system is network-mounted. There are a couple of copy-on-write filesystems on Linux (btrfs and ZFS if memory serves) which can do this locally, but it's not mainstream.
- avar 8y agoYou can assume that git packfiles are immutable. They may disappear from under you as a repack happens, but they will not be changed. They even have a name like pack-<SHA-1>.pack where that <SHA-1> is a SHA-1 of the contents of the pack (minus the last 20 bytes, the checksum SHA-1 is also part of the pack itself).
- loeg 8y agoTrue for this workload, iff you assume the filesystem and/or media protects you from corruption (it probably doesn't). I would guess OP is commenting on mmap IO in general, rather than TFA's specific git use case.
- theoh 8y agoSounds like https://en.wikipedia.org/wiki/Snapshot_isolation https://en.wikipedia.org/wiki/Snapshot_isolation.
- klodolph 8y agoThe mistake here is using longjmp / siglongjmp. This is a possible way to handle SIGBUS, but in practice it will be intractable in larger programs written in C or C++. The compiler is generally free to move loads and stores around, and you might be completely blindsided by how the compiler has reordered your memory operations once you add side effects to one of the operations. Theoretically, if accessing a memory location can siglongjmp out then at least the memory location should be volatile. A better way to handle SIGBUS is to just map zeroes over the offending pages using MAP_FIXED and then setting a flag. After every operation that works on a file, you check the flag.
- planteen 8y agoThis sounds interesting. Do you know of an example that does this? So if I have multiple threads reading the same mmap'd file, I use si_addr in the signal handler to know which page to call MAP_FIXED on?
- klodolph 8y agoI don’t know an open-source example off the top of my head. But that’s the gist of it… you keep an array somewhere with all the address ranges mapped. If you get SIGBUS, find the corresponding map, replace it with zeroes, and mark it as having an error. If there is no corresponding map, uninstall the signal handler and return—the thread will SIGBUS immediately after return and the default action will kill the process.
- tobias3 8y agoI solved the problem this way recently as well. This solution would also work with multiple threads (you can probably set the flag in thread local storage, though I haven't done that yet - currently using a global map with thread id as key). Though still not with multiple different signal handlers. That said, if I could do it over I wouldn't use mmap again. Especially since io_uring is around the corner (on Linux) that allows zero-copy reads/writes with no syscall overhead.
- wahern 8y ago
- rocqua 8y agoHonestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of library precludes you from using certain features. Combined, this means that the use of signal handlers is simply toxic. Which makes them an anti-feature. It feels to me like having the restrictions of kernel-space, with all of the downsides of user-space. Are there any plans in linux to replace signals?
- qiqitori 8y agoNormal libraries should never register signal handlers. Google Breakpad is a crash-reporting system and as such requires signal handling to function. Also since we're on the topic, here's a nice vulnerability caused by bad signal handling: https://news.ycombinator.com/item?id=16753013 https://news.ycombinator.com/item?id=16753013 Pretty sure there are no plans to replace signals, but maybe there are libraries that make signal handling easier?
- loeg 8y ago> Pretty sure there are no plans to replace signals, but maybe there are libraries that make signal handling easier? Non-portable, but sigprocmask() SIG_BLOCK plus signalfd() (Linux) or kqueue() EVFILT_SIGNAL (BSDs). Neither is a good solution for handling mmap SIGBUSes, but they're generally good for handling most signals (USRn, TERM, HUP, CHLD, etc) more similarly to other kinds of events.
- wahern 8y agosignalfd does NOT work the same as kqueue EVFILT_SIGNAL. On Linux signalfd basically behaves the same as installing a signal handler--only a single signalfd object will receive a signal, so whether installing a signal handler or a signalfd listener you're changing behavior globally. It's even messier than that, because signalfd only works if there are no other signal handlers installed. By contrast, with kqueue all EVFILT_SIGNAL listeners will be notified of a signal, even if the signal was also delivered to a handler. Big difference. This is one among many reasons why Linux's event syscalls are widely considered inferior to BSD kqueue. It's ridiculous considering that kqueue not only predated epoll by several years and signalfd by nearly a decade, with ample use cases, it was well documented in a paper that described the rationale for all the semantics. Somehow the authors of epoll, signalfd, etc couldn't even bothered do to basic research; they just spitballed semantics without having much real-world experience about what was needed and most useful.
- z3t4 8y agoOr you could stream parse the file. eg. parse it one chunk at a time.
- saagarjha 8y agoThat’s what they were doing before?
- 0xffff2 8y agoBefore they were reading the whole file into memory and parsing it. Now, they are using mmap to read parts of the file into memory when they are used. If you're only doing a single linear pass through a file there is a third option; you could read part of the file in a single chunk at a time, parse that chunk, then read the next part of the file into the same chunk of memory. I don't know enough about the process being discussed here to know whether it needs to do lookups at random offsets in the file, but iff it doesn't then the chunked read solution could be a simpler way to reduce memory usage.
- gok 8y agoIt would be really nice if there was a POSIX equivalent to Foundation's NSDataReadingMappedIfSafe, which uses mmap() when the file isn't backed by NFS and falls back to read() when unsafe or not worth it.
- thomasjudge 8y agoIs there a way for a process to know what kind of filesystem something lives on, or is this just a developer-provided clue?
- jfhufl 8y agogetmntent(3)
- beat 8y agoThe first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be cleared. My first being original-idea development was rewriting with mmap so I could use an arbitrary (and programmable) number of buffers, tuning blocking against memory consumption, with logging to track performance for tuning. It was very cool. Worked great! Until it went to production. In production, it crashed every time it ran, shortly after starting up. Since we were doing seasonal production, backing out my change also backed out other necessary changes. It was very embarrassing and frustrating. Worse, I could not replicate the problem in testing! It only happened on the prod servers. And, as a wet behind the ears junior programmer, everyone assumed I'd just screwed up and was too dumb to understand how. So I wrote a test program, divorced from our regular code, to test mmap() itself. Turned out that it ran fine on dev/test servers, but on the prod servers, it would randomly overwrite 1k memory pages with nulls. Yeah. Once i convinced the senior engineers and my manager, I got to report an OS bug to IBM. Who were like "What's wrong with your code, really?" I wound up sending them my C test code and the compiled executable, along with results. It turned out the bug was caused by the order in which OS patches had been applied on the servers. One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. And if a junior programmer came to me today reporting a bug in OS memory management, my first response would be "What's wrong with your code, really?"
- ajross 8y agoYeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client system lasting a week between some failure or voodoo reboot was laughable. Now... I dunno, when was the last time I bounced my phone? Genuinely can't remember.
- cryptonector 8y agoThe problem here was wanting to parse a large file at all. You need to be able to do a) online parsing (so you don't need to read the whole file into memory first), b) stream parse (so you don't need to build a parsed representation of the whole thing before you can do anything).
- dicroce 8y agoI've successfully used mmap() a few times in the last few years... Luckily for me the use cases I've had weren't really subject to the same problems I've since read about here and elsewhere: 1) I'm always mmap()ing the whole file (and the files are power of 2 sized). 2) The files I'm mapping are stored on file systems I control (and so are never on NFS). 3) In one case, my use of mmap() is limited to read only.
- loeg 8y agoI'd go a step further and broadly recommend not using mmap to access files, unless there's a really good overriding reason. (E.g., if the file is some sort of special virtual device/filesystem that by nature cannot have errors.) Mmap is not good for writing to files — pages may be persisted to disk in arbitrary order, which makes it harder for filesystems to coalesce adjacent writes into fewer larger (and faster) IOs. (This is still an issue on SSDs, although not as bad as on HDDs.) This is a performance issue and can result in bad file layout (making future reads slower). As this article describes, the mmap() model sort of assumes no errors happen. This breaks when files are truncated, even on local POSIX filesystems, and you get SIGBUS. It can also break if files disappear, such as a failing media or removed USB stick, or network filesystem. It doesn't mesh well with distributed filesystems either, for obvious reasons. If a page is to be writeable, you must take an exclusive data lock on that page's region across your distributed filesystem (and read access requires a shared data lock, to prevent corruption from concurrent writers). What if you lose quorum / availability? TFA's trick to use thread-specific longjmps around specific virtual memory accesses probably works out ok on POSIX platforms[1] but it requires wrapping all of your mmap'd regions carefully. You can't just cast portions to a struct and access directly, except in small critical regions protected by the sigsetjmp. And as they point out, SIGBUS is global — it can conflict with error catchers (mentioned in TFA) but also can be raised for reasons other than mmap IO failure, such as attempting to access a non-canonical virtual address, and thus a long-lived global handler may mask other bugs. (Also, if you mmap many files and install a single long-lived handler in a multi-threaded program, it can become difficult to determine which file-access raised the signal.) rtorrent, for example, used to have a ton of reports of SIGBUS due to mmap'd file access failure. I don't know if they've addressed that in some way (perhaps by simply masking SIBGUS) or continue to ignore it. TFA claims pread was about 2/3 as fast as mmap'd access; some slightly clever application-specific use of caching, prefetch, or larger IOs might help eliminate that gap by reducing syscall overhead and/or disk wait. The best thing about pread/pwrite is they return have error reporting in the interface, and you can actually check that your IO did what you wanted. [1]: http://man7.org/linux/man-pages/man7/signal-safety.7.html http://man7.org/linux/man-pages/man7/signal-safety.7.html : If a signal handler interrupts the execution of an unsafe function, and the handler terminates via a call to longjmp(3) or siglongjmp(3) and the program subsequently calls an unsafe function, then the behavior of the program is undefined.
- adamc 8y agoReally well-written, easy to follow piece.
- strictfp 8y agoTRWTF is Breakpad, trying to deal with the process crashing from within the process itself. Even requires PTRACE to work.