9 ms·

Two new ways to read a file quickly

181 points by chmaynard 7y ago

adrianmonk 7y ago
I'm not a kernel hacker, but the Linux kernel now has an in-kernel VM (BPF), doesn't it? As I understand, it started with letting you load programs into the kernel to filter network packets, but now it allows several other things. So, readfile() combines 3 syscalls into 1 by manually implementing a new system call. Would another approach be to make these same calls (openat(), read(), and close()) and some others available to BPF programs? The idea is to provide a more general mechanism, rather than solving this one case. Obviously, the kernel change would need to be feasible, and the result would need to be safe and fast. Then a user mode program could effectively just write its own readfile(), or some other variation in case that exact sequence of calls isn't what it needs. (An even more out-there idea is to have the kernel auto-detect situations like this, examine the native code, then after verifying that it's simple and harmless, just move a whole chunk of code or tight loop into the kernel. In effect, it would dynamically generate a new syscall.)
- dmitrygr 7y ago
  > verifying that it's simple and harmless I look forward to your PhD thesis disproving Rice's theorem
  - klyrs 7y ago
    Funny thing about Rice's theorem. It says that semantic properties (such as halting) are undecidable in general. That says nothing about specific programs. It's easy to prove that certain programs halt; it's easy to prove that others never halt. In fact, there are whole classes of programs for which it is trivial to prove halting (e.g. bf programs that do not contain [ or ]), or non-halting (e.g. bf programs that contain a single + and the remaining instructions are [ or ]). One need not waste their time proving an impossible generality along the path to proving a specific possibility.
    - deleted 7y ago
      [deleted]
    - dTal 7y ago
      Yes, but if you enforce that the program be decidable, as part of a dynamic kernel API, then you are no longer offering a Turing-complete language. Are there so many use cases that would benefit from this nonetheless, that it wouldn't be better to just hard-code them?
      - klyrs 7y ago
        That's not how I (generously) read GP's suggestion. It could be worthwhile to implement heuristics that can be used to prove certain classes safe; and those could be lifted to a "new syscall". If the heuristics give a non-result, gracefully fall back to not pulling userspace crap into the kernel. Not saying I'd be interested in doing this work, but there's a vast middle ground between "the halting problem is undecideable" and "we can never know if while(1); will halt"
        adrianmonk 7y ago
        Exactly. That's what I meant. I tried to hint at that by saying you'd verify that it's simple and harmless, meaning you solve the simple cases and leave the others as-is. But then I also described it as a more general mechanism, and I guess people ran with that and kinda understandably interpreted it to mean a fully general mechanism. About the halting problem, I'm not necessarily convinced that prevents this from being done. At least not in theory. (This is getting impractical, but it's interesting to think about.) Let's say I have an infinite loop in user space. I then move the infinite loop to kernel space. What specific harm have I caused? I already had an infinite loop; now I still have one. Though there might be some consequences other than burning CPU cycles. I think "kill -9" may not take effect until system calls return. Also, there's such a thing as scheduler priorities and CPU quotas, and those might be impacted. Of course it would always be possible to just exclude anything with a loop from what you try to move into the kernel. Or it might also be possible to handle infinite loops with some kind of complicated thing involving a timeout and restarting system calls.
  - dTal 7y ago
    The snark was perhaps a touch excessive, but this is indeed an excellent reason why the radical suggestion proffered shall remain radical. (isn't this basically just a kernel module?)
    - im3w1l 7y ago
      It's very easy to misinterpret what a theorem means. And doing this can derail a conversation for decades. I remember two such cases. The theorem that two hidden layer neural networks can approximate any function - Common interpretation: two layers are enough. And a proof of well I don't recall the details... being interpreted as imaging being limited by wavelength. Then came super resolution which worked around it by not using some of the implicit assumptions of the common interpretation. So I think it's justified to take issue.
- rjeli 7y ago
  One possibility is integrating io_uring with bpf: https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/
- Elv13 7y ago
  eBPF isn't safe to load without root privilege. This means no containers and no users process. From what I saw so far, there is fear that letting Turing complete programs in the kernel would make some future CPU/hardware vulnerabilities more dangerous. While the compiler makes sure the program return, it cannot make sure it isn't nefarious. ref: https://lwn.net/Articles/808048/ https://lwn.net/Articles/808048/
  - loeg 7y ago
    One can easily imagine a restricted subset of eBPF that is not turing complete and could be safe for unprivileged users to use.
    - amscanne 7y ago
      That’s what BPF is already. It’s not Turing complete. Unbounded loops and backward branching is not allowed. The problem isn’t Turing completeness. It’s that there’s are too many micro-architectural side channels and unknowns.
      - monocasa 7y ago
        They currently go out of their way to remove microarchitectural side channels. Yes, there's almost certainly others, but it's pretty clear that the end goal is to allow non root creation of bpf programs.
        nickodell 7y ago
        Is it really possible to remove the side channels? Even if you formally prove that it does not access memory outside of authorized regions, that doesn't prevent it from speculatively accessing unauthorized memory. We've seen Spectre exploits through sandboxed Javascript. Unless you separated the eBPF VM address space from the kernel address space, I suspect you would have the exact same problems that browsers have had. But if you did put them in separate address spaces, then why bother with eBPF? You could run a normal userspace program.
- hinkley 7y ago
  I remember a paper decades ago now about a way to reduce RPC call round-tripping by sending something akin to promises back and forth, so you could pass the result of one remote call to another one without transferring it across the network. Kinda think something similar might be useful here. What typically ends up happening in “services” (or any distributed system) is that you start creating convenience APIs to speed things up that are mostly just combinations of other calls, and you stay with the ones that seem like they might be the most useful, so you don’t slam into combinatorics face first. Given the amount of talk I’ve heard about system call overhead lately, it sounds like the magnitude of the delays is creeping up again (especially post-Meltdown).
  - exrook 7y ago
    I don't know if this is what you were thinking of, but capnproto[0] does exactly that with a promise like interface. Actually this may be the paper you were referencing, from the bottom of the capnproto page: > Cap’n Proto’s RPC protocol is based heavily on CapTP[1], the distributed capability protocol used by the E programming language[2]. Lots of useful material for understanding capabilities can be found at those links. [0] https://capnproto.org/rpc.html https://capnproto.org/rpc.html [1] http://www.erights.org/elib/distrib/captp/index.html http://www.erights.org/elib/distrib/captp/index.html [2] http://www.erights.org/index.html http://www.erights.org/index.html
    - hinkley 7y ago
      Very interesting, but if memory serves, Microsoft had this idea maybe in ‘96. Though patent expiry-wise, the timing of cap’n proto might make a bit of sense.
      - barrkel 7y ago
        E is from '97 and built on top of ideas implemented before that with extensions to Java; not sure of the date, but could easily predate '96.
        7y ago
- fragmede 7y ago
  BPF is not run inside of a virtual machine (the same way that Java is). It is not safe to load untrusted code into it.
ape4 7y ago
Maybe the new readfile() could be used to implement php's readfile().
- 131hn 7y ago
  PHP readfile is all about « streaming » GB size files to stdout, not so much returning a whole file in a single buffer (but file_get_contents is and might, indeed, benefit from the new readfile syscall )
  - ape4 7y ago
    Thanks, that was what I meant.
userbinator 7y ago
The idea of making file reads stateless is interesting. I wonder if it's atomic too? I think that might be very useful to have.
- loeg 7y ago
  It does sort of raise the question of how interrupts are handled. Short reads / EINTR and suddenly this mechanism is less efficient than open+read+close. Depending on how SA_RESTART is implemented, it may not help. Do consumers have to mask signals around readfile(), and if so, are we saving any system calls?
mayoff 7y ago
This article says `readfile` would return the number of bytes read. It seems like it would be better for it to return the number of bytes in the file, but only write at most `bufsize` bytes into the buffer. That way, if the buffer isn't big enough, you can allocate a correctly-size buffer without needing to `stat` the file. This is how `snprintf` works.
- a1369209993 7y ago
  That wouldn't work well because you'd need to re-open and seek (or re-read) before continuing. Ideally readfile would take and return a filefd (default -1); if present, skip the open and just read (closing on EOF). This also fixes the EINTR/SA_RESTART issue loeg[0] raised. From a C perspective, it does have a problem of needing to return multiple values (both a fd and a bytecount), though. 0: https://news.ycombinator.com/item?id=22509329 https://news.ycombinator.com/item?id=22509329
- im3w1l 7y ago
  I think some "files" that are actually devices or pipes and stuff, for them the number of bytes in the file doesn't make sense, while bytes read does make sense. That may be why.
correct_horse 7y ago
If the problem is slow switching between user space and kernel space, why not have a syscall that allows you to batch process multiple syscalls? It might work like this: user space allocates a buffer, fills it with syscall numbers and arguments, then passes it to the kernel. The kernel executes each syscall in series, placing the return values in the array originally containing arguments. This syscall would not return to user space until all syscalls in the buffer are processed. Does this idea have any potential?
- nominatronic 7y ago
  This is basically the idea of io_uring [0], although that mechanism is specific to certain I/O operations rather than being a generic queuing system for syscalls. [0] https://lwn.net/Articles/776703/ https://lwn.net/Articles/776703/
  - wahern 7y ago
    Before io_uring there were many failed proposals for a general purpose, asynchronous interface to wrap blocking syscalls. One of these, "syslets", was literally a mechanism for batching arbitrary syscalls: https://lwn.net/Articles/221887/ https://lwn.net/Articles/221887/ > Syslets are small, simple, lightweight programs (consisting of system-calls, 'atoms') that the kernel can execute autonomously (and, not the least, asynchronously), without having to exit back into user-space. .... > Syslets consist of 'syslet atoms', where each atom represents a single system-call. These atoms can be chained to each other: serially, in branches or in loops.
    - correct_horse 7y ago
      Thanks for sharing. It's encouraging that something at least resembling what I came up with was considered.
    - anonymousDan 7y ago
      Any thoughts on why they failed?
longtermd 7y ago
TLDR: speed-up happens as common-combinations of syscalls (open, read, close) are combined to one single syscall (read file contents). It violates the unix philosophy of "one call for one specific thing", of course, but massively speeds up performance in practical applications. In fact, you could get a LOT of speedup at Linux/Microsoft/... (the OS level) by figuring out highly common combinations of syscalls typically all called in a row with mostly the same arguments, and combining them into one single syscall.
- wahern 7y ago
  Once upon a time Linux solved the "syscalls are slow" problem by making syscalls fast, rather than introducing convoluted APIs as a workaround, like other operating systems did. Perhaps one day we can get back to that model, or maybe now that Linux has gone corporate those days are gone. Even when you can make something faster with a simple, straight-forward solution, there's always that one corporate sponsor who doesn't benefit. Before Linux would just tell them sod-off.
  - KarlKemp 7y ago
    that’s (at best) a gross oversimplification of the process, idealizing the past while ignoring how expectation for the kernel have increased (I e. security & privacy), all colored by a layer of bad faith that, to be frank, seems to originate with some non-specific grievances.
    - wahern 7y ago
      I'm simply channeling Linus Torvalds. Here's a sample of a rant/boast from 2000 where Linus defends the "heavy-weight" threading approach against claims that a proper user-space threading architecture could provide better performance by requiring fewer syscalls and less memory: > Yes, you could do it differently. In fact, other OS's _do_ do it differently. But take a look at our process timings (ignoring wild claims by disgruntled SCO employees that can probably be filed under "L" for "Lies, Outright"). Think about WHY our system call latency beats everybody else on the planet. Think about WHY Linux is fast. It's because it's designed right. Source: https://lkml.org/lkml/2000/8/25/80 https://lkml.org/lkml/2000/8/25/80 The same architectural approach was taken with fork: Linux optimized the heck out of fork to the point where fork on Linux was faster than thread creation on Windows. On multiple occasions Linus has argued (and proven), that optimizing the simple but "heavy-weight" approach can reap dividends at least as great as more complex, more flexible architectures. Of course, that was then and this is now. And many things have changed, including the success of Linux. I'm just suggesting, or perhaps implying, that there's more eagerness to entertain the replacement of traditional syscall semantics and other subsystems with more complex frameworks than there once used to be. In the context of Spectre, arguably it would have been easier for a younger Linus (and younger Linux) to refuse to rearchitect syscalls (with the concomitant additional kernel and semantic complexity), to tell everyone to sit tight and wait for AMD and Intel to come up with mitigating hardware fixes, and in the intervening years just take the performance hit.
cryptonector 7y ago
> the cost of reading a lot of little files would be too high The `readfile()` system call will reduce that by some factor, though probably not as high as a factor of one third (it replaces three system calls with two, assuming it turns out to be useful). But getting all the data in one call will be even faster. Incidentally, a typical read-all-of-this-file function in user-land allocates a buffer of the right size after doing an `open()` and then `fstat()` to find the file's size. Whereas a system call to read a file in one go really can't do that unless it establishes a memory mapping (which is... expensive). So I rather question the utility of this thing. Also, since one might not know what size buffer to give it, it might as well have an option to return an open FD. Or the app should have to `stat()` then `readfile()`, thus replacing three syscalls (`open()`, `fstat()`, `close()` with two -- bummer. Or the caller can `readfile()` repeatedly if the buffer given was not large enough (oof). No, I'm pretty sure this `readfile()` can't possibly make reading lots of tiny files fast enough to not want a better solution for getting mounted filesystem metadata out of the kernel.
- ploxiln 7y ago
  One of the cases posited is `ps` or `top` which (on linux) read /proc/[0-9]+/stat and /proc/[0-9]+/cmdline, and in these cases it knows the the max it will want to read (which is pretty small, maybe 512 bytes at most). So this can make `ps` or `top` 3x faster / more efficient.
  - bicolao 7y ago
    There must be other use cases besides ps and top to justify new syscalls right? ps and top are not exactly slow to even make these optimizations visible.
- deleted 7y ago
  [deleted]
ncmncm 7y ago
This sort of thing leads me to think Linux is well down the path to irrelevance. Io_uring, for all its baroque elaboration, saves millions or billions of system calls. Bloating the kernel to save two of three system calls on filesystem operations seems nothing short of foolish. I would welcome being proved wrong.
yxhuvud 7y ago
O_SPECIFIED seems like a strange solution to me. Sure, I see why the kernel people wouldn't mind having people specify their own fd numbers, but doing the accounting necessary for that (to avoid conflicts) in userspace doesn't seem nice. Especially when there will be libraries that also may make use of the functionality. Is there even any way to see what file descriptors that are open without doing a syscall?
- StrangeDoctor 7y ago
  It looks like it’s intended to be used with a batched creation of file descriptors that are known to allocated but not yet used (to cut down on total syscalls?) https://lore.kernel.org/io-uring/20200211223235.GA25104@localhost/ https://lore.kernel.org/io-uring/20200211223235.GA25104@loca... At least I think I’m reading this correctly.
shiblukhan 7y ago
Hmm, how is one actually supposed to use O_SPECIFICFD in real life though? I mean, picking some specific FD nr can work in trvial, single threaded programs maybe, but how is that supposed to work on typical generic userspace code that has many threads and many libraries/subsystems all running in the same address space (and fd space) that want to take possession of some fd? I mean are apps supposed to block fd ranges ahead of time, by dup()ing /dev/null a couple of times, or how is that supposed to work? not getting this...
willvarfar 7y ago
Are there any mainstream databases etc adopting io_uring yet? I first heard about io_uring here on HN and a rust DB lib called Sled was used as an example. But when do we get fast MySQL etc? I recall a paper from 2012 or so that implemented a PoC syscall buffer so a bunch of syscalls could be scheduled and executed sequentially with just one syscall. They claimed 40%+ performance improvements in various programs, including MySQL. This was of course before meltdown and other cache mitigations were a thing. So programs, including MySQL and Postgres (hey, I know something about various DB storage engines), would benefit massively from scheduled sequential syscalls (easy to adopt, big win) and true async io (io_uring, harder to adopt, even bigger win).
- antpls 7y ago
  At that point, wouldn't it be simpler to run mysql or postgresql on their own bare metal VM with a specialized uni-kernel that only runs the db process very efficiently (without notion of kernel space and user space and without context switching) ?
- zuzun 7y ago
  RocksDB supports it for reading multiple chunks from a file.
- anarazel 7y ago
  I am working on optional io_uring support for PostgreSQL. Hope to get that stable for PG 14 (PG 13 enters feature freeze soon).
tyingq 7y ago
"On its face, readfile() adds nothing new; an application can obtain the same result with calls to openat(), read(), and close(). But it reduces the number of system calls required from three to one" A little surprised they didn't also roll in an optional lseek() offset in case you wanted a specific area of the file. Then it would roll 4 syscalls into 1. The somewhat similar sendfile() accepts an offset.
- masklinn 7y ago
  If you look at the thread, readfile() would be pretty specifically for extremely short files e.g. procfs and sysfs. It would not be intended as a convenience API for slurping an entire file but as a performance API for very short reads (straight open(); read(); close() where the syscall overhead is enormous). The implementation(s) by Greg Kroah-Hartman don't even bother looping until read returns empty or the buffer is full.
  - tyingq 7y ago
    Sure, but it would be simple to add, and perhaps then useful for lots of small reads from bigger files. Or things like /proc/pid/mem.
nibbula 7y ago
I'm not a kernel hacker either, and even though a readfile system call might speed some things up, one of the reasons it's desired is really wallpapering over a deeper, much harder issue, that of it being a terrible hack to write the 'ps' command, and the more general issue of returning structured data from a unix-like kernel. Also it seems like a bad design not to know how big to make the buffer, which means if you want to reliably get the whole file you either have to put it in a loop reallocating the buffer, or do a stat system call before be lucky with the race condition. To be fair, this has been an issue since the first unix kernels, and even though /proc and sysctl are an improvement over grepping kernel memory from user space (the incredibly hackish way old ps worked), in my opinion it's still a big mess in various ways. Just in case you aren't aware, linux ps/top/etc. has to open possibly hundreds of fake files, parsing from text a bunch of stuff which may or may not be there or valid or the same data type depending on your kernel configuration or version. Just because you could write 'ps' with awk, doesn't make it good. I wouldn't object at all to the /proc interface, as a way of enabling simple tools, if there was also a decent function call interface. Modern BSDs generally use sysctl, which although being better in theory, that you don't have the overhead of useless translating numbers back and forth from text and looking for space characters, it still has the problem that it's very dependent on subtle changes in the C data structures which can easily happen between version and architectures. But it also has the terrible drawback of not knowing how big the buffer should be and therefore having to be in an allocation fail loop, probably exactly when you don't want it to happen, when you system is overloaded with way too many processes. I really don't see why one couldn't pass a memory allocator function, which could be called in the same manner as a signal handler. I don't know how windows stuff works inside, but to get process information you ask for a snapshot of the system from which you can return a set of handles to processes, which you can get information from. In other words, it allocates the appropriate things for you and has well defined set of information which you can query, all with a relatively simple function call interface from C. I wrote a 'ps' that works on linux/bsd/macos/windows, and even though I'm not at all fan of windows, the windows kernel does it better than all the unix kernels. It's actually not that hard to make an acceptable C interface to to say process information, if you ignore the problems of data structure variance. But ignoring it is really the root of the issues, so it seems fairly pointless. sysctl sort of tries to have a metadata system, but it doesn't really address the data type problem well, and can end up using C structs from the kernel with same variance problems. For example task_struct in linux or struct proc on macos is kind of a big mess, but very important. You wouldn't want to restrict it from changing in any way, but you do want to get some of that data to users. Making a good C metadata interface is complex, and although it's been done many times, it's quite tricky to do well, and you would really want a kernel interface to be done very well. I like to imagine that there could be something less bloated than gtk gobject, but more featurefull than sysctl. Unfortunately when I look at code involved I start feeling like C isn't really a good language for writing an operating system. But I still think that a well designed structured meta data system would be overall less overhead than /proc and could even achieve better speed and reliability.
trgfs85 7y ago
A lot of comments in this topic seem to forget that syscalls don't always succeed. Interupts are the biggest issue. If an interrupt occurs and you read something you get a partial read. If you open something you tend to get EINTER(very system dependent.) You can't really queue those. And those shouldn't be the OS's job to deal with.

181 points by chmaynard 7y ago

I'm not a kernel hacker, but the Linux kernel now has an in-kernel VM (BPF), doesn't it? As I understand, it started with letting you load programs into the kernel to filter network packets, but now it allows several other things. So, readfile() combines 3 syscalls into 1 by manually implementing a new system call. Would another approach be to make these same calls (openat(), read(), and close()) and some others available to BPF programs? The idea is to provide a more general mechanism, rather than solving this one case. Obviously, the kernel change would need to be feasible, and the result would need to be safe and fast. Then a user mode program could effectively just write its own readfile(), or some other variation in case that exact sequence of calls isn't what it needs. (An even more out-there idea is to have the kernel auto-detect situations like this, examine the native code, then after verifying that it's simple and harmless, just move a whole chunk of code or tight loop into the kernel. In effect, it would dynamically generate a new syscall.)

dmitrygr 7y ago
> verifying that it's simple and harmless I look forward to your PhD thesis disproving Rice's theorem
- klyrs 7y ago
  Funny thing about Rice's theorem. It says that semantic properties (such as halting) are undecidable in general. That says nothing about specific programs. It's easy to prove that certain programs halt; it's easy to prove that others never halt. In fact, there are whole classes of programs for which it is trivial to prove halting (e.g. bf programs that do not contain [ or ]), or non-halting (e.g. bf programs that contain a single + and the remaining instructions are [ or ]). One need not waste their time proving an impossible generality along the path to proving a specific possibility.
  - deleted 7y ago
    [deleted]
  - dTal 7y ago
    Yes, but if you enforce that the program be decidable, as part of a dynamic kernel API, then you are no longer offering a Turing-complete language. Are there so many use cases that would benefit from this nonetheless, that it wouldn't be better to just hard-code them?
    - klyrs 7y ago
      That's not how I (generously) read GP's suggestion. It could be worthwhile to implement heuristics that can be used to prove certain classes safe; and those could be lifted to a "new syscall". If the heuristics give a non-result, gracefully fall back to not pulling userspace crap into the kernel. Not saying I'd be interested in doing this work, but there's a vast middle ground between "the halting problem is undecideable" and "we can never know if while(1); will halt"
      - adrianmonk 7y ago
        Exactly. That's what I meant. I tried to hint at that by saying you'd verify that it's simple and harmless, meaning you solve the simple cases and leave the others as-is. But then I also described it as a more general mechanism, and I guess people ran with that and kinda understandably interpreted it to mean a fully general mechanism. About the halting problem, I'm not necessarily convinced that prevents this from being done. At least not in theory. (This is getting impractical, but it's interesting to think about.) Let's say I have an infinite loop in user space. I then move the infinite loop to kernel space. What specific harm have I caused? I already had an infinite loop; now I still have one. Though there might be some consequences other than burning CPU cycles. I think "kill -9" may not take effect until system calls return. Also, there's such a thing as scheduler priorities and CPU quotas, and those might be impacted. Of course it would always be possible to just exclude anything with a loop from what you try to move into the kernel. Or it might also be possible to handle infinite loops with some kind of complicated thing involving a timeout and restarting system calls.
- dTal 7y ago
  The snark was perhaps a touch excessive, but this is indeed an excellent reason why the radical suggestion proffered shall remain radical. (isn't this basically just a kernel module?)
  - im3w1l 7y ago
    It's very easy to misinterpret what a theorem means. And doing this can derail a conversation for decades. I remember two such cases. The theorem that two hidden layer neural networks can approximate any function - Common interpretation: two layers are enough. And a proof of well I don't recall the details... being interpreted as imaging being limited by wavelength. Then came super resolution which worked around it by not using some of the implicit assumptions of the common interpretation. So I think it's justified to take issue.
rjeli 7y ago
One possibility is integrating io_uring with bpf: https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/
Elv13 7y ago
eBPF isn't safe to load without root privilege. This means no containers and no users process. From what I saw so far, there is fear that letting Turing complete programs in the kernel would make some future CPU/hardware vulnerabilities more dangerous. While the compiler makes sure the program return, it cannot make sure it isn't nefarious. ref: https://lwn.net/Articles/808048/ https://lwn.net/Articles/808048/
- loeg 7y ago
  One can easily imagine a restricted subset of eBPF that is not turing complete and could be safe for unprivileged users to use.
  - amscanne 7y ago
    That’s what BPF is already. It’s not Turing complete. Unbounded loops and backward branching is not allowed. The problem isn’t Turing completeness. It’s that there’s are too many micro-architectural side channels and unknowns.
    - monocasa 7y ago
      They currently go out of their way to remove microarchitectural side channels. Yes, there's almost certainly others, but it's pretty clear that the end goal is to allow non root creation of bpf programs.
      - nickodell 7y ago
        Is it really possible to remove the side channels? Even if you formally prove that it does not access memory outside of authorized regions, that doesn't prevent it from speculatively accessing unauthorized memory. We've seen Spectre exploits through sandboxed Javascript. Unless you separated the eBPF VM address space from the kernel address space, I suspect you would have the exact same problems that browsers have had. But if you did put them in separate address spaces, then why bother with eBPF? You could run a normal userspace program.
hinkley 7y ago
I remember a paper decades ago now about a way to reduce RPC call round-tripping by sending something akin to promises back and forth, so you could pass the result of one remote call to another one without transferring it across the network. Kinda think something similar might be useful here. What typically ends up happening in “services” (or any distributed system) is that you start creating convenience APIs to speed things up that are mostly just combinations of other calls, and you stay with the ones that seem like they might be the most useful, so you don’t slam into combinatorics face first. Given the amount of talk I’ve heard about system call overhead lately, it sounds like the magnitude of the delays is creeping up again (especially post-Meltdown).
- exrook 7y ago
  I don't know if this is what you were thinking of, but capnproto[0] does exactly that with a promise like interface. Actually this may be the paper you were referencing, from the bottom of the capnproto page: > Cap’n Proto’s RPC protocol is based heavily on CapTP[1], the distributed capability protocol used by the E programming language[2]. Lots of useful material for understanding capabilities can be found at those links. [0] https://capnproto.org/rpc.html https://capnproto.org/rpc.html [1] http://www.erights.org/elib/distrib/captp/index.html http://www.erights.org/elib/distrib/captp/index.html [2] http://www.erights.org/index.html http://www.erights.org/index.html
  - hinkley 7y ago
    Very interesting, but if memory serves, Microsoft had this idea maybe in ‘96. Though patent expiry-wise, the timing of cap’n proto might make a bit of sense.
    - barrkel 7y ago
      E is from '97 and built on top of ideas implemented before that with extensions to Java; not sure of the date, but could easily predate '96.
      - 7y ago
fragmede 7y ago
BPF is not run inside of a virtual machine (the same way that Java is). It is not safe to load untrusted code into it.

ape4 7y ago

Maybe the new readfile() could be used to implement php's readfile().

131hn 7y ago
PHP readfile is all about « streaming » GB size files to stdout, not so much returning a whole file in a single buffer (but file_get_contents is and might, indeed, benefit from the new readfile syscall )
- ape4 7y ago
  Thanks, that was what I meant.

userbinator 7y ago

The idea of making file reads stateless is interesting. I wonder if it's atomic too? I think that might be very useful to have.

loeg 7y ago
It does sort of raise the question of how interrupts are handled. Short reads / EINTR and suddenly this mechanism is less efficient than open+read+close. Depending on how SA_RESTART is implemented, it may not help. Do consumers have to mask signals around readfile(), and if so, are we saving any system calls?

mayoff 7y ago

This article says `readfile` would return the number of bytes read. It seems like it would be better for it to return the number of bytes in the file, but only write at most `bufsize` bytes into the buffer. That way, if the buffer isn't big enough, you can allocate a correctly-size buffer without needing to `stat` the file. This is how `snprintf` works.

a1369209993 7y ago
That wouldn't work well because you'd need to re-open and seek (or re-read) before continuing. Ideally readfile would take and return a filefd (default -1); if present, skip the open and just read (closing on EOF). This also fixes the EINTR/SA_RESTART issue loeg[0] raised. From a C perspective, it does have a problem of needing to return multiple values (both a fd and a bytecount), though. 0: https://news.ycombinator.com/item?id=22509329 https://news.ycombinator.com/item?id=22509329
im3w1l 7y ago
I think some "files" that are actually devices or pipes and stuff, for them the number of bytes in the file doesn't make sense, while bytes read does make sense. That may be why.

correct_horse 7y ago

If the problem is slow switching between user space and kernel space, why not have a syscall that allows you to batch process multiple syscalls? It might work like this: user space allocates a buffer, fills it with syscall numbers and arguments, then passes it to the kernel. The kernel executes each syscall in series, placing the return values in the array originally containing arguments. This syscall would not return to user space until all syscalls in the buffer are processed. Does this idea have any potential?

nominatronic 7y ago
This is basically the idea of io_uring [0], although that mechanism is specific to certain I/O operations rather than being a generic queuing system for syscalls. [0] https://lwn.net/Articles/776703/ https://lwn.net/Articles/776703/
- wahern 7y ago
  Before io_uring there were many failed proposals for a general purpose, asynchronous interface to wrap blocking syscalls. One of these, "syslets", was literally a mechanism for batching arbitrary syscalls: https://lwn.net/Articles/221887/ https://lwn.net/Articles/221887/ > Syslets are small, simple, lightweight programs (consisting of system-calls, 'atoms') that the kernel can execute autonomously (and, not the least, asynchronously), without having to exit back into user-space. .... > Syslets consist of 'syslet atoms', where each atom represents a single system-call. These atoms can be chained to each other: serially, in branches or in loops.
  - correct_horse 7y ago
    Thanks for sharing. It's encouraging that something at least resembling what I came up with was considered.
  - anonymousDan 7y ago
    Any thoughts on why they failed?

longtermd 7y ago

TLDR: speed-up happens as common-combinations of syscalls (open, read, close) are combined to one single syscall (read file contents). It violates the unix philosophy of "one call for one specific thing", of course, but massively speeds up performance in practical applications. In fact, you could get a LOT of speedup at Linux/Microsoft/... (the OS level) by figuring out highly common combinations of syscalls typically all called in a row with mostly the same arguments, and combining them into one single syscall.

wahern 7y ago
Once upon a time Linux solved the "syscalls are slow" problem by making syscalls fast, rather than introducing convoluted APIs as a workaround, like other operating systems did. Perhaps one day we can get back to that model, or maybe now that Linux has gone corporate those days are gone. Even when you can make something faster with a simple, straight-forward solution, there's always that one corporate sponsor who doesn't benefit. Before Linux would just tell them sod-off.
- KarlKemp 7y ago
  that’s (at best) a gross oversimplification of the process, idealizing the past while ignoring how expectation for the kernel have increased (I e. security & privacy), all colored by a layer of bad faith that, to be frank, seems to originate with some non-specific grievances.
  - wahern 7y ago
    I'm simply channeling Linus Torvalds. Here's a sample of a rant/boast from 2000 where Linus defends the "heavy-weight" threading approach against claims that a proper user-space threading architecture could provide better performance by requiring fewer syscalls and less memory: > Yes, you could do it differently. In fact, other OS's _do_ do it differently. But take a look at our process timings (ignoring wild claims by disgruntled SCO employees that can probably be filed under "L" for "Lies, Outright"). Think about WHY our system call latency beats everybody else on the planet. Think about WHY Linux is fast. It's because it's designed right. Source: https://lkml.org/lkml/2000/8/25/80 https://lkml.org/lkml/2000/8/25/80 The same architectural approach was taken with fork: Linux optimized the heck out of fork to the point where fork on Linux was faster than thread creation on Windows. On multiple occasions Linus has argued (and proven), that optimizing the simple but "heavy-weight" approach can reap dividends at least as great as more complex, more flexible architectures. Of course, that was then and this is now. And many things have changed, including the success of Linux. I'm just suggesting, or perhaps implying, that there's more eagerness to entertain the replacement of traditional syscall semantics and other subsystems with more complex frameworks than there once used to be. In the context of Spectre, arguably it would have been easier for a younger Linus (and younger Linux) to refuse to rearchitect syscalls (with the concomitant additional kernel and semantic complexity), to tell everyone to sit tight and wait for AMD and Intel to come up with mitigating hardware fixes, and in the intervening years just take the performance hit.

cryptonector 7y ago

> the cost of reading a lot of little files would be too high The `readfile()` system call will reduce that by some factor, though probably not as high as a factor of one third (it replaces three system calls with two, assuming it turns out to be useful). But getting all the data in one call will be even faster. Incidentally, a typical read-all-of-this-file function in user-land allocates a buffer of the right size after doing an `open()` and then `fstat()` to find the file's size. Whereas a system call to read a file in one go really can't do that unless it establishes a memory mapping (which is... expensive). So I rather question the utility of this thing. Also, since one might not know what size buffer to give it, it might as well have an option to return an open FD. Or the app should have to `stat()` then `readfile()`, thus replacing three syscalls (`open()`, `fstat()`, `close()` with two -- bummer. Or the caller can `readfile()` repeatedly if the buffer given was not large enough (oof). No, I'm pretty sure this `readfile()` can't possibly make reading lots of tiny files fast enough to not want a better solution for getting mounted filesystem metadata out of the kernel.

ploxiln 7y ago
One of the cases posited is `ps` or `top` which (on linux) read /proc/[0-9]+/stat and /proc/[0-9]+/cmdline, and in these cases it knows the the max it will want to read (which is pretty small, maybe 512 bytes at most). So this can make `ps` or `top` 3x faster / more efficient.
- bicolao 7y ago
  There must be other use cases besides ps and top to justify new syscalls right? ps and top are not exactly slow to even make these optimizations visible.
deleted 7y ago
[deleted]

ncmncm 7y ago

This sort of thing leads me to think Linux is well down the path to irrelevance. Io_uring, for all its baroque elaboration, saves millions or billions of system calls. Bloating the kernel to save two of three system calls on filesystem operations seems nothing short of foolish. I would welcome being proved wrong.

yxhuvud 7y ago

O_SPECIFIED seems like a strange solution to me. Sure, I see why the kernel people wouldn't mind having people specify their own fd numbers, but doing the accounting necessary for that (to avoid conflicts) in userspace doesn't seem nice. Especially when there will be libraries that also may make use of the functionality. Is there even any way to see what file descriptors that are open without doing a syscall?

StrangeDoctor 7y ago
It looks like it’s intended to be used with a batched creation of file descriptors that are known to allocated but not yet used (to cut down on total syscalls?) https://lore.kernel.org/io-uring/20200211223235.GA25104@localhost/ https://lore.kernel.org/io-uring/20200211223235.GA25104@loca... At least I think I’m reading this correctly.

shiblukhan 7y ago

Hmm, how is one actually supposed to use O_SPECIFICFD in real life though? I mean, picking some specific FD nr can work in trvial, single threaded programs maybe, but how is that supposed to work on typical generic userspace code that has many threads and many libraries/subsystems all running in the same address space (and fd space) that want to take possession of some fd? I mean are apps supposed to block fd ranges ahead of time, by dup()ing /dev/null a couple of times, or how is that supposed to work? not getting this...

willvarfar 7y ago

Are there any mainstream databases etc adopting io_uring yet? I first heard about io_uring here on HN and a rust DB lib called Sled was used as an example. But when do we get fast MySQL etc? I recall a paper from 2012 or so that implemented a PoC syscall buffer so a bunch of syscalls could be scheduled and executed sequentially with just one syscall. They claimed 40%+ performance improvements in various programs, including MySQL. This was of course before meltdown and other cache mitigations were a thing. So programs, including MySQL and Postgres (hey, I know something about various DB storage engines), would benefit massively from scheduled sequential syscalls (easy to adopt, big win) and true async io (io_uring, harder to adopt, even bigger win).

antpls 7y ago
At that point, wouldn't it be simpler to run mysql or postgresql on their own bare metal VM with a specialized uni-kernel that only runs the db process very efficiently (without notion of kernel space and user space and without context switching) ?
zuzun 7y ago
RocksDB supports it for reading multiple chunks from a file.
anarazel 7y ago
I am working on optional io_uring support for PostgreSQL. Hope to get that stable for PG 14 (PG 13 enters feature freeze soon).

tyingq 7y ago

"On its face, readfile() adds nothing new; an application can obtain the same result with calls to openat(), read(), and close(). But it reduces the number of system calls required from three to one" A little surprised they didn't also roll in an optional lseek() offset in case you wanted a specific area of the file. Then it would roll 4 syscalls into 1. The somewhat similar sendfile() accepts an offset.

masklinn 7y ago
If you look at the thread, readfile() would be pretty specifically for extremely short files e.g. procfs and sysfs. It would not be intended as a convenience API for slurping an entire file but as a performance API for very short reads (straight open(); read(); close() where the syscall overhead is enormous). The implementation(s) by Greg Kroah-Hartman don't even bother looping until read returns empty or the buffer is full.
- tyingq 7y ago
  Sure, but it would be simple to add, and perhaps then useful for lots of small reads from bigger files. Or things like /proc/pid/mem.

nibbula 7y ago

I'm not a kernel hacker either, and even though a readfile system call might speed some things up, one of the reasons it's desired is really wallpapering over a deeper, much harder issue, that of it being a terrible hack to write the 'ps' command, and the more general issue of returning structured data from a unix-like kernel. Also it seems like a bad design not to know how big to make the buffer, which means if you want to reliably get the whole file you either have to put it in a loop reallocating the buffer, or do a stat system call before be lucky with the race condition. To be fair, this has been an issue since the first unix kernels, and even though /proc and sysctl are an improvement over grepping kernel memory from user space (the incredibly hackish way old ps worked), in my opinion it's still a big mess in various ways. Just in case you aren't aware, linux ps/top/etc. has to open possibly hundreds of fake files, parsing from text a bunch of stuff which may or may not be there or valid or the same data type depending on your kernel configuration or version. Just because you could write 'ps' with awk, doesn't make it good. I wouldn't object at all to the /proc interface, as a way of enabling simple tools, if there was also a decent function call interface. Modern BSDs generally use sysctl, which although being better in theory, that you don't have the overhead of useless translating numbers back and forth from text and looking for space characters, it still has the problem that it's very dependent on subtle changes in the C data structures which can easily happen between version and architectures. But it also has the terrible drawback of not knowing how big the buffer should be and therefore having to be in an allocation fail loop, probably exactly when you don't want it to happen, when you system is overloaded with way too many processes. I really don't see why one couldn't pass a memory allocator function, which could be called in the same manner as a signal handler. I don't know how windows stuff works inside, but to get process information you ask for a snapshot of the system from which you can return a set of handles to processes, which you can get information from. In other words, it allocates the appropriate things for you and has well defined set of information which you can query, all with a relatively simple function call interface from C. I wrote a 'ps' that works on linux/bsd/macos/windows, and even though I'm not at all fan of windows, the windows kernel does it better than all the unix kernels. It's actually not that hard to make an acceptable C interface to to say process information, if you ignore the problems of data structure variance. But ignoring it is really the root of the issues, so it seems fairly pointless. sysctl sort of tries to have a metadata system, but it doesn't really address the data type problem well, and can end up using C structs from the kernel with same variance problems. For example task_struct in linux or struct proc on macos is kind of a big mess, but very important. You wouldn't want to restrict it from changing in any way, but you do want to get some of that data to users. Making a good C metadata interface is complex, and although it's been done many times, it's quite tricky to do well, and you would really want a kernel interface to be done very well. I like to imagine that there could be something less bloated than gtk gobject, but more featurefull than sysctl. Unfortunately when I look at code involved I start feeling like C isn't really a good language for writing an operating system. But I still think that a well designed structured meta data system would be overall less overhead than /proc and could even achieve better speed and reliability.

trgfs85 7y ago

A lot of comments in this topic seem to forget that syscalls don't always succeed. Interupts are the biggest issue. If an interrupt occurs and you read something you get a partial read. If you open something you tend to get EINTER(very system dependent.) You can't really queue those. And those shouldn't be the OS's job to deal with.