7 ms·

> POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most li

> POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most likely very rare, so it’s not entirely impossible for this state to be moved to userspace). Normally it’s rare, except shell functionality crucially depends on it[1]. The way to think about this, IMO, is that while a disk file is more or less random-access, when you open it unavoidably turns into strange hybrid of a random-access byte array and a (seekable) byte stream (classically, the “file description”), and that is what the resulting fd and any of its copies point at. Seekable streams made some sense in the age of tape drives, but nowadays it seems to me that random-access files and non-seekable streams have enough of a gulf between them that forcing both to appear like the same kind of object is more confusing than helpful. (I don’t know what my ideal world would look like, given I still like to direct streams from/to files. I still dislike the “file description” business rather strongly.) [1] https://utcc.utoronto.ca/~cks/space/blog/unix/DupSharedOffsetNecessary https://utcc.utoronto.ca/~cks/space/blog/unix/DupSharedOffse...

raggi 2y ago
absolutely right re. shells, there's a lot in core old unix that depends on dup and a lot from the 90s/00s that depends on dup2 as well, with all the pain in the butt that it comes with (for multi-threaded and userspace implementors). readat and friends are certainly becoming more popular in various domains, particularly as they're cheaper in highly parallel programs avoiding the locking on seeks, and avoiding the stalls on mappings (unless you can afford gigantic / up-front & static mappings). At this point what we really need is thread independent mapping solutions, a solution to request a mapping be bound to a thread, and the ability to also pass that around. Mapping will never be free, but if we can avoid whole process stalls and giant mapping pressure both of which are painfully common today that's a big step forward.
- wahern 2y ago
  How would that work in practice? Each thread has their own memory map, but with one shared branch where all the normal process-global virtual memory is mapped, and a non-shared branch for per-thread virtual memory? Seems like there's the potential for surprise, accidental overlapping of non-shared VM mappings. But I guess this is purely a performance thing; caveat emptor, etc.... How easy would that be to hack into Linux's existing page table structures?
  - Veserv 2y ago
    That is not how MMUs work, so it would go very poorly. MMUs expect a tree. You can not give them “Tree A, except at Node N substitute a different sub-tree”. To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees. To do what you are expecting, you would actually want multiple processes with their own mappings/tree and you would share a substantial common/shared mapping/sub-tree. The difficulty there is making sure all the processes are aware of the common sub-tree and properly share and synchronize with it.
    - jart 2y ago
      Yeah per thread virtual memory is called a process. Within a process the memory manager has to maintain a single virtual address space. You could do it with a red black tree and a global lock. Or you could use potentially cleverer algorithms. Skip lists are friendly to scalable concurrency. You could also do a radix tree of page tables, similar to what x86-64 processors do internally, then have each item be its own atomic long, and use some kind of optimistic transactional locking strategy when allocating mappings that span multiple pages. Personally, I think the rbtree is easiest, since it's better to leave memory scalability to malloc(). The way I like to make malloc() scale is having an arena for each CPU and then indexing them using sched_getcpu().
      - immibis 2y ago
        The Linux kernel actually doesn't know the difference between a thread and a process. The clone syscall (which starts a new thread) takes a bitmask specifying what should be shared between the parent and child (e.g. address space, open files, thread group ID (what most people call a process ID))
    - wahern 2y ago
      > To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees. That's what I meant by each thread having their own map (i.e. root), and within each map they have a shared branch, a branch being a pointer to a subtree. In this case, what I had in mind was the shared branch being at the root; basically slicing the user land address space in half (or 1/radix size...), though, that's probably getting too detailed. Perhaps by "branch" you thought I meant a jump operation in code. That's certainly not how most MMUs work, which walk a data structure constructed by the OS. But some MMUs actually fault to OS provided code to resolve an address, in which case "branch" in the meaning of software code isn't necessarily wrong. (But to be clear, it's not what I had in mind.)
deleted 2y ago
[deleted]
nolist_policy 2y ago
Passing around fd's via unix sockets is important for sandboxing in e.g. Chrome and Firefox.
seeknotfind 2y ago
Ah! Well as a systems programmer, it's not so rare. FDs are really not a friendly interface, and I'd love a better interface (FDs have their quirks), but as an application programmer, it's really sad we don't take advantage of FD passing to build more modular programs. For instance on desktop operating systems, you pass paths, not capabilities to use files, and the norm is giving programs access to read large swaths of user data. This is not secure, and it's one of the issues well-passed FDs could help solve. In general, despite the challenges in making them secure, I'd strongly advocate for more modular and integrated applications. As for the seeking abstraction, it fits well with other buffered device driver information streams. Yes, it's a complicated and confusing interface, but the key thing is it allows you to share an OS/system/hardware level resource between multiple programs. We want to take advantage of that. It's the abstraction we've got sure, but it's what we do with it that counts!
- mananaysiempre 2y ago
  I have nothing against FD passing, and indeed agree it’s unfortunate we don’t do more of it. An (almost-)everything-is-a-string (shell) language with object-capability powers is still something I’d like to figure out someday. The Lisp-2-ish way Tcl object systems approach this feels interesting, but still a bit off from a real solution. My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position. And shell-like redirection use cases really are the only case I can think of where that’s important. > As for the seeking abstraction, it fits well with other buffered device driver information streams. I don’t think I understand what you’re getting at here. My point was that having some objects (fds, whatever) support {fstat, read/write} only and others {fstat, pread/pwrite, mmap} only would get rid of the confusing user-visible notion of “file description”. Obviously I don’t expect this to happen, but it’s still nice to dream.
  - tbrownaw 2y ago
    > My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position. It reads like this was an assumption rather than an observation. > And shell-like redirection use cases really are the only case I can think of where that’s important. .xsession-errors , or really anything of that nature where a process tree shares an error log file on stderr.
Cloudef 2y ago
Linux has had plan9 namespaces for long time which you can use from userspace as well. With this you can give a process isolated view of the filesystem, network etc .. Unfortunately not enabled on AWS lambda runner <https://github.com/aws/aws-lambda-base-images/issues/143 https://github.com/aws/aws-lambda-base-images/issues/143>
XorNot 2y ago
I don't know that I see the practical difference here: a seekable stream and a random access byte array are one very thin abstraction away from each other, with the important caveat that a stream can represent a byte array which is being appended to while you're using it.
- layla5alive 2y ago
  That's true in serial computations, but not true in concurrent ones - the abstraction now includes mutual exclusion in the stream case, and does not require it in the array case.
- mananaysiempre 2y ago
  As long as only one thread of execution is using it, yes, there’s essentially no difference. When it’s shared between several, though, seekable streams become annoying (and difficult to impossible to use correctly). When the sharing spans process boundaries, we get the well-known bane of microkernel Unices that TFA describes—you need every file position for every Unix process to be (at least potentially) tracked by a single systemwide “Unix server”. That’s a lot of pain for not a lot of win. My point was that I can’t think of any non-niche thing that’s naturally a seekable stream—they’re usually either nonseekable or outright random-access. Tape was the natural example when the API was invented, but it is niche now.
  - XorNot 2y ago
    But if the stream represents a file that can receive writes, then one way or another you need to keep track of the order in which writes and reads happen - i.e. you need locks - which means you need global state. If the access to a file was truly read only, then trivially every reader can just have its own seekable FD tracking position locally (or permissions to access the underlying object and open new ones).
    - mananaysiempre 2y ago
      From the point of view of the application, yes, you need to coordinate. But that’s a concern for the application. From the point of view of the kernel / device server / however your microkernel OS works, requests for disk I/O arrive in some order, probably update some cache pages or whatnot, then go onto disk’s queue. That’s arguably global, but it feels like a logical extension of the fact that you only have a single physical disk. However you’re going to multiplex it, something like that is still going to happen, and it has little to do with Unix. What is not a natural extension of the whole thing is that, even on an otherwise completely quiescent system, int fd1 = open("foo", O_RDWR), fd2 = dup(fd1); behaves very differently from int fd1 = open("foo", O_RDWR), fd2 = open("foo", O_RDWR); (imagine these fds are then passed out to other processes or whatnot). Even with O_RDONLY, these are still not at all the same. Witness the epitome of CLI design that is OpenSSL /s : { openssl x509 -out "/etc/swanctl/x509/$1.pem" while openssl x509 -out "$t" 2>/dev/null; do fp=$(openssl x509 -in "$t" -noout -md5 -fingerprint | sed 's/.*=//; s/://g') mv -f -- "$t" /etc/swanctl/x509ca/$fp.pem done } < "/etc/ssl/uacme/$1/cert.pem" This is how you pick apart a PEM cert bundle using OpenSSL: the shell spawns openssl, which reads a single PEM block from stdin, does its dirty deeds with it, and leaves stdin pointing to the next one, ready to be consumed by the next instance of itself that’s yet to be spawned by the shell. You can’t do that with a per-process file position, trivially or otherwise. Returning to the application side, imagine you’re making a concurrent B-tree. If one thread wants to write out page X and the other page Y, they have presumably already used some locking to make sure that’ll leave the data structure consistent, so they’re free to just issue their pwrite()s and let them happen in whatever order. On the other hand, if all they have is write() and seek(), they have to hit a global lock even if X and Y are completely unrelated.
4lDO2 2y ago
I'm aware of this example, this was discussed earlier on HN [1]. I think it would be reasonable to (try to) enforce O_APPEND in such scenarios, in which case libc might internally open a pipe (configured to passively or actively be flushed to the underlying file, non-concurrently). A pipe would also be more reliable (and secure, though programs in shell pipelines are usually trusted), since it's no longer possible for isolated programs to change the global cursor, which could overwrite other processes' written data. [1] https://news.ycombinator.com/item?id=38009458 https://news.ycombinator.com/item?id=38009458 Also, I feel like you are misquoting me, by not including the whole sentence (judging from the subcomments). I implied shared fds with shared cursors are probably a rare use cases, not shared fds in general (as you explained later on). Shared fds, and especially the ability to send fds, are obviously very fundamental for Unix-like systems, and for moving towards capability-oriented security.
- mananaysiempre 2y ago
  Yeah, cutting the sentence that way evidently did not work out, sorry. For posterity: > This cursor unfortunately cannot be stored in userspace without complex coordination, since POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most likely very rare, so it’s not entirely impossible for this state to be moved to userspace).
layer8 2y ago
When you write “file description”, do you mean “file descriptor”?
- 4lDO2 2y ago
  File descriptors and file descriptions are not the same thing. Descriptors are references to descriptions, along with some metadata, and multiple descriptors can point to the same description.
  - layer8 2y ago
    > multiple descriptors can point to the same descriptor. You mean to the same description? And is this the same as what is referred to as “open file description” in the open(2) man page, or are there also other “file descriptions”?
    - 4lDO2 2y ago
      Yes, and yes.
- mananaysiempre 2y ago
  Crucially, no. “File descriptions” (or “open file descriptions”; classically, struct file[1]) are what file descriptors designate. The whole hubbub is because a file description is not just a device number, an inode number, and a set of already-checked permissions; they are all that and a position in the file, and that position gets shared along with the rest of the description when you dup(), fork(), or sendmsg() the file descriptor—but not if you open() the same file again, not even[2] via /proc/self/fd. As I’ve said, I think the most helpful way to think about this is that a file description is a seekable stream object on top of the underlying random-access disk file, and a reference to that stream object is what you’re passing around when you’re handling (hah) file descriptors. [1] https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/fio.c https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/fi... [2] https://blog.gnoack.org/post/proc-fd-is-not-dup/ https://blog.gnoack.org/post/proc-fd-is-not-dup/

raggi 2y ago

absolutely right re. shells, there's a lot in core old unix that depends on dup and a lot from the 90s/00s that depends on dup2 as well, with all the pain in the butt that it comes with (for multi-threaded and userspace implementors). readat and friends are certainly becoming more popular in various domains, particularly as they're cheaper in highly parallel programs avoiding the locking on seeks, and avoiding the stalls on mappings (unless you can afford gigantic / up-front & static mappings). At this point what we really need is thread independent mapping solutions, a solution to request a mapping be bound to a thread, and the ability to also pass that around. Mapping will never be free, but if we can avoid whole process stalls and giant mapping pressure both of which are painfully common today that's a big step forward.

wahern 2y ago
How would that work in practice? Each thread has their own memory map, but with one shared branch where all the normal process-global virtual memory is mapped, and a non-shared branch for per-thread virtual memory? Seems like there's the potential for surprise, accidental overlapping of non-shared VM mappings. But I guess this is purely a performance thing; caveat emptor, etc.... How easy would that be to hack into Linux's existing page table structures?
- Veserv 2y ago
  That is not how MMUs work, so it would go very poorly. MMUs expect a tree. You can not give them “Tree A, except at Node N substitute a different sub-tree”. To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees. To do what you are expecting, you would actually want multiple processes with their own mappings/tree and you would share a substantial common/shared mapping/sub-tree. The difficulty there is making sure all the processes are aware of the common sub-tree and properly share and synchronize with it.
  - jart 2y ago
    Yeah per thread virtual memory is called a process. Within a process the memory manager has to maintain a single virtual address space. You could do it with a red black tree and a global lock. Or you could use potentially cleverer algorithms. Skip lists are friendly to scalable concurrency. You could also do a radix tree of page tables, similar to what x86-64 processors do internally, then have each item be its own atomic long, and use some kind of optimistic transactional locking strategy when allocating mappings that span multiple pages. Personally, I think the rbtree is easiest, since it's better to leave memory scalability to malloc(). The way I like to make malloc() scale is having an arena for each CPU and then indexing them using sched_getcpu().
    - immibis 2y ago
      The Linux kernel actually doesn't know the difference between a thread and a process. The clone syscall (which starts a new thread) takes a bitmask specifying what should be shared between the parent and child (e.g. address space, open files, thread group ID (what most people call a process ID))
  - wahern 2y ago
    > To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees. That's what I meant by each thread having their own map (i.e. root), and within each map they have a shared branch, a branch being a pointer to a subtree. In this case, what I had in mind was the shared branch being at the root; basically slicing the user land address space in half (or 1/radix size...), though, that's probably getting too detailed. Perhaps by "branch" you thought I meant a jump operation in code. That's certainly not how most MMUs work, which walk a data structure constructed by the OS. But some MMUs actually fault to OS provided code to resolve an address, in which case "branch" in the meaning of software code isn't necessarily wrong. (But to be clear, it's not what I had in mind.)

deleted 2y ago

[deleted]

nolist_policy 2y ago

Passing around fd's via unix sockets is important for sandboxing in e.g. Chrome and Firefox.

seeknotfind 2y ago

Ah! Well as a systems programmer, it's not so rare. FDs are really not a friendly interface, and I'd love a better interface (FDs have their quirks), but as an application programmer, it's really sad we don't take advantage of FD passing to build more modular programs. For instance on desktop operating systems, you pass paths, not capabilities to use files, and the norm is giving programs access to read large swaths of user data. This is not secure, and it's one of the issues well-passed FDs could help solve. In general, despite the challenges in making them secure, I'd strongly advocate for more modular and integrated applications. As for the seeking abstraction, it fits well with other buffered device driver information streams. Yes, it's a complicated and confusing interface, but the key thing is it allows you to share an OS/system/hardware level resource between multiple programs. We want to take advantage of that. It's the abstraction we've got sure, but it's what we do with it that counts!

mananaysiempre 2y ago
I have nothing against FD passing, and indeed agree it’s unfortunate we don’t do more of it. An (almost-)everything-is-a-string (shell) language with object-capability powers is still something I’d like to figure out someday. The Lisp-2-ish way Tcl object systems approach this feels interesting, but still a bit off from a real solution. My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position. And shell-like redirection use cases really are the only case I can think of where that’s important. > As for the seeking abstraction, it fits well with other buffered device driver information streams. I don’t think I understand what you’re getting at here. My point was that having some objects (fds, whatever) support {fstat, read/write} only and others {fstat, pread/pwrite, mmap} only would get rid of the confusing user-visible notion of “file description”. Obviously I don’t expect this to happen, but it’s still nice to dream.
- tbrownaw 2y ago
  > My reading of TFA was that it’s rare for it to be important that descriptors sharing a description thus also share a file position. It reads like this was an assumption rather than an observation. > And shell-like redirection use cases really are the only case I can think of where that’s important. .xsession-errors , or really anything of that nature where a process tree shares an error log file on stderr.

Cloudef 2y ago

Linux has had plan9 namespaces for long time which you can use from userspace as well. With this you can give a process isolated view of the filesystem, network etc .. Unfortunately not enabled on AWS lambda runner <https://github.com/aws/aws-lambda-base-images/issues/143 https://github.com/aws/aws-lambda-base-images/issues/143>

XorNot 2y ago

I don't know that I see the practical difference here: a seekable stream and a random access byte array are one very thin abstraction away from each other, with the important caveat that a stream can represent a byte array which is being appended to while you're using it.

layla5alive 2y ago
That's true in serial computations, but not true in concurrent ones - the abstraction now includes mutual exclusion in the stream case, and does not require it in the array case.
mananaysiempre 2y ago
As long as only one thread of execution is using it, yes, there’s essentially no difference. When it’s shared between several, though, seekable streams become annoying (and difficult to impossible to use correctly). When the sharing spans process boundaries, we get the well-known bane of microkernel Unices that TFA describes—you need every file position for every Unix process to be (at least potentially) tracked by a single systemwide “Unix server”. That’s a lot of pain for not a lot of win. My point was that I can’t think of any non-niche thing that’s naturally a seekable stream—they’re usually either nonseekable or outright random-access. Tape was the natural example when the API was invented, but it is niche now.
- XorNot 2y ago
  But if the stream represents a file that can receive writes, then one way or another you need to keep track of the order in which writes and reads happen - i.e. you need locks - which means you need global state. If the access to a file was truly read only, then trivially every reader can just have its own seekable FD tracking position locally (or permissions to access the underlying object and open new ones).
  - mananaysiempre 2y ago
    From the point of view of the application, yes, you need to coordinate. But that’s a concern for the application. From the point of view of the kernel / device server / however your microkernel OS works, requests for disk I/O arrive in some order, probably update some cache pages or whatnot, then go onto disk’s queue. That’s arguably global, but it feels like a logical extension of the fact that you only have a single physical disk. However you’re going to multiplex it, something like that is still going to happen, and it has little to do with Unix. What is not a natural extension of the whole thing is that, even on an otherwise completely quiescent system, int fd1 = open("foo", O_RDWR), fd2 = dup(fd1); behaves very differently from int fd1 = open("foo", O_RDWR), fd2 = open("foo", O_RDWR); (imagine these fds are then passed out to other processes or whatnot). Even with O_RDONLY, these are still not at all the same. Witness the epitome of CLI design that is OpenSSL /s : { openssl x509 -out "/etc/swanctl/x509/$1.pem" while openssl x509 -out "$t" 2>/dev/null; do fp=$(openssl x509 -in "$t" -noout -md5 -fingerprint | sed 's/.*=//; s/://g') mv -f -- "$t" /etc/swanctl/x509ca/$fp.pem done } < "/etc/ssl/uacme/$1/cert.pem" This is how you pick apart a PEM cert bundle using OpenSSL: the shell spawns openssl, which reads a single PEM block from stdin, does its dirty deeds with it, and leaves stdin pointing to the next one, ready to be consumed by the next instance of itself that’s yet to be spawned by the shell. You can’t do that with a per-process file position, trivially or otherwise. Returning to the application side, imagine you’re making a concurrent B-tree. If one thread wants to write out page X and the other page Y, they have presumably already used some locking to make sure that’ll leave the data structure consistent, so they’re free to just issue their pwrite()s and let them happen in whatever order. On the other hand, if all they have is write() and seek(), they have to hit a global lock even if X and Y are completely unrelated.

4lDO2 2y ago

I'm aware of this example, this was discussed earlier on HN [1]. I think it would be reasonable to (try to) enforce O_APPEND in such scenarios, in which case libc might internally open a pipe (configured to passively or actively be flushed to the underlying file, non-concurrently). A pipe would also be more reliable (and secure, though programs in shell pipelines are usually trusted), since it's no longer possible for isolated programs to change the global cursor, which could overwrite other processes' written data. [1] https://news.ycombinator.com/item?id=38009458 https://news.ycombinator.com/item?id=38009458 Also, I feel like you are misquoting me, by not including the whole sentence (judging from the subcomments). I implied shared fds with shared cursors are probably a rare use cases, not shared fds in general (as you explained later on). Shared fds, and especially the ability to send fds, are obviously very fundamental for Unix-like systems, and for moving towards capability-oriented security.

mananaysiempre 2y ago
Yeah, cutting the sentence that way evidently did not work out, sorry. For posterity: > This cursor unfortunately cannot be stored in userspace without complex coordination, since POSIX allows file descriptors to be shared by an arbitrary number of processes, after e.g. forks or SCM_RIGHTS transfers (even though this use case is most likely very rare, so it’s not entirely impossible for this state to be moved to userspace).

layer8 2y ago

When you write “file description”, do you mean “file descriptor”?

4lDO2 2y ago
File descriptors and file descriptions are not the same thing. Descriptors are references to descriptions, along with some metadata, and multiple descriptors can point to the same description.
- layer8 2y ago
  > multiple descriptors can point to the same descriptor. You mean to the same description? And is this the same as what is referred to as “open file description” in the open(2) man page, or are there also other “file descriptions”?
  - 4lDO2 2y ago
    Yes, and yes.
mananaysiempre 2y ago
Crucially, no. “File descriptions” (or “open file descriptions”; classically, struct file[1]) are what file descriptors designate. The whole hubbub is because a file description is not just a device number, an inode number, and a set of already-checked permissions; they are all that and a position in the file, and that position gets shared along with the rest of the description when you dup(), fork(), or sendmsg() the file descriptor—but not if you open() the same file again, not even[2] via /proc/self/fd. As I’ve said, I think the most helpful way to think about this is that a file description is a seekable stream object on top of the underlying random-access disk file, and a reference to that stream object is what you’re passing around when you’re handling (hah) file descriptors. [1] https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/fio.c https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/fi... [2] https://blog.gnoack.org/post/proc-fd-is-not-dup/ https://blog.gnoack.org/post/proc-fd-is-not-dup/