6 ms·

> Continuing to use a file descriptor after it has been closed It would be nice if file descriptors on 64-bit platforms used tagging so the full value could ne

> Continuing to use a file descriptor after it has been closed It would be nice if file descriptors on 64-bit platforms used tagging so the full value could never be reused. E.g. lower 32 bits is the normal fd and the 32-bit is a serial number (or random value). When using the fd the kernel can use the lower 32-bit as index into the table and check the upper 32 bits are correct.

fanf2 7y ago
Sadly that isn't allowed by POSIX https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/functions/V2_chap02.html#tag_15_14 https://pubs.opengroup.org/onlinepubs/9699919799.2018edition... > 2.14. File Descriptor Allocation > All functions that open one or more file descriptors shall, unless specified otherwise, atomically allocate the lowest numbered available file descriptor
- codetrotter 7y ago
  Would badness ensue if we allowed ourselves to define all previously used fds as “unavailable”, and thus conforming to the rules as stated when we increment our serial number before handing out the next fd? If yes, what sorts of badness and why?
  - samatman 7y ago
    It strikes me that you'd use up STDIN, STDOUT, and STDERR pretty fast that way. One could make an unprincipled exception for them, of course. And SQLITE would still need the special-casing to prevent using low-numbered fds. But this would prevent other problems and is worth considering.
    - mras0 7y ago
      Didn't see your reply before commenting, but it gave me a thought: We really only need a "local" solution. If there were a way of requesting fd's that weren't required to be the lowest available ones (perhaps even 64-bit ones with a process unique uppper part), that'd probably be good enough here. Of course we're still sharing address space with buggy code, so maybe it's not worth it.
      - codetrotter 7y ago
        I’ll reply here since then both you and parent can see it and comment further. In the case that you linked to in the sibling comment that you wrote in reply to me, the dup2 syscall is used. So the question then that I have is, in the case of foo 2&>1 Does bash then: a) First close fd 2, and then call dup(1), or does it b) Call dup2(1, 2), or c) Do something else entirely, or something more complex or convoluted? In case the answer is b) then it seems to me, though I may be overlooking something, that we could allow calls like b) and inside of the dup2 syscall keep the id of the fd that we are using as newfd, since it is unambiguously requested that this is intended, while at the same never reusing fd id in the dup syscall. In this comment I am referring to the dup and dup2 syscalls as defined in the Linux man pages: dup() uses the lowest-numbered unused descriptor for the new descriptor. dup2() makes newfd be the copy of oldfd, closing newfd first if necessary, but note the following: [...] https://linux.die.net/man/2/dup2 https://linux.die.net/man/2/dup2 With the modification that “lowest-numbered unused descriptor” would be unique as previously stated in the comment that the two of you are replying to. More generally, the question is, does any wide-spread piece of code manually close an fd and then call the dup syscall, and expect the fd id to be exactly the same as the one they themselves closed, instead of using dup2? And furthermore, likewise, does any widespread piece of code use the dup2 syscall to specify the newfd where actually it should not have cared about the id of the newfd and should have used dup instead and let the system decide?
        mras0 7y ago
        I don't think it matters what bash actually does, the issue (as I see it) is that when a program calls write(N, ...) it sometimes wants to refer to the "stale" file descriptor N. Consider this contrived example: write(STDERR_FILENO,"test",4); // OK, write to normal stderr close(STDERR_FILENO); open(....); // Open some log file // later write(STDERR_FILENO,"x",1); // OK, write to log file Even though fd 2 refers to different files I think the above is required to work for POSIX compliance.
        codetrotter 7y ago
        That case is covered too as long as the program makes use of the dup2 syscall to make the switch instead of close followed by open. But it is probable that some pieces of code that are currently in widespread use do it by close and expecting a call to open to give them fd id for stderr like you say indeed I guess. So the conclusion is that the answer to my original question is probably like you say, we can’t just do that. But it would be interesting to find out how many wide-spread pieces of open source code do it that way, and for all code that does one could submit patches to change them to using the explicit dup2.
  - mras0 7y ago
    I think quite a bit of software actually relies on closing and re-opening file descriptors, especially stdin/stdout/stderr just before/after forking. This was the first example I found via a quick web search: https://unix.stackexchange.com/a/148410 https://unix.stackexchange.com/a/148410 but it suggest that e.g. bash would break :) It might be possible with some hacks/heuristics to catch many errors (perhaps one could create a valgrind tool/santizier), but I suspect it's not possible in general since there's no way of knowing if the call to write(2,....) meant the old stderr or the new one.
- marshray 7y ago
  What would it take for a valid program to depend on this behavior? How weird would it look? How likely is it to arise in practice? I'm a believer in standards in general, but there's a time and place to move beyond them too.
dezgeg 7y ago
File descriptors are held in type int in most APIs, which is only 32 bits wide even in 64-bit systems.

fanf2 7y ago

Sadly that isn't allowed by POSIX https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/functions/V2_chap02.html#tag_15_14 https://pubs.opengroup.org/onlinepubs/9699919799.2018edition... > 2.14. File Descriptor Allocation > All functions that open one or more file descriptors shall, unless specified otherwise, atomically allocate the lowest numbered available file descriptor

codetrotter 7y ago
Would badness ensue if we allowed ourselves to define all previously used fds as “unavailable”, and thus conforming to the rules as stated when we increment our serial number before handing out the next fd? If yes, what sorts of badness and why?
- samatman 7y ago
  It strikes me that you'd use up STDIN, STDOUT, and STDERR pretty fast that way. One could make an unprincipled exception for them, of course. And SQLITE would still need the special-casing to prevent using low-numbered fds. But this would prevent other problems and is worth considering.
  - mras0 7y ago
    Didn't see your reply before commenting, but it gave me a thought: We really only need a "local" solution. If there were a way of requesting fd's that weren't required to be the lowest available ones (perhaps even 64-bit ones with a process unique uppper part), that'd probably be good enough here. Of course we're still sharing address space with buggy code, so maybe it's not worth it.
    - codetrotter 7y ago
      I’ll reply here since then both you and parent can see it and comment further. In the case that you linked to in the sibling comment that you wrote in reply to me, the dup2 syscall is used. So the question then that I have is, in the case of foo 2&>1 Does bash then: a) First close fd 2, and then call dup(1), or does it b) Call dup2(1, 2), or c) Do something else entirely, or something more complex or convoluted? In case the answer is b) then it seems to me, though I may be overlooking something, that we could allow calls like b) and inside of the dup2 syscall keep the id of the fd that we are using as newfd, since it is unambiguously requested that this is intended, while at the same never reusing fd id in the dup syscall. In this comment I am referring to the dup and dup2 syscalls as defined in the Linux man pages: dup() uses the lowest-numbered unused descriptor for the new descriptor. dup2() makes newfd be the copy of oldfd, closing newfd first if necessary, but note the following: [...] https://linux.die.net/man/2/dup2 https://linux.die.net/man/2/dup2 With the modification that “lowest-numbered unused descriptor” would be unique as previously stated in the comment that the two of you are replying to. More generally, the question is, does any wide-spread piece of code manually close an fd and then call the dup syscall, and expect the fd id to be exactly the same as the one they themselves closed, instead of using dup2? And furthermore, likewise, does any widespread piece of code use the dup2 syscall to specify the newfd where actually it should not have cared about the id of the newfd and should have used dup instead and let the system decide?
      - mras0 7y ago
        I don't think it matters what bash actually does, the issue (as I see it) is that when a program calls write(N, ...) it sometimes wants to refer to the "stale" file descriptor N. Consider this contrived example: write(STDERR_FILENO,"test",4); // OK, write to normal stderr close(STDERR_FILENO); open(....); // Open some log file // later write(STDERR_FILENO,"x",1); // OK, write to log file Even though fd 2 refers to different files I think the above is required to work for POSIX compliance.
        codetrotter 7y ago
        That case is covered too as long as the program makes use of the dup2 syscall to make the switch instead of close followed by open. But it is probable that some pieces of code that are currently in widespread use do it by close and expecting a call to open to give them fd id for stderr like you say indeed I guess. So the conclusion is that the answer to my original question is probably like you say, we can’t just do that. But it would be interesting to find out how many wide-spread pieces of open source code do it that way, and for all code that does one could submit patches to change them to using the explicit dup2.
- mras0 7y ago
  I think quite a bit of software actually relies on closing and re-opening file descriptors, especially stdin/stdout/stderr just before/after forking. This was the first example I found via a quick web search: https://unix.stackexchange.com/a/148410 https://unix.stackexchange.com/a/148410 but it suggest that e.g. bash would break :) It might be possible with some hacks/heuristics to catch many errors (perhaps one could create a valgrind tool/santizier), but I suspect it's not possible in general since there's no way of knowing if the call to write(2,....) meant the old stderr or the new one.
marshray 7y ago
What would it take for a valid program to depend on this behavior? How weird would it look? How likely is it to arise in practice? I'm a believer in standards in general, but there's a time and place to move beyond them too.

dezgeg 7y ago

File descriptors are held in type int in most APIs, which is only 32 bits wide even in 64-bit systems.