2 ms·

Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor manageme

Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor management which only gets triggered when it inherits an extra fd. The developers might never notice because they never run it under your shell, and then someday some user does. An idea I've had before: the shell should listen on a Unix domain socket, and put the path to that in an environment variable. Then commands could talk to the shell which invoked them. That could be used for this kind of content negotiation stuff. It could also be used to do other things – for example, a subprocess could modify the current directory of the user's shell, or its environment – there are scenarios in which being able to do that would be useful. A Unix domain socket path in an environment variable seems less likely to cause unexpected issues than inheriting an extra file descriptor.

hnlmorg 4y ago
Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.
- skissane 4y ago
  Not trying to be dismissive of your approach either. While I can think of ways it might cause problems in theory, I really have no idea how likely those problems would be in practice. > I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. A socket is just a file descriptor at the end of the day. Some shells (I think some variants of ksh?) have actually used socketpair() to build pipelines instead of pipe(). > the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? The parent process might just be some kind of launcher and the child might be the one that does all the actual work. In any event, what's to stop fd3 being inherited by grandchild processes too?
  - hnlmorg 4y ago
    > A socket is just a file descriptor at the end of the day They are. But I don’t have code to manage sockets whereas I already do manage STDOUT et al so adding another pipe there doesn’t increase the code complexity. It’s a bit of a lazy excuse I’ll grant you. But it also wasn’t the deciding reason, just an added incentive. > Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. I wasn’t aware of this. That’s handy to know. Thank you. > However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? There’s a lot of scope for accidental data corruption. If you’ve just got a byte stream of text then it’s very easy for data to be safely concatenated. But if you’re streaming objects then the last thing you want is is those objects concatenated by dumb POSIX pipes. An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. If the kernel was object aware then this wouldn’t be an issue. But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. > In any event, what's to stop fd3 being inherited by grandchild processes too? Indeed. Just as you can write directly to fd{0..2} on any given process if you wanted. But the developer would have to go out of their way to expose (or locate another processes) fd3. My goal wasn’t to prevent deliberate exposure but rather to prevent accidental unwanted exposure. I do agree there is the risk that some processes might intentionally create an fd3, unaware of my shell. But at the time I felt that risk was lower than the risk of processes accidentally misusing a socket…though I will look into the point you made about the PID of connecting process being available as that might swing my decision the other way.
    - skissane 4y ago
      > An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. The same thing could happen with fd3: Your shell spawns programA, which knows nothing about fd3. programA then spawns children programB and programC in parallel, both of which do. If programA does nothing to stop programB and programC from inheriting fd3, they both will, and then you are in the same situation as with a Unix domain socket path in an environment variable. Actually, here's a way in which fd3 is worse than a domain socket path in an environment variable: with a Unix domain socket, you'll get two independent connections (on different FDs) from programB and programC, so even if their output gets mixed up, at least their control messages won't be. Whereas, they'll share the same fd3, and any message you receive, you won't be able to tell which one it came from–there is even some risk you'll get part of a message from one and then part of a message from another (although whether that happens or not depends on the details of buffering). > But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. Linux supports SOCK_SEQPACKET Unix domain sockets, in which the kernel is aware of explicit message boundaries. Unfortunately, most other Unix-like systems don't (most significantly, macOS).

hnlmorg 4y ago

Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.

skissane 4y ago
Not trying to be dismissive of your approach either. While I can think of ways it might cause problems in theory, I really have no idea how likely those problems would be in practice. > I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. A socket is just a file descriptor at the end of the day. Some shells (I think some variants of ksh?) have actually used socketpair() to build pipelines instead of pipe(). > the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? The parent process might just be some kind of launcher and the child might be the one that does all the actual work. In any event, what's to stop fd3 being inherited by grandchild processes too?
- hnlmorg 4y ago
  > A socket is just a file descriptor at the end of the day They are. But I don’t have code to manage sockets whereas I already do manage STDOUT et al so adding another pipe there doesn’t increase the code complexity. It’s a bit of a lazy excuse I’ll grant you. But it also wasn’t the deciding reason, just an added incentive. > Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. I wasn’t aware of this. That’s handy to know. Thank you. > However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? There’s a lot of scope for accidental data corruption. If you’ve just got a byte stream of text then it’s very easy for data to be safely concatenated. But if you’re streaming objects then the last thing you want is is those objects concatenated by dumb POSIX pipes. An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. If the kernel was object aware then this wouldn’t be an issue. But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. > In any event, what's to stop fd3 being inherited by grandchild processes too? Indeed. Just as you can write directly to fd{0..2} on any given process if you wanted. But the developer would have to go out of their way to expose (or locate another processes) fd3. My goal wasn’t to prevent deliberate exposure but rather to prevent accidental unwanted exposure. I do agree there is the risk that some processes might intentionally create an fd3, unaware of my shell. But at the time I felt that risk was lower than the risk of processes accidentally misusing a socket…though I will look into the point you made about the PID of connecting process being available as that might swing my decision the other way.
  - skissane 4y ago
    > An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. The same thing could happen with fd3: Your shell spawns programA, which knows nothing about fd3. programA then spawns children programB and programC in parallel, both of which do. If programA does nothing to stop programB and programC from inheriting fd3, they both will, and then you are in the same situation as with a Unix domain socket path in an environment variable. Actually, here's a way in which fd3 is worse than a domain socket path in an environment variable: with a Unix domain socket, you'll get two independent connections (on different FDs) from programB and programC, so even if their output gets mixed up, at least their control messages won't be. Whereas, they'll share the same fd3, and any message you receive, you won't be able to tell which one it came from–there is even some risk you'll get part of a message from one and then part of a message from another (although whether that happens or not depends on the details of buffering). > But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. Linux supports SOCK_SEQPACKET Unix domain sockets, in which the kernel is aware of explicit message boundaries. Unfortunately, most other Unix-like systems don't (most significantly, macOS).