3 ms·

Two big differences between PowerShell (even on Linux) and traditional Unix shells: (1) multithreaded single process architecture, as opposed to multi-process a

Two big differences between PowerShell (even on Linux) and traditional Unix shells: (1) multithreaded single process architecture, as opposed to multi-process architecture; (2) runs under a VM (.Net CLR) as opposed to native code. I see nushell is written in Rust, so it doesn't have (2), although I'm less sure about (1). I think it would be cool if you could add structured pipeline support to traditional Unix tools (bash, coreutils, etc). The problem is, Unix pipes don't have any support for passing metadata (performing content negotiation between the two ends of the pipe). Given they are a facility provided by the kernel, adding that would likely require kernel changes. One day I started daydreaming about prototyping content negotiation support for Unix pipes. I was thinking of using CUSE (Linux character device driver in userspace) to do it. Never actually got around to trying though, other more pressing things to work on. Maybe some day I will get around with it, or maybe someone else will feel more enthusiastic about that than I do.

hnlmorg 4y ago
It's possible without any kernel changes. My shell (https://github.com/lmorg/murex https://github.com/lmorg/murex) already supports doing that. The way it works is it uses fd3 to communicate schema information so it can natively support all the existing "dumb" pipes without any modification but any new tools can be written to send objects instead (albeit byte encoded). It's not as elegant as PowerShell sending .NET objects natively, but then PowerShell doesn't work with existing CLI tools natively (it needs wrapper scripts to convert them into PowerShell commands). Whereas my shell is fully backwards compatible while still supporting a suite of additional functionality too.
- skissane 4y ago
  Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor management which only gets triggered when it inherits an extra fd. The developers might never notice because they never run it under your shell, and then someday some user does. An idea I've had before: the shell should listen on a Unix domain socket, and put the path to that in an environment variable. Then commands could talk to the shell which invoked them. That could be used for this kind of content negotiation stuff. It could also be used to do other things – for example, a subprocess could modify the current directory of the user's shell, or its environment – there are scenarios in which being able to do that would be useful. A Unix domain socket path in an environment variable seems less likely to cause unexpected issues than inheriting an extra file descriptor.
  - hnlmorg 4y ago
    Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.
    - skissane 4y ago
      Not trying to be dismissive of your approach either. While I can think of ways it might cause problems in theory, I really have no idea how likely those problems would be in practice. > I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. A socket is just a file descriptor at the end of the day. Some shells (I think some variants of ksh?) have actually used socketpair() to build pipelines instead of pipe(). > the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? The parent process might just be some kind of launcher and the child might be the one that does all the actual work. In any event, what's to stop fd3 being inherited by grandchild processes too?
      - hnlmorg 4y ago
        > A socket is just a file descriptor at the end of the day They are. But I don’t have code to manage sockets whereas I already do manage STDOUT et al so adding another pipe there doesn’t increase the code complexity. It’s a bit of a lazy excuse I’ll grant you. But it also wasn’t the deciding reason, just an added incentive. > Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. I wasn’t aware of this. That’s handy to know. Thank you. > However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? There’s a lot of scope for accidental data corruption. If you’ve just got a byte stream of text then it’s very easy for data to be safely concatenated. But if you’re streaming objects then the last thing you want is is those objects concatenated by dumb POSIX pipes. An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. If the kernel was object aware then this wouldn’t be an issue. But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. > In any event, what's to stop fd3 being inherited by grandchild processes too? Indeed. Just as you can write directly to fd{0..2} on any given process if you wanted. But the developer would have to go out of their way to expose (or locate another processes) fd3. My goal wasn’t to prevent deliberate exposure but rather to prevent accidental unwanted exposure. I do agree there is the risk that some processes might intentionally create an fd3, unaware of my shell. But at the time I felt that risk was lower than the risk of processes accidentally misusing a socket…though I will look into the point you made about the PID of connecting process being available as that might swing my decision the other way.
pjmlp 4y ago
.NET CLR compiles to native code via JIT, and PowerShell also has support for calling into COM and DLL libraries. Additionally the way Xerox and ETHZ workstations did their REPLs was native code all the way, as Interlisp, Smalltalk, Mesa XDE, Mesa/Cedar, Oberon, Oberon-2, AOS all had AOT/JIT workflows.

hnlmorg 4y ago

It's possible without any kernel changes. My shell (https://github.com/lmorg/murex https://github.com/lmorg/murex) already supports doing that. The way it works is it uses fd3 to communicate schema information so it can natively support all the existing "dumb" pipes without any modification but any new tools can be written to send objects instead (albeit byte encoded). It's not as elegant as PowerShell sending .NET objects natively, but then PowerShell doesn't work with existing CLI tools natively (it needs wrapper scripts to convert them into PowerShell commands). Whereas my shell is fully backwards compatible while still supporting a suite of additional functionality too.

skissane 4y ago
Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor management which only gets triggered when it inherits an extra fd. The developers might never notice because they never run it under your shell, and then someday some user does. An idea I've had before: the shell should listen on a Unix domain socket, and put the path to that in an environment variable. Then commands could talk to the shell which invoked them. That could be used for this kind of content negotiation stuff. It could also be used to do other things – for example, a subprocess could modify the current directory of the user's shell, or its environment – there are scenarios in which being able to do that would be useful. A Unix domain socket path in an environment variable seems less likely to cause unexpected issues than inheriting an extra file descriptor.
- hnlmorg 4y ago
  Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.
  - skissane 4y ago
    Not trying to be dismissive of your approach either. While I can think of ways it might cause problems in theory, I really have no idea how likely those problems would be in practice. > I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. A socket is just a file descriptor at the end of the day. Some shells (I think some variants of ksh?) have actually used socketpair() to build pipelines instead of pipe(). > the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? The parent process might just be some kind of launcher and the child might be the one that does all the actual work. In any event, what's to stop fd3 being inherited by grandchild processes too?
    - hnlmorg 4y ago
      > A socket is just a file descriptor at the end of the day They are. But I don’t have code to manage sockets whereas I already do manage STDOUT et al so adding another pipe there doesn’t increase the code complexity. It’s a bit of a lazy excuse I’ll grant you. But it also wasn’t the deciding reason, just an added incentive. > Unix domain sockets let you know the PID/UID/GID at the other end (SO_PEERCRED, SCM_CREDENTIALS, etc). So when the shell gets a connection, it can tell whether it is coming from one of its direct children, or a grandchild. I wasn’t aware of this. That’s handy to know. Thank you. > However, I'm not sure if one actually should restrict subprocess. If a child process spawns a subprocess, why shouldn't the subprocess be allowed to negotiate the input/output format, etc? There’s a lot of scope for accidental data corruption. If you’ve just got a byte stream of text then it’s very easy for data to be safely concatenated. But if you’re streaming objects then the last thing you want is is those objects concatenated by dumb POSIX pipes. An example of this is if one data type aware shell script forks a bash shell script, which in turn forks processes that are again aware of data types. You’d end up having each process forked from bash writing schemas to the socket (because they inherited the env cars from bash) while their actual data is being written in the pipeline. This would lead to corrupted data. If the kernel was object aware then this wouldn’t be an issue. But since we are hacking objects into a byte stream you have to be a little more careful about how the data is segmented. > In any event, what's to stop fd3 being inherited by grandchild processes too? Indeed. Just as you can write directly to fd{0..2} on any given process if you wanted. But the developer would have to go out of their way to expose (or locate another processes) fd3. My goal wasn’t to prevent deliberate exposure but rather to prevent accidental unwanted exposure. I do agree there is the risk that some processes might intentionally create an fd3, unaware of my shell. But at the time I felt that risk was lower than the risk of processes accidentally misusing a socket…though I will look into the point you made about the PID of connecting process being available as that might swing my decision the other way.

pjmlp 4y ago

.NET CLR compiles to native code via JIT, and PowerShell also has support for calling into COM and DLL libraries. Additionally the way Xerox and ETHZ workstations did their REPLs was native code all the way, as Interlisp, Smalltalk, Mesa XDE, Mesa/Cedar, Oberon, Oberon-2, AOS all had AOT/JIT workflows.