4 ms·

I’m an outsider to os Dev and have never read posix standards or similar but I’m curious if an entirely different operating system would achieve benefits worth

I’m an outsider to os Dev and have never read posix standards or similar but I’m curious if an entirely different operating system would achieve benefits worth the cost of incompatibility. Things like the concept of treating everything as a binary stream (shell output, data transfers etc) seem like a design choice with roots in early computing which doesn’t lend well to complex and secure systems. An example is that if we move to shells where everything is an object with a toString method for stdout, we would avoid many bash bugs in piped commands or shell scripts. I’m sure many concepts such as file io etc could achieve different but worthwhile benefits if reengineered from a modern perspective, even tcp/ip has its warts (but I’ll acknowledge we will never get away from tcp/ ip). Am I just a clueless outsider?

bee_rider 4y ago
I’m also not an OS dev. However, sometimes while cutting, pasting, and grepping my program outputs I’m vaguely jealous of powershell I think they use some sort of objects over there…
- dflock 4y ago
  You can install PowerShell on Linux now, if you like - or there's nushell: https://www.nushell.sh/ https://www.nushell.sh/
  - skissane 4y ago
    Two big differences between PowerShell (even on Linux) and traditional Unix shells: (1) multithreaded single process architecture, as opposed to multi-process architecture; (2) runs under a VM (.Net CLR) as opposed to native code. I see nushell is written in Rust, so it doesn't have (2), although I'm less sure about (1). I think it would be cool if you could add structured pipeline support to traditional Unix tools (bash, coreutils, etc). The problem is, Unix pipes don't have any support for passing metadata (performing content negotiation between the two ends of the pipe). Given they are a facility provided by the kernel, adding that would likely require kernel changes. One day I started daydreaming about prototyping content negotiation support for Unix pipes. I was thinking of using CUSE (Linux character device driver in userspace) to do it. Never actually got around to trying though, other more pressing things to work on. Maybe some day I will get around with it, or maybe someone else will feel more enthusiastic about that than I do.
    - hnlmorg 4y ago
      It's possible without any kernel changes. My shell (https://github.com/lmorg/murex https://github.com/lmorg/murex) already supports doing that. The way it works is it uses fd3 to communicate schema information so it can natively support all the existing "dumb" pipes without any modification but any new tools can be written to send objects instead (albeit byte encoded). It's not as elegant as PowerShell sending .NET objects natively, but then PowerShell doesn't work with existing CLI tools natively (it needs wrapper scripts to convert them into PowerShell commands). Whereas my shell is fully backwards compatible while still supporting a suite of additional functionality too.
      - skissane 4y ago
        Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor management which only gets triggered when it inherits an extra fd. The developers might never notice because they never run it under your shell, and then someday some user does. An idea I've had before: the shell should listen on a Unix domain socket, and put the path to that in an environment variable. Then commands could talk to the shell which invoked them. That could be used for this kind of content negotiation stuff. It could also be used to do other things – for example, a subprocess could modify the current directory of the user's shell, or its environment – there are scenarios in which being able to do that would be useful. A Unix domain socket path in an environment variable seems less likely to cause unexpected issues than inheriting an extra file descriptor.
        hnlmorg 4y ago
        Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.
yjftsjthsd-h 4y ago
No, this idea comes up semi regularly. Honestly Windows does alright for that; not POSIX, very object oriented, has powershell which you should look at.
junon 4y ago
No, many OS devs are converging on a similar idea, myself included.
moring 4y ago
> we would avoid many bash bugs in piped commands or shell scripts Food for thoughts: you take a step away from byte-array files. Why stick with the notion of using pipes for composition, or using the same language for the (command-line) UI and for scripting? Or, for that matter, with a universal "file" concept that tries to treat videos, pictures, executables, application-internal data files, database files and system configuration files the same (even before taking into account Unix's "special files" that aren't persistent, and are just high-level APIs sharing the same low-level APIs as actual files).
- skissane 4y ago
  > Food for thoughts: you take a step away from byte-array files. > Or, for that matter, with a universal "file" concept that tries to treat videos, pictures, executables, application-internal data files, database files and system configuration files the same POSIX and Windows have what are called "stream-oriented" or "bytestream" files – as far as the filesystem is concerned, the file is just an unstructured stream of bytes; applications can require those bytes to be in a certain format, but the filesystem is ignorant of that and won't enforce it. Many mainframe/minicomputer operating systems support additional types of files – such as record-oriented files, in which a file is divided into records – either fixed length (every record in file must be same size) or variable length (every record is preceded by a header which stores its length in bytes, sometimes also flags). The filesystem forces all reads/writes to be of whole records (or groups of them), partial reads/writes are prohibited. Some also had indexed files, where you basically have a key-value store built into the filesystem, with each record divided into a key and a value, the filesystem had an API to retrieve records by key instead of location in the file; commonly the file was internally structured as a B-tree or hashtable, but that was hidden from applications, the filesystem looked after those details. The designers of Unix knew about this complexity – they had extensive experience with IBM's OS/360 mainframe operating system which included it – and intentionally decided to avoid it, and give Unix only byte-stream files. The same design choice was made by designers of microcomputer/PC operating systems such as CP/M, MS-DOS and OS/2. Dave Cutler, chief architect of Windows NT, knew all about record-oriented files (OpenVMS has them), but decided against including them in Windows NT. We can question whether those were the right decisions – or, even if they were the right decisions at the time, they are not guaranteed to forever more remain so.
pjmlp 4y ago
No, that is how REPL worked on Xerox workstation environments, and was also copied into ETHZ Oberon based workstations. Nowadays PowerShell is the only experience that ships in the box that is somehow similar to that. Not only you have structured data, by having first class access to .NET, COM and DLLs, it means any OS API or modules from other applications, are directly accessible for scripting, besides some type declarations.
hnlmorg 4y ago
Personally I think a lot of the warts with Bash are more down to it's "language" design rather than pipes. Things like how $VARIABLES are expanded before a statement is compiled into a argument (which leads to spaces in $VARIABLES breaking things) are entirely fixable and quite a number of alternative Linux/UNIX shells do address those kinds of problems directly. That all said, I'm not disagree with your point about smarter pipelines (rather than dumb byte streams) would be advantageous too.
- hsbauauvhabzb 4y ago
  The very notion that $PATH is a list encoded as a colon separated string is a hack ;)
  - hnlmorg 4y ago
    I wasn't disagreeing with your point that there are plenty of rough edges. I was just saying that a lot of common problems are really more Bashisms. Though to address your $PATH point directly, in the literal decades that I've been using Linux and UNIX systems, I can't recall a single time when I've ran into a problem with $PATH values being colon delimited. And the reason it is structured that way is again, to work around a limitation of the shells of that time (ie they didn't support arrays). Though if we are going to talk about problems with environmental variables then $PATH would be somewhere at the bottom of the list while the structure those values are stored at would be top (you think it's a pain parsing $PATH, trying parsing raw env var "structs" -- I've seen far more bugs introduced there) along with the fact that it was never intended to be secure yet that passing secrets seems to be one of it's primary use cases these days.

bee_rider 4y ago

I’m also not an OS dev. However, sometimes while cutting, pasting, and grepping my program outputs I’m vaguely jealous of powershell I think they use some sort of objects over there…

dflock 4y ago
You can install PowerShell on Linux now, if you like - or there's nushell: https://www.nushell.sh/ https://www.nushell.sh/
- skissane 4y ago
  Two big differences between PowerShell (even on Linux) and traditional Unix shells: (1) multithreaded single process architecture, as opposed to multi-process architecture; (2) runs under a VM (.Net CLR) as opposed to native code. I see nushell is written in Rust, so it doesn't have (2), although I'm less sure about (1). I think it would be cool if you could add structured pipeline support to traditional Unix tools (bash, coreutils, etc). The problem is, Unix pipes don't have any support for passing metadata (performing content negotiation between the two ends of the pipe). Given they are a facility provided by the kernel, adding that would likely require kernel changes. One day I started daydreaming about prototyping content negotiation support for Unix pipes. I was thinking of using CUSE (Linux character device driver in userspace) to do it. Never actually got around to trying though, other more pressing things to work on. Maybe some day I will get around with it, or maybe someone else will feel more enthusiastic about that than I do.
  - hnlmorg 4y ago
    It's possible without any kernel changes. My shell (https://github.com/lmorg/murex https://github.com/lmorg/murex) already supports doing that. The way it works is it uses fd3 to communicate schema information so it can natively support all the existing "dumb" pipes without any modification but any new tools can be written to send objects instead (albeit byte encoded). It's not as elegant as PowerShell sending .NET objects natively, but then PowerShell doesn't work with existing CLI tools natively (it needs wrapper scripts to convert them into PowerShell commands). Whereas my shell is fully backwards compatible while still supporting a suite of additional functionality too.
    - skissane 4y ago
      Interesting idea. Although I worry about relying on fd3, it seems like it could be a bit fragile. An executable might have a bug in its file descriptor management which only gets triggered when it inherits an extra fd. The developers might never notice because they never run it under your shell, and then someday some user does. An idea I've had before: the shell should listen on a Unix domain socket, and put the path to that in an environment variable. Then commands could talk to the shell which invoked them. That could be used for this kind of content negotiation stuff. It could also be used to do other things – for example, a subprocess could modify the current directory of the user's shell, or its environment – there are scenarios in which being able to do that would be useful. A Unix domain socket path in an environment variable seems less likely to cause unexpected issues than inheriting an extra file descriptor.
      - hnlmorg 4y ago
        Yeah that's another possible approach. I did consider that myself but I didn't want to manage sockets in addition to processes, plus file descriptors would be idiomatic to pipelines. Another issue was that if a child process then spawns another process, the environmental variables might also be carried over, so how does that grandchild process get restricted from writing to the socket? (it is a fixable problem but fd's solved that problem a little easier). Please don't take this as a dismissive comment though. I'm definitely not suggesting that your method has more problems nor that my method is better. There is a lot I do like about your suggestion and it might be an option I pivot to if/when I do run into issues with fd3.

yjftsjthsd-h 4y ago

No, this idea comes up semi regularly. Honestly Windows does alright for that; not POSIX, very object oriented, has powershell which you should look at.

junon 4y ago

No, many OS devs are converging on a similar idea, myself included.

moring 4y ago

> we would avoid many bash bugs in piped commands or shell scripts Food for thoughts: you take a step away from byte-array files. Why stick with the notion of using pipes for composition, or using the same language for the (command-line) UI and for scripting? Or, for that matter, with a universal "file" concept that tries to treat videos, pictures, executables, application-internal data files, database files and system configuration files the same (even before taking into account Unix's "special files" that aren't persistent, and are just high-level APIs sharing the same low-level APIs as actual files).

skissane 4y ago
> Food for thoughts: you take a step away from byte-array files. > Or, for that matter, with a universal "file" concept that tries to treat videos, pictures, executables, application-internal data files, database files and system configuration files the same POSIX and Windows have what are called "stream-oriented" or "bytestream" files – as far as the filesystem is concerned, the file is just an unstructured stream of bytes; applications can require those bytes to be in a certain format, but the filesystem is ignorant of that and won't enforce it. Many mainframe/minicomputer operating systems support additional types of files – such as record-oriented files, in which a file is divided into records – either fixed length (every record in file must be same size) or variable length (every record is preceded by a header which stores its length in bytes, sometimes also flags). The filesystem forces all reads/writes to be of whole records (or groups of them), partial reads/writes are prohibited. Some also had indexed files, where you basically have a key-value store built into the filesystem, with each record divided into a key and a value, the filesystem had an API to retrieve records by key instead of location in the file; commonly the file was internally structured as a B-tree or hashtable, but that was hidden from applications, the filesystem looked after those details. The designers of Unix knew about this complexity – they had extensive experience with IBM's OS/360 mainframe operating system which included it – and intentionally decided to avoid it, and give Unix only byte-stream files. The same design choice was made by designers of microcomputer/PC operating systems such as CP/M, MS-DOS and OS/2. Dave Cutler, chief architect of Windows NT, knew all about record-oriented files (OpenVMS has them), but decided against including them in Windows NT. We can question whether those were the right decisions – or, even if they were the right decisions at the time, they are not guaranteed to forever more remain so.

pjmlp 4y ago

No, that is how REPL worked on Xerox workstation environments, and was also copied into ETHZ Oberon based workstations. Nowadays PowerShell is the only experience that ships in the box that is somehow similar to that. Not only you have structured data, by having first class access to .NET, COM and DLLs, it means any OS API or modules from other applications, are directly accessible for scripting, besides some type declarations.

hnlmorg 4y ago

Personally I think a lot of the warts with Bash are more down to it's "language" design rather than pipes. Things like how $VARIABLES are expanded before a statement is compiled into a argument (which leads to spaces in $VARIABLES breaking things) are entirely fixable and quite a number of alternative Linux/UNIX shells do address those kinds of problems directly. That all said, I'm not disagree with your point about smarter pipelines (rather than dumb byte streams) would be advantageous too.

hsbauauvhabzb 4y ago
The very notion that $PATH is a list encoded as a colon separated string is a hack ;)
- hnlmorg 4y ago
  I wasn't disagreeing with your point that there are plenty of rough edges. I was just saying that a lot of common problems are really more Bashisms. Though to address your $PATH point directly, in the literal decades that I've been using Linux and UNIX systems, I can't recall a single time when I've ran into a problem with $PATH values being colon delimited. And the reason it is structured that way is again, to work around a limitation of the shells of that time (ie they didn't support arrays). Though if we are going to talk about problems with environmental variables then $PATH would be somewhere at the bottom of the list while the structure those values are stored at would be top (you think it's a pain parsing $PATH, trying parsing raw env var "structs" -- I've seen far more bugs introduced there) along with the fact that it was never intended to be secure yet that passing secrets seems to be one of it's primary use cases these days.