6 ms·

It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever ha

It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever have so many pitfalls. Take n random languages and m random developers and tell them to loop over a string array and print its contents, and count how many correct programs you get on average per language. There will be easy languages, then difficult languages, then a huge gap, then Unix shell because in your random sample you managed to get one guy who has PhD in bash.

blueflow 2y ago
Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell. I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.
- anal_reactor 2y ago
  The bigger issue is the sheer momentum of Unix shell. Even if you come up with an alternative that is better by every objectively measurable metric, it's still going to be a monumental task to have it packages with commonly used distros. Kinda like the "why can't the US switch to the metric system" problem.
  - blueflow 2y ago
    People already use different shells, mksh, fish, and so on. With fish there is a non-posix shell in wide use.
    - oguz-ismail 2y ago
      >wide use Five people around the globe isn't wide use.
      - blueflow 2y ago
        I'm sure you might get more than 5 people on HN replying to you that they are using fish right now. Say something discrediting about fish and they show up.
        fragmede 2y ago
        Heh, reminds me of how to get help with Linux back in the day. If you directly asked for help, you'd be told to RTFM. If you stayed confidently that Windows could do something and that Linux sucks because it can't, you'd get users tripping over themselves with details and instructions,'just to prove you wrong. Human psychology is fascinating!
  - azalemeth 2y ago
    There's a direct cost in money, time and lives that has come from the US's adherence to their US Customary Units (which are often different to the old imperial units). People have literally died because of the confusion caused by having multiple systems of units in common use with ambiguous names (degrees, gallons, etc). Each year industry worldwide spends an enormous amount of money indirectly precisely because of this problem and it's still incredibly unlikely to be fixed within my lifetime. Bash-alternatives that are not completely compatible frankly just don't have a chance.
  - stephenr 2y ago
    If it isn't distributed out of the box with every nix-like OS, it inherently isn't* “better by every objectively measurable metric" - distribution of a common, stable standard is a huge benefit in and of itself.
    - blueflow 2y ago
      > distributed out of the box with every nix-like OS, Python and lua are pretty close to that.
      - stephenr 2y ago
        > Python and lua are pretty close to that. Python maybe often installed by default but it's definitely not an essential/required package "out of the box" on every install. Also, in a thread where one topic is how POSIX shell handles whitespace in filenames, it's hilarious (not in a good way) that someone suggests a language that handles whitespace the wrong way in it's own code. Yes, significant whitespace is objectively wrong. What OS/distro is Lua included on out of the box? That doesn't mean "available in a package". I mean literally included in every single install and cannot reasonably be omitted? Regardless of the availability, the parent comment says > better by every objectively measurable metric Neither Python nor Lua are "better" than shell, at the types of things shell is commonly used for - they're objectively worse.
        blueflow 2y ago
        Lua gets onto every other Linux distro as dependency of some base system component. For example, rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default. You should use the word "objectively" less.
        stephenr 2y ago
        > Lua gets onto every other Linux distro Just FYI, there are UNIX-like, POSIX compatible systems that are not a Linux distro. > rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default. Pipewire? Do you mean this? https://packages.debian.org/bookworm/pipewire https://packages.debian.org/bookworm/pipewire That isn't even close to "installed on every system". Best I can tell from the reverse dependencies, it's required for some Gnome Remote Desktop tool, and best I can tell, it doesn't rely on Lua anyway (at least on Debian). > You should use the word "objectively" less. I specifically used the word objectively, because the original comment that I replied to, said this: > better by every objectively measurable metric
  - Netch 2y ago
    OK let them add an explicit check to standard tools, and/or to open(), mkdir(), etc. with O_PORTABLECHARS. And an environment option to disable this check. Why they force the restriction at syscall level?
- nly 2y ago
  Oil shell? https://www.oilshell.org/ https://www.oilshell.org/ Compatible with most bash scripts
- throw16180339 2y ago
  I certainly have my complaints about Powershell, but it's got pretty good coverage, decent documentation, and cross platform support.
  - felixgallo 2y ago
    if it weren't so irregular, inconsistent, spotty and tasteless, it'd be a great option.
throwaway19972 2y ago
> the design of Unix shell is bonkers Compared to what?
- mdavid626 2y ago
  Powershell?
  - oguz-ismail 2y ago
    Verbosity is a huge problem there
    - consteval 2y ago
      Modern programming language designers have a bad relationship with verbosity. I don't know why they do this. It's a lang for an interactive shell, typing literally translates to developer speed. I understand the want for clarity and maybe that's nice in large scripts, but the main goal is to be a shell. So, optimize for that. Also, you probably shouldn't be using powershell for large scripts anyway. The only recent lang I've seen that has a handle on this is Rust. You can tell they put a lot of thought into having keywords be as short as possible while still being descriptive.
  - ggm 2y ago
    FoundTheCamelCaseConvert. My God next you will say getopt() --longform is the bestest
    - throw16180339 2y ago
      It's been years since I used Powershell, but IIRC there are shortcuts for the common commands, e.g. cat, ls, mv, rm, and such DTRT.
      - Diti 2y ago
        Those aliases are, I believe, only defined on Windows PowerShell (the closed-source version 5; not PowerShell 7). I wish those default aliases you mentioned weren’t a thing. Especially `curl` (people should use `iwr` instead), which is an alias of `Invoke-WebRequest`, because it makes the `curl.exe` shipped with Windows nearly undiscoverable.
enriquto 2y ago
> loop over a string array Dear anal_reactor, what is a "string array"? I have used unix shells since nearly 30 years and never heard about them. And I consider myself a script-fu master! There are two array-like constructions in the shell: list of words (separated by spaces) and list of lines (separated by newlines). Both cases are implemented as a single string, and the shell makes it trivial to iterate through its components.
- ManBeardPc 2y ago
  That is exactly the problem many people have with it. Encoding „arrays“ this way is foreign to everyone who comes from „normal“ programming languages. Both variants lead to problems because either character can occur in elements, worst case scenario they contain both at the same time. I can see why this leads to confusion and bugs.
  - skydhash 2y ago
    It’s like people saying they won’t learn French because it has a different grammatical structure. There’s no “normal” natural language. If you’re used to the C-like syntax, learning C-like language will be easy. But that’s not an argument to say Lisp is confusing.
    - ManBeardPc 2y ago
      That's why I put normal in quotes. There is however more to it than having a different grammatical structure: It works different from many commonly used languages that have actual arrays/lists where elements can contain anything the type allows. If you come from any of the common modern programming languages (lets say Java, Kotlin, C#, JS/TS, Python, Swift, Go, Rust, etc.) and expect something similar (because many of them are very similar) you will be confused. Using spaces or newlines to encode elements in a single string is just not robust and leads to easy to make mistakes.
      - skydhash 2y ago
        Most of these languages were created long after bash and the other shells. The fact is that shell scripts allows for unquoted strings and quoting is a specific operation, not syntax. Also shell scripts were meant for automations, not for writing general programs. The basic units are commands, arguments, input, output, files,… so the design makes these easy to manipulate. I’m not saying that we can’t improve, but I’m more in favor of making the tool more apt to solve a problem than making it easier to learn. Because the latter often wants to forego the requirement of understanding the problem space.
dailykoder 2y ago
Works on my machine!
akira2501 2y ago
> I haven't seen any other tool ever have so many pitfalls. I haven't seen any other tool with so much general utility and availability. > to loop over a string array and print its contents Is incredibly easy in bash and bash like shells. As highlighted the issue is that tools like 'ls' don't create "a string array." They create one giant string that has to be parsed. The rules in the shell are different than in other languages but it /will/ do most of the parsing for you, or all of it, if you do it carefully. This is a fine tradeoff. As evidenced by it's wide usage and lack of convincing replacements.
- anal_reactor 2y ago
  > I haven't seen any other tool with so much general utility and availability. > availability That's the real reason why we use Unix shell. It's not good, but it's available. Like a cheap hooker. > but it /will/ do most of the parsing for you, or all of it, if you do it carefully. "It mostly works if you're careful" doesn't sound very convincing to me.
  - stephenr 2y ago
    > but it's available. Like a cheap hooker. Username checks out.
  - akira2501 2y ago
    > "It mostly works if you're careful" doesn't sound very convincing to me. Would you rather write your own parser?
vbezhenar 2y ago
The main problem is using text as a common format between different applications. First: text is not well defined. Is it ASCII? Is it UTF-8? Some programs can spew UTF-32 with proper locale configured, it's a mess. Second: encoding and decoding of objects to text is not defined at all. Those problems with filenames is just one example. Using newline as a separator is a natural thing that is easy to implement, yet it is wrong. In my opinion two things should be done: 1. Standardise on UTF-8. No other encodings allowed. 2. Standardise on JSON. It is good enough to serve as universal exchange format, tools like `jq` exist for some time now. So any utility must read and write JSON objects with some standard env set. And shells can be developed with better syntax to deal with JSON. This way you can write something like `ps aux | while read row; do echo ${row.user} ${row.pid}; done`
- ezoe 2y ago
  Don't even assume UTF-something is the only character encoding. There are so many existing character encodings before Unicode. It's still widely used.
- nly 2y ago
  The primary purpose of command line program output is to convey information to a human, not to other programs. Command line scripting is supposed to be adhoc and hack.
  - mdavid626 2y ago
    I disagree that it supposed to be adhoc and hack. Look at PowerShell.
  - anthk 2y ago
    That under limited OSes such as DOS. Under Unix, piping has been the philosophy.
  - consteval 2y ago
    There are exchange formats that are well-defined enough to be useful to many computers while also being readable enough to be traversed by human eyes. There's no reason to everything ad-hoc, you don't get much by that. You also control the shell itself - there's no reason you can't display object representations in a pretty way.
zelphirkalt 2y ago
This should not be as downvoted as it is. In a way shell is broken. The brokenness is in that it requires each command to serialize and deserialize again, considering all the weird things that can happen with the "all is a string" kind of approach, instead of having a proper data interchange format or even sending objects to next steps in the pipeline. This behavior is what necessitates even thinking about the changes listed in the post. We wouldn't even have that problem, if the design of shell was better thought out. Now we are dealing with decades of legacy built on these shaky foundations. I hate to admit it, but seems at least this aspect Powershell got right, whatever one may think about the rest of it.
- chasil 2y ago
  On my rhel7 system, the Debian dash shell is this large: $ ll /bin/dash -rwxr-xr-x. 1 root root 113536 Nov 5 2018 /bin/dash I happen to have an old powershell installed: $ rpm -qi powershell | grep Size Size : 126588370 A strict POSIX shell is always going to be vastly smaller, for many reasons. I would prefer that the POSIX shell was an LR-parsed language, but you can't have everything.
dangsux 2y ago
[dead]

blueflow 2y ago

Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell. I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.

anal_reactor 2y ago
The bigger issue is the sheer momentum of Unix shell. Even if you come up with an alternative that is better by every objectively measurable metric, it's still going to be a monumental task to have it packages with commonly used distros. Kinda like the "why can't the US switch to the metric system" problem.
- blueflow 2y ago
  People already use different shells, mksh, fish, and so on. With fish there is a non-posix shell in wide use.
  - oguz-ismail 2y ago
    >wide use Five people around the globe isn't wide use.
    - blueflow 2y ago
      I'm sure you might get more than 5 people on HN replying to you that they are using fish right now. Say something discrediting about fish and they show up.
      - fragmede 2y ago
        Heh, reminds me of how to get help with Linux back in the day. If you directly asked for help, you'd be told to RTFM. If you stayed confidently that Windows could do something and that Linux sucks because it can't, you'd get users tripping over themselves with details and instructions,'just to prove you wrong. Human psychology is fascinating!
- azalemeth 2y ago
  There's a direct cost in money, time and lives that has come from the US's adherence to their US Customary Units (which are often different to the old imperial units). People have literally died because of the confusion caused by having multiple systems of units in common use with ambiguous names (degrees, gallons, etc). Each year industry worldwide spends an enormous amount of money indirectly precisely because of this problem and it's still incredibly unlikely to be fixed within my lifetime. Bash-alternatives that are not completely compatible frankly just don't have a chance.
- stephenr 2y ago
  If it isn't distributed out of the box with every nix-like OS, it inherently isn't* “better by every objectively measurable metric" - distribution of a common, stable standard is a huge benefit in and of itself.
  - blueflow 2y ago
    > distributed out of the box with every nix-like OS, Python and lua are pretty close to that.
    - stephenr 2y ago
      > Python and lua are pretty close to that. Python maybe often installed by default but it's definitely not an essential/required package "out of the box" on every install. Also, in a thread where one topic is how POSIX shell handles whitespace in filenames, it's hilarious (not in a good way) that someone suggests a language that handles whitespace the wrong way in it's own code. Yes, significant whitespace is objectively wrong. What OS/distro is Lua included on out of the box? That doesn't mean "available in a package". I mean literally included in every single install and cannot reasonably be omitted? Regardless of the availability, the parent comment says > better by every objectively measurable metric Neither Python nor Lua are "better" than shell, at the types of things shell is commonly used for - they're objectively worse.
      - blueflow 2y ago
        Lua gets onto every other Linux distro as dependency of some base system component. For example, rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default. You should use the word "objectively" less.
        stephenr 2y ago
        > Lua gets onto every other Linux distro Just FYI, there are UNIX-like, POSIX compatible systems that are not a Linux distro. > rpm or pipewire depend on lua. Ubuntu and Debian ship with pipewire per default. Pipewire? Do you mean this? https://packages.debian.org/bookworm/pipewire https://packages.debian.org/bookworm/pipewire That isn't even close to "installed on every system". Best I can tell from the reverse dependencies, it's required for some Gnome Remote Desktop tool, and best I can tell, it doesn't rely on Lua anyway (at least on Debian). > You should use the word "objectively" less. I specifically used the word objectively, because the original comment that I replied to, said this: > better by every objectively measurable metric
- Netch 2y ago
  OK let them add an explicit check to standard tools, and/or to open(), mkdir(), etc. with O_PORTABLECHARS. And an environment option to disable this check. Why they force the restriction at syscall level?
nly 2y ago
Oil shell? https://www.oilshell.org/ https://www.oilshell.org/ Compatible with most bash scripts
throw16180339 2y ago
I certainly have my complaints about Powershell, but it's got pretty good coverage, decent documentation, and cross platform support.
- felixgallo 2y ago
  if it weren't so irregular, inconsistent, spotty and tasteless, it'd be a great option.

throwaway19972 2y ago

> the design of Unix shell is bonkers Compared to what?

mdavid626 2y ago
Powershell?
- oguz-ismail 2y ago
  Verbosity is a huge problem there
  - consteval 2y ago
    Modern programming language designers have a bad relationship with verbosity. I don't know why they do this. It's a lang for an interactive shell, typing literally translates to developer speed. I understand the want for clarity and maybe that's nice in large scripts, but the main goal is to be a shell. So, optimize for that. Also, you probably shouldn't be using powershell for large scripts anyway. The only recent lang I've seen that has a handle on this is Rust. You can tell they put a lot of thought into having keywords be as short as possible while still being descriptive.
- ggm 2y ago
  FoundTheCamelCaseConvert. My God next you will say getopt() --longform is the bestest
  - throw16180339 2y ago
    It's been years since I used Powershell, but IIRC there are shortcuts for the common commands, e.g. cat, ls, mv, rm, and such DTRT.
    - Diti 2y ago
      Those aliases are, I believe, only defined on Windows PowerShell (the closed-source version 5; not PowerShell 7). I wish those default aliases you mentioned weren’t a thing. Especially `curl` (people should use `iwr` instead), which is an alias of `Invoke-WebRequest`, because it makes the `curl.exe` shipped with Windows nearly undiscoverable.

enriquto 2y ago

> loop over a string array Dear anal_reactor, what is a "string array"? I have used unix shells since nearly 30 years and never heard about them. And I consider myself a script-fu master! There are two array-like constructions in the shell: list of words (separated by spaces) and list of lines (separated by newlines). Both cases are implemented as a single string, and the shell makes it trivial to iterate through its components.

ManBeardPc 2y ago
That is exactly the problem many people have with it. Encoding „arrays“ this way is foreign to everyone who comes from „normal“ programming languages. Both variants lead to problems because either character can occur in elements, worst case scenario they contain both at the same time. I can see why this leads to confusion and bugs.
- skydhash 2y ago
  It’s like people saying they won’t learn French because it has a different grammatical structure. There’s no “normal” natural language. If you’re used to the C-like syntax, learning C-like language will be easy. But that’s not an argument to say Lisp is confusing.
  - ManBeardPc 2y ago
    That's why I put normal in quotes. There is however more to it than having a different grammatical structure: It works different from many commonly used languages that have actual arrays/lists where elements can contain anything the type allows. If you come from any of the common modern programming languages (lets say Java, Kotlin, C#, JS/TS, Python, Swift, Go, Rust, etc.) and expect something similar (because many of them are very similar) you will be confused. Using spaces or newlines to encode elements in a single string is just not robust and leads to easy to make mistakes.
    - skydhash 2y ago
      Most of these languages were created long after bash and the other shells. The fact is that shell scripts allows for unquoted strings and quoting is a specific operation, not syntax. Also shell scripts were meant for automations, not for writing general programs. The basic units are commands, arguments, input, output, files,… so the design makes these easy to manipulate. I’m not saying that we can’t improve, but I’m more in favor of making the tool more apt to solve a problem than making it easier to learn. Because the latter often wants to forego the requirement of understanding the problem space.

dailykoder 2y ago

Works on my machine!

akira2501 2y ago

> I haven't seen any other tool ever have so many pitfalls. I haven't seen any other tool with so much general utility and availability. > to loop over a string array and print its contents Is incredibly easy in bash and bash like shells. As highlighted the issue is that tools like 'ls' don't create "a string array." They create one giant string that has to be parsed. The rules in the shell are different than in other languages but it /will/ do most of the parsing for you, or all of it, if you do it carefully. This is a fine tradeoff. As evidenced by it's wide usage and lack of convincing replacements.

anal_reactor 2y ago
> I haven't seen any other tool with so much general utility and availability. > availability That's the real reason why we use Unix shell. It's not good, but it's available. Like a cheap hooker. > but it /will/ do most of the parsing for you, or all of it, if you do it carefully. "It mostly works if you're careful" doesn't sound very convincing to me.
- stephenr 2y ago
  > but it's available. Like a cheap hooker. Username checks out.
- akira2501 2y ago
  > "It mostly works if you're careful" doesn't sound very convincing to me. Would you rather write your own parser?

vbezhenar 2y ago

The main problem is using text as a common format between different applications. First: text is not well defined. Is it ASCII? Is it UTF-8? Some programs can spew UTF-32 with proper locale configured, it's a mess. Second: encoding and decoding of objects to text is not defined at all. Those problems with filenames is just one example. Using newline as a separator is a natural thing that is easy to implement, yet it is wrong. In my opinion two things should be done: 1. Standardise on UTF-8. No other encodings allowed. 2. Standardise on JSON. It is good enough to serve as universal exchange format, tools like `jq` exist for some time now. So any utility must read and write JSON objects with some standard env set. And shells can be developed with better syntax to deal with JSON. This way you can write something like `ps aux | while read row; do echo ${row.user} ${row.pid}; done`

ezoe 2y ago
Don't even assume UTF-something is the only character encoding. There are so many existing character encodings before Unicode. It's still widely used.
nly 2y ago
The primary purpose of command line program output is to convey information to a human, not to other programs. Command line scripting is supposed to be adhoc and hack.
- mdavid626 2y ago
  I disagree that it supposed to be adhoc and hack. Look at PowerShell.
- anthk 2y ago
  That under limited OSes such as DOS. Under Unix, piping has been the philosophy.
- consteval 2y ago
  There are exchange formats that are well-defined enough to be useful to many computers while also being readable enough to be traversed by human eyes. There's no reason to everything ad-hoc, you don't get much by that. You also control the shell itself - there's no reason you can't display object representations in a pretty way.

zelphirkalt 2y ago

This should not be as downvoted as it is. In a way shell is broken. The brokenness is in that it requires each command to serialize and deserialize again, considering all the weird things that can happen with the "all is a string" kind of approach, instead of having a proper data interchange format or even sending objects to next steps in the pipeline. This behavior is what necessitates even thinking about the changes listed in the post. We wouldn't even have that problem, if the design of shell was better thought out. Now we are dealing with decades of legacy built on these shaky foundations. I hate to admit it, but seems at least this aspect Powershell got right, whatever one may think about the rest of it.

chasil 2y ago
On my rhel7 system, the Debian dash shell is this large: $ ll /bin/dash -rwxr-xr-x. 1 root root 113536 Nov 5 2018 /bin/dash I happen to have an old powershell installed: $ rpm -qi powershell | grep Size Size : 126588370 A strict POSIX shell is always going to be vastly smaller, for many reasons. I would prefer that the POSIX shell was an LR-parsed language, but you can't have everything.

dangsux 2y ago

[dead]