4 ms·

Every time I write software, in the beginning there are command line flags. Then someone goes, can I put them in a file. Then later, someone comes along and goe

Every time I write software, in the beginning there are command line flags. Then someone goes, can I put them in a file. Then later, someone comes along and goes, it would be nice if that file could be dynamic in some way (i.e. let me insert stuff from environment variables). The next request is usually something like, could you let me have conditionals, so if SOME_VARIABLE is true one thing happens, and if false, another. Then someone comes along and goes, THIS IS DANGEROUS, probably NOT VERY SECURE, and your FORMAT SUCKS because you use some character I have to escape when templating, so could you use something modern and trendy, that totally doesn't suck like XML (or whatever format is in the engineering doghouse)? What is funny is that a shell script does a pretty good job at giving you a nice, programmable way to invoke software. But, that is too complicated :-)

stelonix 6y ago
I believe what puts people off shell (myself included) are the various arcane caveats (eg. spaces after or before operators) and little language quirks. It does not seem intuitive and since there are better alternatives like Python, I won't bother to learn some 70s string-based language.
- indymike 6y ago
  The irony in this is that I'm pretty sure that 70's era string-based languages are better designed than many, many configuration file formats.
- pwm 6y ago
  I highly recommend https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck It's a wonderful piece of software to help and guide users writing shell scripts. Hook it up to your editor/IDE and all those arcane caveats will be things of the past.
- chubot 6y ago
  I agree that's what puts people off, and that's the rationale for Oil, an upgrade of Unix shell. In fact I mention the spaces issue in the The Simplest Explanation of Oil: https://www.oilshell.org/blog/2020/01/simplest-explanation.html https://www.oilshell.org/blog/2020/01/simplest-explanation.h... More: http://www.oilshell.org/blog/2021/01/why-a-new-shell.html http://www.oilshell.org/blog/2021/01/why-a-new-shell.html You can use it right now as a dev tool. If you use ShellCheck to statically check it, then running your script under Oil is complementary (and it also has some static checks): https://www.oilshell.org/why.html https://www.oilshell.org/why.html
  - stelonix 6y ago
    Yeah I'm a big fan of Oil and hopefully in 10-20 years it'll be the standard language of shells. Also I did notice the spaces issues when attempting shell but it was only from reading your post sometime ago that I found out it was a deliberate language design choice.
    - chubot 6y ago
      It should be usable for "cloud config" files long before that! :) But yes changing existing code at the lower distro level has a lot more inertia. The cloud is basically built on top of Linux distros, so that part is easier to change and is rapidly evolving. You could say that the = issue is deliberate, but I'd say the core problem is that shell didn't start out as a programming language, or at least it was a very impoverished one without variables. The original paper from the 70's on the Thompson shell shows that. Shell had "goto" but no variables! So name=value had to be grafted on later without breaking too many things. Words were already split by spaces, so I guess they just made name=value a "pseudo-word" that becomes an assignment, whereas name = value remained 3 words.
spaetzleesser 6y ago
"The next request is usually something like, could you let me have conditionals, so if SOME_VARIABLE is true one thing happens, and if false, another." I remember one project that started out with simple XML config, then I added conditionals. when I was starting o work on loops and reusable variables I realized that I was writing a programming language in XML. So I started to write config in C# and compiled that dynamically instead.
TimTheTinker 6y ago
This is actually one reason Ruby DSLs became so popular. You could take config from an XML file, translate it to the DSL, and have all the basic Ruby features (flow control included) available as well. For example, there was an ANT library for JRuby that worked super well for using ANT constructs/libraries but in a sane language. What's old is new again :)
sgtnoodle 6y ago
Lol, this seems pretty accurate. I implemented a simulation framework once, where the core simulation process took in all parameters as command line options. A runner program would read a YAML file describing the parameters in arbitrarily complex ways, and invoke the sim process potentially thousands of times (doing parameter sweeps). The intent was for the underlying sim process to always be directly runnable for debugging just by copy/pasting the generated list of options, regardless of how complex the config file got. It was a year or so before somebody started passing in an intermediate config file to the sim process by command line argument...
chubot 6y ago
What is funny is that a shell script does a pretty good job at giving you a nice, programmable way to invoke software. But, that is too complicated :-) Ha, I agree, although I also think shell is a bit impoverished. It works but there are valid reasons people don't use it. I hope to add the "missing declarative part" to shell in https://www.oilshell.org https://www.oilshell.org : https://lobste.rs/s/6oxpe3/s_lot_yaml#c_mje209 https://lobste.rs/s/6oxpe3/s_lot_yaml#c_mje209 Prediction: we'll see a lot more shell embedded in YAML in the coming years, with the same examples shown in the original article (Github Actions, didn't know about Helm Charts) IMO we should get rid of the YAML! https://lobste.rs/s/v4crap/crustaceans_2021_will_be_year_technology#c_ker4jn https://lobste.rs/s/v4crap/crustaceans_2021_will_be_year_tec...

stelonix 6y ago

I believe what puts people off shell (myself included) are the various arcane caveats (eg. spaces after or before operators) and little language quirks. It does not seem intuitive and since there are better alternatives like Python, I won't bother to learn some 70s string-based language.

indymike 6y ago
The irony in this is that I'm pretty sure that 70's era string-based languages are better designed than many, many configuration file formats.
pwm 6y ago
I highly recommend https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck It's a wonderful piece of software to help and guide users writing shell scripts. Hook it up to your editor/IDE and all those arcane caveats will be things of the past.
chubot 6y ago
I agree that's what puts people off, and that's the rationale for Oil, an upgrade of Unix shell. In fact I mention the spaces issue in the The Simplest Explanation of Oil: https://www.oilshell.org/blog/2020/01/simplest-explanation.html https://www.oilshell.org/blog/2020/01/simplest-explanation.h... More: http://www.oilshell.org/blog/2021/01/why-a-new-shell.html http://www.oilshell.org/blog/2021/01/why-a-new-shell.html You can use it right now as a dev tool. If you use ShellCheck to statically check it, then running your script under Oil is complementary (and it also has some static checks): https://www.oilshell.org/why.html https://www.oilshell.org/why.html
- stelonix 6y ago
  Yeah I'm a big fan of Oil and hopefully in 10-20 years it'll be the standard language of shells. Also I did notice the spaces issues when attempting shell but it was only from reading your post sometime ago that I found out it was a deliberate language design choice.
  - chubot 6y ago
    It should be usable for "cloud config" files long before that! :) But yes changing existing code at the lower distro level has a lot more inertia. The cloud is basically built on top of Linux distros, so that part is easier to change and is rapidly evolving. You could say that the = issue is deliberate, but I'd say the core problem is that shell didn't start out as a programming language, or at least it was a very impoverished one without variables. The original paper from the 70's on the Thompson shell shows that. Shell had "goto" but no variables! So name=value had to be grafted on later without breaking too many things. Words were already split by spaces, so I guess they just made name=value a "pseudo-word" that becomes an assignment, whereas name = value remained 3 words.

spaetzleesser 6y ago

"The next request is usually something like, could you let me have conditionals, so if SOME_VARIABLE is true one thing happens, and if false, another." I remember one project that started out with simple XML config, then I added conditionals. when I was starting o work on loops and reusable variables I realized that I was writing a programming language in XML. So I started to write config in C# and compiled that dynamically instead.

TimTheTinker 6y ago

This is actually one reason Ruby DSLs became so popular. You could take config from an XML file, translate it to the DSL, and have all the basic Ruby features (flow control included) available as well. For example, there was an ANT library for JRuby that worked super well for using ANT constructs/libraries but in a sane language. What's old is new again :)

sgtnoodle 6y ago

Lol, this seems pretty accurate. I implemented a simulation framework once, where the core simulation process took in all parameters as command line options. A runner program would read a YAML file describing the parameters in arbitrarily complex ways, and invoke the sim process potentially thousands of times (doing parameter sweeps). The intent was for the underlying sim process to always be directly runnable for debugging just by copy/pasting the generated list of options, regardless of how complex the config file got. It was a year or so before somebody started passing in an intermediate config file to the sim process by command line argument...

chubot 6y ago

What is funny is that a shell script does a pretty good job at giving you a nice, programmable way to invoke software. But, that is too complicated :-) Ha, I agree, although I also think shell is a bit impoverished. It works but there are valid reasons people don't use it. I hope to add the "missing declarative part" to shell in https://www.oilshell.org https://www.oilshell.org : https://lobste.rs/s/6oxpe3/s_lot_yaml#c_mje209 https://lobste.rs/s/6oxpe3/s_lot_yaml#c_mje209 Prediction: we'll see a lot more shell embedded in YAML in the coming years, with the same examples shown in the original article (Github Actions, didn't know about Helm Charts) IMO we should get rid of the YAML! https://lobste.rs/s/v4crap/crustaceans_2021_will_be_year_technology#c_ker4jn https://lobste.rs/s/v4crap/crustaceans_2021_will_be_year_tec...