5 ms·

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and hel

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck

aidenn0 7y ago
Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/ https://www.oilshell.org/
- butterthebuddha 7y ago
  I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.
  - chubot 7y ago
    (Oil author here) Yes I think that's a great idea, and I hope to release a "liboil" so that people can do that. Let a thousand flowers bloom, etc. The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like: https://www.oilshell.org/blog/2020/01/making-plans.html https://www.oilshell.org/blog/2020/01/making-plans.html So I encourage anyone who's interested in that to get involved now as the interactive shell is also a years-long effort :) There are many links on the home page about how to get involved, and I wrote many blog posts so people can understand how it works. ---- Here are a couple posts on why Oil is a good foundation for an interactive shell: https://www.oilshell.org/blog/2020/01/history-and-completion.html https://www.oilshell.org/blog/2020/01/history-and-completion... https://www.oilshell.org/blog/2020/01/alias-and-prompt.html https://www.oilshell.org/blog/2020/01/alias-and-prompt.html
    - silly_giraffe 7y ago
      I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.
      - chubot 7y ago
        Thank you! Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program. I think Linux, git, and GNU are the prototypical examples of the latter. Those projects are too big for one person, so the code is set up in a way to allow hacking locally. They're also arguably a mess. Anyone who's ever written against the container APIs in Linux knows what I mean (namespaces, cgroups, etc.) The git UI is another famous example, i.e. the plethora of commands and flags that make little sense. Empirically it seems that you can move faster if you disregard consistency and coherence. The shell language of course grew like that over 50 years. Bourne shell is pretty consistent, but one thing I learned is that ksh added a lot of bash misfeatures (they're ksh-isms not bash-isms), and it's pretty bad. But shell is actually a significantly smaller problem than git (after all bash is maintained more or less by one person). So it should be possible to fix it and redesign it. Unfortunately I'm not sure if the lessons generalize to bigger projects, but I would like it if they do. ----- On a more practical note, feel free to send more people this way if you want to see the project succeed :) The design is there, and it will hold up, but it needs a bit more manpower, especially for things like the interactive shell.
        kortilla 7y ago
        The trick is separating out external facing (api/ux) consistency from code style/data-structure/etc consistency. It doesn’t really matter if you don’t have coherency across the whole project for the latter because that can easily be swapped out later if it’s even a problem. Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.
- chubot 7y ago
  FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this". The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly. In other words, it was the typical "static analysis has annoying false positives" issue. Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that. So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).
  - jonahx 7y ago
    > The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly. Once it becomes habit the cost is zero, though. And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too." I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people. Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.
    - acqq 7y ago
      Exactly. If one "expert" is developing only for himself he can allow himself not to bother writing a script "clearly" (i.e. failing the programs that perform source checks). But as soon as the script is something that should be potentially sooner or later maintained by anybody else than the original author (and in any successful project that's practically sure) the only right policy is to produce the source code that's obvious and passes the source checks. An "expert" would have no problem to in no time adjust his three-line code to fulfill all the expectations of ShellCheck. If it takes more than that, that only proves that the checks are actually beneficial and even more important than the "expert" tells to himself: it proves that writing that specific code "cleanly" is actually hard for everybody.
rapfaria 7y ago
Using shellshock.net can actually teach you a bunch about scripting as well
sirtoffski 7y ago
Couldn’t agree more! I also really enjoy shfmt, integrating both ShellCheck and shfmt had really helped me to save time and write safer, readable scripts. https://github.com/mvdan/sh https://github.com/mvdan/sh
urda 7y ago
Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.
- filesystemdude 7y ago
  > For many professionals your employer will often block / avoid GPL code, tools, and libraries. Indeed, which is why Linux is never found in a professional setting. /s
  - urda 7y ago
    1) The sarcasm isn't appreciated here on HN. 2) Not everything in the FOSS world is GPL'd. 3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling. 4) That's just how the code is licensed and is related to the topic at hand.
- ahnick 7y ago
  Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
  - bradknowles 7y ago
    Apple is strongly anti-GPL. And they’re pretty open about that. Most companies that are anti-GPL aren’t so open about it.
    - ahnick 7y ago
      I get that Apple doesn't want GPL code included in their software, but their engineers can still use GPL tools without fear of opening up their proprietary code. I know of plenty of companies where you can not INCLUDE GPL code in the software the company makes, but I don't know of any company that says you can not USE any GPL software to do your job. Those are two different things.

aidenn0 7y ago

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/ https://www.oilshell.org/

butterthebuddha 7y ago
I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.
- chubot 7y ago
  (Oil author here) Yes I think that's a great idea, and I hope to release a "liboil" so that people can do that. Let a thousand flowers bloom, etc. The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like: https://www.oilshell.org/blog/2020/01/making-plans.html https://www.oilshell.org/blog/2020/01/making-plans.html So I encourage anyone who's interested in that to get involved now as the interactive shell is also a years-long effort :) There are many links on the home page about how to get involved, and I wrote many blog posts so people can understand how it works. ---- Here are a couple posts on why Oil is a good foundation for an interactive shell: https://www.oilshell.org/blog/2020/01/history-and-completion.html https://www.oilshell.org/blog/2020/01/history-and-completion... https://www.oilshell.org/blog/2020/01/alias-and-prompt.html https://www.oilshell.org/blog/2020/01/alias-and-prompt.html
  - silly_giraffe 7y ago
    I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.
    - chubot 7y ago
      Thank you! Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program. I think Linux, git, and GNU are the prototypical examples of the latter. Those projects are too big for one person, so the code is set up in a way to allow hacking locally. They're also arguably a mess. Anyone who's ever written against the container APIs in Linux knows what I mean (namespaces, cgroups, etc.) The git UI is another famous example, i.e. the plethora of commands and flags that make little sense. Empirically it seems that you can move faster if you disregard consistency and coherence. The shell language of course grew like that over 50 years. Bourne shell is pretty consistent, but one thing I learned is that ksh added a lot of bash misfeatures (they're ksh-isms not bash-isms), and it's pretty bad. But shell is actually a significantly smaller problem than git (after all bash is maintained more or less by one person). So it should be possible to fix it and redesign it. Unfortunately I'm not sure if the lessons generalize to bigger projects, but I would like it if they do. ----- On a more practical note, feel free to send more people this way if you want to see the project succeed :) The design is there, and it will hold up, but it needs a bit more manpower, especially for things like the interactive shell.
      - kortilla 7y ago
        The trick is separating out external facing (api/ux) consistency from code style/data-structure/etc consistency. It doesn’t really matter if you don’t have coherency across the whole project for the latter because that can easily be swapped out later if it’s even a problem. Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.
chubot 7y ago
FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this". The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly. In other words, it was the typical "static analysis has annoying false positives" issue. Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that. So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).
- jonahx 7y ago
  > The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly. Once it becomes habit the cost is zero, though. And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too." I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people. Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.
  - acqq 7y ago
    Exactly. If one "expert" is developing only for himself he can allow himself not to bother writing a script "clearly" (i.e. failing the programs that perform source checks). But as soon as the script is something that should be potentially sooner or later maintained by anybody else than the original author (and in any successful project that's practically sure) the only right policy is to produce the source code that's obvious and passes the source checks. An "expert" would have no problem to in no time adjust his three-line code to fulfill all the expectations of ShellCheck. If it takes more than that, that only proves that the checks are actually beneficial and even more important than the "expert" tells to himself: it proves that writing that specific code "cleanly" is actually hard for everybody.

rapfaria 7y ago

Using shellshock.net can actually teach you a bunch about scripting as well

sirtoffski 7y ago

Couldn’t agree more! I also really enjoy shfmt, integrating both ShellCheck and shfmt had really helped me to save time and write safer, readable scripts. https://github.com/mvdan/sh https://github.com/mvdan/sh

urda 7y ago

Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.

filesystemdude 7y ago
> For many professionals your employer will often block / avoid GPL code, tools, and libraries. Indeed, which is why Linux is never found in a professional setting. /s
- urda 7y ago
  1) The sarcasm isn't appreciated here on HN. 2) Not everything in the FOSS world is GPL'd. 3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling. 4) That's just how the code is licensed and is related to the topic at hand.
ahnick 7y ago
Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
- bradknowles 7y ago
  Apple is strongly anti-GPL. And they’re pretty open about that. Most companies that are anti-GPL aren’t so open about it.
  - ahnick 7y ago
    I get that Apple doesn't want GPL code included in their software, but their engineers can still use GPL tools without fear of opening up their proprietary code. I know of plenty of companies where you can not INCLUDE GPL code in the software the company makes, but I don't know of any company that says you can not USE any GPL software to do your job. Those are two different things.