3 ms·

How do you deal with it, when you are writing bash scripts?

by tutfbhuf 5y ago

DonHopkins 5y ago
By immediately invoking Python and getting the hell out of bash.
- FranchuFranchu 5y ago
  Or xonsh
- Spivak 5y ago
  Python is my absolute favorite language but it's not suitable for the kinds of things you would use bash for. This is the real code that a Ansible uses to run a shell command correctly and is 350 lines and is still a small subset of the features of a single line of bash. https://github.com/ansible/ansible/blob/a2776443017718f6bbd88bd4ca044e3a71c6a8bf/lib/ansible/module_utils/basic.py#L1833 https://github.com/ansible/ansible/blob/a2776443017718f6bbd8... The Python code to do what a single mv invocation does is 120 lines https://github.com/ansible/ansible/blob/a2776443017718f6bbd88bd4ca044e3a71c6a8bf/lib/ansible/module_utils/basic.py#L1645 https://github.com/ansible/ansible/blob/a2776443017718f6bbd8... People always focus on the footguns that exist in Bash the language but ignore how much systems programming is abstracted away from you in the shell environment. In Bash you can enter a Linux namespace with a single nsenter invocation. If you want to do the same in Python you have use ctypes and call libc.clone manually.
  - overtomanu 5y ago
    i think its not fair comparison. mv command implementation in 'C' might have more lines of code. Maybe we should complain that there are no OOTB library functions in python to move the file.
    - Spivak 5y ago
      I don't really care how many lines of code there are in an implementation. I care how many lines of code I actually have to write. Python has shutil.move and os.rename but the Ansible example is to illustrate that there's a lot of code that needs to surround those calls to make them useful and they're not 1-1.
  - BiteCode_dev 5y ago
    The code you are showing does a lot more than MV: - it's portable accross OSes (and keep flags if the OS supports it, deal with encoding, etc) - it ensures selinux context is saved if there is such a thing - it has proper and rich error communication with the calling code - it's includes documentation and comments - it outputs json par parsing and storage No to say "mv" is not awesome, because it is. There is much more boiler plate in python, and is why I'll often do subprocess.check_call(['mv', 'src', 'dst']) if my script is linux only. But you are pushing it
  - nonameiguess 5y ago
    How is that a remotely real comparison? The ansible mv function deals with preserving SELinux context, which mv doesn't do, and it automatically deals with a whole lot of common error conditions, whereas mv just fails. If you just want to replicate mv, Python has shutil.move, one line of code. Ansible is trying to do a lot more. By the way, I don't know if this is the canonical implementation, but FreeBSD mv is 481 lines of C: https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv.c https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv...
pwdisswordfish8 5y ago
By not using eval (and sh -c, and everything equivalent) unless it’s absolutely unavoidable and always quoting variables. The $ construct acts basically like (the shell’s) eval. Proper escaping is an absolute must here.
- 40four 5y ago
  Indeed, I’m not a bash expert, but I’ve heard multiple times that using eval is a bad idea. Your point is a good reminder of why.
  - TechBro8615 5y ago
    I'm pretty sure that "eval is a bad idea" was mentioned in the "thought terminating cliches" topic on Ask HN a few days ago :) In bash, `eval` is a footgun like any other. You can use it, but you just need to be aware of where your toes are when it shoots. It's usually a "bad idea" in the sense that if you think you need to use it, you probably don't, and 90% of the time, there is an easier way to accomplish what you want to do. The next 5% of the time, using `eval` might be easier but will also create maintenance debt with its overgeneralization. And the final 5% of the time might be actual legit use cases for `eval`. I just grepped my codebase for `eval` and I almost never use it. One example of the "overgeneralized" 5% might be when I realized I could use `eval` to set "variable variables" (i.e. the name of the variable is itself a variable, taken from a function argument). It was cool, but I ended up deleting it in favor of a more concrete solution. Personally, if I'm hesitating to use `eval`, it's usually not for any security reasons. In general, my bash scripts only exist in dev machines and CI runners, and I don't copy them into the application containers that are exposed to a live runtime environment with untrusted users. So for CI/dev scripts, I can safely assume the code will only run in CI/dev, and therefore I can trust arbitrary user input (which I can of course still validate).
    - 40four 5y ago
      Thanks for the detailed response! I’ve been trying to level up my bash lately, and I definitely have seen a lot of ‘poor’ examples where they bailed out & use eval plus something else, for something native bash can in fact handle just fine, if you dig deep enough!
selfhoster11 5y ago
This is usually answered with "poorly", or "with great difficulty".

How do you deal with it, when you are writing bash scripts?

DonHopkins 5y ago

By immediately invoking Python and getting the hell out of bash.

FranchuFranchu 5y ago
Or xonsh
Spivak 5y ago
Python is my absolute favorite language but it's not suitable for the kinds of things you would use bash for. This is the real code that a Ansible uses to run a shell command correctly and is 350 lines and is still a small subset of the features of a single line of bash. https://github.com/ansible/ansible/blob/a2776443017718f6bbd88bd4ca044e3a71c6a8bf/lib/ansible/module_utils/basic.py#L1833 https://github.com/ansible/ansible/blob/a2776443017718f6bbd8... The Python code to do what a single mv invocation does is 120 lines https://github.com/ansible/ansible/blob/a2776443017718f6bbd88bd4ca044e3a71c6a8bf/lib/ansible/module_utils/basic.py#L1645 https://github.com/ansible/ansible/blob/a2776443017718f6bbd8... People always focus on the footguns that exist in Bash the language but ignore how much systems programming is abstracted away from you in the shell environment. In Bash you can enter a Linux namespace with a single nsenter invocation. If you want to do the same in Python you have use ctypes and call libc.clone manually.
- overtomanu 5y ago
  i think its not fair comparison. mv command implementation in 'C' might have more lines of code. Maybe we should complain that there are no OOTB library functions in python to move the file.
  - Spivak 5y ago
    I don't really care how many lines of code there are in an implementation. I care how many lines of code I actually have to write. Python has shutil.move and os.rename but the Ansible example is to illustrate that there's a lot of code that needs to surround those calls to make them useful and they're not 1-1.
- BiteCode_dev 5y ago
  The code you are showing does a lot more than MV: - it's portable accross OSes (and keep flags if the OS supports it, deal with encoding, etc) - it ensures selinux context is saved if there is such a thing - it has proper and rich error communication with the calling code - it's includes documentation and comments - it outputs json par parsing and storage No to say "mv" is not awesome, because it is. There is much more boiler plate in python, and is why I'll often do subprocess.check_call(['mv', 'src', 'dst']) if my script is linux only. But you are pushing it
- nonameiguess 5y ago
  How is that a remotely real comparison? The ansible mv function deals with preserving SELinux context, which mv doesn't do, and it automatically deals with a whole lot of common error conditions, whereas mv just fails. If you just want to replicate mv, Python has shutil.move, one line of code. Ansible is trying to do a lot more. By the way, I don't know if this is the canonical implementation, but FreeBSD mv is 481 lines of C: https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv.c https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv...

pwdisswordfish8 5y ago

By not using eval (and sh -c, and everything equivalent) unless it’s absolutely unavoidable and always quoting variables. The $ construct acts basically like (the shell’s) eval. Proper escaping is an absolute must here.

40four 5y ago
Indeed, I’m not a bash expert, but I’ve heard multiple times that using eval is a bad idea. Your point is a good reminder of why.
- TechBro8615 5y ago
  I'm pretty sure that "eval is a bad idea" was mentioned in the "thought terminating cliches" topic on Ask HN a few days ago :) In bash, `eval` is a footgun like any other. You can use it, but you just need to be aware of where your toes are when it shoots. It's usually a "bad idea" in the sense that if you think you need to use it, you probably don't, and 90% of the time, there is an easier way to accomplish what you want to do. The next 5% of the time, using `eval` might be easier but will also create maintenance debt with its overgeneralization. And the final 5% of the time might be actual legit use cases for `eval`. I just grepped my codebase for `eval` and I almost never use it. One example of the "overgeneralized" 5% might be when I realized I could use `eval` to set "variable variables" (i.e. the name of the variable is itself a variable, taken from a function argument). It was cool, but I ended up deleting it in favor of a more concrete solution. Personally, if I'm hesitating to use `eval`, it's usually not for any security reasons. In general, my bash scripts only exist in dev machines and CI runners, and I don't copy them into the application containers that are exposed to a live runtime environment with untrusted users. So for CI/dev scripts, I can safely assume the code will only run in CI/dev, and therefore I can trust arbitrary user input (which I can of course still validate).
  - 40four 5y ago
    Thanks for the detailed response! I’ve been trying to level up my bash lately, and I definitely have seen a lot of ‘poor’ examples where they bailed out & use eval plus something else, for something native bash can in fact handle just fine, if you dig deep enough!

selfhoster11 5y ago

This is usually answered with "poorly", or "with great difficulty".