5 ms·

> An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unf

> An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. > big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. > It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment. I would say the ease depends on the relative difficulty and functionality of the languages. Casting bash as the insane bad language doesn't win you any points: we all know bash is a hot mess of a garbage fire, now let's use it to do useful good work. I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process.

vog 9y ago
> This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. I don't think so. It is a very common mistake to miss the point where you should switch from a shell script to Python. (... or Ruby, Haskell, OCaml, or any other language with good libraries, sane error handling and sane data structures. And yes, I'm saying this being aware of, and always using, "set -eu -o pipefail".) In other words: A complex collection of shell code is a very good warning indicator that somebody missed that point.
- yorwba 9y ago
  Using a thin layer of bash to feed the output of a command into a short Python script is an excellent use of shell in my opinion.
  - Karunamon 9y ago
    But why would you, when you can just do subprocess.popen on the command and then slice and dice the result in a language that can do regexes, splitting, format strings, and so forth? I mean, if you're going to be using Python anyways, you might as well just use the standard library and skip the extra bash step entirely. The code might be a bit longer, but infinitely safer and more maintainable.
    - yorwba 9y ago
      If all you're doing is rewriting the pipelines you'd have written in Bash using subprocess in Python, you're not making your code safer or more maintainable. Unless I'm missing something, subprocess.Popen requires you to explicitly wire up the inputs and outputs, whereas Bash has clear and readable syntax for the most common idioms. For a program that mostly just calls other programs with the right parameters, Python only obfuscates the important bits with details that a shell abstracts away. For a program that has to do some non-trivial processing, pure Bash+coreutils definitely loses to a general-purpose programming language with good standard library. But that doesn't mean that you have to completely switch over. You can leave the parts in Bash that are just pipelines of other programs (again, Python can't improve anything there) and do your further processing in Python. Whether that's implemented as a Bash script calling Python in a pipeline or Python delegating part of it's processing to a Bash script just depends on the most natural entry point. It might feel "impure" to mix two different languages in a program, but I think using each language in the domain where it's strengths shine is a much better idea than attempting to emulate the core features of one language in the other.
      - mbrock 9y ago
        Often it's a really nice and interesting exercise to approach a problem by thinking along the lines of: "This shell script is a bit complex. Instead of rewriting the shell script in another language, is there some simple and standalone way I could massage the data, in order to make the shell script simple again?" It's quite a lot like thinking like this: "This Python script is getting a bit slow. Instead of rewriting it in a faster language, is there some little part of it that I could isolate and only write that in a faster language?"
- falsedan 9y ago
  > I don't think so. You think it's necessary to let sysadmins know their place as "not real developers"? That's elitist.
  - vog 9y ago
    > That's elitist Wait, what?! I do think that every "sysadmin" who is able to write super-complex shell scripts with large sed/awk scripts is also able to learn Python or Ruby. And I'm convinced that this knowledge will enable them to produce rock-solid tools (which they need!) more easily than with Shell scripts. Splitting the world into admins and programmers - that is elitist. You won't find this distinction in small companies, and you find that in large companies only because that's how they happen to be organized - and are heavily criticized for that, mostly by Agile and DevOps movements. Moreover, removing that distiction is one of the core values of the Free Software movement. The early essays of RMS show this in a very clear way. Many people are doing "real programming" without noticing, and this is why they are using improper tools for that, and should start to learn at least one general-purpose language to make things easier for themselves. It is all about using the right tool for the job, and for complex jobs, a generic programming language with proper libraries is the right tool - far better suited than a shell script gluing together highly specialized tools, which run in separate processes and communicate via ad-hoc text formats rather than proper data structures. For every complex system (OS, games, 3D modelling tools), the emerging power users are also beginning programmers, and the more complex things you are able to build within one DSL or (embeeded) scripting language, the easier it is to jump to a general-purpose programming language. Some systems are so nice that they use a general-purpose programming language for scripting in the first place - such as Lua for many game engines, Python for Blender and Gimp, Scheme (Guile) for Guix package building, and so on.
    - qznc 9y ago
      My sysadmin at the university has explicit orders "not to program". Ironically, he is allowed to write configuration, e.g. Ansible playbooks. So I guess, the trick is to use a turing-complete configuration language which is "totally not programming".
    - 9y ago
jacobparker 9y ago
> This belittles their and anyone else's bash scripts for git as 'toys'. They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Robustness may not matter (e.g. I have tonnes of scripts I use for doing work) but if you're generating HTML for a website and are worried about escaping and security then I think there are more suitable tools. > Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. It's also possible to do that in programming languages (maybe not as succiently.) But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick. I've replaced reasonable (i.e. weren't badly mishandling the data) shell scripts that procesed and formatted the output of git utilities (on large repos) and in one case shrunk a background job from 5 minutes to a few seconds. This is common. > Casting bash as the insane bad language I never said that. I said it was great for prototyping. It is bad for the task FTA, though. > I would struggle to justify the work I'd need to do to write the equivalent in C I didn't recommend C (I did link to some Go stuff, though.) Unless your company is all bash scripts (heaven forbid) you're probably already using some language. C# at my work. PHP, Ruby, JavaScript, whatever.
- falsedan 9y ago
  > They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Programming is hard, regardless. You're being very dismissive towards shell scripters, and it comes off as elitist. > But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick. What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent. > xargs Did you have a bad experience with xargs? > extra processes Did you have a bad experience with… running out of… PIDs? > > Casting bash as the insane bad language > I never said that You implied it by calling a non-bash language 'sane'; thus, bash must be insane. > I didn't recommend C Your blanket recommendation of 'anything but bash' did. I feel that sometimes you need to have more context of a situation before dictating a course of action that must be followed.
kbenson 9y ago
> Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. Yeah, there was an article a few years back about that, Command-line tools can be faster than your Hadoop cluster [1], where the punchline is that because of the inherent stream processing because of pipelining, the shell pipeline was 235 times faster than Hadoop. 1: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462
chubot 9y ago
(author here) Thanks for the comment, yes a lot of people seem to be missing the point of the article, and taking extreme positions on one side or the other. Bash is indeed a hot mess, but it gets work done! This is a trivial part of the release process and shell lets me cut right through it. It's also pretty much correct and I analyze the limitations. And of course I think we can do better than bash while still retaining the shell paradigm -- hence the Oil shell.

vog 9y ago

> This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. I don't think so. It is a very common mistake to miss the point where you should switch from a shell script to Python. (... or Ruby, Haskell, OCaml, or any other language with good libraries, sane error handling and sane data structures. And yes, I'm saying this being aware of, and always using, "set -eu -o pipefail".) In other words: A complex collection of shell code is a very good warning indicator that somebody missed that point.

yorwba 9y ago
Using a thin layer of bash to feed the output of a command into a short Python script is an excellent use of shell in my opinion.
- Karunamon 9y ago
  But why would you, when you can just do subprocess.popen on the command and then slice and dice the result in a language that can do regexes, splitting, format strings, and so forth? I mean, if you're going to be using Python anyways, you might as well just use the standard library and skip the extra bash step entirely. The code might be a bit longer, but infinitely safer and more maintainable.
  - yorwba 9y ago
    If all you're doing is rewriting the pipelines you'd have written in Bash using subprocess in Python, you're not making your code safer or more maintainable. Unless I'm missing something, subprocess.Popen requires you to explicitly wire up the inputs and outputs, whereas Bash has clear and readable syntax for the most common idioms. For a program that mostly just calls other programs with the right parameters, Python only obfuscates the important bits with details that a shell abstracts away. For a program that has to do some non-trivial processing, pure Bash+coreutils definitely loses to a general-purpose programming language with good standard library. But that doesn't mean that you have to completely switch over. You can leave the parts in Bash that are just pipelines of other programs (again, Python can't improve anything there) and do your further processing in Python. Whether that's implemented as a Bash script calling Python in a pipeline or Python delegating part of it's processing to a Bash script just depends on the most natural entry point. It might feel "impure" to mix two different languages in a program, but I think using each language in the domain where it's strengths shine is a much better idea than attempting to emulate the core features of one language in the other.
    - mbrock 9y ago
      Often it's a really nice and interesting exercise to approach a problem by thinking along the lines of: "This shell script is a bit complex. Instead of rewriting the shell script in another language, is there some simple and standalone way I could massage the data, in order to make the shell script simple again?" It's quite a lot like thinking like this: "This Python script is getting a bit slow. Instead of rewriting it in a faster language, is there some little part of it that I could isolate and only write that in a faster language?"
falsedan 9y ago
> I don't think so. You think it's necessary to let sysadmins know their place as "not real developers"? That's elitist.
- vog 9y ago
  > That's elitist Wait, what?! I do think that every "sysadmin" who is able to write super-complex shell scripts with large sed/awk scripts is also able to learn Python or Ruby. And I'm convinced that this knowledge will enable them to produce rock-solid tools (which they need!) more easily than with Shell scripts. Splitting the world into admins and programmers - that is elitist. You won't find this distinction in small companies, and you find that in large companies only because that's how they happen to be organized - and are heavily criticized for that, mostly by Agile and DevOps movements. Moreover, removing that distiction is one of the core values of the Free Software movement. The early essays of RMS show this in a very clear way. Many people are doing "real programming" without noticing, and this is why they are using improper tools for that, and should start to learn at least one general-purpose language to make things easier for themselves. It is all about using the right tool for the job, and for complex jobs, a generic programming language with proper libraries is the right tool - far better suited than a shell script gluing together highly specialized tools, which run in separate processes and communicate via ad-hoc text formats rather than proper data structures. For every complex system (OS, games, 3D modelling tools), the emerging power users are also beginning programmers, and the more complex things you are able to build within one DSL or (embeeded) scripting language, the easier it is to jump to a general-purpose programming language. Some systems are so nice that they use a general-purpose programming language for scripting in the first place - such as Lua for many game engines, Python for Blender and Gimp, Scheme (Guile) for Guix package building, and so on.
  - qznc 9y ago
    My sysadmin at the university has explicit orders "not to program". Ironically, he is allowed to write configuration, e.g. Ansible playbooks. So I guess, the trick is to use a turing-complete configuration language which is "totally not programming".
  - 9y ago

jacobparker 9y ago

> This belittles their and anyone else's bash scripts for git as 'toys'. They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Robustness may not matter (e.g. I have tonnes of scripts I use for doing work) but if you're generating HTML for a website and are worried about escaping and security then I think there are more suitable tools. > Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. It's also possible to do that in programming languages (maybe not as succiently.) But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick. I've replaced reasonable (i.e. weren't badly mishandling the data) shell scripts that procesed and formatted the output of git utilities (on large repos) and in one case shrunk a background job from 5 minutes to a few seconds. This is common. > Casting bash as the insane bad language I never said that. I said it was great for prototyping. It is bad for the task FTA, though. > I would struggle to justify the work I'd need to do to write the equivalent in C I didn't recommend C (I did link to some Go stuff, though.) Unless your company is all bash scripts (heaven forbid) you're probably already using some language. C# at my work. PHP, Ruby, JavaScript, whatever.

falsedan 9y ago
> They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts. Programming is hard, regardless. You're being very dismissive towards shell scripters, and it comes off as elitist. > But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick. What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent. > xargs Did you have a bad experience with xargs? > extra processes Did you have a bad experience with… running out of… PIDs? > > Casting bash as the insane bad language > I never said that You implied it by calling a non-bash language 'sane'; thus, bash must be insane. > I didn't recommend C Your blanket recommendation of 'anything but bash' did. I feel that sometimes you need to have more context of a situation before dictating a course of action that must be followed.

kbenson 9y ago

> Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. Yeah, there was an article a few years back about that, Command-line tools can be faster than your Hadoop cluster [1], where the punchline is that because of the inherent stream processing because of pipelining, the shell pipeline was 235 times faster than Hadoop. 1: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462

chubot 9y ago

(author here) Thanks for the comment, yes a lot of people seem to be missing the point of the article, and taking extreme positions on one side or the other. Bash is indeed a hot mess, but it gets work done! This is a trivial part of the release process and shell lets me cut right through it. It's also pretty much correct and I analyze the limitations. And of course I think we can do better than bash while still retaining the shell paradigm -- hence the Oil shell.