6 ms·

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines lo

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines long, you should have written it in Python." I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or Javascript, it would have been far more obvious what the resulting output would have been, considering that those languages already use {} for objects and [] for lists. For example, take this example: jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false In Python you would write it like this: json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}) Only a bit more code, but at least it won't suffer from the Norway problem. Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. There is no need to construct a fork bomb just to generate some JSON.

naniwaduni 5y ago
For programs this trivial, startup time so dominates runtime, and Python's startup time is so incredibly awful, that you can often fork 10-20 low-overhead processes before Python even starts executing user code.
deleted 5y ago
[deleted]
zimpenfish 5y ago
> Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. Taking these two command lines: jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null For jo (x86_64, Rosetta2), python2 (x86_64, Rosetta2), jo (arm64), and python3 (arm64), running 1000 iterations, with `tai64n` doing the timing. 2022-02-05 21:25:38.357228500 start-jo-x86 2022-02-05 21:25:45.319337500 stop-jo 2022-02-05 21:25:45.319338500 start-python2-x86 2022-02-05 21:26:18.876235500 stop-python2-x86 2022-02-05 21:26:18.876235500 start-jo-arm 2022-02-05 21:26:22.316063500 stop-jo-arm 2022-02-05 21:26:22.316064500 start-python3-arm 2022-02-05 21:26:40.379063500 stop-python3-arm I make it: 7s for jo-x86, 33.5s for python2-x86, 3.5s for jo-arm, 18s for python3-arm. Test script is at https://pastebin.com/4tTVrDia https://pastebin.com/4tTVrDia
- EdSchouten 5y ago
  Now put a thousand of those JSON objects in a list, invoking jo for every element.
  - zimpenfish 5y ago
    That wasn't the claim made in the original post though, was it? The claim was that the Python snippet would be quicker than the jo snippet. "Even though Python isn't the fastest language out there, it's likely still faster than the shell command above." Which is most definitely is not - it's 5x slower. (Probably not a huge issue in the real world if you're writing a shell script, mind, given that bash itself isn't a performance demon. But claims have to be tested.)
    - EdSchouten 5y ago
      That’s because you’re making a false assumption about the environment prior to executing the statement. If you are in a shell session and have to choose between executing python -c or calling jo, the latter is faster as you’ve demonstrated. But that’s not a realistic assumption. Statements like these are almost certainly part of some combined work. The data you’re feeding to jo comes from somewhere. Its output is written somewhere. You can’t convince me that if you’re already inside some Python script, that invoking json.dumps() is slower than calling jo from within a shell script. At no point did I claim that launching Python AND running that json.dumps() is faster than running that shell command. I only stated that the json.dumps() is.
      - zimpenfish 5y ago
        > if you’re already inside some Python script [...] You're not going to shell out to `jo` and that's fine - it's not what `jo` was created for; it's explicitly a shell command to help you work around the annoyance of getting quoting right when constructing JSON from the command line (which I've had to do a lot and I'm pretty sure many people have to.) > If you are in a shell session [and want to create JSON] ... that’s not a realistic assumption. Of course it is. People create JSON in shell scripts all the time! That's why things like `jq` exist - because this is what people do!
motoboi 5y ago
Hard to write oneliners in python, though. And this lets you write json without (or with less) braces, commas and quotes. That alone is already a big win.
mmgutz 5y ago
If you're going to use string literal, then echo '{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": false}' is fine as a script
- anamexis 5y ago
  All fun and games until you need quoting. Quoting in shell scripts is already hellish enough, but layering JSON quoting on top of that is a road to madness.
  - mkdirp 5y ago
    Combining here-string with `jq` (if you need variables), is plenty good enough. You'd need a single jq invocation.
    - anamexis 5y ago
      Yes, `jq` would work fine, but so would `jo`. The point is, if there is anything more than the simplest dynamic values, constructing valid JSON just with POSIX shell is a huge pain.
  - deleted 5y ago
    [deleted]
pkulak 5y ago
Sure, but using Python gives you 100 more problems. Would you like to set pip and your package manager on a fight to the death? Or is today the day you learn all about venv? Might as well use Docker. Oh nice, my 600MB script is ready!
- detaro 5y ago
  That's not really an issue for small scripts, no.
  - deathanatos 5y ago
    Since I have no idea why this is downvoted: it really isn't for small scripts: if you just ignore pip/packages, Python is going to give you way more functionality out of the box than shell would, with a lot fewer sharp corners that will translate into a less buggy/more correct script at the end of day. If you do take the time to deal with pip (which, yes, is a problem) you get access to even more batteries that would have been a pain or just flat out impossible with shell. (& on many distros, you can use your system package manager. I'm not seeing a material difference between a shell script that requires "apt-get install jo" and a Python script that requires "apt-get install python3-requests" or something.) But either way, for circumstances where shell is the wrong tool for the job, "invoke python in the middle of this shell script because it's a better tool for this particular part of the job" is a strategy I've used before & will keep using, b/c it produces code that isn't riddled with bugs.
    - bb88 5y ago
      It's behavior is really well defined and will stop on a syntax error, type error, or some other error that wasn't handled. Bash on error will just move on to the next line as if nothing ever happened unless you: set -eu set -o pipefail
nfw2 5y ago
I feel this whenever anyone advocates using jq. It boggles my mind that anyone would want to learn a whole new DSL for something that js makes trivial, especially considering js a much more expressive scripting language anyway
- viraptor 5y ago
  JS is not that easy to embed in scripting though for trivial situations. After seeing a couple of jq examples I can run `jq '.foo[] | {bar, baz}' without really "learning" its DSL. But doing the same with node? That would be much larger.
  - nfw2 5y ago
    And now everyone who needs to read your script needs to also find those examples to figure out what your code does. Even your not-real-world example isn't obvious to anyone unfamiliar with jq what the intent is
    - viraptor 5y ago
      That's about the only example you need explained to understand ~99% of real world jq usage. $dayjob has quite a bit of it around in various repos, and this actually is as real-world as it gets in my experience. Comparing that to having to learn enough JS to do the same thing in a verbose way, I'm still on the side of jq having an advantage in that case.
deleted 5y ago
[deleted]
synergy20 5y ago
that's probably you're at google, however there are more embedded devices on earth than whatever google has times 1+ billion. in those devices python is too heavy, and a posix shell along with jo fits perfectly.
skybrian 5y ago
The longest shell script I ever wrote was at Google, because at the time, my co-worker and I didn't know anyone with Python readability.
Galanwe 5y ago
> there was this common saying: > "If the first version of your shell script is more than five lines long, you should have written it in Python." Seriously, these kind of "common knowledge", "universal truth in a sentence" sayings are often the mark of wannabe guru mid career engineers that have no clue what they are talking about.
- jsnelgro 5y ago
  This is a toxic comment that adds nothing to the conversation.
  - goldenkey 5y ago
    I don't know, the jo command seems a lot more readable to me than compressing json to a one line string. How is it toxic to call out a silly truthism?
    - MathMonkeyMan 5y ago
      I agree with Galanwe, but > [...] are often the mark of wannabe guru mid career engineers that have no clue what they are talking about. is the "toxic" part. At least he says "are often," as not to accuse the anonymous mid career Google engineer.
jsnelgro 5y ago
I just assume at this point that a new python script from a coworker won’t run without an hour of tinkering and yelling obscenities at my screen. Or resorting to running in docker, which seems asinine. Python’s everywhere and does everything though, so I don’t have a good alternative. Shell scripts definitely aren’t it, but they generally hold up better when sharing in my experience.
- Too 5y ago
  Shell scripts can't declare dependencies though, in the same way a pip package can. A shell script using this tool requires one to manually apt install it first, or run in a common docker image - asinine. If you don't, your script will fail halfway through during runtime (actually, likely it will not fail, just produce corrupted output, since shell by default ignores errors), a python IDE or mypy will tell you about missing packages during analysis before you try to build and run it. Besides that, looking at json only, it's part of the standard library so is more likely to already exist on any given machine rather than this.
- c0npr 5y ago
  That depends on how well your bash script is constructed. If you carefully handle the falling case such as missing commands, non-root permissions, etc. It can be easy to use and kind of portable. Of course python scripts have better error trace so if the script doesnt work others can debug with relatively easily.
michaelcampbell 5y ago
> it's likely still faster than the shell command above. That's not a shell command any more than running "python" is. `jo` is its own executable. And doing the sub-commands are not necessary; `jo` supports nested data natively.
1vuio0pswjnm7 5y ago
"Even though Python isn't the fastest language out there, it's likely still faster than the shell command above." That is going a bit far. By all means use Python. Go ahead and attack people who use the shell. But let's be honest. The shell is faster, assuming one knows how to use it. A similar claim is often made by Python advocates, something along the lines of Python is not slow if one knows how to use it. The startup time of a Python interpreter is enormous for someone who is used to a Bourne shell. This is what always stops me from using Python as a shell replacement for relatively simple jobs; I have never written a large shell script and doubt I ever will. I write small scripts. If anyone knows how to mitigate the Python startup delay, feel free to share. I might become more interested in Python. Anyway, this "jo" thing seems a bit silly. Someone at Google spent their 20% time writing a language called jsonnet to emit JSON. It has been discussed on HN before. People have suggested dhall is a perhaps better alternative. https://jsonnet.org https://jsonnet.org https://dhall-lang.org https://dhall-lang.org
jasfi 5y ago
Although good advice, this is also an area where Nim could shine.
ZeroGravitas 5y ago
Often when writing scripts, I'm chaining tools together, e.g. using git to find a thing in a specific commit, using curl to grab something from the web, decoding some json, maybe unzipping a file. I've never really found any language that feels good for that kind of thing, there's definately a middle ground where it's getting too much for bash, but jumping to a language loses too much at the initial conversion to make it feel worth it until you are well past the point where your future self will think you should have made the switch. Some languages have things like backticks in php to inter-operate but it's still not great experience to mix between them. For my own little things I'm currently looking at fish, but bash is omnipresent. This tool seems to delay that point even further, as currently dealing with generating json is definately a pain point (whereas manipulating it in jq is often really good). But if anyone can point to good examples of this transition in python then I'd be very interested. edit: jq is more powerful than I thought for creating json, see https://spin.atomicobject.com/2021/06/08/jq-creating-updating-json/ https://spin.atomicobject.com/2021/06/08/jq-creating-updatin...
- bb88 5y ago
  Typically if you have bash you have a bunch of other utilities installed too. The problem with python here is that while python-sh might be nice, you have to install any extra libraries you need with it, and that's not a trivial problem for installing scripts into prod. Xonsh is better since you kind of get both the benefits of python and a shell like language, but frankly it's broken in a number of ways still. I use it daily, but hopefully you don't need to ctrl-c out of something since signal handling is iffy at best. It is kind of nice to be able to import python directly on the command line and use it as python though...
- Corrado 5y ago
  Thank you for pointing out that jq can create JSON! I use jq all the time for working with the AWS CLI and a big pain point has always been sending JSON. If you don't know, the AWS CLI depends on JSON arguments for quite a few common tasks, and the JSON needed can be quite lengthy. Up until now I've been creating temp JSON files to feed the commands and I thought jo would be a great tool to make this easier. Now that I know jq can also create JSON, I'll just use that instead.

naniwaduni 5y ago

For programs this trivial, startup time so dominates runtime, and Python's startup time is so incredibly awful, that you can often fork 10-20 low-overhead processes before Python even starts executing user code.

deleted 5y ago

[deleted]

zimpenfish 5y ago

> Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. Taking these two command lines: jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null For jo (x86_64, Rosetta2), python2 (x86_64, Rosetta2), jo (arm64), and python3 (arm64), running 1000 iterations, with `tai64n` doing the timing. 2022-02-05 21:25:38.357228500 start-jo-x86 2022-02-05 21:25:45.319337500 stop-jo 2022-02-05 21:25:45.319338500 start-python2-x86 2022-02-05 21:26:18.876235500 stop-python2-x86 2022-02-05 21:26:18.876235500 start-jo-arm 2022-02-05 21:26:22.316063500 stop-jo-arm 2022-02-05 21:26:22.316064500 start-python3-arm 2022-02-05 21:26:40.379063500 stop-python3-arm I make it: 7s for jo-x86, 33.5s for python2-x86, 3.5s for jo-arm, 18s for python3-arm. Test script is at https://pastebin.com/4tTVrDia https://pastebin.com/4tTVrDia

EdSchouten 5y ago
Now put a thousand of those JSON objects in a list, invoking jo for every element.
- zimpenfish 5y ago
  That wasn't the claim made in the original post though, was it? The claim was that the Python snippet would be quicker than the jo snippet. "Even though Python isn't the fastest language out there, it's likely still faster than the shell command above." Which is most definitely is not - it's 5x slower. (Probably not a huge issue in the real world if you're writing a shell script, mind, given that bash itself isn't a performance demon. But claims have to be tested.)
  - EdSchouten 5y ago
    That’s because you’re making a false assumption about the environment prior to executing the statement. If you are in a shell session and have to choose between executing python -c or calling jo, the latter is faster as you’ve demonstrated. But that’s not a realistic assumption. Statements like these are almost certainly part of some combined work. The data you’re feeding to jo comes from somewhere. Its output is written somewhere. You can’t convince me that if you’re already inside some Python script, that invoking json.dumps() is slower than calling jo from within a shell script. At no point did I claim that launching Python AND running that json.dumps() is faster than running that shell command. I only stated that the json.dumps() is.
    - zimpenfish 5y ago
      > if you’re already inside some Python script [...] You're not going to shell out to `jo` and that's fine - it's not what `jo` was created for; it's explicitly a shell command to help you work around the annoyance of getting quoting right when constructing JSON from the command line (which I've had to do a lot and I'm pretty sure many people have to.) > If you are in a shell session [and want to create JSON] ... that’s not a realistic assumption. Of course it is. People create JSON in shell scripts all the time! That's why things like `jq` exist - because this is what people do!

motoboi 5y ago

Hard to write oneliners in python, though. And this lets you write json without (or with less) braces, commas and quotes. That alone is already a big win.

mmgutz 5y ago

If you're going to use string literal, then echo '{"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": false}' is fine as a script

anamexis 5y ago
All fun and games until you need quoting. Quoting in shell scripts is already hellish enough, but layering JSON quoting on top of that is a road to madness.
- mkdirp 5y ago
  Combining here-string with `jq` (if you need variables), is plenty good enough. You'd need a single jq invocation.
  - anamexis 5y ago
    Yes, `jq` would work fine, but so would `jo`. The point is, if there is anything more than the simplest dynamic values, constructing valid JSON just with POSIX shell is a huge pain.
- deleted 5y ago
  [deleted]

pkulak 5y ago

Sure, but using Python gives you 100 more problems. Would you like to set pip and your package manager on a fight to the death? Or is today the day you learn all about venv? Might as well use Docker. Oh nice, my 600MB script is ready!

detaro 5y ago
That's not really an issue for small scripts, no.
- deathanatos 5y ago
  Since I have no idea why this is downvoted: it really isn't for small scripts: if you just ignore pip/packages, Python is going to give you way more functionality out of the box than shell would, with a lot fewer sharp corners that will translate into a less buggy/more correct script at the end of day. If you do take the time to deal with pip (which, yes, is a problem) you get access to even more batteries that would have been a pain or just flat out impossible with shell. (& on many distros, you can use your system package manager. I'm not seeing a material difference between a shell script that requires "apt-get install jo" and a Python script that requires "apt-get install python3-requests" or something.) But either way, for circumstances where shell is the wrong tool for the job, "invoke python in the middle of this shell script because it's a better tool for this particular part of the job" is a strategy I've used before & will keep using, b/c it produces code that isn't riddled with bugs.
  - bb88 5y ago
    It's behavior is really well defined and will stop on a syntax error, type error, or some other error that wasn't handled. Bash on error will just move on to the next line as if nothing ever happened unless you: set -eu set -o pipefail

nfw2 5y ago

I feel this whenever anyone advocates using jq. It boggles my mind that anyone would want to learn a whole new DSL for something that js makes trivial, especially considering js a much more expressive scripting language anyway

viraptor 5y ago
JS is not that easy to embed in scripting though for trivial situations. After seeing a couple of jq examples I can run `jq '.foo[] | {bar, baz}' without really "learning" its DSL. But doing the same with node? That would be much larger.
- nfw2 5y ago
  And now everyone who needs to read your script needs to also find those examples to figure out what your code does. Even your not-real-world example isn't obvious to anyone unfamiliar with jq what the intent is
  - viraptor 5y ago
    That's about the only example you need explained to understand ~99% of real world jq usage. $dayjob has quite a bit of it around in various repos, and this actually is as real-world as it gets in my experience. Comparing that to having to learn enough JS to do the same thing in a verbose way, I'm still on the side of jq having an advantage in that case.

deleted 5y ago

[deleted]

synergy20 5y ago

that's probably you're at google, however there are more embedded devices on earth than whatever google has times 1+ billion. in those devices python is too heavy, and a posix shell along with jo fits perfectly.

skybrian 5y ago

The longest shell script I ever wrote was at Google, because at the time, my co-worker and I didn't know anyone with Python readability.

Galanwe 5y ago

> there was this common saying: > "If the first version of your shell script is more than five lines long, you should have written it in Python." Seriously, these kind of "common knowledge", "universal truth in a sentence" sayings are often the mark of wannabe guru mid career engineers that have no clue what they are talking about.

jsnelgro 5y ago
This is a toxic comment that adds nothing to the conversation.
- goldenkey 5y ago
  I don't know, the jo command seems a lot more readable to me than compressing json to a one line string. How is it toxic to call out a silly truthism?
  - MathMonkeyMan 5y ago
    I agree with Galanwe, but > [...] are often the mark of wannabe guru mid career engineers that have no clue what they are talking about. is the "toxic" part. At least he says "are often," as not to accuse the anonymous mid career Google engineer.

jsnelgro 5y ago

I just assume at this point that a new python script from a coworker won’t run without an hour of tinkering and yelling obscenities at my screen. Or resorting to running in docker, which seems asinine. Python’s everywhere and does everything though, so I don’t have a good alternative. Shell scripts definitely aren’t it, but they generally hold up better when sharing in my experience.

Too 5y ago
Shell scripts can't declare dependencies though, in the same way a pip package can. A shell script using this tool requires one to manually apt install it first, or run in a common docker image - asinine. If you don't, your script will fail halfway through during runtime (actually, likely it will not fail, just produce corrupted output, since shell by default ignores errors), a python IDE or mypy will tell you about missing packages during analysis before you try to build and run it. Besides that, looking at json only, it's part of the standard library so is more likely to already exist on any given machine rather than this.
c0npr 5y ago
That depends on how well your bash script is constructed. If you carefully handle the falling case such as missing commands, non-root permissions, etc. It can be easy to use and kind of portable. Of course python scripts have better error trace so if the script doesnt work others can debug with relatively easily.

michaelcampbell 5y ago

> it's likely still faster than the shell command above. That's not a shell command any more than running "python" is. `jo` is its own executable. And doing the sub-commands are not necessary; `jo` supports nested data natively.

1vuio0pswjnm7 5y ago

"Even though Python isn't the fastest language out there, it's likely still faster than the shell command above." That is going a bit far. By all means use Python. Go ahead and attack people who use the shell. But let's be honest. The shell is faster, assuming one knows how to use it. A similar claim is often made by Python advocates, something along the lines of Python is not slow if one knows how to use it. The startup time of a Python interpreter is enormous for someone who is used to a Bourne shell. This is what always stops me from using Python as a shell replacement for relatively simple jobs; I have never written a large shell script and doubt I ever will. I write small scripts. If anyone knows how to mitigate the Python startup delay, feel free to share. I might become more interested in Python. Anyway, this "jo" thing seems a bit silly. Someone at Google spent their 20% time writing a language called jsonnet to emit JSON. It has been discussed on HN before. People have suggested dhall is a perhaps better alternative. https://jsonnet.org https://jsonnet.org https://dhall-lang.org https://dhall-lang.org

jasfi 5y ago

Although good advice, this is also an area where Nim could shine.

ZeroGravitas 5y ago

Often when writing scripts, I'm chaining tools together, e.g. using git to find a thing in a specific commit, using curl to grab something from the web, decoding some json, maybe unzipping a file. I've never really found any language that feels good for that kind of thing, there's definately a middle ground where it's getting too much for bash, but jumping to a language loses too much at the initial conversion to make it feel worth it until you are well past the point where your future self will think you should have made the switch. Some languages have things like backticks in php to inter-operate but it's still not great experience to mix between them. For my own little things I'm currently looking at fish, but bash is omnipresent. This tool seems to delay that point even further, as currently dealing with generating json is definately a pain point (whereas manipulating it in jq is often really good). But if anyone can point to good examples of this transition in python then I'd be very interested. edit: jq is more powerful than I thought for creating json, see https://spin.atomicobject.com/2021/06/08/jq-creating-updating-json/ https://spin.atomicobject.com/2021/06/08/jq-creating-updatin...

bb88 5y ago
Typically if you have bash you have a bunch of other utilities installed too. The problem with python here is that while python-sh might be nice, you have to install any extra libraries you need with it, and that's not a trivial problem for installing scripts into prod. Xonsh is better since you kind of get both the benefits of python and a shell like language, but frankly it's broken in a number of ways still. I use it daily, but hopefully you don't need to ctrl-c out of something since signal handling is iffy at best. It is kind of nice to be able to import python directly on the command line and use it as python though...
Corrado 5y ago
Thank you for pointing out that jq can create JSON! I use jq all the time for working with the AWS CLI and a big pain point has always been sending JSON. If you don't know, the AWS CLI depends on JSON arguments for quite a few common tasks, and the JSON needed can be quite lengthy. Up until now I've been creating temp JSON files to feed the commands and I thought jo would be a great tool to make this easier. Now that I know jq can also create JSON, I'll just use that instead.